The use of spoken language requires sophisticated perception and processing of speech sounds and sound patterns. It is still a matter of debate to what extent speech sound perception is based on special mechanisms unique to humans or on more general auditory processing mechanisms also present in other species. This article provides an overview of comparative research on the abilities of non-human animals (mammals and birds) to perceive and process speech sounds. It shows that phenomena like categorical perception of speech sounds, biases in vowel perception, context dependency of sound categorization, speaker normalization as well as speaker identification can all be found in several animal species. Also, like humans, animals can categorize speech sounds flexibly along different orthogonal dimensions (e.g., categorize speech sounds by word or by speaker sex) and are sensitive to prosodic patterns. While the perceptual sensitivities of humans and animals, as well as among different animal species, are not necessarily identical, they seem equivalent in their degree of sophistication. Although there is a need for more extensive and systematic comparative studies, human speech perception thus seems based on broadly shared sensitivities. These are likely to have arisen because all animals that use vocalizations to communicate need to be able to identify meaningful variations in sounds and sound patterns. Such sensitivities may have guided the evolution of speech sounds, rather than being a consequence of the evolution of spoken language.