When we talk with a friend in a noisy restaurant, we find ourselves watching the other person's mouth without even realizing it. Seeing the shape of the lips makes speech much easier to understand, even when the sound is partly drowned out. But what happens if the sound reaching your ears and the lip movements in front of your eyes don't match? Surprisingly, the brain doesn't pick one of the two; it lets you hear a completely different third sound.
This curious illusion is the McGurk effect. Put simply, the McGurk effect is a phenomenon in which, when the sound you hear and the lip movements you see don't match, the visual information changes the very way you hear speech, so that you hear a sound that was never actually made.
In this article, we will look in turn at how the McGurk effect was discovered by chance, how the third sound is created, the conditions that strengthen or weaken the effect, and the role this phenomenon plays in everyday life.
Speech Sounds Made by the Ears and Eyes Together
A Discovery That Began with a Video Editing Mistake
The McGurk effect was not something anyone set out to find.
(1) Research on How Babies Learn to Speak
Psychologist Harry McGurk and his research assistant John MacDonald were studying how babies come to understand speech sounds as they grow. For this research, they asked a technician to dub the sound of one syllable onto a video of a person saying a different syllable.
(2) A Strange Sound Both of Them Heard
When they played the finished video, both researchers heard a third sound that was neither the sound matching the lip movements in the video nor the sound that had been dubbed in. What looked like an editing mistake became the starting point for new research.
(3) The 1976 Paper
In 1976, the two published a paper titled “Hearing Lips and Seeing Voices” in the scientific journal Nature, introducing the phenomenon to the world. The illusion was later named after its discoverer.
How “Ba” and “Ga” Combine to Become “Da”
The best-known experiment dubs the sound “ba-ba” onto a video of lips saying “ga-ga.” Most people who watch this video say they heard “da-da.”
- Fusion: The sound “ba” and the lip movement “ga” blend together and are heard as “da,” which lies somewhere in between.
- Combination: Conversely, when the sound “ga” is paired with the lip movement “ba,” the two may be heard strung together, like “bga.”
(1) The Brain's Best Guess
When different information comes in from the eyes and the ears, the brain combines the two to produce the most plausible answer. Because sound and lip movements are merged at a very early stage of understanding speech, we experience that answer as if we had heard it directly with our ears. In this case, information from the eyes outweighs information from the ears, and a third sound is produced.
(2) An Illusion You Can't Escape Even When You Know It
Knowing how the McGurk effect works does not make it go away. Even researchers who have studied the phenomenon for more than 20 years say they still hear the third sound when they watch the video. Only when they close their eyes do they hear the original sound correctly.
Conditions That Strengthen or Weaken the Effect
The McGurk effect does not appear with the same strength in everyone; it becomes stronger or weaker depending on various conditions.
- Gaze: The effect appears wherever you look on the speaker's face, but it weakens when your gaze strays far from the mouth.
- Temporal Synchrony: The effect occurs even when the sound and lip movements are not perfectly aligned. The illusion still happens if the sound lags behind the lip movements by about 0.2 seconds, but if the sound comes before the lip movements, the illusion breaks down more easily.
- Audio Quality: The more blurry the sound is or the more it is buried in noise, the more you rely on lip movements, and the stronger the effect becomes.
- Familiarity: The effect weakens when the experiment uses the face of someone you know well.
- Attention: The effect drops sharply when your eyes are drawn to another object or when you focus on touching something with your hands.
Different Strengths in Different Languages
The effect is pronounced among speakers of English, German, Spanish, and other languages, but it appears relatively weaker among speakers of Japanese and Chinese. Researchers point to the sound structure of these languages and to cultures in which people avoid staring directly at another person's face.
The McGurk Effect in Everyday Life
The McGurk effect is not just a curious illusion in the laboratory; it reveals how we understand speech as a whole.
(1) Conversation in Noise
In a noisy place, we understand speech much better when we can see lip movements. The McGurk effect is evidence that we normally hear speech not only with our ears but also with our eyes.
(2) Language Development in Babies
Similar responses have been observed even in babies 4–5 months old, and the effect grows stronger as they get older. This means that while learning to speak, we learn sounds and lip movements together.
(3) Eyewitness Memory
A 2005 study showed that mismatched visual information can even change memories of words heard at a crime scene. In other words, testimony based on what was heard can be swayed by what the eyes saw.
The McGurk effect tells us that hearing is not the work of the ears alone. Our brain constantly weaves together information coming in from the eyes and ears, interpreting the world before we even realize it.
Knowing that what I heard is actually the product of my eyes and ears working together is the first step toward the wisdom of trusting my senses without blindly believing them.
