Seeing While Hearing Speeds Brain's Processing of Speech

David Poeppel, associate professor of linguistics at Maryland and senior author of the study, says the study indicates that when a listener can see the speaker's mouth, the listener's brain predicts what sound is about to be heard, a process called predictive coding.

"Moving the mouth comes before the sound," Poeppel said. "The brain uses the slightly preceding visual information to make a prediction, almost instantaneously, of what the sound will be."

That combination of visual and auditory speech, says Poeppel, "gives you the information to get to recognition faster and more accurately" than is done by hearing alone.

Poeppel, along with his Ph.D. student Virginie van Wassenhove and Ken W. Grant, of the Walter Reed Army Medical Center, arrived at their findings by doing an EEG (Electroencephalographic) study of 26 participants, all native American English speakers.

Their brain activity was measured while they listened to just auditory speech and again while they watched video in which a woman spoke "pa," "ta" and "ka," common English syllables.

Not only did the brain process the sounds faster when visual and auditory speech were combined, it took "less effort" to reach recognition earlier.

"This discovery contradicts the commonly held notion that audio and visual speech together are more than the sum of their parts," Poeppel said. "It actually takes 'less' brain activity to process the information and do it in less time."

The study also gives the first neurological evidence to support the Analysis-by-Synthesis model of speech processing, Morris Halle's and Ken Stevens's 1950's model based on the notion of predictive coding.

"This is the first time to connect audiovisual speech to the theory," says Poeppel. "We're increasingly learning about the importance of multi-sensory integration."

The study was supported by a grant from the National Institutes of Health.

Source: University of Maryland