Does What You See Alter What You Hear?
Does what you see alter what you hear? A new study published by Hans Rutger Bosker of the Max Planck Institute for Psycholinguistics and Radboud University and David Peeters from Tilburg University set out to answer that question.
When you watch a politician speak you may notice them make a hand gesture when they say important words.
Bosker and Peeters wanted to discover how listeners use different streams of information, such as body language and facial expressions when they listen to someone speak.
Famously The McGurk effect has discovered that we hear different sounds when they’re combined with certain lip movements.
The researchers wanted to find out if there was a similar effect with hand gestures. They chose Dutch words that differed only by stress patterns.
Where it’s only the emphasis on specific syllables that separates the words.
For example, “PLAto” refers to the Greek philosopher, while putting emphasis on the second syllable plaTO, refers to the word plateau. Bosker then used hand motions coupled with an ambiguous emphasis on syllables to find out whether the hand gestures made a difference.
Concluding that “Listeners listen not only with their ears but also with their eyes.” When beat gestures were made at the same time as a syllable, it changed what people heard.
Bosker explains “Our findings also have the potential to enrich human-computer interaction and improve multimodal speech recognition systems.
It seems clear that such systems should take into account more than just speech.” The team are now looking to virtual reality to discover whether what we hear is also determined by other gestures, like head nods and eyebrow movements.
