Seeing two figures in coordinated action helps brain pick out movements of one

September 7, 2006

Researchers used colored dots to represent the movement of dancers (above) and fighters in a pair of videos illustrating visual discrimination.

A new study by vision scientists at the University of California, Berkeley, finds that the human visual system is better able to discriminate the movements of a single person when his or her actions are coordinated in a meaningful way with a second individual.

When shown a pair of figures in motion, for instance, the brain is better able to pick out an individual if it perceives that one person is throwing a punch while another is making a defensive block, the researchers said.

This is especially important when the view is somehow obscured or impaired, according to the study, published in the September issue of the journal Nature Neuroscience.

"Our study reveals a greater degree of complexity in human visual processing than previously understood," said Dennis Levi, UC Berkeley dean of optometry and principal investigator of the study. "When we watch two people interacting, knowing what one is doing helps us understand the actions of the other. We think of it as 'It takes two to Tango.'"

Other researchers on the team are Peter Neri, a UC Berkeley post-doctoral fellow in Levi's laboratory, and Jennifer Luu, an undergraduate who joined the group as part of the UC Berkeley Undergraduate Research Apprentice Program. Luu is now a UC Berkeley Ph.D. student in bioengineering.

This research provides insight into how accurately we can interpret what we see from grainy security cameras, particularly when identifying whether a crime is taking place. There is even research taking place on the development of artificial intelligence computer programs that can automatically detect which actions are suspicious.

At its most fundamental level, the human brain's ability to interpret and react to the action it sees within a fraction of a second is a matter of life and death, such as identifying that a tiger lunging forward with teeth bared is a threat and then deciding how best to get away.

But this skill - when not impaired by such disorders as autism - is also essential to our success as a social species, allowing us to determine whether someone is happy, sad or nervous based upon visual cues both subtle and obvious.

"Our study is the first to provide quantitative evidence that this ability to interpret action enhances the processing of visual discrimination," said Neri, the lead author.

To build their dataset of human actions, the researchers outfitted members of the UC Martial Arts Program and the UC Ballroom Dance Team with sports suits that each contained 13 lights marking key body parts, such as the head, hands, knees and feet. The "actors" were then asked to fight or dance in a dark room while the researchers filmed the movements.

Using motion capture technology, the lights in the video were converted into dots that the researchers could manipulate. The original samples of each action - fighting or dancing - lasted 22 seconds each. In the synchronized video files, figures' actions corresponded naturally with each other.

The researchers also created desynchronized files in which the movements of one figure filmed in the first 11 seconds were paired with those for the other figure in the second 11 seconds. In these files, the movements of the two figures were no longer coordinated with each other.

Four UC Berkeley undergraduate students with normal vision who had been recruited for the study were then shown 1.5- to 3-second sequences that were randomly selected from either the synchronized or desynchronized files. In half of the sequences, the researchers had further scrambled the dots of one figure and added many other dots to the sequence to mask the action. The researchers were able to vary the amount of dots to measure participants' tolerance for "noise" or "fog."

The participants had to determine whether the dots they were viewing represented one or two figures.

"We found that people can tolerate more masking dots and answer correctly when the figures' actions are interacting in a meaningful way," said Neri, who also participated in the study. "If the sequence is desynchronized, we can tolerate 10 dots. When viewing synchronized action, we can tolerate a minimum of 20 and up to 100 dots. Having the action coordinated improves the performance by at least a factor of two."

The researchers noted that the relationship between two figures held true whether the movements were coordinated but antagonistic, as in the fighting sequences, or coordinated and cooperative, as in the dancing movements.

"This study shows that the cortex encodes complex information about actions, and that there can be a clear evolutionary advantage - improved visual discrimination - to doing so," said Neri.

The researchers pointed out that the human brain is constantly adapting to limited amounts of visual information, enabling us to imagine a full picture of a tiger, for instance, even if trees and bushes obscure parts of its body.

The brain also allows us to understand when a group of dots or lines represents a human being. When presented with a group of dots in motion, the human visual system can distinguish whether the dots comprise a male or a female, and even judge the figure's emotional state, the researchers said.

"We used to talk about how the eye is a camera that passes information intact to the brain, but that's actually not how we see," said Levi. "Things are always changing before our eyes, and our brain is constantly making best guesses about what it's seeing."

When the view is somehow obscured, that task naturally becomes more difficult. This guessing game played by the visual system is also why the brain is susceptible to optical illusions, Levi added. This best-guess approach to interpreting what we see is also an important factor when considering eyewitness accounts of events, the researchers said.

Source: UC Berkeley, By Sarah Yang