Mixed reality cookbook

What we perceive in the world is highly influenced by what we are looking for. That is old news. Now European researchers have used this theory to create a convincing and engaging 'mixed reality', and they have put together a cookbook so others can do it, too. That is new news.

Reality is WYSIWYG: What You See Is What You Get. But what you see depends largely on what you are looking for.

In a famous experiment, a group of volunteers observed a video of two teams, one dressed in black and one in white, passing a ball between them. The volunteers had to count the number of times the ball was passed directly from one player in black to another player in black. They performed the task excellently.

What they failed to notice was the man in the gorilla suit who walked on screen and jumped up and down during the game. It proved that what you see is strongly influenced by what you are looking for.

Non-linear fuzziness

In ophthalmology, researchers have found the eye does not see everything you perceive; neural processing fills in parts of the scene by inferring from those bits that are observed. In quantum physics, researchers discovered that particles change behaviour depending on whether you are looking at them or not.

In field after field researchers have discovered that perception is not linear; it is fuzzy; and it can be strongly influenced by carefully choosing the right cues.

The cues do not necessarily require complex technology. The Wii, a very popular gaming platform, abandoned the arms race of ever-more powerful processors and graphics cards and instead incorporated a simple motion sensor. Now users' gestures and reflexes drive the game, changing the pastime from a solitary, passive experience into an active, social one.

Those two additions, sociability and physicality, dramatically enhance the sense of experienced reality engendered by the game. This is very interesting. Up to now technologies, such as virtual and mixed reality, were thought by most to rely on more power, more technologically advanced interfaces, more animation and textures; but it now seems mixed reality is more powerfully and realistically evoked by combining perceptual dimensions with novel technologies in order to create a greater depth of experience.

“The greater the combination of senses engaged, the greater the chance of the user feeling immersed or present in the experience,” explains Rod McCall, a researcher at the Fraunhofer Institute and coordinator of the IPCity project.

In IPCity, a major EU-funded mixed reality project, researchers studied dozens of technologies to find those that dramatically enhance a user’s experience of a given task, all in an effort to increase citizens’ participation in civic life.

V-Ex and the city

Using virtual experiences (or V-Ex if you want) like this to bring citizens closer to the city, the project  embarked on what is probably the largest concerted effort, looking at the widest variety of mixed reality implementations, in recent times.

The project created applications for town planning, gaming, environmental awareness and storytelling. It enhanced engagement with the social, cultural and historical fabric of a city through location awareness and mapping, and it developed social storytelling rooted at locations within the streetscape.

Using a combination of easy-to-understand yet state-of-the-art technologies and location sensing, the researchers were able to create convincing cross-reality experiences by engaging multiple senses in parallel.From the lab to the real world

The project took perceptual and mixed reality research out of the lab and into the real world with a combination of large-scale field trials and longitudinal studies. As a result, the IPCity team has developed cookbook-like guidelines for creating mixed reality experiences.

Take Urban Renewal, an urban redesign application. Here, the researchers used a wide variety of media and interfaces to engage citizens in an exercise for redesigning an urban space.

IPCity’s Colour Table is a particularly innovative interface, using tokens to represent elements within a scene, such as buildings or other objects. An overhead camera projected the design table onto a wall, revealing changes as they developed from a bird’s eye view.

Another camera ‘interprets’ the tokens and projects virtual mock-ups onto a backdrop of the real site. Meanwhile on a screen, users can see how they have arranged the tokens, and on another they see how that would impact the real landscape.

The entire set-up, along with other tools, is part of a mobile tent that is transported to the actual location for the new building, so participants can visualise the real-world environment. The combination of these technologies, along with subtle audio streams, evokes a very convincing air of engagement in the task.

Token reality

“In the Urban Renewal showcase you have coloured tokens on the table and these represent buildings or another object in the space. So rather than having participants moving graphical objects on the screen you have them physically moving real objects on the table,” stresses McCall.

This physicality makes it easier for other people who are there to grasp what's going on - it starts a much richer discussion around the design through this physical relationship with it.

Another showcase by IPCity, CityWall in Helsinki, uses a very large multi-touch screen on a central street. “People can do whatever they want with content which is uploaded to flickr. But they appropriate the CityWall in hundreds of different ways. People play 'Pong' with the content, throwing it across the screen to one another, and it becomes [an] expressive space, far more perhaps than people showing pictures on their computer monitor,” McCall notes.

A sociable science

Social elements form a part of all the applications developed by IPCity, deliberately so. “We figured out pretty early on that a shared experience is much richer, that is why two people work together in our game, TimeWarp. It becomes a shared, new reality,” McCall explains.

Active, physical engagement, too, is important. McCall reveals that because it is impossible to cover a city entirely in virtual objects, the project developed activities that players need to complete, such as walking through a time portal or following a beer cart through the city. As non-players cannot see what the players are doing it often led to confused looks by passers-by in Cologne when it was demonstrated there.

In many respects, IPCity’s work represents the state of the art for mixed reality experiences and promises to offer a lot of food for thought to tourism, social gaming and mobile phone companies, among many others, including the performing arts. In fact IPCity’s work could be applied in some fashion to almost any area, to dramatically enhance the experience.

IPCity has applied the theory on perception in novel, compelling ways and its work will go on to enhance the theory further and lead to even more subtle and ingenious applications of V-Ex, in the city and elsewhere.

This is the first of a two-part special feature on IPCity.