Cut-and-paste simplicity for computer animation

Tools developed by European researchers bring cut-and-paste simplicity to gaming and animation. Users will be able to cut-and-paste complex elements like emotion, tone of voice and facial expression, making compelling new content, cheaply and quickly.

There is an explosion. Victims lie maimed or dead around the street. You are part of the emergency response and you have to quickly decide who needs urgent medical treatment. Breathing rate, skin tone, the wound severity and mental acuity are key indicators, and you quickly perform brief confirming tests as you move through the scene.

This is an animation, but it is much richer and more detailed than the current triage simulations available on the healthcare training market. What’s more, the simulation was developed more quickly and cheaply than the current state of the art.

It is also a clear demonstration of 21st century audiovisual animation, where recently developed European technology has brought the cut-and-paste simplicity of the internet to animation, audio-processing and semantic search for audiovisual elements.

Cut-and-paste simplicity

It is the work of the Salero project, a large continent-wide effort to streamline the production process and maximise content reuse in computer gaming and animation for online and broadcast entertainment.

Thirteen partners spent over €13 million (€8 million provided by the EU) to create two dozen applications, tools and showcases in an all-points efforts to dramatically improve audiovisual animation workflow. Audio and video transmission, processing and transformation are all affected by the broad range of tools and techniques developed by the project over the last four years.

The fundamental concept behind Salero was to create systems that would allow audio and video content to be easily and quickly repurposed for many different scenes within one project, and for transferring and adapting content between different projects.

In the long term, film and TV producers, animators, video game developers and performing arts groups will be able to mix and match digital elements simply. And they will be able to quickly and convincingly modify and adapt the content to suit their specific needs.

Synthetic emotions

The project focused on three main areas; audio processing, computer animation and semantic search. In each area, they produced tools that will enable existing media to be found and adapted to new content, either automatically or semi-automatically.

With the synthetic emotional model, for example, artists can animate a large emotional range by simply changing two variables, called activation and evaluation. Activation controls the strength of the emotion while evaluation controls the positive and negative balance between, say, a smile and a frown. Using just these two variables an animator is able to create dozens of expressions in a fraction of the time it takes currently.

Another tool, the Maskle, provides an easy way to transfer already defined emotions to a new face. In a first step, the animator selects ‘handles’ on the characters face, at the top, bottom and sides of the lips, for instance, and so defines basic expressions simply by moving these handles. The Maskle is then applied to a newly designed face and can help to specify a spectrum of emotions based on the defined basic expressions using the synthetic emotional model.

But Salero’s animation tools go way beyond facial expression. They work on body mechanics, too. The user can set a range of variables for a character, including his or her gender, ethnicity, age and weight, with each element modifying the appearance, according to Georg Thallinger, a researcher with Joanneum Research and coordinator of the Salero project.

“But these variables also impact the character’s gait and body movement. Using other tools we have developed, the animator can then simply point to an area where the character must go, and the software chooses the path and animates the character variables, and the quality of the surface along the route,” he tells ICT Results.

The required gait is determined automatically, so if a character must move across sand, or tarmac, body movement adapts appropriately.

Sophisticated semi-automation

This is an enormously sophisticated, semi-automated process, and it will save thousands of production man-hours. Tools like these, and others developed by Salero, will ultimately mean that state-of-the-art animation effects reach beyond big budget movie-making.

Audio processing, too, received a big boost in Salero. Here the team developed audio processing and synthesising tools to make voice recording cheaper and faster. Audio transformation, for example, can take a voice recording and change the gender, age, speed, timbre and pitch of the voice, easily adapting it from a strong, young woman to a weak old man.

Even the emotional stress of the voice can be adapted, from joyful to angry, by adjusting certain values. Computer-generated voices can be created and then transformed, too.

“It means you do not need as many voice actors, which is expensive, and it does not take as long to create a wide variety of characters expressing a broad range of emotions,” notes Thallinger.

Intelligent content

Work within the project even developed a system to identify emotional stress, so that the character animation can be perfectly synched with tone of voice. “It leads to a much more realistic and convincing animation when movement is synched across voice and video in this way,” Thallinger stresses.

Underlying all these efforts is a broad thrust to create intelligent content, which advertises its elements or component structure. For this task, Salero borrowed ideas from the ‘semantic web’, developing an ontology, or dictionary of specific terms, to describe audiovisual content.

It also developed archiving and retrieval tools and, particularly, a very clever annotating tool. “We knew that there is no way an animator is going to spend much time tagging content in a real work environment, there is just no time, so we used a statement-based annotating tool that quickly populates the content with relevant metadata by getting the animator to make specific statements about it.”

Excited studios

The work of the Salero project has generated a lot of interest. “In particular, studios are very excited about the audio and animating tools,” reveals Thallinger. “For the annotating work, they were less enthusiastic, mainly because I think they are not familiar with it. But towards the end of the project, audiovisual companies started expressing a lot more interest about semantic technologies, because I think the concepts around that field start to get known also in industry.”

Indeed, over time the annotating tools promise to deliver the greatest impact to the industry, allowing producers to rapidly create content from an ever-expanding library of earlier work. It could lead to stock animation galleries, in the same manner that stock photo galleries exist now.

In the meantime, Salero has permitted studios to create compelling, Hollywood-quality content quickly, efficiently and cost-effectively.

This is the first of a two-part special feature on Salero.