Researchers use Web images to add realism to edited photos

July 10, 2007

Computer graphics researchers at Carnegie Mellon University have developed systems for editing or altering photographs using segments of the millions of images available on the Web.

Whether adding people or objects to a photo, or filling holes in an edited photo, the systems automatically find images that match the context of the original photo so they blend realistically. Unlike traditional photo editing, these results can be achieved rapidly by users with minimal skills.

“We are able to leverage the huge amounts of visual information available on the Internet to find images that make the best fit,” said Alexei A. Efros, assistant professor of computer science and robotics. “It’s not applicable for all photo editing, such as when an image of a specific object or person is added to a photo. But it’s good enough in many cases,” he added. “Why Photoshop if you can ‘photoswap’ instead?”

Efros and his colleagues will present papers on two related systems at the Association for Computing Machinery’s Special Interest Group on Graphics and Interactive Techniques (SIGGRAPH) annual conference Aug. 5–9 in San Diego.

One system, called Photo Clip Art (http://graphics.cs … hotoclipart/), was developed with graduate students Jean-François Lalonde and Derek Hoiem, and with Carsten Rother, John Winn and Antonio Criminisi of Microsoft Research Cambridge. It uses thousands of labeled images from a Web site called LabelMe as clip art that can be added to photos. A photo showing a vacant street, for instance, might be populated with images of people, vehicles and even parking meters derived from the LabelMe database (http://labelme.csail.mit.edu/).

To make the resulting image appear as realistic as possible, the system analyzes the original photo to estimate the camera angle and lighting conditions, and then looks in the clip art library for an object — a car, for instance — that matches those criteria. The user need only identify the horizon in the original photo to orient the system. Using previously developed Carnegie Mellon technology for analyzing the geometric context of a photo, the system can then place the object within the scene, adjusting its size as necessary to put it in proportion to other objects of equal distance from the camera.

“Matching an object with the original photo and placing that object within the 3-D landscape of the photo is a complex problem,” said Lalonde, who led development of the system. “But with our approach, and a lot of clip art data, we can hide the complexity from the user and make the process simple and intuitive.”

The other system, called Scene Completion (http://graphics.cs … -completion/), was developed by graduate student James Hays, another member of Efros’ research team. It draws upon millions of photos from the Flickr Web site to fill in holes in photos. Some of the holes might be from damage to a physical photograph, but more often they are created when an editor cuts out part of an image to eliminate an unsightly truck from a picturesque street scene, or removing a passerby from a group shot of friends. Photo editors often try to fill in those holes with sections derived elsewhere in the same image, but Efros said that a better match can often be found in a different photo.

The system looks for image segments that match the colors and textures that surround the hole on the original photo. It also looks for image segments that make sense contextually — in other words, it wouldn’t put an elephant in a suburban backyard or a boat in a desert.

In the case of well-photographed cities or popular tourist attractions, Efros said, the system might get lucky and find a photo of the same scene on the Web. In other cases, it might offer a number of possible images that could fill in the hole. A retaining wall edited out of one photo, for instance, might be replaced by the image of a building, a grassy slope or a rock outcropping. The system typically gives the user 20 different choices for filling in the hole.

The success of this approach depends on the number of photos available to the system, Hays said. “We saw a dramatic improvement when we moved from a database of 10,000 images to two million images,” he noted. “And that is just a tiny fraction of the hundreds of millions of images already available on sites like Picasa and Flickr. We have tons of photos from which to choose.”

Source: Carnegie Mellon University

3.9 /5 (8 votes)  

Filter


Move the slider to adjust rank threshold, so that you can hide some of the comments.


Display comments: newest first

kellimaier
Nov 01, 2007

Rank: not rated yet
These photos ( those on Flckr...can't speak for the others, since I am unfamiliar with that service/site) are copyright protected...seems a sticky situation to determine the permissions for each photo...or are they just simply not looking at that aspect? In that case they are breaking the law?
Rank 3.9 /5 (8 votes)
Tags

Related Stories
Relevant PhysicsForums posts

More news stories

Google users warned of threat to smartphone wallets

Users of Google smartphone wallets were being warned on Friday that there is a way to crack pass codes intended to thwart thieves from going on illicit shopping sprees.

Technology / Internet

created 1 hour ago | popularity 5 / 5 (1) | comments 0

Anonymous knocks CIA website offline (Update)

The website of the Central Intelligence Agency was inaccessible on Friday after the hacker group Anonymous claimed to have knocked it offline.

Technology / Internet

created 2 hours ago | popularity 5 / 5 (4) | comments 8

New error-correcting codes guarantee the fastest possible rate of data transmission

Error-correcting codes are one of the triumphs of the digital age. They’re a way of encoding information so that it can be transmitted across a communication channel — such as an optical fiber o ...

Technology / Computer Sciences

created 11 hours ago | popularity 5 / 5 (4) | comments 5 | with audio podcast

New power source discovered

(PhysOrg.com) -- Researchers at the Massachusetts Institute of Technology (MIT) and RMIT University have made a breakthrough in energy storage and power generation.

Technology / Energy & Green Tech

created 10 hours ago | popularity 4.8 / 5 (16) | comments 5 | with audio podcast

Small modular reactor design could be a 'SUPERSTAR'

(PhysOrg.com) -- Though most of today's nuclear reactors are cooled by water, we've long known that there are alternatives; in fact, the world's first nuclear-powered electricity in 1951 came from a reactor ...

Technology / Energy & Green Tech

created 10 hours ago | popularity 4.2 / 5 (10) | comments 19 | with audio podcast


Complex wiring of the nervous system may rely on a just a handful of genes and proteins

Researchers at the Salk Institute have discovered a startling feature of early brain development that helps to explain how complex neuron wiring patterns are programmed using just a handful of critical genes. ...

NASA sees wide-eyed cyclone Jasmine

Cyclone Jasmine's eye has opened wider on NASA satellite imagery, as it moves through the Southern Pacific Ocean.

NASA sees Giovanna reach cyclone strength, threaten Madagascar

Tropical Storm 12S built up steam and became a cyclone on February 10, 2012 as NASA's Terra satellite passed overhead. Residents of east-central Madagascar should prepare for this cyclone to make landfall ...

Putting the squeeze on planets outside our solar system

(PhysOrg.com) -- Using high-powered lasers, scientists at Lawrence Livermore National Laboratory and collaborators discovered that molten magnesium silicate undergoes a phase change in the liquid state, abruptly ...

The power of estrogen -- male snakes attract other males

A new study has shown that boosting the estrogen levels of male garter snakes causes them to secrete the same pheromones that females use to attract suitors, and turned the males into just about the sexiest ...

Grass to gas: Researchers' genome map speeds biofuel development

Researchers at the University of Georgia have taken a major step in the ongoing effort to find sources of cleaner, renewable energy by mapping the genomes of two originator cells of Miscanthus x giganteus, a large perenn ...