Stanford site advances science of turning 2-D images into 3-D models

January 23, 2008
Stanford site advances science of turning 2-D images into 3-D models

A three-dimensional 'fly around' image, above, was created from a two-dimensional image using an algorithm developed by Stanford computer scientists. Credit: Ashutosh Saxena

An artist might spend weeks fretting over questions of depth, scale and perspective in a landscape painting, but once it is done, what's left is a two-dimensional image with a fixed point of view. But the Make3d algorithm, developed by Stanford computer scientists, can take any two-dimensional image and create a three-dimensional "fly around" model of its content, giving viewers access to the scene's depth and a range of points of view.

"The algorithm uses a variety of visual cues that humans use for estimating the 3-D aspects of a scene," said Ashutosh Saxena, a doctoral student in computer science who developed the Make3d website with Andrew Ng, an assistant professor of computer science. "If we look at a grass field, we can see that the texture changes in a particular way as it becomes more distant."

The algorithm runs at http://make3d.stanford.edu .

The applications of extracting 3-D models from 2-D images, the researchers say, could range from enhanced pictures for online real estate sites to quickly creating environments for video games and improving the vision and dexterity of mobile robots as they navigate through the spatial world.

Extracting 3-D information from still images is an emerging class of technology. In the past, some researchers have synthesized 3-D models by analyzing multiple images of a scene. Others, including Ng and Saxena in 2005, have developed algorithms that infer depth from single images by combining assumptions about what must be ground or sky with simple cues such as vertical lines in the image that represent walls or trees. But Make3d creates accurate and smooth models about twice as often as competing approaches, Ng said, by abandoning limiting assumptions in favor of a new, deeper analysis of each image and the powerful artificial intelligence technique "machine learning."

Restoring the third dimension

To "teach" the algorithm about depth, orientation and position in 2-D images, the researchers fed it still images of campus scenes along with 3-D data of the same scenes gathered with laser scanners. The algorithm correlated the two sets together, eventually gaining a good idea of the trends and patterns associated with being near or far. For example, it learned that abrupt changes along edges correlate well with one object occluding another, and it saw that things that are far away can be just a little hazier and more bluish than things that are close.

To make these judgments, the algorithm breaks the image up into tiny planes called "superpixels," which are within the image and have very uniform color, brightness and other attributes. By looking at a superpixel in concert with its neighbors, analyzing changes such as gradations of texture, the algorithm makes a judgment about how far it is from the viewer and what its orientation in space is. Unlike some previous algorithms, the Stanford one can account for planes at any angle, not just horizontal or vertical. This allows it to create models for scenes that have planes at many orientations, such as the curved branches of trees or the slopes of mountains.

A paper on the algorithm by Ng, Saxena and a fellow student, Min Sun, won the best paper award at the 3-D recognition and reconstruction workshop at the International Conference on Computer Vision in Rio de Janeiro in October 2007.

On the Make3d website, the algorithm puts images uploaded by users into a processing queue and will send an e-mail when the model has been rendered. Users can then vote on whether the model looks good, and can see an alternative rendering and even tinker with the model to fix what might not have been rendered right the first time.

Photos can be uploaded directly or pulled into the site from the popular photo-sharing site Flickr.

Although the technology works better than any other has so far, Ng said, it is not perfect. The software is at its best with landscapes and scenery rather than close-ups of individual objects. Also, he and Saxena hope to improve it by introducing object recognition. The idea is that if the software can recognize a human form in a photo it can make more accurate distance judgments based on the size of the person in the photo.

For many panoramic scenes, there is still no substitute for being there. But when flat photos become 3-D, viewers can feel a little closer—or farther.

Source: By David Orenstein, Stanford University

4.5 /5 (22 votes)  

Filter


Move the slider to adjust rank threshold, so that you can hide some of the comments.


Display comments: newest first

freemind
Jan 23, 2008

Rank: 5 / 5 (1)
I've been amazed by youtube videos showing how it works. Follow the link in the article.
nilbud
Jan 23, 2008

Rank: 5 / 5 (2)
Must resist urge to submit the last supper.
gopher65
Jan 23, 2008

Rank: not rated yet
I uploaded a few and tested it out. Pretty cool, but, as they say on their site, it *is* still in development, and they have a long way to go. I think the best one I submitted was a picture of an Egyptian statue. That one worked ok:). This software freaks with photoshopped stuff though hehe.

This is an awesome idea and I'm glad someone is working on this technology:).
Ashibayai
Jan 24, 2008

Rank: not rated yet
If they take this and add the ability to compile data from multiple photos, we'll have an extremely impressive system for generating 3D models in everyday photos.
Rank 4.5 /5 (22 votes)
Tags

Relevant PhysicsForums posts

More news stories

Advanced power-grid model finds low-cost, low-carbon future in West

(PhysOrg.com) -- The least expensive way for the Western U.S. to reduce greenhouse gas emissions enough to help prevent the worst consequences of global warming is to replace coal with renewable and other ...

Technology / Energy & Green Tech

created 1 hour ago | popularity 5 / 5 (1) | comments 1 | with audio podcast

Small modular reactor design could be a 'SUPERSTAR'

(PhysOrg.com) -- Though most of today's nuclear reactors are cooled by water, we've long known that there are alternatives; in fact, the world's first nuclear-powered electricity in 1951 came from a reactor ...

Technology / Energy & Green Tech

created 1 hour ago | popularity 5 / 5 (3) | comments 3 | with audio podcast

Engineering images bring life to submerged city

(PhysOrg.com) -- Photo-realistic 3D mapping and digital reconstruction of an ancient underwater city in Greece have earned a team from the University of Sydney's Faculty of Engineering and Information Technologies ...

Technology / Engineering

created 36 minutes ago | popularity not rated yet | comments 0

New power source discovered

(PhysOrg.com) -- Researchers at the Massachusetts Institute of Technology (MIT) and RMIT University have made a breakthrough in energy storage and power generation.

Technology / Energy & Green Tech

created 36 minutes ago | popularity 5 / 5 (1) | comments 0

World's first 300mm-fab compatible directed self-assembly process line

At next week’s SPIE Advanced Lithography conference (San Jose, CA), imec announces the successful implementation of the world first 300mm fab-compatible Directed Self-Assembly (DSA) process line all-under-one-roof ...

Technology / Semiconductors

created 42 minutes ago | popularity not rated yet | comments 0


Mars Science Laboratory computer issue resolved

(PhysOrg.com) -- Engineers have found the root cause of a computer reset that occurred two months ago on NASA's Mars Science Laboratory and have determined how to correct it.

Clam fields found at deep, low-temperature Mariana vents

(PhysOrg.com) -- Scientists have marveled at the unusual life forms thriving at high temperature hydrothermal vents of the deep ocean.

Seeing colors in music, tasting flavors in shapes may happen in life's early months

Famed violinist Itzhak Perlman sees a deep forest green whenever he plays a B-flat on his Stradivarius' G string. The A on the E string is red.

Could Venus be shifting gear?

(PhysOrg.com) -- ESA’s Venus Express spacecraft has discovered that our cloud-covered neighbour spins a little slower than previously measured. Peering through the dense atmosphere in the infrared, the ...

The question of life in the ancient world

There’s a general feeling that we don’t get the Greeks – ancient or modern. Many, including heads of state like Angela Merkel, visibly shake their head in exasperation, rightly or wrongly, at ...

Study suggests girls can 'rewire' brains to ward off depression

(Medical Xpress) -- What if you could teach your brain to respond differently to things that make you feel sad, down or stressed out? What if doing that helped ward off depression?