Machine Learning by Watching and Listening
October 5, 2009(PhysOrg.com) -- To expand the boundaries of machine intelligence, Ben Taskar is using television shows with large fan bases like CSI, Alias, and Lost to teach computers how to be smarter about what they see, hear and read.
Ben Taskar is teaching computers how to watch television. Not, as you may think, because they need to relax after reading all that code, but because through this research, Taskar, the Magerman Term Assistant Professor in the Department of Computer and Information Science, is taking machine learning to the next level. Using novel learning algorithms combining video, sound and text streams, his team has shown that computers can be taught to associate what is in a video clip with existing descriptions of characters and actions and then infer information about new material and categorize it according to what it has already learned.
Currently, to categorize videos, photos and other electronic media, computers are “told” through assigned tags the contents of an image. Even new “self-tagging” technologies rely on existing labels to tag new media as it is saved. This is not at all similar to how a human learns and infers. For example, when we watch an episode of our favorite show and we hear one character say to the other, “Joe is coming over soon,” we are able to infer when a new character arrives that his name is Joe. We do not need an explicit label of “Joe” over his face or a subtitle “Joe” at the bottom of the screen.
To expand the boundaries of machine intelligence, Taskar and a team comprised of graduate students Timothee Cour and Benjamin Sapp, along with undergraduate Chris Jordan, are using television shows with large fan bases like CSI, Alias, and Lost to teach computers how to be smarter about what they see, hear and read. Take, for example, the show Lost. Hundreds of thousands of viewers enjoy spending hours of their time writing and posting scripts of episodes on fan sites, video clips on YouTube, and information in discussion boards. Taskar is taking this collective “wisdom of the crowds” and entering the massive quantities of digitized knowledge and the associated scenes and clips into computers.
From there, computers are given specialized algorithms to be able to combine the information with the video and “learn” which person is which character, what each character is doing, and with whom. At no time does anyone in the research team tag anything. This is known as “unsupervised” or “weakly” supervised learning.
Once this learning has taken place, researchers can ask the computer, “show all scenes where Kate is talking to Jack,” or “produce a montage of all scenes with swimming,” and the computer will generate the sequence. By checking on what is produced, the team then looks for patterns containing errors that suggest the algorithms and models need fine-tuning. Once the algorithm is perfected, the computer can then watch new material and add to the already known information, using its past learning to amass more knowledge.
As you can imagine, using algorithms to teach a computer to learn the nuances of language and parts of speech in written data, along with different camera angles, lighting and other filming conditions, is a daunting task. Taskar compares it to how children learn about their environments. At first, a young child may call all moving vehicles with four wheels “cars,” and later learns to distinguish “trucks” or “vans” from the group. Similarly, computers are given simpler distinctions and tasks at the beginning of learning and more and more complicated ones as patterns to “teach” are better identified.
Future applications of this research go far beyond the “cool” factor of being able to get a computer to show all the scenes in which a favorite character appears. Two areas that will likely benefit are general image and audio search. In order to develop more accurate technologies that can robustly recognize and correctly analyze immense collections of images, videos and spoken language, computers will need to learn to identify hundreds of thousands of different concepts. By tapping into contributions of millions of people on the web and burgeoning data from multiple modalities, the research of Ben and his team will push the field of machine learning towards unsupervised techniques to make computers learn about our complex world.
Computers can only take us so far. We still can’t figure out where those Lost writers are going with that island.
Provided by University of Pennsylvania (news : web) Original story can be found here.
-
Researchers teach computers to perceive three dimensions in 2-D images
Jun 13, 2006 |
not rated yet |
0
-
The New Science of Learning
Sep 11, 2009 |
not rated yet |
0
-
Computers learn art appreciation
Nov 05, 2007 |
not rated yet |
0
-
New algorithm found for learning languages
Sep 06, 2005 |
not rated yet |
0
-
'Rich interaction' may make computers a partner, not a product
Aug 19, 2009 |
not rated yet |
0
-
Engineers build first sub-10-nm carbon nanotube transistor
Feb 01, 2012 |
4.9 / 5 (30) |
30
-
Something old, something new: Evolution and the structural divergence of duplicate genes
Jan 31, 2012 |
4.6 / 5 (7) |
1
-
The hidden nanoworld of ice crystals: Revealing the dynamic behavior of quasi-liquid layers
Jan 30, 2012 |
5 / 5 (3) |
1
-
Stock market network reveals investor clustering
Jan 27, 2012 |
3.9 / 5 (23) |
8
-
Of microchemistry and molecules: Electronic microfluidic device synthesizes biocompatible probes
Jan 26, 2012 |
5 / 5 (1) |
0
-
Synergistic relations between computer science and technology.
Feb 06, 2012
-
how do iphone gloves work?
Feb 05, 2012
-
iPhone battery over time
Jan 30, 2012
-
Best alternate Tablet to an iPad for writing math or physics equations?
Jan 26, 2012
-
Sending SMS to a website
Jan 20, 2012
-
Need help with my technical fest!
Jan 19, 2012
- More from Physics Forums - Computing & Technology
More news stories
First Google hire leaving for online academy
The first person hired by Google's founders is leaving the Internet giant to devote himself to an innovative online education website called Khan Academy.
2 hours ago |
not rated yet |
0
FBI file: Steve Jobs was considered for govt post
(AP) -- FBI background interviews of some people who knew Apple co-founder Steve Jobs reveal a man driven by power and alienating some of the people who worked with him.
2 hours ago |
2.3 / 5 (3) |
0
NY attorney general ends lawsuit against Intel
(AP) -- Intel Corp. is paying $6.5 million as part of a deal to terminate an antitrust lawsuit filed against the chip maker by the New York attorney general's office.
2 hours ago |
not rated yet |
0
LinkedIn's 4Q earnings strong, revenue doubles
(AP) -- LinkedIn reported a strong fourth quarter as the online professional-networking service added 14 million members. Its net income and revenue beat Wall Street's expectations.
2 hours ago |
not rated yet |
0
New integrated building model may improve fish farming operations
Today's "locavore" movement with its emphasis on eating more locally-produced food is a natural fit for fruits and vegetables in nearly every region, but few entrepreneurs have dared to apply the concept to ...
2 hours ago |
not rated yet |
0
'Dark plasmons' transmit energy
Microscopic channels of gold nanoparticles have the ability to transmit electromagnetic energy that starts as light and propagates via "dark plasmons," according to researchers at Rice University.
Anyone can learn to be more inventive, cognitive researcher says
There will always be a wild and unpredictable quality to creativity and invention, says Anthony McCaffrey, a cognitive psychology researcher at the University of Massachusetts Amherst, because an "Aha moment" is rare and ...
Ultraviolet protection molecule in plants yields its secrets
Lying around in the sun all day is hazardous not just for humans but also for plants, which have no means of escape. Ultraviolet (UV) radiation from the sun can damage proteins and DNA inside cells, leading ...
New method makes culture of complex tissue possible in any lab
Scientists at the University of California, San Diego have developed a new method for making scaffolds for culturing tissue in three-dimensional arrangements that mimic those in the body. This advance, published online in ...
Cell biologists describes mechanism by which some people may be more susceptible to colon cancer
An international research team led by cell biologists at the University of California, Riverside has uncovered a new insight into colon cancer, the third leading cause of cancer-related deaths in the United ...
Hydrogen from acidic water: Researchers develop potential low cost alternative to platinum for splitting water
A technique for creating a new molecule that structurally and chemically replicates the active part of the widely used industrial catalyst molybdenite has been developed by researchers with the Lawrence Berkeley ...