Fermilab and Caltech successfully used UltraScience Net, achieved 7 Gigabits per second

September 1, 2005

Preparing for an onslaught of data to be processed and distributed in the upcoming years, scientists at the Department of Energy's Fermi National Accelerator Laboratory and at the California Institute of Technology successfully tested a new ultrafast data transfer connection developed by the Office of Science of the Department of Energy.

The new DOE UltraScience Net, which allows for the development and testing of dynamically switched fiber-optic links with speeds of up to 20 Gigabits per second, connects cities such as Atlanta, Chicago, Seattle, and Sunnyvale, CA. In a recent test, the transfer of data from a system at Fermilab to a system at Caltech achieved a rate of 7 Gigabits per second, over 15 times faster than with a comparable production Energy Science Network, and thousands of times faster than the usual high-speed Internet connections at private residences.

"With the UltraScience Net, the Department of Energy is at the leading edge of high-speed and high-capacity network research for science applications in the country," said Fermilab's Matt Crawford, who led the effort of connecting Fermilab to the Chicago node of the UltraScience Net. "Fermilab is among the first users of the UltraScience Net."

The UltraScience Net was developed under the leadership of scientists at DOE's Oak Ridge National Laboratory.

"About two years ago, we looked at various needs for high-speed networking capabilities within the Department of Energy's science program," said ORNL's Nageswara Rao, one of the principal investigators of the UltraScience Net. "From terascale supernova simulations to visualization of genomics data to data from particle collisions, there is a demand for stable, dedicated, high-bandwidth connections to exchange data. With the UltraScience Net, research institutions can book a connection, and nobody can interfere with your circuit while you transfer data at high speed."

Institutions wanting to use the UltraScience Net are responsible for developing and establishing the links from their sites to the nodes of the UltraScience Net, known as the "last-mile problem." During the early stage of the UltraScience Net, scientists had to reserve a connection using a Web-based system, and network specialists on both ends of the connection had to manually establish the link at the corresponding time. Since then Fermilab's network computing experts have developed an automated system called Lambda Station. As its first application, the system automatically can connect computers across Fermilab's local network with the UltraScience Net when a connection is available.

"The DOE UltraScience Net enables us to think cleverly about networks. We thought of a way to use these ultraspeed channels when they are available, and do without them when they are not," said Fermilab's Don Petravick, principal investigator of the Lambda Station project. "The Lambda Station system automatically coordinates the local networks and applications, dynamically assigning network channels using both shared networks and dedicated pipelines. It is a clever package we developed using the wonderful infrastructure provided by the UltraScience Net."

The successful test of the UltraScience Net and the Lambda Station system is an important milestone for US particle physicists involved in experiments at the Large Hadron Collider, currently under construction at the European laboratory CERN, in Switzerland. In 2007, when the LHC begins producing data, Fermilab will assume the role of a Tier 1 computing center for the CMS experiment at the LHC. As part of its Tier 1 responsibilities, Fermilab will distribute vast amounts of data to universities and other CMS institutions (Tier 2 centers) for analysis.

"The LHC experiments produce large amounts of data, and there is a demand for large data flows among the participating institutions," said US-CMS physicist, Collaboration Board chair and networking expert Harvey Newman of Caltech, who helped to develop and test the infrastructure of the Lambda Station project. "The UltraScience Net and the Lambda Station infrastructure are key technologies for supporting LHC science. Ultimately, we plan to make the Lambda Station infrastructure available to everybody in the science community who has large data-transfer needs and who has access to high-bandwidth networks."

Further information is available at:
UltraScience Net: http://www.csm.ornl.gov/ultranet/
Lambda Station: http://www.lambdastation.org
CMS experiment at the LHC: http://cmsdoc.cern.ch/cms/outreach/html/