High energy physics team captures network prize

December 7, 2005

Researchers from the U.S. Department of Energy's Fermi National Accelerator Laboratory and Stanford Linear Accelerator Center recently joined an international team in shattering the world network speed record. To capture first place in the SC|05 Bandwidth Challenge, the team of high energy physicists, computer scientists and network engineers led by the California Institute of Technology transferred physics data at a rate of over 150 gigabits per second--equivalent to downloading over 130 DVD movies in one minute.

The Bandwidth Challenge was held in Seattle November 12-18, 2005 at SC|05, the International Conference for High Performance Computing, Networking, Storage and Analysis. The contest challenged scientists and networking engineers to create the best and most advanced techniques for using vast amounts of data and showcasing it on advanced networks. Bandwidth Challenge entries were evaluated based on their ability to support the mission of science and to maintain sustained data throughput.

In preparation for its third consecutive win, the Caltech-led team transferred 475 terabytes of high energy physics data in 24 hours and sustained average data rates of more than 100 gigabits per second for several hours. Data from particle physics experiments were transferred from laboratories and universities worldwide, routed through the SC|05 exhibit hall in Seattle, and stored either in the hall or at other institutions around the globe. The official peak throughput was 131.6 gigabits per second, measured during the Challenge on 17 of the 22 optical fiber links used by the team.

"This is much more than measuring how quickly we can send bits," said Fermilab's Don Petravick. "Our Bandwidth Challenge entry used applications that transport actual physics data at high rates, essential to the high energy physics experiments that we support. We brought together people and resources from universities and laboratories on four continents, demonstrating the value of collaboration and our ability to achieve records while transferring data from running experiments."

SLAC transferred data from its BaBar experiment, and Fermilab transferred data from its CDF and DZero experiments, the CMS experiment at the CERN laboratory in Switzerland, and the Sloan Digital Sky Survey through seven 10-gigabit optical links in the Fermilab/SLAC booth at the SC|05 exhibit hall. Software and systems developed at the two laboratories aided the team's high-speed disk transfers, some of which took place over transoceanic network links.

"Demonstrating sustained application throughputs of well over 100 gigabits per second for real physics data over international links will help open the doors for the next generation of data-intensive science," said Les Cottrell, assistant director of Scientific Computing and Computer Services at SLAC. "We learned valuable lessons about the current limitations, bottlenecks and instabilities in today's high-performance components that can be applied to improving management, monitoring and diagnosis and identifying future requirements."

The next generation of data-intensive science will arrive in 2007, when the world's highest-energy particle accelerator, the Large Hadron Collider, begins operating at CERN in Geneva, Switzerland. The Bandwidth Challenge entry was part of the preparation for the torrent of data that will come from the LHC experiments. Thousands of physicists around the globe will need to access and analyze LHC data in their quest to solve the mysteries of the universe. The processing, distribution and analysis of the data will be completed using high-speed optical networks, software to monitor and manage the data flows across the networks, and grid computing.

"This demonstration allowed us to preview the globally distributed grid system of more than 100 laboratory- and university-based computing facilities that is now being developed in the U.S., Europe, Asia and Latin America in preparation for the next generation of high energy physics experiments at the LHC," said Caltech's Harvey Newman, head of the Bandwidth Challenge team.

To achieve extraordinary rates of data transfer in the Bandwidth Challenge, the team used technology developed by SLAC and Fermilab researchers.

As high energy physics data is made up of many event files, accessing all the necessary events from disk every time they are requested takes longer than scientists would like. The PetaCache, a cluster of machines with large amounts of memory developed at SLAC, ships high energy physics data 10 times faster than from disk. SLAC data were transferred using the bbcp and xrootd applications developed at SLAC and INFN-Padova in Italy.

Data from Fermilab were transferred using the dCache data transport application developed by Fermilab and the DESY laboratory in Germany. Lambda Station, also developed at Fermilab, was exhibited as a means to route high-volume data flows between the high-speed optical networks and their ultimate endpoints.

The winning Bandwidth Challenge team also included researchers from CERN in Switzerland; the University of Manchester in the UK; the Universities of Michigan and Florida, Vanderbilt University and Brookhaven National Laboratory in the U.S.; the University of Sгo Paulo, Sгo Paulo State University and Rio de Janeiro State University in Brazil; Kyungpook National University and the Korea Institute of Science and Technology Information in Korea; and the KEK Laboratory in Japan.

Source: Fermi National Accelerator Laboratory