Giants Draft Storage System to Keep an Eye on the Game

With Barry Bonds on the verge of making baseball history, the S.F. Giants are bringing state-of-the-art video capabilities to AT&T Park, making it one of the world's most well-equipped high-tech sports venues.

The sporting eyes of the world will be on AT&T Park this baseball season. Five years after playing in the World Series against the Los Angeles Angels, the resident San Francisco Giants reworked their lineup during the off-season to make another run at the National League pennant, including signing starting pitcher Barry Zito to a whopping $126 million, seven-year contract. In addition, AT&T Park will host Major League Baseball's All-Star Game July 10.

At the center of attention will be outfielder Barry Bonds and his quest to overtake one of the most hallowed records in sports: Hank Aaron's career home-run mark of 755. As of press time, the 42-year-old Bonds was within 20 "dingers" of tying Aaron's mark.

Watching it all very carefully will be not only the media worldwide and tens of thousands of fans at each game but also the ballpark itself, which is one of the most well-equipped high-tech sports venues in the world. Every minute of every game is videotaped and archived from the perspective of five high-definition cameras, and all that production work is handled on-site in a state-of-the-art video editing and production facility, located high above home plate.

The club also has a second video editing room located inside the players' clubhouse. Players can use the video room anytime they desire for scouting upcoming pitchers, reviewing some of their previous performances or comparing their defensive play with that of players from opposing teams.

And all that digital data - not only video but also the club's daily e-mail and archives, ticketing database, customer data, scouting database, retail database, organizational accounting data, sponsorship project data, and day-of-game employee scheduling - is handled by Data Domain's DD430 system, alongside a couple of Nexsan Technologies arrays. The DD430 has eight 400GB drives configured as RAID 6 (six data drives, plus two parity drives). Counting all eight drives, one DD430 unit provides 3.2TB of raw capacity.

Last year, the Giants decided to update and centralize their disparate IT systems, centering on the storage servers from Data Domain and Nexsan Technologies. The servers would be used as "home base," with the Giants' IT personnel redirecting the team's sophisticated video system into a "hub" on those servers. The club's business systems, Web site, on-site operations, Wi-Fi LAN and retail operation also were added. In October, the existing Hewlett-Packard application server and tape backups were taken offline, and the Data Domain-Nexsan arrays and storage servers were switched on. All the club's data is backed up nightly and sent via a T-1 line to the backup Data Domain-Nexsan storage servers in its spring training facility in Scottsdale, Ariz.

The Giants had been planning this consolidation for about a year. They looked at several other vendors and tested a few, but none of them could offer the scalability and pricing that Data Domain did. They declined to discuss the cost of the system.

The team has two DD430s: one at the ballpark in San Francisco, and one in Scottsdale. The Scottsdale unit is used for backup and disaster recovery during the regular season; in the spring, the players and view crew also use it for storing developmental video.

AT&T Park - originally known as Pac Bell Park, then SBC Park for two seasons - has undergone some ambitious upgrades since its own Opening Day in April 2000. Improvements include a huge - 31.5-by-103-foot - high-definition video screen overlooking center field, a full-scale LAN that extends into the surrounding neighborhood, 275 flat-screen TV monitors for the corridors and Stadium Club level, and a large number of IT infrastructure enhancements.

Only a year ago, the team was still storing data on DVDs and 8-inch floppy disks, the latter of which were introduced during the Reagan administration. However, since last fall, the new Data Domain and Nexsan servers handle the storage for about 15TB of video for the Giants, plus all the team's business data. The Giants are able to easily transport - via a single T-1 line - any amount of digital video footage on demand from the stadium to the Arizona training facility.

"We're the first team to go digital with all our video," said Giants' Network Engineer Dave Woolley. "We used to have a 750-slot DVD jukebox ... which we sold on Craigslist awhile ago. Somebody must have it in their living room by now. We stored 4TB of video on that box. Over the years, the system has evolved, and now we have a more traditional storage array system.

The NFL's Baltimore Ravens are scoring points on the bottom line - and improving security - by outsourcing all of their data storage and management services. Click here to read more.

"The video quality has improved; the compression technology has improved. Now we log a game - during the game - and associate metadata with each video - clip - , such as who was up at bat, what was the outcome, what was the speed and location of the pitch, etc. All of that - can be reviewed/studied - by the players after the fact."

The Giants have several "loggers" entering all the information, Woolley said. "They become highly skilled at this," he said. "But that's just during the games. Between games, we have our TiVo boxes capturing all the other games as well. We have to have recent video of the teams we're about to play.

"There are two functions for the video coaching system: There's player development - capturing video of our own players, comparing that video to other players, to past images of themselves, to past outings against another team and so on. Then there's video from other major-league teams, from minor-league teams, and more stored video from amateurs for our player draft. ... So this is really serving as a 'video hub' for our baseball operations," Woolley said.

Any Giants player can come into the video room at any time, put in a query to see a particular video clip and then view it as many times as he wants.

"We have standard and custom requests," Woolley said. "The standard - video - stuff is already prepared for each player before each game. We know the standard slate of things they might want to see. If they come in with any kind of custom request, then they can ask it of the video-operations folks, or they can do it themselves, if they have the skill set."

When a Giants batter is about to face a pitcher such as future Hall of Famer Greg Maddux of the San Diego Padres, for example, he can observe how Maddux tends to pitch in certain situations and how the player did against him in the past, Woolley said.

"They can view the video right here in the video room, or they can burn a DVD on the spot and take it home with them," Woolley said, showing a visitor to the clubhouse how it all works.

The same kind of access to all the video archives is available at the spring training facility in Scottsdale. A T-1 line and WAN connect the two facilities, so there is little latency in moving the digital content from one place to another.

The Giants also use a Nexsan RAID array, coupled with the Data Domain storage. Sydex Sports Software, in Grand Rapids, Mich., handles the Giants' baseball analysis software. "We were one of their early clients; now they have half of baseball," Woolley said.

The IT operation throughout the stadium uses Fibre Channel connectivity - from the data center to the business office to the video production rooms to the Sony and Panasonic high-definition television screens scattered throughout the park to the main scoreboard above center field to running the club's Web site - which is hosted on-site.

The AT&T Wi-Fi network, which added a few dozen new access points during the off-season and now has more than 150 of them in operation, also connects to the main data center.

"The network is fed by four T-1 lines out to the Internet," said Giants Senior Vice President and CIO Bill Schlough. "All the concession booths, our retail store, the press box - and - the clubhouse are hooked up. We're actually installing a bunch of new APs - access points - for remote vendors to go wireless, so they're not restricted anymore for their point-of-sale work," he said.

AT&T Park is adjacent to a large marina on San Francisco Bay. Boat owners can sail right up to the stadium on game days, dock their vessels and go watch the ballgame. While docked, they can check their e-mail or browse the Web using the Giants' free local Wi-Fi access; this includes those hearty souls who venture out into McCovey Cove, just beyond the right-field wall, in small boats and inflatables to station themselves in hopes of recovering a ball hit into the bay.

As of April 10, there have been 41 so-called splash hits - where a home run lands in McCovey Cove without hitting anything at the stadium. Bonds has had 33 of them. Only six other players have had splash hits.

The Giants are pushing out more wireless point-of-sale accessibility, even to fans during the games.

"It always used to be that the vendors - selling food and drinks - walking the aisles could only take cash, but fans will - eventually - be able to use their - credit - cards to buy refreshments," Woolley said. The need for vendors to carry piles of cash will disappear. The timetable for that isn't set yet, he said.

The wireless capabilities extend all the way to the parking lots, Woolley said. "They have wireless scanners out in the parking lots - to take payments at the gates - that do connect - to the data center - . ... There are some access points on the B level of the parking lot that are fed from here."

The Giants have a total of about 20TB of storage for video and for all business systems at AT&T Park's small but efficient data centers, Woolley said.

At the moment, he said, the system is using only about 30 percent of its raw storage capacity. "We're getting like 27-1 compression ratio, due to the efficiency of the system," Woolley said.

The key ingredient making this data center work? The superefficient data deduplication process, which boils down each data file into digital 4K to 12K blocks, compresses them and doesn't allow any of them to be replicated unnecessarily. The result is a more efficient data center that can store more data more quickly.

"We also use the Data Domain storage server for staging and nightly backups, so we can do our off-site backup - at - and vaulting of data over to our Scottsdale facility," Woolley said. "This eliminates the need to take tapes home or store tapes anywhere."

At the moment, the Giants are using only about 30 percent of their storage capacity (20TB of a possible 65TB overall), thanks largely to the Data Domain deduplication feature in the system. The system, as they see it, is now complete; however, they do anticipate buying another storage server later this year to keep plenty of overhead raw storage capacity.

Deduplication replicates the unique segments of data that need to be stored, which doesn't allow unnecessary bits of information to build up and slow down the system.

"None of the other systems we looked at could do replication over a single T-1 line," Woolley said. "They said, 'Well, you're going to need a DS3 or something else.' ... It just got so expensive so quick."

The deduplication efficiency of the Data Domain-Nexsan system results in a lean stream of data that requires only that one line each evening for the nightly backup to Scottsdale. Thanks to its high-performance system architecture, the Data Domain system delivers 6.4TB per hour of throughput.

"From a bandwidth perspective, it was cost-prohibitive to have installed one of those other VTL - virtual tape library - solutions," Schlough said.

After all, a baseball club - like any other business - has to watch carefully where its capital is invested.

Copyright 2007 by Ziff Davis Media, Distributed by United Press International