Scientists across the US will soon have access to new, ultra-high-speed network links spanning the Atlantic Ocean, thanks to a project currently underway to extend the Department of Energy's Energy Sciences Network (ESnet) to London, Amsterdam, and Geneva. The new links will be heavily used by particle physicists conducting research at the Large Hadron Collider (LHC) at CERN, near Geneva Switzerland - the world's most powerful particle collider.
Investment in science
The transatlantic extension will deliver a total capacity of 340 gigabits-per-second (Gbps), and serve dozens of scientific collaborations. "Particle physicists have been pushing the boundaries of networking technology for decades, and they will make use of our new extension almost immediately," says ESnet director Greg Bell. "Very soon, other data-intensive fields will benefit as well. We expect to see significant network traffic across the Atlantic from the astrophysics, materials science, genomics, and climate science communities." To maximize the resiliency of the new infrastructure, ESnet equipment in Europe will be interconnected with dedicated 100 Gbps links from the pan-European networking organization GÉANT.
The first European network node was installed at CERN in mid-September and will come online in October. The plan is for all links to be commissioned and in production by January 2015, says Bell. Brookhaven National Laboratory in New York, and Fermi National Accelerator Laboratory near Chicago - major US computing centers for the LHC's ATLAS and CMS experiments - will use the links as soon as they are tested and commissioned.
The US LHCNet, led by Harvey Newman and managed by the California Institute of Technology (Caltech), provided transatlantic connectivity during the first successful run of LHC experiments that culminated in the discovery of the Higgs boson in 2012. "This project is greatly indebted to Caltech's Harvey Newman, and to his talented colleagues in the US LHCNet project," Bell says. "Today, we take it for granted that large scientific facilities can be coupled together by high-performance research networks. Harvey championed that idea powerfully, and US LHCNet has provided exceptional service for LHC science."
When the LHC is up and running again in spring 2015, estimates project that data output from the detectors will expand to approximately 40 petabytes of raw data per year, compared with at total of 20 petabytes of raw data for all of the previous lower-energy collisions produced over the first run between 2010 and 2012. "The ability to distribute this on a global scale is the key to its analysis," says David Foster, who is deputy head of CERN's IT department and responsible for international network strategy. "Close collaboration between Caltech and CERN, led by Newman, created transatlantic networking capabilities since the start of LHC operations and enabled a detailed understanding of the computing models. The European extension of ESnet will continue this work and provide increased levels of capacity to explore new approaches to analyzing the LHC data."
Technical challenges of transatlantic connectivity
When building a network between the US and Europe, network engineers are keen to ensure that the service will be robust and built from multiple underlying links - so that if one goes down, researchers can still rely on sufficient bandwidth. Based on the pioneering work and data collected by Caltech physicists and Newman, they also recognize that multiple cables can be cut accidentally - simultaneously.
Undersea (submarine) cables face a number of threats, from ship anchors to undersea earthquakes and landslides. Ship anchors cause most damage near the coastline. Even though undersea cables near land are designed with extra armor and buried up to a meter deep, anchors dropped by supertankers riding out storms can still damage the fiber optics inside the cables.
Determining the safest routes for network links can be difficult - while each cable operator knows the locations of their own cables, there's no accurate database covering all cable paths. Another major consideration is the potential time needed to repair a break. In networks connecting sites across land, engineers can be at the site within a few hours of a problem being reported. With an undersea break, it can take a repair ship up to a week just to find the problem.
After locating a break, the repair ship uses an underwater robot to cut the cable and affix a buoy, which then raises one end of the cable to the ocean surface. The process is repeated for the other end. Once the damaged section is removed and a new piece of cable spliced in, the repaired cable is then lowered back to the seabed. The procedure can be delayed by storms and the number of available repair ships. In some cases, it can take four weeks or longer to repair an undersea break.
The primary challenge for ESnet, then, was designing a transatlantic extension with four separate links to minimize the probability of a complete network outage. That meant each link had to follow a separate path. After getting proposals from a number of vendors, ESnet staff wrote a computer program to analyze and rank all of the detailed cable maps. They then selected the four connections that had the least amount of overlap. To provide an extra measure of safety, ESnet is also working with other research networks in North America and Europe to arrange mutual backup in case of catastrophic failure.
Joe Metzger, ESnet lead engineer, will give a presentation on the groundbreaking, two-year project at the 2014 Technology Exchange conferencein late October.