- Antarctica perfect place to build observatory large enough to detect elusive neutrinos
- Scientific cooperation necessary to supply sufficient computing power
- Neutrino detection confirmed by scientists around the world, pointing to origin in a blazar
Four billion years ago—before the first life had developed on Earth—a massive black hole shot out a proton at nearly the speed of light.
Fast forward—way forward—to 45.5 million years ago. At that time, the Antarctic continent had started collecting an ice sheet. Eventually Antarctica would capture 61 percent of the fresh water on Earth.
Thanks to XSEDE resources and help from XSEDE Extended Collaborative Support Service (ECSS) experts, scientists running the IceCube Neutrino Observatory in Antarctica and their international partners have taken advantage of those events to answer a hundred-year-old scientific mystery: Where do cosmic rays come from?
Making straight the path
First identified in 1912, cosmic rays have puzzled scientists. The higher in the atmosphere you go, the more of them you can measure. The Earth’s thin shell of air, scientists came to realize, was protecting us from potentially harmful radiation that filled space. Most cosmic ray particles consist of a single proton. That’s the smallest positively charged particle of normal matter.
Cosmic ray particles are ridiculously powerful. Gonzalo Merino, computing facilities manager for the Wisconsin IceCube Particle Astrophysics Center at the University of Wisconsin-Madison (UW), compares the force of a proton accelerated by the LHC, the world’s largest atom-smasher, as similar to the force of a mosquito flying into a person. By comparison, the “Oh-My-God” cosmic ray particle detected by the University of Utah in 1991 hit with the force of a baseball flying at 58 miles per hour.
Because cosmic-ray particles are electrically charged, they would be pushed and pulled by every magnetic field they encounter along the way. Cosmic rays would not travel in a straight line, particularly if they came from some powerful object far away in the Universe. You can’t figure out where they originated from by their direction when they hit Earth.
Particle-physics theorists came to the rescue.
“If cosmic rays hit any matter around them, the collision will generate secondary products,” Merino says. “A byproduct of any high-energy interaction with the protons that make up much of a cosmic ray will be neutrinos.”
Neutrinos respond to gravity and to what’s known as the weak subatomic force, like most matter. But they aren’t affected by the electromagnetic forces that send cosmic rays on a drunkard’s walk. Scientists realized that the intense showers of protons at the source of cosmic rays had to be hitting matter nearby, producing neutrinos that can be tracked back to their source.
The shape of water
But if the matter that makes up your instrument can’t interact with an incoming neutrino, how are you going to detect it? The answer lay in making the detector big.
“The probability that a neutrino will interact with matter is extremely low, but not zero,” Merino explains. “If you want to see neutrinos, you need to build a huge detector so that they collide with matter at a reasonable rate.”
Enter the Antarctic ice shelf. The ice here is nearly pure water and could be used as a detector. From 2005 through 2010, a UW-led team created the IceCube Neutrino Observatory by drilling 86 holes deep in the ice, re-freezing detectors in the holes. Their new detector consisted of 5,160 detectors suspended in a huge ice cube six-tenths of a mile on each side.
The IceCube scientists weren’t quite ready to detect cosmic-ray-associated neutrinos yet. While the IceCube observatory was nearly pure water, it wasn’t completely pure. As a natural formation, its transparency might differ a bit from spot to spot, which could affect detection.
“Progress in understanding the precise optical properties of the ice leads to increasing complexity in simulating the propagation of photons in the instrument and to a better overall performance of the detector,” says Francis Halzen, a UW professor of physics and the lead scientist for the IceCube Neutrino Observatory.
GPUs to the rescue
The collaborators simulated the effects of neutrinos hitting the ice using traditional supercomputers containing standard central processing units (CPUs). They realized, though, that portions of their computations would instead work faster on graphics-processing units (GPUs), invented to improve video-game animation.
“We realized that a part of the simulation is a very good match for GPUs,” Merino says. “These computations run 100 to 300 times faster on GPUs than on CPUs.”
Madison’s own GPU cluster and collaborators’ campuses’ GPU systems helped, but it wasn’t enough.
Then Merino had a talk with XSEDE ECSS expert Sergiu Sanielevici from the Pittsburgh Supercomputing Center (PSC), lead of XSEDE’s Novel and Innovative Projects. Sanielevici filled him in on the large GPU capability of XSEDE supercomputing systems. The IceCube team wound up using a number of XSEDE machines for GPU and CPU computations: Bridges at PSC, Comet at the San Diego Supercomputer Center (SDSC), XStream at Stanford University and the collection of clusters available through the Open Science Grid Consortium.
The IceCube scientists could not assume that their computer code would run well in the XSEDE system. Their massive and complex flow of calculations could have slowed down considerably had the new machines conflicted with it. ECSS expertise was critical to making the join-up smooth.
“XSEDE’s resources integrated seamlessly; that was very important for us,” Merino says. “XSEDE has been very collaborative, extremely open in facilitating that integration.”
Their detector built and simulated, the IceCube scientists had to wait for it to detect a cosmic neutrino. On Sept. 22, 2017, it happened. An automated system tuned to the signature of a cosmic-ray neutrino sent a message to the members of the IceCube Collaboration, an international team with more than 300 scientists in 12 countries.
This was important. A single neutrino detection would not have been proof by itself. Scientists at observatories that detect other types of radiation expected from cosmic rays needed to look at the same spot in the sky.
They found multiple types of radiation coming from the same spot in the sky as the neutrino. At this spot was a “blazar” called TXS 0506+056, about 4 billion light years from Earth. A type of active galactic nucleus (AGN), a blazar is a huge black hole sitting in the center of a distant galaxy, flaring as it eats the galaxy’s matter. Blazars are AGNs that happen to be pointed straight at us.
The scientists think that the vast forces surrounding the black hole are likely the catapult that shot cosmic-ray particles on their way toward Earth. After a journey of 4 billion years across the vastness of space, one of the neutrinos created by those particles blazed a path through IceCube’s detector.
The IceCube scientists went back over nine and a half years of detector data, before they’d set up their automated warning. They found several earlier detections from TXS 0506+056, greatly raising their confidence.
The findings led to papers in the prestigious journal Science in July 2018. Future work will focus on confirming that blazars are the source—or at least a major source—of the high-energy particles that fill the Universe.