- LIGO's gravitational wave detection has sparked a hunt for the wave's source.
- Dark Energy Camera (DECam) joins the hunt, sending cosmic images from Chile to scientists at Fermilab.
- The Open Science Grid enables rapid processing of the dense DECam images.
On September 14, 2015, gravitational waves were directly observed for the first time by both detectors of the Laser Interferometer Gravitational-wave Observatory (LIGO). The detection confirmed Einstein’s proposal in his general theory of relativity. Now, scientists are seeking the wave source.
(Check out our coverage of the historic LIGO announcement here.)
Seeking the source
While LIGO, funded by the US National Science Foundation (NSF), can pick out the general direction of the source of gravitational waves, it can’t identify the exact location. So, LIGO scientists are coordinating their measurements with observations made by the Dark Energy Camera (DECam) on the Blanco Telescope at the Cerro Tololo Inter-American Observatory in Chile.
Scientists at Fermilab and other institutions in the Dark Energy Survey use the camera to understand dark energy — a force scientists believe is helping the universe expand. A subset of members known as the Dark Energy Survey-Gravitational Wave (DES-GW) group are using the camera and the Open Science Grid (OSG) to build on LIGO’s groundbreaking findings.
“Our focus primarily is the search for dark energy,” says Marcelle Soares-Santos, associate scientist at the US Department of Energy’s Fermilab. “Since we have experience detecting things through magnetic emissions, we coordinated with LIGO to find a source that we would find useful in our own research. Unfortunately this time we did not see anything, but we are now much better prepared when LIGO becomes active again later this year.”
How to find a needle in a universe-sized haystack
The area of sky DES-GW members observe is very large — 700 square degrees of sky, or about 2,800 times the size of the full moon — and requires rapid image processing. That’s where the OSG comes in. Without OSG, Soares-Santos says they couldn’t keep up.
“For this event, we had something like 4-5,000 jobs. We must break every image down into smaller parts and process them in parallel on the OSG. It is critical to get our results fast — within 24 hours.”
Scientists then analyze their observations with a spectrograph — which is expensive — so it’s important to narrow the choices down to only a few candidates. “At first, our turnaround time was not very fast, but thanks to our close partnership with the computing side here at Fermilab, now it is. We have great confidence that when LIGO observations start again in early August, we will be ready and hopefully see something.”
Kenneth Herner, an application developer and systems analyst at Fermilab, is one of those key partners on the computing side. He makes sure the DES group has as many resources as they need and devotes part of his time to OSG.
“Opportunistic OSG resources really help with the computing needs and the time crunch,” says Herner. “When we submit jobs, we get the first resources that meet the requirements no matter where they may be. We use the CernVM File System to pull in a code repository over HTTP to a local cache on a worker node. It only pulls down what it needs as it needs it. We don’t have to configure each OSG site — it just works. All OSG sites then look the same and all the site has to do is mount a repository.”
In preparation for the LIGO partnership, Herner’s group prepared a code pipeline and made sure everything would work. The LIGO alert came on the 14th. “We had to wait on the telescope—and on top of that an earthquake in Chile,” says Herner. “We worked our plan, checked our code, transferred images from Chile up to the US, and submitted our jobs.”
"We couldn’t do it without the OSG" ~ Marcelle Soares-Santos
Almost all the jobs ran at Fermilab, but Herner says they could have gone anywhere on the OSG. “This was our shakedown cruise,” said Herner. “The first event used about 15,000 CPU hours for a full pass over all nights, but with multiple passes and preprocessing it was over 25,000 hours.” Without OSG resources, the group would have taken Fermilab computing resources away from other experiments, he says.
Observing the sources of these gravitational waves will tell Soares-Santos how systems work and give her and her colleagues deeper insight into the physics. “It is quite challenging to observe these events,” says Soares-Santos.
“We have to be quick to respond to see them. We have to be on the spot sooner and it is the computing that makes that possible. We couldn’t do it without the OSG because of the volume of data. We must have massive parallel computing and quick turnaround and hopefully next time we will see something exciting.”
Read the original version of this OSG science highlight here.