Stories of the single-handed contributions made by astronomy’s greats conjure images of lone figures leaning over telescopes, peering into space. But, in truth, the field is quite collaborative — necessarily so, given the diverse expertise needed to support its increasingly complex technologies.

Such is the case with a 2022 digital sky survey, which is set to capture a full image of the Southern Hemisphere every three to four nights for ten years. The project, known as the Legacy Survey of Space and Time (LSST), will be run by Vera C. Rubin Observatory’s team of 170, using an 8m telescope on top of Cerro Pachón mountain in Chile.

Dense galaxies and fine pixels

Part of what makes this project unique is the sheer amount of data it will collect. For comparison, the Sloan Digital Sky Survey (SDSS) — another big-data survey — collected 116 terabytes, which is significantly less than what Rubin will collect in just one year.

The size of the data is, “mostly all due to [the telescope’s] 3,200-megapixel camera, which is the biggest camera ever made,” says Dino Bektešević, a fellow at the Data Intensive Research in Astrophysics and Cosmology (DiRAC) Institute at the University of Washington and presenter at Gateways 2020. “It takes just humongous pictures. A single image would take 378 4K televisions to display in its actual size.”

All of the data contained in those images — the cosmic structures and their measurements, overlap, and relationships — is useful, so Rubin will be cataloguing it all. This dramatically increases the size of the data, which is already quite large given the enormity of the images.

“The depth that they're going to when they’re stacking all of the data, the density of stars and galaxies that they are actually beginning to detect — it’s a different regime we’re in than what we’ve been in previously,” says Andrew Connolly, researcher at DiRAC and professor at the University of Washington . 

Given the density of the images, data-processing speed was a big focus. And this focus yielded an exciting result: a transient alert system. Transients are temporary space objects and phenomenon that last anywhere from seconds to years. Unfortunately, they often elapse unnoticed, vanishing before researchers spot them. 

"[The camera at Rubin]  takes just humongous pictures. A single image would take 378 4K televisions to display in its actual size.” ~ Bektešević

However, since LSST’s telescope will capture images of the entire Southern Hemisphere on a regular basis, even slight changes will be detectable. 

“Nothing that's transient should pass by LSST unnoticed,” says Bektešević. The Rubin team will process the images in real time and send out alerts within sixty seconds. Astronomers can then zoom in on the phenomena using smaller telescopes. 

Streamlining computational astronomy

Astronomy’s growing demand for big data processing requires new algorithms which can detect, measure, and characterize objects within large space and sky images. The fact that many of these objects overlap makes this an especially complex challenge.

“So, the idea of how you actually separate the light out from two galaxies that are close to one another… This becomes a computational and an astrophysical challenge; because how do you actually assign which light is coming from one galaxy compared to the one lying next to it,” says Connolly.

As a result, one of Rubin’s primary focuses has been to develop state-of-the-art algorithms and models for large-scale cataloguing. These catalogs will be made accessible to astronomers through a science gateway.

The catalogues are detailed enough that most astronomers will be able to find everything they are looking for within them. However, some scientists may want to run custom analyses on all, or some subset, of Rubin’s original data — to test the scalability of a deblending (separating overlapping objects) algorithm, for example. 

That’s where the cloud-based gateway, developed in collaboration with DiRAC, becomes especially useful. It makes large amounts of space data readily accessible but charges only for the selection used, thus ushering in a new level of flexibility and accessibility, economical and otherwise, in astronomical studies. As researchers themselves, these were big concerns for those on the project:

“I think it would be remarkable if a graduate student could get an answer from these massive datasets in hours or minutes compared to the months or the years that it used to take me to work with datasets like this one,” says Connolly.

Access to large amounts of information like this is changing the way astronomers think and the questions they ask:

“We’re getting astronomers no longer looking in telescopes at single objects, but we’re actually giving them a statistical buffet of data that they can do proper statistics on,” says William O’Mullane, head of the Rubin Data Management Team. The ability to look at hundreds of millions of objects in one sitting is allowing astronomers to ask larger, more complex questions about solar system formation, galaxy mergers, and dark energy.

After a sustained effort by so many, the team is excited to see the field changes and discoveries that LSST will make possible. “What I hope the long-term impact of LSST to be, well at least something as cool as the discoveries of stellar streams that came from SDSS,” says Bektešević. “And I'm pretty sure something like that will be done by LSST.”