• Subscribe

Evolving the early universe

The Texascale Days event in December 2020 provided an opportunity for nine research groups to use large sections of the National Science Foundation-funded Frontera supercomputer at the Texas Advanced Computing Center (TACC) to solve problems that in many cases have never been attempted.

Simeon Bird, professor of physics and astronomy at the University of California, Riverside used his one-day access to the full Frontera system to run the largest cosmological simulation his team — or any team — have ever performed at this resolution. What follows is a first-person account of the experience. 

 <strong>A 50 Mpc/h square box</strong> centered on the largest galactic halo showing in turn the dark matter, temperature, metallicity and neutral hydrogen fraction, and including a zoomed-in image of the host galaxy. Courtesy Yueying Ni, Carnegie Mellon University

During the Texascale days event we made progress on our "big run," now dubbed the "Asterix" simulation (named after the cartoon Gaul!). This is the largest cosmological simulation yet performed at this resolution, with 5,5003 dark matter and gas particles and a plethora of stars and black holes, adding up to almost 400 billion elements.

The calculation was carried out under a Large Resource Allocation (LRAC) grant of supercomputer time to support "super-resolution cosmological simulations of quasars and black holes." The team includes Tiziana Di Matteo (principal investigator), Yueying Ni and Rupert Croft at Carnegie Mellon University (CMU), Yu Feng at the University of California, Berkeley, and Yin Li at the Flatiron Institute.

The simulation volume — 250 Mpc/h [megaparsec/hubble constant; an alternative measurement to light-years] across — includes for the first time a statistical sample of high redshift supermassive black holes at sufficient resolution to follow their host galaxies. We include a variety of physical models, including black hole growth, star formation, enrichment of intergalactic gas with metals, and gas physics.

The black hole modeling in particular was based on a new treatment developed by graduate students Nianyi Chen and Yueying NI (both at CMU), tested just in time for this run. The treatment will allow us to make predictions for black hole mergers across cosmic history that the next-generation, space-based gravitational wave detector space, LISA (Laser Interferometer Space Antenna), will detect.

<strong>A histogram</strong> of the number of galaxies in the Universe by stellar mass at z=6.1. The results achieved by Bird's group demonstrate reasonable agreement with the real Universe. Courtesy Yueying Ni, Carnegie Mellon University

Our simulation ran from the beginning until redshift 6, a time when the Universe has just become ionized and the era targeted by the upcoming James Webb Space Telescope. This is already a longer run than was achieved in our previous simulation on the Blue Waters machine, which reached z=7 over the course of a year (the previous simulation was lower resolution but with more elements, so it is not fully comparable, but this illustrates that the improvement is an order of magnitude). The largest previous simulation of this class to reach z < 6 contains about an order of magnitude fewer resolution elements.

We have obtained our first science results (right) which shows a histogram of the number of galaxies in the Universe by stellar mass at z=6.1 and demonstrates reasonable agreement with the real Universe.

We used the full Frontera supercomputer for 24 hours, peaking at 7,911 nodes (or more than 400,000 processors) in a single job. This was our first time attempting usage at this scale. I fixed a couple of bugs in the code over the course of the day that only manifest at this scale. We require at least 2,048 nodes to start our job due to the memory requirements of the simulation and we scaled pretty close to linearly up to the full machine.

Our code is efficient, but without access to a machine of this size, it would not matter: memory requirements are driven by the number of resolution elements and cannot easily be reduced. With a machine of Frontera's size, we are able to really push the boundaries of cosmological simulations in a way that is not possible otherwise.

<strong>A 3 Mpc/h region</strong> around the largest galactic halo in the simulation, showing gas temperature, gas metallicity, neutral hydrogen fraction and star formation rate.  Courtesy Yueying Ni, Carnegie Mellon University

We have also used the GPU cluster on Frontera to good effect and have trained a machine learning-based model on small simulation runs. Yueying Ni and Yin Li have led efforts that resulted in a paper which demonstrated that we could combine high resolution simulations with large volume simulations, approximating a single simulation with a volume one thousand times greater than the individual runs. In the future, we plan to apply this technique to our big run and further increase its dynamic range.

Frontera has really opened up new scientific opportunities. At these high redshifts, the Universe is quite smooth and galaxies are rare. In order to be able to generate predictions for upcoming high redshift telescopes (James Webb Space Telescope and Nancy Grace Roman Space Telescope), we are forced to look at very large volumes and thus very large simulations.

Without the continuing advance of computational capacity this would just not be possible at this resolution and so we would have no way to interpret the upcoming flood of data from new missions.

Read the original article on TACC's website.

Join the conversation

Do you have story ideas or something to contribute? Let us know!

Copyright © 2021 Science Node ™  |  Privacy Notice  |  Sitemap

Disclaimer: While Science Node ™ does its best to provide complete and up-to-date information, it does not warrant that the information is error-free and disclaims all liability with respect to results from the use of the information.


We encourage you to republish this article online and in print, it’s free under our creative commons attribution license, but please follow some simple guidelines:
  1. You have to credit our authors.
  2. You have to credit ScienceNode.org — where possible include our logo with a link back to the original article.
  3. You can simply run the first few lines of the article and then add: “Read the full article on ScienceNode.org” containing a link back to the original article.
  4. The easiest way to get the article on your site is to embed the code below.