When Lawrence Livermore National Laboratory (LLNL) in California, US, began offering outside researchers compute time on Sequoia- the second fastest supercomputer in the world with 1.6 million processor cores capable of 16 thousand trillion calculations per second (16.32 petaFLOPS), Stanford University's School of Engineering Center for Turbulence Research (CTR) in California, US, was one of the first to jump on board.
Researchers at CTR started by working on computational fluid dynamics (CFD) code for simulating and testing noise from high performance jet aircraft. By February, they had successfully rendered complex flow simulations using more than one million compute cores. Joe Nichols, a research associate at CTR spoke about their achievements at this month's HPC Advisory Council Conference at Stanford University. "We really want to highlight the predictive science Sequoia is enabling, and to assess the performance of our tools at the extreme scale of parallelism."
Predictive simulations enable the researchers to model complex jet engine environments (which cannot be observed firsthand), testing different mechanisms and measuring their impacts. In this case, chevrons have been added to the engine nozzle to enhance turbulence in and around the exhaust stream. "The turbulence helps shape the exhaust jet, which has strong consequences downstream. In supersonic jets, this results in interrupting shock cells and disorganizing them, which in turn affects what we hear," adds Nichols.
Nichols's simulation ran successfully using 1,048,576 cores. "The more cores you have, the more complex these domains become," explains Nichols. "Every computational domain is broken up into subdomains that can compute in parallel, but the cores have to be coupled with their neighbors. In this case, at every time step there are waves propagating inside the larger computational domain, so the processors have to let their neighbors know what might be coming."
"This is really an indicator of what we can achieve in the future," says Nichols. "We were able to run the calculation previously on the Blue Gene/P,which had 131,072 cores and took about 100 hours over an entire month. On Blue Gene/Q(Sequoia), it can run in about 12 hours," notes Nichols. "The number of cores became eight times what we were previously running - and we would expect to see an eight-fold increase in speed as well, but what we saw was closer to six or about 83% efficiency. This is quite remarkable considering the amount of communication each core is handling." Nichols works closely with Sanjiva Lele, who has joint appointments in the department of mechanical engineering and the department of aeronautics and astronauticsat Stanford.
The researchers are working with a number of CFD codes developed at CTR: CharlesX - (based on CharLES code developed at Stanford by Frank Ham) - for large eddy simulation and compressible flows; Joe, the RANS solver, for complex geometry; and Hybrid for structured mesh. Ivan Bermejo-Moreno and Julien Bodart, both at CTR, are also actively testing Sequoia. Their projects include simulations of scramjet propulsion systems and of airflow over a full airplane wing. The projects are funded in part by the Predictive Science Academic Alliance Program (PSAAP).
"We are helping design hypersonic scramjets by defining the margins of uncertainty - how far we can push, but still safely operate the system. We are able to use high-fidelity simulations to inform lower-fidelity simulations and ultimately achieve a full system characterization of the scramjet. We're also working on a predictive simulation of a completely new scramjet system," says Bermejo-Moreno. "We are currently running code on Sequoia and taking advantage of the times when we can run on the system at full capacity. The early tests are certainly returning excellent news; we've run simulations on up to 4.1 trillion grid points."
Many CFD breakthroughs involve using inventive numerical algorithms to break down fluid dynamics equations into finite numbers of standard differential equations. "CFD simulations are incredibly complex. Only recently, with the advent of massive supercomputers boasting hundreds of thousands of computing cores, have engineers been able to model jet engines and the noise they produce with accuracy and speed," said Parviz Moin, the Franklin M. and Caroline P. Johnson Professor in the School of Engineering and director of CTR at Stanford.
PSAAP and the US Air Force Office of Science and Researchsupport the jet noise research. Located at LLNL, Sequoia is funded by the Advanced Simulation and Computing programof the US National Nuclear Security Administration.The recent successful tests of extreme CFD calculations were made possible by a collaboration of Stanfordresearchers and LLNL computing staff.