- Hikari is the first supercomputer in US to use solar and HVDC for power.
- Data centers consumed 1.8 percent of all US electricity in 2014.
- Hikari's success should spur greater adoption of HVDC by data center industry.
The roar can be deafening. Cooling fans and power supplies whoosh and whine from rows and rows of supercomputers at the main data center of the Texas Advanced Computing Center (TACC) in Austin. The power bill at TACC can reach over a million dollars a year keeping the machines humming. But there’s a stranger in town that might change how data centers power their systems.
A new kind of advanced computing system called Hikari (Japanese for the word ‘light’) came online at TACC late August, 2016. What’s new is that Hikari runs on solar power and high voltage direct current (HVDC.)
Hikari is a microgrid that supports a supercomputer, a first for the US. By day, solar panels that shade a TACC parking lot provide nearly all of Hikari’s power, up to 208 kilowatts. At night, it switches back to conventional AC power from the utility grid.
“The Hikari project is a Japan and Texas collaboration project, and it aims to demonstrate the potential of the HVDC system,” says Toshihiro Hayashi, assistant manager in the engineering divisions of Nippon Telegraph and Telephone Corporation (NTT) Facilities, Inc..
Engineers of the Hikari HVDC power feeding system predict it will save 15 percent compared to conventional systems. “The 380 volt design reduces the number of power conversions when compared to AC voltage systems,” says James Stark, director of Engineering and Construction at the Electronic Environments Corporation (EEC).
“What's interesting about that,” Stark adds, “is the computers themselves – the supercomputer, the blade servers, cooling units, and lighting – are really all designed to run on DC voltage. By supplying 380 volts DC to Hikari instead of having an AC supply with conversion steps, it just makes a lot more sense. That's really the largest technical innovation.”
Data centers in the US consumed an estimated 70 billion kilowatt hours in 2014, which represents about 1.8 percent of total U.S. electricity consumption. HVDC also allows for ease in connection to renewable energy, solar for Hikari but potentially other sources like wind and hydrogen fuel cells.
“That's really one of our main focuses, trying to make data centers more sustainable so that we're reducing overall power consumption within an industry that has traditionally consumed a lot of power,” Stark says.
Data centers in the US consumed an estimated 70 billion kilowatt hours in 2014.
The collaboration started with a visit to TACC by engineers at NTT Facilities, Inc. in early 2014, fresh off the heels of basic studies they had done for installing HVDC at US sites. TACC also shares a strong interest in developing new technologies, including energy savings.
“We're very interested in UT Austin's motto,” says Hayashi, “which is ‘what starts here changes the world.’ We very much agree with this motto.”
Hayashi's team worked with TACC to develop feasibility studies of the Hikari HVDC project from December 2014 to May 2015. This effort led to a Memorandum of Understanding between the State of Texas and New Energy and Industrial Technology Development Organization (NEDO) in August of 2015.
NTT Facilities, Inc. worked with EEC to build out Hikari, which completed the late August 2016 system installation. “If there wasn't such partnership, we wouldn't have launched this project. I would like to express my gratitude to NEDO for establishing the partnership,” Hayashi says.
The Hikari supercomputer cluster consists of 432 Hewlett Packard Enterprise (HPE) Apollo 8000 XL730f servers coupled with HPE DL380 and DL360 nodes that are interconnected with a first-of-its-kind Mellanox End-to-End EDR InfinBand at 100 gigabytes per second. Over 10,000 cores from ‘Haswell’ Xeon processors will deliver more than 400 teraFLOPS.
The Hikari project also aims to demonstrate energy efficiency through more than just HVDC. The HPE Apollo 8000 systems use a warm water-based liquid cooling system that eliminates the need for fans within the nodes and reduces the energy that would normally be required for water refrigeration and excess heat removal. The solar energy that would have been used for powering fans and chillers can be used for computational work.
When it reaches production later in 2017, Hikari will be used by the University of Texas medical researchers to make progress on diseases like cancer and disorders like autism.
“We really hope this project will demonstrate the efficiency advantages of using 380 volt DC, not only in data centers, but in any commercial building,” James Stark says. “The hope is that the research that comes out of this demonstration project will help to open the door to more widespread use of 380 volt systems throughout data centers and commercial buildings worldwide.”
