Navigation Links
New ultra-efficient HPC data center debuts

Scientists and researchers at the U.S. Department of Energy's National Renewable Energy Laboratory (NREL) are constantly innovating, integrating novel technologies, and "walking the talk." Since 1982, NREL has won 52 R&D 100 Awards known in the research and development community as "the Oscars of Innovation" for its groundbreaking work.

When it came time for the lab to build its own high performance computing (HPC) data center, the NREL team knew it would have to be made up of firsts: The first HPC data center dedicated solely to advancing energy systems integration, renewable energy research, and energy efficiency technologies. The HPC data center ranked first in the world when it comes to energy efficiency. The first petascale HPC to use warm-water liquid cooling and reach an annualized average power usage effectiveness (PUE) rating of 1.06 or better.

To accomplish this, NREL worked closely with industry leaders to track rapid technology advances and to develop a holistic approach to data center sustainability in the lab's new Energy Systems Integration Facility (ESIF).

"We took an integrated approach to the HPC system, the data center, and the building as part of the ESIF project," NREL's Computational Science Center Director Steve Hammond said. "First, we wanted an energy-efficient HPC system appropriate for our workload. This is being supplied by HP and Intel. A new component-level liquid cooling system, developed by HP, will be used to keep computer components within safe operating range, reducing the number of fans in the backs of the racks."

Next, the NREL team, which included the design firms SmithGroupJJR and the Integral Group, created the most energy-efficient data center it could to house and provide power and cooling to the HPC system. High-voltage (480 VAC) electricity is supplied directly to the racks rather than the typical 208 V, which saves on power electronics equipment, power conversions, and losses. Energy-efficient pumps largely replace noisy, less-efficient fans.

"Last but not least, we wanted to capture and use the heat generated by the HPC system," Hammond said. "Most data centers simply throw away the heat generated by the computers. An important part of the ESIF is that we will capture as much of the heat as possible that is generated by the HPC system in the data center and reuse that as the primary heat source for the ESIF office space and laboratories. These three things manifest themselves in an integrated 'chips-to-bricks' approach."

Like NREL's Research Support Facility, the ESIF HPC data center did not cost more to build than the average facility of its kind. It actually cost less to construct than comparable data centers and will be much cheaper to operate. NREL's approach was to minimize the energy needed, supply it as efficiently as possible, and then capture and reuse the heat generated.

"Compared to a typical data center, we may save $800,000 of operating expenses per year," Hammond said. "Because we are capturing and using waste heat, we may save another $200,000 that would otherwise be used to heat the building. So, we are looking at saving almost $1 million per year in operation costs for a data center that cost less to build than a typical data center."

Warm-Water Cooling Boosts Data Center Efficiency

The ultra-efficient HPC system in NREL's new data center has been designed in collaboration with HP and Intel. The HPC system will be deployed in two phases that will include scalable HP ProLiant SL230s and SL250s Generation 8 (Gen8) servers based on eight-core Intel Xeon E5-2670 processors as well as the next generation of servers using future 22nm Ivy Bridge architecture-based Intel Xeon processors and Intel Many Integrated Core architecture-based Intel Xeon Phi coprocessors. The first phase of the HPC installation began in November 2012, and the system will reach petascale capacity in the summer of 2013.

In the spirit of overall energy efficiency, the Intel Xeon Phi coprocessor delivers on several fronts. According to Intel, it can easily port complete applications in a short time, so software engineers won't need specialized tools or new languages to support significant software packages. "Intel coprocessors also increase the efficiency of computer resource usage," said Stephen Wheat, general manager of high performance computing at Intel. "The methods of code optimization for Xeon Phi are identical to what one does to make the most of Xeon processors. Finely tuned optimizations for Xeon Phi almost always result in a better-performing source code for Xeon processors. As the optimized and tuned application is run in production, the achieved performance per watt on both Xeon Phi and Xeon processors allows achieving the results with the lowest energy use."

While some of the NREL HPC components may be off the shelf, the team is taking a different approach in cooling this supercomputer.

"In traditional computer systems, you have a mechanical chiller outside that delivers cold water into the data center, where air-conditioning units blow cold air under a raised floor to try to keep computer components from overheating," Hammond said. "From a data center perspective, that's not very efficient; it's like putting your beverage on your kitchen table and then going outside to turn up the air conditioner to get your drink cold."

"NREL's ultimate HPC system is currently under development and will be a new, warm-water cooled high-performance system," said Ed Turkel, group manager of HPC marketing at HP. "It will be a next-generation HPC solution that's specifically designed for high power efficiency and extreme density, as well as high performance things that NREL requires."

Starting this summer, NREL's HPC data center will require just over 1 megawatt of power to operate. "That's a lot of power; the heat dissipated from that is very substantial," Hammond said. "Getting the heat directly to liquid rather than through air first and then to liquid is the most efficient way to utilize it."

Water being supplied to the servers will be approximately 75 degrees Fahrenheit; the water returning from the HPC will be in excess of 100 degrees Fahrenheit and is designed to be the primary source of heat for ESIF's office and lab spaces. Data-center waste heat is even used under the front plaza and walkway outside the building to help melt snow and ice. Thus, the heat byproduct from the data center will also improve safety around the ESIF.

Compared to a typical data center, NREL's HPC data center will be much warmer. The 75-degree-Fahrenheit design point is a higher starting temperature for computer cooling. Starting at this temperature allows NREL to eliminate compressor-based cooling systems and instead use cooling towers. In a data center, this is comparable to a homeowner using an energy-efficient swamp cooler rather than an air conditioner. In addition, the pump energy needed to move liquid in the cooling system is much less than the fan energy needed to move the air in a traditional data center. Water is about 1,000 times more effective than air in terms of the thermodynamics, or the heat exchange.

"We're quite enamored with NREL being able to reuse the heat for the building and parts of the campus," Wheat said. "While others have done this before, here we are looking at a combined total efficiency goal and not just harvesting heat. We're looking to see how this can be the best solution for the entire campus."

Using the HPC's waste heat to boost the ESIF's sustainability and finding unique solutions to cut the data center's PUE are just what NREL does. "This is in our DNA; it is part of our mission at the lab, and we want others to follow suit," Hammond said. "NREL isn't one-of-a kind in what we are doing but we've set out to be the first of a kind. For us, it just makes sense. Others can follow suit if it makes dollars and sense."

The lab's industry partners also see a long-term relationship for energy efficiency and HPC, especially when it comes to exascale computing.

"We see the area of HPC as being insatiable; people will have a need for ever-greater performance," Wheat said. "One of the things we are mindful of is that while our systems are becoming denser in terms of footprint, they are becoming more power efficient. NREL is the premiere place to demonstrate a means to continue the growth of HPC capability in an environmentally friendly way."

HP's Turkel echoes that sentiment: "As power-efficient and dense as our HPC systems are, to meet our customer's rapidly expanding requirements for performance, we would need to grow even our most powerful and efficient system to be impractically large and complex, while consuming enormous amounts of energy.

"To get to the levels of scale that our customers are demanding of us, we have to fundamentally change the dynamic around power, density, and performance," Turkel added. "We have to be able to do it in a much smaller package using less energy. This project is a step in that direction and it's apropos that NREL is a partner in the effort."

"eBay, Facebook, and others have data centers that are water capable, but there aren't any products on the market now that are providing liquid cooling," Hammond said. "NREL is getting the first product that is direct-component liquid cooled. We're going to show it's possible, efficient, safe, and reliable."

Expanding NREL's View into the Unseen

The $10 million HPC system will support the breadth of research at NREL, leading to increased efficiency and lower costs for research on clean energy technologies including solar photovoltaics, wind energy, electric vehicles, buildings technologies, and renewable fuels.

The new system is crucial to advancing NREL's mission and will enable scientists to address challenges that have been intractable to date. The new system will greatly expand NREL's modeling and simulation capabilities, including advancing materials research and developing a deeper understanding of biological and chemical processes. "Modeling and simulation capability is a key part of advancing our technologies," Hammond said. "It allows us to do research we can't do experimentally because it would be too expensive, or would take too long if actual systems were built. We can mathematically model and run numerical simulations that allow us to understand things through direct observation."

Before building an HPC data center in the ESIF, NREL had a small system on its campus, while collaborating with Sandia National Laboratories on the RedMesa supercomputer to bridge the gap until NREL had a facility to house its own HPC system. For the past two years, the NREL/Sandia solution has been oversubscribed. "We averaged 92% utilization day in and day out; we needed much more capable systems to meet growing demand for modeling and simulation," Hammond said.

According to Hammond, NREL will also reach out to the local utility to study demand-response scenarios. "There are times in summer when electricity demand is high that we could shed load with the data center to help Xcel Energy." NREL could alter workloads and schedule particular jobs to run in mornings when there's a high demand for heat and cooling is less expensive. In another scenario, NREL could schedule workloads to take advantage of lower electricity costs or be mindful of when rates are higher to help reduce operating expenses. "There is a lot of interest in looking at how to integrate the HPC system in the building automation system as part of the energy systems integration work that we're doing," Hammond said.

"The computational activities at NREL had to be part of the efficiency equation," said Wheat. "A motivation for Intel to work with NREL was the ability to work together to validate how to do an efficient data center. We needed to be able to assure that we had the right balance of processor performance for the workload with a performance-per-watt focus. Being a partner with NREL in this process is of value to us for demonstrating leadership in our community. Others are taking notice of what NREL has done; I believe we all benefit from that."


Contact: David Glickson
DOE/National Renewable Energy Laboratory

Related biology news :

1. New Center for Carbon Measurement to drive UKs low carbon economy
2. Israel names Tel Aviv Universitys Renewable Energy Center a Center of Research Excellence
3. Heart study suggests city center pollution doubles risk of calcium build-up in arteries
4. Mainz University Medical Center attracts funding of Alexander von Humboldt Professorship
5. UC Santa Cruz builds national data center for cancer genome research
6. UF to establish Faroe Island research center with help of baseball star
7. DOE Plasma Science Center Annual Meeting at Princeton Plasma Lab
8. UH Case Medical Center, CardioKinetix reveal promising data for treatment for heart failure
9. Tufts Medical Center researchers finds marker in premies saliva predicts readiness to feed by mouth
10. University of Hawaii Cancer Center researchers create new anti-cancer drug
11. Center for North Sea oil expertise set to boost jobs and growth
Post Your Comments:
(Date:4/11/2017)... 11, 2017 Research and Markets has announced ... report to their offering. ... global eye tracking market to grow at a CAGR of 30.37% ... Tracking Market 2017-2021, has been prepared based on an in-depth market ... landscape and its growth prospects over the coming years. The report ...
(Date:4/5/2017)... April 5, 2017  The Allen Institute for Cell ... Explorer: a one-of-a-kind portal and dynamic digital window into ... data, the first application of deep learning to create ... cell lines and a growing suite of powerful tools. ... these and future publicly available resources created and shared ...
(Date:4/3/2017)... , April 3, 2017  Data ... precision engineering platform, detected a statistically significant ... product prior to treatment and objective response ... the potential to predict whether cancer patients ... to treatment, as well as to improve ...
Breaking Biology News(10 mins):
(Date:10/10/2017)... , Oct. 10, 2017 SomaGenics announced the ... NIH to develop RealSeq®-SC (Single Cell), expected to be ... small RNAs (including microRNAs) from single cells using NGS ... the need to accelerate development of approaches to analyze ... "New techniques for measuring levels of mRNAs ...
(Date:10/9/2017)... ... , ... The award-winning American Farmer television series will feature 3 Bar Biologics ... at 8:30aET on RFD-TV. , With global population estimates nearing ten billion people ... to feed a growing nation. At the same time, many of our valuable resources ...
(Date:10/7/2017)... , ... October 06, 2017 ... ... experience providing advanced instruments and applications consulting for microscopy and surface analysis, ... in application consulting, Nanoscience Analytical offers a broad range of contract analysis ...
(Date:10/6/2017)... , ... October 06, 2017 , ... ... a lunch discussion and webinar on INSIGhT, the first-ever adaptive clinical trial for ... Dana-Farber Cancer Institute. The event is free and open to the public, but ...
Breaking Biology Technology: