Slimming Down Supercomputers

Click to enlarge photo.Enlarge Photo
Mira ranks among the world’s most energy-efficient supercomputers, making it within the top 100 of the green supercomputing list.

Photo courtesy of Argonne National Laboratory

Mira ranks among the world's most energy-efficient supercomputers, making it within the top 100 of the green supercomputing list.

Supercomputers are hungry for power. Titan -- the fastest supercomputer in the country -- located in Tennessee consumes about 8 megawatt-hours of electricity. In return, it solves quadrillions of calculations a second. Thousands of people use Titan at the Oak Ridge Leadership Computing Facility and supercomputers at two other DOE Office of Science centers to simulate events that are impossible, impractical, or too dangerous to research any other way -- from volcanoes erupting to supernova slamming into a nearby star.

These computers and their predecessors, stretching back 40 years in some cases, have led to Nobel-Prize-winning breakthroughs. But it's not enough according to a report by the Secretary of Energy Advisory Board. An exascale system, a radical leap that could solve a quintillion calculations a second, is vital to tackle the remaining massive yet nuanced problems that face the world.

However, more computing power often means higher electric bills. "As someone who has to pay the power bill, I want to afford to run these machines for the nation," says Barbara Helland, facilities division director at the Advanced Scientific Computing Research Office of DOE's Office of Science.

Not that long ago, some people thought that exascale systems would need their own dedicated power plants. In 2008, an exascale computer was predicted to gulp 1000 megawatt-hours. Scientists put supercomputers on a strict diet. New predictions are for around 20 megawatt-hours, leading to supercomputers that are 100 times faster, but only use 1.5 times more power.

How was such a diet possible? When your office's name features the words "energy" and "science," you work with vendors, researchers, and others to make computers gourmets, not gourmands.

Video Games and People Reduce Power Use. Smart computer design saves energy and increases computing power. Titan uses a device optimized by video games: the graphical processing unit. Each unit crunches numbers without going back to the memory for more digits. It's the difference between cooking in a well-stocked kitchen and running to the store for oregano.

Also, each DOE supercomputer has a user support team: experts who help users get the most out of their allotted time. When a real-world event — that happens in a fraction of a second — is simulated on a supercomputer in a day-long effort with the electric meter running, efficiency is serious business.

Click to enlarge photo.Enlarge Photo
The National Energy Research Scientific Computing Center.

Photo courtesy of Berkeley Lab

At the National Energy Research Scientific Computing Center, the air cooling grates, just below the windows on the left take in the naturally cool breezes from the San Francisco Bay in the background to chill the two supercomputers and massive data center inside on the lower levels.

Cold Sips and Ocean Breezes Cool Supercomputers. These basketball-court-sized systems pump out heat. At the Argonne Leadership Computing Facility in Illinois, supercomputing systems like Mira and its cohorts take on cooling directly. Instead of first chilling a stream of air and then puffing it into the computer's racks, Mira pulls heat out directly using copper tubes woven through the racks. The result? Far less energy use than less direct methods.

As an added bonus, direct cooling allows more processors to fit in each rack, offering a memorable advantage. The distance the data travels from the memory to the processors is shorter. So if the chef needs pasta, it's across the kitchen, not across town. The less distance data travels, the faster a processor can solve a problem and move on. This benefit has helped Mira secure a place on the annual Top500 list of the world's fastest supercomputers.

At the National Energy Research Scientific Computing Center in California, lower energy bills come with ocean breezes and superb views. Opened in 2015, the center's new home perches above the San Francisco Bay. The building takes in breezes that cool two supercomputers: Edison and Cori. The cooler air pushes warm air up to offices above and leads to a power bill that's about half of that at NERSC's old home, while the computing power has been more than doubled.

Necessity Is a Mother. Any performance improvements that could be wrung out of supercomputers by adding more power have long been exhausted. New supercomputers demand new options that will give scientists a sleek, efficient partner in making new discoveries such as the new supercomputer called Summit that's being developed and is to arrive at Oak Ridge National Lab in the next couple of years.

"If necessity is the mother of invention, we'll have some inventions happening soon," says deputy division director of Argonne Leadership Computing Facility Susan Coghlan.

The Argonne Leadership Computing Facility, the Oak Ridge Leadership Computing Facility, and the National Energy Research Scientific Computing Center are all DOE Office of Science User Facilities. The Office of Science is the single largest supporter of basic energy research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information please visit http://science.energy.gov.

Kristin Manke is a Communications Specialist at Pacific Northwest National Laboratory on detail to the U.S. Department of Energy's Office of Science, [email protected].