Water cooling with NeXtScale WCT

NeXtScale Water Cool Technology

By Duncan Furniss, Consulting Client Technical Specialist for Lenovo in Canada

NeXtScale WCT (Water Cool Technology) is the most efficient NeXtScale systems yet, due to both the efficiency of direct water cooling, and the additional performance and power savings achieved by running the high power electronics at lower temperatures than possible with air cooling.

I made the long trek from Vancouver, Canada to Morrisville, North Carolina to visit the new Lenovo facilities and work with the NeXtScale WCT team and the Lenovo Press team (formerly the Redbooks team) to produce the NeXtScale WCT Product Guide. This blog highlights what I learned during my week-long visit.

Here’s a look at one of new buildings at the Lenovo campus - the new home of product development for NeXtScale WCT


The new Lenovo campus in Morrisville, North Carolina

Most of development of the water-cooled  variant of the NeXtScale n1200 chassis, and nx360 M5 compute nodes was in the thermal lab. It is here in the thermal lab that the manifolds for the chassis were designed, the circulation path for the water was determined, the front and rear conduction plates were shaped, and other specialized parts originated. Not only did the engineers assure that the systems were adequately cooled, but they studied the effect of temperature on the performance of the system.

Here’s a photo of a rack of prototype nodes under test, along with thermal probes and an instrumented water system.

We were in the lab doing the video shoot while they are conducting stress tests. All the compute node were running at 100% CPU utilization, but the rack was quiet enough that we could still shoot our video right in front of the rack!

While I was in Raleigh working on the Product Guide, we also shot a video where we went through all the components that make up the WCT system. We actually filmed it right next to the rack while it was fully loaded and running at maximum CPU utilization. Because the WCT is cooled with water and not with dozens of fans, we were able to carry on a conversation without having to shout at each other. Very cool!

Here’s a screen grab from our video where we’re going through the water circulation components. I’m with David Watts from the Lenovo Press team (white shirt) and Mark Steinke, senior engineer from the thermal lab (blue shirt).


The water hoses use special couplings that don’t inhibit the water flow and allow you to connect and disconnect even while the water pressure is at operational levels

The photo below shows two NeXtScale compute nodes on the one tray. Two NeXtScale nx360 M5 WCT nodes are paired together on a full wide compute tray to make most efficient use of the water flow. Each tray has one water inlet and one water outlet; the cooling water flow is split for each node inside the tray, and collected from each node before exiting.


Two independent NeXtScale servers on the one compute tray. Direct water cooling maximizes heat transfer and energy efficiency.

The processor heat sinks and the conduction plates (front and rear) are directly cooled with water. The conduction plates are shaped to directly contact several integrated circuits on the main board of the servers, and copper tubes connect the thermally conductive material between the DIMMs, and other heat sinks, to the conduction plates.

One of the interesting findings from the testing in the thermal lab is that the cooling water can be as warm as 45 ºC (113 ºF) and still cool everything more effectively than air, while providing lower power consumption (about 5%) and higher performance (thanks to Intel Turbo Boost) than air cooled systems. This means that, in most climates, the water used to cool the servers can then be cooled with ambient air, no chillers required.

The photo below shows the rear of a rack of water cooled NeXtScale systems. Notable are the filler panels where the chassis fans used to be. Cooling fans are noisy and inefficient and just not needed when you have water cooling!


Water is fed to the servers via hoses and manifolds at the back of the rack

The two vertical hoses  connect all the water manifolds.  There is one external connection to each chassis for the inlet water manifold, and one external connection to the outlet water manifold. Each manifold has 6 internal connectors, one for each compute tray location in the chassis. These connectors enable the compute trays to be installed or removed while there is water flowing through other nodes.

Unique to the WCT chassis are drip sensors, one under each manifold; if any drips are detected, the Fan and Power Controller (FPC) is notified, and the FPC can send an alert.

The power supplies used are the same as for the air cooled NeXtScale systems. They have variable speed fans, and pull cooling air through the nodes to cool the few non-water cooled components as well as the power supplies themselves.

 

If you are looking for a solution with a chilled water loop and cooling distribution unit, like the one in the lab – but without the instrumentation – the team can provide the specifications and suggest commercially available systems.

The NeXtScale nx360 M5 nodes support processors up to 18 cores and 256 GB RAM, so a rack with 6 full chassis – 72 nodes – can provide 2,592 cores and 18,432 GB RAM. A very compute dense solution is manageable with the cooling made possible by the WCT approach.

For more information about the WCT solution, see the NeXtScale M5  product page. And stay tuned for the Product Guide and our walk-through video.

Duncan FurnissDuncan Furniss is a Consulting Client Technical Specialist for Lenovo in Canada. He provides technical sales support for iDataPlex, NeXtScale, BladeCenter, Flex and System x products, and has co-authored several IBM Redbooks publications, including the IBM NeXtScale System Planning and Implementation Guide. Duncan has designed and provided oversight for the implementation of many large-scale solutions for HPC, distributed databases, and rendering of computer generated images.