Using Simulation to Design Efficient Data Center Cooling Systems
By 2025, about half of the world’s data1 will reside in the cloud. It is estimated that this market will be worth a huge $24.8 billion2, which puts significant pressure on designers and builders to construct new data centers more efficiently and to overcome known challenges:
- Optimized use of data center facilities: taking advantage of all available space and using power more efficiently, while maintaining sufficient cooling
- Ensuring physical and digital security to protect hardware and data from threats including fire, natural disaster, electromagnetic pulse (EMP), remote attacks, and on-site incursions
- Hyperscaling: agile practices to scale rapidly and match the cooling and power supplies to the demand, moment-to-moment
- Preparing for changes to the semiconductor industry as the growth in transistor density slows, necessary hardware requires increased space and power (as described by Moore’s law)
For data center owner-operators, contractors and builders, managing cooling through Heating, Ventilation, and Air Conditioning (HVAC) and rack layout is one of the most important factors in meeting these challenges.
This article will cover HVAC design for data centers and show how simulation helps to make cooling more efficient to improve data center performance and reduce operating costs.
How Simulation can help address Data Center Challenges
According to the IEA3, data centers alone consumed 200-250 TWh, which is equivalent to 1% of global electricity use (not including cryptocurrency mining-related facilities, which consumed an additional 100 TWh). This has remained fairly constant over the past decade, despite the huge growth in internet traffic, due to the ever-increasing efficiency of data centers. Nevertheless, continuing improvements in efficiency will be necessary to stay competitive, as data usage increases ever faster.
At present, the largest and most efficient data centers have a power usage effectiveness (PUE) of 1.14, meaning that for every 10 W of power used in computing, they consume an additional 1 W of power for cooling and other uses. Smaller data centers tend to be far less efficient and have even higher relative cooling needs, as much as half and half between compute power and cooling consumption5. Reducing the power consumption of cooling systems can therefore significantly reduce a data center’s operating costs.
Integrating simulation from the start of the design process means that cooling requirements can be considered at every stage of development, accelerating deployment times and reducing the risk of issues emerging later. It ensures that data center facilities meet the specifications and helps engineers to cut power usage and operating costs, by matching the design of the cooling to the design of the compute infrastructure.
Engineers can use a Virtual Twin of the HVAC system and the room in which it is housed, which includes all the relevant details and data about the system. This approach reduces prototyping costs and accelerates design, and is already established in industries such as automotive and aerospace in designing HVAC for efficiency, cooling performance and comfort.
Simulation helps to identify several approaches to reduce power consumption and cut operating costs:
- Simulating heat sources such as servers in racks, and optimizing rack layouts to avoid heating hotspots. This can not only reduce the cooling fan requirement and the resulting power consumption, but also the risk of compute power loss or even early component failure from overheating
- Modeling different HVAC configurations and room layouts, and optimizing the design of control algorithms, helping to improve airflow and cooling efficiency, and to reduce air leaks
- Analyzing the “side effects” of cooling, such as the cost of materials, the potential fire hazard caused by layout choice, and the risk of a cooling failure
Identify Heat Sources
Servers generate significant amounts of heat from electric resistance, and this heat needs to be removed through air or liquid cooling. Server internal design affects rack hotspots and, if left unchecked, these can cause early component failure or excessive power consumption.
Heat sources in individual servers can be identified with 3D electromagnetic simulation, but with multiple servers in a single rack and numerous racks in a room, cooling becomes a multiscale problem. It is dependent on airflow at every scale, from individual server components, up through racks to the entire room layout. It also covers several areas of physics including fluid dynamics, thermodynamics and electromagnetics. A simulation solution therefore needs to connect all scales of the problem and include all the relevant physics. This can be done with coupled 3D CFD and 1D system modeling. The cost-effective system model represents the server and HVAC systems, whilst the advanced 3D CFD provides the detail needed about the thermodynamics of the room in order to make informed engineering decisions.
Manage Cooling Efficiently
There are a number of different cooling techniques for data centers. One of the most important factors in how effectively and efficiently the centers are cooled is their layout. Commonly, data centers are designed with alternating cold and hot aisles. Cold aisles are created by placing cold air intakes on the front of the racks, and hot aisles are created by placing hot air exhausts on the back of the racks. The hot aisles expel hot air into the air conditioning intakes, where it is cooled, recirculated, and vented into the cold aisles. This approach can reduce cooling costs by 40%.
This sounds simple, but optimizing the layout of a data center is complex, with multiple factors needing to be considered, such as spacing, leakage6 between aisles, sensor positioning, cabling, power backup, fire prevention, and security. Generative design is one solution, also provided by the 3DEXPERIENCE platform, for making layout decisions. The platform’s tools include a configurator that proposes the optimal layout of racks in any given space.
A user can feed in any kind of footprint into the configurator and it will automatically perform rack layout, not only placing the racks but also generating all dependent subsystems, such as HVAC and firefighting, in a multidisciplinary, complete solution. A configurator for individual racks is also available.
Optimization scenarios can take into account many factors that affect cooling performance, including weather conditions, humidity, and the flow through small gaps. Other aspects, such as noise, can also be analyzed.
Failure Management and Other Issues
The cost of a data center can reach tens or hundreds of millions of US dollars, and this investment must be protected. There are threats to data center operations, both natural and human-caused. Many of these can be virtually tested and mitigated in order to reduce maintenance requirements and loss of service.
With so much electrical equipment in densely spaced rooms, fire is a very serious risk. Data centers need to be designed for very stringent levels of fire safety to prevent damage or loss of life. HVAC affects the spread of fire and smoke, as well as the effectiveness of gas-based fire suppression systems. Simulation can be used to analyze these risks.
Cutting the Carbon Footprint
The energy consumption density of data centers is currently about 40 times higher than that of residential buildings. Improving the performance of HVAC through operations optimization is estimated to be able to increase HVAC efficiency by 20%7, which reinforces the importance of simulation for data center design, especially with respect to reducing operating costs.
To address the various challenges data center design and operation pose, a new approach is required. Engineers, owners and operators must move away from traditional fragmented and siloed industry methods; in this, simulation and digitization are key. Dassault Systèmes’ 3DEXPERIENCE platform enables the entire value chain to be digitally connected, and the virtual twin experience it provides allows all stakeholders to collaborate more efficiently and effectively.
Virtual twins let designers and engineers digitally create and modify an entire data center, optimizing the design before any physical structures are built. These digital twins can be used for thermal and flow simulation as well as layout, enabling a plan for effective temperature regulation to be put in place. Simulation using the virtual twin can help reduce operating costs and make data centers efficient and fit for the demands of the future.
References
1 https://blog-assets.3ds.com/uploads/2022/03/idc-seagate-dataage-whitepaper.pdf
3 https://www.iea.org/reports/data-centres-and-data-transmission-networks
5 https://www.sciencedirect.com/science/article/pii/S0306261921003019
6 https://www.datacenterdynamics.com/en/news/server-leakage-and-cooling
7 https://dteenergy.bizenergyadvisor.com/article/data-centers
SIMULIA offers an advanced simulation product portfolio, including Abaqus, Isight, fe-safe, Tosca, Simpoe-Mold, SIMPACK, CST Studio Suite, XFlow, PowerFLOW, and more. The SIMULIA Community is the place to find the latest resources for SIMULIA software and to collaborate with other users. The key that unlocks the door of innovative thinking and knowledge building, the SIMULIA Community provides you with the tools you need to expand your knowledge, whenever and wherever.
Charles Luzzato is a senior manager responsible for climate system methodologies worldwide and across industries. After completing his PhD in Thermo-Acoustic Instabilities at Imperial College London, Charles joined Exa Corporation, working on Aero-acoustics simulation topics for big German OEMs and suppliers. In 2017 he became the team leader for the aeroacoustics team in Munich, working intensively on climate system applications throughout Europe. After the acquisition of Exa Corporation by Dassault Systèmes, he moved into the worldwide Industry Process Success team of SIMULIA.