Twenty years in this industry, and the thermal problem was always the same: you push more compute into less space, and heat becomes your enemy. We played games with hot-aisle/cold-aisle containment, raised floors, in-row cooling units, and precision air conditioning. We got pretty good at it. Then AI happened, and all of that institutional knowledge aged out in about eighteen months.
The numbers tell the story plainly. A well-tuned enterprise rack in 2018 ran at 8 to 12 kW. A rack of H100 servers runs at 40 to 60 kW depending on configuration. NVIDIA’s GB200 NVL72, the current generation ultra-dense GPU rack for training and inference, draws roughly 120 to 132 kW nominal, with HPE’s documentation noting electrical provisioning at peak electrical design power significantly exceeding the nominal thermal load. Air cooling stops being practical somewhere around 50 kW per rack. That threshold is not a conspiracy – it is physics. You cannot move enough cold air past those fins fast enough to keep junction temperatures in a safe range when you are generating that much heat in a 20U chassis.
If you are planning GPU infrastructure for AI workloads and you have not started thinking seriously about liquid cooling, this article is for you. Not a vendor brochure, not a trends overview: a practitioner’s guide to what actually matters when you are making decisions that will shape your infrastructure for five years.
The Physics Problem You Cannot Route Around
Before getting into the approaches, it helps to understand why this is hard in a way that did not apply to CPU-era infrastructure.
A data center cools servers primarily through convective heat transfer: fans push air across hot components, and that warm air moves to a cooling unit where heat is rejected to the outside world. The efficiency of this depends on the temperature differential between the component surface and the incoming air, the volume of air you can move, and the thermal resistance of the heatsink. Air has a specific heat capacity of about 1 kJ per kilogram per Kelvin. Water has a specific heat capacity of roughly 4.18 kJ per kilogram per Kelvin. Water is about 800 times denser than air. The result is that water can carry away roughly 3,000 times more heat than the same volume of air. When you are trying to remove 130 kW from a single rack, you can do the math on how much airflow that requires versus how much water flow.
The GB200 NVL72 makes the physics visceral. The chassis ships with an integrated liquid cooling manifold. There is no air-cooled variant. NVIDIA made a product decision that for this generation of hardware, the thermal envelope simply cannot be met with air. You are not choosing between liquid cooling and air cooling for GB200 clusters; you are choosing between different liquid cooling architectures.
Even for H100 clusters, the industry largely converged on liquid cooling for production deployments. An eight-GPU H100 server generates roughly 5 to 6 kW of GPU heat alone, before accounting for CPUs, NVMe, and networking. Rack densities for H100 vary, but 40 to 80 kW per rack is common for production AI workloads. Air cooling can handle this range, but you pay for it in fan power, noise, and the overhead of extreme airflow management. The operating economics shift toward liquid well before you physically cannot cool with air.
The Three Architectures
Liquid cooling for data centers breaks into three main categories, and each makes different trade-offs on retrofit difficulty, maintenance overhead, density ceiling, and cost.
Direct Liquid Cooling (DLC)
Direct-to-chip cooling, often called DLC, is the most widely deployed approach for AI server clusters today. Cold plates – machined metal blocks with internal channels for coolant – are mounted directly on the GPU dies and CPU sockets. The fluid (typically water with corrosion inhibitors, or a proprietary water-glycol mix) circulates through the cold plate, absorbs heat, and returns to a heat exchanger or cooling distribution unit (CDU). The fluid never contacts the electronics. Pumps, valves, and manifolds distribute flow to each server.
The key strength of DLC is compatibility with standard rack-mount form factors. You can retrofit DLC into an existing data center more readily than immersion, because the servers themselves remain largely conventional. The plumbing is the new element: a manifold running down the side of the rack, quick-disconnect fittings at each server, and a CDU either integrated into the rack or in a row unit handling multiple racks.
At Computex 2026, CoolIT demonstrated what it describes as a 15 kW-per-cold-plate direct liquid cooling platform, which they claim delivers roughly four times the performance of their previous-generation single-phase DLC. This matters because individual GPU die heat flux is increasing faster than die area, and the cold plate thermal resistance is a real constraint. Vendors are engineering their way around it.
The limitation of DLC is that it does not cool everything. Fans still run on most DLC servers, handling the residual heat from components the cold plates do not cover: memory, power delivery, networking, drives. You still need some airflow in the rack and some room-level air handling. DLC typically achieves PUE values in the range of 1.1 to 1.2 in well-designed deployments, a meaningful improvement over all-air at 1.4 to 1.6, but not the theoretical minimum possible with full immersion.
For GB200 NVL72 deployments, DLC is the primary architecture. The integrated manifold in the chassis connects to facility water supply, and you deploy a CDU sized to the rack load. HPE’s documentation describes in-rack CDUs rated at 200 kW and in-row CDUs rated at 1.3 MW, with a single in-row CDU serving up to eight racks. You provision facility water meeting ASHRAE Liquid Cooling Guideline quality specifications, maintain inlet temperatures (specifications vary by OEM configuration, so check your specific deployment guide), and manage the hydronic loop with appropriate chemical treatment and flow monitoring.
Immersion Cooling
Immersion cooling submerges entire servers in a bath of dielectric fluid. The fluid is electrically non-conductive, so it can directly contact the electronics without causing shorts. There are two variants with meaningfully different characteristics.
Single-phase immersion keeps the fluid in liquid state throughout. You fill a tank with a mineral oil derivative or a synthetic dielectric such as 3M Novec or Engineered Fluids EC-110, submerge the servers, and pump the heated fluid to an external heat exchanger where it rejects heat to facility water. Single-phase is generally simpler to operate than two-phase because there is no phase-change management. Tank pressure is atmospheric. Maintenance involves pulling servers out of an oil bath, which requires procedures for fluid handling and contamination management.
The thermal ceiling of single-phase immersion is a genuine concern. Some vendors claim a per-chip TDP limit around 700W for single-phase immersion before convective heat transfer within the fluid becomes insufficient to keep junction temperatures in range. That is a vendor-side claim from cooling equipment manufacturers, not a neutral benchmark, so verify it against your specific hardware and fluid combination. What I have seen in practice: single-phase immersion works well for dense CPU and GPU deployments up to a certain heat flux per chip, and starts to struggle with the current generation of highest-TDP accelerators unless supplementary forced fluid circulation is added.
Two-phase immersion is more exotic. The fluid has a low boiling point, so heat from the components causes it to vaporize at the chip surface. The vapor rises to a condenser at the top of the tank, which rejects heat (typically to facility water), and the liquid rains back down. This evaporation-condensation cycle is highly efficient at removing heat from hotspots, because the latent heat of vaporization does the work rather than pure sensible heat capacity. Two-phase immersion can handle very high local heat flux.
The complications with two-phase are real. The fluids (historically fluorocarbon-based) are expensive. Environmental regulations around certain fluorocarbons have been tightening in the EU and elsewhere, which affects long-term fluid supply economics. The sealed system is harder to service. And the capital cost per kW of cooling capacity is higher than single-phase or DLC. This is why two-phase remains a niche approach used in specialized HPC environments and a few flagship AI research facilities, not the default for production AI clusters.
Rear-Door Heat Exchangers
The third option worth mentioning is the rear-door heat exchanger (RDHx): a radiator that mounts on the back of the rack and through which facility water circulates. The servers continue to use air cooling internally, but the hot exhaust air is cooled before it enters the room. RDHx does not touch the chips, so there are no per-server plumbing modifications. You just replace the rear door.
I include RDHx in this discussion because it is the transitional choice for brownfield data centers that cannot immediately commit to full DLC deployment. It handles the heat rejection problem at the room level without requiring per-server liquid integration. The practical ceiling for RDHx is generally around 30 to 40 kW per rack: enough to handle some GPU deployments, but not GB200 class hardware.
For anyone managing a mixed fleet of legacy infrastructure and new AI servers, RDHx often makes sense for the legacy racks while you deploy DLC or immersion for new AI capacity. Phased transitions are real; not every organization can rip and replace all at once.
What the Hyperscalers Are Doing
The hyperscalers have been designing for high-density liquid cooling for several years. They have advantages the rest of us do not: they can design purpose-built AI factory buildings from the ground up, negotiate custom OEM configurations, and spread the engineering cost across hundreds of megawatts of deployment.
For production GPU clusters at hyperscale, direct liquid cooling integrated at the chassis level has become the standard. The OEM server designs for GB200 come from the factory with DLC integrated. You are not retrofitting; you are deploying a fully liquid-cooled system from day one, connected to a facility water loop you build out to support it.
What is interesting is the cooling water source and treatment requirements. High-quality liquid cooling needs a consistent water supply with controlled chemistry: pH, conductivity, dissolved oxygen, microbiological contamination, and inhibitor levels all matter for long-term reliability of the aluminum cold plates and copper manifolds. This is a building infrastructure problem as much as an IT problem. Your facilities team and your IT team need to be in the same room planning this. If they are not talking to each other by the time you deploy, you will find out why they needed to be.
The neocloud providers – the GPU-focused cloud platforms that emerged to serve AI training and inference demand – have generally built on top of DLC from the start for their latest hardware generations. If you are renting H100 or GB200 capacity from neocloud providers like CoreWeave, Nebius, or Lambda Labs, the cooling infrastructure is their problem. If you are building bare metal infrastructure or planning a hybrid deployment with on-premises GPU clusters, it is yours.
The Economics and the PUE Argument
Power Usage Effectiveness is the ratio of total facility power to IT equipment power. An air-cooled enterprise data center typically runs at PUE 1.4 to 1.6. An efficient facility with hot-aisle containment and economizers might hit 1.2. Well-designed liquid-cooled facilities, including DLC deployments with efficient CDUs and facility hydronic design, commonly achieve PUE in the range of 1.05 to 1.15. Two-phase immersion under ideal conditions can approach 1.02 to 1.05.
This matters because of AI FinOps. At the power densities of AI infrastructure, that PUE difference is not cosmetic. If you are running 10 MW of GPU load at PUE 1.5, you are consuming 15 MW total. At PUE 1.1, that drops to 11 MW. At many data center locations, power is the dominant operating cost. The cooling infrastructure is a significant lever on total cost of ownership.
The capital cost argument runs the other direction. DLC infrastructure – CDUs, piping, manifolds, quick-disconnects, chemical treatment systems, sensors, and controls – adds cost over air-cooled deployment. Immersion adds more. The capital cost per kW of cooling capacity varies substantially by vendor and configuration, and I will not fabricate a figure here. What I will say is: the TCO equation increasingly favors liquid cooling for AI workloads when you factor in operating cost savings over a five-year depreciation cycle. This is especially true as power costs rise and power availability becomes a site constraint rather than a given.
The GreenOps angle is increasingly real too. Regulators in the EU and some US jurisdictions are starting to require water usage reporting and energy efficiency metrics from data center operators. A facility running PUE 1.5 for AI training is going to face questions it would not have faced three years ago.
What the Facility Actually Needs
Let me get concrete about what building a liquid-cooled AI cluster requires at the facility level, because this is where many operators discover they have a problem after it is too late to fix cheaply.
Structural load. Immersion tanks are heavy. A fully loaded single-phase immersion tank can weigh two to five tonnes or more depending on size. Your raised floor or concrete slab needs to support that. DLC CDUs and piping add weight too, though more manageably. If you are retrofitting into an existing facility, get a structural engineer involved before you commit to a configuration.
Hydronic infrastructure. You need a facility water loop: supply and return piping sized for the total flow, pumping equipment, water treatment and chemical dosing, expansion tanks, pressure relief valves, and flow sensors. For a serious AI cluster, this is industrial plumbing. It is not the same kind of piping your building uses for HVAC. Get a mechanical engineer who has designed data center hydronic systems, not a general HVAC contractor.
Heat rejection. All that heat has to go somewhere. Cooling towers, dry coolers, or plate heat exchangers reject heat to the outside environment. In cold climates, free cooling through economization is available for much of the year: the outdoor air or water source is cold enough to cool the facility water loop without mechanical refrigeration. This is one reason hyperscale AI infrastructure is concentrated in cold climates; the free cooling availability dramatically improves economics. Your CDU connects to this facility-level heat rejection loop.
Water quality. This is underestimated. DLC cold plates, manifolds, and quick-disconnect fittings are made from aluminum and copper. Mixing these metals in an aqueous system creates galvanic corrosion if chemistry is not controlled. The industry standard is to follow ASHRAE Liquid Cooling Guidelines and work from there. Ongoing monitoring and periodic draining, flushing, and inhibitor replenishment are operational requirements, not optional.
Electrical provisioning. A rack drawing 132 kW nominal needs circuit capacity to match, with headroom for peaks. This is often a bigger surprise than the cooling for organizations coming from CPU-era infrastructure where a rack might draw 15 to 20 kW. Your power distribution, UPS, and generator capacity may need significant upgrades before your AI cluster can run at nameplate capacity.
Connecting to the Rest of the AI Stack
Cooling architecture does not exist in isolation from the rest of your GPU cluster design. GPU cluster networking decisions affect rack layout, which affects cooling geometry. InfiniBand switch placement, optical interconnect runs, and the physical arrangement of compute nodes in the fabric all have to be designed alongside cooling. If you are running RDMA over Ethernet or InfiniBand for distributed training, the latency between nodes can be affected by how they are physically grouped in racks and rows, which is in turn constrained by cooling topology.
Parallel file systems for AI training introduce their own heat loads. A Lustre or WEKA cluster serving a GPU training job at full throughput generates significant heat in its own right. Storage nodes often do not need liquid cooling at the same density as GPU nodes, but if they share the same physical space, the room-level thermal design still needs to account for them.
For organizations running mixed AI and conventional workloads in the same facility, the cooling design for the AI zone needs to be isolated from conventional air-cooled areas. Hot containment failures in an AI zone can dump significant heat into adjacent spaces, which will cause problems you will not diagnose quickly.
A Decision Framework
When I am advising on cooling architecture for a new AI cluster deployment, the decision tree looks roughly like this:
If you are buying GB200 NVL72 or equivalent ultra-dense hardware: DLC is not a choice, it is the default. The hardware comes configured for liquid cooling. Your decisions are about CDU sizing, facility hydronic design, and heat rejection strategy.
If you are deploying H100-generation hardware at 40 to 80 kW per rack: You have genuine options. DLC delivers the best PUE economics and sets you up well for future hardware generations. Air cooling can work if your facility is already set up for it and you are not near capacity on power or cooling headroom. If you are deploying in an existing facility with limited modification budget, RDHx is worth considering as a transitional measure.
If you are running a new greenfield facility: Design for liquid from the start. The delta cost of including the hydronic infrastructure at greenfield versus retrofitting it later is significant. Retrofitting means disruption, potential facility downtime, and often paying two to three times more than doing it right the first time.
If you are evaluating immersion: Single-phase immersion is mature enough to run in production for dense CPU and moderate-TDP GPU workloads, and it is being actively deployed by some hyperscale operators. For the highest-TDP current-generation AI accelerators, verify thermal limits with your specific hardware before committing to a tank-based approach. Two-phase immersion is best reserved for specialized HPC and research environments where the engineering resources to support it are present.
If you are on cloud: The hyperscalers and neoclouds handle this for you. Your decision is which cloud to use and how to optimize costs for the GPU workloads you run on liquid-cooled infrastructure you never have to maintain. That is a legitimate choice, particularly for organizations without the facilities engineering expertise to run liquid cooling in-house. The trade-off is density control and the capital repatriation argument some organizations make when their GPU spend reaches the scale where on-premises becomes economically attractive again.
What Comes Next
The thermal trajectory is not getting easier. Industry speculation about NVIDIA’s forthcoming Rubin generation points to rack-scale power loads that would push beyond what current liquid cooling at the CDU level can handle, requiring a fundamental rethink of heat rejection at the row and facility level. I will not repeat specific numbers that come from a single uncorroborated source, but the direction is clear: each successive GPU generation increases power density faster than the market develops cooling infrastructure, and every few years we end up having this conversation again at a higher wattage.
CXL memory pooling, which is getting serious industry attention in 2026, adds another thermal element: CXL switch fabrics and pooled memory modules generate their own heat loads in the rack or at the row level. Designing cooling systems that can handle heterogeneous rack configurations with CPUs, GPUs, CXL switches, and pooled memory requires more careful thermal modeling than simply specifying a CDU rated for GPU TDP alone.
The AI infrastructure stack for GPU cloud workloads has become one of the most technically complex builds in our industry. The networking is hard. The software stack is hard. The cooling and power infrastructure is hard in ways that are unfamiliar to anyone whose data center experience is primarily software-defined. Knowing which problems you are solving with DLC versus immersion versus air versus hybrid, and which facility investments are worth making now versus later, is not glamorous work. It is the kind of work that determines whether your training jobs complete successfully or whether you are getting called at 2am about a cluster that throttled itself to 40% performance because a CDU valve failed and nobody noticed.
Build the thermal foundation right. Everything running on top of it will thank you.
This article covers cooling infrastructure for on-premises and colocation AI deployments. Cloud GPU rental from hyperscalers and neoclouds abstracts cooling away from the customer, but the economics of that abstraction ultimately trace back to the same physics discussed here.
Get Cloud Architecture Insights
Practical deep dives on infrastructure, security, and scaling. No spam, no fluff.
By subscribing, you agree to receive emails. Unsubscribe anytime.