Every production LLM deployment I have seen in the last two years hits the same wall. The GPU memory is full. The model weights eat the first slice, the KV cache from active requests chews through the rest, and you are left making hard choices: reduce batch sizes, cut context windows, or throw more GPUs at it. For teams running large-scale inference, this is not an occasional inconvenience. It is the dominant cost driver.
The hardware industry has known this problem was coming for a long time. CPU vendors and memory manufacturers have been working on a coherent interconnect that can attach large pools of cheap DRAM to compute nodes as a first-class memory tier. That standard is Compute Express Link, and in 2026 it has finally moved from research papers and trade show demos to the first real production deployments. Microsoft Azure is running Astera Labs CXL controllers in M-series virtual machines. Marvell’s Structera S 30260 CXL switch was announced at OFC 2026 and is sampling with customers. The software stack in the Linux kernel has stabilized enough to actually use.
This is not a solution for every AI workload, and I will be honest about where the limitations bite. But if you are running large LLM inference clusters, building out AI data center infrastructure, or just trying to understand where the memory hierarchy is going over the next three years, CXL memory pooling is the story you need to understand now.
Why the Memory Wall Exists
The memory wall is not a new concept in computer science, but the AI era has made it acutely painful in a specific way. Modern LLMs store two things in GPU memory: model weights and the KV cache. For a large model, the weights alone can consume most of the available HBM on a single GPU. Then every inference request generates a KV cache proportional to its context length. A 128K-token context at bfloat16 with a 70B-class model can consume tens of gigabytes of KV cache by itself.
The hardware has not kept pace. HBM bandwidth has grown impressively across H100 and B200 generations, but HBM capacity has grown much more slowly because stacking more DRAM dies is expensive and physically constrained. An H100 SXM carries 80 GB of HBM3. An H200 scales that to 141 GB of HBM3e. The Blackwell B200 generation adds further capacity, with various sources citing figures between 180 and 192 GB depending on variant and configuration. Those are real improvements, but they are chasing model sizes and context lengths that keep growing faster.
The result is that decode throughput, which is largely memory-bandwidth-bound, often underutilizes GPU compute because you cannot fit enough KV cache to saturate the attention computation. Teams work around this with paged attention (see LLM inference engine comparisons), speculative decoding (see speculative decoding in production), and disaggregated prefill/decode architectures. All of these help. None of them eliminate the fundamental constraint that you have a fixed amount of fast memory attached to each GPU.
CXL addresses this from a different angle. Instead of compressing what fits in existing memory, it expands the memory pool itself by adding a coherent, byte-addressable memory tier that CPUs can access at DRAM latency rather than NVMe or network latency.
What CXL Actually Is
Compute Express Link is a coherent interconnect protocol built on the PCIe physical layer. It was first published by the CXL Consortium in 2019 and has gone through several specification versions since. The key insight is that it reuses the PCIe electrical and physical layer, which means it can run on the same connectors and cables, while adding semantic layers on top that PCIe alone does not provide.
CXL defines three sub-protocols that are layered and named with the naming convention you would expect from a standards committee:
CXL.io is essentially PCIe semantics. It handles configuration, interrupts, and DMA, the same things PCIe does today.
CXL.cache lets a device maintain a cache of host memory with coherent snoop-based synchronization. This is what allows an accelerator to read and write host DRAM without going through explicit DMA transfers.
CXL.mem lets the host CPU access device-attached memory with full cache coherence. From the OS’s perspective, this memory shows up as a NUMA node.
The combination of these protocols gives you different device types. A Type 1 device uses CXL.cache, typically accelerators that leverage host memory. A Type 2 device uses both CXL.cache and CXL.mem, covering accelerators like GPUs that have their own memory but want coherent access to host DRAM. Type 3 devices use only CXL.mem and are pure memory expanders: DRAM modules behind a controller that expose their capacity to the host as additional memory.
For memory pooling discussions, Type 3 devices are the main character. You are adding cheap DDR5 capacity to a server via a CXL interface.
The versioning matters for what you can actually do:
CXL 1.x established the basics. One host, one device, coherent access.
CXL 2.0 introduced pooling semantics. A CXL switch can connect one host to multiple memory devices, and multiple hosts can connect to the switch with distinct, non-shared allocations. This is memory expansion at rack scale.
CXL 3.0 took the final leap to true shared memory. Multiple hosts can access the same CXL memory with coherent peer-to-peer semantics. The bandwidth specification doubles to 64 GT/s per lane. CXL 3.0 also adds fabric routing so you can build multi-hop topologies.
CXL 4.0 doubles the bandwidth again to 128 GT/s per lane, putting it in the same range as PCIe 7.0.
The Vendor Landscape in 2026
The CXL ecosystem has matured considerably. A few players are worth understanding at the controller and switch level.
Astera Labs is the company that brought CXL to production in the cloud first. Their Leo series of CXL Smart Memory Controllers support CXL 2.0 with up to 2 TB of memory capacity per controller. Astera Labs announced in November 2025 that their Leo controllers are deployed in Microsoft Azure M-series virtual machines, which the company describes as the first public cloud deployment of CXL-attached memory. The Azure deployment uses these controllers to expand memory capacity for memory-intensive workloads without requiring larger physical hosts.
Marvell is attacking the switching problem. Their existing Structera S 20256 is a CXL 2.0 switch in production today. At OFC 2026 in March, Marvell announced the Structera S 30260, a 260-lane CXL 3.x switch. Marvell claims it supports 16 or 32 CPUs or GPUs with up to 48 TB of shared memory and 4 TB/second of cumulative bandwidth. As of the time of writing, the S 30260 is sampling with customers and is not yet generally available. I treat that number as a vendor claim on a reference architecture, not a measured production figure.
Samsung, SK Hynix, and Micron are the memory manufacturers producing CXL DRAM modules. SK Hynix demonstrated pooled-memory AI agent serving and long-context inference at FMS 2026. The memory itself is commodity DDR5 DRAM packaged with a CXL controller die. These modules plug into standard E1.S or E3.S slots or conventional DIMM-like form factors.
The ecosystem extends to host-side support. AMD’s EPYC Genoa and Turin generations include native CXL 2.0 lanes, as do Intel’s Sapphire Rapids and Granite Rapids Xeons. This matters because CXL is a CPU-attached interconnect. Your GPU nodes still communicate with their HBM via NVLink and PCIe; the CXL memory tier is attached to the CPU complex that hosts the inference software, not directly to the GPU.
How CXL Memory Pooling Fits Into LLM Inference
This is where the architecture gets interesting, and also where I need to be honest about what CXL can and cannot do for LLM workloads.
The typical LLM inference serving setup has a CPU host managing inference servers like vLLM or SGLang, while the actual tensor computation runs on GPU hardware over NVLink and PCIe. The KV cache lives in GPU HBM because attention computation requires it to be there during the decode loop. CXL memory does not attach to the GPU HBM bus. It attaches to the CPU NUMA domain.
So CXL does not eliminate the GPU HBM constraint directly. What it does is enable tiered KV cache architectures where you can offload inactive or less-recently-used KV cache entries to the CXL memory tier instead of evicting them or pushing them to NVMe.
Think of it this way: when you are running many concurrent inference requests, some of those requests are between tokens during their decode step. Their KV cache is not being actively read by the attention kernel at that moment. Today you have two choices with that idle cache: keep it in HBM (expensive, limited), or evict it to host DRAM or NVMe (fast to evict, slow to restore). CXL memory gives you a third option: host-side memory that is faster than NVMe by a significant margin, coherent with the CPU, and cheap enough to deploy in very large capacities.
The latency numbers from research papers suggest CXL 3.0 Type 3 expanders operate around 100 to 200 nanoseconds for random reads, compared to roughly 70 nanoseconds for local DRAM and roughly 100 microseconds for NVMe SSDs. That is not as fast as local DRAM, but it is orders of magnitude faster than storage. For the pattern of restoring a KV cache to GPU HBM before the next prefill, the restoration time from CXL memory is dominated by the PCIe or NVLink DMA transfer to the GPU anyway, not the CXL memory access latency itself.
The research group at the University of Washington’s Allen School has published work showing that CXL-backed KV cache overflow can allow serving longer context lengths and higher concurrency without adding GPUs. The SNIA storage community has explored architectures where a block-level cache manager in CXL memory handles shared prefixes, so multiple GPU instances running independent requests can all reference the same system prompt or few-shot examples without each storing a separate copy. Given that shared prefixes in production deployments can represent a significant fraction of total KV cache capacity, this is a meaningful efficiency gain.
The Software Stack: Linux CXL Subsystem
The hardware story is only half of it. CXL memory needs OS and kernel support to be useful, and this is where the picture is more nuanced than the vendor slides suggest.
Linux has had a CXL subsystem since kernel 5.12, and it has matured substantially through the 5.x and 6.x series. The core functionality you need is there: CXL Type 3 devices enumerate as DAX (Direct Access) devices, which can be configured as memory via the KMEM driver, appearing to the OS as additional NUMA nodes.
The tiering challenge is that you want hot data in local DRAM and warm or cold data in the CXL memory tier, with the OS making those placement decisions automatically. Linux has two relevant mechanisms here:
Auto NUMA (automatic NUMA balancing) will migrate pages between NUMA nodes based on access frequency. On a system with local DRAM as NUMA node 0 and CXL memory as NUMA node 1, Auto NUMA will tend to pull frequently-accessed pages toward the CPU-local node. This works, but the default migration thresholds are not tuned for CXL workloads, and the overhead of page migration can affect latency-sensitive inference serving.
DAMON (Data Access Monitor) is a more recent kernel mechanism that provides higher-level access pattern monitoring with policy-based actions. With DAMON, you can write tiering policies that match your specific workload: for instance, keeping inference engine overhead in local DRAM while placing KV cache buffers in CXL memory by default.
For inference workloads specifically, the most practical approach I have seen in early-stage deployments is explicit memory placement using numactl or mmap with mbind system calls to pin allocations to specific NUMA nodes. The inference framework manages which tier each allocation lives in, rather than relying on the OS to make the right call. vLLM and SGLang are both adding CXL-aware memory management capabilities, though as of mid-2026 this is more in the prototype stage than production-hardened.
The kernel CXL subsystem also provides tools for managing memory pools. The cxl command-line utility can enumerate devices, configure memory regions, and query DPA (Device Physical Address) allocations. For teams used to managing NVMe namespaces, the operational model is familiar.
When CXL Memory Pooling Actually Makes Sense
My twenty years in data center architecture have taught me to be skeptical of technology that solves every problem. CXL memory pooling is a real solution, but it has a specific sweet spot.
CXL is a good fit when you are running LLM inference with long context windows where KV cache overflow is your primary constraint. If your decode step is memory-bandwidth-bound rather than compute-bound, the ability to run larger effective batch sizes by spilling KV cache to a fast local memory tier translates directly into better throughput per GPU. This is the story that Astera Labs and Microsoft Azure are telling with the M-series deployment, and it is credible.
CXL is also a fit for memory-intensive CPU workloads that have grown beyond what DRAM alone can provide: large in-memory databases, graph analytics, or certain ETL jobs that want to hold large datasets resident in memory for performance. These workloads often tolerate the slightly higher latency of CXL memory because they are not microsecond-sensitive.
CXL makes less sense for GPU training workloads. Training is compute-bound and heavily dependent on NVLink and InfiniBand for gradient synchronization across GPUs (see GPU cluster networking). The bottleneck is not CPU-attached memory capacity. CXL memory pooling would not improve training throughput.
CXL also makes less sense if your inference workload is prefill-heavy rather than decode-heavy. Prefill is compute-bound and does not stress the KV cache in the same way. If you are running a lot of short-context requests with minimal decode steps, the benefit of expanded KV cache capacity is marginal.
The cost calculus is worth doing carefully. DDR5 DRAM behind a CXL controller is cheaper per gigabyte than HBM, but it is not free. A CXL memory expander module with 512 GB of DDR5 costs real money, and you need a CXL-capable switch to pool it across multiple hosts. The total system cost comparison versus simply buying additional GPU memory capacity (where available) or adding more GPU nodes is not obvious without workload-specific modeling.
For the GPU FinOps picture, CXL memory is most compelling when you are compute-limited on GPU utilization but memory-limited on batch size. If your GPUs are sitting at 30% utilization because you cannot fit enough concurrent requests due to KV cache overflow, that is the signal that CXL memory is worth evaluating. I cover the broader GPU cost picture in AI FinOps.
What the Cooling and Power Picture Looks Like
CXL memory expanders consume power, and in dense AI data center configurations, thermal density is already a constraint. A typical 512 GB CXL DRAM module might draw 15 to 25 watts. At rack scale, if you are filling a CXL switch with 48 TB of memory across many modules, the aggregate thermal load is meaningful.
This is compounded by the CXL switch itself, which Marvell has not published detailed power figures for on the Structera S 30260 at the time of this writing. PCIe switch chips of similar lane counts typically consume 30 to 50 watts, so a CXL switch at that scale is likely in the same ballpark.
For data centers already deploying direct-to-chip cooling for GPU nodes (see AI data center liquid cooling), the incremental thermal load from CXL memory expanders is manageable. For air-cooled environments already at the thermal limit of their racks, adding CXL memory capacity requires careful power budget planning.
The Microsoft Azure Proof of Concept
The most concrete production evidence available is the Azure M-series deployment with Astera Labs Leo controllers. What Microsoft has deployed is a CXL 2.0 expansion architecture, not the full CXL 3.0 shared-memory pooling model. One host accesses one or more CXL memory controllers as additional NUMA nodes. This is memory expansion, not rack-level pooling.
That distinction matters. The marketing material around CXL sometimes conflates these two things. Memory expansion (one host, more DRAM via CXL) is available today and production-deployed. True memory pooling (multiple hosts sharing a common CXL memory pool via a switch, with coherent peer-to-peer access) requires CXL 2.0 or 3.0 switch hardware and more complex software coordination. The Marvell S 20256 switch enables CXL 2.0 pooling, but production deployments of multi-host CXL pooling are not yet common outside of research environments.
Microsoft’s Pond research project, published at ASPLOS 2023, modeled the economics of disaggregated memory for cloud data centers. Their analysis suggested that pooling memory across servers could reduce total DRAM requirements in a cloud fleet while improving overall memory utilization. The Azure deployment is presumably the operational follow-through on that research direction.
For teams outside of hyperscaler scale, the near-term practical opportunity is CXL 2.0 memory expansion: adding 512 GB to 2 TB of CXL DRAM to compute nodes that are memory-constrained, without the complexity of a pooling switch. This is the version you can actually buy and deploy today.
The Parallel File System Comparison
There is a useful analogy to parallel file systems here. When teams building AI training clusters first realized that local NVMe was not fast enough to feed GPU training jobs, they turned to parallel file systems like Lustre and WEKA (see parallel file systems for AI training). These systems provide a shared, high-bandwidth storage tier that multiple nodes can access concurrently. The operational model was unfamiliar to most infrastructure teams, but the performance need was real enough that adoption happened.
CXL memory pooling is following a similar arc. The need is clear: inference workloads want more memory capacity than a single server’s DIMM slots can provide. The technology is arriving: controllers and switches are in production or sampling. The operational patterns are unfamiliar: NUMA-aware memory tiering, CXL device management, coherent pooling semantics. Infrastructure teams that build expertise in this area now will be in a strong position when the workloads push them there.
What I Would Do Right Now
If you are evaluating your LLM inference infrastructure for the next twelve to eighteen months, here is my practical take.
Instrument your inference workload to understand where HBM goes. Tools like DCGM (see GPU infrastructure observability) can show you memory utilization at the per-request level. If you are evicting KV cache entries because of HBM pressure, you have a real CXL use case.
Start with CXL memory expansion, not pooling. The single-host CXL 2.0 model is simpler, available, and gives you operational experience with the technology before you take on the switch management complexity of rack-scale pooling. Astera Labs Leo and similar controllers can be evaluated without a specialized CXL switch.
Talk to your server vendors about CXL-capable platforms. AMD EPYC Genoa and Turin, and Intel Sapphire Rapids and Granite Rapids, all have native CXL support. If you are buying server hardware now for an inference cluster, CXL-capable CPUs are worth specifying even if you are not deploying CXL memory yet.
Watch the Linux kernel tiering tooling closely. The auto-tiering story in the Linux kernel is improving with each release. By kernel 6.12 and 6.13, the DAMON-based tiering policies are materially better than the 6.6 baseline. If you deploy CXL memory today, expect to do manual placement via numactl. In a year, that overhead should be much more automated.
For the large-scale data center build, the CXL 3.0 pooling story with switches like the Marvell Structera S 30260 is genuinely compelling for reducing the total DRAM you need per server. But treat this as a 2027 architecture rather than a 2026 deployment. The switch hardware is sampling, the software is research-grade, and the operational playbook does not exist yet. Evaluate it in a test environment. Do not build your production inference fleet around it.
The AI memory wall is real, and CXL is one of the most technically coherent solutions to it that the hardware industry has produced. The deployment timeline is measured in years, not quarters. Getting your hands dirty with it now is the right call. Building critical infrastructure dependencies on it today is premature.
For teams also evaluating custom AI silicon like Trainium and TPUs, CXL memory pooling and custom accelerators are complementary, not competing, answers to the inference efficiency problem. The memory architecture question and the compute architecture question are separable, and both are worth working on in parallel.
The CXL Consortium publishes the specifications at computeexpresslink.org. The Linux CXL subsystem documentation is maintained in the kernel source under Documentation/driver-api/cxl/. Marvell’s Structera S 30260 product brief is available on Marvell’s website. The Astera Labs Leo deployment on Azure was announced by Astera Labs in November 2025.
Get Cloud Architecture Insights
Practical deep dives on infrastructure, security, and scaling. No spam, no fluff.
By subscribing, you agree to receive emails. Unsubscribe anytime.