llm-d: The Kubernetes-Native Framework Solving Disaggregated LLM Inference Before Your GPU Budget Explodes

I have been deploying LLM inference infrastructure since before most organizations had a firm opinion about what a GPU even was. And in twenty years of building distributed systems, I have watched the same pattern repeat: a technology that works beautifully at small scale develops structural problems at production concurrency, and the community has to build new abstractions to fix them. That is exactly what happened to LLM inference in 2025, and it is exactly what llm-d is designed to solve.

If you are running vLLM on Kubernetes today and you are happy with your p99 latency at moderate concurrency, good. Enjoy it. But when you push past a few hundred simultaneous requests, or when you start mixing short chat completions with long document ingestion jobs on the same cluster, you will hit the wall that makes disaggregated inference necessary. llm-d is the Kubernetes-native answer to that wall, and it became a CNCF Sandbox project in March 2026 with backing from IBM, Red Hat, Google Cloud, NVIDIA, CoreWeave, AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI. That is not a collection of names assembled for a press release. It is the list of organizations that had all tried to solve this problem independently and decided to stop reinventing the wheel.

The Prefill-Decode Saturation Problem

To understand why llm-d exists, you need to understand why monolithic LLM serving falls apart under load.

When a request arrives at a vLLM instance, it goes through two distinct computational phases. Prefill processes the prompt: it is compute-bound, parallelizable across the input tokens, and produces the initial KV cache. Decode generates the response tokens one at a time: it is memory-bandwidth-bound, iterative, and its computational footprint grows as the output length increases.

The problem is that these two phases compete for the same GPU. A single vLLM pod handles both, which sounds efficient until you think about what happens under concurrency. A long-context prefill for a 64K token document ties up the GPU while decode requests queue. The decode phase for a running generation blocks new prefills from starting promptly. The GPU utilization metric looks fine at the aggregate level while actual latency for individual request types is drifting badly.

I saw this firsthand running inference for an enterprise document processing pipeline. We had a mixed workload: short conversational completions and long-context summarization requests hitting the same vLLM endpoints. Average GPU utilization looked healthy on the dashboard. But our conversational p99 latency was dramatically worse than the SLO because those requests were queuing behind document summarizations that had grabbed the GPU for prefill. The fix was to separate the workloads manually, which required operational overhead and static capacity decisions that aged poorly as traffic patterns shifted.

Disaggregated inference solves this by running prefill and decode on separate GPU pools that can scale independently. Prefill pods are compute-optimized; decode pods are memory-bandwidth-optimized. Requests route to prefill first, which computes the KV cache, then hands off to a decode pod that generates the response. The handoff itself requires transferring the KV cache, which is the engineering challenge, but the throughput improvement at scale justifies the complexity.

What llm-d Actually Is

llm-d is not another inference engine. It does not replace vLLM. It is the orchestration and routing layer that makes disaggregated inference work in Kubernetes, connecting the Gateway API inference extension to vLLM pods and managing the KV cache routing logic that would otherwise require you to build something bespoke.

The project was launched in May 2025 as a collaboration between Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. It was accepted into the CNCF Sandbox in March 2026. The timing matters: this is not a research project. It was built by people running production inference workloads who needed a standardized way to do disaggregated serving and decided to build it as an open standard rather than keep their implementations proprietary.

The architecture has three layers. The Kubernetes Gateway API Inference Extension provides the custom resource definitions and routing protocol. The llm-d Router is the intelligent control plane that makes routing decisions. And vLLM (or another compliant inference engine) does the actual work on the GPU.

Understanding the separation between those layers is important. The Gateway API Inference Extension is a kubernetes-sigs project, not part of llm-d itself. llm-d builds on top of it. This is a deliberate design decision: the routing protocol is open and standardized, so that multiple inference engines and multiple gateway implementations can participate in the same ecosystem. KServe v0.17 integrates with llm-d using the same extension. Other serving frameworks are following.

The Three Core Abstractions

If you are going to deploy llm-d, you need to understand its three primary Kubernetes abstractions before you touch a YAML file.

InferencePool is the operator-level abstraction. It describes a group of model server pods as a unit, analogous to a Kubernetes Service but aware of the LLM serving protocol. An InferencePool knows what model is loaded, what hardware the pods run on, and what the health semantics are for an LLM backend. Platform teams own InferencePools. They control the infrastructure decisions: GPU type, pod count, resource limits, scaling policy.

InferenceModel is the developer-facing abstraction. It maps a stable public name (think “gpt-4-chat” or “llama-3-70b-instruct”) to an InferencePool. AI and ML engineers manage InferenceModels. They specify priority tiers for different request types, criticality levels, and optionally the target model within a pool. This separation is intentional: the team that trains and deploys the model should not need to understand GPU node affinity rules, and the platform team that manages GPU capacity should not need to understand the difference between model variants.

The split between InferencePool and InferenceModel is the same separation of concerns that made Kubernetes Deployments and Services useful. Platform engineers manage infrastructure, application engineers manage workloads. llm-d applies that principle to inference.

The llm-d Router is the intelligent data plane component sitting between the Gateway and the inference pods. It receives inference requests, examines their context, and makes routing decisions based on KV cache state, pod load, request priority, and hardware characteristics. This is where the intelligence in “intelligent routing” actually lives.

KV-Cache Aware Routing

KV-cache aware routing is the capability that makes llm-d’s latency improvements concrete rather than theoretical.

When a vLLM pod has already processed a prefix (a system prompt, a document chunk, a conversation history), that prefix lives in its KV cache. Routing a new request with the same prefix to that pod avoids recomputing the prefix entirely, which for long system prompts can be significant savings. Without intelligent routing, you route round-robin and probabilistically lose that cache advantage as your pod count grows.

The llm-d Router tracks which pods have cached which prefixes and prefers routing requests to pods that will get a cache hit. For deployments with stable system prompts (which is most of them; the same few system prompt templates account for the majority of requests in practice), this dramatically reduces time-to-first-token for requests with long prompts.

This is not a new idea. I had teams implementing hash-based routing for KV cache locality in 2024. The problem was that hand-rolled solutions were fragile: they did not handle pod restarts gracefully, they did not account for load, and they did not work across disaggregated prefill and decode phases. The llm-d Router handles all of that, updates its routing table as pods restart, and balances cache affinity against load to avoid overloading any single pod even when it has a coveted cache entry.

The Kubernetes Gateway API is the transport layer under all of this. The inference extension adds two things to the Gateway API: the InferencePool target backend type, and the ext-proc filter hook that allows the router to intercept requests before they are forwarded and make routing decisions. Gateway implementations that support the extension (kgateway and Envoy Gateway are the current production-ready options) handle the actual request forwarding.

Disaggregated Prefill and Decode in Practice

The disaggregation feature is the more advanced deployment mode. Standard single-phase serving with KV cache routing is useful at moderate scale; disaggregated serving is for high-concurrency workloads where the prefill-decode competition is measurably hurting your latency.

In disaggregated mode, you deploy two sets of vLLM pods: prefill pods and decode pods. The prefill pods receive the request, run the prefill computation, produce the KV cache, and transfer it to a decode pod. The decode pod picks up from the cached state and generates the response tokens.

The KV cache transfer is the expensive part. It happens over the network between pods, and for large models with long contexts, that transfer is not trivial. The tradeoffs here are real: you are exchanging transfer overhead for the ability to scale prefill and decode independently. If your workload has short contexts and low concurrency, disaggregated serving may add overhead without meaningful benefit. This is not a configuration you turn on by default.

For the workloads where it makes sense: sustained high concurrency, long-context prompts, or mixed workloads with significantly different compute profiles, the throughput improvement is measurable. The prefill GPUs can be saturated with compute work without blocking decode progress, and the decode pool can be sized for memory bandwidth rather than compute, which means you can choose different GPU types for each phase.

Integration with the Kubernetes AI Stack

llm-d does not exist in isolation. Understanding where it fits in the broader Kubernetes AI infrastructure stack is necessary before you commit to deploying it.

vLLM is the default and recommended inference engine. vLLM added native support for the llm-d disaggregation protocol, and the llm-d Router was designed around vLLM’s serving semantics. You can run llm-d with other OpenAI-compatible inference servers, but vLLM is where the integration is deepest. If you are already reading the LLM inference engines comparison, understand that vLLM plus llm-d is the Kubernetes-native inference stack as of mid-2026.

KServe integrates with llm-d through the LLMInferenceService CRD introduced in KServe v0.17. KServe handles model storage, model loading, autoscaling, and canary deployment logic; llm-d handles the runtime routing and disaggregation. The combination gives you a full operator-managed inference lifecycle. If your organization already runs KServe for classical ML model serving, adding llm-d for generative workloads is the natural path.

Kubernetes DRA (Dynamic Resource Allocation) is the GPU scheduling mechanism underneath. The DRA guide covers this in depth, but the short version is that DRA gives the scheduler structured knowledge about GPU resources so it can make intelligent placement decisions. llm-d pod specs use DRA resource claims to request GPU capacity, and the scheduler places prefill and decode pods on appropriate hardware.

Kueue manages batch admission control for long-running inference jobs and batch inference workloads on the same cluster. If you are running Kueue for batch scheduling alongside llm-d for online inference, you need to think carefully about how they interact with your cluster’s GPU capacity. The GPU cluster networking guide is also relevant: if you are doing disaggregated inference, KV cache transfers between pods benefit from high-bandwidth, low-latency interconnects.

AI Gateways sit above the llm-d layer. The AI gateway architecture covers semantic caching, rate limiting, and model routing at the API level. An AI gateway like Kong AI Gateway or LiteLLM Proxy handles the concerns above the inference layer: which model to route to, rate limiting per tenant, cost allocation. llm-d handles the concerns below: which pod within a model’s serving fleet to route to. These layers compose rather than compete.

llm-d vs. Alternatives

When I talk to teams evaluating llm-d, the same comparison questions come up. Here is how I think about them.

vs. Ray Serve: Ray distributed computing is a general-purpose distributed compute framework that can serve LLMs. Ray Serve handles disaggregated inference via Ray’s actor model. It is more general, which means more flexibility and more operational complexity. llm-d is Kubernetes-native: it uses standard CRDs, Gateway API, and Helm charts. If your team already knows Kubernetes operations and does not need Ray’s broader data processing capabilities, llm-d has a lower operational surface area. If you are already running a Ray cluster for training and offline processing, Ray Serve may be a natural fit for inference too.

vs. NVIDIA Dynamo: NVIDIA Dynamo is NVIDIA’s disaggregated inference framework. It also disaggregates prefill and decode, but it runs above Kubernetes as an orchestration layer rather than as a Kubernetes-native citizen. It integrates deeply with NVIDIA’s hardware and NIM (NVIDIA Inference Microservices). llm-d runs on any GPU vendor’s hardware (AMD, Intel Gaudi, and others have committed support in addition to NVIDIA). If you are committed to a pure NVIDIA stack and want the deepest possible hardware integration, Dynamo is worth evaluating. If you need hardware flexibility or want to avoid vendor lock-in, llm-d is the better choice.

vs. plain vLLM with manual sharding: This is the most common baseline. Running vLLM without llm-d works fine at moderate concurrency and gives you maximum control. The question is what “moderate” means for your workload. Teams that have hit the prefill-decode saturation problem know when they need llm-d. If you are not sure whether you need it, you probably do not yet.

Production Deployment Considerations

Deploying llm-d in production requires thinking through a few things that the getting-started guide does not cover.

Cluster prerequisites: llm-d requires Kubernetes 1.29 or later, a Gateway API implementation that supports the inference extension (kgateway or Envoy Gateway with the ext-proc filter), and vLLM 0.8 or later on your inference pods. The Gateway API version matters: you need v1.1 or later for the InferencePool backend type. Check your managed Kubernetes offering’s Gateway API support before starting.

Monitoring: The llm-d Router exposes Prometheus metrics for routing decisions, cache hit rates, and request latencies per pool and model. Wire these into your observability stack before you need them. KV cache hit rate is the metric to watch first: if it is lower than you expect, your routing configuration or pod count may need adjustment.

Scaling policy: InferencePools support KEDA-based autoscaling. The right scaling metric for LLM inference is usually queue depth or time-to-first-token percentile rather than CPU or GPU utilization. GPU utilization is a lagging indicator for inference workloads. Build your scaling policy around request latency, not resource saturation.

The FinOps angle: Disaggregated inference changes your cost model. Instead of a single GPU type for all inference, you now have two pools with potentially different hardware. Prefill pods benefit from high compute throughput; decode pods benefit from memory bandwidth. This means you can use compute-optimized instances for prefill and memory-bandwidth-optimized instances for decode, which may reduce cost compared to running everything on the most expensive GPU tier. The AI FinOps guide has relevant background on GPU cost optimization strategy.

GitOps and deployment lifecycle: The Helm chart structure introduced in the llm-d project is modular: the router, the gateway extension, and the monitoring components are separate subcharts. This works well with GitOps tooling because you can deploy and manage each layer independently. Manage InferencePools and InferenceModels through Git like any other Kubernetes resource.

Is It Ready for Production?

This is the honest question, and the honest answer is: yes, with the caveat that it is a CNCF Sandbox project and the API surface will stabilize over the next two to three releases.

The code is real and production-tested. The founding organizations donated it because they were running it in production, not because they thought it was a good idea in theory. KServe’s adoption of llm-d as its generative inference backend is a meaningful signal: KServe is not a research project.

The areas to watch are the disaggregated mode specifically (more mature organizations have had it in production for longer; general availability adoption is newer) and the Gateway API implementation support (kgateway and Envoy Gateway are solid; Istio’s support is coming). The InferencePool and InferenceModel APIs should be treated as stable; the disaggregation-specific configuration options are more likely to evolve.

My recommendation for teams evaluating llm-d: start with single-pool deployment with KV-cache aware routing. That is the less operationally complex mode, it delivers measurable benefits for most workloads, and it gets your team familiar with the CRDs and routing behavior. Add disaggregated prefill/decode once you understand your traffic patterns well enough to know whether the added complexity is justified.

For teams already hitting the prefill-decode saturation problem with hand-rolled solutions or with static workload segregation: llm-d is the right abstraction. The community building it is serious, the Kubernetes integration is first-class, and the alternative is maintaining bespoke routing logic that will not benefit from any of the ecosystem improvements landing over the next year.

The inference infrastructure problem in Kubernetes is being solved in the open. That is new. Take advantage of it.

Updated September 3, 2026: This article reflects llm-d’s current state as a CNCF Sandbox project (accepted March 2026) with production integrations in KServe v0.17 and kgateway.