Context Engineering for Production AI: How to Treat the Token Window as the Scarce Resource It Actually Is

Twenty years in this industry and I have watched the bottleneck move around like a hot potato. First it was CPU. Then memory. Then network. Then storage IOPS. Now, with AI workloads taking over every roadmap, the new scarcest resource on the board is the token window. More specifically: what you put into it, how you structure it, and what you leave out.

This discipline has a name now. Context engineering. The term has been floating around since at least 2024, but it crystallized into something practitioners actually argue about in 2026. And in my opinion, getting it right is the single biggest lever teams have on both the cost and the reliability of their LLM-based systems. Not model choice. Not inference engine tuning. Context engineering.

Let me be direct about what I mean. Prompt engineering is about crafting good instructions. Context engineering is about managing the entire information environment that surrounds those instructions: the system prompt, the retrieved documents, the conversation history, the tool definitions, the agent scratchpad, and the structured data you pulled from your database five seconds ago. All of that competes for space in a finite window. Every token you add has a cost. Every token you add also changes what the model pays attention to. Context engineering is the practice of making those tradeoffs deliberately.

Why This Suddenly Matters More

Context windows have grown dramatically. As of late 2026, frontier closed-source models from Anthropic, Google, and OpenAI are largely operating at 1 million token windows, with some advertised at 2 million or higher. (A few vendors claim 10 million token windows, though no published benchmark I have seen demonstrates quality holding at anything close to that range.) This sounds like it solves the problem. In practice, it makes the problem worse, for two reasons.

First, filling a million-token window on every request is expensive. The cost is not uniform across providers and changes frequently, so I will not quote specific numbers here; check your vendor’s current pricing documentation. But the order-of-magnitude point holds: a system that sends 800,000 tokens of context on every API call will quickly produce infrastructure bills that make leadership uncomfortable. I have watched engineering teams build genuinely impressive AI features only to discover their marginal cost per user session is structurally unsustainable. Context engineering is, at its core, a FinOps discipline as much as a reliability one. For a broader view of where AI costs concentrate, see our guide to AI FinOps and GPU cost optimization.

Second, more tokens does not mean better reasoning. Models have limits on what they actually attend to effectively within a large context. Burying a critical piece of information deep in a 500,000-token context is not the same as putting it at the top of a tight, focused 8,000-token context. One of the consistent patterns I see in production debugging is agents that fail not because the model is bad but because the relevant fact was present in context and the model still missed it. That is a context engineering failure, not a model failure.

The Four Layers of Context

When I audit an AI system’s context strategy, I think in four layers, stacked from most stable to most dynamic.

System prompt (near-static). This is the identity and behavioral layer: who the agent is, what it can and cannot do, output format expectations, tool definitions, and any durable policies. The system prompt changes rarely, maybe on deploys or user configuration changes. Because of this stability, it is the single best candidate for prefix caching. If your inference stack supports it (and most production deployments should be using a stack that does), a stable system prompt means the KV cache of that prefix gets reused across virtually every request. For a deep dive on the mechanics of this, see our guide to LLM prompt caching and prefix caching.

Retrieved knowledge (request-scoped). This is where RAG lives: documents retrieved from a vector store, rows from a database query, records from an API call, chunks of a user’s uploaded file. This layer is dynamic per request but its content comes from a managed corpus. The key architectural tension here is between retrieval precision and retrieval recall. If you retrieve too few chunks, you miss the answer. If you retrieve too many, you fill the context with noise, raise costs, and degrade reasoning quality. My rule of thumb from production systems: retrieved content should be the minimum amount that would allow a thoughtful human expert to answer the question correctly, not the maximum amount that might theoretically be relevant. For the full treatment of retrieval architecture, see our RAG architecture production guide.

Conversation history (session-scoped). For multi-turn interactions, you need some representation of what has been said. But verbatim conversation history compounds quickly. A hundred-turn conversation can easily consume more tokens than your entire system prompt and retrieved context combined. Compaction is the standard solution: summarize older turns rather than carrying them verbatim. The question is when and how. I have seen teams do this naively with a rolling summary that loses important details, and I have seen teams do it with structured episodic compression that maintains a retrievable record. The structured approach wins on reliability. See our dedicated article on AI agent memory architecture for the implementation patterns.

Agent scratchpad (step-scoped). This is the working memory of a running agent: tool outputs, intermediate reasoning, partial results from previous steps in a multi-step task. This layer is the wildest and most frequently mis-engineered. Tool outputs in particular can be enormous: a web page fetch might return 50,000 tokens of HTML, a database query might return thousands of rows. If you stuff raw tool outputs directly into context at every step, you will blow your token budget within a few agent turns. The correct pattern is to process and summarize tool outputs before they re-enter context. A web page should be reduced to its relevant passages. A database result set should be reduced to the aggregate or the specific rows that matter.

KV Cache Awareness Is Not Optional

If you are running inference at any meaningful scale, you need to understand how your context structure interacts with KV cache behavior. I will keep this practical.

The KV cache stores the intermediate attention computations for tokens that have already been processed. When a new request arrives with a prefix that matches what is already cached, the model can skip recomputing those tokens and jump straight to the new content. This is the mechanism behind prefix caching, and it is why keeping your system prompt byte-for-byte identical across requests matters so much architecturally.

The practical implication is that context assembly order matters beyond just “what makes sense to the model.” You want your most stable, least-changing content at the beginning of the context, and your most dynamic content at the end. System prompt first. Retrieved documents (if the same documents appear across requests) next. Conversation history after that. The freshest, most request-specific content last.

A concrete failure mode I have encountered: a team was injecting a timestamp into their system prompt at every request for audit reasons. This broke prefix caching on every call because the first token of context was different for every request. Moving the timestamp to a user message instead of the system prompt was a one-line fix that dramatically improved cache hit rates. The model’s behavior was unchanged. The infrastructure cost dropped substantially.

For teams using vLLM, SGLang, or TensorRT-LLM as their inference backend, each handles prefix caching somewhat differently, with different eviction policies and granularities. If you have not read the documentation for your specific inference engine on this topic, stop here and go do that. Our LLM inference engines comparison covers the relevant differences.

Context Compression Techniques

When you cannot avoid having a large amount of content that needs to be in context, compression is the next tool. There are several techniques, each with different tradeoffs.

Summarization. The most obvious approach: use the LLM itself (or a smaller, cheaper model) to summarize long content before injecting it into the primary context. Works well for narrative content like emails, documents, and conversation history. Loses precision on structured data, code, and content where exact wording matters.

Extraction. Rather than summarizing, identify and extract only the specific fields, sentences, or passages that are relevant to the current task. This requires knowing what the task is before assembling context, which works well for structured agentic workflows but is harder to apply in open-ended conversational systems.

Truncation with smart selection. When a retrieved document is too long, truncate it, but truncate intelligently: prefer passage-level chunking during indexing so you are selecting whole meaningful units, and prefer chunks from the beginning of documents (where authors tend to put the most important information) when relevance scores are similar.

Lossless token reduction. There are techniques for reducing token count without losing semantic content: removing whitespace from structured data, using abbreviated forms of long but predictable strings, compressing repeated patterns in structured output. These are less dramatic than summarization but add up on high-volume systems.

My general guidance: be more aggressive about compression in the agent scratchpad layer, less aggressive in the retrieved knowledge layer. The scratchpad is working memory; losing a little fidelity on intermediate steps is usually acceptable. Retrieved knowledge is where the answer actually lives; losing fidelity there causes factual errors.

The Long Context vs. RAG Decision

One of the architectural questions that comes up constantly is whether to use a large context window to hold all the relevant information directly, or whether to use retrieval to pull in only the most relevant pieces. This used to be a clear tradeoff: RAG added complexity but large context was prohibitively expensive. Now that 1-million-token windows are common, the calculus is more nuanced.

My current guidance: use RAG as the default for any corpus that changes frequently or is large relative to what any single request actually needs. Long context works well when the entire corpus is small enough to fit, relatively stable, and when tasks require the model to reason across many parts of the content simultaneously (pattern detection, synthesis across documents, code comprehension across a large codebase).

Do not treat long context as a replacement for good retrieval design. I have seen teams adopt the “just put everything in context” approach as a way to avoid building a proper retrieval pipeline, and it works, right up until the corpus grows, the cost becomes unsustainable, or quality degrades because the model starts hallucinating patterns it thinks it saw somewhere in a 900,000-token window.

The hybrid approach, where you use retrieval to identify relevant sections and then include a larger window of surrounding context around those sections, often gives you the best of both worlds. This is sometimes called “contextual chunking” in the RAG literature.

Context Poisoning and Robustness

There is a security dimension to context engineering that I do not see discussed enough in architecture reviews. If your agent accepts any external content into its context, that content is an attack surface.

Context poisoning is the introduction of instructions or misleading information into retrieved content, user input, or tool outputs that causes the model to behave in unintended ways. This is the mechanism behind most prompt injection attacks on agentic systems. A malicious document in your retrieval corpus, a manipulated API response, or injected instructions in a user-supplied field can all redirect an agent’s behavior if you are not careful about how external content is demarcated in context.

The architectural mitigations are not complicated, but they require deliberate design. External content should always be clearly delimited from instructions; most providers have guidance on this in their system prompt design documentation. Tool outputs should be treated as data, not as instructions, by clearly labeling them as such in context structure. For agents with access to sensitive capabilities, consider running external content through a filter model before it enters the primary agent’s context.

This intersects with the broader topic of securing AI agents in production, which goes deeper on the guardrails layer. But context structure is a prerequisite; you cannot bolt security on after the fact if your context assembly is already treating external content as trusted instruction.

Observability for Context

You cannot improve what you cannot measure. In my experience, most teams that have sophisticated LLM observability in place (traces, span-level token counts, latency breakdowns) still have blind spots around context quality. They can tell you how many tokens a request used; they often cannot tell you which tokens were actually relevant to the answer, or which retrieved chunks were present but unused by the model.

The emerging practice I find most valuable is context relevance scoring: after each successful response, backfilling a relevance label on the retrieved chunks that were included. This is cheap to compute with a small reranking model, and over time it gives you a signal on whether your retrieval strategy is sending useful content or filling context with noise. Teams that build this feedback loop into their LLM observability stack end up with retrieval pipelines that compound in quality over time rather than staying static.

Token budget attribution is the other gap. When a request runs over budget or hits a latency threshold, you want to know which layer of context was responsible. Was it the system prompt growing over time with accumulated feature additions? Was it a retrieved document set that is growing as the corpus expands? Was it the conversation history that compresses poorly for a certain interaction type? Instrumentation at the layer boundary, not just the total token count, gives you the data to make targeted improvements.

Practical Architecture: A Starting Point

When I set up context engineering for a new production AI system, here is the default architecture I start from and adjust based on workload.

The system prompt lives in a versioned configuration store, not hardcoded in application code. It has a defined structure: persona and constraints at the top, tool definitions in a separate section, and output format instructions at the end. No dynamic content. No timestamps. No per-user customization. Those go elsewhere. The goal is maximum stability for prefix cache efficiency.

Retrieved content goes through a retrieval pipeline that returns ranked chunks with associated relevance scores. There is a hard token budget for the retrieval layer: the pipeline fills up to that budget, highest relevance first, and stops. If the top result alone exceeds the budget (long documents), it extracts the relevant passage rather than truncating arbitrarily. This is the minimum viable RAG architecture and it avoids the most common retrieval mistakes.

Conversation history is compacted after a configurable number of turns, with the most recent turns kept verbatim and older turns converted to a structured summary. The summary includes key decisions made, key facts established, and any outstanding commitments the agent made to the user. This is enough for continuity without carrying hundreds of turns verbatim.

Tool outputs are processed immediately upon return: long outputs are summarized by a small model before being injected into the agent scratchpad, with the full output stored externally if needed for audit. The agent sees a compact representation. If a step genuinely requires the full output, the agent can request it explicitly.

The whole pipeline is instrumented at layer boundaries. Every request logs: system prompt token count, retrieved content token count and relevance distribution, history token count, scratchpad token count, and total context token count. Alerts fire when any layer trends upward over a rolling window, catching context bloat before it becomes a billing surprise.

This is not the final architecture for every use case. Agentic systems with complex tool graphs need more sophisticated scratchpad management. Reasoning models with visible chain-of-thought require special handling of the thinking token budget. For the full picture on scaling these systems, see our guide to agentic AI in production.

The Shift in Mental Model

What I find most useful about the “context engineering” framing is that it forces a shift in how teams think about LLM applications. Prompt engineering treats the model as a clever text processor that you are instructing. Context engineering treats the model as a reasoning engine operating on an information environment that you have carefully curated. The second framing is more honest about what these systems actually are, and it leads to better architectural decisions.

When a production AI system fails, the root cause is almost always one of a small set of context problems: the right information was not present, the wrong information crowded out the right information, the information was present but structured in a way that made it hard for the model to use, or external content manipulated the agent’s behavior in unintended ways. None of those are prompt problems. They are information architecture problems.

Twenty years ago I was building search indexes and arguing about term frequency weighting and document freshness signals. This feels like the same problem at a different level of abstraction. You are deciding, on behalf of a reasoning system, what information it gets to see and how that information is organized. Get that right and the model largely takes care of itself. Get it wrong and no amount of model tuning or prompt iteration will save you.

Context engineering is not glamorous work. It does not make conference talks the way new model launches do. But in my experience, it is the difference between a prototype that impresses in demos and a production system that reliably serves real users at real scale.