LLM Distillation in Production: How to Build a Task-Specific Student Model That Costs 1% of the Frontier

Somewhere around eighteen months ago, I started sitting in quarterly business reviews where the LLM API line on the cloud bill had grown to the size of an engineering headcount. The pattern was always the same: a team had proven out a workflow using a frontier model, the product had grown, and now the call volume was high enough that the economics felt broken. “Can we just use a smaller model?” was always the question. The honest answer was: maybe, but not by just swapping the model name and hoping.

Model distillation is how you get there. Not just swapping a frontier model for a cheaper generic small model and accepting the performance cliff, but actually capturing what the big model knows about your specific task and baking it into a compact model that costs a fraction to serve. This guide is the practical side of that: what distillation actually is, how the pipeline works, the frameworks you will use in 2026, and the operational decisions that determine whether you end up with something useful or a forgetting-prone wreck.

I have spent most of my twenty years working on infrastructure. But inference cost pressure has made LLM model lifecycle management an infrastructure problem, and distillation sits at the intersection of data engineering, training infrastructure, and deployment. If you own the platform, you need to understand it.

What Distillation Actually Is

The core idea is simple: you have a large, capable “teacher” model and a smaller, cheaper “student” model. The student is trained not just on ground-truth labels but on the teacher’s outputs, its probability distributions, and sometimes its intermediate reasoning. The student learns not just the right answer but how the teacher approaches the problem.

Geoffrey Hinton formalized this in 2015, but the technique has evolved significantly for the transformer era. Three distinct approaches matter in 2026:

Response distillation is the simplest. You run your production traffic (or synthetic queries) through the teacher, collect the outputs, and fine-tune the student on those input-output pairs. The student sees polished, high-quality answers and learns to reproduce them on your task distribution. This works well for structured tasks: classification, extraction, summarization, code generation within a defined scope. It is essentially supervised fine-tuning where the teacher generates the labels.

Reasoning distillation targets the chain-of-thought. Instead of just collecting final answers, you capture the teacher’s step-by-step reasoning traces and include those in the training signal. When you are working with reasoning models like o3 or Claude Extended Thinking, this is how you get the student to internalize systematic problem decomposition rather than just surface-level outputs. The open-weight reasoning model proliferation in 2025-2026 (DeepSeek-R1, Qwen3, and others) has made this approach more accessible, because teams can now legally collect reasoning traces from open-weight teachers.

Logit distillation is the most technically thorough approach. Instead of just the final output, the student is trained to match the teacher’s full probability distribution across the vocabulary at each token position. Matching the “soft targets” rather than just the hard answers contains more information: you learn which answers the teacher considered plausible, not just which one it picked. TRL’s DistillationTrainer implements this well, but it requires access to the teacher’s logits, which limits you to open-weight teachers or self-hosted models. You cannot pull logits from a closed API.

In practice, most teams start with response distillation because it requires no teacher model access beyond text outputs, and move to reasoning or logit distillation if they need to squeeze out more student performance.

The Production Pipeline

A distillation run is a data engineering problem wrapped around a training problem. I have seen both halves fail independently. Here is how the pipeline actually works:

Phase 1: Dataset Generation

Your student model’s ceiling is set by the quality and coverage of the dataset you distill from. Generate too little data and the student overfits to a narrow slice of the task. Generate from the wrong query distribution and the student performs well in testing but badly on the queries that actually matter in production.

The best source of queries is your production traffic. If you are already running the teacher in production, log every request and response. OpenAI’s stored completions feature (the basis of their distillation API) works exactly this way: you opt in, the API logs your completions, and you use that corpus to train a smaller model. Azure AI Foundry implements the same concept for their Azure OpenAI customers. The advantage is the query distribution is real, not synthetic, which means the student sees what users actually ask.

If you are building a new product or your teacher is not yet in production, you need synthetic query generation. The argilla/distilabel framework is the tool I reach for here. It is a pipeline framework built specifically for AI feedback workflows: you define a prompt format, configure a teacher model, and it generates queries, runs them through the teacher, and builds structured datasets. The framework integrates with OpenAI, Anthropic APIs, and local vLLM deployments. Critically, it is reproducible and auditable. You can run it again and compare.

For reasoning distillation, the prompts matter even more. You need prompts that elicit multi-step chains of thought from the teacher, and you need to verify the reasoning is actually sound before including it in training. Including bad reasoning traces is one of the fastest ways to produce a student that confidently follows the wrong steps to the wrong answer.

One practical rule: aim for at least 5,000-10,000 high-quality examples per task type before you expect meaningful student performance. Below that you are likely fine-tuning on noise rather than signal.

Phase 2: Training

For teams doing open-weight distillation, HuggingFace TRL is the default framework in 2026. TRL’s DistillationTrainer implements both response and logit distillation, integrates with Accelerate for multi-GPU training, and works with LoRA/QLoRA adapters so you do not need to train the full student from scratch. I pair this with the LoRA and QLoRA infrastructure patterns described in our fine-tuning guide, since the hardware setup is nearly identical.

For teams using closed teacher models (GPT-4o, Claude Sonnet), you are limited to response distillation and you will use the provider’s distillation tooling. OpenAI’s model distillation workflow captures completions from GPT-4o or o-series models and fine-tunes a smaller model (gpt-4o-mini or similar) on them. Azure AI Foundry’s distillation offering lets you use stored completions from GPT-4.5 or o1 to fine-tune smaller Azure OpenAI models. Both are pay-per-token for the training run, similar to standard fine-tuning pricing.

Arcee AI’s DistillKit is worth mentioning for teams doing this at serious scale. It uses polynomial approximation of logit distributions and bit-level packing to make offline distillation workflows practical when you have very large teachers and need to minimize storage of the intermediate activations.

For student model selection: start smaller than you think you need to. The instinct is to pick a 13B or 30B model because it feels safer, but a well-distilled 3B-7B model on a specific task often outperforms a generic 13B model. The student is specializing; the larger baseline model is not the only thing that matters. Good starting points in 2026: Qwen3-4B/8B, Llama 3.2 3B, Phi-4-mini for English-language tasks.

Phase 3: Evaluation

This is where distillation projects collapse most often. Teams train a student model, run a quick spot-check, declare victory, and deploy. Two months later, product quality has degraded and nobody knows why.

Evaluation has to cover both task performance and failure mode coverage. Task performance is the easy part: compare student and teacher outputs on a held-out evaluation set using LLM-as-judge evaluation with the teacher itself scoring. For RAGAS-style RAG tasks, use context precision and faithfulness metrics. For generation tasks, use semantic similarity and human-rated samples.

Failure mode coverage is harder. You need to specifically test the cases where you expect the student to fail: out-of-distribution queries, adversarial prompts, requests the teacher would have refused, domain boundaries. A student trained on customer support queries for a B2B SaaS product will confidently give wrong answers when a user asks about something outside that domain. You need to know where the edges are before you put the student in front of users.

I run evaluations in tiers: automated metrics on the full evaluation set, LLM-as-judge on a stratified sample, and human review on the tail cases. The human review tier is the one teams skip, and it is the one that catches the failure modes that will embarrass you.

Build regression detection into your CI pipeline from day one. Every distillation update should run the full evaluation suite, and any regression beyond a defined threshold should block promotion to production. This is the same discipline as database schema migrations: making it hard to accidentally break things.

When to Distill vs When Not To

Distillation is not always the right tool. Here is how I think about the decision tree:

Distill when the task is well-defined and repetitive, your query distribution is stable enough to generate a representative dataset, and you expect high call volume that will amortize the training cost. Customer support bots, document extraction pipelines, code review assistants with a fixed scope, classification and routing layers in AI agents. These are excellent distillation candidates.

Use quantization instead when you need to reduce inference cost and memory footprint without the complexity of a training run, and you are willing to accept a modest quality drop uniformly across all tasks. Quantization is much faster to implement. If you are serving a general-purpose assistant and need to run it on more constrained hardware, quantize first and evaluate whether the quality hit is acceptable before investing in distillation.

Use RAG instead when the task is knowledge-heavy and the knowledge changes frequently. A distilled model bakes knowledge into weights at training time. If your queries depend on information that updates daily or weekly, a retrieval-augmented pipeline with a cheaper base model will be more accurate and easier to maintain than a distilled model that goes stale.

Combine them when you need maximum inference efficiency. A common pattern: distill a task-specific student model, then apply 4-bit or 8-bit quantization to the student. You get the quality benefit of task specialization and the hardware efficiency of quantization. The distilled student is smaller to begin with, so quantization costs you less absolute quality than it would on a larger general-purpose model. This plays well with speculative decoding too, where the student becomes the draft model for a larger verifier.

Do not distill when you are still exploring task requirements. Distillation is a capital investment: it takes time, compute, and ongoing maintenance. If you are not confident in what the model needs to do, running the frontier teacher directly is the right choice. The cost of being wrong is lower. Distill once the requirements are stable and the call volume justifies it.

Infrastructure for Training and Serving

The compute picture for training a student is more tractable than it looks. A full fine-tuning run for a 3B-7B parameter student on 10,000-50,000 examples with LoRA adapters typically takes a few hours on a single A100 or H100. This is not the multi-node training job that frontier model training requires. Most teams can run distillation training jobs on serverless GPU platforms like Modal or RunPod for a cost well within a single day’s frontier API bill at meaningful inference volume.

For serving, a distilled 3B-7B model runs comfortably on a single GPU, often a lower-tier one. vLLM or SGLang with continuous batching will handle the serving. The per-token cost at this model size is orders of magnitude below frontier API pricing, which is the entire point.

Track your costs explicitly from the start. The AI FinOps practices for tracking frontier API costs apply equally to self-hosted inference: cost per thousand tokens, cost per feature, anomaly detection on usage spikes. The difference is you now own the infrastructure, so cost visibility requires active instrumentation rather than just reading a cloud bill.

One infrastructure consideration that surprises teams: managing model versions. A distilled student is a trained artifact tied to a specific version of the teacher, a specific query distribution, and specific evaluation thresholds. When you retrain, you need to version and track all three. MLflow or Weights and Biases for experiment tracking, a model registry for artifact management, and a clear policy on when retraining is triggered. “We retrain quarterly” is a policy. “We retrain when we feel like it” is how you end up running a stale student in production while the teacher has been updated with improvements the student never learned.

The Ethics Dimension

2026 brought this into sharp focus in a way that had not been visible before. Anthropic published findings documenting coordinated, large-scale campaigns where actors used Claude outputs to train competing models, extracting millions of conversations specifically to capture reasoning traces and agent-style workflows. This is sometimes called “model distillation attacks” in the security community, and it is explicitly prohibited by every major provider’s terms of service.

The distinction matters for practitioners: legitimate distillation uses your own production traffic, synthetic queries you generate, or datasets you have rights to. It does not involve systematically querying a provider’s API to build a dataset for training a competing model. Most distillation use cases teams actually encounter, building a specialized customer support bot or document pipeline, are straightforwardly legitimate. But if you are building a general-purpose assistant and planning to train it on GPT-4o’s outputs so you can avoid paying OpenAI, you are in terms-of-service violation territory regardless of whether anyone detects it.

Practically, the ethical and technically sound approaches align: task-specific distillation with your own production data produces better students than broad distillation of frontier outputs anyway. The task-specific distribution is cleaner training signal.

The Real Value Proposition

The teams I have seen do this well are the ones that treat distillation as a recurring engineering discipline rather than a one-time optimization project. They build the pipeline, establish the evaluation benchmarks, automate the training runs, and treat student model updates the same way they treat application deployments: versioned, tested, gradually rolled out.

The cost economics are compelling when the volume justifies it. A well-distilled task-specific model that handles ninety percent of your call volume while routing edge cases to the frontier teacher is a genuinely different economic profile than pure frontier API usage. The key words are “when the volume justifies it.” Distillation has a fixed cost that pays off over inference volume. Do the math honestly before you start: estimate call volume, estimate the cost delta, see how long it takes to amortize the training investment. If the answer is three years, wait until volume grows.

The teams I have seen do this poorly are the ones who distill early (before the task is stable), skip evaluation rigor (the student sounds good in demos but fails on production queries), or treat the distilled model as set-and-forget (the teacher gets better, the distribution drifts, the student stops being competitive without anyone noticing).

Distillation is not magic. It is a disciplined transfer of capability from a model you cannot afford to run at scale into a model that fits your infrastructure budget. Get the discipline right and the economics follow.

For teams just starting, the fastest path to production is the OpenAI distillation API or Azure AI Foundry’s distillation offering if you are already in those ecosystems: low operational overhead, no training infrastructure to manage, and the teacher is already the model you are paying for. Once you have demonstrated the ROI, moving to open-weight distillation with TRL gives you more control over the process and removes the dependency on the provider’s tooling.

The synthetic data generation practices I described in our earlier guide are directly applicable here, especially for cold-start situations where you do not yet have production traffic. Build the synthetic query pipeline first, use it to generate the distillation dataset, and you have something to train on even before a single user hits the system.

And when you are evaluating whether the student is good enough, treat it like any other production service: SLOs on quality metrics, regression gates in CI, gradual rollout with traffic shadowing before full promotion. The LLM observability stack you built for the teacher applies to the student too. The only difference is the cost line in your metrics dashboard, which should be going in the right direction.