Two years ago I was called into a war room at eleven in the evening because a financial services client’s AI assistant had started giving customers incorrect account balance information. The system had been running fine for months. Nothing was deployed. No one changed a prompt. The underlying model provider had done a quiet model update, and every faithfulness score we had been manually spot-checking looked fine in our weekly review spreadsheet. The problem was that we had no automated evaluation whatsoever. We were flying blind.
That incident is what made me take LLM evaluation seriously as infrastructure, not as an afterthought. In the twenty years I have been building production systems, the lesson keeps repeating itself: the things you do not measure will eventually bite you. With traditional software, you measure behavior with unit tests and integration tests. With LLMs, the output space is too large for deterministic tests. You need a different framework entirely, and most teams I talk to still do not have one.
This guide covers the tools and patterns that have actually worked in production: RAGAS for evaluating RAG pipelines, DeepEval for CI-integrated testing, Braintrust for production monitoring, and the LLM-as-judge pattern that ties it all together.
Why Traditional Testing Fails for LLMs
The core problem is non-determinism. If I ask an LLM “what is the capital of France?” it will always say Paris, so I could write a deterministic test for that. But for anything complex, the same input produces different outputs across invocations, model versions, and temperature settings. You cannot write assertEqual(response, expected) when the expected output is a paragraph of text that could be expressed a hundred ways.
The second problem is that LLMs can fail in ways that are correct but wrong. Your RAG system retrieves perfect context, generates fluent grammatical prose, and then quietly hallucinates a number or a date. The output passes a semantic similarity check against your expected answer but is factually wrong. Detecting this requires understanding the relationship between the retrieved context and the generated output, not just comparing strings.
The third problem is silent model drift. Your model provider updates the underlying weights, or you switch from one model to another for cost reasons (this happens constantly in teams doing AI FinOps work). The new model might perform better on average but regress on your specific use cases. Without systematic evaluation, you will not find out until a user complains.
The Two-Layer Eval Architecture
The mature pattern I have settled on is a two-layer architecture. The first layer runs during development and CI: automated evals that gate every prompt change, model swap, or retrieval configuration update. The second layer runs in production: sampling real traffic, evaluating it asynchronously, and alerting when quality metrics degrade below thresholds.
These two layers serve different purposes. Development evals are about catching regressions before they ship. Production monitoring is about catching regressions that only manifest on real-world inputs you did not anticipate. You need both. A test suite that passes in CI but misses the weird edge cases your actual users generate is only half the system.
This architecture connects directly to your observability layer. If you have already built out LLM observability with tracing, your production eval layer can hook into the same trace pipeline, sampling spans and evaluating them without adding latency to the hot path. The eval happens asynchronously after the response has been returned to the user.
RAGAS: The RAG Evaluation Toolkit
If you are building RAG applications, and at this point most teams with internal AI tools are, RAGAS should be your first stop. It is purpose-built for measuring the two components of a RAG system: retrieval quality and generation quality. Getting this split right matters because these components fail independently and require different fixes.
The four core RAGAS metrics are: context precision, context recall, faithfulness, and answer relevancy.
Context precision measures whether the retrieved chunks are actually relevant to the question. High context precision means your retrieval is pulling signal, not noise. If this metric is low, your chunking strategy or vector similarity threshold needs tuning. You can dig into the chunking and retrieval patterns in more detail in the RAG architecture guide.
Context recall measures whether all the information needed to answer the question exists in the retrieved context. Low recall means your retrieval is missing relevant chunks. This often points to a chunking problem: relevant information got split across boundaries, or your embedding model does not represent a certain class of queries well.
Faithfulness is the metric I care about most in regulated industries. It measures whether every claim in the generated response is grounded in the retrieved context. A faithfulness score of 1.0 means everything the model said can be traced back to the retrieved documents. A score of 0.6 means 40% of the response is potentially hallucinated.
Answer relevancy measures whether the response actually answers the question asked. This catches a specific failure mode I see in production: the model retrieves good context and generates faithful text, but the text is an oblique non-answer to the actual question.
RAGAS calculates these metrics by using an LLM as an evaluator internally. It does not require ground-truth answer labels to calculate faithfulness or relevancy, which is a major practical advantage since building labeled datasets is expensive. You do need ground truth for context recall, but you can get useful signal on faithfulness and answer relevancy from unlabeled production data.
The operational pattern I recommend is to run RAGAS on 100-200 test cases in CI, covering your known failure modes and representative queries. Keep these test cases in version control alongside your prompts. When a RAGAS score drops on a PR, that PR does not merge.
DeepEval: CI Integration and Agent Testing
RAGAS is specialized for RAG. For broader coverage, including chat, summarization, classification, and agentic workflows, I use DeepEval. It provides over 50 built-in metrics and integrates cleanly into pytest, which means it drops into any existing CI pipeline without ceremony.
The metric I use most from DeepEval is G-Eval. G-Eval is a framework for defining custom evaluation criteria in natural language and having an LLM score the output against those criteria on a 0-to-1 scale. This is the LLM-as-judge pattern formalized into a reusable structure.
For example, if I am evaluating a customer support bot, I might define a G-Eval metric like: “The response is professionally worded, acknowledges the customer’s concern, and provides a specific actionable next step. Score: 1 if all three are present, 0.5 if two are present, 0 if fewer than two.” That criterion is too complex for a regex but straightforward for a capable LLM evaluator.
DeepEval also ships purpose-built hallucination detection that works without a RAG context. It feeds the input, the retrieved context (if any), and the generated output to an evaluator model and asks it to identify any claims in the output that are not supported by either the input or the context. This is more expensive than RAGAS faithfulness since it requires more evaluator calls, but it catches a wider class of hallucinations.
For agentic AI systems where the model takes multi-step actions, DeepEval has task completion metrics that assess whether the agent accomplished the stated goal given a sequence of tool calls and intermediate outputs. This is genuinely hard to evaluate and the coverage is still imperfect, but having any metric here is better than none.
Braintrust: Production Monitoring and the Full Eval Lifecycle
DeepEval is excellent for development-time evaluation, but its open-source version is designed around local execution. When you need production monitoring with shared dashboards, regression tracking across model versions, and automated alerts, you need something like Braintrust.
Braintrust positions itself as the full eval lifecycle platform: you define your eval suite once, it runs in CI, and the same suite samples production traffic continuously. You get a unified view of how your metrics trend over time, version-to-version comparisons when you roll out a new model, and the ability to drill into individual failing examples to understand what went wrong.
The feature I find most valuable in production is the experiment comparison view. When I swap from one model version to another or tune a prompt, I run an experiment in Braintrust against my golden dataset. Braintrust shows me a side-by-side diff of every metric, flagging regressions and improvements. This turns a subjective “does the new model feel better?” conversation into a data-driven decision.
Arize Phoenix is the other platform I see used heavily in this space, particularly in teams that are already invested in OpenTelemetry for their broader observability stack. Phoenix integrates with your OpenTelemetry pipeline and evaluates LLM spans as they flow through. If you have already built an OTel pipeline, extending it to cover LLM quality evaluation is a lower-friction path than adopting a new platform.
The LLM-as-Judge Pattern
Both RAGAS and DeepEval rely on a foundational pattern worth understanding: using a capable LLM to evaluate the output of another LLM. This is the LLM-as-judge pattern, and it is the only practical way to evaluate complex, open-ended outputs at scale.
The practical setup is to designate a judge model, typically one step more capable and more expensive than your production model. If you are serving responses with Claude Haiku or GPT-4o Mini, your judge might be Claude Sonnet or GPT-4o. The judge receives the original input, any relevant context, the generated output, and a rubric expressed in natural language. It returns a score and a brief explanation.
The criticisms of LLM-as-judge are real and worth knowing. Judge models have biases: they prefer longer responses, they favor responses that match their own style, and they can be swayed by formatting. Mitigation strategies include swapping judge model and evaluatee model to check for self-preference bias, using multiple independent judge calls and averaging, and calibrating your judge against a small set of human-labeled examples.
Despite these limitations, LLM-as-judge achieves 80-90% agreement with human evaluators on well-defined rubrics, which is sufficient for regression detection. The goal is not perfect evaluation; it is catching when quality degrades meaningfully. A metric dropping from 0.85 to 0.65 is a signal worth investigating regardless of whether the exact scores are ground truth.
One practical consideration: judge model calls have costs. If you are evaluating a high-volume production system, sampling matters. I typically evaluate 1-5% of production traffic, focusing the budget on low-confidence responses where the production model expressed uncertainty, and on responses in categories historically prone to failure.
Building Your Evaluation Dataset
The weakest point in most teams’ eval pipelines is not the evaluation framework; it is the dataset. Evaluating against ten examples tells you almost nothing. Evaluating against a thousand well-curated examples tells you a great deal.
I build eval datasets from three sources. First, seed examples: hand-crafted inputs covering your core use cases, known edge cases, and the failure modes you discovered during development. Second, adversarial examples: inputs designed to probe specific failure modes like prompt injection, off-topic requests, or questions at the boundaries of your system’s knowledge. Third, production sampling: real inputs that came through your system, selected to cover the diversity of what users actually ask. The securing AI agents guide covers prompt injection testing patterns that belong in your adversarial set.
The dataset should live in version control. When you discover a new failure mode in production, you add a regression example to the dataset before you fix the bug. This is the AI equivalent of test-driven development.
For RAG systems specifically, I maintain a separate retrieval evaluation dataset that tests the retrieval component in isolation from the generation component. This lets me distinguish between “the retrieval was bad” and “the generation was bad” without running the full pipeline.
Cost Management for Production Evals
Running evaluation at scale is not free. Every judge model call costs tokens. For a system handling 100,000 requests per day at a 2% sampling rate, you are running 2,000 eval calls per day. If each eval involves three judge model calls of ~2,000 tokens each at $15 per million tokens, that is about $0.18 per day, or under $70 per year. Cheap.
The cost concern is actually at the CI layer, not production. If you have 500 test cases and each requires 3-5 judge calls, a full eval run costs around 3,000 judge calls. At high-volume usage with multiple PRs per day, this can add up. The mitigation is tiered CI evals: run a fast subset of 50-100 critical test cases on every commit, and run the full suite only before merges to main.
The other cost lever is judge model selection. The LLM prompt caching guide applies directly here: if your rubric is the same across all eval calls, you can cache the rubric prefix and pay significantly less per eval call. For RAGAS and DeepEval, where the system prompt is fixed per metric type, prompt caching gives you 60-80% savings on judge model costs.
Integrating Evals into Your Deployment Pipeline
The operational goal is to make eval results block deployments when quality regresses, the same way failing tests block deployments. In practice, this means defining threshold values for each metric and treating a drop below threshold as a build failure.
The threshold values are context-dependent. For a financial compliance system, I set faithfulness at 0.95 and treat anything below that as a deployment blocker. For a creative writing assistant, faithfulness is not meaningful and I focus on user satisfaction proxies like response length, relevancy, and diversity. Know what matters for your use case before you set thresholds.
One pattern that works well in practice is relative thresholds rather than absolute ones. Instead of “faithfulness must be above 0.90,” you define “faithfulness must not drop more than 5% relative to the baseline.” This handles natural variation in eval results and focuses attention on actual regressions rather than noise. Your CI/CD pipeline needs to store the baseline metrics alongside your artifacts for this comparison to work.
The War Story I Should Have Started With
Back to the financial services incident. After that night, we rebuilt the evaluation layer from scratch. We defined a set of 200 golden test cases covering every account-related query type, calibrated faithfulness thresholds against known-good responses, and wired RAGAS into the CI pipeline. We also added a production sampling job that evaluated 3% of live traffic every hour and pushed metrics to Grafana.
Three weeks later, the model provider did another silent update. This time, we caught it. Faithfulness on account balance queries dropped from 0.94 to 0.81 overnight. The on-call alert fired before any user noticed. We rolled back to the previous model version while we investigated, and the investigation found that the updated model was more likely to interpolate from training data rather than strictly following retrieved context. We adjusted our system prompt to reinforce grounding and re-evaluated. The fix took four hours instead of an all-night war room session.
That is what systematic LLM evaluation gets you. Not certainty, but early warning. The same kind of early warning that SLOs and error budgets give you for service reliability, applied to AI quality.
Choosing Your Stack
For most teams starting from zero, I recommend this path. Start with RAGAS if you are building RAG systems or DeepEval for broader use cases. Get 100 test cases into version control and running in CI before you invest in anything else. That baseline alone will catch the majority of the regressions that matter.
Once you have CI evals working, add production sampling. If you have an OpenTelemetry pipeline, Arize Phoenix is the lowest friction addition. If you do not, Braintrust or Langfuse provide good end-to-end solutions that handle both tracing and evaluation.
Do not let the perfect be the enemy of the good here. A team running 50 test cases with a single faithfulness metric in CI is in a dramatically better position than a team doing nothing while they wait to implement the theoretically perfect eval framework. The eval dataset and the threshold values will improve over time. The important thing is to start.
The same instinct that makes you add monitoring to a new service, write tests for a new function, or define SLOs for a new API endpoint should make you add evaluation to a new LLM feature. These systems fail in ways you do not expect, often quietly, and often slowly. Without measurement, you will not know until your users tell you. By then, the damage is done.
Get Cloud Architecture Insights
Practical deep dives on infrastructure, security, and scaling. No spam, no fluff.
By subscribing, you agree to receive emails. Unsubscribe anytime.