How to Build an AI Agent Eval Harness (2026)

How do you build an eval harness for an AI agent?

Build an eval harness as six wired-together parts. Start with a versioned dataset of 20–50 tasks (inputs plus expected outcomes) seeded from real production traffic. Write a runner that resets a clean environment, invokes the same agent you ship, and captures a full trace: every thought, tool call, argument, result, token count, and latency. Score each run two ways — deterministic assertions (tool-call correctness, argument and final-state checks, JSON-schema validation) and an LLM-as-judge with a decomposed rubric. Aggregate pass rates into a report, gate pull requests on thresholds in CI, and feed every production failure back as a permanent regression case.

Key facts

• An eval harness has six components: dataset → runner → scorers → reporting → CI gate → regression store.
• Start with 20–50 cases for one important workflow — hand-written across your top intents and seeded from real traffic — then grow from production failures.
• Use deterministic checks for anything verifiable (tool calls, arguments, final state, JSON schema) and reserve LLM-as-judge for open-ended quality, ideally triggered only when deterministic checks flag a drop.
• Invoke the production agent, not a stripped-down copy, and reset an isolated environment before every trial.
• Agents are non-deterministic — run each case N times and gate CI on a statistical lower bound, not a single pass.
• Log traces against OpenTelemetry GenAI semantic conventions (emerging/stabilizing) so they're portable across Langfuse, Phoenix, and other backends.

What is an AI agent eval harness?

An eval harness is the test rig for an agent — the standing infrastructure that runs your agent against a fixed set of tasks, scores both the outcome and the path it took, and tells you whether a change made things better or worse. It's what turns "it worked when I tried it" into a repeatable measurement you can gate releases on.

The distinction from a one-off eval: a harness is wired into your workflow. It loads version-controlled cases, resets a clean environment per trial, invokes the same agent you ship (not a stripped-down copy), captures the full trace, grades it with deterministic and model-based scorers, compares against an approved baseline, and blocks releases when critical thresholds are missed. Build it once and every future change is measured automatically.

What are the components of an eval harness?

Six parts, wired in a pipeline:

How do you build an eval dataset for an agent?

Seed from reality, not imagination. Take your last ~200 real interactions, read them, and pull the ~20 that made you uncomfortable — the messy, edge, and failure cases. Cover happy paths, messy inputs, safety boundaries, tool failures, and missing-data cases. Each record stores the input, the context the system sees, and enough to judge the output: expected tool calls, expected final state, or a reference answer.

Then version it — so a score drop means the model regressed, not that your test set moved — and grow it from production. Route failed traces (low eval score or negative user feedback) into an annotation queue, label them with two-reviewer agreement, and promote them into the dataset. On an active product expect to add roughly 30–80 cases a week. The best datasets mix curated edge cases, real production failures, and a few targeted synthetic scenarios for known gaps.

How many eval cases should you start with?

Fewer than you think. 20–50 cases for a single important workflow is the recommended starting band — hand-written across your top user intents, with clear labels and a short labeling guideline so scoring is consistent. Resist the urge to build a thousand cases up front; a small, high-signal set you actually run on every change beats a large one you run once. Let the dataset grow from real production failures rather than speculation.

How do you write deterministic scorers and assertions?

Deterministic scorers are fast, cheap, and exact — use them for anything with a checkable right answer:

• Tool-call correctness — did the agent call the right tool? (Ragas ships a tool-call-accuracy metric; DeepEval scores tool selection at the span level.)
• Argument checks — assert the tool was called with the expected arguments.
• Final-state checks — assert the resulting world or database state matches expected. For action agents this is the strongest signal.
• JSON-schema validation — validate structured output against a schema.
• Exact/substring/similarity — for reference strings.

The code-level pattern is plain Python assert functions, one per dimension, run inside pytest — parametrize one test per case, and run with parallel workers for speed. Everything a string comparison or schema check can confirm should be scored this way, not by a model.

When should you use LLM-as-judge instead of deterministic checks?

Reach for an LLM-judge only where there's no single correct string — open-ended quality like coherence, helpfulness, faithfulness, and overall trajectory quality. It's the practical default for trajectory scoring because human labeling can't scale to agent-trace volume. But it's slower, costs money, and carries bias, so a good pattern is to trigger the judge only when deterministic checks show a quality drop, keeping the expensive scorer off the happy path.

How do you design an LLM-judge rubric and reduce bias?

Decompose the rubric into independent dimensions — accuracy, coherence, safety, conciseness — scored separately and then aggregated. This stops one strong trait from masking a failure and makes debugging easier. Use simple, explicit language, separate correctness from style, and explicitly penalize known failure modes.

Then mitigate the documented biases:

• Position bias — score both orders (A,B) and (B,A) and accept only a stable preference.
• Verbosity bias — add conciseness as a rubric dimension or use length-controlled scoring.
• Self-preference bias — use a cross-family judge model; don't let a model grade its own family.

Pin the judge model ID, version the rubric, hash the prompt template, and calibrate against ~100–300 human labels per rubric, re-checking monthly or whenever the rubric changes. Carefully designed judges report over 80% agreement with human preferences — treat that as a conditional result, not a guarantee.

How do you capture agent traces?

Log everything the agent did, not just what it returned: every reasoning step, each tool call with arguments and result, handoffs, the final response, state changes, safety events, token usage, latency, and cost. That full trace is what lets you score the path, not only the outcome.

For structure, lean on the OpenTelemetry GenAI semantic conventions (an emerging, still-stabilizing standard): they add span types distributed tracing lacked — a model-call span, an agent-invocation span (invoke_agent), and a tool-call span (execute_tool) — each first-class rather than buried. Message content is captured through a separate opt-in mechanism, so PII redaction stays your decision. The payoff is portability: instrument once against gen_ai.* and the same spans flow to Langfuse, Arize Phoenix, or a self-hosted store with no re-instrumentation.

Which eval framework should you use?

Match the tool to how you want to build:

For a code-first, in-repo harness, start with DeepEval (pytest-native) and add Phoenix or Langfuse for self-hosted OTel tracing and dataset versioning; choose LangSmith or Braintrust if you want a hosted eval-driven-dev platform.

How do you run agent evals in CI?

Wire the suite into CI as a required check: on every pull request, install the package and run pytest (deepeval test run or equivalent) against your dataset. Gate on thresholds — fail the build when the pass rate drops below your bar or any critical case fails — and compare the candidate against an approved baseline, blocking releases on regressions. Keep the fast deterministic checks on every run and the expensive LLM-judge calls scoped to where they're needed, so the gate stays quick enough that engineers don't route around it.

How do you gate pull requests and track regressions?

Because agents are non-deterministic, a single pass or fail is noisy. Run each case N times and gate on a statistical lower bound — for example, require the Wilson lower confidence bound of the pass rate to clear your threshold, rather than trusting one run. That separates a real behavior shift from normal variance.

The regression loop is what makes the harness compound: route every production failure (low score or negative feedback) into the annotation queue, label it, and promote it into the versioned dataset. Each incident becomes a permanent test that can never silently regress again. (The pattern is now mainstream — cloud providers have shipped agent regression testing directly into CI pipelines.)

How do you evaluate an agent in production (online eval)?

Offline evals prove readiness; online evals catch what your dataset didn't anticipate. In production there's no reference answer, so you sample live traffic — roughly 10–15% of sessions to control LLM-judge cost — and score it reference-lessly, tracking the distribution over time rather than any single run.

Think in three pillars: tracing (capture what happened), scheduled evals (score the sampled traffic), and guardrails (block harmful output in real time, inline as generation happens). Trigger the expensive model-based checks only when cheaper deterministic signals show a drop, and feed anything that fails back into the offline dataset.

Frequently asked questions

How many eval cases do I need to start?

20–50 for a single important workflow — hand-written across your top intents and seeded from real traffic, then grown from production failures (~30–80 a week on an active product).

Deterministic checks or LLM-as-judge?

Both. Use deterministic asserts for anything verifiable (tool calls, arguments, final state, JSON schema); reserve the LLM-judge for open-ended quality and trajectory scoring, ideally triggered only when deterministic checks flag a drop.

How do I stop my LLM judge from being biased?

Decompose the rubric into independent dimensions, swap-augment for position bias, add conciseness to fight verbosity bias, use a cross-family judge model, and calibrate against ~100–300 human labels per rubric.

How do I handle non-deterministic agents in CI?

Run each case multiple times and gate on a statistical lower bound (e.g., a Wilson lower-bound pass rate) against an approved baseline, not a single run.

What should I log in a trace?

Every thought, tool call with arguments and result, handoffs, the final response, state changes, safety events, tokens, latency, and cost — ideally against OpenTelemetry GenAI semantic conventions so traces are portable across backends.

Which tool should I pick?

For an in-repo, code-first harness start with DeepEval (pytest-native, open-source); add Phoenix or Langfuse for self-hosted OTel tracing and dataset versioning; choose LangSmith or Braintrust for a hosted eval-driven-dev platform.

Related reading

• How to evaluate AI agents — the concepts (outcome vs trajectory, benchmarks) this harness operationalizes.
• How to build an AI agent — the agent whose traces you'll be scoring.
• Multi-agent systems explained — why multi-agent traces are harder to capture and grade.

Build the harness, not just the agent

The engineers who ship reliable agents own the test rig, not just the demo. Dexity's Ship Production Code with AI course has you build and evaluate a real agent end to end — dataset, scorers, CI gate, and all — so you leave with the eval discipline every AI-engineering team now hires for.

Sources: harness architecture and dataset/scorer guidance synthesized from production eval-harness writeups and golden-dataset guides (Towards Data Science, Langfuse, DEV, qaskills, 2026); LLM-as-judge rubric and bias practices from openlayer, autorubric, and 2026 judge-bias research; trace structure from the OpenTelemetry GenAI semantic conventions (emerging); tooling capabilities verified against DeepEval, Ragas, Langfuse, Arize Phoenix, and OpenAI Evals docs. Numeric guidance (20–50 seed cases, 10–15% sampling, 100–300 labels) is recommended practice, not law; verify tool specifics before relying on them. US-only. · Dexity.com