How to Evaluate AI Agents: Trajectories, Tool Use, and Task Success (2026 Guide)

How do you evaluate an AI agent?

Evaluating an AI agent means judging both what it produced and how it got there. Unlike a single LLM call, an agent plans, calls tools, and takes many steps, so a correct final answer can hide a broken path. Combine outcome evaluation (task success rate, final-answer correctness) with trajectory evaluation (tool selection, tool-argument accuracy, step efficiency, recovery from errors). Score deterministic things with code — tool names, required parameters, expected outputs — and reserve LLM-as-a-judge for subjective quality. Track cost, latency, and step count alongside quality, not as an afterthought. Run offline evals on a fixed dataset before shipping, then monitor online in production and guard against regressions.

Key facts

• Agent evaluation has two axes: what you measure (outcome vs. trajectory) × when (offline vs. online).
• A right answer via a wrong path is a false positive — trajectory checks (tool selection, argument accuracy, efficiency) catch what outcome-only evals miss.
• Single-run success overstates reliability. τ-bench introduced pass^k — success across k repeated trials — and consistency collapses well below the one-shot rate.
• Use code for what's checkable, a judge for what isn't. LLM-as-a-judge has documented position, verbosity, and self-preference biases (some studies report judge error rates above 50%).
• Even frontier systems struggle on hard agent benchmarks: GAIA (humans 92% vs. GPT-4-with-plugins 15%), WebArena (best GPT-4 agent 14.41% vs. humans 78.24%).
• Princeton's AI Agents That Matter shows accuracy-only evaluation is misleading — evaluate on an accuracy-vs-cost curve, and keep a regression suite built from real failures.

What makes evaluating an agent different from evaluating an LLM?

An LLM evaluation scores a single input → output. An agent evaluation must also score the process — the plan, the sequence of tool calls, the arguments passed, error recovery, and efficiency — because an agent takes many steps and a right answer can come from a wrong path.

Evaluate at three levels: end-to-end (did the task succeed?), trajectory-level (was the path sound and efficient?), and component-level (which retriever, tool, or sub-agent broke?). Outcome-only evaluation is cheap and objective but blind to how — it will wave through an agent that reached the right answer by luck through a broken path, and that false positive will break under different conditions in production.

Outcome vs. trajectory evaluation: what's the difference?

Outcome (final-answer) evaluation asks: did the agent achieve the goal, independent of path? This is functional correctness — the database ended in the right state, the correct answer was returned, the pull request passed its tests. WebArena is a pure outcome benchmark: success means the final goal was reached, regardless of how.

Trajectory (step) evaluation grades the ordered execution trace — reasoning steps, tool calls, observations, and the order of decisions. The sub-metrics engineers actually use:

You need both axes. Outcome tells you whether it worked; trajectory tells you why, and whether it'll keep working.

How do you measure task success (and reliability)?

Task success rate — the fraction of tasks completed correctly — is the headline outcome metric. But for agents it's not enough on its own, because agents are non-deterministic: a single pass rate overstates dependability.

This is why τ-bench introduced pass^k: the probability the agent succeeds on the same task across k independent trials. Reported results make the gap vivid — the best model in the original τ-bench averaged under 50% success, and pass^8 fell below ~25% in the retail domain. An agent that passes once often fails on re-runs. Always pair success rate with a reliability view across repeated runs before you trust a number.

How do you measure tool-selection and argument accuracy?

These are the trajectory checks that catch silent tool misuse, and most of them are deterministic — you don't need a judge. Verify with code whether the agent selected the right tool for the step, and whether the arguments it passed were correct and well-formed. Eval libraries ship exactly these metrics: DeepEval has an argument-correctness metric plus tool-call accuracy and F1; Ragas has a tool_call_accuracy metric.

A nuance that matters: grade tool correctness, not a rigid tool order. Demanding an exact call sequence penalizes an agent that found a smarter route. Check that the right tools were used with the right arguments; only enforce order when the order is genuinely required.

What is LLM-as-a-judge, and what are its limits?

LLM-as-a-judge uses an LLM to score traces or outputs against a rubric you define. It's become the practical default for subjective grading because human labeling can't scale to trace volume. The rule: use it only for what code can't check — don't pay a judge to verify a tool name or a required parameter a string comparison confirms for free.

Its limits are real and documented. Judges show position bias (favoring the first or second response by order), verbosity bias (preferring longer answers), and self-enhancement bias (rating their own outputs higher); some studies have found evaluator error rates above 50%. Mitigations reduce but don't eliminate the bias: randomize answer order, mask model identity, penalize length explicitly, and calibrate the judge against human labels.

When should you use human evaluation?

Human evaluation is the ground-truth anchor for subjective quality and the way you calibrate an LLM judge. It doesn't scale, so you don't run it on everything — you run it on a curated slice of representative or high-stakes cases, and you use it to seed and validate your automated metrics. Think of humans as the calibration standard for your cheaper automated evals, not the everyday measurement.

How do you measure agent efficiency (steps, cost, latency)?

Two runs can both pass task-completion while one burns far more tokens, takes far more steps, or runs far slower. Efficiency metrics — steps/turns, token cost, latency, number of tool calls, retries — belong in the same traces as quality, not in a separate report.

Princeton's AI Agents That Matter makes this the central failure of agent evaluation: a simple baseline matched a complex state-of-the-art agent's accuracy at a fraction of the cost. The fix is to evaluate on an accuracy-vs-cost Pareto curve rather than accuracy alone — an agent that's 2% more accurate for 10× the cost is usually the wrong choice.

What agent benchmarks exist, and what does each test?

Only real benchmarks below; match the benchmark to your domain, and treat public scores as directional.

To show even frontier systems struggle, the safest attributable scores from primary papers: GAIA — humans 92% vs. GPT-4-with-plugins 15%; WebArena — best GPT-4 agent 14.41% vs. humans 78.24%; BrowseComp — at release, GPT-4o-with-browsing 1.9% vs. OpenAI Deep Research 51.5% (persistent multi-step navigation mattered more than mere browser access).

A caution on SWE-bench Verified: it's now near saturation, and OpenAI published a note titled Why we no longer evaluate SWE-bench Verified, arguing it no longer distinguishes frontier coding ability. Cite its provenance (500 human-validated tasks, released Aug 2024) and check the official leaderboard for any current number rather than quoting a stale one.

What's the difference between offline and online evaluation?

Offline evaluation runs a fixed dataset of tasks with known-good outcomes, under controlled conditions, before deployment. It verifies baseline standards, powers regression testing, and gates CI. Online evaluation continuously monitors real production traffic — live quality and safety signals, latency, cost, escalations, and drift.

You need both. Offline proves readiness; online catches what your dataset didn't anticipate. Neither substitutes for the other: an agent can ace your offline set and still fail on the long tail of real inputs, and it can look fine in aggregate production metrics while regressing on a case your offline suite would have caught.

How do you build an eval harness and a regression suite?

Start by turning real tasks into a fixed offline dataset with known-good outcomes. Score the objective parts with code (tool selection, arguments, final-state checks) and the subjective parts with a calibrated judge. Run the whole set in CI on every change — a new prompt, a new model, a new tool — so regressions block the merge (tools like Braintrust are built to gate on this).

The discipline that compounds: convert every production failure into a permanent test case. A user hits a bug, you reproduce it, you add it to the suite — and it can never silently regress again. Over time your eval set becomes a precise map of your agent's real failure surface, which no public benchmark can give you.

What are the most common mistakes when evaluating AI agents?

Frequently asked questions

What's the difference between evaluating an LLM and evaluating an agent?

An LLM eval scores a single input → output. An agent eval must also score the process — the plan, the sequence of tool calls, the arguments, error recovery, and efficiency — because an agent takes many steps and a right answer can come from a wrong path.

Should I grade the agent's trajectory or just the final outcome?

Both. Use outcome checks (task success, final-answer correctness) for objective pass/fail, and trajectory checks (tool selection, argument accuracy, step efficiency) to see how it succeeded or failed. Grade tool correctness, not a rigid tool order, unless the order is required.

Is LLM-as-a-judge reliable enough for agent evaluation?

For subjective quality it's the practical default, but it has real biases (order, length, self-preference) and can be wrong often. Use deterministic code for anything checkable, reserve the judge for subjective parts, and calibrate it against human labels.

What agent benchmarks should I look at?

Match the benchmark to your domain: τ-bench/τ²-bench for tool-agent-user flows, WebArena/BrowseComp/OSWorld for web and computer-use, SWE-bench Verified for coding, GAIA for general assistants, BFCL for raw function-calling. Treat leaderboard scores as directional and build your own eval set from real tasks.

How do I catch regressions when I change a prompt or model?

Keep a fixed offline dataset of tasks with known-good outcomes and run it in CI on every change. Add every production failure to the set so it can never regress unnoticed.

What metrics matter most in production?

Task success and goal completion, tool-call accuracy, reliability across repeated runs, and the operating envelope — latency, token cost, step count, retries — plus business signals like escalation and satisfaction. Track cost and latency in the same traces as quality.

Related reading

• How to build an AI agent — the loop and tool calls you'll be evaluating.
• Multi-agent systems explained — why multi-agent traces are harder to grade.
• Production RAG interview guide — evaluation as the thing that separates a demo from a system.

Learn to evaluate, not just build

The engineers who ship reliable agents are the ones who can measure them — trajectory, cost, and regressions, not just a demo that worked once. Dexity's Ship Production Code with AI course has you build and evaluate a real agent end-to-end, so you leave with an eval harness and the judgment that every AI-engineering interview and job actually tests.

Sources: benchmark provenance and scores from primary papers — τ-bench, WebArena, GAIA, SWE-bench Verified (OpenAI), BrowseComp; the accuracy-vs-cost argument from Princeton's AI Agents That Matter; LLM-as-a-judge biases from Justice or Prejudice?; tooling and taxonomy from Anthropic, LangChain, Confident AI/DeepEval, and Ragas documentation. Attributed scores carry their paper's date; leaderboards move — verify current numbers before quoting. US-only. · Dexity.com