AI Evals in Production: The Error-Analysis-First Playbook (2026)
Updated August 17, 2026·8 min read
TL;DR
Evals — not model choice, not prompt cleverness — decide whether AI features work in production. In Dexity's analysis of live job descriptions, evals now appear in 56% of AI-engineer and 32% of product-manager postings, up from near-zero two years ago. Here's the error-analysis-first method teams use to ship AI they can measure instead of hope for.
Product managers evaluate AI and LLM features by starting with error analysis, not tooling: read a sample of real production traces, note what went wrong in plain language, group the failures into recurring categories, and count them to get your actual failure distribution. In Dexity's analysis of live job descriptions, evals now appear in 56% of AI-engineer and 32% of product-manager postings — up from near-zero in 2024. Only then do you build measurement — LLM-as-a-judge scorers, assertion checks, and monitoring — around the failures that matter. Evals, not model choice or prompt cleverness, are the difference between an AI feature that works in production and a demo that falls over. As models commoditize, the durable advantage is knowing whether your system actually works — and why.
Key facts
- In Dexity's analysis of live job descriptions, evals now appear in 56% of AI-engineer postings.
- In Dexity's analysis of live job descriptions, evals now appear in 32% of product-manager postings.
- Evals went from a research topic to a hiring requirement in under two years, up from near-zero in 2024, according to Dexity's job-description analysis.
- There are three canonical types of AI eval — human evals, code/assertion-based evals, and LLM-as-a-judge — and mature systems combine all three rather than picking one.
- In the error-analysis-first method Dexity teaches, reviewing just a few dozen production traces is often enough to surface a system's recurring failure patterns.
Why did AI evals become a hiring requirement?
Two years ago "evals" was a research word. Today it's on the job description.
The reason is simple: AI systems are probabilistic. You can't ship them the way you ship deterministic code — you have to measure whether they produce correct, useful output, on your data, for your users.
Employers are now writing this into the job itself:
"Help customers develop evaluation frameworks to measure Claude's performance for their specific use cases." — Anthropic, Applied AI Architect job description (2026)
How do you run error analysis on your AI outputs?
The most common mistake is reaching for an eval tool first. The teams that ship reliable AI invert that — they start by looking at their own data:
- Read real traces. After any significant change, manually review a sample of your system's outputs (a few dozen is enough to see patterns).
- Note what's wrong, in open-ended language — don't force categories yet.
- Categorize the failures into recurring buckets (wrong retrieval, hallucinated field, tone, format, broken tool call).
- Count them. Now you know your actual failure distribution, not your imagined one.
Only then do you build measurement — because now you know what to measure. This error-analysis-first approach is the method practitioners like Hamel Husain and Shreya Shankar have made the standard for applied LLM evals (Hamel Husain, "Your AI Product Needs Evals"), and it's the same discipline Aman Khan lays out in Lenny's Newsletter's guide to evals for PMs.
What kinds of AI evals are there?
Once error analysis tells you what to measure, you choose how to measure it. There are three canonical approaches, and mature systems use all three in combination rather than picking one:
| Eval type | How it works | Best used when |
|---|---|---|
| Human evals | A person reviews outputs against a rubric and labels each one | Early on; for subjective quality (tone, helpfulness, safety); and to produce the ground-truth labels the other methods are validated against |
| Code / assertion-based evals | Deterministic checks assert on structure or content — valid JSON, required fields present, exact match, regex, no forbidden strings | The correct answer is objectively checkable: output format, schema, tool-call arguments, presence of a required key |
| LLM-as-a-judge | A model scores each output against a rubric, calibrated against your human labels | Subjective criteria that need to run at scale, after you have human labels to validate the judge against |
The practical ordering follows from that: humans first (to see failures and create labels), assertion-based checks wherever correctness is deterministic (they're cheap, fast, and never drift), and LLM-as-a-judge to scale the subjective judgments once a human-labeled set exists to keep the judge honest.
Dexity Intel · free newsletter
Liking this? Get the next one in your inbox.
JD-backed career reads, AI market signals, and field-tested tool guides — a few times a month. No fluff, no spam.
How do you scale from manual review to automated evals?
Once error analysis reveals the failure modes, you scale from manual review to automated measurement:
- LLM-as-a-judge — encode each recurring failure as a judge prompt, validated against your human labels, so every new output gets scored.
- Synthetic data — manufacture edge cases you can't yet collect in production.
- Production monitoring — keep sampling live traces; the failure distribution drifts as usage grows.
- A data flywheel — each round of analysis → judges → fixes → new traces compounds into a system that improves over time.
Open frameworks from the major AI labs make the plumbing straightforward (OpenAI Evals, Anthropic evaluation docs) — but the frameworks are the easy part. The judgment about what to measure comes from error analysis.
How do you design a good eval?
How do you write an eval rubric?
A rubric turns a vague "is this good?" into specific, independently checkable dimensions. Break "good" apart into the qualities that actually matter for your feature — for example correctness, retrieval relevance, format, tone, and safety — and score each one separately instead of collapsing everything into a single blurry number. For each dimension, write down what a pass and a fail look like, anchored to concrete examples pulled straight from your error analysis. Prefer a binary pass/fail or a short ordinal scale per dimension, and tie every criterion back to a real trace so two reviewers reading the same output land on the same label.
How big should your eval dataset be?
There's no universal number, and bigger isn't automatically better. Start small: as the error-analysis step above notes, a few dozen traces is often enough to surface your recurring failure patterns. Size the set for coverage of the failure modes you found — a dataset that contains real examples of each category beats a larger random sample that misses the cases you care about. Then grow it over time as production surfaces new failure modes, so the set keeps reflecting how the system actually behaves.
What are evals not?
- Not public benchmarks. An MMLU score doesn't tell you whether your RAG bot answers your customers correctly. Evals are application-specific.
- Not vibes. "It looks good in the demo" is not an eval — the point is a repeatable, counted failure distribution.
- Not an infra project first. Tooling comes after error analysis; the reverse is the most common failure.
- Not one-and-done. The failure distribution drifts; evals are a continuous loop, not a launch gate.
Why is the window to learn evals closing?
Evals went from research topic to hiring requirement in under two years — 56% of AI-engineer and 32% of PM JDs now call for them. The builders who learn error-analysis-first evals now are scarce; the ones who wait will be competing against teams whose products measurably improve every week.
Related reading
- AI Engineer Career Path in 2026 — evals appear in 56% of these roles.
- What Does a Product Manager Career Look Like in 2026? — 32% of PM JDs demand evals.
- How to Become an AI Product Manager in 2026.
Build a real eval system for your AI product
Reading about evals isn't the same as running them on your own product. Dexity's AI Evals for PMs course walks you through the full loop — data collection, error analysis, architecture-specific eval strategies, and regression detection — so you ship AI you can actually stand behind.
Frequently asked questions
What is an AI eval?
An application-specific test of whether your AI system produces correct, useful output — built from real failure analysis of your own traces, not public benchmarks.
How do you start doing evals?
Start with error analysis, not tooling: review a sample of real outputs, note and categorize failures, count them, then encode the recurring failures as LLM-as-a-judge prompts validated against your labels.
What are the main types of AI evals?
Three canonical approaches: human evals (a person labels outputs against a rubric), code/assertion-based evals (deterministic checks for format, schema, or exact content), and LLM-as-a-judge (a model scores outputs against a rubric, calibrated on human labels). Mature systems combine all three.
Are evals a PM skill or an engineering skill?
Both — evals appear in 56% of AI-engineer and 34% of product-manager job descriptions in our analysis.
Source: Dexity analysis of live AI-engineer and product-manager job descriptions across company career boards, US-inclusive, July 2026 (keyword-coded from full JD text; shares directional). Technical references: OpenAI Evals · Anthropic evaluation docs · Expert sources: Hamel Husain — Your AI Product Needs Evals · Aman Khan / Lenny's Newsletter — a PM's guide to evals · JD datasets & methodology · Dexity.com
Go from reading to doing · Dexity Course
AI Evals for PMs
One of the biggest gaps in AI product development is not models or features — it is evaluation. Traditional product metrics break down in AI systems: accuracy is incomplete, user feedback is noisy, behavior is inconsistent.
