AI Evals Kickoff: Stop Shipping AI on Vibes
Join Kadamb Goswami (Product Leader · Amazon) for a free live session.
Kadamb Goswami
Product Leader · Amazon
⭐ 4.8 / 5
If you can't measure AI quality, you can't ship AI products.
One of the biggest gaps in AI product development is not models or features — it is evaluation. Traditional product metrics break down in AI systems: accuracy is incomplete, user feedback is noisy, behavior is inconsistent. This leaves most teams guessing whether what they built is actually better, reliable enough to ship, or what 'good' even means. This sprint teaches you how to think about quality, reliability, and performance in AI systems from a product perspective — designing evaluation frameworks that combine quantitative metrics, human judgment, and real-world usage signals. Not about becoming an ML engineer. About becoming a product leader who can make informed decisions about AI behavior.
What You'll Learn
Data Collection for Evals
Instrument your AI system to capture the right signals. Generate synthetic data to bootstrap evals before you have real users.
Error Analysis at Speed
Apply data analysis techniques to find systematic failures in your AI product — fast — regardless of use case.
LLM-as-a-Judge
Build custom, high-quality evaluators aligned to your product goals and stakeholder trust — not generic off-the-shelf benchmarks.
Architecture-Specific Evals
Measure RAG retrieval quality, debug multi-step pipelines, and handle multi-modal settings with the right eval strategies.
Who Is This For?
This sprint is designed for:
AI PMs Shipping on Vibes
Who know their AI feature could be better but have no systematic way to measure or improve it.
Senior PMs & Leads
Who need to set quality standards for AI teams and brief engineering on what 'good' looks like.
Technical PMs & AI Leads
Who want rigorous eval frameworks they can implement themselves — not just theory.
Sprint Outline
6 weeks · 3 sessions per week
Projects You'll Ship
Leave with real work to show, not just a certificate.
Eval Instrumentation Plan
Design the observability layer for a real AI product — what to log, how to sample, and what to synthesize when you have no data.
LLM-as-a-Judge Scorecard
Build a custom evaluator for a specific AI use case, validated against human judgment and stakeholder sign-off.
Production Eval Pipeline
End-to-end: automated eval gates, experiment tracking, and safety guardrails — presented live in the final session.
Your Instructors
Kadamb Goswami
Product Leader · Amazon
⭐ 4.8 / 5
Kadamb is a Product Leader at Amazon, specializing in building and scaling AI-driven systems for high-volume, mission-critical workflows. He focuses on helping PMs evaluate AI products rigorously — turning ambiguity into clear decisions and measurable outcomes.
What Students Say
⭐⭐⭐⭐⭐
"I went from shipping AI features and hoping they worked to having a systematic way to know they work. This is the missing piece for every AI PM."
Tyler Bennett
Product Manager · Databricks
⭐⭐⭐⭐⭐
"The LLM-as-a-judge module alone was worth the entire sprint. We shipped it to production two weeks after the session."
Sara Hoffman
Senior PM · Cloudflare
⭐⭐⭐⭐⭐
"Finally a sprint that treats AI quality as a PM problem, not just an engineering problem. Priya's frameworks are immediately usable."
James Kim
AI Product Lead · Rippling
Sprint Schedule
All sessions are instructor-led and live. Recordings available within 24 hours.
SUNDAY
9:00 AM PDT
Live ClassPrimary topic deep dive with instructor. Includes lecture, case studies, and live Q&A.
WEDNESDAY
6:00 PM PDT
Coaching SessionSmall group coaching. Bring your eval questions and current blockers.
THURSDAY
6:00 PM PDT
Practice SessionHands-on practice and peer review. Build your evals with cohort support.
Frequently Asked Questions
AI Evals Kickoff: Stop Shipping AI on Vibes
with Kadamb Goswami · Product Leader, Amazon
What you'll walk away with:
🎁 Bonus for attendees:
Get "The AI PM Eval Starter Kit"
Templates for annotation, LLM-as-a-judge, and CI/CD eval gates
Claim your free seat
Skills you can deploy on Monday morning.