Skip to content

Learning: Evaluation + Quality Assurance

How to Use This Page

  • Build small eval sets first, then expand.
  • Keep scoring deterministic where possible.
  • Tie quality metrics to release decisions.

Lesson 1: Fundamentals

Learn case design, rubric scoring, and reproducible evaluation runs.

What You Learn

  • How to define clear expected outcomes per case
  • Why rubric criteria should be explicit and measurable
  • How to separate correctness, latency, and safety metrics

Walkthrough

  1. Create an initial eval set with representative cases.
  2. Define rubric dimensions and scoring rules.
  3. Run baseline evaluation and store results.

Try It Yourself

  • Add edge-case inputs and observe score shifts.
  • Tighten one rubric criterion and compare pass rates.
  • Ensure repeated runs produce stable scores.

Lesson 2: Intermediate Patterns

Learn baseline comparison and regression triage.

What You Learn

  • How to compare candidate changes to fixed baselines
  • Why delta analysis is better than raw score snapshots
  • How failure categorization speeds remediation

Walkthrough

  1. Select a baseline run as reference.
  2. Evaluate a new prompt/model/version candidate.
  3. Classify regressions by failure type.

Try It Yourself

  • Create one intentional regression and verify it is detected.
  • Measure false alarm rate for your gate thresholds.
  • Build a simple top-3 failure category report.

Lesson 3: Production Patterns

Learn release gating, adversarial evals, and drift monitoring.

What You Learn

  • How to enforce hard quality thresholds before release
  • Why adversarial subsets are required for safety-critical paths
  • How judge/model drift can invalidate historical comparisons

Walkthrough

  1. Define gate thresholds for critical metrics.
  2. Add adversarial cases to eval suite.
  3. Monitor trend lines and drift indicators over time.

Try It Yourself

  • Run gate checks on two candidate versions and choose one.
  • Add a "block release" rule for severe safety regressions.
  • Document a rollback trigger based on eval drift.

Code Examples

  • Starter script: docs/starter/stage08_eval_qa.py
  • Stage architecture notes: docs/roadmap/stage-08-eval-design-and-regression-gates/architecture.md