Stage 1: Python + Async Foundations
Theory
Stage Outcome
Build reliable async Python services that run concurrent API work with bounded latency and graceful failure handling.
Competency Snapshot
| Skill |
Target |
Evidence |
asyncio concurrency |
Intermediate |
bounded semaphore execution |
| Timeout and retry design |
Intermediate |
transient failures recover safely |
| FastAPI async behavior |
Intermediate |
sync vs async benchmark results |
Deep-Dive Index
| Topic |
Why it matters |
Link |
| Async concurrency control |
Prevents stalled or overloaded pipelines |
→ Deep Dive |
Hands-On Exercises
| Exercise |
Input |
Expected output |
| Concurrency tuning |
docs/starter/stage01_async_foundations.py |
success/fail counters and elapsed time |
| Reliability pass |
retry plus timeout changes |
fewer hard failures |
| Endpoint benchmark |
sync vs async endpoint load test |
measurable throughput difference |
Starter Code
- Starter script:
docs/starter/stage01_async_foundations.py
- Optional input seed:
data/roadmap/stage-01/urls.txt
Project
| Scope |
Deliverables |
Stretch |
| Concurrent API aggregator |
merged response plus per-source status |
adaptive concurrency control |
Exit Criteria
- [ ] Concurrent requests run with a clear cap.
- [ ] Partial failures do not crash the API response.
- [ ] Run logs include latency and failure counts.
Stage 2: LLM Fundamentals for Agents
Theory
Stage Outcome
Control prompt quality, token budgets, and model routing decisions to balance accuracy, latency, and cost.
Competency Snapshot
| Skill |
Target |
Evidence |
| Token and cost estimation |
Intermediate |
deterministic cost reports |
| Prompt structure |
Intermediate |
stable outputs under repeated runs |
| Model routing |
Intermediate |
tier selection by task complexity |
Deep-Dive Index
| Topic |
Why it matters |
Link |
| Prompt design and context control |
Avoids quality drift and runaway cost |
→ Deep Dive |
Hands-On Exercises
| Exercise |
Input |
Expected output |
| Cost estimator run |
docs/starter/stage02_llm_fundamentals.py |
three complexity tiers printed |
| Routing policy edits |
route-map tuning |
changed cost and tier lines |
| Needle-in-noise test |
data/roadmap/stage-02/needle_noise_cases.jsonl |
recall score by case |
Starter Code
- Starter script:
docs/starter/stage02_llm_fundamentals.py
- Optional dataset seed:
data/roadmap/stage-02/needle_noise_cases.jsonl
Project
| Scope |
Deliverables |
Stretch |
| Cost and latency comparison tool |
CSV plus model-tier summary |
route optimization by quality score |
Exit Criteria
- [ ] Prompt workload is scored consistently.
- [ ] Routing is deterministic for known task classes.
- [ ] Cost/latency tradeoffs are documented.
Theory
Stage Outcome
Build schema-first tool workflows that validate calls, recover from malformed inputs, and return machine-friendly outputs.
Competency Snapshot
| Skill |
Target |
Evidence |
| Tool schema design |
Intermediate |
strict input contracts |
| Validation and repair |
Intermediate |
malformed calls handled safely |
| Structured responses |
Intermediate |
consistent output payloads |
Deep-Dive Index
| Topic |
Why it matters |
Link |
| Structured tool calling |
Improves reliability and debuggability |
→ Deep Dive |
Hands-On Exercises
| Exercise |
Input |
Expected output |
| Registry run |
docs/starter/stage03_tool_calling.py |
valid tool outputs |
| Malformed-call repair |
data/roadmap/stage-03/broken_tool_calls.json |
bounded recovery attempts |
| Output normalization |
structured response formatting |
consistent final schema |
Starter Code
- Starter script:
docs/starter/stage03_tool_calling.py
- Optional dataset seed:
data/roadmap/stage-03/broken_tool_calls.json
Project
| Scope |
Deliverables |
Stretch |
| Utility tool agent |
validated tools plus error policy |
confidence-based fallback strategy |
Exit Criteria
- [ ] All tool calls are validated before execution.
- [ ] Invalid payloads are handled without crashes.
- [ ] Final outputs are structured and reusable.
Stage 4: Memory + State Management
Theory
Stage Outcome
Design short-term and persisted memory patterns that survive restarts and preserve relevant context quality.
Competency Snapshot
| Skill |
Target |
Evidence |
| Session persistence |
Intermediate |
restart-safe state recovery |
| Summary compression |
Intermediate |
concise context retention |
| Retrieval relevance |
Intermediate |
focused context recall |
Deep-Dive Index
| Topic |
Why it matters |
Link |
| Memory and checkpointing |
Enables reliable long-running workflows |
→ Deep Dive |
Hands-On Exercises
| Exercise |
Input |
Expected output |
| Persisted run loop |
docs/starter/stage04_memory_state.py |
incrementing saved message count |
| Summary tuning |
summary logic edits |
better signal-to-noise summary |
| Notes retrieval |
data/roadmap/stage-04/notes_corpus/ |
relevant note snippets |
Starter Code
- Starter script:
docs/starter/stage04_memory_state.py
- Optional dataset seed:
data/roadmap/stage-04/notes_corpus/
Project
| Scope |
Deliverables |
Stretch |
| Local memory assistant |
persisted state plus retrieval layer |
checkpoint versioning strategy |
Exit Criteria
- [ ] State survives process restart.
- [ ] Summaries remain concise and useful.
- [ ] Retrieval adds relevant context, not noise.
Stage 5: Single Agent Workflows
Theory
Stage Outcome
Implement bounded planner-executor loops that collect evidence before final answers and fail safely when stalled.
Competency Snapshot
| Skill |
Target |
Evidence |
| Planner loop control |
Intermediate |
bounded step execution |
| Evidence sufficiency checks |
Intermediate |
final answers with rationale |
| Fallback design |
Intermediate |
deterministic safe exits |
Deep-Dive Index
| Topic |
Why it matters |
Link |
| Planner-executor loops |
Core control pattern for reliable single agents |
→ Deep Dive |
Hands-On Exercises
| Exercise |
Input |
Expected output |
| Loop baseline |
docs/starter/stage05_single_agent.py |
step-by-step execution trace |
| Stall detection |
repeated-step test prompts |
fallback triggers |
| Evidence threshold |
observation count rules |
withheld low-confidence answers |
Starter Code
- Starter script:
docs/starter/stage05_single_agent.py
- Optional dataset seed:
data/roadmap/stage-05/loop_prompts.jsonl
Project
| Scope |
Deliverables |
Stretch |
| Bounded research assistant |
planning, evidence checks, safe stop |
adaptive stopping by progress score |
Exit Criteria
- [ ] Agent loop is bounded and inspectable.
- [ ] Repetition or stalling triggers safe fallback.
- [ ] Final outputs include evidence-aware confidence.
Stage 6: Multi-Agent Orchestration
Theory
Stage Outcome
Coordinate specialists through explicit role contracts, shared state boundaries, and deterministic conflict policies.
Competency Snapshot
| Skill |
Target |
Evidence |
| Role contract design |
Intermediate |
stable handoff payloads |
| Conflict resolution |
Intermediate |
deterministic final decisions |
| Orchestration visibility |
Intermediate |
traceable role interactions |
Deep-Dive Index
| Topic |
Why it matters |
Link |
| Role contracts and messaging |
Prevents handoff ambiguity and silent conflicts |
→ Deep Dive |
Hands-On Exercises
| Exercise |
Input |
Expected output |
| Supervisor baseline |
docs/starter/stage06_multi_agent.py |
specialist outputs plus final choice |
| Contract enforcement |
fixed payload schema |
fewer merge errors |
| Conflict handling |
data/roadmap/stage-06/conflict_cases.json |
policy-based decisions |
Starter Code
- Starter script:
docs/starter/stage06_multi_agent.py
- Optional dataset seed:
data/roadmap/stage-06/conflict_cases.json
Project
| Scope |
Deliverables |
Stretch |
| Multi-agent planner |
supervisor plus specialists and merge policy |
HITL escalation for ambiguous conflicts |
Exit Criteria
- [ ] Handoffs use clear role contracts.
- [ ] Conflicts are resolved via documented policy.
- [ ] Final result includes conflict rationale.
Stage 7: Human-in-the-Loop Systems
Theory
Stage Outcome
Design approval gates and escalation logic that improve safety while preserving workflow speed and traceability.
Competency Snapshot
| Skill |
Target |
Evidence |
| Confidence gating |
Intermediate |
deterministic routing decisions |
| Review queue design |
Intermediate |
resumable pending items |
| Auditability |
Intermediate |
append-only decision traces |
Deep-Dive Index
| Topic |
Why it matters |
Link |
| Approval and escalation design |
Enables safe automation with human control |
→ Deep Dive |
Hands-On Exercises
| Exercise |
Input |
Expected output |
| Routing policy run |
docs/starter/stage07_human_loop.py |
auto-approve vs review split |
| Queue resume test |
data/roadmap/stage-07/approval_queue_seed.json |
resumed status updates |
| Audit logging |
approval events |
traceable decision history |
Starter Code
- Starter script:
docs/starter/stage07_human_loop.py
- Optional dataset seed:
data/roadmap/stage-07/approval_queue_seed.json
Project
| Scope |
Deliverables |
Stretch |
| Approval workflow agent |
confidence routing plus resumable queue |
reviewer-side quality analytics |
Exit Criteria
- [ ] Confidence gates are explicit and testable.
- [ ] Review outcomes resume correct workflow state.
- [ ] Audit logs support full reconstruction of decisions.
Stage 8: Evaluation + Quality Assurance
Theory
Stage Outcome
Create repeatable evaluation pipelines with baseline comparison and regression gates for safer prompt and agent changes.
Competency Snapshot
| Skill |
Target |
Evidence |
| Eval dataset design |
Intermediate |
normal plus adversarial coverage |
| Scoring and rubric |
Intermediate |
deterministic pass-rate reporting |
| Regression gating |
Intermediate |
baseline-delta release decisions |
Deep-Dive Index
| Topic |
Why it matters |
Link |
| Eval design and regression gates |
Prevents quality drift in production paths |
→ Deep Dive |
Hands-On Exercises
| Exercise |
Input |
Expected output |
| Harness baseline |
docs/starter/stage08_eval_qa.py |
case-by-case pass report |
| Dataset expansion |
data/roadmap/stage-08/eval_cases.jsonl |
broader failure coverage |
| Baseline diff |
changed agent behavior |
measurable regression deltas |
Starter Code
- Starter script:
docs/starter/stage08_eval_qa.py
- Optional dataset seed:
data/roadmap/stage-08/eval_cases.jsonl
Project
| Scope |
Deliverables |
Stretch |
| Eval CLI and score pipeline |
baseline tracking plus regression alerts |
CI gate integration |
Exit Criteria
- [ ] Eval set covers representative and adversarial cases.
- [ ] Scores are reproducible across repeated runs.
- [ ] Regression thresholds are explicit and enforced.
Stage 9: Observability + Tracing
Theory
Stage Outcome
Instrument agent workflows with step-level traces and SLO-aligned alerting for latency, cost, and reliability.
Competency Snapshot
| Skill |
Target |
Evidence |
| Correlation-id tracing |
Intermediate |
linked per-step logs |
| Latency and cost metrics |
Intermediate |
per-step and total reports |
| Alert policy design |
Intermediate |
threshold-driven anomaly detection |
Deep-Dive Index
| Topic |
Why it matters |
Link |
Tracing and SLO dashboards |
Makes bottlenecks and regressions visible fast |
→ Deep Dive |
Hands-On Exercises
| Exercise |
Input |
Expected output |
| Trace baseline run |
docs/starter/stage09_observability.py |
step latency plus total latency |
| Anomaly threshold test |
data/roadmap/stage-09/anomaly_runs.csv |
threshold alert output |
| Cost attribution |
per-step tags |
cost breakdown by step |
Starter Code
- Starter script:
docs/starter/stage09_observability.py
- Optional dataset seed:
data/roadmap/stage-09/anomaly_runs.csv
Project
| Scope |
Deliverables |
Stretch |
| Observability layer for one agent |
trace, metrics, and alert summary |
dashboard auto-refresh pipeline |
Exit Criteria
- [ ] Each run has a stable correlation ID.
- [ ] Step-level latency and cost are captured.
- [ ] Alert thresholds catch anomalous runs.
Stage 10: Security + Guardrails
Theory
Stage Outcome
Apply layered safeguards for input policy, output redaction, and restricted tool execution against adversarial and accidental misuse.
Competency Snapshot
| Skill |
Target |
Evidence |
| Injection defense design |
Advanced |
blocked red-team attack patterns |
PII redaction |
Advanced |
sanitized logs and outputs |
| Tool policy enforcement |
Advanced |
allowlist-gated execution |
Deep-Dive Index
| Topic |
Why it matters |
Link |
| Injection defense and guardrails |
Reduces security and compliance risk in agent runtimes |
→ Deep Dive |
Hands-On Exercises
| Exercise |
Input |
Expected output |
| Guardrail baseline run |
docs/starter/stage10_security_guardrails.py |
redacted output and command policy result |
| Red-team replay |
data/roadmap/stage-10/redteam_prompts.jsonl |
deny decisions with reasons |
| Policy hardening |
allowlist and pattern updates |
fewer unsafe pass-throughs |
Starter Code
- Starter script:
docs/starter/stage10_security_guardrails.py
- Optional dataset seed:
data/roadmap/stage-10/redteam_prompts.jsonl
Project
| Scope |
Deliverables |
Stretch |
| Hardened agent runtime |
layered policy checks and audit outputs |
policy tuning via false-positive tracking |
Exit Criteria
- [ ] Known attack prompts are blocked.
- [ ] Sensitive output is redacted before storage.
- [ ] Tool execution obeys explicit allowlist policy.
Stage 11: Production Deployment
Theory
Stage Outcome
Ship agent services safely with CI gates, canary rollout checks, and rollback automation tied to quality and latency signals.
Competency Snapshot
| Skill |
Target |
Evidence |
| Release gate design |
Advanced |
tests and eval checks before deploy |
| Canary analysis |
Advanced |
promote vs rollback decisions |
| Rollback readiness |
Advanced |
scripted and repeatable fallback path |
Deep-Dive Index
| Topic |
Why it matters |
Link |
| Deployments, CI, and rollbacks |
Limits production blast radius and downtime |
→ Deep Dive |
Hands-On Exercises
| Exercise |
Input |
Expected output |
| Release decision baseline |
docs/starter/stage11_deployment.py |
deterministic deploy decision |
| Canary metric replay |
data/roadmap/stage-11/canary_metrics.csv |
promote or rollback outcome |
| Gate tuning |
threshold edits |
changed rollout behavior |
Starter Code
- Starter script:
docs/starter/stage11_deployment.py
- Optional dataset seed:
data/roadmap/stage-11/canary_metrics.csv
Project
| Scope |
Deliverables |
Stretch |
| Deployable agent service |
CI gates, canary checks, rollback path |
automated release report generation |
Exit Criteria
- [ ] Build and eval gates are enforced pre-release.
- [ ] Canary metrics drive promotion decisions.
- [ ] Rollback path is documented and scriptable.
Stage 12: Portfolio + Open Source
Theory
Stage Outcome
Package your best agent work into clear case studies and contribution artifacts that are interview-ready and collaboration-ready.
Competency Snapshot
| Skill |
Target |
Evidence |
| Case-study writing |
Intermediate |
architecture and outcomes summary |
| Demo storytelling |
Intermediate |
concise problem-to-results narrative |
| Open-source workflow |
Intermediate |
contribution-ready artifact |
Deep-Dive Index
| Topic |
Why it matters |
Link |
| Portfolio case study and OSS workflow |
Turns project output into professional signal |
→ Deep Dive |
Hands-On Exercises
| Exercise |
Input |
Expected output |
| Template generation |
docs/starter/stage12_portfolio_open_source.py |
generated README outline |
| Artifact package |
one prior stage project |
case-study plus demo notes |
| Contribution rehearsal |
focused change summary |
reviewable OSS-style writeup |
Starter Code
- Starter script:
docs/starter/stage12_portfolio_open_source.py
Project
| Scope |
Deliverables |
Stretch |
| Public project portfolio |
README, ADR, metrics, and demo artifact |
contribution issue and PR draft |
Exit Criteria
- [ ] Portfolio entry includes measurable outcomes.
- [ ] Demo narrative is concise and reproducible.
- [ ] Contribution artifact is clear and review-ready.