Skip to content

Learning: Observability + Tracing

How to Use This Page

  • Instrument first, optimize second.
  • Correlate traces, logs, and metrics by request id.
  • Focus dashboards on user-impacting signals.

Lesson 1: Fundamentals

Learn request tracing and structured event logging.

What You Learn

  • How correlation ids connect distributed workflow steps
  • Why structured logs outperform plain text for debugging
  • Which step-level fields are mandatory for troubleshooting

Walkthrough

  1. Add a request id to every run.
  2. Emit one structured event per major step.
  3. Capture status, duration, and key context fields.

Try It Yourself

  • Reconstruct one failing request end-to-end from logs.
  • Remove one required field and observe analysis friction.
  • Add a simple trace viewer sorted by timestamp.

Lesson 2: Intermediate Patterns

Learn cost and latency analytics across runs.

What You Learn

  • How to compute per-step and per-request cost
  • Why P95 latency reveals user pain better than average
  • How to identify expensive bottlenecks quickly

Walkthrough

  1. Aggregate latency by stage and operation type.
  2. Add token and infrastructure cost estimates.
  3. Plot median and P95 trends over time.

Try It Yourself

  • Compare two workflow variants using the same test set.
  • Identify one step that dominates tail latency.
  • Propose one optimization and estimate impact.

Lesson 3: Production Patterns

Learn SLO policy, alerting, and anomaly response.

What You Learn

  • How to define SLO targets tied to user experience
  • Why error budgets help prioritize incident response
  • How anomaly detection reduces blind spots

Walkthrough

  1. Define SLOs for latency and success rate.
  2. Configure alert thresholds and severity levels.
  3. Route high-severity events to incident workflow.

Try It Yourself

  • Simulate an SLO breach and verify alert flow.
  • Measure alert noise over one test window.
  • Adjust thresholds to improve signal-to-noise ratio.

Code Examples

  • Starter script: docs/starter/stage09_observability.py
  • Stage architecture notes: docs/roadmap/stage-09-tracing-and-slo-dashboards/architecture.md