Skip to content

Distributed Tracing — Deep Dive

Level: Intermediate
Pre-reading: 07 · Observability


What is Distributed Tracing?

Distributed tracing tracks a request as it flows through multiple services, showing where time is spent.

graph LR
    subgraph Trace abc123
        S1[Gateway - 10ms] --> S2[Order - 45ms]
        S2 --> S3[Payment - 200ms]
        S2 --> S4[Inventory - 30ms]
    end

Tracing Concepts

Concept Description
Trace End-to-end journey of a request
Span Single operation within a trace
Trace ID Unique identifier for the trace
Span ID Unique identifier for each span
Parent Span ID Links child spans to parent

flowchart TB
    T["trace-id: 4bf92f…<br/>ONE request"] --> S1["span · GET /checkout<br/>api-gateway · 120ms"]
    S1 --> S2["span · POST /orders<br/>order-svc · 90ms"]
    S1 --> S5["span · GET /inventory<br/>inventory-svc · 40ms"]
    S2 --> S3["span · INSERT<br/>postgres · 12ms"]
    S2 --> S4["span · publish OrderCreated<br/>kafka · 5ms"]
    S5 -.->|"gap = uninstrumented<br/>hop or slow code"| G(("?"))

Context Propagation

Trace context travels in HTTP headers:

W3C TraceContext (Standard)

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: vendor=value

B3 (Zipkin)

X-B3-TraceId: 463ac35c9f6413ad48485a3953bb6124
X-B3-SpanId: a2fb4a1d1a96d312
X-B3-ParentSpanId: 0020000000000001
X-B3-Sampled: 1

OpenTelemetry

OpenTelemetry (OTel) is the CNCF standard for traces, metrics, and logs.

graph LR
    A[Application + OTel SDK] --> C[OTel Collector]
    C --> J[Jaeger]
    C --> P[Prometheus]
    C --> L[Loki]

Spring Boot Integration

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
management:
  tracing:
    sampling:
      probability: 1.0  # 100% sampling (reduce in prod)
  otlp:
    tracing:
      endpoint: http://otel-collector:4318/v1/traces

Sampling Strategies

Strategy Description Use Case
Head-based Decide at request entry Simple; may miss errors
Tail-based Decide after seeing full trace Captures errors; more complex
Rate limiting Sample N traces/second Control cost

Tracing Tools

Tool Type Notes
Jaeger OSS CNCF; Kubernetes-native
Zipkin OSS Twitter origin; simple
Tempo OSS Grafana; cost-effective storage
AWS X-Ray Managed AWS native
Honeycomb SaaS High-cardinality analysis

What's the difference between tracing and logging?

Tracing shows the flow and timing of a request across services. Logging captures discrete events with context. Traces answer "where did time go?"; logs answer "what happened?". Use both together; link logs to traces via trace ID.

How do you decide on sampling rate?

Start with 100% in dev. In production: (1) Cost constraints — traces are expensive. (2) Traffic volume — high-traffic needs lower rate. (3) Error capture — use tail-based to always capture errors. Common: 1-10% for high-traffic, 100% for critical paths.

Head-based vs tail-based sampling?

Head-based decides at trace start — simple but may miss errors. Tail-based decides after seeing complete trace — captures all errors and slow traces. Use head-based for most cases; add tail-based for critical paths to ensure error capture.


📝 Knowledge Check

#

Question 1

What is a span, and how do spans become a trace across five services?

#

Question 2

W3C TraceContext vs B3 — why does the header standard matter?

#

Question 3

100% trace sampling vs 1% vs head-based vs tail-based — when does each make sense?

#

Question 4

Tracing an async (Kafka) hop — what does the producer/consumer need to do to keep the trace continuous?

#

Question 5

A trace shows 2s inside 'OrderService' with no child spans during the gap. What are the likely explanations?