Skip to content

20 · Saga Pattern — Compensating Distributed Transactions

Distributed Transactions · Topic 20 of 20


The Problem with 2PC at Scale

Two-Phase Commit holds locks and blocks on coordinator failure — unacceptable for long-running business workflows across microservices.

The Saga pattern breaks a distributed transaction into a sequence of local transactions, each with a corresponding compensating transaction that can undo its effect.


How Saga Works

Each step is a local DB transaction. If a step fails, compensating transactions run in reverse to undo the completed steps.

graph LR
    T1["T1: Place Order\n(Compensate: Cancel Order)"]
    T2["T2: Reserve Inventory\n(Compensate: Release Inventory)"]
    T3["T3: Charge Payment\n(Compensate: Refund)"]
    T4["T4: Ship Order\n(Compensate: Recall Shipment)"]

    T1 --> T2 --> T3 --> T4

If T3 definitively fails before charging, compensate completed steps: C2 (Release Inventory) → C1 (Cancel Order). If the charge outcome is unknown, reconcile it first; refund only if a charge actually occurred.


Saga Coordination Models

Choreography

Each service publishes events and reacts to events from other services. No central coordinator.

  • ✅ Decoupled, no single point of failure
  • ❌ Hard to trace end-to-end flow; risk of cyclic dependencies

Orchestration

A Saga orchestrator service explicitly tells each participant what to do.

  • ✅ Centralized visibility and control
  • ❌ Orchestrator is a new service to maintain
sequenceDiagram
    Orchestrator->>OrderService: Place Order
    Orchestrator->>InventoryService: Reserve Inventory
    Orchestrator->>PaymentService: Charge Payment
    PaymentService-->>Orchestrator: FAILED
    Orchestrator->>InventoryService: Release Inventory (compensate)
    Orchestrator->>OrderService: Cancel Order (compensate)

Idempotency is Required

Since messages can be retried, every step must be idempotent: processing the same message twice must produce the same result.

-- Idempotent reservation
INSERT INTO reservations (id, item_id, qty, saga_id)
VALUES (?, ?, ?, ?)
ON CONFLICT (saga_id) DO NOTHING;

Cloud Implementations

  • Native saga/orchestration engine
  • State machine DSL; retries and error handling built-in
  • Integrates with DynamoDB, Lambda, SQS, SNS
  • Orchestration service for multi-step GCP workflows
  • Compensation steps defined as on-error branches
  • Each service writes events to its local DB (outbox table) atomically with business data
  • CDC (Debezium) reads outbox and publishes to Kafka
  • Next service consumes and processes
  • Write business record + event atomically with DynamoDB Transactions
  • Lambda processes SQS events per saga step
BEGIN;
  INSERT INTO orders (...) VALUES (...);
  INSERT INTO outbox (event_type, payload) VALUES ('OrderPlaced', '...');
COMMIT;
-- Debezium/CDC picks up outbox row and publishes to Kafka

Saga vs 2PC

2PC Saga
Locks May be held across participants Local transactions may lock; no global lock held for the workflow
Failure May block awaiting a durable decision Retry, reconcile or compensate; compensation can also fail
Atomicity Atomic commit across participating resources No global ACID atomicity; intermediate states are visible
Complexity Protocol complexity Compensation logic complexity
Latency High Lower per step
Best for Short OLTP transactions Long-running business workflows

Compensation Is a New Business Action

A refund is not time travel: fees, emails and external observations may remain. A shipped parcel may not be recallable at all; treat irreversible steps as a pivot, placing reversible reservations before them when possible. Persist workflow state, step idempotency keys, deadlines and retry counts so the orchestrator can recover after a crash.

The reservation snippet above assumes a unique constraint on saga_id; for multiple line items use a suitable per-step/per-item unique key. Duplicate suppression must verify that a reused key carries the same intent, not silently accept a different quantity. Bound retries, surface stuck compensation to operators and reconcile uncertain payment outcomes.

Check Your Understanding

#

Question 1

What prevents two sagas from overselling the same item?

#

Question 2

What should happen if a compensation fails?