Distributed Transactions

Steps: T1=ReserveStock(order_id, items), T2=ChargeCard(order_id, amount), T3=CreateShipment(order_id, address). Compensations: C1=ReleaseReservation(order_id), C2=RefundCharge(order_id), C3=CancelShipment(order_id). T3 failure path: C2 (refund) → C1 (release reservation) — executed in reverse order. Orchestration is better here: a single Order Saga Orchestrator service tracks state (pending/compensating/completed), retries failed steps, and triggers compensations. With choreography, debugging "which service is stuck" across 3 event-driven services is hard. Inconsistency window: between T2 success and C2 refund completion = typically 2–10 seconds (async compensation). User sees "order processing" spinner, then receives "order failed, refund issued" email. The refund itself may take 3–5 business days on the card statement — a real UX problem worth highlighting.

Rate limiting: 3,200 / 500 = 6.4 seconds minimum clearance time. Use a token bucket rate limiter with 500 tokens/sec on the compensation queue consumer. Separate queues for compensation (priority) vs normal cancellations (lower priority). State machine: PENDING_PAYMENT → PAYMENT_FAILED (on explicit failure or timeout) → COMPENSATING_INVENTORY → COMPENSATION_COMPLETE. Ambiguous payments: call Payment API GET /payments/{idempotency_key} to check status. If still ambiguous after 3 retries over 60 seconds, assume FAILED and compensate (refund if needed). Better to over-refund than over-charge. Compensation failure: use exponential backoff (1s, 2s, 4s, 8s, max 5 retries). After 5 failures, push to a dead-letter queue with full context. A human operator reviews DLQ items and manually releases inventory or flags the order for customer service resolution within 1 hour SLA.

2PC flow: Coordinator → Prepare(debit) to A, Prepare(credit) to B → A and B write provisional state and reply VoteYes → Coordinator → Commit to A (done), Commit to B. If coordinator crashes after A commits but before B commits: B is stuck in "prepared" state with locks held indefinitely until coordinator recovers — this is the "blocking" problem. With 1,200 transfers/sec, held locks during 2PC (typically 50–200ms) mean thousands of concurrent locked rows — a performance disaster. Saga at 1,200/sec: T1 debits sender atomically (local ACID transaction), publishes FundsDebited event. T2 credits receiver. If T2 fails: C1 refunds sender. Window: sender is debited for ~100ms–2s while T2 is in-flight. User sees "transfer pending." Recommendation: Saga with idempotent steps is standard for microservices. Use 2PC only within a single database cluster (e.g., Postgres with XA transactions to a linked DB), never across independently-deployed microservices.

TCC phases for Hotel: Try = INSERT INTO reservations (booking_id, room_id, status='TENTATIVE', expires_at=now()+120s). Confirm = UPDATE reservations SET status='CONFIRMED'. Cancel = DELETE FROM reservations WHERE booking_id=? AND status='TENTATIVE'. Flight service: identical pattern with seat reservations. TTL: Hotel and Flight services run a background job that auto-cancels TENTATIVE reservations past expires_at — coordinator sets TTL to 2 minutes. Partial Confirm failure: the coordinator must retry Confirm to Flight with idempotency_key=booking_id indefinitely (with exponential backoff) until Flight responds definitively. The Flight service must handle Confirm(booking_id) idempotently: if already confirmed, return success; if cancelled/unknown, return error. If Flight has actually auto-cancelled (TTL expired during network partition), the coordinator must Cancel Hotel and notify the user to rebook. TCC superiority: Try phase holds inventory without charging — users only pay at Confirm, so no refunds needed. Cost: all third-party APIs must expose Try/Confirm/Cancel — not always possible with external vendors.