Interview Prep Zoneby CuriouserLabs

Question 4 of 5 · architect level

Timeouts, retries, circuit breakers, bulkheads: compose resilience for a service calling three downstreams.

🎤 Say this first

Order the layers from inside out: timeout on every remote call (no exceptions — the un-timed-out call is the outage seed) → retry only idempotent operations, few attempts, exponential backoff with jitter, budgeted (retry storms take down recovering systems) → circuit breaker per dependency to fail fast and give it air → bulkhead (separate pools/semaphores per dependency) so one slow downstream can't consume all capacity → fallback where the business allows degradation (cached prices, default recommendations, queue-for-later). Resilience4j provides all five as decorators/annotations; the design work is choosing per-dependency policies and testing them with fault injection.

The full picture

Composed, per-dependency, observable
resilience4j:
  circuitbreaker.instances.inventory:
    slidingWindowSize: 50
    failureRateThreshold: 50        # open at 50% failures
    slowCallDurationThreshold: 800ms
    slowCallRateThreshold: 80       # slow IS failure — brownouts count
    waitDurationInOpenState: 10s    # then half-open probes
  retry.instances.inventory:
    maxAttempts: 3
    waitDuration: 200ms
    enableExponentialBackoff: true
    retryExceptions: [com.acme.TransientInventoryException]  # NEVER on 4xx
  timelimiter.instances.inventory:
    timeoutDuration: 1s

@CircuitBreaker(name = "inventory", fallbackMethod = "cachedLevels")
@Retry(name = "inventory")
@TimeLimiter(name = "inventory")           // order: TL( CB( Retry( call )))
public CompletableFuture<Levels> levels(SkuId sku) { … }
Levels cachedLevels(SkuId sku, Throwable t) { return cache.lastKnown(sku); }
  • Retries are dangerous by default: only idempotent ops (GET, or POST with idempotency keys); cap total attempts across the call chain (A retries 3× calling B which retries 3× = 9× load on C — the multiplication that turns a blip into an incident); jitter or synchronized retry waves hammer the recovering service.
  • Slow-call thresholds matter more than error thresholds: dependencies rarely die cleanly — they brown out; a breaker that only counts exceptions stays closed while p99 goes to 30s.
  • Bulkheads = blast-radius design: separate thread pools (or semaphores/VT limits) per downstream; the 'recommendations is slow so checkout is down' incident is a missing bulkhead, every time.
  • Fallbacks are product decisions: stale-but-available vs error — decide per capability with the business, and make degraded mode visible (metric + banner), or you'll run degraded for a week unnoticed.
  • Verify with chaos: fault-injection tests (Toxiproxy latency/failures in integration tests) prove the composition; untested resilience config is decorative YAML.

🔄 Likely follow-up questions

  • Derive a sane end-to-end timeout budget for a 3-deep call chain with a 2s SLO.
  • Where do idempotency keys live and who generates them?
  • Circuit breaker per-instance vs shared state across pods — trade-offs?