Interview Prep Zoneby CuriouserLabs

Question 5 of 5 · architect level

Design the observability strategy for a 30-service platform: signals, SLOs, alerting and cost.

🎤 Say this first

Start from questions, not tools: detect (is it broken? → SLO-based alerts on burn rate), triage (where? → RED dashboards per service + trace exemplars), diagnose (why? → traces + correlated logs + profiles). Standardize via a platform starter — one logging format (structured JSON + trace ids), one metrics convention, one propagation format — so every service is observable by default. Alert only on symptoms (SLO burn), page humans for user-impact, ticket the rest. Manage cost deliberately: sampling tiers, log levels + retention classes, metric cardinality budgets — observability spend can rival compute if unmanaged.

The full picture

  1. SLOs first: per service, 2–4 user-journey SLIs (availability, latency p99) with error budgets — they define what's alert-worthy and end the 'CPU is 80%!' paging culture.
  2. Paved road: acme-starter-observability wires Micrometer + OTLP + JSON logging + MDC correlation + standard tags (service, version, region). Observability by dependency, not by per-team diligence.
  3. Alert taxonomy: page on burn rate (fast+slow windows), ticket on trends (cert expiry, disk), dashboard the rest. Every page must be actionable — an alert nobody acts on is deleted or demoted.
  4. Correlation contract: trace id in every log line, exemplars linking metrics→traces, deploy markers on dashboards — the 3am path is metric alert → trace → logs in three clicks.
  5. Cost levers: tail-sampled traces (keep errors/slow), log retention tiers (7d hot / 90d cold), cardinality linting in CI, and per-team showback — visibility of spend changes behavior.
QuestionSignalOwner artifact
Is it broken for users?SLO burn-rate alertsSLO definitions per service
Where is it broken?RED dashboards + service mapStandard Grafana folder per service
Why is it broken?Traces + logs + continuous profilingCorrelation via trace id everywhere
Did the deploy do it?Deploy markers + version tag on all signalsCD pipeline annotations

🔄 Likely follow-up questions

  • Walk through a real 3am debug using the three signals — where does each hand off?
  • How do you retrofit this onto 30 services that all log differently today?
  • What would you cut first when the observability bill doubles?