Interview Prep Zoneby CuriouserLabs

Question 3 of 5 · staff level

How does Micrometer work, and how do you design good metrics — tags, percentiles, and the mistakes that lie to you?

🎤 Say this first

Micrometer is SLF4J-for-metrics: you instrument against one API (Counter, Timer, Gauge, DistributionSummary), a registry exports to Prometheus/OTLP/Datadog. Boot auto-instruments HTTP server/client, JVM, pools and more. Design rules: low-cardinality tags only (status, method, outcome — never userId/URL-with-ids), latency as percentiles/histograms, never averages, and RED (rate/errors/duration) per endpoint + USE for resources. The classic lies: averaged latency hiding a bimodal p99, unbounded tag cardinality melting the TSDB, and gauges sampled at the wrong moment.

The full picture

Instrumenting with discipline
// Timer with bounded tags + histogram for percentile aggregation
Timer.builder("payment.charge")
     .tag("provider", provider.key())          // ~5 values ✅
     .tag("outcome", outcome)                  // success|declined|error ✅
     // .tag("orderId", orderId)               // ❌ cardinality bomb
     .publishPercentileHistogram()             // aggregatable across pods
     .register(registry)
     .record(() -> provider.charge(order));

// Gauge: reference must be strongly held — lambda over a weak ref silently NaNs
Gauge.builder("orders.queue.depth", queue, Queue::size).register(registry);
  • Cardinality math: series ≈ product of tag values × metrics × pods. uri templated (/orders/{id}) is fine; raw paths are not — Boot templates for you; keep it that way in custom metrics.
  • Percentiles across instances: client-side precomputed percentiles can't be averaged — publish histograms and let Prometheus compute histogram_quantile across the fleet.
  • Counters vs gauges: counters monotonic + rate(); gauges for levels (queue depth, pool usage). Deriving rates from gauges or levels from counters both mislead.
  • @Timed/@Counted + Observation API: annotation-driven for the common case; Observation.createNotStarted(…) gives one instrumentation emitting metrics and trace spans (Boot 3).
  • SLO wiring: alerts on burn rate against SLOs, not on every metric wiggle — metrics design and alert design are one exercise.

🔄 Likely follow-up questions

  • How do you cap cardinality when a tag is business-required (top-N + 'other')?
  • MeterFilter — how do you enforce metric conventions fleet-wide?
  • How do JVM/Hikari auto-metrics help you debug a pool exhaustion live?