The full picture
// Timer with bounded tags + histogram for percentile aggregation
Timer.builder("payment.charge")
.tag("provider", provider.key()) // ~5 values ✅
.tag("outcome", outcome) // success|declined|error ✅
// .tag("orderId", orderId) // ❌ cardinality bomb
.publishPercentileHistogram() // aggregatable across pods
.register(registry)
.record(() -> provider.charge(order));
// Gauge: reference must be strongly held — lambda over a weak ref silently NaNs
Gauge.builder("orders.queue.depth", queue, Queue::size).register(registry);- Cardinality math: series ≈ product of tag values × metrics × pods.
uritemplated (/orders/{id}) is fine; raw paths are not — Boot templates for you; keep it that way in custom metrics. - Percentiles across instances: client-side precomputed percentiles can't be averaged — publish histograms and let Prometheus compute
histogram_quantileacross the fleet. - Counters vs gauges: counters monotonic +
rate(); gauges for levels (queue depth, pool usage). Deriving rates from gauges or levels from counters both mislead. - @Timed/@Counted + Observation API: annotation-driven for the common case;
Observation.createNotStarted(…)gives one instrumentation emitting metrics and trace spans (Boot 3). - SLO wiring: alerts on burn rate against SLOs, not on every metric wiggle — metrics design and alert design are one exercise.