The full picture
- Mechanics to name:
traceparent: 00-<traceId>-<spanId>-<flags>header;ObservationAPI creates spans + metrics from one instrumentation; MDC getstraceId/spanIdso every log line joins the trace (logging.pattern.correlationout of the box in Boot 3.2+). - The propagation gotchas: thread hops (
@Async, executors) need context-propagation wrappers (ContextExecutorService, Reactor hooks); Kafka consumers start new traces linked via headers; virtual threads work but ThreadLocal-heavy custom propagation deserves an audit. - Sampling strategy: 100% in staging; production head-sampling (1–10%) + always-sample-on-error/slow via tail sampling in the OTel Collector — the traces you keep should be the ones you'd grep for.
- What tracing answers that metrics can't: 'which downstream of the 7 in this request was slow, for THIS request' — metrics aggregate, traces narrate. Logs answer 'what did the code decide'. Three signals, three questions.
- Baggage (propagated key-values, e.g., tenantId) is powerful and dangerous — every hop carries it; keep the allowlist tiny.
private final ObservationRegistry observations;
Order process(Command cmd) {
return Observation.createNotStarted("order.process", observations)
.lowCardinalityKeyValue("type", cmd.type().name()) // metric tag + span tag
.highCardinalityKeyValue("orderId", cmd.orderId()) // span-only (cost-aware!)
.observe(() -> doProcess(cmd)); // span + timer, one API
}