Liveness vs readiness: how do you design health checks for Kubernetes, and what goes into a custom HealthIndicator?

The full picture

Probe groups + a disciplined custom indicator

management:
  endpoint.health.probes.enabled: true
  endpoint.health.group:
    readiness:
      include: readinessState, db, kafkaConsumerLag   # required deps only
    liveness:
      include: livenessState                          # NOTHING external!

@Component("kafkaConsumerLag")
class ConsumerLagHealth implements HealthIndicator {
    private volatile Health cached = Health.up().build();   // refreshed async
    @Scheduled(fixedDelay = 10_000)
    void refresh() {                                        // probe never blocks
        long lag = lagMonitor.maxLag();
        cached = lag < 10_000 ? Health.up().withDetail("lag", lag).build()
                              : Health.outOfService().withDetail("lag", lag).build();
    }
    @Override public Health health() { return cached; }
}

The cascade failure to name: DB in liveness → DB hiccup → kubelet restarts every pod → connection storm on recovering DB → longer outage. Liveness checks dependencies is the #1 probe anti-pattern.
Readiness nuance: only hard dependencies (can't do anything without the DB); optional dependencies (recommendation service) belong in degraded-mode logic + alerts, not readiness — or you turn partial degradation into full outage.
Timing: probes run every few seconds from kubelet — health() must be O(cached-read); Boot's AvailabilityState events let graceful shutdown flip readiness before stopping the server (drain window).
Startup probes (or initialDelay) protect slow-booting JVMs from liveness-killing them mid-warmup.

Liveness vs readiness: how do you design health checks for Kubernetes, and what goes into a custom HealthIndicator?

🎤 Say this first

The full picture

🔄 Likely follow-up questions