Interview Prep Zoneby CuriouserLabs

Question 2 of 5 · staff level

Why can't you order events by `System.currentTimeMillis()` across machines?

🎤 Say this first

Because wall-clock time on different machines disagrees, and it doesn't even move monotonically on one machine. NTP typically holds skew to single-digit milliseconds in a datacenter but tens to hundreds across the internet — and it fixes drift by stepping or slewing the clock, so time can jump forward or backward. Consequences: a timestamp-ordered comparison between two hosts is unreliable at sub-second granularity, last-write-wins can pick the older write, and a duration measured with wall-clock time can come out negative (use a monotonic clock — System.nanoTime() — for elapsed time, never for timestamps). The distributed fix is to stop pretending physical time is an ordering primitive: use logical clocks for causality, or hybrid logical clocks when you need timestamps that are both roughly-real and causally correct. Spanner's alternative is to make uncertainty explicit — TrueTime returns an interval and commit-wait sleeps it out.

The full picture

Two clocks, two jobs — mixing them up is a real bug
// WRONG: wall clock for elapsed time. NTP steps the clock mid-measurement and
// this can be negative, or wildly wrong, on a machine that just synced.
long t0 = System.currentTimeMillis();
doWork();
long elapsed = System.currentTimeMillis() - t0;

// RIGHT: monotonic clock for durations. Never jumps, never goes backwards,
// meaningless as an absolute time — which is exactly the correct contract.
long t0 = System.nanoTime();
doWork();
Duration elapsed = Duration.ofNanos(System.nanoTime() - t0);

// And for cross-machine ordering, neither clock is sufficient:
// use a logical/hybrid clock, or a sequence assigned by a single writer.
  • Leap seconds and clock steps are not hypothetical: the 2012 leap second took down Reddit, Mozilla and others via a kernel livelock. Modern practice is leap smearing (Google/AWS spread the second over hours) precisely because 'time goes backwards for one second' broke too much software.
  • LWW conflict resolution inherits every clock bug: with 100ms of skew, a write made later in real time can carry an earlier timestamp and lose. This is the concrete link between this question and conflict resolution — clock skew silently becomes data loss.
  • Hybrid Logical Clocks (HLC) are the practical sweet spot and worth naming: a physical component (so timestamps are human-meaningful and roughly sortable) plus a logical counter (so causality is never violated). CockroachDB and YugabyteDB rely on them; MongoDB's cluster time is the same idea.
  • Spanner's TrueTime is the 'buy your way out' answer: GPS and atomic clocks bound uncertainty to a few milliseconds, the API returns [earliest, latest], and a commit waits out that interval before releasing — trading latency for globally consistent timestamps. It's the clearest demonstration that real-time ordering has a literal price.
  • Practical hygiene worth stating: run NTP/chrony everywhere with the same upstream sources, alarm on offset (not just 'is ntpd running'), and never let a business rule depend on sub-second cross-host ordering. If two events must be ordered, give them a sequence from a single authority rather than a timestamp from two.

🔄 Likely follow-up questions

  • How does an HLC guarantee causality while still looking like a real timestamp?
  • What exactly does Spanner's commit-wait wait for, and what does it cost?
  • Where in your stack would a 200ms backward clock step cause a correctness bug?