Interview Prep Zoneby CuriouserLabs

Question 5 of 5 · architect level

Zero-downtime deployments: rolling updates, backwards-compatible migrations, and the expand/contract pattern.

🎤 Say this first

During any deploy, old and new code run simultaneously against one database — that sentence generates all the rules. Schema changes must be backwards-compatible per release: expand/contract — add nullable column/new table (expand, old code ignores it) → deploy code writing both/reading new → backfill → deploy code on new only → drop old (contract, releases later). Same discipline for APIs (additive changes; never repurpose fields) and events (schema evolution rules). Plus the mechanics: rolling/canary with health gates, graceful shutdown (earlier question), Flyway migrations that never lock tables for minutes (CREATE INDEX CONCURRENTLY, batched backfills), and a rollback story that acknowledges migrations don't roll back — code does.

The full picture

Renaming a column with zero downtime (5 steps, 3 releases)
-- R1 (expand): V12__add_customer_email.sql
ALTER TABLE orders ADD COLUMN customer_email varchar(320);   -- nullable! instant.
-- R1 code: write BOTH email + legacy contact_info; read legacy.

-- between releases: backfill in batches (job, not migration — no long locks)
UPDATE orders SET customer_email = extract_email(contact_info)
 WHERE customer_email IS NULL AND id BETWEEN :lo AND :hi;

-- R2: code reads customer_email (fallback legacy), still writes both.
-- R3 (contract): V15__drop_contact_info.sql   -- only after R2 is EVERYWHERE
ALTER TABLE orders DROP COLUMN contact_info;
-- Rollback story at every step: previous code still works with current schema ✅
  • The compatibility window is bidirectional: new schema must serve old code (rollback!) and old schema must serve new code (rolling window). 'Compatible one release backwards and forwards' is the contract; N-2 if you batch deploys.
  • Migration operational limits: Flyway runs at startup by default — a 20-minute backfill inside a migration = a 20-minute deploy freeze with pods failing readiness. Schema shape in migrations; data movement in resumable jobs.
  • Locking literacy: know your DB — adding a NOT NULL column with default, index creation, type changes: which lock, how long, at your table sizes. CONCURRENTLY, NOT VALID + VALIDATE, and pt-osc/gh-ost exist for the hard cases.
  • Beyond the DB: cache key versions, session/serialization compatibility, event consumers deployed before producers (or schemas additive), and feature flags decoupling release from deploy for the risky switches.
  • Prove it in CI: a pipeline stage that runs old-code-against-new-schema tests catches the incompatible migration before the 2am page does.

🔄 Likely follow-up questions

  • How do you handle a migration that MUST be breaking (data model redesign)?
  • Blue/green vs rolling vs canary — how does the database story differ?
  • How do feature flags interact with expand/contract sequencing?