The full picture
| Routing model | How the client finds data | Trade-off |
|---|---|---|
| Proxy tier | client → proxy → shard | one extra hop; easy central control (Vitess, mongos) |
| Cluster-aware client | client caches the map, connects directly | fastest; map staleness must be handled |
| Any-node forwarding | any node proxies to the owner | simplest client; internal traffic amplification |
| Redirect (MOVED/ASK) | node replies 'go there instead' | self-healing client caches — Redis Cluster's model |
- Stale maps are the normal case, so design for them: clients will hold an old map during a move. Systems handle this with redirects (Redis
MOVED/ASK), with retries on a versioned map epoch, or with the source node forwarding during migration. 'How does a client with a stale map behave?' is the question that separates a design from a sketch. - Fixed partitions are the operationally boring choice, which is the point: pick 1024 partitions on day one for a 4-node cluster, and each node owns 256. Grow to 16 nodes and each owns 64 — partitions never split, they only move. The constraint is that the partition count caps your maximum node count and is painful to change (Kafka partitions, Elasticsearch shards).
- Automatic rebalancing plus an automatic failure detector is a dangerous combination: a node is slow, the detector marks it dead, rebalancing kicks off, the resulting data movement makes the cluster slower, more nodes look dead — a cascade. Many mature systems deliberately keep rebalancing manual or approval-gated for exactly this reason, and knowing why is a strong architect-level point.
- The safe migration sequence is the same everywhere: dual-write (or replicate) to the new owner → backfill historical data → verify by comparison → shift reads gradually → cut over writes → keep the old copy readable until you're confident → delete. Every step is reversible except the last, and the verification step is the one teams skip and regret.
- Throttle and observe: rebalancing competes with production for disk, network and page cache. Rate-limit the transfer, run it off-peak, and watch p99 latency rather than just transfer progress. A rebalance that finishes in one hour and breaks your SLO is worse than one that takes a day.