Interview Prep Zoneby CuriouserLabs

Question 4 of 5 · staff level

How do secondary indexes work across shards — local vs global, and what does each cost?

🎤 Say this first

Two designs. A local (document-partitioned) index lives on the same shard as the data it indexes: writes are cheap and atomic (index and row are on one node), but a query on a non-partition-key attribute must scatter-gather to every shard and merge — cost and tail latency grow with cluster size. A global (term-partitioned) index partitions by the indexed term instead, so a lookup reads exactly one shard — but every write now touches a different shard than the row, making writes distributed and the index asynchronous (or you pay two-phase commit). The rule of thumb: local indexes make writes fast and reads expensive; global indexes make reads fast and writes expensive-or-stale. Elasticsearch and Cassandra's SASI are local; DynamoDB GSIs are global and explicitly eventually consistent.

The full picture

Local (document-partitioned)Global (term-partitioned)
Partitioned bythe document's partition keythe indexed term
Write costlocal, atomic with the rowcross-shard, usually async
Read costscatter-gather across all shardssingle shard
Consistencyindex matches the roweventually consistent
ExamplesElasticsearch shards, Cassandra 2iDynamoDB GSI, term-partitioned search
  • Scatter-gather latency is governed by the slowest shard, so p99 read latency degrades as you add shards — the counter-intuitive result that growing the cluster makes these queries worse. With 100 shards, a query is only as fast as the unluckiest of 100 GC pauses. This is the single most important consequence to state.
  • DynamoDB GSIs are the concrete case to reason about: a GSI is a separate table maintained asynchronously, with its own partition key, its own throughput, and its own hot-partition risk. It can be behind the base table, and — the operational trap — if the GSI throttles, writes to the base table can be throttled too. Real constraints like this beat abstract comparisons.
  • Cassandra's built-in secondary indexes are the classic anti-pattern: local indexes that scatter to every node, so they only perform when combined with the partition key (which largely defeats the purpose). The idiomatic answer there is a manually maintained inverted table — write the data twice, keyed differently. Denormalization is the index.
  • The write-twice pattern generalizes far beyond Cassandra: maintaining your own query-shaped table (or a CQRS read model, or a search index fed by CDC) is how most large systems handle secondary access paths. The cost moves from query time to write time, plus the eventual-consistency window and the reconciliation job to detect drift.
  • Bounded fan-out is the middle ground: if a query can be narrowed to a few shards (routing by tenant, or a coarse hash bucket), scatter-gather stays cheap. Elasticsearch's routing key does exactly this — same index design, dramatically better tail latency, at the cost of skew risk.

🔄 Likely follow-up questions

  • Why does adding shards make a scatter-gather query slower at p99?
  • What happens to base-table writes when a DynamoDB GSI is throttled?
  • How would you detect and repair drift between a table and its async index?