Interview Prep Zoneby CuriouserLabs

Question 5 of 5 · architect level

From malloc to the OOM killer: how is process memory actually managed, and who dies when RAM runs out?

🎤 Say this first

malloc is a library (glibc/jemalloc) managing arenas it got from the kernel via brk/mmap — most allocations never enter the kernel. Freed memory usually returns to the allocator, not the OS — why RSS stays high after load drops (fragmentation + retention policy, not a leak). Because Linux overcommits virtual memory, the reckoning comes at touch time: if RAM + swap truly run out, the OOM killer picks a victim by oom_score (mostly: biggest RSS, adjustable via oom_score_adj) and SIGKILLs it. In containers it's per-cgroup: exceed your memory limit and your cgroup's process dies — exit 137, no SIGTERM courtesy, no shutdown hooks.

The full picture

  • Allocator anatomy: size-class bins, per-thread caches (tcache/arenas) to avoid lock contention, small allocations from sliced arenas, huge ones straight to mmap. Fragmentation — external (holes) and internal (rounding) — is why long-lived mixed-size workloads bloat; jemalloc/tcmalloc exist largely as better fragmentation/concurrency trades.
  • 'Free' doesn't shrink RSS: allocators keep pages for reuse (MADV_FREE lets the kernel reclaim lazily). Judge leaks by trend under stable load, not by 'it never goes down'.
  • Container OOM math (the JVM classic): limit must cover heap + metaspace + thread stacks + code cache + direct buffers + allocator overhead. Exit 137 with a healthy heap = native side or limit math, not OutOfMemoryError — different diagnosis, different fix (this is the ops mirror of the JVM container-memory question).
  • Defensive knobs: oom_score_adj to shield critical daemons, cgroup memory.high (throttle-before-kill), PSI alerts before the killer wakes, and -XX:+ExitOnOutOfMemoryError so the JVM dies loudly on its OOM rather than limping.
Reading an OOM kill correctly
# Pod restarted, exit code 137, no heap dump, no OutOfMemoryError in logs
kubectl describe pod api-7f...   # -> Last State: OOMKilled

# This was the CGROUP killer, not the JVM:
#   limit 1Gi  vs  RSS = 780Mi heap-touched + 120Mi metaspace/code
#                + 200 threads × 1MB stacks + direct buffers  -> >1Gi on a burst
# Fixes: raise limit / trim stacks (-Xss512k) / cap direct buffers /
#        -XX:MaxRAMPercentage=70 so heap leaves room for everything else.

🔄 Likely follow-up questions

  • Why can RSS grow while the Java heap is stable? (native, direct buffers, allocator arenas)
  • What does memory.high do differently from memory.max in cgroup v2?
  • How would you decide between glibc malloc and jemalloc for a native-heavy service?