Interview Prep Zoneby CuriouserLabs

Question 4 of 5 · staff level

ASCII to Unicode to UTF-8: how does text actually work, and where do Java strings lie to you?

🎤 Say this first

Separate the two layers everyone conflates: Unicode assigns numbers (code points, ~1.1M possible) to characters; encodings (UTF-8/16/32) turn code points into bytes. UTF-8 won the world: ASCII-compatible, byte-oriented (no endianness), self-synchronizing, 1–4 bytes per code point. Java's lie: char is a UTF-16 code unit, not a character — anything beyond U+FFFF (emoji, many CJK) is a surrogate pair: "🚀".length() == 2, charAt hands you half a character, and naive reversal corrupts text. And user-perceived characters (grapheme clusters — 👨‍👩‍👧 is one 'character', many code points) are yet another layer up. Correct code iterates code points or graphemes, never chars.

The full picture

Where String's abstraction leaks
"🚀".length()                    // 2   (UTF-16 code units)
"🚀".codePointCount(0, 2)        // 1   (code points — the truth)
"🚀".charAt(0)                   // '\uD83D' — half a rocket (high surrogate)
new StringBuilder("🚀x").reverse()  // "x🚀"? Java handles pairs since 1.5… but
                                    // "👨‍👩‍👧" (ZWJ sequence) still shatters.

// Iterate honestly:
"a🚀b".codePoints().forEach(cp -> ...);          // code points
BreakIterator.getCharacterInstance()             // grapheme clusters (user chars)

// Bytes require a contract — NEVER default charset (pre-18 platform roulette;
// JEP 400 made UTF-8 the default, but be explicit anyway):
string.getBytes(StandardCharsets.UTF_8);
new String(bytes, StandardCharsets.UTF_8);
  • UTF-8's design is worth admiring aloud: prefix bits encode sequence length, continuation bytes are recognizable (10xxxxxx) → you can resync mid-stream, sort by bytes ≈ sort by code point, and pure ASCII is valid UTF-8 unchanged — engineering elegance that explains its 99% web share.
  • The practical bug catalog: length checks for DB columns (chars? code points? bytes in the column's charset — MySQL utf8 vs utf8mb4!), substring truncation splitting surrogates/graphemes (broken emoji in notifications), regex . matching half a pair, and mojibake from double-encoding (UTF-8 read as Latin-1 re-encoded: 'é').
  • JVM internals footnote: since Java 9, compact strings store Latin-1 text as one byte per char internally — memory win, invisible semantically; char's UTF-16 contract is unchanged.
  • Normalization exists: 'é' is one code point or e+combining-accent — equal to humans, unequal to equals(); Normalizer.normalize(s, NFC) before comparing/deduplicating user text (and Unicode-aware casing: Turkish i is the canonical war story).

🔄 Likely follow-up questions

  • utf8 vs utf8mb4 in MySQL — what actually happened there?
  • How do you truncate a string to N bytes safely for a fixed-size field?
  • What does normalization (NFC/NFD) mean for usernames and search?