Unicode and Text Encoding Without Surprises

Text bugs become mysterious when software treats “character” as a precise unit. It is not. A user sees letters, marks, emoji, and words; a program may store bytes, index UTF-16 code units, iterate Unicode code points, or segment extended grapheme clusters. Those units coincide for simple ASCII and diverge as soon as text contains an accented letter, a supplementary-plane symbol, a combining mark, or an emoji sequence.

Unicode solves the problem of assigning interoperable identities to text elements, but it does not make every string operation culturally or visually obvious. An encoding maps those identities to storage units. Normalization reconciles some canonically equivalent sequences. Segmentation approximates what users perceive as characters. Collation supplies language-sensitive ordering and equality. Each layer answers a different question.

The central thesis is therefore: choose a text unit and a comparison rule for the operation at hand, and make transcoding an explicit validated boundary. Byte length is right for a network limit, code-point iteration is useful for scalar processing, grapheme clusters are usually right for cursor movement and truncation, and locale-aware collation is right for many human-facing comparisons. One universal length operation cannot stand in for all four.

Separate Bytes, Code Units, Code Points, and Graphemes #

A Unicode code point is an integer in the range U+0000 through U+10FFFF. Code points name letters, marks, punctuation, control characters, emoji components, and many other elements. Surrogate code points are reserved for UTF-16 mechanics and are not Unicode scalar values. Text interchange should encode scalar values, not isolated surrogates.

An encoding represents scalar values as sequences of code units. A UTF-8 code unit is an 8-bit byte. A UTF-16 code unit is a 16-bit value. A UTF-32 code unit is a 32-bit value. A runtime’s string indexing usually follows its internal code-unit model, not visual characters. JavaScript and Java expose UTF-16 code-unit indexing; many systems APIs and files expose bytes; other languages may hide representation behind iterators.

A grapheme cluster is a sequence intended to behave as one user-perceived character under Unicode segmentation rules. The visible é can be one precomposed code point, U+00E9, or two code points, U+0065 U+0301, where the second is a combining acute accent. A family emoji can contain several people joined by zero-width joiners. A flag commonly consists of two regional-indicator code points. None of those facts is captured by “one code point equals one character.”

The layers can be summarized by the questions they answer:

Glyphs add another layer. A font shaping engine can render several code points as one ligature or one code point with different contextual shapes. Conversely, a missing font can show a replacement box. Unicode defines text data and algorithms; it does not promise one code point, one glyph, or one column on a terminal.

Understand the UTF Encodings #

UTF-8 represents each scalar value in one to four bytes. ASCII values use the same single bytes as ASCII, making valid ASCII a subset of UTF-8. Higher values use leading-byte patterns followed by continuation bytes. The representation is self-synchronizing enough that a decoder can identify character boundaries from byte patterns, but cutting an arbitrary byte range can still produce an incomplete sequence.

UTF-16 represents values in the Basic Multilingual Plane with one 16-bit code unit, except reserved surrogates. Supplementary values use a high-surrogate and low-surrogate pair. Consequently, JavaScript’s "😀".length is 2, and indexing either position yields half of the pair. for...of uses the string iterator and advances by code point, which is safer for scalar processing but still splits multi-code-point graphemes.

UTF-32 uses one 32-bit code unit per scalar value. That makes scalar indexing straightforward, but it consumes four bytes even for ASCII and does not solve grapheme segmentation, normalization, collation, or display width. “Fixed width” applies to code points in that encoding, not to user-perceived characters or rendered glyphs.

UTF-16 and UTF-32 byte streams require an endianness convention. A byte-order mark can signal byte order, and U+FEFF has a historical dual role, but protocols should define encoding and endianness instead of guessing. UTF-8 has no byte-order ambiguity; a leading UTF-8 byte-order mark is legal in some contexts but can surprise parsers that treat the first bytes as content.

A decoder must specify malformed-input behavior. Strict decoding rejects invalid lead bytes, invalid continuations, overlong encodings, encoded surrogates, out-of-range values, and truncated sequences. Replacement decoding emits U+FFFD for malformed portions and continues. Replacement is useful for displaying damaged text, but it is lossy: re-encoding cannot recover the original bytes. Security-sensitive identifiers, signatures, and source formats should usually reject malformed input rather than silently repair it.

Do not infer an encoding from text appearance. A sequence of bytes may be valid under several legacy encodings and produce different text. Prefer protocols that declare UTF-8, validate once at ingress, and keep strings in the runtime’s native representation internally. When legacy conversion is unavoidable, record the declared source encoding and surface conversion errors.

Worked Example: One String Through Every Layer #

Consider the conceptual string Café 👩🏽‍💻, where the accent is stored as e followed by U+0301 COMBINING ACUTE ACCENT. The emoji sequence contains U+1F469 WOMAN, U+1F3FD MEDIUM SKIN TONE, U+200D ZERO WIDTH JOINER, and U+1F4BB LAPTOP.

The first word has five code points but four grapheme clusters. The emoji has four code points but one extended grapheme cluster. Including the space, the complete string has ten code points and six grapheme clusters. In UTF-16 it has thirteen code units: each of the woman, skin-tone, and laptop values requires a surrogate pair, while the other code points use one unit each. Its UTF-8 byte length is larger again. The exact number relevant to a database column or message limit must be computed in that boundary’s encoding.

JavaScript makes the distinctions visible:

const text = 'Cafe\u0301 👩🏽‍💻';

const utf16Units = text.length;
const codePoints = Array.from(text);
const utf8Bytes = new TextEncoder().encode(text);
const graphemes = Array.from(
  new Intl.Segmenter('en', { granularity: 'grapheme' }).segment(text),
  (part) => part.segment,
);

console.log({
  utf16Units,             // 13
  codePointCount: codePoints.length, // 10
  utf8ByteCount: utf8Bytes.length,
  graphemeCount: graphemes.length,   // 6
  graphemes,
});

Now imagine a UI that permits six “characters.” text.slice(0, 6) slices six UTF-16 code units. It happens to end after the space for this value, but changing the limit by one can retain only the high surrogate of the woman emoji. Slicing Array.from(text) avoids broken surrogate pairs, yet a cut inside the four-code-point emoji leaves a woman, a skin-tone component, or an unfinished joiner sequence rather than the original visible symbol. Grapheme segmentation is the relevant boundary for UI truncation.

The example also exposes why visual inspection is an inadequate equality test. The precomposed string Café and the decomposed first word above can render identically while comparing unequal byte-for-byte and code-point-for-code-point. Normalization can make those particular forms comparable, but it does not collapse every pair of strings that users may regard as equivalent.

Normalize for a Declared Invariant #

Unicode normalization defines standard transformations. NFC performs canonical decomposition followed by canonical composition where possible. NFD leaves text canonically decomposed. NFKC and NFKD additionally apply compatibility mappings, which can replace formatting distinctions such as certain presentation forms with a more basic representation.

Canonical equivalence is useful for storage, lookup, and deduplication when the domain considers canonically equivalent sequences the same. A common policy is to normalize ordinary authored text to NFC at a well-defined boundary. That tends to preserve conventional composed forms while ensuring combining marks have canonical ordering. The important part is consistency: normalizing only query strings or only new records creates a mixed corpus.

Compatibility normalization is more aggressive. It may be appropriate for a search key or identifier profile, but it should not automatically replace the original display value. Typography, mathematical notation, and other compatibility distinctions can carry meaning in a particular domain. Preserve the authored string when fidelity matters and derive a normalized comparison key under a documented rule.

Normalization does not perform case folding, accent removal, transliteration, spell correction, or language-aware equivalence. NFC does not make A equal a, and it does not make the German sharp s equal a two-letter spelling. It also does not prevent visually confusable characters from different scripts. Those are separate policies.

Normalize at stable boundaries rather than repeatedly in arbitrary helpers. For example, a username service can validate scalar input, apply its chosen normalization and case policy, create a canonical uniqueness key, and retain the requested display form. Every service that consumes the identifier should share that exact profile. Changing the profile is a data migration because existing keys and uniqueness decisions can change.

Compare and Search According to Purpose #

There is no context-free definition of string equality. Protocol tokens, cryptographic inputs, and source identifiers often require exact code-point or byte equality after a narrowly specified encoding. User-facing names may require normalization, case handling, and locale-aware collation. Search may intentionally ignore accents or punctuation. These operations must not share an accidental default comparator.

Binary comparison is deterministic and fast, but its ordering reflects encoded values rather than dictionary order. A database’s collation can change equality and sorting behavior, affect whether an index supports a query, and differ across operating-system or Unicode-library upgrades. Application and database rules must agree. Creating a normalized key in application code while a unique database index uses another collation can admit duplicates or reject distinct values unexpectedly.

Locale matters. Case conversion for I is the classic warning, but it is not the only tailoring. Accents, contractions, numeric substrings, punctuation, and script conventions affect ordering. JavaScript’s Intl.Collator and equivalent platform libraries expose tested collation algorithms and options. Avoid implementing human sorting with lowercasing plus <; lowercasing is a transformation, not a collation algorithm.

Search has another choice: code-point matching versus linguistic matching. A parser looking for a delimiter should use the protocol’s exact representation. A contact search may derive case-folded, normalized, and perhaps accent-insensitive tokens while still returning the original text. Record the search transformation version so an algorithm upgrade can trigger reindexing instead of silently mixing keys.

Security identifiers should use a restricted, explicit profile. Normalization alone does not stop spoofing: Latin a, Cyrillic а, and other confusables can look alike while remaining different code points. Depending on the domain, defenses include limiting scripts, flagging mixed-script labels, computing confusable skeletons, reserving protected names, and showing unambiguous forms during sensitive confirmation. Do not apply a destructive “ASCII cleanup” to arbitrary human names.

Truncate, Index, and Display Without Breaking Text #

Truncation has at least three legitimate meanings. A storage or network boundary needs a maximum encoded byte count. An algorithm may need a maximum number of code points. A UI normally needs a maximum number of grapheme clusters, lines, or rendered pixels. State which limit is being enforced.

For a UTF-8 byte budget, iterate complete scalar values, encode them, and stop before the next value exceeds the remaining bytes. If the result is user-visible, also avoid splitting a grapheme cluster: segment first, then add each complete cluster only if its encoded bytes fit. Reserve space for an ellipsis if one will be appended. Never truncate an encoded byte array and ask a replacement decoder to conceal the broken tail; the displayed result may fit but no longer represents the intended prefix.

For cursor movement, selection, backspace, and text-field limits, use a maintained grapheme segmenter. Segmentation rules evolve as Unicode adds emoji sequences. A cluster can contain many code points, so a hostile input can still make “one grapheme” expensive. Place separate byte or scalar limits at untrusted boundaries to cap memory and processing.

Display width is not grapheme count. East Asian wide characters, emoji presentation, combining marks, tabs, control characters, bidirectional layout, and font shaping all affect columns or pixels. Terminal libraries need width rules appropriate to their environment; graphical UIs should measure shaped text with the actual font and layout constraints. Padding by string.length produces misaligned tables across scripts.

Bidirectional text is another rendering dimension. Logical storage order and visual order can differ, and directional control characters can make filenames or source snippets misleading. Preserve logical text, escape or reveal controls in security-sensitive tooling, and rely on a Unicode-aware layout engine instead of reversing strings manually.

Failure Modes at System Boundaries #

Many Unicode failures are actually contract failures. An HTTP body is labeled UTF-8 but decoded as a local legacy encoding. A database column counts bytes while the API documents characters. A JSON parser accepts replacement characters, then a signature verifier signs the repaired string rather than the original bytes. A service normalizes identifiers while another compares raw values. Every component behaves consistently with its own assumption, yet the system disagrees.

Serialization formats usually define an encoding, but APIs can still contain unpaired surrogates internally. Some encoders replace them, some reject them, and some emit escape sequences that another implementation interprets differently. Validate strings at interoperability boundaries and include malformed surrogate cases in cross-language tests. For signatures, define whether the signed material is the original bytes or a canonical serialization; “sign the text” is not precise enough.

Filesystem names are sequences governed by the host filesystem and API. Different platforms can return differently normalized names, treat case differently, or allow byte sequences that a language string cannot represent losslessly. Do not assume a filename round-trips across platforms merely because it displays the same. Archive extraction and path-security checks must apply their validation to the same canonical representation used for path resolution.

Length limits can become denial-of-service or data-loss bugs. A database field measured in bytes can reject a value that passed a code-unit check. An input containing enormous combining sequences can stress segmentation or rendering despite having few graphemes. Limit raw input bytes first, decode strictly where appropriate, then enforce domain and display limits at their own units.

Test Text as a Matrix, Not a Handful of Accents #

A useful test matrix crosses representation with operation. Include ASCII; precomposed and decomposed accents; supplementary-plane symbols; combining-mark sequences; emoji with skin tones and joiners; flags; right-to-left text; mixed scripts; control characters; empty input; and malformed byte or surrogate sequences where the API can receive them. The expected result should name its unit.

Property tests can assert strong invariants: UTF-8 encoding followed by strict decoding round-trips every valid scalar string; NFC is idempotent; canonically equivalent fixtures share the same NFC result; grapheme concatenation reconstructs the original string; byte-budget truncation always yields valid UTF-8 and never exceeds its budget. Fuzz decoders with arbitrary bytes and parsers with arbitrary scalar strings, placing time and allocation bounds around the test.

Run interoperability fixtures through every participating language, database, queue, and filesystem adapter. Check exact bytes at protocol boundaries, not only rendered output. For database upgrades, test uniqueness and sort order under the new collation version before rollout. If search keys depend on Unicode data, version the transformation and rehearse a complete reindex.

Operations need visibility without leaking text. Count decoding failures, replacement events, rejected identifiers, normalization migrations, and truncations by boundary and client version. Log escaped code points or hashes for sensitive values rather than copying arbitrary user text. A spike in U+FFFD is evidence of upstream corruption, not a cosmetic font issue.

The durable model is layered: bytes are transported, encodings produce code units and scalar values, normalization reconciles defined equivalences, segmentation identifies interaction boundaries, collation applies language-sensitive comparison, and shaping produces glyphs. Once an API names the layer it operates on, Unicode stops being a collection of exceptional characters and becomes a set of explicit transformations with testable contracts.