Normalizes an origin document's text for encoding, never for content.
Does: folds every line-break form to LF; strips the byte-order mark,
control characters other than LF and tab, bidirectional controls,
zero-width characters, tag characters, and stray variation selectors —
while preserving emoji ZWJ, keycap, CJK, Mongolian, and emoji-tag
sequences; composes to Unicode NFC; trims leading and trailing whitespace.
Does not: collapse internal whitespace, reflow paragraphs, fold smart
quotes or dashes, case-fold, or strip punctuation, diacritics, emoji, or
non-ASCII characters. The document is meant to be the original.
Idempotent — normalizeOriginText(normalizeOriginText(t)) equals
normalizeOriginText(t) — which is what makes it safe to apply both at an
application's import boundary and again on document creation.
Two things make it idempotent, and both are load-bearing.
The step order. Line breaks are folded first because a lone carriage return
is itself a control character, so stripping first would delete the break
rather than convert it. Stripping precedes NFC because removing an invisible
character can leave a base letter adjacent to a combining mark it was
previously separated from; composing first would leave that pair for a second
application to compose. NFC never emits a control, an invisible, or a line
break, so the reverse hazard does not exist.
And the rule that a removal candidate never legitimizes another removal
candidate — see isLegitimateInContext. Without it a character survives on
the strength of a neighbour deleted in the same pass, and the next pass
deletes it too.
Normalizes an origin document's text for encoding, never for content.
Does: folds every line-break form to LF; strips the byte-order mark, control characters other than LF and tab, bidirectional controls, zero-width characters, tag characters, and stray variation selectors — while preserving emoji ZWJ, keycap, CJK, Mongolian, and emoji-tag sequences; composes to Unicode NFC; trims leading and trailing whitespace.
Does not: collapse internal whitespace, reflow paragraphs, fold smart quotes or dashes, case-fold, or strip punctuation, diacritics, emoji, or non-ASCII characters. The document is meant to be the original.
Idempotent —
normalizeOriginText(normalizeOriginText(t))equalsnormalizeOriginText(t)— which is what makes it safe to apply both at an application's import boundary and again on document creation.Two things make it idempotent, and both are load-bearing.
The step order. Line breaks are folded first because a lone carriage return is itself a control character, so stripping first would delete the break rather than convert it. Stripping precedes NFC because removing an invisible character can leave a base letter adjacent to a combining mark it was previously separated from; composing first would leave that pair for a second application to compose. NFC never emits a control, an invisible, or a line break, so the reverse hazard does not exist.
And the rule that a removal candidate never legitimizes another removal candidate — see
isLegitimateInContext. Without it a character survives on the strength of a neighbour deleted in the same pass, and the next pass deletes it too.