Unicode Normalization Length: NFC, NFD and NFKC Counts

Unicode normalization can change both a text’s code-point count and its UTF-8 byte count. Measure the value after the transformations required by the receiving system. If the rule covers the original input, keep and measure that original value instead.

Choose the normalization form before counting

Unicode Normalization Forms defines NFD as canonical decomposition and NFC as canonical decomposition followed by composition. For é (U+00E9), NFD produces e plus U+0301: two code points and three UTF-8 bytes. NFC produces the precomposed letter: one code point and two bytes.

NFKC adds compatibility mappings before composition. Those mappings can remove distinctions that matter in the text. It is not a universal way to shorten a string or preserve its original spelling. Choose it because the application’s contract calls for compatibility normalization, rather than because a count is too high.

Compare six exact inputs

Each cell below is code points / UTF-8 bytes after the named form. The first column lists the original input’s exact code points, so visually similar accent spellings stay distinguishable. Counts exclude quotation marks, a BOM and file or protocol overhead.

U+FF21 is fullwidth Ａ; its NFKC result is ASCII A, still one code point but fewer bytes. U+FB03 is the ffi ligature; NFKC produces ffi, increasing the code-point count from one to three while keeping three UTF-8 bytes. These two cases show why a smaller byte count and a smaller text count are separate outcomes.

NFC need not compose every base-and-accent pair. The final row, q followed by U+0301, remains two code points. Normalization also does not turn a code-point total into a grapheme count. The String Length guide explains that separate choice of unit.

Measure the normalized value in Python

unicodedata.normalize returns the requested normalized string. In this in-memory example, \u0301 in the Python literal denotes the actual combining acute accent. Each result is measured separately:

import unicodedata

text = "e\u0301"
for form in ("NFC", "NFD", "NFKC"):
    value = unicodedata.normalize(form, text)
    print(form, len(value), len(value.encode("utf-8")))

The output is NFC 1 2, NFD 2 3 and NFKC 1 2. In Python’s text and bytes types, len(value) counts code points and len(value.encode("utf-8")) counts the encoded bytes. Python 3.12 uses Unicode 15.0 data; unicodedata.unidata_version reports the database version. These examples use that data, rather than assuming every runtime implements the latest Unicode version.

Recheck after joining normalized pieces

Two individually normalized strings can require normalization again after concatenation. For NFC, A and a standalone U+030A combining ring are each normalized. Joined together, they contain two code points and three UTF-8 bytes; normalizing the joined value produces Å, one code point and two bytes.

If a field’s contract requires NFC, normalize the assembled value and then apply its counting rule. Preserve the original separately when exact spelling or bytes matter. Compatibility normalization is not reversible text preservation, and normalization is not a general ASCII transliteration.

Re-encoding a normalized value measures its new text bytes. That count describes the transformed value; it does not establish the original file’s byte length or a service’s acceptance limit. Use the UTF-8 file-size guide when the constraint applies to the original file.