Detect JSON key collisions after Unicode normalization

By Batu · English technical notes

Also published in our primary archive.

Detect JSON key collisions after Unicode normalization

To detect JSON key collisions after Unicode normalization, normalize each name inside object_pairs_hook before checking uniqueness. This preserves pair order and lets the diagnostic retain both source spellings instead of discovering the problem only after ordinary dictionary decoding.

The JSON fixture uses two escaped names: \u00e9 decodes to the composed character é, while e\u0301 decodes to an e followed by a combining acute accent. The default decoded dictionary contains two distinct Python strings, so the first assertion and output show that raw equality is False. An exact duplicate-key policy would accept that pair.

The hook applies unicodedata.normalize('NFC', raw_key) to each name before using it as the policy key. It prints a small trace of the raw and normalized forms. The first key is stored with its original spelling; the second normalizes to the same NFC string and raises ValueError. The exception reports the prior raw form, the current raw form, and their shared normalized key. This is a key-name policy only: it does not decide whether NFC is appropriate for every identifier system, nor does it normalize values.

object_pairs_hook was added in Python 3.1. The json documentation specifies that it receives ordered object pairs before a dictionary is made, and the unicodedata documentation describes normalization forms including NFC. No newer API is used.

AI assistance disclosure: This article was drafted with AI assistance and verified with the synthetic fixture shown.

Example

import json
import unicodedata

source = r'{"\u00e9":"composed","e\u0301":"decomposed"}'

raw_object = json.loads(source)
raw_keys = list(raw_object)
assert raw_keys[0] != raw_keys[1]
print("raw keys equal:", raw_keys[0] == raw_keys[1])


def reject_nfc_collisions(pairs):
    result = {}
    for raw_key, value in pairs:
        normalized = unicodedata.normalize("NFC", raw_key)
        print("hook: raw={!r}, nfc={!r}".format(raw_key, normalized))
        if normalized in result:
            first_raw = result[normalized][0]
            raise ValueError(
                "NFC collision: {!r} and {!r} -> {!r}".format(
                    first_raw, raw_key, normalized
                )
            )
        result[normalized] = (raw_key, value)
    return {key: item[1] for key, item in result.items()}


try:
    json.loads(source, object_pairs_hook=reject_nfc_collisions)
except ValueError as error:
    print("validated:", error)

Expected output:

raw keys equal: False
hook: raw='é', nfc='é'
hook: raw='é', nfc='é'
validated: NFC collision: 'é' and 'é' -> 'é'