Treat JSON iterencode chunks as fragments of one document

By Batu · English technical notes

Also published in our primary archive.

Treat JSON iterencode chunks as fragments of one document

JSONEncoder.iterencode() supports incremental output, but its yielded strings are fragments of one JSON document, not records to pass independently to json.loads(). For {'items': [1, 2]}, this run exposes the fragments. Punctuation and partial array fragments cannot be decoded alone; the key fragment happens to be a valid JSON string, but it is still not the original value. That contrast is why “try to load every chunk” is the wrong boundary rule.

The correction is to retain the fragments for this one encoder call, join them once, and decode the resulting complete text. The assertions verify that the joined text is {"items": [1, 2]} and that decoding it recreates the original in-memory dictionary. They do not promise that these particular chunk boundaries are a stable protocol: code consuming iterencode() should treat every yield as an opaque string fragment and only attach meaning to the full concatenation.

Python documents iterencode() as yielding each string representation as available, and its own example shows a value split across multiple pieces. See JSONEncoder.iterencode. No newer API is used; this example is compatible with Python 3. AI assistance disclosure: this article was drafted with AI assistance and uses only a synthetic in-memory value.

import json

value = {"items": [1, 2]}
fragments = list(json.JSONEncoder().iterencode(value))

outcomes = []
for fragment in fragments:
    try:
        decoded = json.loads(fragment)
    except json.JSONDecodeError:
        outcomes.append("not a complete document")
    else:
        outcomes.append("wrong standalone value " + repr(decoded))

whole_document = "".join(fragments)
round_trip = json.loads(whole_document)

assert whole_document == '{"items": [1, 2]}'
assert round_trip == value
assert outcomes[1] == "wrong standalone value 'items'"

for index, fragment in enumerate(fragments):
    print(str(index) + ": " + repr(fragment) + " -> " + outcomes[index])
print("joined:", whole_document)
print("round trip:", round_trip)

0: '{' -> not a complete document
1: '"items"' -> wrong standalone value 'items'
2: ': ' -> not a complete document
3: '[1' -> not a complete document
4: ', 2' -> not a complete document
5: ']' -> not a complete document
6: '}' -> not a complete document
joined: {"items": [1, 2]}
round trip: {'items': [1, 2]}