Build a synthetic JSON Lines fixture in memory

By Batu · English technical notes

Also published in our primary archive.

To build a synthetic JSON Lines fixture in memory, serialize each dictionary separately and append a newline after each serialized value. Do not repeatedly call json.dump() into one stream without a delimiter: the resulting adjacent objects are not one JSON document. The Python documentation explicitly describes JSON as unframed and warns that repeated dump() calls to the same stream create invalid JSON. StringIO is appropriate here because json writes text, not bytes. Python JSON documentation

This example first creates the bad text {"id": 1}{"id": 2}{"id": 3} and confirms that json.loads() rejects it. It then constructs the corrected fixture in a StringIO: each line is a complete JSON object. Its printed representation makes the three newline boundaries visible, while the last line shows the ordinary Python list produced by decoding every line.

The assertions check this fixture’s narrow contract: decoding all lines returns the original three dictionaries and the text has exactly three non-empty lines. They do not establish a universal JSON Lines specification, validate object fields, or limit resource use for untrusted input. json.JSONDecodeError, used for the failure branch, is available from Python 3.5; the example otherwise uses longstanding Python 3 standard-library APIs.

AI assistance disclosure: this synthetic example was drafted with AI assistance and should be adapted to the surrounding data contract.

Example

import json
from io import StringIO

records = [{"id": 1}, {"id": 2}, {"id": 3}]

broken_io = StringIO()
for record in records:
    json.dump(record, broken_io)

try:
    json.loads(broken_io.getvalue())
except json.JSONDecodeError:
    print("adjacent objects: invalid JSON")

fixture = StringIO("".join(json.dumps(record) + "\n" for record in records))
text = fixture.getvalue()
decoded = [json.loads(line) for line in fixture]

assert decoded == records
assert len(text.splitlines()) == 3
print(repr(text))
print(decoded)

Expected output:

adjacent objects: invalid JSON
'{"id": 1}\n{"id": 2}\n{"id": 3}\n'
[{'id': 1}, {'id': 2}, {'id': 3}]