Validate a UTF-8 CSV with a byte-order mark

By Batu · English technical notes

Also published in our Blogger archive.

Quick answer

The csv reader consumes decoded text; decoding policy determines whether a UTF-8 byte-order mark becomes part of the first header.

Example

The same bytes are decoded with utf-8 and utf-8-sig. Plain UTF-8 leaves U+FEFF in id, while the signature-aware codec produces the intended label. The script makes that difference visible with repr.

import csv, io
raw = b'\xef\xbb\xbfid,name\n1,A\n'
plain = next(csv.reader(io.StringIO(raw.decode('utf-8'))))[0]
sig = next(csv.reader(io.StringIO(raw.decode('utf-8-sig'))))[0]
assert plain == '\ufeffid' and sig == 'id'
print(repr(plain), repr(sig))

Expected stdout:

'\ufeffid' 'id'

Reading the result

This does not detect every encoding. A file without a BOM still needs an agreed encoding, and accepting utf-8-sig should be an explicit intake rule rather than a hidden repair.

A useful report can include the chosen decoding label and whether a signature was observed. It should not claim that the bytes were rewritten; decoding is a view, not a mutation.

Check the resulting first header before applying a mapping. A BOM mismatch is especially easy to miss because ordinary visual display may not reveal the leading character.

Sources

- Python csv module documentation

Prepared with AI assistance. The example uses synthetic data; its stated limits apply.