Strip a UTF-8 byte-order mark from a fixture
Also published in our Blogger archive.
A UTF-8 byte-order mark is sometimes present at the beginning of exported fixtures. If ordinary utf-8 decoding reads those leading bytes, the resulting text begins with U+FEFF. For a fixture format that permits a UTF-8 signature but expects ordinary text content, read it with the utf-8-sig codec instead. On decoding, that codec skips the three BOM bytes only when they occur at the start of the file.
The example creates an isolated temporary directory and writes known bytes: the UTF-8 BOM followed by a UTF-8 encoded title. Path.read_text(encoding="utf-8-sig") returns the title and newline without a leading U+FEFF. The assertions check both the exact decoded content and the absence of that leading character. The program then prints the fixture text exactly.
Do not apply this choice blindly to arbitrary data. A leading U+FEFF can be intentional content, and utf-8-sig does not detect or fix a different encoding or malformed UTF-8; normal decoding errors still need a deliberate policy. Python’s codecs documentation also notes that a BOM is generally discouraged for UTF-8. pathlib and TemporaryDirectory are available in Python 3.4+; no newer API is required here.
References: Python codecs documentation for utf-8-sig and Python temporary-directory documentation.
AI-assistance disclosure: This article was drafted with AI assistance and should be reviewed in its application context.
from pathlib import Path
from tempfile import TemporaryDirectory
with TemporaryDirectory() as directory:
path = Path(directory) / "fixture.txt"
path.write_bytes(b"\xef\xbb\xbftitle=R\xc3\xa9sum\xc3\xa9\n")
text = path.read_text(encoding="utf-8-sig")
assert text == "title=Résumé\n"
assert not text.startswith("\ufeff")
print(text, end="")title=Résumé