Constrain delimiter detection instead of trusting a heuristic

By Batu · English technical notes

Also published in our Blogger archive.

Quick answer

Sniffer infers a dialect from a sample, but its result is only a heuristic.

Example

The script limits the heuristic to comma and semicolon, then parses the record with the declared semicolon delimiter. The comma inside one,two stays text because the explicit contract takes precedence over inference.

import csv
sample = 'id;note\n1;one,two\n'
detected = csv.Sniffer().sniff(sample, delimiters=',;').delimiter
assert detected == ';'
assert csv.reader(['1;one,two'], delimiter=';').__next__() == ['1', 'one,two']
print(f'detected={detected} contract=;')

Expected stdout:

detected=; contract=;

Reading the result

Use sniffing to flag a possible format change, not to silently alter a known integration. A too-small or unrepresentative sample can select a delimiter that later records contradict.

If the contract is comma-delimited, pass comma directly to reader and report semicolon as a mismatch. Candidate restriction only makes an exploratory check less unconstrained; it does not validate data.

The sample includes the competing comma inside field content specifically to show why a delimiter cannot be chosen by counting punctuation without a format decision.

Sources

- Python csv module documentation

Prepared with AI assistance. The example uses synthetic data; its stated limits apply.