Handle blank components in a CSV duplicate key

By Batu · English technical notes

Also published in our Blogger archive.

Quick answer

CSV cells arrive as strings, so a duplicate policy must define how blank components participate in a key.

Example

The code strips components for its completeness decision, increments skipped for an incomplete tuple, and only inserts complete tuples into seen. That prevents two broken rows from being presented as a meaningful duplicate group.

import csv, io
rows = list(csv.DictReader(io.StringIO('region,id\nEU,7\n,7\nEU,7\nEU,\n')))
seen = set()
skipped = 0
duplicate = []
for row in rows:
    key = (row['region'].strip(), row['id'].strip())
    if not all(key):
        skipped += 1
        continue
    if key in seen:
        duplicate.append(key)
    seen.add(key)
assert (skipped, duplicate) == (2, [('EU', '7')])
print(f'skipped={skipped} duplicates={duplicate}')

Expected stdout:

skipped=2 duplicates=[('EU', '7')]

Reading the result

Some imports may treat blank as a valid key component. If so, remove this skip rule and document that decision; do not inherit it accidentally from a convenience helper.

Report skipped row numbers alongside the count in a real tool. The count explains scope, while individual record references let the producer correct the incomplete key values.

A complete-key duplicate report should state that incomplete rows were excluded. Otherwise a count of duplicates could imply that every source record participated in the comparison.

Sources

- Python csv module documentation

Prepared with AI assistance. The example uses synthetic data; its stated limits apply.