Report duplicate rows without exposing the whole row

By Batu · English technical notes

Also published in our Blogger archive.

Quick answer

DictReader can select the one field needed for duplicate evidence without copying unrelated values into a report.

Example

The fixture counts emails, then emits just the selected key and count. The secret column is read as part of the fixture but is never serialized. That keeps the report focused on the finding.

import csv, io, json
rows = list(csv.DictReader(io.StringIO('email,secret\na@example.test,x\na@example.test,y\n')))
counts = {}
for r in rows:
    counts[r['email']] = counts.get(r['email'], 0) + 1
report = [{'key': {'email': k}, 'count': v} for k, v in counts.items() if v > 1]
assert report == [{'key': {'email': 'a@example.test'}, 'count': 2}]
print(json.dumps(report, separators=(',', ':')))

Expected stdout:

[{"key":{"email":"a@example.test"},"count":2}]

Reading the result

Selected keys can still be sensitive. Decide who may read the report and whether a keyed digest or a local-only report is more appropriate for the actual dataset.

Counts alone are enough for many reviewers, but row references can be added when remediation needs them. Keep every added field justified by the investigation rather than dumping the original record.

The count is accumulated before report construction, so duplicate evidence is deterministic for the fixture. A larger tool should choose a stable ordering for multiple keys.

Sources

- Python csv module documentation

Prepared with AI assistance. The example uses synthetic data; its stated limits apply.