CSV Deduplicator

Exact match only catches the easy half

Removing rows that are byte for byte identical is the simple case, and it is usually the smaller one. The duplicates that survive are the ones where the email is the same but the casing differs, where the company is Acme Ltd in one row and ACME Ltd. in another, or where a trailing space slipped in during an export.

The key column is the real decision

Matching on every column finds identical rows. Matching on the email finds the same person recorded twice with different job titles. Matching on the domain collapses a company down to one row. Which one is correct depends on what the list is for, which is why it is a choice rather than a default.

Normalisation before comparison

Casing, surrounding whitespace and punctuation are noise in almost every identity field. Ignoring them before comparing turns three near misses into one match. Punctuation is left off by default because it matters in some fields, such as product codes, and can be switched on when it does not.

Keep the removed rows

Both halves are exportable. The clean list is what you use, and the removed rows are what you check when a number looks wrong. Deduplication that throws data away without showing you what it took is impossible to audit.