THE DATA DESK · 02
Remove duplicate rows from a CSV file
Exports from CRMs, forms and shops often repeat a record. Commafield removes the later copies, keeps the first, and shows each removed row next to the line it repeats, so you can see exactly what will disappear before you download.
Steps
- Open the CSV cleaner and choose Open CSV…, paste the text, or drop the file on the panel. Comma, semicolon, tab and pipe files are detected.
- Remove duplicate rows is on by default and compares the entire row. To find repeated customers or products, set Match duplicates by to one column, such as
emailorsku. - Tick Ignore upper/lower case if
ANA@example.comandana@example.comshould count as the same. - Check the table. Removed rows are struck through and labelled Removed · repeats line N. Use Only rows with changes to see just those.
- Select Download cleaned CSV. The file contains every kept row, not only the 400 shown in the preview.
Worked example: one file, four settings
Download duplicates.csv. Line 1 is the header; line 5 has a space before the email address.
id,name,email,plan 101,Rae Okafor,rae@example.com,Pro 102,Jin Park,jin@example.com,Free 102,Jin Park,jin@example.com,Free 103,Ana Silva, ana@example.com,Team 104,Ana Silva,ANA@example.com,Team 105,Leo Brandt,,Pro 106,Leo B.,,Free
| Match by | Removed | Rows out | What happens |
|---|---|---|---|
| Entire row | 1 | 6 | Line 4 repeats line 3. Lines 5 and 6 differ in id, so both stay. |
| 1 | 6 | Line 4 again. ana@ and ANA@ differ in case. The two rows with an empty email are kept. | |
| email + ignore case | 2 | 5 | Line 6 now repeats line 5: trimming removed the leading space, and case is ignored. |
| email + ignore case, trimming off | 1 | 6 | " ana@example.com" keeps its space, so it no longer matches line 6. |
Trimming runs before duplicates are compared. If a match you expect is missing, look for spaces at the start or end of the cell; the marked view shows removed spaces as ·.
Matching by name also removes 2 rows here, but it would merge two different people who share a name. Prefer a column that identifies a record: an id, an email address or a SKU.
How rows are matched
- Entire row compares every column of the cleaned row. Two rows are duplicates only if all cells are equal after the clean-up steps that are on.
- One column compares only that column. The first row with each value is kept, whatever the other columns contain.
- Empty values in the chosen column never count as duplicates of each other.
- Case matters unless you tick Ignore upper/lower case. The kept row keeps its original case.
- Empty and whitespace-only rows are handled by Remove empty rows, not by duplicate removal.
- Quoted cells are read as one value, so
"Portland, OR"is one city, and a line break inside quotes stays in its cell.
Values are compared as text: 1.0 and 1 are different, as are 2026-10-01 and 10/1/2026. Normalise such columns first if they should match.
Compared with Remove Duplicates in Excel
Excel’s Data → Remove Duplicates also keeps the first occurrence and lets you pick columns. The differences that tend to matter:
| Commafield | Excel | |
|---|---|---|
| Review before removing | Each removed row is shown with the line it repeats | Shows a count; removed rows are gone |
| Case | Case-sensitive unless you tick Ignore case | Not case-sensitive |
| Leading and trailing spaces | Trimmed first when Trim extra spaces is on | Not trimmed, so they prevent a match |
| Opening the file | Values stay text; leading zeros such as 001 are kept | May convert values, for example dropping leading zeros, unless you import the columns as text |
The Excel behaviour is described from its documented defaults; your version and import settings may differ.
Questions
Which copy of a duplicate is kept?
The first row in the file is kept. Every later row with the same value is removed and marked with the line number it repeats, so you can check it before downloading.
Is the header row ever removed?
No. The first non-empty row is read as the header and is never compared with the data rows.
Are rows with an empty key treated as duplicates?
No. When you match by one column, rows where that column is empty are all kept, and a note says how many there were.
Can I remove duplicates without trimming spaces?
Yes. Untick Trim extra spaces. Values are then compared exactly as they are in the file, so a cell with a leading space differs from the same text without it.
Does it merge duplicate records?
No. It keeps the first row and drops the others. If two rows share an id but differ in other columns, the later values are not merged into the first row, so review those rows before you rely on the result.
Related
Files are processed in this browser tab and are not uploaded. Only your option choices are remembered. Read the privacy notice.