Every list ends up with duplicates. Here's the exit.
Merge two email exports, combine keyword research from three tools, concatenate log files, paste together attendee lists from four spreadsheets — and the same entries now appear two, three, five times. Duplicates skew counts, double-send emails, waste ad budget on repeated keywords and make any “how many?” answer wrong. This tool deduplicates any line-based list in one click, keeps the first occurrence of each entry in original order, and reports exactly how many lines it removed.
How to deduplicate a list
- Paste your list — one item per line.
- Set the two switches: case-sensitive comparison, and whether to trim spaces before comparing (recommended on).
- The cleaned list renders live, with a report: “n duplicates removed, m unique lines kept.”
- Copy result.
The two switches, explained with real lists
Case sensitivity decides whether John@example.com and john@example.com are one entry or two. For emails, URLs and keywords — where case rarely distinguishes real-world identity — leave it off, and the lowercase-blind comparison merges them (keeping the first form encountered). Turn it on for code identifiers, passwords or any list where UserID and userid genuinely differ. Trim before comparing handles the invisible enemy: entries pasted from spreadsheets routinely carry trailing spaces, making apple and apple sneakily distinct. Trimming-for-comparison treats them as the duplicate they really are, while your kept lines remain exactly as pasted.
Order preservation: why it matters
Many dedupe utilities sort the list as a side effect, destroying meaningful ordering — priority rankings, chronological logs, “first come first served” signups. This tool never reorders: each unique line stays where it first appeared. (Need sorting too? That's a deliberate second step with our Sort Lines tool, which also offers its own dedupe switch for the sort-and-clean combo.)
Classic clean-up runs
- Email lists: case-insensitive + trim → every address once, first spelling kept.
- Keyword sets: merge exports from all your research tools, dedupe, then sort.
- URL collections: bookmarks, sitemap inputs, crawl lists — mind that http/https variants are honestly different lines; dedupe won't merge those (nor should it silently).
- Log triage: dedupe repeated error lines to see the distinct failure modes.
Paste thousands of lines, read one honest number — “312 duplicates removed” — and move on with a list you can finally trust.
Quick reference
| Property | Detail |
|---|---|
| Input | Any line-based list (emails, URLs, keywords…) |
| Kept | First occurrence of each entry |
| Order | Always preserved — never re-sorted |
| Case switch | Sensitive or insensitive comparison |
| Trim switch | Ignore leading/trailing spaces when comparing |
| Report | Duplicates removed + unique lines kept |
| Processing | 100% in-browser, no upload |
Frequently asked questions
Which occurrence of a duplicate is kept?
The first one, in its original position and exact original spelling. Everything after it that matches (under your case/trim settings) is removed. Order is never shuffled.
Should comparison be case-sensitive for email addresses?
No — email addresses are effectively case-insensitive in practice, so leave the switch off and John@ and john@ merge into one entry. Reserve case sensitivity for code, tokens and IDs.
Why does my visually-identical line not get removed?
Almost always invisible whitespace — a trailing space or tab from a spreadsheet. Enable “trim before comparing” and the phantom difference disappears. (Different unicode characters that merely look alike are rarer but possible.)
How large a list can I clean?
Tens of thousands of lines process instantly — comparison happens in memory in your browser. Paste entire exports without hesitation; nothing is uploaded anywhere.