LangStop

Remove Duplicate Lines

Remove Duplicate Lines — Deduplicate a List Online

Paste a list and click REMOVE DUPES to keep each line once. The first occurrence of every line is kept and the original order is preserved, so the result is your list minus the repeats. Everything runs in your browser; nothing is uploaded.

What each button does

Button Output
REMOVE DUPES Every distinct line once, in the order it first appeared
KEEP DUPES Only the lines that appear more than once — each listed once
COUNT DUPES How many distinct lines repeat, with each line's count
SHOW DUPES The repeated lines, without counts 
apple
banana
apple
cherry
banana

REMOVE DUPES gives:

apple
banana
cherry

and KEEP DUPES gives apple and banana.

How "duplicate" is decided

Lines are compared exactly:

  • Apple and apple are different. Lowercase the list first with the String Case Converter when case should not matter — email addresses and domain names, for example.
  • apple and apple (with a trailing space) are different. Text copied from spreadsheets and PDFs often hides trailing spaces or tabs.
  • Several blank lines collapse into one blank line.

Remove duplicates, then sort?

Order is preserved on purpose — log lines and ranked lists keep their sequence. For an alphabetical, de-duplicated list, run REMOVE DUPES and then SORT A-Z on the result.

Common uses

  • Mailing lists — one entry per address before an import or a send.
  • Keyword and tag lists — merge lists from several sources without repeats.
  • IDs and SKUs — find out whether an export that should be unique really is (compare COUNT LINES before and after).
  • Log excerpts — collapse repeated messages to see the distinct errors.