Remove Duplicate Lines

Remove duplicate lines from a list or text while keeping the original order. Optionally ignore case and extra spaces, and drop empty lines.

How to use the remove Duplicate Lines

  1. Type or paste into the Text (one item per line) box.
  2. Tick Ignore upper/lower case if you want that option; leave it clear otherwise.
  3. Tick Ignore leading and trailing spaces if you want that option; leave it clear otherwise.
  4. Tick Remove empty lines if you want that option; leave it clear otherwise.
  5. Tick List the removed duplicates if you want that option; leave it clear otherwise.
  6. The result updates instantly as you type. There is no button to press.
  7. Use Copy result to copy the figures, or Copy link to share a link that reopens the remove Duplicate Lines with the same inputs.

How it works

Lists collected from spreadsheets, sign-up forms, log files, keyword research or several merged documents often contain repeated entries. This tool removes them, keeping the first occurrence of each line in its original position, so the order of your list is preserved. If you also need the list sorted, run the result through the sort lines tool.

Real-world duplicates are rarely identical character for character. "Apple" and "apple", or "banana" and "banana " with a trailing space, look the same to a person but not to a computer. By default, this tool ignores case and trims spaces at the start and end of each line when comparing, so near-duplicates like these are caught. The version that is kept is the first one that appears. Untick the options for an exact, case-sensitive comparison, which matters for things like passwords, codes or case-sensitive file names.

Empty lines are removed by default too. Tick List the removed duplicates to see exactly what was taken out, which is a useful check before you delete anything from a real data set. The tool processes tens of thousands of lines instantly, and nothing you paste leaves your browser.

Formula

for each line, in order: key = line (trimmed and lowercased if those options are on) if key has been seen before → remove the line otherwise → keep it and remember the key

This takes one pass through the list using a hash set, so it stays fast even for very long lists.

Example

The list apple, Banana, cherry, apple, banana␣, cherry, (blank), date has 8 lines. With the default options, the second "apple", "banana " (which matches "Banana" ignoring case and the trailing space) and the second "cherry" are removed, as is the blank line. The result is apple, Banana, cherry, date: 4 unique lines, 3 duplicates removed and 1 empty line removed.

Frequently asked questions

Does it keep the original order?

Yes. The first occurrence of each line stays exactly where it was. Only later repeats are removed.

Can I remove duplicates that differ only in capitalisation?

Yes. That's the default: 'Ignore upper/lower case' treats 'Paris' and 'paris' as the same line, and keeps whichever appeared first.

Is there a limit on the number of lines?

There is no fixed limit. Lists of 100,000 lines or more process in a moment in modern browsers.