Remove Duplicate Lines

1. 01Paste your listOne item per line. Length is not a problem.
2. 02Choose how lines are comparedTrim spaces and ignore capitalisation both widen what counts as a duplicate. Turn them on when "Hello" and "hello " should be treated as the same entry.
3. 03Read the counts, then copyBefore, after and removed are shown, so you know how much overlap there actually was.

• Merging two exportsTwo overlapping lists concatenated into one leaves duplicates wherever the exports overlapped. This collapses them and reports how many.
• Cleaning an email or ID listSending twice to the same address is the visible cost of a duplicate. Ignoring capitalisation matters here, since addresses are often stored inconsistently.
• Finding out how much overlap existsThe removed count answers "were these two lists nearly the same, or barely related?" without opening a spreadsheet.

• A short list with repeatsHello, World, Hello, Tooleem, Hello has five lines and three unique ones. The result keeps the first occurrence of each, in original order, and reports two removed.
• Duplicates that do not look identical"hello" and "Hello " differ by a capital and a trailing space, so by default they survive as two lines. Turn on trim spaces and ignore capitalisation and they collapse into one.

• Order is preserved unless you sortThe first occurrence of each line stays where it was. Sorting alphabetically is a separate switch, so you can dedupe without rearranging anything.
• Trim before comparingTrailing whitespace is invisible and is the most common reason two lines that look identical are not treated as duplicates.

By default two lines are duplicates only if they are byte-for-byte identical, which is strict on purpose: it will never merge two entries that were genuinely different. The two switches loosen it in the ways that matter in practice. Trimming spaces removes leading and trailing whitespace before the comparison, which catches the very common case of a trailing space that nobody can see. Ignoring capitalisation makes "Hello" and "hello" the same entry, which is usually right for email addresses and usernames and usually wrong for things like product codes where case is meaningful. Both affect only the comparison — the line that survives keeps its original form, so you are not silently lower-casing your data.

Knowing that eight lines were removed from a list of five hundred is a different situation from eight removed out of twelve, and the difference tells you something about the data you are working with. A very high removal count usually means the same file got concatenated with itself; a very low one on two supposedly different lists usually means they were not as different as expected. Because the tool reports before, after and removed rather than just handing back a shorter list, you can notice both cases without counting rows yourself.

Sorting is offered as its own option because deduplicating and reordering are different jobs and you often want only the first. When you do sort, comparison follows the reading locale rather than raw character codes, so accented and non-Latin text orders the way a reader of that language would expect rather than by byte value. If you want sorting with more control — numeric ordering, direction, case sensitivity — the dedicated sort tool covers it.

Which duplicate is kept?

The first occurrence, in its original position and original form. Later copies are removed.

Does it change the order of my list?

No, unless you turn on sorting. Without it the surviving lines stay exactly where they were.

Why are two identical-looking lines not being merged?

Almost always trailing whitespace or a difference in capitalisation. Turn on trim spaces and ignore capitalisation and they will collapse.

Is my list uploaded?

No. Everything runs in your browser.