Sort, deduplicate and tidy the lines of a text file
Sorts, deduplicates, trims and reverses the lines of a text file in your browser. The six steps run in a fixed order — trim, drop blanks, deduplicate, order, reverse — and the file is never sent anywhere.
Any UTF-8 text file — a list, a log, an export. It is read in this tab and never sent anywhere.
How it works
What this does
Takes a text file — a list of email addresses, a word list, a log, a column pasted out of a spreadsheet — and puts its lines in order, removes the repeats, strips the stray spaces and drops the empty lines. The four jobs travel together because they are almost never wanted apart: a list that needs deduplicating nearly always needs trimming first, or the duplicates do not match.
The reason to do it here rather than in a text editor is that the file does not leave the machine. A list of customer email addresses pasted into an online “remove duplicate lines” box has been sent to somebody else’s server, and that is a disclosable event under most data protection regimes. This one reads the file in the tab you are already looking at.
The method
The steps run in a fixed order, and the order is the whole specification:
Trim, then drop blanks, then deduplicate, then sort, then reverse.
Trimming has to come before deduplicating. milk and milk are the same
entry to the person who typed them and two different strings to a computer, so
a deduplicator that runs first leaves both behind and looks broken. Dropping
blanks before deduplicating is the same argument: otherwise every empty line in
the file collapses into one empty line at the top of the output, which is not
what “remove blank lines” means to anybody.
Reversing comes last, deliberately, so it flips whatever order the previous step produced. That makes Z to A plus reverse the same thing as A to Z, which sounds like a redundancy and is actually the check that the two steps have not been quietly merged.
The ordering is Intl.Collator, which is the Unicode Collation Algorithm the
browser already carries, and not the < operator. Comparing strings with <
compares UTF-16 code units, and that produces two visible failures every time:
every capital letter sorts before every lower-case one, so Zebra lands before
apple; and every accented letter sorts after z, so Ångström lands after
Zulu rather than beside Anchor. The collator is pinned to English rules
rather than following the language of the page, because the same file and the
same settings have to produce the same file — in Swedish collation, Å is a
separate letter that comes after Z, and a download that changes with the
interface language is a bug that is very hard to report.
Sorting is done in blocks of twenty thousand lines that are then merged in pairs, rather than in one call. A single sort of a million lines is one uninterruptible block of work: no progress bar, and a Cancel button that does nothing until it finishes. Merging blocks gives the page a turn between each one. The merge takes the left-hand line whenever two compare equal, which is what keeps the result identical to a single stable sort — and that identity is asserted against fifty thousand pseudo-random lines in the test file rather than assumed.
Before you read on
Four lines — Anchor, apple, Ångström and Zebra — sorted A to Z. Where does Ångström land?
Second, between Anchor and apple — because the ordering here is Intl.Collator, the Unicode Collation Algorithm the browser already carries, rather than the < operator. Comparing with < produces two visible failures every time on this exact list: it sorts capitals before lower-case, so Zebra lands before apple; and it sorts accented letters after z, so Ångström ends up last. The collator is pinned to English rules rather than the language of the page, because the same file and the same settings have to produce the same file — in Swedish collation Å really is a separate letter after Z, and a download that changed with the interface language would be a bug nobody could report.
A worked example
A shopping list, saved as shopping-list.txt, with the defaults left alone —
trim on, remove blank lines on, remove duplicates on, match case off, A to Z:
Bread
milk
Milk
bread
Eggs
milk
Seven lines in. Trimming turns Bread into Bread. The empty line goes,
which is one blank removed. Milk, bread and the second milk are all
repeats once case is ignored, which is three duplicates removed. Three lines
survive, and they come out in English alphabetical order:
Bread
Eggs
milk
| Reading | Value |
|---|---|
| Lines in | 7 |
| Lines out | 3 |
| Duplicates removed | 3 |
| Blank lines removed | 1 |
The download is called shopping-list-lines.txt — the source name with a
suffix, rather than the source name exactly, because two files called
shopping-list.txt in one downloads folder is how the browser ends up silently
naming one of them shopping-list (1).txt.
These are the same numbers asserted in this tool’s test file, so if the formula ever changes without this page changing with it, the build fails.
What it does not do
It does not sort by anything except the whole line. There is no “sort by the
third column”, no “sort by length”, and no key extraction — for that, the
answer is genuinely sort -k. It does not merge two files, compare two files,
or find the lines that appear in one and not the other. It does not understand
CSV quoting, so a comma-separated file whose cells contain line breaks inside
quotes will be split on those line breaks like any other text; convert it
first if that matters. It does not preserve the original line endings or a
byte-order mark, and it will not process a file saved as UTF-16 or a file that
is not text at all — both are refused with a message rather than mangled
quietly.
It does not understand subtitle cue blocks either. Sorting those lines would separate dialogue from its timestamps; shift subtitle timing with a parser that changes the recognised cue times and leaves their text in place.
It treats every line as a line. Where the lines are really rows with columns inside them, CSV to JSON parses the quoting and the embedded commas properly instead of splitting on the first one it sees.
When several cleaned lists need to travel together, create a ZIP file from the finished downloads. Storage keeps each text file’s exact bytes, while DEFLATE usually shrinks repeated text and still gives every list its own safe filename.
How it is done
- Read the chosen file in this tab, refuse it if it is UTF-16 or binary, and decode the rest as UTF-8, discarding any byte-order mark at the front.
- Split the text into lines on a carriage return, a line feed, or the two together, so a Windows, Unix or classic Mac file all give the same lines. A final newline is treated as the terminator of the last line, not as an empty line after it.
- Trim leading and trailing whitespace from each line, if that is switched on. This runs first, because two lines that differ only in trailing spaces are the same line to the person reading them.
- Remove lines holding nothing but whitespace, if that is switched on. A line of three spaces counts as blank whether or not trimming is on.
- Remove repeated lines, keeping the first appearance. Matching ignores case unless "Match case" is on, in which case the comparison is exact.
- Order the lines A to Z or Z to A with the Unicode Collation Algorithm through Intl.Collator, pinned to English rules, or leave the file's own order alone.
- Reverse the whole list last, if that is switched on, so it flips whatever order the previous step produced. Write the lines back out as UTF-8 with a newline after each one.
What it assumes
- Trimming runs before deduplicating and deduplicating runs before sorting. Change that order and the answers change — "milk " and "milk" are one entry with trimming on and two with it off.
- Lines that compare exactly equal keep the order the file had them in, because every sort here is stable. With case ignored, "Milk" and "milk" are equal, so whichever appeared first stays first.
- Ordering uses English collation rules regardless of the language of the page, so the same file and the same settings always produce the same file. Under other locales a handful of letters would sort elsewhere — in Swedish, for example, Å comes after Z rather than beside A.
- Case-insensitive matching uses Unicode lower-casing, so ß and SS are still different lines. Accents are always a real difference — "résumé" and "resume" are never treated as duplicates.
- The whole file is held in memory at once, which is why it is capped at 20 MB. Anything larger belongs in sort, awk or a database rather than a browser tab.
- The file that comes back is UTF-8 with Unix line endings, with no byte-order mark, whatever the file that went in had.
Common questions
Is the list I paste in uploaded anywhere?
No. The file is read in this browser tab and processed by a worker running on your own machine. There is no server to send it to, and the automated test for this page fails if any request leaves this site while a file is loaded.
Why did "Milk" disappear when "milk" was already in the list?
Because matching ignores case unless you turn "Match case" on. A list of names, addresses or products usually wants Milk and milk treated as one entry, so that is the default. Turn "Match case" on and both are kept, and they will also sort apart from each other.
Why is Zebra after apple rather than before it?
Because the lines are ordered by English collation rules rather than by character codes. Every capital letter has a lower character code than every lower-case one, so a raw code order gives Zebra, apple, banana — which is the order a computer sees and nobody wants. Collation also puts Ångström beside Anchor instead of after Zulu.
Why is item10 after item2 instead of between item1 and item2?
Because "Read numbers as numbers" is on by default, so a run of digits inside a line is compared as a number. Turn it off under "More options" for a strict character-by-character order, which gives item1, item10, item2.
What counts as a blank line?
A line containing nothing, or nothing but spaces and tabs. That holds whether or not trimming is on, so the two toggles cannot disagree about what blank means. A line of three spaces is removed by "Remove blank lines" even with trimming switched off.
My file came back with different line endings.
It did. Windows CRLF and classic Mac CR are both read correctly, and what is written back always uses a single newline after each line, with one at the end of the file. Every editor in current use opens that, including Notepad.
Why was my file refused?
Two things are refused rather than mangled. A file saved as UTF-16 begins with a byte-order mark that says so, and decoding it as UTF-8 would produce a page of replacement characters that looks like the tool broke the file — re-save it as UTF-8 instead. A file with a zero byte near the start is binary rather than text, and running line tools over it would produce nonsense.