Why two word counters disagree about the same text

There is no such thing as a word

Ask two tools to count the words in a paragraph and you may get two numbers. Neither is wrong, because the unit is not defined.

Consider what has to be decided:

• Is state-of-the-art one word or four?
• Is don't one or two?
• Is £1,250.00 a word?
• Is https://example.com/a/b one word or several?
• Is sentence—another two words when there are no spaces around the dash?

Word processors, publishers and academic style guides answer these differently, and each answer is reasonable. A tool has to pick.

The most principled available rule is the Unicode text segmentation algorithm, which defines word boundaries in a way that works across scripts — including the ones where whitespace does not separate words at all, such as Thai and Japanese. A tool that splits on spaces gets English roughly right and those languages entirely wrong.

So the useful question about a word count is not “is it correct” but “is it consistent”. For tracking a draft against a limit, a stable rule matters more than a philosophically ideal one.

Before you read on

You paste the same paragraph into two word counters and get 148 and 151. Which one is broken?

• AThe one reporting 148 — it is dropping words.
• BNeither. A word is not a defined unit.

  Right, and this is the whole point of the page.

• CBoth, if they disagree they cannot both be right.

Neither. There is no single definition of a word, so every tool picks a rule: whether a hyphenated compound is one word or four, whether a contraction is one or two, whether a URL counts at all. Different reasonable answers give different totals. The useful question about a word count is not whether it is correct but whether it is consistent.

Where a line ends

Three conventions, all still in circulation:

Bytes	Name	Where
\n	LF	Unix, Linux, macOS since 2001
\r\n	CRLF	Windows, and most internet protocols
\r	CR	classic Mac OS, before 2001

Two files that look identical in an editor can differ in every line ending, and the consequences are real: a different byte count, a different checksum, an extra blank line when a naive split treats \r\n as two terminators.

The reliable approach is to normalise on the way in — fold every convention to a single line feed — and count from there. Every text tool here does that, which is why a pasted block of Windows text and the same block typed fresh produce the same figures.

The other classic is the trailing newline. A file ending with a newline has one by convention rather than by accident, and whether that final empty string counts as a line is a decision a tool has to make and state.

Characters are not bytes

Before words or lines can be counted, the byte sequence has to become Unicode characters. UTF-8, UTF-16 little-endian and UTF-16 big-endian can represent the same text with different bytes; a byte-order mark can identify the UTF-16 order, while unmarked input still needs an explicit contract. Opening UTF-8 bytes as a legacy code page is how café becomes a familiar strip of wrong characters.

That corruption is called mojibake, and saving it again does not repair the original mapping. A safe encoding conversion starts from a known source encoding, rejects byte sequences that are invalid under it, then encodes the resulting Unicode scalar values in the target form. Strict failure is useful: silently inserting replacement characters makes an apparently successful file that cannot be reversed.

Line endings are a separate layer. Converting UTF-16 to UTF-8 can preserve CRLF exactly, or an explicit option can normalise it to LF. Combining those decisions under an unexplained “fix text” button makes it impossible to know whether a checksum changed because the characters, their encoding, or their terminators changed.

Case is not a simple mapping

Uppercasing looks like a lookup table and is not.

• The German sharp s. ß uppercases to SS. The string gets longer, so any code assuming case conversion preserves length is wrong.
• Turkish i. Turkish has a dotted and a dotless i, and they uppercase and lowercase differently from every other language. Applying English rules to Turkish text corrupts it.
• Scripts without case. Arabic, Hebrew, Chinese, Japanese, Korean and Thai have no case at all, so the operation is a no-op rather than an error.

Title case is worse, because it is a style question rather than a character question: which small words stay lowercase is a decision belonging to a style guide, and the guides disagree with each other.

CSV has a specification, and it is not “split on commas”

The format looks like the simplest thing in computing and has one genuinely tricky rule: a field may contain a comma, a line break or a quote character, and it handles that by wrapping the field in quotes and doubling any quote inside it.

name,note
Smith,"Lives at 12 High Street, Ipswich"
Jones,"She said ""no"" twice"
Brown,"Line one
Line two"

Every one of those rows is valid, and every one breaks a parser that splits on commas. The third breaks anything that assumes one record per line.

Add the things the specification does not settle — the delimiter is not always a comma, encodings vary, a byte-order mark may lead the file, and some exporters quote everything while others quote nothing — and it becomes clear why real CSV parsing is a state machine rather than a split.

The lesson generalises past CSV. Text formats that look obvious are where the edge cases hide, precisely because the obvious implementation works on the first file you try.