Why ChatGPT Still Can't Count the Letters in a Word

Ask ChatGPT, Gemini or Claude how many times the letter "r" appears in "strawberry," and there is still a real chance one of them says two. It has been a running joke online for more than two years now, ever since users first noticed frontier chatbots flunking a question a six-year-old answers correctly. What is less understood is why the industry has not simply patched it.

The honest answer is that it is not a bug. It is a side effect of how every major chatbot reads text in the first place.

Why can't AI models see individual letters?

Large language models never take in raw text. Before a word reaches the neural network, a tokenizer chops the input into chunks called tokens — sometimes a whole word, sometimes a fragment, rarely a single letter. OpenAI's own tokenizer library, tiktoken, states the underlying issue directly:

"Language models don't see text like you and I, instead they see a sequence of numbers (known as tokens)."

OpenAI, tiktoken documentation

Run "strawberry" through o200k_base, the encoding behind GPT-4o and newer OpenAI models, and it splits into exactly three tokens — st, raw and berry — according to tokenizer analysis published by the developer-education site AI Behaves. None of those three chunks map to individual letters. Ask the model to count the r's and it is not looking at ten characters; it is being handed three numbers that stand for "st," "raw" and "berry," with no record of what is inside them.

Why does the model guess instead of admitting it can't see the letters?

Because guessing is what a language model does for a living. Its job is predicting the next plausible chunk of text given everything before it, not executing a verifiable counting procedure. Internet text about counting letters is dominated by small, round numbers, and "two" is a common, unremarkable answer to lean on. It is not a lie and not sloppy arithmetic — it is a response to a question the architecture was never built to answer, generated the same way the model generates everything else: by pattern, not by inspection.

That is also why the failure looks so out of place next to what these same tools do well. Summarizing a contract, translating a paragraph, drafting an email — all of that survives tokenization intact, because words, clauses and ideas are exactly the units a token is built to preserve. Ask about the letters buried inside one specific word, and you are asking about the one layer of information tokenization discards on purpose. Each token typically stands in for around four bytes of text, per tiktoken's own documentation — a compression trade-off, not an oversight, and one that also explains why running these models at scale consumes so much computing power in the first place: shorter token sequences are cheaper to process, even when they cost the model its ability to see individual characters.

Does spelling the word out fix the problem?

Usually, yes. Typing "s-t-r-a-w-b-e-r-r-y" instead of "strawberry" splits most tokenizers into ten separate one-letter tokens instead of three fused chunks, so the letters finally exist as distinct units the model can reason over. The same trick works if you ask the model to list every letter with its position before answering; models that reason step by step before responding sometimes stumble onto this workaround on their own, which is why the identical question can succeed one day and fail the next depending on how it happens to be phrased.

The more durable fix, and the one AI companies have leaned on quietly, is routing character-level tasks to an actual tool rather than the language model — having the system write and run a short script to do the counting instead of guessing. A calculator was always going to beat a next-word predictor at counting; what has changed is that chatbots increasingly know to reach for one rather than answer from memory.

Will future models fix this on their own?

Not by getting better at spelling — by getting better at recognizing when spelling is not their job. As AI Behaves' breakdown of the mechanism puts it, any character-level claim a chatbot makes — a string's exact length, whether it contains a double letter, how many words end in "-ing" — is an answer about a layer of text the model was never shown, and should be treated as unverified until checked independently. That gap between appearance and mechanism matters more by the day now that ChatGPT alone counts a billion monthly app users, most of whom have no idea their chatbot is reading in sealed chunks rather than characters. The tokenizer is not going away, and neither, for now, is the strawberry test.