What are tokens in AI? How many words they are and why they matter

Tokens in AI are the chunks ChatGPT, Claude or Gemini break text into to read and write it: a whole word, part of a word or a punctuation mark. In English, a token is about 4 characters or three-quarters of a word, according to OpenAI’s guide to tokens, and other languages need more of them to say the same thing. They matter because AI is billed by the token and can only keep a limited number in view at once, which is why it seems to forget the start of a very long conversation.

To put numbers on it, in October 2026 we counted the tokens in the same 100-word notice written in four languages. With o200k_base, the encoding OpenAI’s official library assigns to GPT-4o and GPT-5, Spanish needed 6% more tokens than English and French 15% more; with cl100k_base, GPT-4’s encoding, 27% and 41% more.

What are tokens in AI, and how do they work?

A token is the unit a language model works with. The model doesn’t read letters or words: it has a fixed catalog of pieces, about 200,000 in o200k_base, the encoding of GPT-4o and GPT-5. It turns your text into a row of those pieces and builds its reply piece by piece, like building with blocks.

This is how a message travels:

How many words is a token?

In English, a token is about 4 characters or three-quarters of a word, so 100 tokens are about 75 words, according to OpenAI. For Gemini, Google’s documentation uses the same 4-character rule and says 100 tokens are about 60 to 80 English words. Other languages need more.

Claude does its own thing. According to Anthropic’s glossary, on its models before Claude 4.7 a token is about 3.5 English characters. From Claude 4.7 on, a newer tokenizer produces roughly 30% more tokens for the same text.

In our tests with o200k_base, 100 tokens covered about 90 words of everyday English. The official rules are round numbers: good for estimating, not for budgeting.

How does AI split a sentence into tokens?

It breaks uncommon words into smaller chunks called subwords. Frequent words take a single token; long or rare ones are built from several and, in the worst case, from the smallest pieces, so no text is left out. What it doesn’t do is split by syllables or by meaning.

The technique OpenAI uses is called BPE (byte pair encoding), according to its tiktoken library. Rico Sennrich, Barry Haddow and Alexandra Birch adapted it to split words in a paper presented at ACL 2016. The method starts from single characters and keeps merging the pairs that appear most often.

Notice: the space sticks to the next word, punctuation marks are pieces of their own, and “otolaryngologist” ended up in four chunks that aren’t syllables. And the newer encoding has more Spanish pieces: “cómo” and “estás” went from two tokens to one.

Why do other languages use more tokens than English?

Because each tokenizer learns its pieces from the text it’s trained on, and English tends to dominate that text. That’s the explanation Ahia and colleagues give in a study presented at EMNLP 2023. So English words fit in a single token more often, while words in other languages get split more.

We measured it in October 2026 with gpt-tokenizer 4.1.0, an open-source library that reproduces OpenAI’s tokenizers. It returns the same figures OpenAI published when it introduced GPT-4o: 29 and 26 tokens for its Spanish example. We counted the same bakery notice in four languages and confirmed the results with a second library, js-tiktoken 1.0.21.

It depends on the text. In a second test, an instruction of about 60 words asking a chatbot to reply to a customer, Spanish needed 21% more tokens than English with o200k_base, Portuguese 15% and French 37%.

Large studies point the same way. With 2,000 sentences translated into 200 languages, Petrov and colleagues (NeurIPS 2023) found that GPT-4’s tokenizer uses 1.55 tokens in Spanish, 1.48 in Portuguese and 1.60 in French for every token in English. And according to Ahia’s team, languages with their own script, such as Telugu, need up to 5 times as many.

Keep in mind: these figures come from OpenAI’s tokenizers. Claude and Gemini use their own, so their numbers differ, and both offer official ways to count: Anthropic’s token counting, which is free, and Google’s countTokens function.

Why does AI charge by the token?

Because tokens measure the model’s work: the more it has to process, the longer it takes, as Petrov and colleagues point out. That’s why providers bill per million tokens, with two rates: one for what you send, the input, and another, usually higher, for what the model writes, the output.

For example, as of October 2026, Claude Sonnet 5.5 costs $2 per million input tokens and $10 per million output tokens, according to Anthropic’s pricing page. Each token it writes costs five times as much as each token it reads.

With models that reason before answering, that reasoning isn’t shown but is billed as output, OpenAI explains. And in the ChatGPT, Claude or Gemini apps you usually don’t see this math: the per-million price applies to their APIs, the connection businesses and developers use.

Not all AI is pay-per-use: a small model can be downloaded once and run on your own phone or computer. That’s how our free tool to remove the background from a photo works: it downloads a neural network of about 14 MB and cuts out photos in your browser, without uploading them to any server.

Why does AI forget what you told it?

Because it can only keep a limited number of tokens in view at once: its context window. Your messages and its replies pile up in it, and when the conversation no longer fits, chat apps may start dropping the oldest parts. And with a huge amount of text, the model recalls details less well.

Anthropic describes it as the model’s “working memory.” Its guide to context windows explains that chat apps such as claude.ai can manage it like a queue: first in, first out.

Today’s windows are huge: as of October 2026, the newest Gemini models share a 1,048,576-token window across what they read, their reasoning and their answer, according to Google, and several Claude models reach 1 million, according to Anthropic. By Google’s rule, a million tokens is 600,000 to 800,000 English words. But Anthropic warns that as the token count grows, accuracy and recall degrade.

In practice: if a conversation gets very long, start a new one with a summary of what matters and repeat the key instructions.

What changes for you and your business

Knowing how tokens are counted helps you spend less and get better answers:

• Estimate with a margin. In everyday English, expect about 1.1 tokens per word with the GPT-4o and GPT-5 encoding; Spanish, Portuguese and French need more. For an exact figure, paste a sample into OpenAI’s Tokenizer.
• Get to the point. Filler is paid for and takes up room in the window: OpenAI suggests removing unnecessary or repeated context and summarizing long text before sending it.
• Ask for the length you need. If output costs five times as much as input, asking for short answers when they’re enough is the most direct way to save.
• Split huge documents. If a contract or a catalog doesn’t fit in the window, send it in parts, with only what the task needs.
• Compare by task, not by price per million. As OpenAI warns, a lower rate doesn’t guarantee a lower total cost, because each model splits text its own way.

What gets exaggerated about tokens

• “A token is a word.” Not quite: everyday English came close in our tests, at 1.1 tokens per word, but Spanish took 1.2 to 1.3, and “otolaryngologist” alone took four.
• “More tokens, better answers.” More text isn’t more quality: with lots of context, the model recalls less well, as Anthropic warns. Give it what it needs, not everything you have.
• “Newer tokenizers always use fewer tokens.” GPT-4o’s cut the tokens for Spanish, but Anthropic warns that the one in Claude 4.7 and later produces roughly 30% more tokens for the same text.

What to remember

A token is a piece of text, not a word, and almost everything in AI is measured in it: what it reads, what it writes, what it charges and what it remembers. Write directly, ask for answers of the right length and, before spending real money, measure your texts with the official counter.

And if you work with texts in several languages, our free translator handles more than 20 of them and lets you hear how they sound. It’s in the Vertize Plus tools library, along with other utilities.

Frequently asked questions