What Is a Token in AI? Tokens Explained in Plain English

What Is a Token in AI? Tokens Explained in Plain English

⚠️ Affiliate Disclosure: This post is not an affiliate associated review. It may become one in the future and this disclosure will reflect that. This does not affect my reviews and I only recommend tools I genuinely believe in.

📅 Last updated: 24 October 2026

What Is a Token in AI?

A token in AI is the small chunk of text a language model actually reads and writes. Not a word, not a letter — something in between. “Cat” is one token. “Unbelievable” might be three. A full stop is a token on its own.

Tokens matter more than most people realise. They decide how much text an AI tool can handle at once, how quickly you hit your usage limit, and — if you pay per use — exactly what you are charged. Once you understand tokens, a lot of confusing AI pricing suddenly makes sense.

How Many Words Is a Token?

For English text, OpenAI’s rule of thumb is that one token is roughly four characters, or about three-quarters of a word. That gives you some handy conversions:

TokensRoughly equals
1 token¾ of a word, or about 4 characters
100 tokensAbout 75 words
1,000 tokensAbout 750 words — a short blog post
100,000 tokensAbout 75,000 words — a full-length novel

These are averages, not rules. Common English words are usually one token each. Long, rare or technical words get split into several. Other languages often use more tokens for the same meaning, and so does code.

Why Do AI Models Use Tokens Instead of Words?

Because a fixed list of whole words would never be big enough. New words, names, typos, product codes and other languages would all fall outside it. Breaking text into reusable pieces — common words whole, rarer words in fragments — lets a model handle any text it meets with a manageable vocabulary.

This process is called tokenisation. Before your prompt reaches the model, a tokeniser chops it into tokens and turns each one into a number. The model works entirely in those numbers, predicts the next token, then the next, and the result is turned back into text for you. Each AI company uses its own tokeniser, so the same sentence can be a slightly different number of tokens in ChatGPT, Claude or Gemini.

What Are Input and Output Tokens?

  • Input tokens are everything you send the AI: your prompt, any document you paste in, and the earlier messages in the conversation.
  • Output tokens are everything the AI writes back.

The distinction matters because AI companies price them differently. On pay-as-you-go plans, output tokens usually cost several times more than input tokens, because generating text takes more computing work than reading it.

Do You Pay for AI Tokens?

It depends how you use AI.

  • Chat apps on a subscription — ChatGPT, Claude, Gemini and similar — do not charge you per token. But tokens still sit behind the “you have reached your limit” message: your plan allows a certain amount of usage, and long conversations and big file uploads use it up faster.
  • API access, used by developers and by many AI tools under the bonnet, is billed per token — usually quoted per million tokens, with separate prices for input and output.
  • AI tools that charge in credits are often passing token costs on to you in a friendlier unit. One credit might cover a fixed number of tokens, words or images.

That last point is why AI credits come up again and again in my reviews. When a tool’s credits run out faster than you expected, tokens are usually the reason.

What Is a Token Limit?

Every model can only consider a certain number of tokens at once. This is called the context window, and it covers both what you send and what the AI writes back. Go past it and something has to give: older parts of the conversation drop out of view, long documents get cut off, or the tool refuses the request.

It is the reason a very long chat can start to “forget” what you said at the beginning. The model has not lost interest — your early messages have fallen outside the window. Context windows have grown enormously since 2023, but they are still finite, and a bigger window does not guarantee the model pays equal attention to everything inside it.

How to Use Fewer Tokens

Whether you are watching a usage cap or a bill, the same habits help:

  • Start a new chat for a new task — every message in a long conversation is re-read each time, so old context costs tokens on every reply.
  • Paste only the part that matters, not the whole document.
  • Ask for the length you need — “in three bullet points” uses far fewer output tokens than an open question.
  • Avoid repeating instructions that the tool already has in its custom instructions or project settings.

Frequently Asked Questions

How many words are 1,000 tokens?

About 750 words of ordinary English. The exact figure depends on the text: plain everyday words use fewer tokens, while technical terms, names, other languages and code use more.

How many tokens is one word?

On average a little more than one — about 1.3 tokens per English word. Short common words are usually a single token; longer or unusual words are split into two or more.

Do ChatGPT and Claude count tokens the same way?

No. Each AI company uses its own tokeniser, so the same text produces slightly different token counts in ChatGPT, Claude and Gemini. The differences are small for everyday English but can matter for pricing at scale.

Are AI tokens the same as crypto tokens?

No. In AI, a token is a piece of text a language model processes. Crypto tokens are digital assets on a blockchain. The shared word is a coincidence and the two are unrelated.

Why does my AI chat forget earlier messages?

Because the conversation has outgrown the model’s context window — the maximum number of tokens it can consider at once. Earlier messages fall out of view. Starting a fresh chat with a short summary of what matters usually fixes it.

More AI Explained guides: What Is an LLM? · What Is an AI Agent? · AI Hallucinations: Why AI Makes Things Up

James Huntington
James Huntington
AI Tools Expert

James is an SEO and content specialist with over 12 years’ experience, and tests AI tools so everyday people don’t have to. WhatAITools.co.uk covers honest, no-jargon reviews to help you find the right tool for your needs.

LinkedIn · Follow on 𝕏 (Twitter / X)


Tried this tool?

Share your experience below — your review helps other everyday people make better decisions.

Leave a Reply

Your email address will not be published. Required fields are marked *