Tokens
The units of text, roughly word fragments, that a language model reads and writes, and the unit in which context limits and API prices are measured.
Definition
Language models do not read letters or whole words. Before text reaches the model, a tokeniser splits it into tokens: common words become one token, rarer words become two or more pieces, and punctuation, spaces and numbers get their own. "Invoicing" might be "Invo" + "icing"; "the" is one token. Each model family has its own tokeniser, so the same text can have slightly different counts on different models.
Tokens are the currency of everything else. The context window is a token limit. API prices are quoted per million input and output tokens. Speed is measured in tokens per second. For English, three to four characters per token is a workable estimate; code, other languages and tables are often less efficient.
Example: a support assistant handles 5,000 conversations a day. Each one sends a 1,200-token system prompt, 800 tokens of retrieved help-centre text and 150 tokens of question, and generates a 250-token answer. That is roughly 11 million input and 1.25 million output tokens a day, which the model's price list turns into a monthly budget.
Why it matters for your business
- Cost is a function of tokens, so prompt length and answer length are budget decisions.
- Long system prompts repeated on every request are the most common hidden cost; prompt caching can reduce it.
- Token limits determine how much document text can be processed per call and how chunking is sized.
- Non-English content and structured data tokenise less efficiently; estimates should use your real text.
How Teksmobile uses it
We measure token usage from day one of a build and set budgets per feature, as described in our guide to the cost of LLM applications. The LLM cost calculator and our LLMOps and monitoring service keep spend visible once a system is live.
Related terms
Part of the Teksmobile AI Glossary. For longer reads, see our guides.
Frequently asked questions
How many tokens are in a page of text?
For English, a rough rule is three to four characters per token, so a typical page of around 500 words is roughly 650 to 750 tokens. Other languages and code often use more tokens per word.
Why are input and output tokens priced differently?
Generating a token costs more compute than reading one, so providers usually charge several times more per output token. Long answers cost more than long questions.
How do we estimate our monthly token usage?
Multiply requests per day by average input and output tokens per request, then apply the model's prices. Our LLM cost calculator does this for you.