Token (LLM)

Language models don't read text the way humans do. They break text into tokens, which are typically 3-4 characters long. The word "understanding" might be two tokens: "under" and "standing." Common words like "the" are a single token. Rare or technical terms get split into more tokens.

Tokenization is handled by a tokenizer specific to each model. OpenAI's GPT-4 family uses cl100k_base or o200k_base (tiktoken). Anthropic's Claude uses its own tokenizer, exposed through the count_tokens API. Google's Gemini uses SentencePiece variants. These tokenizers assign different token counts to the same text. A paragraph that's 200 tokens for GPT-4o might be 210 for Claude and 190 for Gemini. Close but not identical, which matters when you're comparing prices across providers.

This matters for two practical reasons. First, pricing. API providers charge per token, with separate rates for input tokens (your prompt) and output tokens (the model's response). Output tokens are typically 3-5x more expensive than input. A long prompt with lots of context costs more than a short one. Understanding your token usage is essential for managing AI costs at scale.

Second, token limits. Every model has a maximum number of tokens it can handle in a single request (its context window). This includes both your input and the model's output. If your prompt is too long, you need to either shorten it, use a model with a larger context window, or redesign your approach.

In RAG systems, token management is a key design decision. You need to fit the user's question, the retrieved documents, the system prompt, and leave room for the model's response, all within the context window. This is why chunking strategy and retrieval count matter: you want to maximize relevant context without hitting the token limit or burning budget.

As a rough rule of thumb, 1 token is about 0.75 words in English. A 1,000-word document is roughly 1,300 tokens. Non-English text, code, and structured data often tokenize less efficiently. Chinese and Japanese text can use 2-3x more tokens per character than English. JSON with deeply nested keys tokenizes worse than flat Markdown. If your workload has a lot of non-English content or code, benchmark actual token usage rather than estimating from the English rule of thumb.

A SaaS company builds an AI-powered email reply assistant embedded in their customer support tool. In the first month of production, usage ramps to 14,000 assistant invocations per day. Each invocation uses roughly 3,500 input tokens (system prompt, 3 retrieved policy docs, conversation history) and produces about 400 output tokens.

Total daily tokens: 49M input plus 5.6M output. On Claude 3.5 Sonnet at $3 per million input and $15 per million output, that's $147 + $84 = $231 per day, or about $7,000 per month. The finance team flags the spend. The engineering team looks at traces in Langfuse and finds three optimizations.

One: the system prompt and the retrieved policy docs are identical across the first few turns of a conversation. Adding Anthropic prompt caching on that prefix drops input costs on cached hits to about $0.30 per million tokens. About 70% of invocations are follow-up turns that hit the cache. Effective input cost drops by roughly 55%. Two: they route simple acknowledgement replies ("got it, thanks") through a small router classifier to Claude Haiku at $0.25/$1.25 per million. That's about 30% of traffic at 20x lower cost. Three: max_tokens is set to 500 instead of unlimited, preventing the occasional 4,000-token response that nobody reads.

Post-optimization: monthly cost drops from $7,000 to $2,100 with no quality regression. The team adds a per-tenant token budget in the API gateway as a safety net.