
Context window is the specification everyone quotes and almost nobody interprets correctly. A model advertising two million tokens sounds like it can hold a shelf of books in working memory and reason across all of it equally well.
It cannot, and the gap between the number and the behaviour is where a lot of disappointing AI projects live.
This is the practical explanation: what a context window actually is, why usable capacity is smaller than advertised capacity, and how to size your inputs so you get reliable output instead of confident nonsense.
Table of Contents
ToggleA context window is the total amount of text a model can consider at once, measured in tokens. It covers everything in the exchange:
That last item is the one people forget. Output consumes the same budget as input. A model with a 128K window generating a 16K-token response has 112K available for everything else, not 128K.
Tokens are not words. In English, one token is roughly ¾ of a word — about 750 words per 1,000 tokens. Code tokenises less efficiently, often 1.5–2× more tokens for the same visual length, because symbols, indentation, and identifiers fragment. Non-English text, especially languages not written in Latin script, can be substantially worse.
Practical conversions worth memorising:
| Content | Approximate tokens |
| One page of prose | 500–750 |
| A 20-page report | 10,000–15,000 |
| A 300-page book | 120,000–180,000 |
| A 500-line source file | 6,000–10,000 |
| A one-hour meeting transcript | 8,000–12,000 |
Here is the part the spec sheet does not tell you: retrieval accuracy is not uniform across the window.
Models tend to process information most reliably when it appears near the start or end of the context. Material in the middle is retrieved less reliably, and the effect gets more pronounced as the window fills. This has been documented repeatedly, and it is often called the “lost in the middle” problem.
The practical consequence is blunt: a fact placed at 60% depth in a very long context is meaningfully less likely to be used correctly than the same fact placed at the start or the end.
A second effect compounds it. Long contexts do not just risk missed facts — they degrade reasoning quality. A model tracking 400,000 tokens of loosely related material produces vaguer, more hedged, less specific answers than the same model given the 20,000 tokens that actually matter. More context doesn’t always add value; beyond a certain point, it simply creates more noise.
So the honest reading of a large context window is this: it is a capacity figure, not a fidelity figure. It tells you what will fit, not what will be reasoned over well.
Context limits vary by roughly an order of magnitude across mainstream models, from a few hundred thousand tokens at the low end to two million at the high end. Maximum output length varies too, and independently — a model with a very large input window may still cap responses at a few thousand tokens, which matters enormously if your task is generating a long document rather than analysing one.
Because both numbers move with every release, check them against a current source rather than memory. Side-by-side model comparison tools that list context window and maximum output tokens together are the fastest way to do this, and looking at both columns at once prevents the common mistake of choosing a model on input capacity for a task that is actually output-bound.
Five rules that reliably improve output quality.
Structure long prompts as: context first, question last. Models attend strongly to the end of the window, and burying your actual request under 50,000 tokens of source material is the single most common cause of a model “ignoring” an instruction.
The instinct with a large window is to include everything and let the model sort it out. This produces worse results than including only what matters. Twenty relevant pages beat two hundred pages containing those twenty. If you find yourself relying on the model to filter, do the filtering yourself — you are better at it and it costs nothing.
Input tokens are billed, and long-context work is expensive in a way that surprises people. At typical rates of a few dollars per million input tokens, a single 500,000-token analysis costs real money, and repeating it across a document set costs real money repeatedly. Long context is a genuine capability with a genuine bill attached.
If your task can be split — summarise each section, then combine the summaries — do that instead of one enormous pass. You get better attention on each part, lower cost, and the ability to inspect intermediate results when something goes wrong. Reserve full-context passes for tasks that genuinely require cross-referencing distant parts of the material.
Before trusting a long-context workflow, verify it. Place a specific, unusual fact at the midpoint of a representative document and ask for it. If the model misses it, your effective working limit is lower than the advertised one, and you should design around that rather than hope.
These are routinely confused and are completely different mechanisms.
Context window is working memory for a single exchange. It resets. Nothing persists between conversations unless it is re-sent.
Memory features are a storage layer built on top. The system saves summaries or facts and re-injects relevant ones into future context. That is a retrieval system, not a larger window — and it is why an assistant can “remember” your preferences across months while still having a fixed per-conversation limit.
Both matter, and they fail differently. Context limits cause truncation and lost detail within a task. Memory limits cause forgetting between tasks. Diagnosing which one you are hitting saves a lot of pointless prompt tuning.
For a large, stable corpus — a documentation set, a knowledge base, a year of transcripts — stuffing context is usually the wrong architecture. Retrieval is better: index the corpus, fetch the relevant fragments per query, and send only those. It is cheaper, faster, more accurate, and it scales past any window size.
Long context is the right tool when you need genuine cross-referencing over a bounded set of material, when the relationships between distant parts matter, or when you simply need an answer once and do not want to build anything. It is the wrong tool as a substitute for retrieval on a corpus you query repeatedly.
Three criteria, in priority order:
Testing across several models is worth the effort here, because long-context behaviour varies more between models than headline benchmarks suggest. Multi-model platforms such as Perspective AI make that comparison practical by putting several labs’ models behind one account, so running the same document through three of them is a dropdown change rather than three subscriptions.
What is a token?
The unit models process text in — roughly ¾ of an English word. Code and non-Latin scripts consume more tokens per character.
Does a bigger context window mean a better model?
No. It means more capacity. Reasoning quality, coding ability, and factual reliability are separate properties, and a model can lead on window size while trailing on all three.
Why does the model forget things from earlier in a long chat?
Either the conversation exceeded the window and early messages were truncated, or the material fell into the low-attention middle region. Re-stating key facts near your question fixes both.
Is retrieval better than a long context window?
For large, stable, repeatedly queried corpora, generally yes — cheaper and more accurate. For one-off cross-referencing over a bounded set, long context is simpler and often better.
Treat the context window as a ceiling, not a target. Curate your input, put the instruction last, stay well under the limit, and verify retrieval at the depth you actually use.
The model that fits your document is not automatically the model that understands it.