Tokens and context windows: budget a real document task

A context window is shared space. System instructions, conversation history, retrieved evidence, tool schemas and generated output all consume it. Sending a document that fits by itself can still overflow once the application adds everything else.

Reserve the budget before selecting evidence

The offline foundations lab uses whitespace-delimited teaching units to make the arithmetic inspectable.

from foundation_cases import context_budget_case

result = context_budget_case()
assert result["context_limit"] == 1000
assert result["evidence_budget"] == 500
assert result["selected_sections"] == ["contract", "exceptions"]
assert result["provider_tokenizer_executed"] is False

From a 1,000-unit limit, the fixture reserves 120 for system instructions, 80 for the user request, 220 for output and 80 as margin. That leaves 500 for evidence. The 180-unit contract and 210-unit exception section fit; adding the 260-unit appendix would exceed the budget.

These are not provider token counts. Production code must use the tokenizer and context rules supported by the selected model/version.

Budget for the whole serialized request

Measure the exact messages and tool definitions the API receives. Reserve output explicitly; a request that fills the input window can leave no room for the answer. Include a margin for escaping, wrappers or provider-specific formatting. Log the model/version, tokenizer/version and counted components with the run.

When evidence exceeds the budget, do not truncate blindly. Rank sections against the question, preserve headings and source identifiers, and split work where independent subproblems exist. Summaries can reduce size, but they become derived evidence that needs provenance and checks for lost constraints.

Test important boundaries

Create cases just below, at and above the limit. Include long tables, non-English text, code, repeated metadata and a large tool schema. Verify that the application rejects or reduces an oversized request before an expensive call. Measure answer quality as context grows; fitting within a limit does not prove the model uses every part reliably.

The Generative & Agentic AI course connects context budgeting to RAG ingestion, evaluations and cost control. A deliverable should include its budget calculation and truncation policy.

Exercise

Choose a real public document and three questions. Count the complete serialized requests with your selected model’s supported tokenizer, allocate output/margin, and compare section selection with naive first-*n* truncation on a fixed evidence set.

Continue learning

This article is part of the LLM fundamentals and prompt design sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Continue with Temperature is not a reliability control.
• Then apply it in Prompt instructions versus retrieved evidence.

Reference: OpenAI tokenizer tool.