Context ordering: test rather than assume

Moving the question before evidence, placing critical instructions last or sorting retrieved chunks by similarity can change model behaviour. There is no reason to promote one arrangement from a single example into a universal rule.

Treat order as an experiment

The offline foundations lab compares two authored outcome arrays.

from foundation_cases import context_order_case

result = context_order_case()
assert result["accuracy"] == {"question_first": 0.75, "evidence_first": 0.5}
assert result["selected_for_fixture"] == "question_first"
assert result["universal_order_claim"] is False
print(result["scope"])

Question-first wins on this four-case fixture only. No LLM ran, and the result cannot support a general ordering recommendation.

Hold everything else fixed

Use identical task cases, model snapshot, decoding settings, evidence and output schema. Change only the order. Log the fully rendered request so hidden framework templates do not invalidate the comparison. Run repeats if sampling can vary.

Measure exact-match or business invariants where possible, plus citation support and per-slice results. Add long-context cases with relevant evidence near the beginning, middle and end. Include distractors and contradictory versions. An average can hide a severe drop when the decisive paragraph appears in one position.

Design for robust use, not a winning prompt

Even after selecting an arrangement, reduce dependence on position. Retrieve fewer higher-quality sections, label sources clearly, repeat critical application instructions in trusted channels where appropriate and validate evidence references. Use document structure to preserve relationships instead of flattening every chunk.

If one order wins narrowly with a small set, record uncertainty and keep the decision provisional. Rerun after changing model, tokenizer, chunking, tool definitions or conversation history.

The Generative & Agentic AI course integrates ordering tests with context budgets and retrieval evaluation rather than presenting prompt folklore as fact.

Exercise

Build 20 tasks and render three orders: question-first, evidence-first and grouped-by-source. Rotate the location of decisive evidence, preserve request logs and report per-position transitions plus critical failures.

Continue learning

This article is part of the LLM fundamentals and prompt design sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Evaluate prompt changes on a fixed task set.
• Continue with Compress a conversation without losing task constraints.

Reference: “Lost in the Middle” paper.