Create a small labelled dataset for an LLM application

A small reviewed dataset is more useful than a large unlabeled folder when you need to compare prompts, models or retrieval choices. Start with the decision and failure boundaries, then sample cases deliberately.

Preserve disagreement

The offline foundations lab contains four authored rows.

from foundation_cases import labelled_dataset_case

result = labelled_dataset_case()
assert result["slices"] == ["conflict", "injection", "supported", "unsupported"]
assert result["agreement"] == 0.75
assert result["disagreements"] == ["L3"]
assert result["adjudication_pending"] == ["L3"]

Two reviewers disagree on the conflicting-evidence case, so it remains pending. The lab does not silently choose the label that favours a candidate. This is a four-row teaching fixture, not a measured annotation study.

Write the label guide first

Define the task, supported inputs, output fields, evidence rules, abstention criteria and examples of ambiguity. Give cases stable IDs and separate raw input from labels. Record source, permissions, date, reviewer and adjudication state.

Sample common cases and rare high-impact boundaries: missing evidence, conflict, prompt injection, language variation, long context and malformed inputs. Avoid building the set entirely from current system failures; include prospective coverage so it can detect new regressions.

Protect the evaluation boundary

Split development examples from a release set. Prompt authors can use the former; repeated inspection of the latter turns it into another development set. Version labels and explain corrections. Measure per-slice denominators rather than reporting one score from four rows.

Remove unnecessary personal data and respect dataset licences, retention and access. Synthetic cases can cover security boundaries, but must be labelled as synthetic and cannot establish performance on the real population.

The Generative & Agentic AI course uses labelled sets across prompt, retrieval, agent and evaluator work.

Exercise

Write a label guide and 40 cases across eight slices. Double-label ten, calculate agreement, preserve disputed IDs and adjudicate with reasons. Freeze a release subset before comparing two candidates.

Continue learning

This article is part of the LLM fundamentals and prompt design sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Choose retrieval, fine-tuning or a better task definition.
• Continue with Write a model selection decision record.

Reference: OpenAI evaluation best practices.