Validate JSON output beyond syntactic correctness

Valid JSON can carry an invalid decision. {"decision":"approve","evidence_ids":[]} parses and may satisfy field types, yet it violates a policy requiring evidence for approval.

Apply three validation layers

The offline foundations lab records three authored payload states.

from foundation_cases import json_semantics_case

result = json_semantics_case()
assert result["accepted"] == 1
assert result["accepted_indices"] == [1]
print(result["rows"])

The first row passes syntax and schema but fails the business rule. The second is a valid abstention without evidence. The third is malformed. Only index 1 is accepted. These are deterministic fixtures, not generated responses.

Validate in order

1. 1Syntax: parse JSON and reject trailing prose or malformed encoding according to the interface contract.
2. 2Schema: enforce required fields, types, enum values, lengths, bounds and additional-property policy.
3. 3Semantics: check cross-field rules, evidence IDs, authorization, timestamps, totals and task-specific invariants.

Then validate external references. If the output cites document P-17, confirm that P-17 was in the authorized retrieval set and supports the stated field. If an extracted total should equal line items, recalculate it independently.

Control repair and failure

A parser error may enter one bounded repair path using the validation message. A semantic failure should not be disguised by repeatedly asking the model until a response passes. Record attempt count and final disposition. Route unsupported or high-impact cases to abstention or human review.

Use property-based or table-driven tests around empty lists, duplicate IDs, negative quantities, future timestamps, incompatible units and enum changes. Track validation failures by rule and model/prompt version; a falling parse-error rate can coexist with worsening business correctness.

The Generative & Agentic AI course treats validated structured output as an integration boundary, especially before agent tools and automation.

Exercise

Create 20 payloads for one task: five malformed, five schema-invalid, five semantically invalid and five valid. Run all layers, save reason codes and prove that the application accepts only the intended five.

Continue learning

This article is part of the LLM fundamentals and prompt design sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Write a structured-output contract for an LLM task.
• Continue with Few-shot examples: choose cases that expose the boundary.

Reference: Pydantic model validation.