Vision-language extraction with field-level validation

A document extraction can be mostly correct while its most important number is wrong. Validate fields by type and business consequence rather than accepting plausible JSON.

Inspect the failed field

The adaptation lab compares a synthetic invoice reference with an authored extraction.

from adaptation_cases import field_validation_case

result = field_validation_case()
assert result["correct"] == 3
assert result["total"] == 4
assert result["field_accuracy"] == 0.75
assert result["failed_fields"] == ["total"]
assert result["model_executed"] is False

The total loses a zero. A 75% field score would be unsafe if downstream payment used that value. No vision-language model ran.

Define the extraction contract

Specify fields, types, units, formats, null behaviour, page/region provenance and confidence or review rules. Create human references under a rubric and adjudicate disagreements. Split documents by source/template so near-identical pages do not leak into evaluation.

Report exact and normalized accuracy per field, missing/spurious fields and critical-value error. Numeric tolerances must reflect the use case; do not apply a broad fuzzy match to invoice totals or identifiers. Validate cross-field rules such as subtotal plus tax equals total.

Retain source pointers so a reviewer sees the crop and page beside the proposed value. Never auto-accept solely because the model emits high confidence. Log corrections with provenance and keep them outside training until reviewed and split safely.

The Generative & Agentic AI course connects multimodal extraction to schemas, human review and evidence.

Exercise

Label ten varied local documents with four fields, inject formatting and numeric errors, calculate per-field metrics and implement a hard review rule for one critical field.

Continue learning

This article is part of the Model adaptation and multimodal tasks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Compare prompt caching and fine-tuning from workload economics.
• Continue with Evaluate OCR on difficult document layouts.

Reference: NIST AI RMF Generative AI Profile.