Evaluate OCR on difficult document layouts

OCR quality varies with columns, tables, rotation, scans, handwriting, fonts and image degradation. A clean-page average can conceal the layout that breaks the downstream workflow.

Calculate character error by layout

The adaptation lab computes edit distance for three authored reference/output pairs.

from adaptation_cases import ocr_layout_case

result = ocr_layout_case()
assert result["exact_matches"] == 1
assert result["worst_layout"] == "two_column"
assert result["rows"][0]["cer"] == 0.0
assert result["ocr_executed"] is False

The two-column fixture has the highest character error rate because reading order changes. No OCR engine processed an image.

Build a layout-labelled set

Sample the documents the application will receive and label scan quality, rotation, columns, tables, stamps, handwriting, language and template. Keep related pages from one document in the same split. Preserve a human transcription plus structural annotations such as cell, region and reading order.

Report character and word error with insertions, deletions and substitutions. Add exact identifiers, numeric fields, table-cell accuracy and reading-order metrics. Normalise whitespace or case only when the downstream task also ignores it. Show counts and results per layout; a small hard slice needs uncertainty and examples.

Separate OCR from downstream reasoning. A system may read text correctly but map it to the wrong field, or vice versa. Store page/region provenance and send low-quality or critical documents to a correction queue.

The Generative & Agentic AI course evaluates OCR as one measurable stage of multimodal RAG.

Exercise

Create five synthetic layouts from your own text, establish references and evaluate one permitted OCR engine. Publish layout counts, CER/WER and structural failures without generalizing beyond the sample.

Continue learning

This article is part of the Model adaptation and multimodal tasks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Vision-language extraction with field-level validation.
• Continue with Multimodal RAG: connect an image claim to its source.

Reference: NIST AI RMF Generative AI Profile.