Speech transcription evaluation beyond one accuracy number

Word error rate is useful, but one corpus number can conceal poor performance for a language, accent, noise condition or critical identifier. Evaluate transcription in the context of the task it feeds.

Inspect a slice error

The adaptation lab calculates edit distance for two authored transcripts.

from adaptation_cases import transcription_case

result = transcription_case()
assert result["overall_wer"] == 0.125
assert result["worst_slice"] == "south_street"
assert result["critical_entity_correct"] == {"A1": True, "A2": True}
assert result["transcriber_executed"] is False

One of eight reference words is substituted, while both fixture critical entities remain correct. No audio or speech model ran.

Design the reference set

Sample consented audio across languages, accents, speaking rates, microphones, noise, overlapping speech and domain vocabulary. Keep speakers in one split to avoid voice leakage. Define transcription conventions for fillers, punctuation, numbers and code-switching before annotation.

Report insertions, deletions and substitutions, WER by slice, and confidence intervals where data supports them. Add named-entity or slot accuracy for account numbers, medicines, dates or commands. Measure downstream intent/task success and review burden. A low WER can still corrupt the one word that changes an action.

Test empty audio, long silence, unsupported language and adversarial background content. Protect biometric and personal data with access, retention and deletion controls; do not publish raw samples without rights.

The Generative & Agentic AI course connects speech metrics to multilingual evaluation and human correction.

Exercise

Record or synthesize a rights-cleared ten-clip set, label conditions, compare exact and normalized WER, and build a critical-entity gate that blocks one dangerous substitution.

Continue learning

This article is part of the Model adaptation and multimodal tasks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Multimodal RAG: connect an image claim to its source.
• Continue with Build a human correction loop for extracted documents.

Reference: NIST AI RMF Generative AI Profile.