Handle long documents with explicit truncation rules

Text models have input limits. Silently taking the first tokens assumes the beginning contains the evidence. In support threads, the newest update or resolution may appear at the end. Define a truncation policy and measure what it removes.

Put the signal at the tail

The NLP lab creates 40 documents with 80 filler tokens followed by either duplicate charge or password locked. The maximum input is 40 tokens.

from nlp_cases import truncation_case

result = truncation_case()
assert result["max_tokens"] == 40
assert result["signal_visible_rate"]["head_only"] == 0.0
assert result["signal_visible_rate"]["head_tail"] == 1.0
print(result["signal_visible_rate"])

Head-only truncation preserves none of the authored signals. Keeping 20 tokens from the head and 20 from the tail preserves all of them. This demonstrates one constructed layout; it does not prove head-plus-tail is optimal for every document.

Measure position in real data

Calculate token-length distribution and locate annotated evidence. Compare head, tail, head-plus-tail, sliding windows and section-aware extraction. Report the fraction truncated and signal coverage by document type and language.

Chunking creates another aggregation problem. A document label may depend on one chunk, several chunks or their order. Define how chunk scores combine and split by document before chunking so sibling chunks cannot cross partitions.

Make the policy visible

Store original length, kept ranges and truncation flag with predictions. Test exact boundary lengths, empty sections and very long inputs. Version the tokenizer because token counts can change across models.

If summaries are used, evaluate whether they retain task evidence and introduce unsupported text. A generated summary is another model output, not a lossless compression.

The Data Science course connects sequence limits to reproducible preprocessing and slice evaluation.

Exercise

Annotate evidence spans in 100 long documents. Compare four truncation or chunking rules on evidence coverage, model metric, latency and number of windows. Freeze the rule before final test.

Continue learning

This article is part of the NLP and text analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Classify multilingual tickets without hiding language imbalance.
• Continue with Fine-tune a text classifier with a held-out evaluation set.

Reference: Hugging Face padding and truncation documentation.