Detect drift in incoming support language

Support language changes with products, payment methods, campaigns and channels. Input drift can warn that a text model sees unfamiliar language, but it does not prove predictions have degraded. Pair drift indicators with delayed labels, reviewed samples and performance monitoring.

An authored vocabulary shift

The local NLP lab uses 80 reference messages about passwords, login, refunds and invoices. The current window adds terms such as OTP, UPI, WhatsApp and dark store.

from nlp_cases import drift_case

result = drift_case()
assert result["reference_rows"] == result["current_rows"] == 80
assert result["current_oov_token_rate"] > 0
print(result["current_oov_token_rate"], result["jensen_shannon_on_known_terms"])

The current out-of-vocabulary token rate is 84.6% under the reference word vocabulary. Jensen-Shannon divergence over known terms is 0.454. The high OOV rate is intentionally created and indicates a changed lexical population; it is not a live-system alert.

Monitor several views

Track token or subword distribution, document length, language/script, predicted class, confidence and abstention. Slice by channel and product so mix changes do not masquerade as within-segment drift. Compare like-for-like time windows and retain counts.

Character or subword models may have few literal unknown tokens while meanings still change. Embedding-distance drift can help, but it also requires a stable reference and threshold. Review examples from the changed region.

Link drift to response

Define alert threshold, persistence window, owner and action. An alert might trigger label sampling, vocabulary review or temporary expansion of human review. Retraining automatically on unlabelled drift can reinforce errors.

When outcomes arrive, measure class metrics and calibration by time. Distinguish data-pipeline breakage from real user-language change. Preserve model and tokenizer versions.

The Data Science course places drift within a monitoring and retraining runbook.

Exercise

Create gradual and sudden drift scenarios. Set thresholds on historical validation windows, measure false alerts and detection delay, then attach a labelled-performance confirmation step.

Continue learning

This article is part of the NLP and text analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build a review-mining project that produces actionable themes.
• Continue with Privacy-aware text preprocessing for a portfolio dataset.

Reference: scikit-learn text vectorization documentation.