Classify multilingual tickets without hiding language imbalance

A multilingual classifier can inherit the volume of its largest language. An overall metric dominated by English says little about a smaller Hindi or Hinglish population. Record language counts and report each slice with its denominator.

A deliberately uneven fixture

The local NLP lab creates 96 short billing and access phrases: 60 English, 24 Hinglish and 12 Hindi. A character 2–5-gram logistic model receives a stratified split.

from nlp_cases import multilingual_case

result = multilingual_case()
assert sum(result["language_rows"].values()) == result["total_rows"] == 96
for language, row in result["test_by_language"].items():
    print(language, row["rows"], row["accuracy"])

The test contains 18 English, 7 Hinglish and only 4 Hindi rows. All three slices score 100% because phrases repeat simple class terms. Four Hindi examples cannot support a broad performance claim, and templated phrases do not represent code-switching, spelling variety or real support language.

Define language and script policy

Language detection itself can fail on short or mixed text. Preserve a mixed/unknown route rather than forcing every ticket into one language. Decide whether transliteration is normalized, retained or modeled separately. Keep emoji, product names and identifiers that carry task meaning.

Stratify and group by source. If translated copies of one ticket cross partitions, evaluation leaks content. Evaluate native text and machine-translated pipelines separately, including translation failure and latency.

Report uncertainty and action

For each language, show class counts, macro F1, confusion and confidence coverage. Set a minimum evaluation size before automation. Low-resource languages may remain review-only while more labelled data is gathered.

Involve fluent reviewers in label rules and error analysis. A model can classify an authored phrase correctly while missing politeness, negation or regional variation.

The Data Science course uses language slices to connect representation with accountable evaluation.

Exercise

Add natural spelling variants, mixed-script messages and multi-intent tickets. Double-label each language, use source-disjoint splits and define automation gates separately by language.

Continue learning

This article is part of the NLP and text analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Topic modelling: validate topics with human inspection.
• Continue with Handle long documents with explicit truncation rules.

Reference: scikit-learn classification report.