Stemming versus lemmatization for a specific task

Stemming removes character suffixes using rules; lemmatization maps inflected forms to a dictionary base form, often using part of speech. Both reduce vocabulary, but both can merge distinctions or create awkward outputs. Choose them for a task and language, not as mandatory NLP cleaning.

Make mappings visible

The local lab applies an intentionally simple suffix stemmer and a small authored lemma dictionary to 11 forms.

from nlp_cases import stemming_case

result = stemming_case()
assert result["stem_vocabulary"] == 5
assert result["lemma_vocabulary"] == 3
print(result["stem_mapping"]["studies"], result["lemma_mapping"]["studies"])

The stemmer maps studies and studied to studi but studying to study, splitting forms that share a lemma. It maps charges, charging and charged to charg while leaving charge separate. The dictionary maps all four charge forms to charge.

This does not prove lemmatization is better. The dictionary is tiny and authored for these words. A production lemmatizer may need language, part-of-speech and domain handling.

Check lost distinctions

Normalization can merge words that matter differently. “Billing” as a business process and “bill” as a document may or may not be interchangeable. In sentiment, “better” and “good” have related meaning but different intensity. In entity extraction, modifying identifiers or names is dangerous.

Create a mapping audit: original token, normalized form, frequency, class distribution and representative sentences. Review high-frequency merges and collisions. Keep raw text for traceability.

Compare with no normalization

Word and character n-grams often perform well without stemming. Run no normalization, stemming and lemmatization with the same split and classifier. Report vocabulary, memory, validation metric and important errors. Do not use final test to choose.

For multilingual data, one English stemmer is not a multilingual strategy. Detect or route language where justified, preserve script and evaluate each language separately.

The Data Science course treats linguistic preprocessing as an evaluated, reversible choice.

Exercise

Build a 100-token collision report from a target corpus. Have a reviewer label helpful and harmful merges, then compare three pipelines on validation and error slices.

Continue learning

This article is part of the NLP and text analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Tokenization choices that change a text model.
• Continue with Prevent duplicate text leakage across data splits.

Reference: scikit-learn text feature extraction and preprocessing.