Choose a quantization level using measured task quality

Fewer bits can reduce weight memory, but the best level depends on model, method, hardware and task. Choose from measured eligible candidates rather than assuming four-bit is always the answer.

Apply quality before memory

The adaptation lab compares authored 16-, 8- and 4-bit records under a 0.89 task threshold and 10 GB limit.

from adaptation_cases import quantization_choice_case

result = quantization_choice_case()
assert result["eligible_bits"] == [8]
assert result["selected_bits"] == 8
assert result["records"][2]["task_pass"] == 0.83
assert result["fixture_measurements"] is True

Eight-bit is the only eligible option. Four-bit uses less fixture memory but fails quality; 16-bit exceeds the limit. None of the values were measured here.

Benchmark the deployable artifact

Pin checkpoint/revision, quantizer and config, compute dtype, device, runtime, drivers and prompt settings. Measure peak memory, cold load, tokens per second, time to first token and p95 end-to-end latency. Include representative context and output lengths.

Run the same fixed evaluation for every candidate. Report exact-schema, reasoning, multilingual, retrieval/tool and critical safety slices. Quantization error can affect tasks unevenly, so an aggregate alone is insufficient. Repeat variable outputs or use deterministic decoding where suitable.

Test operational compatibility: serialization and reload, device placement, batching, adapter loading and fallback. Keep the full-precision or approved prior artifact available for comparison and rollback.

The Generative & Agentic AI course makes resource savings conditional on measured acceptance.

Exercise

Choose a small permitted model and evaluate two supported precisions on one machine. Publish the manifest, task IDs, quality slices, memory and latency; reject the smaller artifact if it crosses a critical threshold.

Continue learning

This article is part of the Model adaptation and multimodal tasks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Detect overfitting in a small instruction dataset.
• Continue with Run an open model locally with explicit resource limits.

Reference: Hugging Face Transformers quantization overview.