Run an open model locally with explicit resource limits

Local inference removes a hosted call, but it does not remove capacity, licensing, privacy or safety responsibilities. Admission control should reject work that cannot fit within known bounds before memory is exhausted.

Test an admission policy

The adaptation lab checks three synthetic requests against RAM, input, output and concurrency limits.

from adaptation_cases import local_limits_case

result = local_limits_case()
assert result["limits"]["max_ram_gb"] == 12
assert result["allowed_ids"] == ["L1"]
assert result["decisions"][1]["allowed"] is False
assert result["decisions"][2]["allowed"] is False
assert result["model_loaded"] is False

The logic denies an over-memory estimate and oversized input. No model was downloaded or loaded.

Specify the execution envelope

Record checkpoint and revision, license review, quantization, runtime, CPU/GPU/accelerator, available RAM/VRAM, storage, context cap, output cap, batch/concurrency, thread count and deadline. Bind the service to an intended interface and authenticate it if other users can reach it.

Measure actual cold load, peak memory and throughput with representative lengths. Keep headroom for the operating system and concurrent work. Enforce request size before tokenization where possible, propagate cancellation and kill work after the total deadline. Queue boundedly; an unlimited queue only moves the outage.

Treat prompts, files, cache and traces as sensitive. Define storage and deletion, disable unintended outbound access and scan model/runtime dependencies. Validate outputs and tool authority exactly as with a hosted model.

The Generative & Agentic AI course connects local execution to measured quantization and operational runbooks.

Exercise

For a small permitted checkpoint, create a launch manifest and preflight command. Test one accepted and three rejected requests, then capture measured peak memory without publishing unsupported performance generalizations.

Continue learning

This article is part of the Model adaptation and multimodal tasks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Choose a quantization level using measured task quality.
• Continue with Document model licensing before using a checkpoint.

Reference: Hugging Face Transformers quantization overview.