LoRA explained through trainable parameter scope

Low-Rank Adaptation freezes a pretrained weight matrix and learns a low-rank update. The useful first calculation is how many parameters the update introduces, not a claim that every workload becomes cheap or accurate.

Calculate one layer

For a 4,096 by 4,096 weight and rank 16, the adaptation lab compares full and adapter parameter counts.

from adaptation_cases import lora_scope_case

result = lora_scope_case()
assert result["full_parameters"] == 16_777_216
assert result["lora_parameters"] == 131_072
assert result["trainable_fraction"] == 0.0078125
assert result["base_frozen"] is True
assert result["model_instantiated"] is False

The low-rank update uses rank × (input + output) parameters, 0.78125% of this matrix. No checkpoint or GPU ran, so the calculation does not establish memory, speed or task quality.

Scope the real adapter

An implementation must choose target modules, rank, scaling, dropout, bias treatment and any modules saved in full. Count trainable parameters from the instantiated model and list every target; one-layer arithmetic cannot stand in for the architecture total.

Freezing base weights reduces optimizer state and gradients for those weights, but activations, adapters, temporary buffers and the base checkpoint still consume memory. Hardware, precision, sequence length, batch size, optimizer and checkpointing determine the measured peak.

Compare LoRA with a prompt/workflow baseline and the untouched base on identical splits. Evaluate regressions and unseen slices. Record base revision, adapter revision and whether weights were merged for serving.

The Generative & Agentic AI course connects adapter mechanics to experiment design and deployment evidence.

Exercise

Calculate parameter scope for ranks 4, 8, 16 and 32 across query and value projections. Then use a small local network to verify the framework’s reported trainable count matches your formula.

Continue learning

This article is part of the Model adaptation and multimodal tasks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Prepare instruction examples without duplicating evaluation data.
• Continue with QLoRA trade-offs: memory savings and evaluation requirements.

Reference: LoRA paper.