Measure inference latency with warmup and repeated trials

One timer reading is not a latency benchmark. First calls may initialize libraries or caches, operating-system scheduling adds noise, and accelerators may execute asynchronously. A useful report states hardware, software, batch, input shape, warmup, trial count and percentile.

A repeatable microbenchmark

The local lab measures its small NumPy convolutional forward pass on the executing CPU. It warms up eight calls and records 60 trials for batches 1 and 32.

from deep_learning_cases import latency_case

result = latency_case()
for batch, row in result["results"].items():
    assert row["warmups"] == 8 and row["trials"] == 60
    print(batch, round(row["median_ms"], 4), round(row["p95_ms"], 4))

The code prints live median and 95th-percentile milliseconds. The values are deliberately not copied into a portable claim because CPU load and machine state change. The saved verification artifact records the observed run for audit.

Match the deployment boundary

A model-only timer excludes image decode, resize, normalization, request parsing, network transfer and queueing. Measure both kernel latency and end-to-end service latency. Use realistic concurrency and input sizes. If requests arrive singly, a batch-32 throughput result does not describe user response time.

For accelerators, synchronize around timing calls. Record device, precision, thread settings, model mode and compiler state. Run enough trials to show median, p95 or p99 and distribution, not just the fastest result. Separate cold-start from warm latency.

Interpret batching carefully

Batching often reduces per-image cost while increasing the time an individual waits for a batch to fill. Report total batch latency, per-item throughput and queue policy. Memory limits can create nonlinear failures at larger batches.

Establish a service objective from user and system needs, then test on target hardware. Re-run after quantization or graph conversion and confirm prediction quality. A faster operation that changes outputs beyond tolerance is a different model.

The Data Science course connects local measurement protocols to deployment evidence and monitoring.

Exercise

Benchmark batch sizes 1, 8, 32 and 128 with model-only and full preprocessing timers. Run cold and warm trials, plot distributions, and choose a batching policy for a declared request-arrival rate.

Continue learning

This article is part of the Deep learning and computer vision sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Save and reload a neural model with equivalent predictions.
• Continue with Quantization: compare size, speed and prediction changes.

Reference: PyTorch benchmark utility documentation.