Load-test a prediction API with realistic payloads

A useful load test reflects payload sizes, validity mix, concurrency and arrival patterns expected in operation. It reports throughput, latency distribution and error codes under a defined environment. Calling a prediction function in a loop is a local benchmark, not an API capacity test.

Reconcile a local contract load

The deployment lab sends 300 sequential in-process requests: 95% valid and 5% with negative spend.

from deployment_cases import load_case

result = load_case()
assert result["requests"] == 300
assert sum(result["status_counts"].values()) == result["requests"]
assert result["status_counts"] == {200: 285, 422: 15}
print(round(result["median_ms"], 2), round(result["p95_ms"], 2))

The code prints live CPU timing. Values vary by machine and load. Status counts are the reproducible contract result. There is no HTTP stack, concurrency, network, container or shared dependency, so the lab makes no capacity claim.

Model the workload

Use observed or explicitly assumed request rates, payload distribution, batch sizes and bursts. Include invalid, maximum-size and unknown-category requests. Separate open-loop arrival tests from closed-loop clients that wait before sending another request.

Warm up the model and report cold start separately. Measure end-to-end p50, p95 and p99, throughput, timeout, error code, CPU, memory and dependency saturation. Maintain a unique request ID and verify response correctness under load.

Test failure and recovery

Slow or fail a dependency, restart instances and observe queue growth and recovery. Check rate limits and backpressure. Ensure load generators do not hit production or create side effects without explicit authorization.

The Data Science course connects model latency with the actual service path and release checklist.

Exercise

Build a local HTTP service in an isolated environment. Test steady, burst and invalid workloads at several concurrency levels and plot latency/error versus offered load. Keep this in-process result as the lowest-level baseline.

Continue learning

This article is part of the Model deployment and MLOps sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Dockerize a model service with a reproducible environment.
• Continue with Explain model latency percentiles to a product owner.

Reference: k6 load-testing documentation.