Batch size: compare throughput and validation behaviour

Batch size changes the gradient estimate, number of optimizer updates, memory use and hardware utilization. Comparing only epochs is misleading: on 360 training rows, 60 epochs create far more updates with batch 16 than with batch 256.

Count updates and measure locally

The neural lab holds the eight-unit network, learning rate, momentum, seed and 60-epoch budget constant.

The small batch gets more than eleven times as many updates as the large batch, so its lower loss cannot be attributed to batch noise alone. A fair compute-budget comparison might fix optimizer steps or elapsed resources rather than epochs.

from deep_learning_cases import batch_size_case

result = batch_size_case()
assert result["rows"] == 360
assert result["results"]["16"]["optimizer_steps"] == 1380
for size, row in result["results"].items():
    print(size, row["optimizer_steps"], round(row["elapsed_ms"], 2))

The elapsed values are measured by the executing CPU process and will change across runs and machines. They establish the measurement protocol, not a portable throughput claim.

Measure end-to-end work

Record examples per second, optimizer steps per second, peak memory and validation metric. Include data loading and augmentation when they are part of training. Warm up accelerators, synchronize before timing asynchronous work and repeat trials. State precision, device and software versions.

Large batches may increase hardware utilization but require learning-rate adjustment. That becomes a joint experiment. Small batches provide noisier gradients and more updates; they may help or hurt validation depending on the problem. Do not convert a heuristic into a rule.

Match the real constraint

If the constraint is a two-hour training window, compare candidates within two hours. If it is memory, find the largest safe batch and consider gradient accumulation, while verifying that optimizer semantics remain appropriate. For online inference, training batch results do not establish request latency.

The Data Science course links these measurements to reproducible experiment budgets and deployment decisions.

Exercise

Repeat the experiment with a fixed 1,000 optimizer steps for each batch. Record wall time, examples processed and validation loss across three seeds. Explain which budget matches the intended production retraining process.

Continue learning

This article is part of the Deep learning and computer vision sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Choose a learning rate using training evidence.
• Continue with Dropout and weight decay: different regularization mechanisms.

Reference: scikit-learn MLP parameters.