Sequence models: distinguish padding from real observations

Batches often require equal sequence lengths, so shorter examples receive padding. If zero is also a legitimate observation or token, its numeric value cannot tell the model which positions are real. Supply sequence lengths or a boolean mask and carry it through attention, recurrence, pooling and loss.

One sequence, two padded lengths

The local lab starts with the real sequence [2, 0, 4]. The middle zero is an observation. It pads the sequence to lengths five and eight with more zeros.

from deep_learning_cases import padding_case

result = padding_case()
assert result["real_zero_observations"] == 1
assert len(set(result["masked_means"].values())) == 1
print(result["unmasked_means"], result["masked_means"])

The unmasked mean changes from 1.2 at padded length five to 0.75 at length eight. The masked mean remains 2.0 because only the three real positions contribute. A prediction should not change solely because another example in the batch required more padding.

Carry the mask end to end

Create masks from trusted lengths before batching. Verify shape and convention: some APIs use True for valid positions while others use it for positions to ignore. Apply the mask before average or attention normalization and exclude padding targets from token-level loss.

For embeddings, a padding index can keep the padding vector fixed, but it does not automatically mask every later operation. Packed-sequence utilities can skip padded recurrent steps; attention needs its own key-padding or attention mask.

Test invariance

Take one example, append additional padding and assert that its prediction remains equal within tolerance. Test an all-padding invalid case and the shortest permitted real sequence. Include a real zero to ensure the implementation does not infer masks by comparing values to zero.

Padding bugs can correlate with sequence length and create misleading slice performance. Report metrics by real length. If truncation is used, record which side is removed and whether critical information tends to appear there.

The Data Science course connects tensor shapes and masks to reproducible sequence-model behavior.

Exercise

Batch three sequences with real zeros and different lengths. Implement masked sum, mean and attention. Append padding to every sequence and write prediction-invariance tests for each operation.

Continue learning

This article is part of the Deep learning and computer vision sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Inspect a model failure without overstating saliency maps.
• Continue with RNNs and LSTMs through a sequence prediction task.

References: PyTorch packed-sequence utility and Transformer key-padding masks.