Cross-validation with repeated customers or patients

When several rows belong to the same entity, random row splitting can put that entity in both training and validation. If the intended claim concerns previously unseen entities, hold out the entity as a group.

This article uses fictional customer snapshots, not patient records or a clinical model. The grouping principle also matters for repeated measurements in other domains, but a customer exercise does not establish clinical validity or an appropriate medical evaluation protocol.

Count entities as well as rows

The January portion of our original lab contains 160 rows: four observations for each of forty customers. A five-fold random row split creates held-out rows, but many belong to customers already represented in the corresponding training fold.

With the declared shuffle seed, the five folds share 25, 25, 23, 22 and 23 customer identities across their training and held-out portions. Those are observed overlap counts, not a measurement of score inflation.

Removing the identifier from the model does not eliminate this structure. Other features may preserve an entity's stable characteristics, so the evaluation population still needs an explicit definition.

Verify group separation directly

from sklearn.model_selection import KFold,GroupKFold
from evaluation_core import load,TARGET

data = load()
train = data[data['split']=='train'].reset_index(drop=True)
random_overlap=[]
for fit,held in KFold(n_splits=5,shuffle=True,random_state=20261005).split(train):
    random_overlap.append(len(set(train.iloc[fit]['customer_id']) & set(train.iloc[held]['customer_id'])))
assert random_overlap==[25,25,23,22,23]
for fit,held in GroupKFold(n_splits=5).split(train,train[TARGET],train['customer_id']):
    assert len(fit)==128 and len(held)==32
    assert not (set(train.iloc[fit]['customer_id']) & set(train.iloc[held]['customer_id']))
print({'row_fold_shared_customers':random_overlap,'group_fold_shared_customers':0})

Run from the evaluation lab. Each group-held-out fold contains eight whole customers and 32 rows. Scikit-learn GroupKFold keeps a group's observations together; the supplied groups must represent the correct entity boundary.

Choose the highest relevant dependency boundary

If one person has multiple account IDs, grouping by account may still allow person-level overlap. If several sites share a common operational process, holding out individual records may not assess transfer to a new site.

Define the entity-resolution policy before splitting. Otherwise two labels for the same entity can cross a boundary while an identifier intersection check reports no overlap. Retain an audit of the grouping key and known limitations without exposing unnecessary personal data.

Group sizes can be very unequal. Check fold row counts, class support and meaningful subgroup composition rather than assuming five folds provide five equally informative measurements. A fold with only one class also makes some classification metrics undefined.

Keep preprocessing inside each training fold

Imputation, scaling, feature selection and other learned transformations must be fitted separately within each fold's training portion. Fitting them on all rows before group splitting leaks information across the boundary you just created.

Group separation also does not impose time order. The January group folds in this example contain the same date range on both sides. To assess future behavior for unseen entities, design a split that enforces both requirements.

Exercise: create a customer with twenty observations while the others retain four. Inspect fold sizes and class counts. Explain whether you will average losses by row or give each customer equal weight, and connect that choice to the intended deployment population.

NeuraPath's Data Science course connects cross-validation with data structure. A complete result reports the grouping key, overlap checks, preprocessing scope and the population represented by the held-out entities.

Continue learning

This article is part of the Machine learning workflow and evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Choose a data split that matches the deployment setting.
• Continue with Time-aware validation for a changing business process.