Why Your Validation Split Leaks

A validation score is only useful if it predicts how the model will do on data it has never seen. When information from the training set reaches the validation set, the score goes up and the prediction gets worse. This is called leakage, and it is one of the most common reasons a model that looked excellent in development disappoints in use.

Leakage rarely comes from one obvious bug. It usually comes from a reasonable-looking choice about how the data was split. This post goes through the usual sources and a quick check for each.

What leakage looks like

The symptoms are familiar. Validation accuracy is close to training accuracy even for a large model. The score barely moves when we remove features that should matter. Or the model does well in validation and then drops sharply on the first batch of new data. None of these prove leakage, but each is a reason to look.

Duplicates and near-duplicates

Exact duplicates

Scraped and logged datasets often contain the same record more than once. If a random split puts one copy in training and another in validation, the model is being asked to recall rather than generalise. Hashing each example after normalising whitespace and case is a cheap first check.

Near-duplicates

Exact hashing misses images that differ by a resize, or documents that differ by a footer. For these we need a similarity measure: perceptual hashes for images, MinHash over word shingles for text.¹ We then treat each cluster of near-duplicates as one unit when splitting.

Grouped data

Many datasets have natural groups: several photos of the same patient, many sessions from the same user, several sentences from the same document. Examples within a group share information that has nothing to do with the label. If a group is spread across both sides of the split, the model can learn to recognise the group instead of the pattern.

The fix is to split by group rather than by example. Most libraries support this directly.

from sklearn.model_selection import GroupShuffleSplit

splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=0)
train_idx, val_idx = next(splitter.split(X, y, groups=patient_ids))

# Every patient appears on exactly one side.
assert set(patient_ids[train_idx]).isdisjoint(patient_ids[val_idx])

Time-ordered data

When the model will be used to predict the future, validation data must come from after the training data. A random split lets the model see tomorrow while predicting today, and features such as rolling averages quietly carry future values backwards. Splitting by a cut-off date, and computing every feature using only data available before that date, removes the problem.²

Preprocessing fitted on everything

Scaling, imputation, vocabulary building and feature selection all learn something from the data they see. If we fit them on the full dataset before splitting, the validation set has influenced the training pipeline. The effect is often small for scaling and large for feature selection, where choosing the features most correlated with the label on all the data is a direct leak.

The safe pattern is to split first and fit every preprocessing step on the training portion only. Wrapping the steps in a pipeline object makes this hard to get wrong.

A short checklist

• Hash examples and count duplicates across the split.
• Name the groups in the data and split by them.
• If the model predicts the future, split by time.
• Fit preprocessing after the split, on training data only.
• Keep one final test set that nobody looks at until the end.

None of these steps take long, and together they remove most of the ways a validation score can flatter a model.