Five Leaks That Inflate Your AUC: I Measured Each One

The best-looking model in the notebook is not always the best model in production. Sometimes the gap is ordinary overfitting. Often it is leakage: information the model will never have at prediction time finding its way into training or evaluation. The score goes up, the deployment falls over.

This is not a rare or academic problem. Kapoor and Narayanan surveyed the machine-learning literature and documented leakage in 294 papers across 17 scientific fields, from bioinformatics to radiology. When they reproduced a well-cited claim that complex models dramatically outperform decades-old logistic regression at predicting civil wars, the advantage disappeared once the leakage was fixed. The models did not fail because they were badly built. They failed because they were graded on data they had effectively seen.

I wanted magnitudes, not just warnings. Tutorials list the leak types, but they rarely say which ones actually hurt. So I built five small experiments with scikit-learn, one per leak, each comparing a leaky evaluation with the honest version on the same underlying data. The table is the short version. The sections below are the recipes, so you can check your own pipelines the same way.

Leak	Honest AUC	Leaky AUC	Fake gain
Target leakage	0.833	0.989	+0.156
Feature selection before the split	0.534	0.798	+0.264
Repeat groups, random split	0.676	0.823	+0.147
Centred time window	0.881	0.945	+0.064
Duplicate rows across the split	0.923	0.966	+0.042

Leak 1: Target leakage, the answer in disguise

Setup: 8,000 rows, a logistic regression on 10 features. The honest AUC was 0.833 on a held-out 2,000 rows. Then I added one column: the label plus a small amount of noise. It stands in for the classic post-outcome fields, a days-overdue column on a default model, a cancellation-reason column on churn, a discharge note on readmission. The leaky AUC was 0.989.

Nothing about the model changed. The new feature is the answer, lightly disguised. That is exactly why target leakage keeps surviving reviews: the feature looks like a strong signal, and in a notebook nothing distinguishes a strong signal from a leaked one.

Why production kills it: in live use the column is empty, or it gets written after the moment you need the prediction. The model's best feature does not exist when it matters.

The detection habit: rank features by single-feature AUC. A column far stronger than everything else is a suspect until proven otherwise. Then ask the process question: when exactly is this value written, and by whom? If the answer is after the outcome you are predicting, drop it.

Leak 2: Feature selection before the split

This was the largest fake gain, and the setup was deliberately harsh: 500 rows, 2,000 features of pure noise, labels of pure noise. The true AUC is 0.50 by construction. Selecting the top 20 features by F-test on all 500 rows, then splitting 60/40, produced a test AUC of 0.798 on 200 held-out rows. Selecting the features inside the training data only gave 0.534, at chance as it should be.

Selection is fitting. Choosing features by their relationship with the label is learning, and learning on the test set is what the test set exists to prevent. With many features and few rows, picking 20 winners from 2,000 on noise alone still finds columns that correlate with the test labels by luck. The model then gets graded on luck it shares.

In real work the effect shrinks when genuine signal exists and features are few relative to rows. But the mechanism is always there: anything that touches the label, feature selection, target encoding, resampling like SMOTE, belongs inside the cross-validation loop, not in front of it.

The fix is mechanical. Put selection inside a pipeline and let the cross-validator refit it per fold:

from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
 
model = Pipeline([
    ("select", SelectKBest(f_classif, k=20)),
    ("clf", LogisticRegression(max_iter=1000)),
])
scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")

Fitting the pipeline inside each fold guarantees the validation fold never influenced the selection. scikit-learn's pipeline docs make this the recommended pattern precisely because the failure is silent: the code runs without errors and produces flattering numbers.

Leak 3: Repeat groups split at random

Setup: 500 groups (think customers), 16 rows each, 8,000 rows total. Every group has its own latent risk level, and rows within a group sit close together in feature space. A random forest trained on a random 75/25 split scored 0.823. Splitting by group, so no customer appears on both sides, scored 0.676. The fake gain was 0.147.

This is the leak that bites hospital readmission models (multiple visits per patient), fraud models (multiple transactions per account), and imaging work (multiple scans per specimen). Near-twin rows let the model memorize group fingerprints. The random forest does not need the group ID as a feature; tight clusters in feature space do the job.

The fix is to split by the unit the model will meet in production. If production means new customers, the evaluation must use new customers:

from sklearn.model_selection import GroupShuffleSplit
 
splitter = GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=42)
train_idx, test_idx = next(splitter.split(X, y, groups=customer_id))

A fair question to ask in review: are the test rows from entities the training set has already seen? If yes, and production brings unseen entities, the score is a fiction.

Leak 4: A centred time window

Setup: a latent daily state with strong autocorrelation (an AR(1) process with rho 0.9), 9,996 days, predicting each day's event from the previous two days' readings. On a chronological final-quarter holdout of 2,499 days, the past-only features scored 0.881. Adding one centred 3-day rolling mean, which quietly uses readings from the current day and the next, raised the score to 0.945.

The centred window is the most polite leak in this list. rolling(center=True) reads beautifully in a notebook, interpolation across a gap does the obvious thing, and full-period aggregates look like clean feature engineering. But at prediction time, tomorrow's reading does not exist yet.

The fix has two parts. Feature windows should trail, never straddle, the prediction point. And on time-ordered data, random splits are the wrong evaluation: scikit-learn's cross-validation docs explicitly warn that shuffling time series gives poor estimates of generalization error, and provide TimeSeriesSplit as the alternative. In this experiment both evaluations used the same chronological holdout, so the 0.064 gap is purely the price of the window itself.

Leak 5: Duplicate rows across the split

Setup: 6,000 records, with 2,400 duplicated once (an intake system double-recording 40% of applications), for 8,400 rows total. A random forest on a random split scored 0.966. Splitting grouped by the original record ID, so a record and its twin stay together, scored 0.923. The fake gain was 0.042, the smallest of the five.

That ordering surprised me. The duplicate-row warning is loud in every tutorial, and here it moved the least. Two honest caveats. First, the base signal in this test was strong, so the ceiling absorbed part of the effect. Second, exact duplicates are the easy version. Near-duplicates, the same form submitted twice with a typo fixed, survive exact deduping entirely and behave more like the group leakage in experiment 3. If anything, treat the 0.042 as a floor, not a ceiling.

The fix: dedupe on business keys before splitting, keep each row's origin ID, and split by origin. Kapoor and Narayanan's taxonomy lists duplicates spanning the split as one of the eight canonical leak types, and their survey found it in real published work, so the real-world rate of this mistake is nonzero.

The famous leak that did not make the list

Fitting a scaler on the full dataset before splitting is the textbook example of train-test contamination, and the scikit-learn docs correctly call it leakage, with a hard rule: never call fit on the test data. So I ran it on the same 8,000-row dataset as experiment 1. The honest and leaky AUCs were 0.8329 and 0.8329, identical to four decimal places.

The rule still stands, and the scaler still goes inside the pipeline, because the habit is what protects you against the leaks where the damage is real: feature selection and target encoding. But rank your audit by measured damage. A team that spends its review budget arguing about the scaler while a target encoder fits on the full table has mispriced the risks.

The six-question audit

Run this list on any tabular project before trusting its offline score:

1. When is each feature written? Compare every feature's timestamp with the prediction timestamp. Anything recorded after the outcome is target leakage until proven otherwise.
2. What is the split unit? Rows, groups, or time periods? Write the answer in the notebook. If production meets new groups or future time, the split must match.
3. Can any row appear twice? Check exact and near duplicates, then split by origin ID.
4. Did anything fit before the split? Scalers, imputers, selectors, encoders, resamplers. All of it goes inside the pipeline, where cross-validation refits it per fold.
5. Does any window look forward? Centred rolling means, interpolation, aggregates computed over periods that include the prediction point.
6. Is one feature suspiciously strong? Score each feature alone. An outlier usually means the label leaked into it.

The verdict

The five experiments share one moral. Every leak here is a violation of the same contract: the evaluation must only use information the model will have when it is actually used. The fake gains ranged from 0.04 to 0.26 AUC points, and the most dangerous one, target leakage, is the one no pipeline construct can catch for you. Pipelines prevent train-test contamination mechanically. Nothing mechanical tells you that a column was written after the outcome. That part requires asking where the data came from, row by row.

A useful side effect of the audit: when you remove a leak and the score drops, that drop is information. The honest score is the first score you can act on. Everything above it was a measurement of your own plumbing.

Key takeaways

1. Target leakage is the largest honest-signal killer: one post-outcome column moved AUC from 0.833 to 0.989 with nothing else changed.
2. Anything that uses the label, feature selection, target encoding, resampling, must run inside the cross-validation loop, inside a pipeline.
3. Split by the unit production will bring: new customers mean group splits, future time means chronological splits, not random ones.
4. Centred windows and interpolation quietly use future data; trailing windows only.
5. Rank leaks by measured damage, not by how often tutorials warn about them. The textbook scaler leak measured exactly zero here.

What would make me wrong

The fake gains above are magnitudes from synthetic setups, not constants. On your data the ordering can change: with a weak base signal, duplicates could outrank a centred window. If your production setting really does score new rows from known customers, a group split is too strict and the random-split score is the honest one, not the other way around. And if a replication with different seeds moves any gap by a point or two, that is expected noise, not a refutation.

The claim that would fail the whole piece: that leakage is a common, high-damage failure mode in applied work. If the literature survey evidence is thin and these mechanisms rarely appear in production pipelines, this audit is theatre. The documented 294-paper count and the civil-war reproduction suggest otherwise, but the direction of the evidence matters more than any number here.

Have you ever watched a score collapse on deployment? Which of the six questions would have caught it, and which one do you now trust least in your own work?

Related on Everyday Data Science

• Your 0.85 Meant 0.98: A 20,000-Row Calibration Test That Caught a Top Model Misstating Its Own Odds
• pandas vs Polars vs DuckDB on 3.5 Million Taxi Rides. DuckDB Won Every Query, and the Timings Were the Least Interesting Part.
• Five JOIN Traps That Quietly Rewrite Your Numbers. One Orders Table Reported $620, $1,240 and $2,040.

Sources: Kapoor and Narayanan, Leakage and the reproducibility crisis in machine-learning-based science, Patterns 2023 · scikit-learn, Common pitfalls and recommended practices · scikit-learn, Cross-validation: evaluating estimator performance