79% Accurate, Give or Take 8 Points: What a Bootstrap Interval Says About Your Test Score

The number everyone quotes, and the number nobody asks for

Model comparisons in notebooks usually end with a table of point estimates: model A 79.1%, model B 78.4%, pick A. The table never says how much of that 0.7 point gap is the model and how much is the luck of which rows landed in the test set. On a small test set, the answer is often: most of it is luck.

This is easy to state and easy to ignore, because computing the uncertainty feels like extra work. It is not. The bootstrap, resampling your test predictions with replacement and recomputing the metric thousands of times, turns one test score into a distribution of plausible scores. The percentile interval from that distribution is the honest version of the number you were going to report.

I measured it on a setup simple enough to check by hand. The point is not the model. The point is the width.

Setup, so you can trust or dismiss the numbers

The data are synthetic, two features, labels drawn from a logistic rule with true weights 2.0 and -1.0. One logistic regression was trained on 10,000 rows with plain NumPy gradient descent (no scikit-learn in this environment; fitted weights came out 1.947 and -1.030, bias 0.018, close to the truth). A held-out pool of 20,000 rows from the same distribution served as the source of test sets. On the full 20,000-row pool the model scores 79.26%.

For each test size (100, 400, 1,600), I took the first n rows of the pool, then bootstrapped: 2,000 resamples with replacement, accuracy recomputed each time, interval taken as the 2.5th and 97.5th percentiles. Seed 42 throughout. The script is archived with the article data, and every number below comes from its output, not from a formula.

One honest caveat up front: this bootstrap measures sampling noise in the test set only. It says nothing about training variance (retrain on different data, weights move, scores move) and nothing about whether the test set resembles production data at all. A biased test set bootstrapped 2,000 times gives a narrow interval around the wrong number. The interval quantifies one uncertainty, not all of them.

Finding 1: quadrupling the test set halves the interval, twice

Test size	Point estimate	95% interval	Width
100	75.0%	[66.0%, 83.0%]	17.0 points
400	78.8%	[74.8%, 82.8%]	8.0 points
1,600	79.6%	[77.6%, 81.6%]	4.0 points

The pattern is the square root law doing its job in public. Each fourfold increase in test rows halves the width: 17 to 8 to 4. That scaling is the practical rule. If your interval is twice as wide as you can tolerate, you need roughly four times the test data, not twice.

Note what the point estimates do across the rows. At n=100 the model scores 75.0%, a full 4.3 points below its 20,000-row score of 79.26%. That is not a worse model. It is the same model meeting an unlucky first 100 rows. Anyone comparing two models on a 100-row test set and celebrating a 3-point lead is celebrating a draw.

Finding 2: fresh test sets confirm the bootstrap, almost exactly

Bootstrap intervals are sometimes dismissed as a resampling trick that invents information. So I checked them against reality: 20 freshly drawn test sets of 400 rows each, same model, no resampling involved. Point estimates ranged from 73.8% to 82.3%, a spread of 8.5 points. The bootstrap interval at n=400 was 8.0 points wide.

The agreement matters. It means the bootstrap is not manufacturing width. It is recovering, from a single test set, the spread you would see if you could afford to draw 20 test sets. When you only have one test set, which is the usual case, the bootstrap is the closest thing to that second, third, and twentieth draw.

Finding 3: compare models on the same rows, not on overlapping intervals

A weaker rival, a threshold on the first feature alone, scores 75.12% on the full pool against the model's 79.26%, a gap of 4.14 points. At n=400 the paired difference (model minus rival, computed row by row on the same test rows) is 4.75 points with a 95% paired bootstrap interval of [1.0%, 8.25%]. At n=1,600 the paired gap is 4.94 points with interval [3.06%, 6.75%].

The paired interval excludes zero at both sizes, so the lead is real in both cases. The reason to pair is that the two models share most of their errors: rows that are hard for one tend to be hard for the other, and that shared difficulty cancels in the difference. Comparing two separate intervals and eyeballing the overlap throws that cancellation away and makes real gaps look undecidable. Rule: same test rows, paired resamples, one interval for the difference.

The code

Ten lines, NumPy only. correct is a 0/1 array of per-row test correctness.

import numpy as np
 
rng = np.random.default_rng(42)
B = 2000
n = len(correct)
idx = rng.integers(0, n, size=(B, n))
boot_acc = correct[idx].mean(axis=1)
lo, hi = np.percentile(boot_acc, [2.5, 97.5])
print(f"accuracy {correct.mean():.3f}  95% CI [{lo:.3f}, {hi:.3f}]")

For a paired comparison, replace correct with the per-row difference (correct_a - correct_b) and read the interval for the difference the same way. If it excludes zero, the lead survives resampling. If it includes zero, say so in the report, even when the point estimates differ.

The verdict

Report the interval with the score, every time, in three cases especially: test sets under about 1,000 rows, model gaps under about 5 points, and any result that will drive a decision (ship, retrain, buy, publish). At n=400 a 4-point lead can be real, as the paired interval here shows, but a 1-point lead cannot be distinguished from noise, and the interval is what tells you which situation you are in.

The cost is trivial and the failure mode of skipping it is concrete: teams ship the model that won a coin flip, then spend weeks explaining why production disagrees with the notebook. Production was never the one that was wrong.

Key takeaways

1. At n=100, the 95% interval for accuracy spanned 17 points; at n=400, 8 points; at n=1,600, 4 points. Width halves with each fourfold increase in test rows.
2. Twenty fresh 400-row test sets ranged from 73.8% to 82.3%, matching the 8-point bootstrap width. The bootstrap recovers real spread from a single test set.
3. Compare models on the same rows with a paired bootstrap interval for the difference. Separate intervals with overlapping ranges hide real gaps.
4. A bootstrap interval measures test-set sampling noise only. It does not cover training variance or a test set that fails to represent production.
5. Under about 1,000 test rows, or for gaps under about 5 points, a point estimate alone is not a comparison. Attach the interval or do not claim a winner.

What would make me wrong

Three things. First, if the metric is not a simple average over rows (AUC, for example, is a pairwise statistic), the plain row bootstrap above is the wrong resampler and needs a paired or stratified variant, so the widths here do not transfer to AUC directly. Second, if test rows are clustered (multiple rows per patient, per account, per document), resampling rows instead of clusters understates the width, and the honest interval is wider than the one shown. Third, if the test distribution shifts in production, the interval is centered on the wrong target entirely, and no resampling fixes that.

What do you report alongside your test scores today: a point estimate, an interval, or nothing at all? And if you ran this bootstrap on your last comparison, did the winner survive it?

Related on Everyday Data Science: Drift Is Three Different Failures, and One Metric Cannot See Them All · Five Aggregation Traps That Rewrite Your Totals. I Measured Each One on Twelve Rows. · Five Leaks That Inflate Your AUC: I Measured Each One

Sources: scikit-learn, Cross-validation: evaluating estimator performance · scikit-learn, Common pitfalls and recommended practices · Efron and Tibshirani, An Introduction to the Bootstrap (1993), the percentile bootstrap method used above