Research Radar — 2026-10-28

Methodology Review: The bootstrap: simulate another sample, not more information

English · About 10 minutes

Main reading: original journal pages 1–4 inspected; related readings: publisher abstracts and bibliographic metadata. Toy calculations are independently derived. No omics benchmark, general coverage guarantee, or causal interpretation is claimed. Classic papers enter through the thematic path, not the news window.

The intuition

Suppose a reported mean changes when a different set of experimental units is measured. The computational question is how much that estimate might move across repeated samples when only one sample is available. A bootstrap creates a miniature sampling world from the observations and repeats the analysis inside it. It does not manufacture new experimental information. That distinction matters whenever a very smooth histogram from thousands of simulations looks more convincing than the small dataset that generated it.

The ordinary nonparametric bootstrap assigns equal probability to each observed unit, draws another sample of the same size with replacement, and recalculates the statistic. Repetition approximates a distribution of possible estimates under this empirical population. The approximation to the real sampling distribution is a separate statistical claim, not a consequence of running the simulation accurately. This plug-in perspective is the central idea in Efron’s construction. [R1]

A small worked example

Consider the deliberately tiny dataset consisting of two independent measurements, zero and two. Its observed mean is one. A bootstrap sample contains two draws, each independently choosing either observed value with probability one half. There are four equally likely ordered samples: zero with zero, zero with two, two with zero, and two with two. Their means are zero, one, one, and two. The bootstrap distribution of the mean therefore puts probabilities one quarter, one half, and one quarter on those three values. This is an exact enumeration, not a simulation.

The bootstrap mean is one. Its conditional variance is \(\frac{1}{4}(0-1)^2+\frac{1}{2}(1-1)^2+\frac{1}{4}(2-1)^2=\frac{1}{2}\), so the bootstrap standard error is \(\sqrt{1/2}\), approximately 0.707. Here conditional means that the observed pair stays fixed while only the resampling changes. The usual sample-variance estimate, dividing squared deviations by one rather than two, is two; dividing that estimate by the sample size gives an estimated mean variance of one. The discrepancy is a finite-sample feature, not an arithmetic mistake. With only two observations, replacing the population by the empirical distribution is a strong approximation.

Now compare two interventions on the calculation. Running one million resamples instead of one thousand makes the simulated frequencies more closely match one quarter, one half, and one quarter. It does not expand the empirical population beyond zero and two. Collecting a third real measurement does change that population and can change the estimated uncertainty itself. This contrast is a useful diagnostic whenever computation is cheap but additional donors, independent cultures, or experimental runs are expensive.

There is also a revealing deliberately invalid target. Ask for the population maximum rather than the mean. Every resampled value remains at or below two, so no bootstrap maximum can exceed the largest observed value. A real population could contain values above two that this sample missed. This support argument alone shows why a seemingly reasonable resampling recipe cannot automatically answer every statistical question. It is an original counterexample illustration here, not a claim that the three readings establish a universal theorem about every extreme-value estimator.

The necessary mathematics

Let \(x_1,\ldots,x_n\) be the observed sample, with \(n\) its size. Let \(F\) denote the unknown population distribution and \(\widehat F_n\) the empirical distribution assigning mass \(1/n\) to each observation. A statistic \(t\) maps a dataset to an estimate \(\widehat\theta=t(x_1,\ldots,x_n)\). For replicate \(b\), draw \(n\) observations independently from \(\widehat F_n\) and call the recalculated statistic \(\widehat\theta_b^*\). The star labels bootstrap randomness, not a new population parameter. The empirical distribution replaces the unknown distribution in the sampling experiment. [R1]

With \(B\) simulated replicates, define their mean as \(\overline\theta^*=B^{-1}\sum_{b=1}^{B}\widehat\theta_b^*\). A Monte Carlo estimate of the bootstrap standard error is

Here \(B\) is greater than one and \(b\) indexes replicates. This is a sample standard deviation across simulated estimates. It is not the standard deviation of the original observations. Nor is it the Monte Carlo uncertainty of the estimated standard error itself. The exact four-outcome calculation above bypasses simulation error entirely. [R2]

Assumptions and failure modes

The resampling unit encodes the assumed experiment. As a practical check proposed here, write its name before writing code. If the intended experiment repeatedly recruits donors, resampling individual cells while keeping donors fixed answers a different question. If observations are paired, resampling the two sides independently destroys the pairing. These are design-level warnings: the ordinary independent-draw construction cannot preserve dependence that it deliberately removes. A specialized dependent-data bootstrap needs its own justification; merely changing the random seed is not such a justification.

A second check is to specify what gets recomputed. If a reported estimator includes a normalization rule, feature selection step, or fitted threshold, freezing that step may target uncertainty conditional on the fitted choice rather than uncertainty of the complete procedure. Neither target is automatically wrong, but their interpretation differs. For a proposed implementation audit, compare a fully refitted pipeline with a deliberately frozen one on synthetic data whose sampling mechanism you control. Label the comparison as a diagnostic, not as a guarantee that either interval has correct coverage.

Standard errors and confidence intervals are different deliverables. Hall compares seven interval constructions and identifies important differences in their treatment of nonnormality and coverage. Consequently, taking arbitrary histogram quantiles is not a universally validated interval procedure. This column does not derive a bias-corrected or studentized interval, and the two-observation example is not offered as a trustworthy confidence interval. Choose an interval method with assumptions appropriate to the estimator and sample size, rather than treating the word bootstrap as sufficient calibration. [R3]

Verified applications

The main paper applies the construction to statistical estimation problems, including the variance of a sample median, discrimination error rates, ratios and regression parameters. Its inspected opening pages describe those examples; this review does not independently reproduce their numerical results. The 1986 companion broadens the practical discussion to accuracy measures and more complicated data structures. These are verified methodological applications, not evidence of a specific single-cell or wet-laboratory result. No omics benchmark is claimed for these readings. [R1] [R2]

Possible transfers — speculative

A possible omics transfer, proposed here rather than validated by these papers, is uncertainty auditing for a prespecified donor-level pathway-score contrast. Start from independent donor summaries and resample donors within the relevant study groups, retaining the intended paired or unpaired design. Decide in advance whether the target averages donors equally or weights their cells. Those targets can differ when donors contribute unequal numbers of cells. The bootstrap cannot resolve that scientific choice for you, and this proposal does not establish valid inference for arbitrary hierarchical data.

Before applying the idea to a real study, simulate several small datasets with known donor effects and repeated cells. Compare resampling donors with resampling cells, then inspect whether repeated-sampling intervals cover the known contrast. This is a suggested falsification exercise, not a validated workflow or a request to execute an analysis today. An apparently narrow interval from the wrong sampling unit should be a reason to inspect the design, not a reason to celebrate precision. Also keep biological confounding separate: resampling observational data does not turn an association into a causal effect.

What to try and what to read next

For a ten-minute exercise, enumerate the four toy samples, reproduce the variance one half, and explain why the familiar sample-variance calculation gives a different finite-sample estimate. Then name one way to reduce simulation noise and one way to gain actual information. Read the main paper’s construction before its applications, use the 1986 overview to separate accuracy targets, and consult Hall before choosing an interval rule. The stopping rule is simple: do not trust a resampling plot until you can state what was sampled, what was refitted, and which repeated experiment it is meant to imitate.

Reading list

• [R1] Primary reading: Bootstrap Methods: Another Look at the Jackknife. B. Efron (1979). Main reading: start with section 2, then section 3. January 1979 journal publication; the Helsinki-hosted PDF is a copy of that published article, not a different preprint.

• [R2] Related reading: Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy. B. Efron and R. Tibshirani (1986). Practical overview companion: compare estimating a standard error with constructing an interval. Evidence here is publisher abstract only, not full-text inspection. Published February 1986.

• [R3] Related reading: Theoretical Comparison of Bootstrap Confidence Intervals. Peter Hall (1988). Calibration companion: bootstrap interval rules are not interchangeable. Abstract-level evidence only; this review does not reproduce the higher-order asymptotic theory. Published September 1988.