01 · the questiontraining only
Can a training signal see deployment?
Generalization measures try to predict a model’s unseen train-to-test gap without using labels from the future test environment. Sharpness, margins, norms, gradient statistics, calibration, and information criteria each offer a different gauge.
The useful version of the question is not “does this metric correlate once?” It is: does its ranking survive the conditions in which the model will actually run?
trained model + training data → predicted unseen gap
02 · familiar dataIID
IID can make a gauge look universal.
On data drawn like the training distribution, model rankings can settle into a clean diagonal: a proxy orders configurations in much the same way as their measured generalization gaps.
The benchmark expands that test to more than 40 measures across more than 10,000 configurations. The diagonal is a useful baseline—but it is not yet evidence that the gauge is robust.
rank agreement in one regime ≠ universal validity
03 · move the worldOOD
Shift the distribution; the ranking can flip.
Under synthetic corruptions and natural domain shifts, configurations trade places. Some associations weaken; others reverse across architecture or training regime. Sharpness is the clearest example: its familiar relationship in scratch-trained CIFAR experiments reverses in the DomainBed fine-tuning setting.
That reversal is an observed association, not a causal mechanism. It shows why a training-time proxy must be evaluated under the shift it is supposed to anticipate.
IID association ≠ OOD guarantee
04 · the decisionseveral signals
A regime-aware answer is plural.
The result is a map of failure modes, not one winning number. Capacity, margin, sharpness, optimization, information, and calibration measures move differently as the regime changes.
For model selection, the practical move is to combine distinct signals and validate them across deployment-relevant shifts—then treat the remaining score as evidence, not a guarantee.
selection = combine signals + validate under shift
What the study establishes
A stress test for the gauges we use before deployment.
40+Generalization measures spanning norm, margin, sharpness, optimization, information, output, and calibration families.
10K+Training configurations across architectures, hyperparameters, and training recipes rather than a single model snapshot.
IID → OODControlled shifts include CIFAR corruptions and natural domains in PACS and VLCS.
NO UNIVERSAL PROXYRank stability is conditional. A useful measure in one regime may be uninformative—or point the other way—in another.
Scope. These are empirical correlations, not causal guarantees. The main experiments use small-to-medium vision models; the DistilGPT2 appendix is IID-only and single-architecture. Robustness of the same relationships in modern large transformers remains open.
Read the manuscript
Paper preview
TMLR manuscript · revised 2026visual preview
Google Scholar currently indexes the February arXiv version under its earlier title, Revisiting Generalization Measures Beyond IID. This page follows the revised manuscript title and links both records.