Temperature scaling and calibration

Maximum-margin methods squeeze scores toward the middle, and a sigmoid undoes it. Boosted-tree scores have almost no mass near 0 or 1 and the reliability curve is a sigmoid on seven of eight problems; boosted stumps are the most extreme, and SVMs behave alike. The companion paper explains this through margins. To widen the margin of hard cases near the boundary, boosting gives up margin on easy ones, so scores pile up on either side of it. Friedman et al.'s reading of AdaBoost as fitting half the log-odds makes a sigmoid of the normalized vote $f=\sum_t\alpha_th_t/\sum_t\alpha_t$ the right family (my derivation):$$\operatorname{logit}P(y=1\mid x)\approx2F=2\Big(\sum_t\alpha_t\Big)(2f-1),$$affine in $f$, leaving a slope and an offset to fit. The slope and offset the theory implies fail for boosted full trees, which separate the training set and inflate $\sum_t\alpha_t$, so the two parameters are fit on held-out data.

Naive Bayes pushes scores to the extremes, averaging keeps them off the edges, and log-loss learners are already calibrated. Naive Bayes adds one log-likelihood ratio per attribute, so correlated attributes count the same evidence several times and the log-odds overshoot; the correction it needs bends the other way from a sigmoid in $f$, so isotonic regression fits it better; a sigmoid in $\log f$ and $\log(1-f)$ can bend either way, which beta calibration later added (Kull et al., 2017). Random forests show a mild sigmoid on five problems because an average reaches 0 only if every tree says 0, so near the edges any noise moves it inward; bagged trees stay close to calibrated. A neural net with a sigmoid output trained on log loss is in effect Platt-scaled during training, so in 2005 it sat on the diagonal with logistic regression.

Platt wins on small calibration sets, isotonic on large ones, and calibrating a calibrated model hurts. The tradeoff is bias against variance in the map. Two parameters with smoothed targets cannot overfit much, while an isotonic fit has as many steps as the data allow. Platt is better for every learner plotted below roughly 200 to 1,000 calibration cases, and isotonic is as good or better from 1,000 on. The paper counts cases, but an isotonic step is determined by the positives in it (my reading): at 3% positives a set of 1,000 holds about 30, and at fraud-like rates of 0.1% it holds one, so the calibration set has to be sized in positives. On already calibrated models, Platt scaling moves mass away from 0 and 1, because a sigmoid of a $[0,1]$ score is never the identity; fitting it on the log-odds instead puts the identity at $A=-1,B=0$. After calibration, boosted trees give the best probabilities of the ten learners, and in the companion paper boosting full trees directly on log loss does badly, pushing predictions toward 0 and 1.

Modern nets are overconfident, and the settings that maximize accuracy are not the ones that calibrate best. Most uncalibrated 2017 models have ECE between 4% and 10%, up to 16.5% on CIFAR-100. There LeNet's average confidence is close to its accuracy, while ResNet-110 sits well above it. On CIFAR-100, ECE grows with depth and width while error falls, batch normalization worsens calibration while slightly improving accuracy, and calibration keeps improving as weight decay rises well past the value that minimizes error. These are single-dataset correlations with no causal claim, as the paper says.

NLL overfits while 0/1 error does not, and this produces the overconfidence. Training a ResNet-110 on CIFAR-100, test NLL rises after the learning rate is lowered while test error keeps falling. Once nearly every training point is classified correctly, the loss can still fall by scaling the logits up, which raises confidence everywhere, including on test points the net gets wrong, and the optimal temperature rises steadily through training. So high-capacity models that seem immune to overfitting do overfit, in log loss. For a linear model on separable data, gradient descent grows the weight norm like $\log t$ while the direction converges (Soudry et al.), and a growing norm at a fixed direction is exactly a falling temperature.

Temperature scaling fits that one degree of freedom, and on benchmarks the simplest method wins. Dividing by a positive scalar keeps the argmax, so accuracy is untouched, and the NLL is convex in $1/T$. Its first-order condition is a moment match, the maximum-entropy characterization the paper derives:$$\sum_{i=1}^nz_i^{(y_i)}=\sum_{i=1}^n\sum_{k=1}^Kz_i^{(k)}\,q_i^{(k)},$$the average logit of the true class equals its expectation under the model. That one parameter suffices is an empirical finding: vector scaling, free to scale each class, lands on nearly constant entries. On CIFAR-100 ResNet-110, ECE is 16.53% uncalibrated, 2.66% with histogram binning, 4.99% isotonic, 1.26% with temperature scaling and 25.49% with matrix scaling, whose $K^2$ parameters overfit with many classes. The one-against-rest binning methods can change the argmax after renormalization: on ImageNet DenseNet-161, error goes from 22.57% to 48.32% with histogram binning. On text temperature scaling is comparable but not always best.