Logit adjustment

Theorem (consistency). For any $\delta\in\mathbb R^L_{>0}$, the pairwise margin loss with $\alpha_y=\delta_y/\pi_y$ and $\Delta_{yy'}=\log(\delta_{y'}/\delta_y)$ is Fisher consistent for the balanced error; $\delta=\pi$ is the logit-adjusted loss with $\tau=1$, and $\delta=1$ the balanced loss. Lemma (binary, if and only if). With temperature $\gamma$, $\ell(+1,f)=\frac{\omega_{+1}}\gamma\log(1+e^{\gamma\delta_{+1}}e^{-\gamma f})$ and $\ell(-1,f)=\frac{\omega_{-1}}\gamma\log(1+e^{\gamma\delta_{-1}}e^{\gamma f})$ are consistent for the balanced error exactly when $$\frac{\omega_{+1}\,\sigma(\gamma\delta_{+1})}{\omega_{-1}\,\sigma(\gamma\delta_{-1})}=\frac{1-\pi}\pi,\qquad\sigma(z)=\frac1{1+e^{-z}}.$$ Neither LDAM nor the equalization loss meets the condition, and post-hoc weight normalization is not consistent either.

The Bayes rule for the balanced error ignores the prior. Averaging the per-class errors with equal weights is the ordinary error under $P_{\mathrm{bal}}(x,y)=\frac1LP(x\mid y)$, whose Bayes classifier is $$\arg\max_yP(x\mid y)=\arg\max_y\frac{\eta_y(x)}{\pi_y}=\arg\max_y\big[s^*_y(x)-\log\pi_y\big]$$ when $\eta_y\propto e^{s^*_y}$. This decision does not move when the priors change with $P(x\mid y)$ fixed, which makes the balanced error the natural target under label shift. Two procedures follow: shift the logits of an ordinary model afterwards, or train a model whose logits already estimate $\log P_{\mathrm{bal}}(y\mid x)$. In the binary case the rule predicts positive when $\eta(x)>\pi$, Elkan's threshold $p^*$ with a false negative costing $(1-\pi)/\pi$ times a false positive (my remark; the paper leaves general costs to future work), and shifting the log-odds by $-\operatorname{logit}p^*$ handles any cost ratio.

Post-hoc adjustment is additive, which is why it beats weight normalization. With $\tau=1$, subtracting $\log\pi_y$ is the Bayes rule applied to the model's own probability estimates, and since $\arg\max_y[f_y/\tau-\log\pi_y]=\arg\max_y[f_y-\tau\log\pi_y]$, any other $\tau$ is the same rule after temperature scaling, the paper's reason for tuning $\tau$ on a miscalibrated net. Weight normalization predicts $\arg\max_yf_y(x)/\nu_y^\tau$, a multiplicative rule: when a rare class has a negative logit while some other class is positive, shrinking $\nu_y$ only pushes it further down, so the rare class can never win. The common choice $\nu_y=\|w_y\|_2$ also assumes the classifier norms track the priors, which on long-tailed CIFAR holds under SGD with momentum and fails under Adam at comparable balanced error, so the method rests on an accident of the optimizer's implicit bias.

The logit-adjusted loss learns the balanced posterior, by two changes of variable. A margin $\Delta_{yy'}=\log(\delta_{y'}/\delta_y)$ is the same as adding $\log\delta_y$ to logit $y$ inside the softmax, so the population minimizer satisfies $f_y+\log\delta_y=\log\eta_y+C(x)$. A class weight $\alpha_y$ turns the risk into unweighted cross-entropy on a distribution with posterior $\bar\eta_y\propto\eta_y\alpha_y$. Together, $$f^*_y(x)=\log\frac{\eta_y(x)\,\alpha_y}{\delta_y}+C(x)=\log\frac{\eta_y(x)}{\pi_y}+C(x)\quad\text{when }\alpha_y=\frac{\delta_y}{\pi_y},$$ whose argmax is the Bayes rule of Step 1. Only the ratio $\alpha_y/\delta_y$ matters, so a change in a weight can be offset one for one by a change in the margin, and a margin that depends on the positive label alone (LDAM) or on the negative alone (equalization) cannot produce the factor $1/\pi_y$. The loss is post-hoc adjustment of the scorer $f+\tau\log\pi$, so the two routes coincide for convex models and differ for nets only through the local minimum reached, and applying both at once double-counts the prior.

In the binary case, consistency is one equation at $f=0$. The binary pairwise loss is proper composite, so its Bayes score is $f^*=\Psi(\eta)$ for an invertible link $\Psi$, and consistency means $\Psi^{-1}(0)=\pi$. The ratio of the two loss derivatives fixes the link (Reid and Williamson), and at $f=0$ it gives the lemma's condition. Pure weighting $\omega_{\pm1}\propto1/\pi_{\pm1}$ satisfies it at every $\gamma$. A pure antisymmetric margin needs $\delta_{+1}=-\delta_{-1}=\gamma^{-1}\log\frac{1-\pi}\pi$, so it must shrink with the temperature, and in the hinge limit $\gamma\to\infty$ fixed positive margins drop out and only the weights decide, the cost-sensitive SVM result. Weights and margins each sized for the full correction correct twice: in a check of mine at $\pi=0.05$ and $\gamma=1$, the Bayes score at $\eta=\pi$ is $0$ for either single correction, $+2.94$ for both and $-2.94$ for plain logistic loss.

In experiments only the consistent methods find the Bayes separator, and on long-tailed benchmarks the gains are a few points. On two Gaussians with 5% positives and an affine classifier, the logit-adjusted loss lands close to the balanced-error Bayes separator, while plain ERM, LDAM and the equalization loss end up away from it; post-hoc adjustment reaches the Bayes error near $\tau=1$, and weight normalization at any $\tau$ makes things worse. On CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT and iNaturalist, balanced error is 27.16, 61.64, 53.11 and 38.66 for ERM, 22.60, 58.24, 49.66 and 33.98 for post-hoc adjustment at $\tau=1$ and 22.33, 56.11, 48.89 and 33.64 for the logit-adjusted loss, with the gain in the rare classes. The method is not uniformly best (weight normalization at $\tau=1$ slightly beats both untuned variants on one CIFAR-100 profile, though post-hoc adjustment with $\tau>1$ overtakes it), and adding the inconsistent LDAM margin on top gives the best iNaturalist number, so consistency does not settle the ranking at finite samples.