Stein's paradox and the James-Stein estimator

$\|X\|^2$ overestimates $\|\theta\|^2$ by about $d$, and the best shrinkage factor is $1-d/\|X\|^2$ (Stein's heuristic). Since $\|X\|^2=d+\|\theta\|^2+O_p(\sqrt{d+\|\theta\|^2})$, the data reveal $\|\theta\|^2$ up to an error small next to $d+\|\theta\|^2$. Try $\delta=[1-h]X$ with $h$ a function of $\|X\|^2/d\approx1+\rho^2$, $\rho^2=\|\theta\|^2/d$: $$\|\delta-\theta\|^2=d\big[(1-h)^2+\rho^2h^2\big]+O_p(\sqrt d),$$ minimized at $h=1/(1+\rho^2)$, that is $\delta=(1-d/\|X\|^2)X$, with loss about $d\rho^2/(1+\rho^2)$, the Bayes risk of Theorem 4 at $A=\rho^2$. This shrinks more than rescaling onto the sphere of radius $\|\theta\|$ because the noise also points $X$ in a wrong direction, and shrinking past the sphere removes more of that error than it adds in bias.

A small bounded shrinkage already beats $X$ (Stein 1956). With $Y=X-\theta$, expanding the square gives $$R(\theta,\delta_{a,b})\le d-2b\,\mathbb E\frac{Y^\top(Y+\theta)-b/2}{a+\|Y+\theta\|^2}.$$ Expand the denominator to second order in $\theta^\top Y$ and use rotational invariance (given $\|Y\|^2$, $\theta^\top Y$ has mean $0$ and second moment $\|\theta\|^2\|Y\|^2/d$); the leading terms give $(d-2-b/2)/(a+\|\theta\|^2)$, the bound of Theorem 1. The remainder is negligible uniformly only because $a\to\infty$ (Stein's bounds, from a Chebyshev bound on $\{\theta^\top Y\lt-\|\theta\|^2/4\}$ when $\|\theta\|^2\gt a$, are of order $1/((a+\|\theta\|^2)a)$ and $(a+\|\theta\|^2)^{-3/2}$), so $a$ serves only to keep the expansion uniform, and Step 3 does without it.

Stein's lemma gives the exact risk of $(1-b/S)X$ in terms of $\mathbb E_\theta[1/S]$ (SURE). Gaussian integration by parts, $\mathbb E[(X_i-\theta_i)g_i(X)]=\mathbb E[\partial_ig_i(X)]$, converts the cross term of $\|X+g-\theta\|^2$ into an expected divergence, which is SURE. For $g=-bX/S$, $\nabla\!\cdot g=-b(d-2)/S$, so $$R\Big(\theta,\big(1-\tfrac bS\big)X\Big)=d-\big[2b(d-2)-b^2\big]\,\mathbb E_\theta\frac1S,$$ and $\mathbb E[1/S]$ is finite exactly when $d\ge3$. Since $\theta$ enters only through one positive factor, the rule beats $X$ iff $0\lt b\lt2(d-2)$, and $b=d-2$ is best at every $\theta$ at once, with no expansion in $a$.

The exact risk is a Poisson mixture (James and Stein 1961). Both terms reduce to $\mathbb E[1/(d-2+2K)]$ through $S\mid K\sim\chi^2_{d+2K}$ and $\mathbb E[1/\chi^2_m]=1/(m-2)$; without Stein's lemma, James and Stein got the cross term $\mathbb E[(X-\theta)^\top X/S]$ from a power series integrated termwise against Beta integrals. The risk is $2$ at $\theta=0$, increases in $\|\theta\|$ because $K$ is stochastically increasing, and by Jensen is at most $2+\frac{(d-2)\|\theta\|^2}{d-2+\|\theta\|^2}$, close to the oracle of Step 1. The whole saving is made along $X$: every rule $c(X)X$ has the same error orthogonal to $X$, while the mean square error along $X$ is $1+(d-2)\mathbb E\frac{1}{d-2+2K}\le2$ for James-Stein. Conditioning on $S_e$ and using its first two moments gives the unknown-variance risk $\sigma^2\{d-[2an(d-2)-a^2n(n+2)]\,\mathbb E\frac{1}{d-2+2K}\}$, now with $K\sim\mathrm{Poisson}(\|\theta\|^2/2\sigma^2)$, for the factor $1-aS_e/\|X\|^2$, optimized at $a=(d-2)/(n+2)$.

$(d-2)^2/\|\theta\|^2$ is the best rate, and $X$ is admissible for $d=2$ (Stein 1956). Averaging a rule over $O(d)$ with Haar measure gives a spherically symmetric rule whose risk at $\theta$, by convexity of the loss, is at most the original risk averaged over the sphere of radius $\|\theta\|$, so it keeps any bound $R\le d-c/\|\theta\|^2$; and for those the information inequality yields a differential inequality with no solution. For bias $-\varphi(t)\theta$, $t=\|\theta\|^2$, Cramér-Rao with the gradient-squared term dropped gives $$R(\theta)\ge d+t\varphi^2-2d\varphi-4t\varphi'.$$ For $d=2$, $R\le2$ forces $\psi'\ge\psi^2/4t$ for $\psi=t\varphi$, so $1/\psi$ would have to change by $\frac14|\log(t/t_0)|$ toward $0$ or $\infty$ while $\psi$ is monotone and keeps its sign; hence $\psi\equiv0$, so $\delta$ is unbiased and $R\ge2$. For $d\ge3$, assume a saving $c/t$ with $\alpha^2=c-(d-2)^2\gt0$ and center at the James-Stein bias, $\varphi=(d-2)/t+f$; the $1/t$ terms collapse to $-(d-2)^2/t$, and with $\psi=tf$ $$4\psi'\ge\frac{\alpha^2+\psi^2}{t}\quad\Longrightarrow\quad\frac4\alpha\Big[\arctan\frac{\psi(t)}\alpha-\arctan\frac{\psi(t_0)}\alpha\Big]\ge\log\frac t{t_0},$$ impossible for large $t$ because the left side is at most $4\pi/\alpha$.

Truncating the shrinkage factor at zero lowers the risk (positive part). When $c=(d-2)/S\gt1$, James-Stein flips the sign of $X$, and replacing it by $0$ lowers the loss by $(c-1)^2S+2(c-1)\,\theta^\top X$. The first term is positive. For the second, $\mathbb E\,\theta^\top X=\|\theta\|^2$ is not enough because $c-1$ is correlated with $\theta^\top X$, so condition on $S$, which fixes $c$: on the sphere $\|x\|^2=S$ the density of $X$ is proportional to $e^{\theta^\top x}$, and reflecting across $\theta^\perp$ pairs $\pm\theta^\top x$ with weights $e^{\pm\theta^\top x}$, so $\mathbb E[\theta^\top X\mid S]\ge0$. Nothing used the form of $c$, which gives the general statement.

James-Stein is the Bayes rule with its one unknown estimated (Efron and Morris 1973). Marginally $X\sim N_d(0,(1+A)I)$, so $S\sim(1+A)\chi^2_d$ and $$\mathbb E\frac{d-2}{S}=\frac{d-2}{1+A}\,\mathbb E\frac1{\chi^2_d}=B.$$ Averaging SURE over the marginal shows that the Bayes risk of James-Stein exceeds that of the Bayes rule by $2B$, against $dB$ for the MLE, whence $2/d$. Estimating the prior mean by $\bar X$ leaves $\sum_i(X_i-\bar X)^2\sim(1+A)\chi^2_{d-1}$, so each estimated location parameter removes one degree of freedom. The MLE of $B$, $\min(1,d/S)$, gives $(1-d/S)_+X$; its untruncated form dominates $X$ for $d\ge5$ but by Step 3 has uniformly larger risk than $\hat\theta^{JS}$. The prior only suggests the rule; domination holds at every fixed $\theta$.