Score-based diffusion

The score is learnable by denoising, and DDPM's loss is a case of it. The conditional score is unbiased for the marginal one, $\mathbb E\big[\nabla\log p_{0t}(x(t)\mid x(0))\,\big|\,x(t)\big]=\nabla\log p_t(x(t))$, by differentiating $p_t=\int p_0\,p_{0t}$ under the integral. So the objective equals $\mathbb E\|s_\theta-\nabla\log p_t\|^2$ plus a constant free of $\theta$, and with enough capacity its population minimizer is the marginal score. Because the kernel is Gaussian the target is explicit, $\nabla\log p_{0t}=-\varepsilon/\sigma_t$, and $\lambda=\sigma_t^2$ equalizes the noise levels. With $s_\theta=-\varepsilon_\theta/\sigma_t$ the weighted loss becomes $\mathbb E\|\varepsilon_\theta(x_t,t)-\varepsilon\|^2$, DDPM's $L_{\rm simple}$ (its weight $1-\alpha_i$ is $\sigma_i^2$), and SMLD's weight $\sigma_i^2$ is the same rule.

Sampling is the time reversal of the forward SDE, and the score is its only unknown. By Anderson's theorem the reversed process $\bar x(s)=x(T-s)$ is again a diffusion, which in forward-running time reads $$d\bar x=\big[-f(\bar x,T-s)+g(T-s)^2\,\nabla\log p_{T-s}(\bar x)\big]\,ds+g(T-s)\,d\bar w(s),\qquad \bar x(0)\sim p_T .$$ A one-line Fokker-Planck computation checks the marginals: $q_s=p_{T-s}$ satisfies $\partial_sq=\nabla\cdot(fq)-\tfrac12g^2\Delta q$, and writing $-\tfrac12g^2\Delta q=-\nabla\cdot(g^2q\nabla\log q)+\tfrac12g^2\Delta q$ turns this into the forward equation of the displayed SDE. So the score term carries twice the diffusion's spreading, half to cancel the noise the reverse SDE injects and half to undo the forward smoothing. With a state-dependent $G(x,t)$ the score term becomes $GG^\top\nabla\log p_t$ and the drift gains $+\nabla\cdot[GG^\top]$. The sampler replaces $\nabla\log p_t$ by $s_\theta$ and starts from the prior.

The same score defines an ODE with the same marginals, and that ODE has exact likelihoods. Put the whole Laplacian into the transport term instead of half of it: $\partial_tp_t=-\nabla\cdot(fp_t)+\tfrac12g^2\Delta p_t=-\nabla\cdot\big[(f-\tfrac12g^2\nabla\log p_t)\,p_t\big]$ is a continuity equation, so the probability-flow ODE $$\frac{dx}{dt}=f(x,t)-\tfrac12g(t)^2\,\nabla\log p_t(x)$$ moves $p_0$ to $p_T$ through the same marginals as the SDE. With $s_\theta$ in place of the score it is a neural ODE, so the instantaneous change of variables gives $\log p_0(x(0))=\log p_T(x(T))+\int_0^T\nabla\cdot\tilde f_\theta(x(t),t)\,dt$, with the divergence estimated by Skilling-Hutchinson. Since the forward SDE has no parameters, the code $x(T)$ of an input is fixed by $p_0$ alone given an exact score, and two independently trained architectures give nearly the same codes. The paper does not note that with $s_\theta$ in place of the true score the ODE and the reverse SDE are different models, so the reported likelihood is the ODE model's, while the best FIDs come from SDE samplers.

DDPM's sampler is one discretization of the reverse VP SDE, and Langevin steps can correct any discretization. Running the forward chain's increments backward with the score added gives the reverse-diffusion sampler, and expanding DDPM's ancestral update $\frac{1}{\sqrt{1-\beta}}(x+\beta s_\theta)+\sqrt\beta z$ to first order in $\beta$ shows it is the same update up to $o(\beta)$. The corrector uses that $s_\theta(\cdot,t)$ is the score of the current marginal: a few Langevin steps $x\leftarrow x+\epsilon\,s_\theta(x,t)+\sqrt{2\epsilon}\,z$ at fixed $t$ target $p_t$ itself and pull the iterate back toward it. This predictor-corrector (PC) sampler has the old methods as special cases, SMLD an identity predictor with a Langevin corrector and DDPM an ancestral predictor with no corrector. On CIFAR-10 with 1000 noise levels, one corrector step per predictor step takes VE reverse diffusion from FID 4.79 to 3.60 and VE probability-flow sampling from 15.41 to 3.51, beating 2000 predictor steps at the same cost, while 2000 corrector-only steps give 20.43; for VP the gains are smaller.

Conditioning needs only a gradient added to the unconditional score, plus an approximation when the condition is part of $x$. By Bayes, $\nabla_x\log p_t(x\mid y)=\nabla_x\log p_t(x)+\nabla_x\log p_t(y\mid x)$, so the conditional reverse SDE uses the unconditional model plus a classifier $p_t(y\mid x(t))$ trained on noised inputs, since the term needed is the label likelihood given the current noisy state. For imputation with known coordinates $\Omega(x(0))=y$ the exact conditional score of the unknown coordinates $z(t)$ is intractable, so the paper conditions instead on $\hat\Omega(x(t))$, a fresh draw of the known coordinates noised to level $t$, and takes the gradient from the joint score. This heuristic has no error control, since it drops the dependence of $z(t)$ on $y$ beyond what $\hat\Omega(x(t))$ carries. Colorization is the same after an orthogonal change of color basis.

The experiments separate sampler, objective and architecture. With the discrete objectives fixed, the PC comparisons isolate the sampler; new architectures then give FID 2.45 (VE) and 2.78 (VP), and the continuous objective with doubled depth 2.20 and 2.41. For likelihood, the exact ODE value for Ho et al.'s own DDPM model is 3.28 bits/dim against its reported ELBO bounds of 3.70 and 3.75 (computed on discrete data, so the comparison is indicative), and sub-VP beats VP in every pairing. A black-box ODE solver cuts score evaluations by over 90% with little visible loss, though ODE samples without a corrector have worse FID than SDE samples.