30 Diffusion Models for Images

Chapter 29 studies diffusion models for images from a mathematical point of view. Diffusion models are probabilistic generative models that learn to reverse a gradual noising process. For computer vision, they provide a common mathematical language for image generation, denoising, inpainting, super-resolution, editing, text-to-image synthesis, inverse problems, and uncertainty-aware reconstruction.

The previous chapters built the ingredients needed here. Chapter 18 introduced likelihoods, priors, MAP estimation, and probabilistic image models. Chapter 20 studied representations. Chapter 22 studied training by stochastic optimization. Chapter 25 studied transformer-based architectures. Chapter 27 studied vision-language conditioning. Chapter 28 studied prompt-conditioned foundation models. This chapter combines those ideas into one probabilistic framework.

A diffusion model begins with a clean image $x_0$ sampled from the unknown data distribution $p_{\text{data}}$. It defines a forward Markov chain that gradually adds noise,

\[ q(x_t \mid x_{t-1}), \qquad t=1,\dots,T, \]

until $x_T$ is close to a simple distribution such as a standard Gaussian. The model then learns a reverse chain

\[ p_\theta(x_{t-1}\mid x_t) \]

that turns noise into an image. The essential statistical problem is therefore:

\[ \text{learn the reverse dynamics of a known noising process from samples of clean images.} \]

This idea turns high-dimensional image generation into a sequence of smaller denoising problems. It also turns many computer vision tasks into conditional sampling problems.

30.1 Learning Goals

After reading this chapter, students should be able to:

define the forward diffusion process and compute the closed-form distribution $q(x_t\mid x_0)$;
explain why the reverse process is generally unknown and must be learned;
derive the common denoising objective used in DDPM-style models;
connect denoising score matching to the score function $\nabla_x \log p_t(x)$;
distinguish $\epsilon$-prediction, $x_0$-prediction, and velocity prediction;
explain DDPM, DDIM, score-based SDE, and probability-flow ODE viewpoints;
formulate conditional diffusion for class labels, text, masks, depth, and other visual prompts;
explain classifier guidance and classifier-free guidance;
describe latent diffusion as diffusion in a learned representation space;
implement small mathematical experiments for noising, denoising, sampling, guidance, and inverse problems;
analyze strengths, limitations, statistical assumptions, and failure modes of diffusion models in vision.

30.2 29.1 Generative Modeling for Images

A generative model tries to approximate an unknown distribution on images. Let

\[ X \in \mathbb R^d, \]

where $d=HWC$ for an image with height $H$, width $W$, and $C$ channels. The data distribution is

\[ p_{\text{data}}(x). \]

A generative model constructs a distribution $p_\theta(x)$ that should be close to $p_{\text{data}}(x)$. The central question is how to represent and sample from $p_\theta$.

Classical choices include:

explicit likelihood models, where $p_\theta(x)$ can be evaluated;
latent-variable models, where $x$ is generated from a latent variable $z$;
adversarial models, where a generator competes with a discriminator;
flow models, where an invertible map transforms Gaussian noise into data;
autoregressive models, where pixels or tokens are generated sequentially;
diffusion models, where a stochastic denoising process transforms noise into data.

Diffusion models are especially attractive in vision because the intermediate states have a simple interpretation: noisy versions of images. The model learns a family of denoisers indexed by noise level.

30.3 29.2 The Forward Noising Process

Let $x_0\sim p_{\text{data}}$. A standard DDPM-style forward process is the Markov chain

\[ q(x_t\mid x_{t-1}) =\mathcal N\left(\sqrt{1-\beta_t}x_{t-1},\,\beta_t I\right), \qquad t=1,\dots,T, \]

where

\[ 0<\beta_t<1. \]

The numbers $\beta_t$ form the noise schedule. Define

\[ \alpha_t=1-\beta_t, \qquad \bar\alpha_t=\prod_{s=1}^t \alpha_s. \]

Then the forward transition can be written as

\[ x_t=\sqrt{\alpha_t}x_{t-1}+\sqrt{1-\alpha_t}\epsilon_t, \qquad \epsilon_t\sim\mathcal N(0,I). \]

The process gradually shrinks the signal and adds independent Gaussian noise.

30.4 29.3 Closed Form of $q(x_t\mid x_0)$

A key algebraic fact is that $x_t$ can be sampled directly from $x_0$ without simulating all previous steps:

\[ q(x_t\mid x_0) =\mathcal N\left(\sqrt{\bar\alpha_t}x_0,\,(1-\bar\alpha_t)I\right). \]

Equivalently,

\[ x_t=\sqrt{\bar\alpha_t}x_0+ \sqrt{1-\bar\alpha_t}\epsilon, \qquad \epsilon\sim\mathcal N(0,I). \]

This formula is the computational heart of diffusion training. We can choose a random time $t$, draw Gaussian noise $\epsilon$, form $x_t$, and train a model to predict something about the clean image or the noise.

30.4.1 Proof by induction

For $t=1$, the formula follows from the definition. Suppose

\[ x_{t-1}=\sqrt{\bar\alpha_{t-1}}x_0+ \sqrt{1-\bar\alpha_{t-1}}\epsilon' \]

with $\epsilon'\sim\mathcal N(0,I)$. Then

\[ \begin{aligned} x_t &=\sqrt{\alpha_t}x_{t-1}+\sqrt{1-\alpha_t}\epsilon_t \\ &=\sqrt{\alpha_t\bar\alpha_{t-1}}x_0 +\sqrt{\alpha_t(1-\bar\alpha_{t-1})}\epsilon' +\sqrt{1-\alpha_t}\epsilon_t. \end{aligned} \]

The two Gaussian noise terms are independent and isotropic. Their sum is isotropic Gaussian with variance

\[ \alpha_t(1-\bar\alpha_{t-1})+(1-\alpha_t) =1-\alpha_t\bar\alpha_{t-1} =1-\bar\alpha_t. \]

Thus

\[ x_t=\sqrt{\bar\alpha_t}x_0+ \sqrt{1-\bar\alpha_t}\epsilon. \]

30.5 29.4 Signal-to-Noise Ratio

The coefficient $\bar\alpha_t$ controls how much signal remains at time $t$. A useful quantity is the signal-to-noise ratio

\[ \operatorname{SNR}(t)=\frac{\bar\alpha_t}{1-\bar\alpha_t}. \]

Early times have large SNR: $x_t$ is mostly image. Late times have small SNR: $x_t$ is mostly noise. Training over many $t$ values teaches the network to denoise at many noise scales.

Noise schedules are not merely implementation details. They determine which denoising problems are emphasized during training. Common choices include linear schedules, cosine schedules, and schedules expressed directly in log-SNR.

30.6 29.5 Interactive Demo: Forward Noising and Noise Schedule

The following demo compares how $\bar\alpha_t$ and SNR change under different noise schedules.

30.7 29.6 The Reverse Process

If the forward process destroys structure, generation requires reversing it:

\[ p_\theta(x_{0:T})=p(x_T)\prod_{t=1}^T p_\theta(x_{t-1}\mid x_t), \]

where typically

\[ p(x_T)=\mathcal N(0,I). \]

A common parameterization is

\[ p_\theta(x_{t-1}\mid x_t) =\mathcal N\left(\mu_\theta(x_t,t),\Sigma_\theta(x_t,t)\right). \]

The forward transition $q(x_t\mid x_{t-1})$ is known. The reverse transition is not known because it depends on the unknown data distribution. Learning $p_\theta$ is the main task.

The reverse process is not simply subtracting Gaussian noise. At late times, many images could have produced the same noisy observation. The model must use learned image priors to decide which denoising direction is plausible.

30.8 29.7 The Exact Posterior $q(x_{t-1}\mid x_t,x_0)$

Although $q(x_{t-1}\mid x_t)$ is unknown, the conditional posterior given both $x_t$ and $x_0$ is tractable:

\[ q(x_{t-1}\mid x_t,x_0) =\mathcal N\left(\tilde\mu_t(x_t,x_0),\tilde\beta_t I\right), \]

where

\[ \tilde\beta_t =\frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t, \]

and

\[ \tilde\mu_t(x_t,x_0) = \frac{\sqrt{\bar\alpha_{t-1}}\beta_t}{1-\bar\alpha_t}x_0 + \frac{\sqrt{\alpha_t}(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}x_t. \]

This formula is useful because if a network can estimate $x_0$ from $x_t$, then it can estimate the reverse mean.

30.9 29.8 Predicting Noise

Instead of directly predicting $x_0$, DDPMs commonly train a neural network $\epsilon_\theta(x_t,t)$ to predict the noise $\epsilon$ in

\[ x_t=\sqrt{\bar\alpha_t}x_0+ \sqrt{1-\bar\alpha_t}\epsilon. \]

Solving for $x_0$ gives

\[ \widehat x_0(x_t,t)= \frac{x_t-\sqrt{1-\bar\alpha_t}\epsilon_\theta(x_t,t)} {\sqrt{\bar\alpha_t}}. \]

Thus noise prediction, clean-image prediction, and reverse-mean prediction are closely related.

The simplified training loss is

\[ L_{\text{simple}}(\theta) =\mathbb E_{x_0,\epsilon,t} \left[\left\|\epsilon- \epsilon_\theta\left(\sqrt{\bar\alpha_t}x_0+ \sqrt{1-\bar\alpha_t}\epsilon,t\right) \right\|_2^2\right]. \]

This objective turns generative modeling into supervised regression: input a noisy image and time index, output the noise that was added.

30.10 29.9 Statistical Meaning of Noise Prediction

For fixed $t$, minimizing squared error implies

\[ \epsilon_\theta^*(x_t,t)=\mathbb E[\epsilon\mid x_t,t]. \]

Thus the network learns a conditional expectation. Since

\[ x_t=\sqrt{\bar\alpha_t}x_0+ \sqrt{1-\bar\alpha_t}\epsilon, \]

noise prediction is equivalent to estimating the direction from noisy observation back toward the data manifold. The model is not memorizing pixels; it is learning a family of denoising vector fields.

This connects diffusion to score matching. For the perturbed distribution $p_t(x)$ of $x_t$, the score is

\[ \nabla_x \log p_t(x). \]

For the Gaussian perturbation model, the optimal noise predictor is related to the score by

\[ \epsilon_\theta^*(x_t,t) \approx -\sqrt{1-\bar\alpha_t}\,\nabla_{x_t}\log p_t(x_t). \]

Thus a diffusion model can be viewed as learning the score field of noisy image distributions.

30.11 29.10 Interactive Demo: Score Field of a Noisy Distribution

This toy example uses a one-dimensional mixture distribution and shows how increasing noise smooths the density and changes the score field.

30.12 29.11 Evidence Lower Bound Viewpoint

Diffusion models can be derived from variational inference. The negative log-likelihood satisfies an evidence lower bound:

\[ -\log p_\theta(x_0) \leq \mathbb E_q\left[ -\log p_\theta(x_{0:T})+ \log q(x_{1:T}\mid x_0) \right]. \]

This decomposes into terms comparing the true reverse posterior $q(x_{t-1}\mid x_t,x_0)$ to the learned reverse model $p_\theta(x_{t-1}\mid x_t)$, plus reconstruction and prior terms. In simplified form, one obtains a weighted sum of denoising losses.

The simplified noise-prediction loss is not arbitrary. It is a practical version of a likelihood-based objective. The weighting of different times $t$ determines which noise levels dominate training.

30.13 29.12 Three Common Parameterizations

Diffusion models often differ in what the network predicts.

30.13.1 Noise prediction

The network predicts $\epsilon$:

\[ \epsilon_\theta(x_t,t)\approx \epsilon. \]

This is simple and stable.

30.13.2 Clean-image prediction

The network predicts $x_0$:

\[ x_\theta(x_t,t)\approx x_0. \]

This can make sampling formulas interpretable because one directly estimates the clean image.

30.13.3 Velocity prediction

Some models predict a velocity-like variable

\[ v=\sqrt{\bar\alpha_t}\epsilon- \sqrt{1-\bar\alpha_t}x_0. \]

Velocity prediction often balances low-noise and high-noise training regimes. It is especially useful when schedules are expressed in angular or log-SNR form.

All three parameterizations are algebraically convertible if $x_t$ and $t$ are known.

30.14 29.13 Time Embeddings

The denoising network must know the noise level. It receives $t$ or a continuous noise parameter as input. A common sinusoidal embedding is

\[ \gamma(t)= (\sin(\omega_1 t),\cos(\omega_1 t),\dots, \sin(\omega_m t),\cos(\omega_m t)). \]

This is analogous to positional encoding in transformers. The embedding is usually processed by a small multilayer perceptron and injected into residual blocks through additive or affine transformations.

From a mathematical point of view, the model approximates a function

\[ f_\theta:\mathbb R^d\times [0,T]\to \mathbb R^d. \]

The time variable indexes a continuum of denoising problems.

30.15 29.14 U-Net Denoisers

Image diffusion models commonly use U-Net-like denoisers. The network receives a noisy image $x_t$ and a time embedding. It outputs a tensor of the same spatial shape as $x_t$.

A schematic form is

\[ \epsilon_\theta(x_t,t)=D_\theta(E_\theta(x_t,t),t), \]

where $E_\theta$ is an encoder and $D_\theta$ is a decoder with skip connections.

The architecture is well matched to images because:

local convolution captures texture and edges;
downsampling captures global structure;
skip connections preserve spatial detail;
attention blocks capture long-range relationships;
time embeddings condition every denoising scale.

Modern diffusion systems may use convolutional U-Nets, transformers, or hybrid architectures.

30.16 29.15 DDPM Sampling

After training, sampling starts from Gaussian noise

\[ x_T\sim\mathcal N(0,I). \]

For $t=T,T-1,\dots,1$, one computes a reverse mean using $\epsilon_\theta(x_t,t)$ and samples

\[ x_{t-1}=\mu_\theta(x_t,t)+\sigma_t z, \qquad z\sim\mathcal N(0,I). \]

A common formula for the mean is

\[ \mu_\theta(x_t,t)= \frac{1}{\sqrt{\alpha_t}} \left( x_t- \frac{\beta_t}{\sqrt{1-\bar\alpha_t}} \epsilon_\theta(x_t,t) \right). \]

The stochastic term $\sigma_tz$ preserves sampling diversity. If $\sigma_t=0$, the process becomes deterministic.

30.17 29.16 DDIM Sampling

DDIM sampling constructs a non-Markovian reverse process that can sample with fewer steps. Given an estimate $\widehat x_0$, one can write

\[ x_{t-1} = \sqrt{\bar\alpha_{t-1}}\widehat x_0 +\sqrt{1-\bar\alpha_{t-1}}\widehat\epsilon \]

for deterministic sampling, where

\[ \widehat\epsilon= \epsilon_\theta(x_t,t). \]

More general DDIM samplers include a parameter controlling stochasticity. The important point is that generation can follow a deterministic path through latent space, making image editing and interpolation more controllable.

30.18 29.17 Interactive Demo: Stochastic vs. Deterministic Sampling

This toy visualization compares several reverse trajectories in a two-dimensional toy distribution. Stochastic sampling can explore multiple modes; deterministic sampling follows smoother paths.

30.19 29.18 Score-Based SDE Viewpoint

Diffusion can also be formulated in continuous time. A forward stochastic differential equation has the form

\[ dx=f(x,t)dt+g(t)dw, \]

where $w$ is Brownian motion. The reverse-time SDE is

\[ dx=\left[f(x,t)-g(t)^2\nabla_x\log p_t(x)\right]dt+g(t)d\bar w, \]

where time runs backward and $\bar w$ is reverse-time Brownian motion.

Thus if one learns the score

\[ s_\theta(x,t)\approx \nabla_x\log p_t(x), \]

one can simulate the reverse SDE to generate samples. This viewpoint clarifies the connection among diffusion models, score matching, Langevin dynamics, and stochastic processes.

30.20 29.19 Probability-Flow ODE

Associated with the reverse SDE is a deterministic probability-flow ODE:

\[ \frac{dx}{dt}=f(x,t)-\frac{1}{2}g(t)^2\nabla_x\log p_t(x). \]

This ODE has the same marginal distributions as the corresponding SDE. It supports deterministic sampling and likelihood computation by change of variables.

For applied mathematics students, this is an important bridge: diffusion models can be studied as numerical solvers for learned vector fields.

30.21 29.20 Numerical Solvers and Discretization Error

Sampling is numerical integration. The learned denoiser or score gives a vector field, and a sampler approximates its flow.

Important questions include:

How many steps are needed?
Is the solver explicit or implicit?
Is the trajectory stochastic or deterministic?
How does discretization error affect image quality?
How do errors accumulate over time?

Modern fast samplers use higher-order ODE/SDE solvers, carefully designed noise schedules, and distillation methods. The mathematical issues resemble those in numerical differential equations: stability, local truncation error, stiffness, and error propagation.

30.22 29.21 Conditional Diffusion

Most vision applications are conditional. The model generates or reconstructs an image given extra information $c$:

\[ p_\theta(x_0\mid c). \]

The condition $c$ may be:

a class label;
a text prompt;
a segmentation mask;
an edge map;
a depth map;
a low-resolution image;
a corrupted image;
a bounding box;
another image;
a prompt embedding from a vision-language model.

The denoising network becomes

\[ \epsilon_\theta(x_t,t,c). \]

Conditional diffusion turns many computer vision tasks into conditional denoising.

30.23 29.22 Classifier Guidance

Suppose a classifier estimates $p_\phi(c\mid x_t)$. To sample images matching class $c$, one can guide the reverse process using

\[ \nabla_{x_t}\log p_\phi(c\mid x_t). \]

Intuitively, the sampler follows both the generative score and a classifier gradient pointing toward images with the desired label.

A guided score has the schematic form

\[ \widetilde s(x_t,t,c) =s_\theta(x_t,t)+w\nabla_{x_t}\log p_\phi(c\mid x_t), \]

where $w\ge 0$ is a guidance scale.

Classifier guidance improves conditioning but requires a noise-aware classifier and can introduce artifacts when the guidance is too strong.

30.24 29.23 Classifier-Free Guidance

Classifier-free guidance avoids training a separate classifier. During training, the condition $c$ is randomly dropped. The same network learns both conditional and unconditional denoising:

\[ \epsilon_\theta(x_t,t,c), \qquad \epsilon_\theta(x_t,t,\varnothing). \]

At sampling time, one uses

\[ \widehat\epsilon_{\text{cfg}} = \epsilon_\theta(x_t,t,\varnothing) +w\left( \epsilon_\theta(x_t,t,c)-\epsilon_\theta(x_t,t,\varnothing) \right). \]

Here $w$ is the guidance scale. Larger $w$ usually improves prompt adherence but can reduce diversity and create over-saturated or distorted images.

30.25 29.24 Interactive Demo: Classifier-Free Guidance Trade-Off

The toy model below illustrates the trade-off between unconditional diversity and conditional concentration.

30.26 29.25 Text-to-Image Diffusion

In text-to-image generation, the condition $c$ is a text embedding. A text encoder maps a prompt to vectors:

\[ \tau = T_\psi(\text{text}). \]

The denoising network uses $\tau$ through cross-attention:

\[ \operatorname{Attention}(Q,K,V) =\operatorname{softmax}\left(\frac{QK^T}{\sqrt d}\right)V, \]

where image features produce queries and text tokens produce keys and values. This allows each image location or latent token to attend to words or phrases.

Text-to-image diffusion combines three modeling components:

a text representation model;
a visual denoising model;
a sampling algorithm with guidance.

The mathematical challenge is not only generation but alignment: the generated image should match the conditional distribution implied by the text prompt.

30.27 29.26 Latent Diffusion

Pixel-space diffusion is expensive because images are high dimensional. Latent diffusion first trains an autoencoder:

\[ z=E_\phi(x), \qquad \widehat x=D_\phi(z), \]

where $z$ has lower spatial resolution or fewer channels than $x$. Diffusion is then performed in latent space:

\[ z_t=\sqrt{\bar\alpha_t}z_0+\sqrt{1-\bar\alpha_t}\epsilon. \]

The final image is decoded by $D_\phi$.

Latent diffusion changes the modeling problem. The diffusion model learns the distribution of latent codes, not raw pixels. This improves efficiency but makes the autoencoder an important part of the generative prior.

30.28 29.27 Interactive Demo: Pixel Diffusion vs. Latent Diffusion Cost

This demo compares the rough dimensional cost of doing diffusion in pixel space versus a compressed latent grid.

30.29 29.28 Diffusion for Inverse Problems

Many vision tasks have the form

\[ y=A(x_0)+\eta, \]

where $A$ is a measurement operator and $\eta$ is noise. Examples include:

super-resolution;
deblurring;
inpainting;
compressed sensing;
MRI or CT reconstruction;
low-light enhancement;
depth-guided reconstruction.

Diffusion models can serve as learned priors. A posterior sampling problem is

\[ p(x_0\mid y)\propto p(y\mid x_0)p(x_0). \]

The diffusion prior supplies $p(x_0)$ implicitly through the learned denoising process. The measurement likelihood supplies consistency with the observation.

A schematic guided score is

\[ \nabla_x\log p_t(x\mid y) \approx s_\theta(x,t)+\lambda \nabla_x\log p(y\mid x). \]

This unifies generative modeling and classical inverse problems.

30.30 29.29 Inpainting and Mask Constraints

For inpainting, the measurement is

\[ y=M\odot x_0, \]

where $M\in\{0,1\}^d$ is a known mask. The unknown region is $(1-M)\odot x_0$.

A diffusion inpainting sampler must preserve known pixels while sampling plausible missing pixels. A simple consistency step is

\[ x_t \leftarrow M\odot y_t+(1-M)\odot x_t, \]

where $y_t$ is the noised version of the known observation at time $t$.

This shows a useful principle: constraints can be enforced during sampling, not only during training.

30.31 29.30 Super-Resolution

In super-resolution, one observes a low-resolution image

\[ y=D x_0+\eta, \]

where $D$ is a downsampling operator. The goal is to sample from

\[ p(x_0\mid y). \]

The problem is ill-posed: many high-resolution images can produce the same low-resolution observation. Diffusion models are natural here because they represent a conditional distribution rather than a single deterministic answer.

For statistics students, this is important: perceptually plausible super-resolution is not the same as estimating the conditional mean. A conditional mean may be blurry, while samples from the posterior can be sharp and diverse.

30.32 29.31 Interactive Demo: Conditional Posterior Ambiguity

This toy visualization compares a conditional mean with multiple posterior samples. In image reconstruction, the mean can be smoother than any plausible sample.

30.33 29.32 Diffusion and Image Editing

Diffusion models can edit images by combining noising, conditioning, and denoising. A typical procedure is:

map an input image to an intermediate noisy state;
modify the condition or prompt;
run the reverse process while preserving selected structure.

The amount of noise controls the size of the edit. Low noise preserves the image closely. High noise allows large semantic changes.

Mathematically, editing is sampling from a conditional distribution with constraints:

\[ p(x_{\text{edited}}\mid x_{\text{input}},c,\text{constraints}). \]

This is close in spirit to classical constrained optimization, but the learned prior is much richer.

30.34 29.33 Evaluation of Diffusion Models

Evaluating generative image models is difficult. Common criteria include:

sample quality;
diversity;
coverage of the data distribution;
conditional alignment;
perceptual realism;
factual consistency;
fairness and bias;
robustness to prompt changes;
downstream utility.

Common metrics include FID, precision/recall for generative models, CLIP-style alignment scores, human preference studies, and task-specific measures. None is complete.

A statistical warning: a model can produce visually impressive samples while failing to represent the true data distribution in important regions.

30.35 29.34 Calibration and Uncertainty

Diffusion models produce samples, so they can represent uncertainty. For a conditional task, one can draw

\[ x_0^{(1)},\dots,x_0^{(m)}\sim p_\theta(x_0\mid y) \]

and estimate posterior summaries:

\[ \widehat\mu(y)=\frac1m\sum_{j=1}^m x_0^{(j)}, \]

\[ \widehat\Sigma(y)=\frac1{m-1}\sum_{j=1}^m (x_0^{(j)}-\widehat\mu)(x_0^{(j)}-\widehat\mu)^T. \]

For images, one usually visualizes marginal variance maps, uncertainty over edges, or uncertainty over downstream predictions.

However, sample diversity is not automatically calibrated uncertainty. The sampler, guidance scale, conditioning, and training distribution all affect the apparent uncertainty.

30.36 29.35 Failure Modes

Diffusion models have several mathematically interpretable failure modes.

30.36.1 Sampling error

Finite-step samplers approximate continuous or discrete reverse dynamics. Too few steps may introduce artifacts.

30.36.2 Score error

The learned score may be inaccurate in low-density regions. Sampling trajectories can pass through such regions.

30.36.3 Guidance over-optimization

Large guidance can overemphasize prompt adherence and reduce naturalness or diversity.

30.36.4 Mode imbalance

The model may underrepresent rare visual configurations in the training distribution.

30.36.5 Prompt ambiguity

A text prompt may correspond to a broad or multimodal conditional distribution.

30.36.6 Dataset bias

The learned prior reflects the data distribution and its biases.

30.36.7 Hallucination in inverse problems

When measurements are weak, the learned prior may invent plausible but incorrect details. This is especially important in medical imaging, scientific imaging, and legal or forensic settings.

30.37 29.36 Python Example: Forward Diffusion on an Image

The following example shows how to sample noisy versions of an image at different time steps.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(29)

H, W = 96, 96
yy, xx = np.mgrid[0:H, 0:W]

# Synthetic clean image in [0, 1]
x0 = (
    0.25
    + 0.55 * np.exp(-((xx - 32) ** 2 + (yy - 45) ** 2) / (2 * 14 ** 2))
    + 0.35 * np.exp(-((xx - 68) ** 2 + (yy - 55) ** 2) / (2 * 10 ** 2))
)
x0 = np.clip(x0, 0, 1)

T = 1000
betas = np.linspace(1e-4, 0.02, T)
alphas = 1.0 - betas
alpha_bars = np.cumprod(alphas)

def q_sample(x0, t):
    """Sample x_t from q(x_t | x_0). t is zero-based."""
    eps = rng.normal(size=x0.shape)
    a_bar = alpha_bars[t]
    xt = np.sqrt(a_bar) * x0 + np.sqrt(1.0 - a_bar) * eps
    return xt, eps

steps = [0, 50, 150, 300, 600, 999]
fig, axes = plt.subplots(1, len(steps), figsize=(13, 3))
for ax, t in zip(axes, steps):
    xt, _ = q_sample(x0, t)
    ax.imshow(xt, cmap="gray")
    ax.set_title(f"t={t+1}")
    ax.axis("off")
plt.tight_layout()
plt.show()

30.38 29.37 Python Example: One Training Batch for Noise Prediction

The following code constructs one training batch for the simplified denoising loss. It is meant to clarify the mathematics, not to be a full training pipeline.

import numpy as np

rng = np.random.default_rng(123)

# Suppose x0_batch has shape (B, H, W, C) and values scaled to [-1, 1].
B, H, W, C = 16, 64, 64, 3
x0_batch = rng.normal(size=(B, H, W, C)).astype("float32")

T = 1000
betas = np.linspace(1e-4, 0.02, T, dtype="float32")
alphas = 1.0 - betas
alpha_bars = np.cumprod(alphas)

# Random time for each image.
t = rng.integers(0, T, size=(B,))
eps = rng.normal(size=x0_batch.shape).astype("float32")

# Broadcast alpha_bar to image shape.
a_bar = alpha_bars[t].reshape(B, 1, 1, 1)
xt = np.sqrt(a_bar) * x0_batch + np.sqrt(1.0 - a_bar) * eps

# A neural network would predict eps_hat = model(xt, t).
# The simplified DDPM loss is mean squared error.
eps_hat = np.zeros_like(eps)  # placeholder
loss = np.mean((eps - eps_hat) ** 2)
print("toy loss:", loss)

30.39 29.38 TensorFlow Example: Tiny Denoising Network Skeleton

This compact example shows the shape of a denoising model. A real diffusion model would use a much larger U-Net or transformer and better time conditioning.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

class SinusoidalTimeEmbedding(layers.Layer):
    def __init__(self, dim=64, **kwargs):
        super().__init__(**kwargs)
        self.dim = dim

    def call(self, t):
        t = tf.cast(tf.reshape(t, [-1, 1]), tf.float32)
        half = self.dim // 2
        freqs = tf.exp(
            tf.linspace(tf.math.log(1.0), tf.math.log(1000.0), half)
        )
        args = t / freqs[None, :]
        return tf.concat([tf.sin(args), tf.cos(args)], axis=-1)

def tiny_denoiser(image_shape=(64, 64, 3), time_dim=64):
    x_in = keras.Input(shape=image_shape, name="x_t")
    t_in = keras.Input(shape=(), dtype="int32", name="t")

    temb = SinusoidalTimeEmbedding(time_dim)(t_in)
    temb = layers.Dense(128, activation="swish")(temb)
    temb = layers.Dense(128, activation="swish")(temb)

    h = layers.Conv2D(64, 3, padding="same", activation="swish")(x_in)
    h = layers.Conv2D(64, 3, padding="same", activation="swish")(h)

    # Inject time information by adding a broadcast vector.
    tb = layers.Dense(64)(temb)
    tb = layers.Reshape((1, 1, 64))(tb)
    h = h + tb

    h = layers.Conv2D(64, 3, padding="same", activation="swish")(h)
    out = layers.Conv2D(image_shape[-1], 3, padding="same")(h)
    return keras.Model([x_in, t_in], out)

model = tiny_denoiser()
model.summary()

30.40 29.39 Python Example: DDPM Reverse Step

The following function implements a simplified reverse step using a predicted noise tensor.

import numpy as np

rng = np.random.default_rng(7)

def ddpm_step(x_t, t, eps_hat, betas, alpha_bars):
    """One DDPM reverse step from x_t to x_{t-1}.

    t is zero-based. This example uses fixed variance beta_t.
    """
    beta_t = betas[t]
    alpha_t = 1.0 - beta_t
    a_bar_t = alpha_bars[t]

    mean = (1.0 / np.sqrt(alpha_t)) * (
        x_t - beta_t / np.sqrt(1.0 - a_bar_t) * eps_hat
    )

    if t == 0:
        return mean

    z = rng.normal(size=x_t.shape)
    return mean + np.sqrt(beta_t) * z

30.41 29.40 Python Example: Classifier-Free Guidance Formula

This example shows the algebra of classifier-free guidance for noise prediction.

import numpy as np

# Suppose a neural network has produced two predictions.
eps_uncond = np.random.normal(size=(4, 64, 64, 3))
eps_cond = np.random.normal(size=(4, 64, 64, 3))

def classifier_free_guidance(eps_uncond, eps_cond, guidance_scale):
    return eps_uncond + guidance_scale * (eps_cond - eps_uncond)

for w in [0.0, 1.0, 3.0, 7.5]:
    eps_guided = classifier_free_guidance(eps_uncond, eps_cond, w)
    print(w, eps_guided.shape, np.linalg.norm(eps_guided.ravel()))

30.42 29.41 AI-Assisted Mathematical Study Prompts

Use AI tools carefully as mathematical collaborators, not as authorities. Good prompts include:

Forward process derivation. Ask for a proof of the formula for $q(x_t\mid x_0)$, then verify every variance term yourself.
Posterior derivation. Ask for the Gaussian conditioning derivation of $q(x_{t-1}\mid x_t,x_0)$ and check dimensions.
Score connection. Ask why the optimal noise predictor is proportional to the score $\nabla_x\log p_t(x)$.
Sampler comparison. Ask for a comparison of DDPM, DDIM, and score-SDE sampling in terms of randomness, numerical integration, and likelihood.
Guidance critique. Ask for examples where classifier-free guidance improves prompt alignment but harms diversity.
Inverse problem audit. Ask how a diffusion prior might hallucinate details in medical image reconstruction, then propose safeguards.

When using AI, always request explicit assumptions, equations, and dimensional checks.

30.43 29.42 Common Misconceptions

Misconception 1. Diffusion models simply remove random noise.

They learn a sequence of conditional denoising distributions shaped by the data distribution. Denoising depends on image priors.

Misconception 2. More guidance is always better.

Large guidance can reduce diversity, distort images, and over-optimize prompt features.

Misconception 3. A generated image is a posterior mean.

A diffusion sample is one draw from a model distribution. It is not necessarily a conditional expectation.

Misconception 4. Latent diffusion is just faster pixel diffusion.

Latent diffusion changes the data space. The autoencoder affects what details can be represented.

Misconception 5. Better visual quality implies better statistical modeling.

Perceptual quality, likelihood, diversity, calibration, and downstream utility are different criteria.

30.44 29.43 Mini-Project: Diffusion as a Learned Image Prior

Design a small experiment on synthetic images.

Generate a dataset of simple shapes.
Define a forward noising process.
Train a small denoising network to predict Gaussian noise.
Compare denoising at low, medium, and high noise levels.
Use the trained denoiser in a simple inpainting or super-resolution task.
Evaluate reconstruction error, diversity, and failure cases.

A mathematically strong report should include:

the data distribution;
the noise schedule;
the training objective;
the network architecture;
the sampling algorithm;
quantitative metrics;
visual examples;
a discussion of uncertainty and hallucination.

30.45 29.44 Exercises

30.45.1 Conceptual Exercises

Explain why the forward diffusion process is easy to sample but the reverse process is hard.
Why does the direct formula for $q(x_t\mid x_0)$ make training efficient?
Explain the difference between $\epsilon$-prediction and $x_0$-prediction.
Why can diffusion models represent multiple plausible solutions to super-resolution?
Explain why text-to-image generation is a conditional distribution modeling problem.
Describe one reason why FID is not a complete measure of generative quality.

30.45.2 Mathematical Exercises

Prove by induction that \[ q(x_t\mid x_0)= \mathcal N(\sqrt{\bar\alpha_t}x_0,(1-\bar\alpha_t)I). \]
Starting from \[ x_t=\sqrt{\bar\alpha_t}x_0+ \sqrt{1-\bar\alpha_t}\epsilon, \] solve for $x_0$ in terms of $x_t$ and $\epsilon$.
Show that the DDPM reverse mean formula can be written in terms of $\epsilon_\theta(x_t,t)$.
For a scalar Gaussian data distribution $x_0\sim\mathcal N(0,\sigma_0^2)$, compute the distribution of $x_t$.
Let $p_t$ be a one-dimensional density. Show that Langevin dynamics \[ x_{k+1}=x_k+\eta\nabla_x\log p_t(x_k)+\sqrt{2\eta}z_k \] uses the score to move samples toward high-density regions while preserving randomness.
Derive the classifier-free guidance formula as an extrapolation from unconditional to conditional noise prediction.
Suppose a downsampling operator $D$ maps high-resolution images to low-resolution images. Explain why the inverse problem $y=Dx$ is not unique.

30.45.3 Python Exercises

Implement linear and cosine noise schedules. Plot $\bar\alpha_t$ and SNR.
Generate noisy versions of a small image using the closed form $q(x_t\mid x_0)$.
Train a tiny denoising network on synthetic disks, squares, and triangles.
Compare MSE loss for predicting $\epsilon$ versus predicting $x_0$.
Implement one DDPM reverse step and test it on synthetic tensors.
Simulate classifier-free guidance using two toy vector fields.
Create a toy inpainting task and enforce mask consistency during sampling.

30.45.4 AI and Modeling Exercises

Ask an AI assistant to explain DDPM training. Identify at least three places where the explanation omits assumptions or dimensions.
Ask for a derivation of the posterior $q(x_{t-1}\mid x_t,x_0)$. Rewrite it in your own notation.
Ask for examples of diffusion hallucination in scientific imaging. Convert the answer into a checklist for responsible use.
Compare a diffusion prior with a total-variation prior for image reconstruction. Which assumptions are explicit in each?
Design a conditional diffusion project for medical imaging, satellite imaging, or microscopy. Specify what evidence would be needed before deployment.

30.46 29.45 Chapter Summary

Diffusion models are probabilistic generative models built from a simple idea: gradually add noise to data, then learn to reverse the noising process. The forward process is known and analytically tractable. The reverse process depends on the unknown image distribution and is learned by denoising.

The most important mathematical formulas are:

\[ q(x_t\mid x_{t-1}) =\mathcal N(\sqrt{1-\beta_t}x_{t-1},\beta_t I), \]

\[ q(x_t\mid x_0) =\mathcal N(\sqrt{\bar\alpha_t}x_0,(1-\bar\alpha_t)I), \]

and

\[ L_{\text{simple}}(\theta)= \mathbb E\left[\|\epsilon- \epsilon_\theta(x_t,t)\|^2\right]. \]

From a statistics viewpoint, diffusion models learn conditional expectations, scores, reverse Markov kernels, and posterior samplers. From an applied mathematics viewpoint, they are learned stochastic or deterministic dynamical systems. From a computer vision viewpoint, they provide a powerful framework for generation, editing, inverse problems, and multimodal visual reasoning.

30.47 References

Core sources for this chapter include the original diffusion probabilistic modeling line of work, denoising diffusion probabilistic models, score-based generative modeling, DDIM sampling, classifier-free guidance, latent diffusion, and modern text-to-image diffusion systems (Sohl-Dickstein et al. 2015; Ho et al. 2020; Y. Song et al. 2021; J. Song et al. 2021; Dhariwal and Nichol 2021; Ho and Salimans 2022; Rombach et al. 2022; Nichol and Dhariwal 2021).

Dhariwal, Prafulla, and Alexander Nichol. 2021. “Diffusion Models Beat GANs on Image Synthesis.” Advances in Neural Information Processing Systems 34: 8780–94. https://arxiv.org/abs/2105.05233.

Ho, Jonathan, Ajay Jain, and Pieter Abbeel. 2020. “Denoising Diffusion Probabilistic Models.” Advances in Neural Information Processing Systems 33: 6840–51. https://arxiv.org/abs/2006.11239.

Ho, Jonathan, and Tim Salimans. 2022. “Classifier-Free Diffusion Guidance.” arXiv Preprint arXiv:2207.12598. https://arxiv.org/abs/2207.12598.

Nichol, Alexander Quinn, and Prafulla Dhariwal. 2021. “Improved Denoising Diffusion Probabilistic Models.” International Conference on Machine Learning, 8162–71. https://arxiv.org/abs/2102.09672.

Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. “High-Resolution Image Synthesis with Latent Diffusion Models.” IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10684–95. https://arxiv.org/abs/2112.10752.

Sohl-Dickstein, Jascha, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. “Deep Unsupervised Learning Using Nonequilibrium Thermodynamics.” International Conference on Machine Learning, 2256–65. https://arxiv.org/abs/1503.03585.

Song, Jiaming, Chenlin Meng, and Stefano Ermon. 2021. “Denoising Diffusion Implicit Models.” International Conference on Learning Representations. https://arxiv.org/abs/2010.02502.

Song, Yang, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2021. “Score-Based Generative Modeling Through Stochastic Differential Equations.” International Conference on Learning Representations. https://arxiv.org/abs/2011.13456.

# Diffusion Models for Images ::: {.hero-box} Chapter 29 studies **diffusion models for images** from a mathematical point of view. Diffusion models are probabilistic generative models that learn to reverse a gradual noising process. For computer vision, they provide a common mathematical language for image generation, denoising, inpainting, super-resolution, editing, text-to-image synthesis, inverse problems, and uncertainty-aware reconstruction. ::: The previous chapters built the ingredients needed here. Chapter 18 introduced likelihoods, priors, MAP estimation, and probabilistic image models. Chapter 20 studied representations. Chapter 22 studied training by stochastic optimization. Chapter 25 studied transformer-based architectures. Chapter 27 studied vision-language conditioning. Chapter 28 studied prompt-conditioned foundation models. This chapter combines those ideas into one probabilistic framework. A diffusion model begins with a clean image $x_0$ sampled from the unknown data distribution $p_{\text{data}}$. It defines a forward Markov chain that gradually adds noise, $$ q(x_t \mid x_{t-1}), \qquad t=1,\dots,T, $$ until $x_T$ is close to a simple distribution such as a standard Gaussian. The model then learns a reverse chain $$ p_\theta(x_{t-1}\mid x_t) $$ that turns noise into an image. The essential statistical problem is therefore: $$ \text{learn the reverse dynamics of a known noising process from samples of clean images.} $$ This idea turns high-dimensional image generation into a sequence of smaller denoising problems. It also turns many computer vision tasks into conditional sampling problems. ## Learning Goals After reading this chapter, students should be able to: 1. define the forward diffusion process and compute the closed-form distribution $q(x_t\mid x_0)$; 2. explain why the reverse process is generally unknown and must be learned; 3. derive the common denoising objective used in DDPM-style models; 4. connect denoising score matching to the score function $\nabla_x \log p_t(x)$; 5. distinguish $\epsilon$-prediction, $x_0$-prediction, and velocity prediction; 6. explain DDPM, DDIM, score-based SDE, and probability-flow ODE viewpoints; 7. formulate conditional diffusion for class labels, text, masks, depth, and other visual prompts; 8. explain classifier guidance and classifier-free guidance; 9. describe latent diffusion as diffusion in a learned representation space; 10. implement small mathematical experiments for noising, denoising, sampling, guidance, and inverse problems; 11. analyze strengths, limitations, statistical assumptions, and failure modes of diffusion models in vision. ## 29.1 Generative Modeling for Images A generative model tries to approximate an unknown distribution on images. Let $$ X \in \mathbb R^d, $$ where $d=HWC$ for an image with height $H$, width $W$, and $C$ channels. The data distribution is $$ p_{\text{data}}(x). $$ A generative model constructs a distribution $p_\theta(x)$ that should be close to $p_{\text{data}}(x)$. The central question is how to represent and sample from $p_\theta$. Classical choices include: - explicit likelihood models, where $p_\theta(x)$ can be evaluated; - latent-variable models, where $x$ is generated from a latent variable $z$; - adversarial models, where a generator competes with a discriminator; - flow models, where an invertible map transforms Gaussian noise into data; - autoregressive models, where pixels or tokens are generated sequentially; - diffusion models, where a stochastic denoising process transforms noise into data. Diffusion models are especially attractive in vision because the intermediate states have a simple interpretation: noisy versions of images. The model learns a family of denoisers indexed by noise level. ## 29.2 The Forward Noising Process Let $x_0\sim p_{\text{data}}$. A standard DDPM-style forward process is the Markov chain $$ q(x_t\mid x_{t-1}) =\mathcal N\left(\sqrt{1-\beta_t}x_{t-1},\,\beta_t I\right), \qquad t=1,\dots,T, $$ where $$ 0<\beta_t<1. $$ The numbers $\beta_t$ form the **noise schedule**. Define $$ \alpha_t=1-\beta_t, \qquad \bar\alpha_t=\prod_{s=1}^t \alpha_s. $$ Then the forward transition can be written as $$ x_t=\sqrt{\alpha_t}x_{t-1}+\sqrt{1-\alpha_t}\epsilon_t, \qquad \epsilon_t\sim\mathcal N(0,I). $$ The process gradually shrinks the signal and adds independent Gaussian noise. ## 29.3 Closed Form of $q(x_t\mid x_0)$ A key algebraic fact is that $x_t$ can be sampled directly from $x_0$ without simulating all previous steps: $$ q(x_t\mid x_0) =\mathcal N\left(\sqrt{\bar\alpha_t}x_0,\,(1-\bar\alpha_t)I\right). $$ Equivalently, $$ x_t=\sqrt{\bar\alpha_t}x_0+ \sqrt{1-\bar\alpha_t}\epsilon, \qquad \epsilon\sim\mathcal N(0,I). $$ This formula is the computational heart of diffusion training. We can choose a random time $t$, draw Gaussian noise $\epsilon$, form $x_t$, and train a model to predict something about the clean image or the noise. ### Proof by induction For $t=1$, the formula follows from the definition. Suppose $$ x_{t-1}=\sqrt{\bar\alpha_{t-1}}x_0+ \sqrt{1-\bar\alpha_{t-1}}\epsilon' $$ with $\epsilon'\sim\mathcal N(0,I)$. Then $$ \begin{aligned} x_t &=\sqrt{\alpha_t}x_{t-1}+\sqrt{1-\alpha_t}\epsilon_t \\ &=\sqrt{\alpha_t\bar\alpha_{t-1}}x_0 +\sqrt{\alpha_t(1-\bar\alpha_{t-1})}\epsilon' +\sqrt{1-\alpha_t}\epsilon_t. \end{aligned} $$ The two Gaussian noise terms are independent and isotropic. Their sum is isotropic Gaussian with variance $$ \alpha_t(1-\bar\alpha_{t-1})+(1-\alpha_t) =1-\alpha_t\bar\alpha_{t-1} =1-\bar\alpha_t. $$ Thus $$ x_t=\sqrt{\bar\alpha_t}x_0+ \sqrt{1-\bar\alpha_t}\epsilon. $$ ## 29.4 Signal-to-Noise Ratio The coefficient $\bar\alpha_t$ controls how much signal remains at time $t$. A useful quantity is the signal-to-noise ratio $$ \operatorname{SNR}(t)=\frac{\bar\alpha_t}{1-\bar\alpha_t}. $$ Early times have large SNR: $x_t$ is mostly image. Late times have small SNR: $x_t$ is mostly noise. Training over many $t$ values teaches the network to denoise at many noise scales. Noise schedules are not merely implementation details. They determine which denoising problems are emphasized during training. Common choices include linear schedules, cosine schedules, and schedules expressed directly in log-SNR. ## 29.5 Interactive Demo: Forward Noising and Noise Schedule The following demo compares how $\bar\alpha_t$ and SNR change under different noise schedules. <div id="ch29-forward-noising" class="plotly-demo"></div> <script> (function(){ const id='ch29-forward-noising'; if(!window.Plotly || !document.getElementById(id)) return; const T=200; const ts=Array.from({length:T},(_,i)=>i+1); function cumprod(arr){let p=1; return arr.map(v=>{p*=v; return p;});} const betaLinear=ts.map(t=>1e-4 + (0.02-1e-4)*(t-1)/(T-1)); const abarLinear=cumprod(betaLinear.map(b=>1-b)); function abarCos(t){ const s=0.008; const f=Math.cos(((t/T+s)/(1+s))*Math.PI/2)**2; const f0=Math.cos((s/(1+s))*Math.PI/2)**2; return f/f0; } const abarCosine=ts.map(t=>abarCos(t)); const snrLin=abarLinear.map(a=>a/(1-a)); const snrCos=abarCosine.map(a=>a/(1-a)); Plotly.newPlot(id,[ {x:ts,y:abarLinear,mode:'lines',name:'linear beta: alpha_bar'}, {x:ts,y:abarCosine,mode:'lines',name:'cosine: alpha_bar'}, {x:ts,y:snrLin.map(v=>Math.log10(v)),mode:'lines',name:'linear beta: log10 SNR',yaxis:'y2'}, {x:ts,y:snrCos.map(v=>Math.log10(v)),mode:'lines',name:'cosine: log10 SNR',yaxis:'y2'} ],{ title:'Forward noising schedule', xaxis:{title:'time step t'}, yaxis:{title:'remaining signal alpha_bar',range:[0,1]}, yaxis2:{title:'log10 SNR',overlaying:'y',side:'right'}, margin:{t:55} },{responsive:true}); })(); </script> ## 29.6 The Reverse Process If the forward process destroys structure, generation requires reversing it: $$ p_\theta(x_{0:T})=p(x_T)\prod_{t=1}^T p_\theta(x_{t-1}\mid x_t), $$ where typically $$ p(x_T)=\mathcal N(0,I). $$ A common parameterization is $$ p_\theta(x_{t-1}\mid x_t) =\mathcal N\left(\mu_\theta(x_t,t),\Sigma_\theta(x_t,t)\right). $$ The forward transition $q(x_t\mid x_{t-1})$ is known. The reverse transition is not known because it depends on the unknown data distribution. Learning $p_\theta$ is the main task. The reverse process is not simply subtracting Gaussian noise. At late times, many images could have produced the same noisy observation. The model must use learned image priors to decide which denoising direction is plausible. ## 29.7 The Exact Posterior $q(x_{t-1}\mid x_t,x_0)$ Although $q(x_{t-1}\mid x_t)$ is unknown, the conditional posterior given both $x_t$ and $x_0$ is tractable: $$ q(x_{t-1}\mid x_t,x_0) =\mathcal N\left(\tilde\mu_t(x_t,x_0),\tilde\beta_t I\right), $$ where $$ \tilde\beta_t =\frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t, $$ and $$ \tilde\mu_t(x_t,x_0) = \frac{\sqrt{\bar\alpha_{t-1}}\beta_t}{1-\bar\alpha_t}x_0 + \frac{\sqrt{\alpha_t}(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}x_t. $$ This formula is useful because if a network can estimate $x_0$ from $x_t$, then it can estimate the reverse mean. ## 29.8 Predicting Noise Instead of directly predicting $x_0$, DDPMs commonly train a neural network $\epsilon_\theta(x_t,t)$ to predict the noise $\epsilon$ in $$ x_t=\sqrt{\bar\alpha_t}x_0+ \sqrt{1-\bar\alpha_t}\epsilon. $$ Solving for $x_0$ gives $$ \widehat x_0(x_t,t)= \frac{x_t-\sqrt{1-\bar\alpha_t}\epsilon_\theta(x_t,t)} {\sqrt{\bar\alpha_t}}. $$ Thus noise prediction, clean-image prediction, and reverse-mean prediction are closely related. The simplified training loss is $$ L_{\text{simple}}(\theta) =\mathbb E_{x_0,\epsilon,t} \left[\left\|\epsilon- \epsilon_\theta\left(\sqrt{\bar\alpha_t}x_0+ \sqrt{1-\bar\alpha_t}\epsilon,t\right) \right\|_2^2\right]. $$ This objective turns generative modeling into supervised regression: input a noisy image and time index, output the noise that was added. ## 29.9 Statistical Meaning of Noise Prediction For fixed $t$, minimizing squared error implies $$ \epsilon_\theta^*(x_t,t)=\mathbb E[\epsilon\mid x_t,t]. $$ Thus the network learns a conditional expectation. Since $$ x_t=\sqrt{\bar\alpha_t}x_0+ \sqrt{1-\bar\alpha_t}\epsilon, $$ noise prediction is equivalent to estimating the direction from noisy observation back toward the data manifold. The model is not memorizing pixels; it is learning a family of denoising vector fields. This connects diffusion to score matching. For the perturbed distribution $p_t(x)$ of $x_t$, the score is $$ \nabla_x \log p_t(x). $$ For the Gaussian perturbation model, the optimal noise predictor is related to the score by $$ \epsilon_\theta^*(x_t,t) \approx -\sqrt{1-\bar\alpha_t}\,\nabla_{x_t}\log p_t(x_t). $$ Thus a diffusion model can be viewed as learning the score field of noisy image distributions. ## 29.10 Interactive Demo: Score Field of a Noisy Distribution This toy example uses a one-dimensional mixture distribution and shows how increasing noise smooths the density and changes the score field. <div id="ch29-score-field" class="plotly-demo"></div> <script> (function(){ const id='ch29-score-field'; if(!window.Plotly || !document.getElementById(id)) return; const xs=Array.from({length:350},(_,i)=>-5+10*i/349); const sigmas=[0.15,0.35,0.70,1.20,2.00]; function normal(x,m,s){return Math.exp(-0.5*((x-m)/s)**2)/(s*Math.sqrt(2*Math.PI));} function density(x,s){ const v1=Math.sqrt(0.25+s*s), v2=Math.sqrt(0.36+s*s), v3=Math.sqrt(0.49+s*s); return 0.35*normal(x,-2.2,v1)+0.45*normal(x,0.4,v2)+0.20*normal(x,2.4,v3); } function score(x,s){ const comps=[[-2.2,Math.sqrt(0.25+s*s),0.35],[0.4,Math.sqrt(0.36+s*s),0.45],[2.4,Math.sqrt(0.49+s*s),0.20]]; let p=0, dp=0; comps.forEach(c=>{ const [m,v,w]=c; const n=w*normal(x,m,v); p+=n; dp+=n*(-(x-m)/(v*v)); }); return dp/Math.max(p,1e-12); } const frames=sigmas.map((s,k)=>({ name:String(k), data:[ {x:xs,y:xs.map(x=>density(x,s))}, {x:xs,y:xs.map(x=>score(x,s))} ], layout:{title:'Noisy density and score field, noise sigma = '+s.toFixed(2)} })); Plotly.newPlot(id,[ {x:xs,y:xs.map(x=>density(x,sigmas[0])),mode:'lines',name:'density p_t(x)'}, {x:xs,y:xs.map(x=>score(x,sigmas[0])),mode:'lines',name:'score d log p_t / dx',yaxis:'y2'} ],{ title:'Noisy density and score field, noise sigma = '+sigmas[0].toFixed(2), xaxis:{title:'x'}, yaxis:{title:'density'}, yaxis2:{title:'score',overlaying:'y',side:'right'}, updatemenus:[{ type:'buttons', x:0.05,y:1.15, buttons:[ {label:'play',method:'animate',args:[null,{frame:{duration:900,redraw:true},fromcurrent:true}]}, {label:'pause',method:'animate',args:[[null],{mode:'immediate',frame:{duration:0,redraw:false}}]} ] }], sliders:[{ active:0, steps:sigmas.map((s,k)=>({label:s.toFixed(2),method:'animate',args:[[String(k)],{mode:'immediate',frame:{duration:0,redraw:true}}]})) }], margin:{t:60} },{responsive:true}); Plotly.addFrames(id,frames); })(); </script> ## 29.11 Evidence Lower Bound Viewpoint Diffusion models can be derived from variational inference. The negative log-likelihood satisfies an evidence lower bound: $$ -\log p_\theta(x_0) \leq \mathbb E_q\left[ -\log p_\theta(x_{0:T})+ \log q(x_{1:T}\mid x_0) \right]. $$ This decomposes into terms comparing the true reverse posterior $q(x_{t-1}\mid x_t,x_0)$ to the learned reverse model $p_\theta(x_{t-1}\mid x_t)$, plus reconstruction and prior terms. In simplified form, one obtains a weighted sum of denoising losses. The simplified noise-prediction loss is not arbitrary. It is a practical version of a likelihood-based objective. The weighting of different times $t$ determines which noise levels dominate training. ## 29.12 Three Common Parameterizations Diffusion models often differ in what the network predicts. ### Noise prediction The network predicts $\epsilon$: $$ \epsilon_\theta(x_t,t)\approx \epsilon. $$ This is simple and stable. ### Clean-image prediction The network predicts $x_0$: $$ x_\theta(x_t,t)\approx x_0. $$ This can make sampling formulas interpretable because one directly estimates the clean image. ### Velocity prediction Some models predict a velocity-like variable $$ v=\sqrt{\bar\alpha_t}\epsilon- \sqrt{1-\bar\alpha_t}x_0. $$ Velocity prediction often balances low-noise and high-noise training regimes. It is especially useful when schedules are expressed in angular or log-SNR form. All three parameterizations are algebraically convertible if $x_t$ and $t$ are known. ## 29.13 Time Embeddings The denoising network must know the noise level. It receives $t$ or a continuous noise parameter as input. A common sinusoidal embedding is $$ \gamma(t)= (\sin(\omega_1 t),\cos(\omega_1 t),\dots, \sin(\omega_m t),\cos(\omega_m t)). $$ This is analogous to positional encoding in transformers. The embedding is usually processed by a small multilayer perceptron and injected into residual blocks through additive or affine transformations. From a mathematical point of view, the model approximates a function $$ f_\theta:\mathbb R^d\times [0,T]\to \mathbb R^d. $$ The time variable indexes a continuum of denoising problems. ## 29.14 U-Net Denoisers Image diffusion models commonly use U-Net-like denoisers. The network receives a noisy image $x_t$ and a time embedding. It outputs a tensor of the same spatial shape as $x_t$. A schematic form is $$ \epsilon_\theta(x_t,t)=D_\theta(E_\theta(x_t,t),t), $$ where $E_\theta$ is an encoder and $D_\theta$ is a decoder with skip connections. The architecture is well matched to images because: - local convolution captures texture and edges; - downsampling captures global structure; - skip connections preserve spatial detail; - attention blocks capture long-range relationships; - time embeddings condition every denoising scale. Modern diffusion systems may use convolutional U-Nets, transformers, or hybrid architectures. ## 29.15 DDPM Sampling After training, sampling starts from Gaussian noise $$ x_T\sim\mathcal N(0,I). $$ For $t=T,T-1,\dots,1$, one computes a reverse mean using $\epsilon_\theta(x_t,t)$ and samples $$ x_{t-1}=\mu_\theta(x_t,t)+\sigma_t z, \qquad z\sim\mathcal N(0,I). $$ A common formula for the mean is $$ \mu_\theta(x_t,t)= \frac{1}{\sqrt{\alpha_t}} \left( x_t- \frac{\beta_t}{\sqrt{1-\bar\alpha_t}} \epsilon_\theta(x_t,t) \right). $$ The stochastic term $\sigma_tz$ preserves sampling diversity. If $\sigma_t=0$, the process becomes deterministic. ## 29.16 DDIM Sampling DDIM sampling constructs a non-Markovian reverse process that can sample with fewer steps. Given an estimate $\widehat x_0$, one can write $$ x_{t-1} = \sqrt{\bar\alpha_{t-1}}\widehat x_0 +\sqrt{1-\bar\alpha_{t-1}}\widehat\epsilon $$ for deterministic sampling, where $$ \widehat\epsilon= \epsilon_\theta(x_t,t). $$ More general DDIM samplers include a parameter controlling stochasticity. The important point is that generation can follow a deterministic path through latent space, making image editing and interpolation more controllable. ## 29.17 Interactive Demo: Stochastic vs. Deterministic Sampling This toy visualization compares several reverse trajectories in a two-dimensional toy distribution. Stochastic sampling can explore multiple modes; deterministic sampling follows smoother paths. <div id="ch29-sampling-trajectories" class="plotly-demo"></div> <script> (function(){ const id='ch29-sampling-trajectories'; if(!window.Plotly || !document.getElementById(id)) return; function path(seed, stochastic){ let x=3*Math.cos(seed), y=3*Math.sin(seed); const xs=[x], ys=[y]; const centers=[[-2,-1],[0.4,1.7],[2.1,-0.8]]; for(let k=0;k<40;k++){ let weights=centers.map(c=>Math.exp(-0.7*((x-c[0])**2+(y-c[1])**2))); const s=weights.reduce((a,b)=>a+b,0); weights=weights.map(w=>w/s); let tx=0,ty=0; centers.forEach((c,i)=>{tx+=weights[i]*c[0]; ty+=weights[i]*c[1];}); const eta=stochastic ? 0.18*(1-k/45) : 0.0; x += 0.10*(tx-x) + eta*Math.sin(13.1*k+seed); y += 0.10*(ty-y) + eta*Math.cos(9.7*k+seed); xs.push(x); ys.push(y); } return {xs,ys}; } const traces=[]; for(let i=0;i<5;i++){ const p=path(i+0.6,true); traces.push({x:p.xs,y:p.ys,mode:'lines+markers',name:'stochastic '+(i+1),marker:{size:4}}); } for(let i=0;i<3;i++){ const p=path(i+1.2,false); traces.push({x:p.xs,y:p.ys,mode:'lines',name:'deterministic '+(i+1),line:{dash:'dash'}}); } traces.push({x:[-2,0.4,2.1],y:[-1,1.7,-0.8],mode:'markers',name:'data modes',marker:{size:16,symbol:'x'}}); Plotly.newPlot(id,traces,{ title:'Toy reverse sampling trajectories', xaxis:{title:'coordinate 1',range:[-4,4]}, yaxis:{title:'coordinate 2',range:[-3,3],scaleanchor:'x'}, margin:{t:55} },{responsive:true}); })(); </script> ## 29.18 Score-Based SDE Viewpoint Diffusion can also be formulated in continuous time. A forward stochastic differential equation has the form $$ dx=f(x,t)dt+g(t)dw, $$ where $w$ is Brownian motion. The reverse-time SDE is $$ dx=\left[f(x,t)-g(t)^2\nabla_x\log p_t(x)\right]dt+g(t)d\bar w, $$ where time runs backward and $\bar w$ is reverse-time Brownian motion. Thus if one learns the score $$ s_\theta(x,t)\approx \nabla_x\log p_t(x), $$ one can simulate the reverse SDE to generate samples. This viewpoint clarifies the connection among diffusion models, score matching, Langevin dynamics, and stochastic processes. ## 29.19 Probability-Flow ODE Associated with the reverse SDE is a deterministic probability-flow ODE: $$ \frac{dx}{dt}=f(x,t)-\frac{1}{2}g(t)^2\nabla_x\log p_t(x). $$ This ODE has the same marginal distributions as the corresponding SDE. It supports deterministic sampling and likelihood computation by change of variables. For applied mathematics students, this is an important bridge: diffusion models can be studied as numerical solvers for learned vector fields. ## 29.20 Numerical Solvers and Discretization Error Sampling is numerical integration. The learned denoiser or score gives a vector field, and a sampler approximates its flow. Important questions include: - How many steps are needed? - Is the solver explicit or implicit? - Is the trajectory stochastic or deterministic? - How does discretization error affect image quality? - How do errors accumulate over time? Modern fast samplers use higher-order ODE/SDE solvers, carefully designed noise schedules, and distillation methods. The mathematical issues resemble those in numerical differential equations: stability, local truncation error, stiffness, and error propagation. ## 29.21 Conditional Diffusion Most vision applications are conditional. The model generates or reconstructs an image given extra information $c$: $$ p_\theta(x_0\mid c). $$ The condition $c$ may be: - a class label; - a text prompt; - a segmentation mask; - an edge map; - a depth map; - a low-resolution image; - a corrupted image; - a bounding box; - another image; - a prompt embedding from a vision-language model. The denoising network becomes $$ \epsilon_\theta(x_t,t,c). $$ Conditional diffusion turns many computer vision tasks into conditional denoising. ## 29.22 Classifier Guidance Suppose a classifier estimates $p_\phi(c\mid x_t)$. To sample images matching class $c$, one can guide the reverse process using $$ \nabla_{x_t}\log p_\phi(c\mid x_t). $$ Intuitively, the sampler follows both the generative score and a classifier gradient pointing toward images with the desired label. A guided score has the schematic form $$ \widetilde s(x_t,t,c) =s_\theta(x_t,t)+w\nabla_{x_t}\log p_\phi(c\mid x_t), $$ where $w\ge 0$ is a guidance scale. Classifier guidance improves conditioning but requires a noise-aware classifier and can introduce artifacts when the guidance is too strong. ## 29.23 Classifier-Free Guidance Classifier-free guidance avoids training a separate classifier. During training, the condition $c$ is randomly dropped. The same network learns both conditional and unconditional denoising: $$ \epsilon_\theta(x_t,t,c), \qquad \epsilon_\theta(x_t,t,\varnothing). $$ At sampling time, one uses $$ \widehat\epsilon_{\text{cfg}} = \epsilon_\theta(x_t,t,\varnothing) +w\left( \epsilon_\theta(x_t,t,c)-\epsilon_\theta(x_t,t,\varnothing) \right). $$ Here $w$ is the guidance scale. Larger $w$ usually improves prompt adherence but can reduce diversity and create over-saturated or distorted images. ## 29.24 Interactive Demo: Classifier-Free Guidance Trade-Off The toy model below illustrates the trade-off between unconditional diversity and conditional concentration. <div id="ch29-guidance" class="plotly-demo"></div> <script> (function(){ const id='ch29-guidance'; if(!window.Plotly || !document.getElementById(id)) return; const ws=[0,0.5,1,2,4,7]; function points(w){ const xs=[], ys=[]; const N=180; for(let i=0;i<N;i++){ const a=2*Math.PI*i/N; const r=1.8+0.35*Math.sin(3*a+0.5*i); const mix=Math.min(w/4,1); const target=[1.8,1.2]; xs.push((1-mix)*r*Math.cos(a)+mix*(target[0]+0.35*Math.cos(a)*(1/(1+w)))); ys.push((1-mix)*r*Math.sin(a)+mix*(target[1]+0.35*Math.sin(a)*(1/(1+w)))); } return {xs,ys}; } const frames=ws.map((w,k)=>{ const p=points(w); const diversity=Math.exp(-0.23*w); const adherence=1-Math.exp(-0.55*w); return {name:String(k),data:[ {x:p.xs,y:p.ys}, {x:[w],y:[adherence]}, {x:[w],y:[diversity]} ],layout:{title:'Classifier-free guidance scale w = '+w}} }); const p0=points(ws[0]); Plotly.newPlot(id,[ {x:p0.xs,y:p0.ys,mode:'markers',name:'samples',marker:{size:5,opacity:0.65}}, {x:[0],y:[0],mode:'markers',name:'prompt adherence',xaxis:'x2',yaxis:'y2',marker:{size:12}}, {x:[0],y:[1],mode:'markers',name:'diversity',xaxis:'x2',yaxis:'y2',marker:{size:12}} ],{ title:'Classifier-free guidance scale w = 0', grid:{rows:1,columns:2,pattern:'independent'}, xaxis:{title:'sample coordinate 1',range:[-3,3]}, yaxis:{title:'sample coordinate 2',range:[-3,3],scaleanchor:'x'}, xaxis2:{title:'guidance scale',range:[0,7]}, yaxis2:{title:'score',range:[0,1]}, sliders:[{ active:0, steps:ws.map((w,k)=>({label:String(w),method:'animate',args:[[String(k)],{mode:'immediate',frame:{duration:0,redraw:true}}]})) }], margin:{t:60} },{responsive:true}); Plotly.addFrames(id,frames); })(); </script> ## 29.25 Text-to-Image Diffusion In text-to-image generation, the condition $c$ is a text embedding. A text encoder maps a prompt to vectors: $$ \tau = T_\psi(\text{text}). $$ The denoising network uses $\tau$ through cross-attention: $$ \operatorname{Attention}(Q,K,V) =\operatorname{softmax}\left(\frac{QK^T}{\sqrt d}\right)V, $$ where image features produce queries and text tokens produce keys and values. This allows each image location or latent token to attend to words or phrases. Text-to-image diffusion combines three modeling components: 1. a text representation model; 2. a visual denoising model; 3. a sampling algorithm with guidance. The mathematical challenge is not only generation but alignment: the generated image should match the conditional distribution implied by the text prompt. ## 29.26 Latent Diffusion Pixel-space diffusion is expensive because images are high dimensional. Latent diffusion first trains an autoencoder: $$ z=E_\phi(x), \qquad \widehat x=D_\phi(z), $$ where $z$ has lower spatial resolution or fewer channels than $x$. Diffusion is then performed in latent space: $$ z_t=\sqrt{\bar\alpha_t}z_0+\sqrt{1-\bar\alpha_t}\epsilon. $$ The final image is decoded by $D_\phi$. Latent diffusion changes the modeling problem. The diffusion model learns the distribution of latent codes, not raw pixels. This improves efficiency but makes the autoencoder an important part of the generative prior. ## 29.27 Interactive Demo: Pixel Diffusion vs. Latent Diffusion Cost This demo compares the rough dimensional cost of doing diffusion in pixel space versus a compressed latent grid. <div id="ch29-latent-cost" class="plotly-demo"></div> <script> (function(){ const id='ch29-latent-cost'; if(!window.Plotly || !document.getElementById(id)) return; const sizes=[64,128,256,512,768,1024]; const pixel=sizes.map(s=>s*s*3/1e6); const latent4=sizes.map(s=>(s/4)*(s/4)*4/1e6); const latent8=sizes.map(s=>(s/8)*(s/8)*4/1e6); const attention=sizes.map(s=>Math.pow((s/16)*(s/16),2)/1e6); Plotly.newPlot(id,[ {x:sizes,y:pixel,mode:'lines+markers',name:'pixel tensor dimension, 3 channels'}, {x:sizes,y:latent4,mode:'lines+markers',name:'latent, factor 4, 4 channels'}, {x:sizes,y:latent8,mode:'lines+markers',name:'latent, factor 8, 4 channels'}, {x:sizes,y:attention,mode:'lines+markers',name:'quadratic token cost at 16x patches',yaxis:'y2'} ],{ title:'Rough dimensional cost: pixel space vs. latent space', xaxis:{title:'image side length'}, yaxis:{title:'millions of scalar variables'}, yaxis2:{title:'millions of token pairs',overlaying:'y',side:'right'}, margin:{t:55} },{responsive:true}); })(); </script> ## 29.28 Diffusion for Inverse Problems Many vision tasks have the form $$ y=A(x_0)+\eta, $$ where $A$ is a measurement operator and $\eta$ is noise. Examples include: - super-resolution; - deblurring; - inpainting; - compressed sensing; - MRI or CT reconstruction; - low-light enhancement; - depth-guided reconstruction. Diffusion models can serve as learned priors. A posterior sampling problem is $$ p(x_0\mid y)\propto p(y\mid x_0)p(x_0). $$ The diffusion prior supplies $p(x_0)$ implicitly through the learned denoising process. The measurement likelihood supplies consistency with the observation. A schematic guided score is $$ \nabla_x\log p_t(x\mid y) \approx s_\theta(x,t)+\lambda \nabla_x\log p(y\mid x). $$ This unifies generative modeling and classical inverse problems. ## 29.29 Inpainting and Mask Constraints For inpainting, the measurement is $$ y=M\odot x_0, $$ where $M\in\{0,1\}^d$ is a known mask. The unknown region is $(1-M)\odot x_0$. A diffusion inpainting sampler must preserve known pixels while sampling plausible missing pixels. A simple consistency step is $$ x_t \leftarrow M\odot y_t+(1-M)\odot x_t, $$ where $y_t$ is the noised version of the known observation at time $t$. This shows a useful principle: constraints can be enforced during sampling, not only during training. ## 29.30 Super-Resolution In super-resolution, one observes a low-resolution image $$ y=D x_0+\eta, $$ where $D$ is a downsampling operator. The goal is to sample from $$ p(x_0\mid y). $$ The problem is ill-posed: many high-resolution images can produce the same low-resolution observation. Diffusion models are natural here because they represent a conditional distribution rather than a single deterministic answer. For statistics students, this is important: perceptually plausible super-resolution is not the same as estimating the conditional mean. A conditional mean may be blurry, while samples from the posterior can be sharp and diverse. ## 29.31 Interactive Demo: Conditional Posterior Ambiguity This toy visualization compares a conditional mean with multiple posterior samples. In image reconstruction, the mean can be smoother than any plausible sample. <div id="ch29-posterior-ambiguity" class="plotly-demo"></div> <script> (function(){ const id='ch29-posterior-ambiguity'; if(!window.Plotly || !document.getElementById(id)) return; const xs=Array.from({length:180},(_,i)=>i/179); const samples=[]; for(let k=0;k<6;k++){ const phase=0.7*k; samples.push(xs.map(x=>0.4+0.25*Math.sin(2*Math.PI*(2*x)+phase)+0.08*Math.sin(2*Math.PI*(7*x)+1.3*k))); } const mean=xs.map((x,i)=>samples.reduce((a,s)=>a+s[i],0)/samples.length); const low=xs.filter((_,i)=>i%15===0); const lowy=low.map(x=>0.4+0.25*Math.sin(2*Math.PI*(2*x))); const traces=samples.map((s,k)=>({x:xs,y:s,mode:'lines',name:'posterior sample '+(k+1),opacity:0.45})); traces.push({x:xs,y:mean,mode:'lines',name:'conditional mean',line:{width:4}}); traces.push({x:low,y:lowy,mode:'markers',name:'low-resolution observations',marker:{size:9}}); Plotly.newPlot(id,traces,{ title:'Posterior samples versus conditional mean', xaxis:{title:'position'}, yaxis:{title:'intensity'}, margin:{t:55} },{responsive:true}); })(); </script> ## 29.32 Diffusion and Image Editing Diffusion models can edit images by combining noising, conditioning, and denoising. A typical procedure is: 1. map an input image to an intermediate noisy state; 2. modify the condition or prompt; 3. run the reverse process while preserving selected structure. The amount of noise controls the size of the edit. Low noise preserves the image closely. High noise allows large semantic changes. Mathematically, editing is sampling from a conditional distribution with constraints: $$ p(x_{\text{edited}}\mid x_{\text{input}},c,\text{constraints}). $$ This is close in spirit to classical constrained optimization, but the learned prior is much richer. ## 29.33 Evaluation of Diffusion Models Evaluating generative image models is difficult. Common criteria include: - sample quality; - diversity; - coverage of the data distribution; - conditional alignment; - perceptual realism; - factual consistency; - fairness and bias; - robustness to prompt changes; - downstream utility. Common metrics include FID, precision/recall for generative models, CLIP-style alignment scores, human preference studies, and task-specific measures. None is complete. A statistical warning: a model can produce visually impressive samples while failing to represent the true data distribution in important regions. ## 29.34 Calibration and Uncertainty Diffusion models produce samples, so they can represent uncertainty. For a conditional task, one can draw $$ x_0^{(1)},\dots,x_0^{(m)}\sim p_\theta(x_0\mid y) $$ and estimate posterior summaries: $$ \widehat\mu(y)=\frac1m\sum_{j=1}^m x_0^{(j)}, $$ $$ \widehat\Sigma(y)=\frac1{m-1}\sum_{j=1}^m (x_0^{(j)}-\widehat\mu)(x_0^{(j)}-\widehat\mu)^T. $$ For images, one usually visualizes marginal variance maps, uncertainty over edges, or uncertainty over downstream predictions. However, sample diversity is not automatically calibrated uncertainty. The sampler, guidance scale, conditioning, and training distribution all affect the apparent uncertainty. ## 29.35 Failure Modes Diffusion models have several mathematically interpretable failure modes. ### Sampling error Finite-step samplers approximate continuous or discrete reverse dynamics. Too few steps may introduce artifacts. ### Score error The learned score may be inaccurate in low-density regions. Sampling trajectories can pass through such regions. ### Guidance over-optimization Large guidance can overemphasize prompt adherence and reduce naturalness or diversity. ### Mode imbalance The model may underrepresent rare visual configurations in the training distribution. ### Prompt ambiguity A text prompt may correspond to a broad or multimodal conditional distribution. ### Dataset bias The learned prior reflects the data distribution and its biases. ### Hallucination in inverse problems When measurements are weak, the learned prior may invent plausible but incorrect details. This is especially important in medical imaging, scientific imaging, and legal or forensic settings. ## 29.36 Python Example: Forward Diffusion on an Image The following example shows how to sample noisy versions of an image at different time steps. ```{python} #| label: ch29-python-forward-diffusion #| eval: false import numpy as np import matplotlib.pyplot as plt rng = np.random.default_rng(29) H, W = 96, 96 yy, xx = np.mgrid[0:H, 0:W] # Synthetic clean image in [0, 1] x0 = ( 0.25 + 0.55 * np.exp(-((xx - 32) ** 2 + (yy - 45) ** 2) / (2 * 14 ** 2)) + 0.35 * np.exp(-((xx - 68) ** 2 + (yy - 55) ** 2) / (2 * 10 ** 2)) ) x0 = np.clip(x0, 0, 1) T = 1000 betas = np.linspace(1e-4, 0.02, T) alphas = 1.0 - betas alpha_bars = np.cumprod(alphas) def q_sample(x0, t): """Sample x_t from q(x_t | x_0). t is zero-based.""" eps = rng.normal(size=x0.shape) a_bar = alpha_bars[t] xt = np.sqrt(a_bar) * x0 + np.sqrt(1.0 - a_bar) * eps return xt, eps steps = [0, 50, 150, 300, 600, 999] fig, axes = plt.subplots(1, len(steps), figsize=(13, 3)) for ax, t in zip(axes, steps): xt, _ = q_sample(x0, t) ax.imshow(xt, cmap="gray") ax.set_title(f"t={t+1}") ax.axis("off") plt.tight_layout() plt.show() ``` ## 29.37 Python Example: One Training Batch for Noise Prediction The following code constructs one training batch for the simplified denoising loss. It is meant to clarify the mathematics, not to be a full training pipeline. ```{python} #| label: ch29-python-training-batch #| eval: false import numpy as np rng = np.random.default_rng(123) # Suppose x0_batch has shape (B, H, W, C) and values scaled to [-1, 1]. B, H, W, C = 16, 64, 64, 3 x0_batch = rng.normal(size=(B, H, W, C)).astype("float32") T = 1000 betas = np.linspace(1e-4, 0.02, T, dtype="float32") alphas = 1.0 - betas alpha_bars = np.cumprod(alphas) # Random time for each image. t = rng.integers(0, T, size=(B,)) eps = rng.normal(size=x0_batch.shape).astype("float32") # Broadcast alpha_bar to image shape. a_bar = alpha_bars[t].reshape(B, 1, 1, 1) xt = np.sqrt(a_bar) * x0_batch + np.sqrt(1.0 - a_bar) * eps # A neural network would predict eps_hat = model(xt, t). # The simplified DDPM loss is mean squared error. eps_hat = np.zeros_like(eps) # placeholder loss = np.mean((eps - eps_hat) ** 2) print("toy loss:", loss) ``` ## 29.38 TensorFlow Example: Tiny Denoising Network Skeleton This compact example shows the shape of a denoising model. A real diffusion model would use a much larger U-Net or transformer and better time conditioning. ```{python} #| label: ch29-tensorflow-denoiser #| eval: false import tensorflow as tf from tensorflow import keras from tensorflow.keras import layers class SinusoidalTimeEmbedding(layers.Layer): def __init__(self, dim=64, **kwargs): super().__init__(**kwargs) self.dim = dim def call(self, t): t = tf.cast(tf.reshape(t, [-1, 1]), tf.float32) half = self.dim // 2 freqs = tf.exp( tf.linspace(tf.math.log(1.0), tf.math.log(1000.0), half) ) args = t / freqs[None, :] return tf.concat([tf.sin(args), tf.cos(args)], axis=-1) def tiny_denoiser(image_shape=(64, 64, 3), time_dim=64): x_in = keras.Input(shape=image_shape, name="x_t") t_in = keras.Input(shape=(), dtype="int32", name="t") temb = SinusoidalTimeEmbedding(time_dim)(t_in) temb = layers.Dense(128, activation="swish")(temb) temb = layers.Dense(128, activation="swish")(temb) h = layers.Conv2D(64, 3, padding="same", activation="swish")(x_in) h = layers.Conv2D(64, 3, padding="same", activation="swish")(h) # Inject time information by adding a broadcast vector. tb = layers.Dense(64)(temb) tb = layers.Reshape((1, 1, 64))(tb) h = h + tb h = layers.Conv2D(64, 3, padding="same", activation="swish")(h) out = layers.Conv2D(image_shape[-1], 3, padding="same")(h) return keras.Model([x_in, t_in], out) model = tiny_denoiser() model.summary() ``` ## 29.39 Python Example: DDPM Reverse Step The following function implements a simplified reverse step using a predicted noise tensor. ```{python} #| label: ch29-python-ddpm-step #| eval: false import numpy as np rng = np.random.default_rng(7) def ddpm_step(x_t, t, eps_hat, betas, alpha_bars): """One DDPM reverse step from x_t to x_{t-1}. t is zero-based. This example uses fixed variance beta_t. """ beta_t = betas[t] alpha_t = 1.0 - beta_t a_bar_t = alpha_bars[t] mean = (1.0 / np.sqrt(alpha_t)) * ( x_t - beta_t / np.sqrt(1.0 - a_bar_t) * eps_hat ) if t == 0: return mean z = rng.normal(size=x_t.shape) return mean + np.sqrt(beta_t) * z ``` ## 29.40 Python Example: Classifier-Free Guidance Formula This example shows the algebra of classifier-free guidance for noise prediction. ```{python} #| label: ch29-python-classifier-free-guidance #| eval: false import numpy as np # Suppose a neural network has produced two predictions. eps_uncond = np.random.normal(size=(4, 64, 64, 3)) eps_cond = np.random.normal(size=(4, 64, 64, 3)) def classifier_free_guidance(eps_uncond, eps_cond, guidance_scale): return eps_uncond + guidance_scale * (eps_cond - eps_uncond) for w in [0.0, 1.0, 3.0, 7.5]: eps_guided = classifier_free_guidance(eps_uncond, eps_cond, w) print(w, eps_guided.shape, np.linalg.norm(eps_guided.ravel())) ``` ## 29.41 AI-Assisted Mathematical Study Prompts Use AI tools carefully as mathematical collaborators, not as authorities. Good prompts include: 1. **Forward process derivation.** Ask for a proof of the formula for $q(x_t\mid x_0)$, then verify every variance term yourself. 2. **Posterior derivation.** Ask for the Gaussian conditioning derivation of $q(x_{t-1}\mid x_t,x_0)$ and check dimensions. 3. **Score connection.** Ask why the optimal noise predictor is proportional to the score $\nabla_x\log p_t(x)$. 4. **Sampler comparison.** Ask for a comparison of DDPM, DDIM, and score-SDE sampling in terms of randomness, numerical integration, and likelihood. 5. **Guidance critique.** Ask for examples where classifier-free guidance improves prompt alignment but harms diversity. 6. **Inverse problem audit.** Ask how a diffusion prior might hallucinate details in medical image reconstruction, then propose safeguards. When using AI, always request explicit assumptions, equations, and dimensional checks. ## 29.42 Common Misconceptions **Misconception 1.** Diffusion models simply remove random noise. They learn a sequence of conditional denoising distributions shaped by the data distribution. Denoising depends on image priors. **Misconception 2.** More guidance is always better. Large guidance can reduce diversity, distort images, and over-optimize prompt features. **Misconception 3.** A generated image is a posterior mean. A diffusion sample is one draw from a model distribution. It is not necessarily a conditional expectation. **Misconception 4.** Latent diffusion is just faster pixel diffusion. Latent diffusion changes the data space. The autoencoder affects what details can be represented. **Misconception 5.** Better visual quality implies better statistical modeling. Perceptual quality, likelihood, diversity, calibration, and downstream utility are different criteria. ## 29.43 Mini-Project: Diffusion as a Learned Image Prior Design a small experiment on synthetic images. 1. Generate a dataset of simple shapes. 2. Define a forward noising process. 3. Train a small denoising network to predict Gaussian noise. 4. Compare denoising at low, medium, and high noise levels. 5. Use the trained denoiser in a simple inpainting or super-resolution task. 6. Evaluate reconstruction error, diversity, and failure cases. A mathematically strong report should include: - the data distribution; - the noise schedule; - the training objective; - the network architecture; - the sampling algorithm; - quantitative metrics; - visual examples; - a discussion of uncertainty and hallucination. ## 29.44 Exercises ### Conceptual Exercises 1. Explain why the forward diffusion process is easy to sample but the reverse process is hard. 2. Why does the direct formula for $q(x_t\mid x_0)$ make training efficient? 3. Explain the difference between $\epsilon$-prediction and $x_0$-prediction. 4. Why can diffusion models represent multiple plausible solutions to super-resolution? 5. Explain why text-to-image generation is a conditional distribution modeling problem. 6. Describe one reason why FID is not a complete measure of generative quality. ### Mathematical Exercises 1. Prove by induction that $$ q(x_t\mid x_0)= \mathcal N(\sqrt{\bar\alpha_t}x_0,(1-\bar\alpha_t)I). $$ 2. Starting from $$ x_t=\sqrt{\bar\alpha_t}x_0+ \sqrt{1-\bar\alpha_t}\epsilon, $$ solve for $x_0$ in terms of $x_t$ and $\epsilon$. 3. Show that the DDPM reverse mean formula can be written in terms of $\epsilon_\theta(x_t,t)$. 4. For a scalar Gaussian data distribution $x_0\sim\mathcal N(0,\sigma_0^2)$, compute the distribution of $x_t$. 5. Let $p_t$ be a one-dimensional density. Show that Langevin dynamics $$ x_{k+1}=x_k+\eta\nabla_x\log p_t(x_k)+\sqrt{2\eta}z_k $$ uses the score to move samples toward high-density regions while preserving randomness. 6. Derive the classifier-free guidance formula as an extrapolation from unconditional to conditional noise prediction. 7. Suppose a downsampling operator $D$ maps high-resolution images to low-resolution images. Explain why the inverse problem $y=Dx$ is not unique. ### Python Exercises 1. Implement linear and cosine noise schedules. Plot $\bar\alpha_t$ and SNR. 2. Generate noisy versions of a small image using the closed form $q(x_t\mid x_0)$. 3. Train a tiny denoising network on synthetic disks, squares, and triangles. 4. Compare MSE loss for predicting $\epsilon$ versus predicting $x_0$. 5. Implement one DDPM reverse step and test it on synthetic tensors. 6. Simulate classifier-free guidance using two toy vector fields. 7. Create a toy inpainting task and enforce mask consistency during sampling. ### AI and Modeling Exercises 1. Ask an AI assistant to explain DDPM training. Identify at least three places where the explanation omits assumptions or dimensions. 2. Ask for a derivation of the posterior $q(x_{t-1}\mid x_t,x_0)$. Rewrite it in your own notation. 3. Ask for examples of diffusion hallucination in scientific imaging. Convert the answer into a checklist for responsible use. 4. Compare a diffusion prior with a total-variation prior for image reconstruction. Which assumptions are explicit in each? 5. Design a conditional diffusion project for medical imaging, satellite imaging, or microscopy. Specify what evidence would be needed before deployment. ## 29.45 Chapter Summary Diffusion models are probabilistic generative models built from a simple idea: gradually add noise to data, then learn to reverse the noising process. The forward process is known and analytically tractable. The reverse process depends on the unknown image distribution and is learned by denoising. The most important mathematical formulas are: $$ q(x_t\mid x_{t-1}) =\mathcal N(\sqrt{1-\beta_t}x_{t-1},\beta_t I), $$ $$ q(x_t\mid x_0) =\mathcal N(\sqrt{\bar\alpha_t}x_0,(1-\bar\alpha_t)I), $$ and $$ L_{\text{simple}}(\theta)= \mathbb E\left[\|\epsilon- \epsilon_\theta(x_t,t)\|^2\right]. $$ From a statistics viewpoint, diffusion models learn conditional expectations, scores, reverse Markov kernels, and posterior samplers. From an applied mathematics viewpoint, they are learned stochastic or deterministic dynamical systems. From a computer vision viewpoint, they provide a powerful framework for generation, editing, inverse problems, and multimodal visual reasoning. ## References Core sources for this chapter include the original diffusion probabilistic modeling line of work, denoising diffusion probabilistic models, score-based generative modeling, DDIM sampling, classifier-free guidance, latent diffusion, and modern text-to-image diffusion systems [@sohldickstein2015deep; @ho2020denoising; @song2021scorebased; @song2020ddim; @dhariwal2021diffusion; @ho2022classifierfree; @rombach2022latent; @nichol2021improved].