19 Probabilistic Models of Images

Classical computer vision often asks for a deterministic answer: an edge map, a segmentation, a depth map, or a reconstruction. Modern vision also asks: How uncertain is the answer? What assumptions produced it? What distribution over images or labels is being learned? This chapter develops probabilistic models of images for MA Applied Mathematics and MS Statistics students, emphasizing likelihoods, priors, posterior inference, MAP estimation, random fields, and connections to modern generative AI.

19.1 Learning Goals

After reading this chapter, students should be able to:

model images, labels, and latent scenes as random variables or random fields;
write likelihoods for common image noise models, including Gaussian, Laplace, Bernoulli, multinomial, and Poisson models;
derive maximum likelihood, MAP, posterior mean, and posterior predictive estimators;
interpret regularization as a negative log-prior;
connect classical denoising, segmentation, stereo, and reconstruction to Bayesian inference;
explain Markov random fields, Gibbs distributions, and pairwise smoothness priors;
analyze uncertainty using covariance, entropy, calibration, and posterior samples;
recognize how modern generative models, including VAEs, normalizing flows, and diffusion models, extend probabilistic vision.

The central mathematical idea is that a vision problem is usually an inverse problem with uncertainty:

\[ y = A(x) + \varepsilon, \]

where $x$ is an unknown image, scene, label field, or representation; $y$ is the observed data; $A$ is a forward measurement process; and $\varepsilon$ is noise or modeling error. Probability turns this schematic equation into likelihoods, priors, posteriors, and decisions.

19.2 18.1 Why Probability Is Needed in Computer Vision

An image rarely determines a unique interpretation. The same 2D photograph can be consistent with many possible 3D scenes, illuminations, textures, segmentations, camera poses, and object identities. This is not merely a computational inconvenience. It is a mathematical feature of the inverse problem.

For example:

a noisy image may come from many possible clean images;
a pixel near an object boundary may have ambiguous class membership;
a short-baseline stereo pair may produce a broad depth distribution;
an occluded object may have several plausible completions;
a low-resolution image may correspond to many high-resolution images.

A deterministic algorithm returns one estimate. A probabilistic model describes a full uncertainty structure.

The probabilistic viewpoint introduces four basic objects.

Probabilistic vision objects

Object	Meaning in vision
$p(y\|x)$	likelihood: how an unknown image/scene $x$ produces observation $y$
$p(x)$	prior: which images/scenes are plausible before seeing $y$
$p(x\|y)$	posterior: uncertainty about $x$ after observing $y$
$\delta(y)$	decision rule: an estimate, prediction, or action based on $y$

Bayes’ rule is the bridge:

\[ p(x|y)=\frac{p(y|x)p(x)}{p(y)}, \qquad p(y)=\int p(y|x)p(x)\,dx. \]

The denominator $p(y)$ is called the evidence or marginal likelihood. For many vision problems, it is difficult to compute exactly because $x$ may be a high-dimensional image, a depth map, a segmentation mask, a set of 3D points, or a neural-network latent code.

19.3 18.2 Images as Random Vectors and Random Fields

Let a grayscale image have height $H$ and width $W$. A deterministic image is an array

\[ I \in \mathbb{R}^{H\times W}. \]

A random image is a random vector

\[ X = (X_1,\ldots,X_n) \in \mathbb{R}^n, \qquad n=HW, \]

where each coordinate $X_i$ corresponds to one pixel after vectorization. A color image is a random vector in $\mathbb{R}^{HWC}$.

This vector notation is useful for linear algebra and statistics, but it hides spatial structure. A more geometric view treats an image as a random field

\[ \{X_s:s\in\Omega_h\}, \]

where $\Omega_h$ is the discrete pixel grid. Neighboring pixels are usually statistically dependent. A model that ignores spatial dependence may be too weak for denoising, segmentation, and texture modeling.

19.3.1 Independent-pixel model

The simplest model assumes

\[ p(x)=\prod_{i=1}^n p_i(x_i). \]

This can be useful for histograms, thresholding, and naive Bayes models, but it cannot express smoothness or texture.

19.3.2 Correlated Gaussian image model

A basic correlated model is

\[ X \sim \mathcal{N}(\mu,\Sigma), \]

where $\Sigma_{ij}$ describes covariance between pixels $i$ and $j$. If nearby pixels have positive covariance, the model favors smooth images.

The negative log-density is, up to an additive constant,

\[ -\log p(x) = \frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu) + \frac{1}{2}\log\det\Sigma. \]

The precision matrix $Q=\Sigma^{-1}$ is especially important. Sparse precision matrices encode conditional independence and lead to Gaussian Markov random fields.

19.3.3 Label fields

For segmentation, we often use a discrete random field

\[ Z_s\in\{1,\ldots,K\}, \qquad s\in\Omega_h. \]

The observed image intensities are $Y_s$, and the unknown labels are $Z_s$. A probabilistic segmentation model seeks

\[ p(z|y) \propto p(y|z)p(z). \]

The likelihood describes how each region generates intensities. The prior describes spatial coherence of labels.

19.4 18.3 Forward Models and Likelihoods

The likelihood $p(y|x)$ encodes the measurement model. It answers the question:

If the true image, scene, or label field were $x$, how likely would the observed data $y$ be?

A common forward model is

\[ y=A x+\varepsilon, \]

where $A$ may represent blurring, downsampling, projection, masking, tomographic measurement, or a camera model.

19.4.1 Gaussian noise

\[ \varepsilon\sim \mathcal{N}(0,\sigma^2I), \]

then

\[ p(y|x) = (2\pi\sigma^2)^{-m/2} \exp\left(-\frac{1}{2\sigma^2}\|y-Ax\|_2^2\right). \]

The negative log-likelihood is

\[ -\log p(y|x) = \frac{1}{2\sigma^2}\|y-Ax\|_2^2+\text{constant}. \]

Thus least squares is maximum likelihood under Gaussian noise.

19.4.2 Laplace noise and robust fitting

If each noise coordinate has a Laplace distribution,

\[ p(\varepsilon_i)=\frac{1}{2b}\exp\left(-\frac{|\varepsilon_i|}{b}\right), \]

then the negative log-likelihood is proportional to an $\ell_1$ residual:

\[ -\log p(y|x) = \frac{1}{b}\|y-Ax\|_1+\text{constant}. \]

This model is more robust to outliers than the Gaussian model.

19.4.3 Poisson noise

Photon counts in low-light imaging are often modeled as Poisson:

\[ Y_i|x_i \sim \operatorname{Poisson}((Ax)_i), \]

\[ p(y|x)=\prod_i \frac{(Ax)_i^{y_i}e^{-(Ax)_i}}{y_i!}. \]

Ignoring constants independent of $x$, the negative log-likelihood is

\[ -\log p(y|x) = \sum_i \left((Ax)_i-y_i\log (Ax)_i\right)+\text{constant}. \]

Poisson models are natural for microscopy, astronomy, medical imaging, and low-light photography.

19.4.4 Bernoulli and categorical likelihoods

For a binary mask $Y_s\in\{0,1\}$ with predicted probability $\pi_s$, the Bernoulli likelihood is

\[ p(y_s|\pi_s)=\pi_s^{y_s}(1-\pi_s)^{1-y_s}. \]

The negative log-likelihood is binary cross-entropy:

\[ -\log p(y|\pi) = -\sum_s \left[y_s\log \pi_s+(1-y_s)\log(1-\pi_s)\right]. \]

For semantic segmentation with $K$ classes, the categorical likelihood gives multiclass cross-entropy:

\[ -\log p(y|\pi) = -\sum_s \sum_{k=1}^K 1\{y_s=k\}\log \pi_{s,k}. \]

This is the probabilistic foundation of many neural segmentation losses.

19.5 Interactive 18.1: Likelihoods for Common Noise Models

The following interactive compares negative log-likelihood shapes for Gaussian, Laplace, and Huber-style robust residuals. The independent variable is the residual $r=y-Ax$.

Interactive 18.1: Negative log-likelihood and residual models

Change the scale and compare how Gaussian, Laplace, and Huber residual penalties treat small and large errors.

scale parameter: 1.00

The graph illustrates a statistical modeling principle. The loss function is not arbitrary: it usually corresponds to an assumed noise distribution.

19.6 18.4 Maximum Likelihood Estimation

Suppose $y$ is observed and $x$ is unknown. The maximum likelihood estimator is

\[ \widehat{x}_{\operatorname{ML}} \in \arg\max_x p(y|x). \]

Equivalently,

\[ \widehat{x}_{\operatorname{ML}} \in \arg\min_x -\log p(y|x). \]

For Gaussian linear observations,

\[ y=Ax+\varepsilon, \qquad \varepsilon\sim\mathcal{N}(0,\sigma^2I), \]

ML becomes ordinary least squares:

\[ \widehat{x}_{\operatorname{ML}} \in \arg\min_x \|y-Ax\|_2^2. \]

If $A$ has full column rank, then

\[ \widehat{x}_{\operatorname{ML}} =(A^TA)^{-1}A^Ty. \]

If $A$ is ill-conditioned or rank deficient, the ML estimate may be unstable. This is exactly why priors and regularization are central in image reconstruction.

19.7 18.5 Priors and Regularization

A prior $p(x)$ describes which images are plausible before observing $y$. The MAP estimator is

\[ \widehat{x}_{\operatorname{MAP}} \in \arg\max_x p(x|y) = \arg\max_x p(y|x)p(x). \]

Equivalently,

\[ \widehat{x}_{\operatorname{MAP}} \in \arg\min_x \left\{-\log p(y|x)-\log p(x)\right\}. \]

Thus the data-fidelity term comes from the likelihood, while the regularization term comes from the negative log-prior.

19.7.1 Gaussian prior and Tikhonov regularization

Assume

\[ x\sim \mathcal{N}(0,\tau^2I) \]

and

\[ y|x\sim \mathcal{N}(Ax,\sigma^2I). \]

Then

\[ \widehat{x}_{\operatorname{MAP}} = \arg\min_x \frac{1}{2\sigma^2}\|y-Ax\|_2^2 + \frac{1}{2\tau^2}\|x\|_2^2. \]

Multiplying by $2\sigma^2$ gives

\[ \widehat{x}_{\operatorname{MAP}} = \arg\min_x \|y-Ax\|_2^2+\lambda\|x\|_2^2, \qquad \lambda=\frac{\sigma^2}{\tau^2}. \]

This is ridge or Tikhonov regularization. The closed form is

\[ \widehat{x}_{\operatorname{MAP}} =(A^TA+\lambda I)^{-1}A^Ty. \]

19.7.2 Smoothness prior

Images are not usually small in Euclidean norm. A more natural prior penalizes roughness:

\[ p(x)\propto \exp\left(-\frac{\alpha}{2}\|Dx\|_2^2\right), \]

where $D$ is a finite-difference gradient operator. Then MAP estimation becomes

\[ \widehat{x}_{\operatorname{MAP}} = \arg\min_x \frac{1}{2\sigma^2}\|y-Ax\|_2^2 + \frac{\alpha}{2}\|Dx\|_2^2. \]

This is a probabilistic version of smooth denoising and deblurring.

19.7.3 Total variation prior

Edges are important in vision. A quadratic smoothness penalty tends to oversmooth edges. Total variation uses an $\ell_1$ penalty on gradients:

\[ \operatorname{TV}(x) = \sum_s \|\nabla x_s\|_2. \]

The corresponding prior has the form

\[ p(x)\propto \exp(-\alpha\operatorname{TV}(x)). \]

The MAP estimator is

\[ \widehat{x}_{\operatorname{MAP}} = \arg\min_x \frac{1}{2\sigma^2}\|y-Ax\|_2^2 + \alpha\operatorname{TV}(x). \]

This is edge-preserving because it allows sparse large gradients.

19.8 Interactive 18.2: MAP Denoising and the Bias-Variance Tradeoff

This demo shows a one-dimensional analogy of image denoising. The noisy observation is smoothed by a MAP estimator with a quadratic smoothness prior. Increasing $\lambda$ reduces noise but also removes detail.

Interactive 18.2: MAP smoothing from a Gaussian prior

The estimator minimizes squared data error plus a quadratic roughness penalty.

regularization strength λ: 1.0

The slider is a concrete version of a general question in probabilistic vision:

How strongly should prior belief constrain the estimate relative to the observed data?

19.9 18.6 Posterior Mean, MAP, and Decision Theory

The MAP estimate is only one Bayesian estimator. If the loss function is squared error,

\[ L(x,a)=\|x-a\|_2^2, \]

then the Bayes estimator is the posterior mean:

\[ \widehat{x}_{\operatorname{PM}}=\mathbb{E}[X|Y=y]. \]

If the loss is absolute error in one dimension,

\[ L(x,a)=|x-a|, \]

then a posterior median is optimal.

If the target is a class label and the loss is zero-one loss, the optimal classifier is

\[ \widehat{z}(y)=\arg\max_k p(z=k|y). \]

This is the maximum posterior probability rule.

The key lesson is that an estimator is not only a formula. It is the solution to a decision problem involving a posterior distribution and a loss function.

19.9.1 MAP can be misleading

For a highly asymmetric or multimodal posterior, the MAP estimate may be unrepresentative. Consider a depth posterior with two modes: one corresponding to a nearby object and one corresponding to a far object. The MAP returns only one mode. The posterior mean may fall between two physically plausible depths. A calibrated uncertainty report may be more informative than either point estimate.

19.10 18.7 Bayesian Linear Inverse Problems

The Gaussian linear model is one of the cleanest mathematical laboratories for probabilistic vision.

Assume

\[ y|x\sim \mathcal{N}(Ax,\Sigma_\varepsilon), \qquad x\sim \mathcal{N}(\mu_0,\Sigma_0). \]

Then the posterior is also Gaussian:

\[ x|y\sim \mathcal{N}(\mu_{\operatorname{post}},\Sigma_{\operatorname{post}}), \]

where

\[ \Sigma_{\operatorname{post}} = \left(\Sigma_0^{-1}+A^T\Sigma_\varepsilon^{-1}A\right)^{-1}, \]

and

\[ \mu_{\operatorname{post}} = \Sigma_{\operatorname{post}} \left(\Sigma_0^{-1}\mu_0+A^T\Sigma_\varepsilon^{-1}y\right). \]

For Gaussian posteriors, the posterior mean and MAP estimate coincide.

19.10.1 Interpretation

The posterior precision is

\[ \Sigma_{\operatorname{post}}^{-1} = \Sigma_0^{-1}+A^T\Sigma_\varepsilon^{-1}A. \]

This formula says that information from the prior and information from the data add in precision space. A strong prior has large $\Sigma_0^{-1}$. Accurate measurements have large $\Sigma_\varepsilon^{-1}$.

19.10.2 Image deblurring example

If $A$ is a blur matrix, then deblurring is ill-posed because high frequencies are strongly attenuated. A prior supplies information about those unstable directions. In Fourier-like coordinates, the posterior variance is large where both the prior and the data provide little information.

19.11 18.8 Markov Random Fields and Gibbs Priors

For label fields and images, spatial dependence is essential. A Markov random field models local interactions.

Let $G=(V,E)$ be a graph whose nodes are pixels and whose edges connect neighboring pixels. A random field $X=(X_s)_{s\in V}$ is Markov with respect to $G$ if

\[ p(x_s|x_{V\setminus\{s\}})=p(x_s|x_{N(s)}), \]

where $N(s)$ is the set of neighbors of $s$.

A Gibbs distribution has the form

\[ p(x)=\frac{1}{Z}\exp(-E(x)), \]

where

\[ Z=\sum_x \exp(-E(x)) \]

or, for continuous variables,

\[ Z=\int \exp(-E(x))\,dx. \]

The normalizing constant $Z$ is called the partition function. It is often computationally difficult.

19.11.1 Pairwise prior for binary labels

For binary labels $z_s\in\{0,1\}$, a common prior is

\[ p(z)\propto \exp\left(-\beta\sum_{(s,t)\in E} 1\{z_s\ne z_t\}\right). \]

When $\beta>0$, neighboring labels prefer to be equal. This is a probabilistic version of segmentation smoothness.

The MAP segmentation problem has energy

\[ E(z) = \sum_s D_s(z_s) + \beta\sum_{(s,t)\in E} 1\{z_s\ne z_t\}, \]

where $D_s(k)$ is a data cost, often $-\log p(y_s|z_s=k)$.

This connects directly to the graph-cut energies from Chapter 8.

19.11.2 Gaussian Markov random fields

For continuous images, a Gaussian Markov random field may use energy

\[ E(x)=\frac{1}{2}\sum_s \frac{(y_s-x_s)^2}{\sigma^2} + \frac{\beta}{2}\sum_{(s,t)\in E}(x_s-x_t)^2. \]

The first term fits the observed data. The second term encourages neighboring pixels to be similar. In matrix form,

\[ E(x)=\frac{1}{2\sigma^2}\|y-x\|_2^2 + \frac{\beta}{2}x^TLx, \]

where $L$ is the graph Laplacian. The MAP estimate solves the linear system

\[ \left(\frac{1}{\sigma^2}I+\beta L\right)x = \frac{1}{\sigma^2}y. \]

19.12 Interactive 18.3: Pairwise MRF Prior for Binary Segmentation

This demo shows how the pairwise parameter $\beta$ rewards spatial coherence. Larger $\beta$ makes isolated label changes more expensive.

Interactive 18.3: MRF pairwise smoothness

The displayed energy is a toy binary segmentation energy: data cost plus a penalty for neighboring label disagreements.

pairwise strength β: 0.60

This small example is intentionally discrete. It prepares students for probabilistic segmentation, graph cuts, belief propagation, and conditional random fields.

19.13 18.9 Mixture Models and Probabilistic Segmentation

A useful model for intensity-based segmentation is a Gaussian mixture model. Suppose each pixel intensity $Y_s\in\mathbb{R}^d$ belongs to a latent class $Z_s\in\{1,\ldots,K\}$. Let

\[ P(Z_s=k)=\pi_k, \qquad Y_s|Z_s=k\sim \mathcal{N}(\mu_k,\Sigma_k). \]

The marginal density is

\[ p(y_s)=\sum_{k=1}^K \pi_k\,\mathcal{N}(y_s;\mu_k,\Sigma_k). \]

The posterior class probability is

\[ \gamma_{s,k} =P(Z_s=k|Y_s=y_s) = \frac{\pi_k\mathcal{N}(y_s;\mu_k,\Sigma_k)}{\sum_{\ell=1}^K \pi_\ell\mathcal{N}(y_s;\mu_\ell,\Sigma_\ell)}. \]

These posterior probabilities are soft segmentations.

19.13.1 EM algorithm

For independent pixels, the log-likelihood is

\[ \ell(\theta) = \sum_s \log\left(\sum_{k=1}^K \pi_k\mathcal{N}(y_s;\mu_k,\Sigma_k)\right). \]

The EM algorithm alternates:

E-step: compute responsibilities

\[ \gamma_{s,k}^{(t)} = \frac{\pi_k^{(t)}\mathcal{N}(y_s;\mu_k^{(t)},\Sigma_k^{(t)})}{\sum_{\ell=1}^K \pi_\ell^{(t)}\mathcal{N}(y_s;\mu_\ell^{(t)},\Sigma_\ell^{(t)})}. \]

M-step: update parameters

\[ N_k^{(t)}=\sum_s \gamma_{s,k}^{(t)}, \]

\[ \pi_k^{(t+1)}=\frac{N_k^{(t)}}{n}, \]

\[ \mu_k^{(t+1)}=\frac{1}{N_k^{(t)}}\sum_s \gamma_{s,k}^{(t)}y_s, \]

\[ \Sigma_k^{(t+1)}=\frac{1}{N_k^{(t)}}\sum_s \gamma_{s,k}^{(t)}(y_s-\mu_k^{(t+1)})(y_s-\mu_k^{(t+1)})^T. \]

This model does not enforce spatial smoothness. Adding an MRF prior on labels produces a more realistic segmentation model but makes inference harder.

19.14 Interactive 18.4: Gaussian Mixture Responsibilities

The next demo shows soft class probabilities in a one-dimensional mixture model. Notice that uncertainty is highest near class overlap.

Interactive 18.4: Mixture-model posterior probabilities

Move the component means closer or farther apart and observe how the posterior responsibilities change.

mean separation: 2.5

The soft responsibilities $\gamma_{s,k}$ are a statistically principled alternative to hard thresholding.

19.15 18.10 Conditional Models and Discriminative Probability

The models above often begin with a generative story $p(y|x)p(x)$ or $p(y,z)$. In supervised computer vision, we often model the conditional distribution directly:

\[ p_\theta(z|y). \]

For classification, a neural network may output logits $a_k(y)$ and probabilities

\[ p_\theta(z=k|y) = \frac{\exp(a_k(y))}{\sum_{\ell=1}^K\exp(a_\ell(y))}. \]

Training by cross-entropy minimizes

\[ \widehat{R}(\theta) = -\frac{1}{n}\sum_{i=1}^n \log p_\theta(z_i|y_i). \]

This is maximum likelihood for the conditional model.

19.15.1 Calibration

A classifier is calibrated if predicted probabilities match empirical frequencies. Informally, among examples for which the model predicts probability $0.8$, approximately $80\%$ should be correct.

One common calibration score is expected calibration error. Partition predictions into confidence bins $B_1,\ldots,B_M$. Then

\[ \operatorname{ECE} = \sum_{m=1}^M \frac{|B_m|}{n} \left|\operatorname{acc}(B_m)-\operatorname{conf}(B_m)\right|. \]

For high-stakes vision systems, such as medical imaging and autonomous driving, calibrated uncertainty may be as important as accuracy.

19.16 Interactive 18.5: Calibration of Vision Predictions

This demo compares an ideal calibrated classifier with an overconfident classifier.

Interactive 18.5: Reliability diagram

Overconfident models have predicted confidence larger than empirical accuracy.

overconfidence level: 0.25

A model can have high accuracy and poor calibration. This distinction becomes important when probabilities are used for downstream decisions.

19.17 18.11 Uncertainty in Vision Models

Uncertainty can be separated into two broad types.

Aleatoric uncertainty is uncertainty caused by noise or inherent ambiguity in the data. For example, a low-light image has photon noise, and an occluded object may be intrinsically ambiguous.

Epistemic uncertainty is uncertainty caused by limited knowledge of model parameters or limited training data. It can often be reduced by collecting more relevant data.

A probabilistic model may represent uncertainty through:

posterior covariance matrices;
marginal posterior variances;
entropy of class probabilities;
predictive distributions;
ensembles;
Bayesian neural networks;
Monte Carlo dropout;
diffusion or generative posterior samples.

19.17.1 Predictive distribution

Given parameters $\theta$, a conditional model predicts

\[ p(y_*|x_*,\theta). \]

In Bayesian modeling, parameters are uncertain, so the predictive distribution integrates over their posterior:

\[ p(y_*|x_*,\mathcal{D}) = \int p(y_*|x_*,\theta)p(\theta|\mathcal{D})\,d\theta. \]

This integral is usually approximated.

19.17.2 Entropy for segmentation

If a neural segmentation model outputs probabilities $p_{s,k}$ at pixel $s$, the pixelwise entropy is

\[ H_s=-\sum_{k=1}^K p_{s,k}\log p_{s,k}. \]

High entropy indicates uncertain class membership. Entropy maps are useful for quality control, active learning, and human review.

19.18 18.12 Probabilistic Graphical Models for Vision

A probabilistic graphical model represents a joint distribution using a graph. Nodes represent random variables. Edges represent dependence.

Examples in computer vision include:

hidden Markov models for video and tracking;
conditional random fields for segmentation;
Bayesian networks for object relationships;
factor graphs for SLAM and bundle adjustment;
diffusion graphical models for generative modeling.

A factor graph writes the joint density as

\[ p(x)\propto \prod_{a\in\mathcal{F}} \psi_a(x_a), \]

where each factor $\psi_a$ depends only on a subset $x_a$ of variables.

Taking negative logarithms gives an additive energy:

\[ E(x)=-\sum_{a\in\mathcal{F}}\log \psi_a(x_a)+\text{constant}. \]

This explains why so many vision objectives are sums of local terms. Local probabilistic dependencies become additive optimization objectives.

19.19 18.13 Sampling and Approximate Inference

Exact posterior inference is often impossible in high-dimensional vision models. Common approximation methods include:

Laplace approximation: approximate the posterior near a mode by a Gaussian;
variational inference: approximate $p(x|y)$ by a simpler distribution $q_\phi(x)$;
Monte Carlo methods: draw approximate samples from the posterior;
belief propagation: pass messages on a graph;
mean-field approximation: replace a coupled distribution by a factorized one.

19.19.1 Laplace approximation

Suppose

\[ p(x|y)\propto \exp(-E(x)). \]

Let $\widehat{x}$ be a local minimizer of $E$. The second-order Taylor approximation is

\[ E(x)\approx E(\widehat{x})+rac{1}{2}(x-\widehat{x})^TH(x-\widehat{x}), \]

where

\[ H=\nabla^2E(\widehat{x}). \]

Then

\[ p(x|y)\approx \mathcal{N}(\widehat{x},H^{-1}). \]

This connects optimization curvature to uncertainty.

19.19.2 Variational inference

Variational inference chooses a tractable family $q_\phi(x)$ and minimizes

\[ \operatorname{KL}(q_\phi(x)\|p(x|y)). \]

Equivalently, it maximizes an evidence lower bound. This idea is central to variational autoencoders and many modern latent-variable models.

19.20 18.14 Modern Generative Models as Probabilistic Image Models

Deep learning did not remove probability from vision. It made probabilistic models more expressive.

19.20.1 Variational autoencoders

A variational autoencoder introduces a latent variable $z$:

\[ z\sim p(z), \qquad x|z\sim p_\theta(x|z). \]

The marginal likelihood is

\[ p_\theta(x)=\int p_\theta(x|z)p(z)\,dz. \]

Because this integral is often intractable, VAEs use an approximate posterior $q_\phi(z|x)$ and optimize the ELBO:

\[ \log p_\theta(x) \ge \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)] - \operatorname{KL}(q_\phi(z|x)\|p(z)). \]

The first term is a reconstruction term. The second term regularizes the latent distribution.

19.20.2 Normalizing flows

A normalizing flow constructs an invertible map

\[ x=f_\theta(z), \qquad z\sim p_Z. \]

By the change-of-variables formula,

\[ p_X(x) = p_Z(f_\theta^{-1}(x)) \left|\det \frac{\partial f_\theta^{-1}}{\partial x}\right|. \]

Flows give exact likelihoods but require invertible architectures.

19.20.3 Diffusion models

Diffusion models define a forward noising process and learn a reverse denoising process. A common forward model is

\[ q(x_t|x_{t-1}) = \mathcal{N}(\sqrt{1-\beta_t}x_{t-1},\beta_tI). \]

After many steps, $x_T$ is approximately noise. The learned reverse model estimates how to move from noise back toward data. Diffusion models can be understood as very high-dimensional probabilistic image models with learned denoising transitions.

Chapter 29 returns to diffusion models in detail. Here the important point is conceptual: modern generative AI is built from likelihoods, latent variables, priors, scores, and approximate inference.

19.21 18.15 AI Components: Probabilistic Thinking for Modern Vision Systems

Modern AI vision systems use probability in several ways.

19.21.1 Cross-entropy training

For classification and segmentation, cross-entropy is negative log-likelihood for categorical outputs. This turns supervised learning into conditional maximum likelihood.

19.21.2 Diffusion and score models

Denoising models estimate information about the score

\[ \nabla_x \log p_t(x), \]

which points toward higher-density regions of a noised data distribution. This gives a probabilistic interpretation of learned denoisers.

19.21.3 Uncertainty-aware neural networks

A model may output both a mean and a variance. For a regression target $y$, suppose

\[ y|x\sim\mathcal{N}(\mu_\theta(x),\sigma_\theta^2(x)). \]

The negative log-likelihood is

\[ \frac{1}{2}\log \sigma_\theta^2(x) + \frac{(y-\mu_\theta(x))^2}{2\sigma_\theta^2(x)} + \text{constant}. \]

This objective lets the model learn data-dependent uncertainty.

19.21.4 Segmentation confidence and active learning

For segmentation, uncertainty maps can identify pixels or images for human annotation. One acquisition score is entropy:

\[ a(y)=\sum_s H(p_\theta(z_s|y)). \]

Images with high entropy may be informative for retraining.

19.21.5 Probabilistic prompts for AI study

When using an AI assistant to study probabilistic vision, ask it to state assumptions explicitly:

What is the random variable?
What is observed and what is latent?
What is the likelihood?
What is the prior?
What loss function justifies the estimator?
Which independence assumptions are being made?
How would the answer change under Gaussian, Laplace, or Poisson noise?

19.22 18.16 Python Example: Noise Models and Negative Log-Likelihoods

The following code compares Gaussian, Laplace, and Poisson negative log-likelihoods. It is written for independent study; execute it in a notebook after installing numpy and matplotlib.

import numpy as np
import matplotlib.pyplot as plt

r = np.linspace(-5, 5, 500)
sigma = 1.0
b = 1.0

nll_gaussian = 0.5 * (r / sigma) ** 2
nll_laplace = np.abs(r) / b

plt.figure(figsize=(7, 4))
plt.plot(r, nll_gaussian, label="Gaussian noise: quadratic loss")
plt.plot(r, nll_laplace, label="Laplace noise: absolute loss")
plt.xlabel("residual r")
plt.ylabel("negative log-likelihood, up to constants")
plt.title("Noise assumptions determine loss functions")
plt.legend()
plt.show()

For Poisson noise:

lam = np.linspace(0.05, 10, 500)
y_obs = 3

# Negative log-likelihood up to constants independent of lambda
nll_poisson = lam - y_obs * np.log(lam)

plt.figure(figsize=(7, 4))
plt.plot(lam, nll_poisson)
plt.axvline(y_obs, linestyle="--", label="MLE lambda = observed count")
plt.xlabel("Poisson rate lambda")
plt.ylabel("negative log-likelihood")
plt.title("Poisson likelihood for a photon count")
plt.legend()
plt.show()

19.23 18.17 Python Example: MAP Denoising with a Graph Laplacian

This example performs one-dimensional MAP denoising using a smoothness prior. The same idea extends to 2D images by replacing the 1D Laplacian with a 2D grid Laplacian.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(7243)

n = 160
t = np.linspace(0, 1, n)
x_true = np.sin(2 * np.pi * t) + 0.6 * (t > 0.55)
y = x_true + 0.25 * rng.normal(size=n)

# First-difference matrix D, shape (n - 1, n)
D = np.zeros((n - 1, n))
for i in range(n - 1):
    D[i, i] = -1
    D[i, i + 1] = 1

lam = 15.0
A = np.eye(n) + lam * D.T @ D
x_map = np.linalg.solve(A, y)

plt.figure(figsize=(8, 4))
plt.plot(t, x_true, label="true signal")
plt.scatter(t, y, s=12, alpha=0.6, label="noisy observations")
plt.plot(t, x_map, linewidth=2, label="MAP estimate")
plt.xlabel("position")
plt.ylabel("intensity")
plt.title("MAP denoising with a quadratic smoothness prior")
plt.legend()
plt.show()

19.24 18.18 Python Example: Gaussian Mixture Segmentation

The next example fits a Gaussian mixture model to pixel intensities and interprets posterior responsibilities as soft labels.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.mixture import GaussianMixture

rng = np.random.default_rng(18)
H, W = 120, 160
Y, X = np.mgrid[0:H, 0:W]

mask = ((X - 80) ** 2 / 45 ** 2 + (Y - 60) ** 2 / 28 ** 2) < 1
image = np.where(mask, 0.75, 0.25) + 0.12 * rng.normal(size=(H, W))
image = np.clip(image, 0, 1)

pixels = image.reshape(-1, 1)
gmm = GaussianMixture(n_components=2, covariance_type="full", random_state=0)
gmm.fit(pixels)
responsibilities = gmm.predict_proba(pixels)

# Choose the component with larger mean as foreground
foreground_component = np.argmax(gmm.means_.ravel())
soft_mask = responsibilities[:, foreground_component].reshape(H, W)

fig, axes = plt.subplots(1, 3, figsize=(11, 3.5))
axes[0].imshow(image, cmap="gray")
axes[0].set_title("noisy image")
axes[1].imshow(mask, cmap="gray")
axes[1].set_title("true mask")
axes[2].imshow(soft_mask, cmap="gray", vmin=0, vmax=1)
axes[2].set_title("posterior foreground probability")
for ax in axes:
    ax.axis("off")
plt.tight_layout()
plt.show()

This example is intentionally simple. It separates intensity modeling from spatial regularization. Adding an MRF prior would reduce isolated mistakes.

19.25 18.19 Python Example: Pixelwise Entropy for Segmentation Uncertainty

Suppose a segmentation model outputs pixelwise class probabilities. Entropy gives a simple uncertainty map.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(9)
H, W, K = 80, 100, 3
logits = rng.normal(size=(H, W, K))

# Make the center more ambiguous by shrinking logits there
Y, X = np.mgrid[0:H, 0:W]
r2 = (X - W / 2) ** 2 + (Y - H / 2) ** 2
ambiguity = np.exp(-r2 / (2 * 18 ** 2))
logits = logits * (1 - 0.8 * ambiguity[..., None])

exp_logits = np.exp(logits - logits.max(axis=-1, keepdims=True))
probs = exp_logits / exp_logits.sum(axis=-1, keepdims=True)
entropy = -np.sum(probs * np.log(probs + 1e-12), axis=-1)

plt.figure(figsize=(6, 4))
plt.imshow(entropy, cmap="magma")
plt.colorbar(label="entropy")
plt.title("Pixelwise segmentation uncertainty")
plt.axis("off")
plt.show()

19.26 18.20 TensorFlow Example: Learning Mean and Uncertainty

A neural model can output both a mean and a variance. This is useful when the noise level depends on the input.

import tensorflow as tf

inputs = tf.keras.Input(shape=(64, 64, 3))

x = tf.keras.layers.Conv2D(16, 3, padding="same", activation="relu")(inputs)
x = tf.keras.layers.MaxPooling2D()(x)
x = tf.keras.layers.Conv2D(32, 3, padding="same", activation="relu")(x)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dense(64, activation="relu")(x)

# Example: predict a scalar depth value and log variance
mu = tf.keras.layers.Dense(1, name="mean_prediction")(x)
log_var = tf.keras.layers.Dense(1, name="log_variance_prediction")(x)

model = tf.keras.Model(inputs=inputs, outputs=[mu, log_var])

def gaussian_nll(y_true, outputs):
    mu_pred, log_var_pred = outputs
    return tf.reduce_mean(
        0.5 * log_var_pred +
        0.5 * tf.square(y_true - mu_pred) * tf.exp(-log_var_pred)
    )

In practice, one must regularize and validate uncertainty predictions. A model can learn to inflate variance unless the objective, data, and evaluation are carefully designed.

19.27 18.21 AI-Assisted Mathematical Study Prompts

Use these prompts to study the chapter with an AI assistant. They are designed to force explicit assumptions and derivations.

Prompt 1: Derive a loss from a likelihood.
I have a vision inverse problem $y=Ax+\varepsilon$. Derive the negative log-likelihood for Gaussian, Laplace, and Poisson noise. Explain which loss function each model produces and when each assumption is reasonable.

Prompt 2: Interpret a regularizer as a prior.
Given the MAP objective $\|y-Ax\|_2^2+\lambda\|Dx\|_2^2$, identify the likelihood and prior assumptions. Explain the role of $D$ and derive the normal equations.

Prompt 3: Compare MAP and posterior mean.
Give examples of posterior distributions for which the MAP estimate and posterior mean are close, and examples where they are very different. Explain the decision-theoretic reason.

Prompt 4: Diagnose uncertainty.
A segmentation model produces high softmax probabilities but makes many mistakes. Explain the difference between accuracy and calibration. Suggest a reliability diagram and entropy-based diagnostics.

Prompt 5: Connect classical and modern vision.
Explain how Markov random fields, graph cuts, cross-entropy neural networks, VAEs, and diffusion models all fit into the probabilistic modeling framework of likelihoods, priors, posteriors, and approximate inference.

19.28 18.22 Common Pitfalls

19.28.1 Pitfall 1: Treating softmax probabilities as automatically calibrated

Softmax outputs sum to one, but that does not imply they are calibrated probabilities. Overconfident deep networks can produce high confidence for wrong predictions.

19.28.2 Pitfall 2: Confusing a loss function with an arbitrary engineering choice

Many losses correspond to negative log-likelihoods. Squared error, absolute error, cross-entropy, and Poisson loss each encode different probabilistic assumptions.

19.28.3 Pitfall 3: Forgetting the prior in ill-posed inverse problems

When $A$ destroys information, the data alone cannot recover $x$. Super-resolution, deblurring, inpainting, and monocular depth all rely on priors, whether explicit or learned.

19.28.4 Pitfall 4: Reporting one estimate without uncertainty

For ambiguous visual tasks, one point estimate may hide important alternatives. Posterior samples, entropy maps, confidence intervals, and calibrated probabilities often reveal more.

19.28.5 Pitfall 5: Assuming independent pixels

Independent-pixel models are mathematically convenient but often miss the spatial structure that makes images images.

19.29 18.23 Summary

This chapter introduced probabilistic image modeling as the statistical foundation of modern computer vision. The main ideas are:

images, labels, scenes, and representations can be modeled as random variables or random fields;
likelihoods encode noise and measurement assumptions;
priors encode plausibility, smoothness, sparsity, edges, geometry, or learned structure;
MAP estimation is regularized optimization;
posterior means, MAP estimates, and classifiers correspond to different decision losses;
Markov random fields connect probability, graph models, and energy minimization;
mixture models produce soft probabilistic segmentations;
calibration and uncertainty matter in real vision systems;
VAEs, flows, and diffusion models are modern probabilistic image models.

The next chapters move from general probabilistic modeling to supervised learning and representation learning for vision.

19.30 Exercises

19.30.1 Conceptual Exercises

Explain why image denoising is an inverse problem. Identify $x$, $y$, $A$, and $\varepsilon$ in the simplest denoising model.
Give one computer vision example where Gaussian noise is reasonable and one where Poisson noise is more reasonable.
Explain why a spatial prior is important for segmentation.
What is the difference between aleatoric and epistemic uncertainty? Give a vision example of each.
Why can MAP estimation be misleading for a multimodal posterior?

19.30.2 Mathematical Exercises

Let $y|x\sim \mathcal{N}(x,\sigma^2I)$ and $x\sim\mathcal{N}(0,\tau^2I)$. Derive the MAP estimate.
Let $y=Ax+\varepsilon$ with $\varepsilon\sim\mathcal{N}(0,\sigma^2I)$ and prior $p(x)\propto\exp(-\alpha\|Dx\|_2^2/2)$. Derive the normal equations for the MAP estimator.
Show that the negative log-likelihood for independent Bernoulli observations is binary cross-entropy.
For a two-class Gaussian mixture with equal variances and equal mixing weights, derive the posterior probability $P(Z=1|Y=y)$.
Let $p(x)\propto\exp(-\beta x^TLx/2)$ where $L$ is a graph Laplacian. Explain why this prior favors smooth signals on the graph.
Derive the posterior covariance for the Gaussian linear inverse problem.
Show that the posterior mean minimizes posterior expected squared error.
For a categorical distribution $p=(p_1,\ldots,p_K)$, prove that entropy is maximized when $p_k=1/K$ for all $k$.

19.30.3 Python Exercises

Simulate Gaussian, Laplace, and Poisson noise on the same grayscale image. Compare the empirical histograms of residuals.
Implement MAP denoising with a 2D finite-difference operator.
Fit a Gaussian mixture model to a color image using RGB features. Display posterior probability maps for each component.
Implement a reliability diagram for a trained classifier or a synthetic classifier.
Compute entropy maps from pixelwise softmax outputs.
Compare quadratic smoothness and total variation denoising on an image with sharp edges.

19.30.4 AI and Modeling Exercises

Ask an AI assistant to derive the MAP objective for a denoising problem. Then check whether it clearly identifies the likelihood and prior.
Ask an AI assistant for a probabilistic model for semantic segmentation. Require it to distinguish $p(y|z)$, $p(z)$, and $p(z|y)$.
Ask an AI assistant to explain why cross-entropy is a negative log-likelihood. Verify the derivation line by line.
Ask an AI assistant to propose uncertainty diagnostics for a medical image classifier. Critique which diagnostics measure calibration and which measure discrimination.
Ask an AI assistant to compare VAEs, flows, and diffusion models as probabilistic image models. Summarize the difference in likelihood tractability.

19.31 Notes and References

The probabilistic view of computer vision is developed in Prince (2012) and connects naturally to statistical learning in Bishop (2006). Markov random fields and Gibbs distributions are classical tools for spatial image modeling; see Geman and Geman (1984), Li (2009), and Rue and Held (2005). Variational and Bayesian deep generative models include VAEs Kingma and Welling (2014), Rezende et al. (2014), normalizing flows Dinh et al. (2017), and diffusion models Ho et al. (2020). Total variation regularization is historically central in image restoration; see Rudin et al. (1992).

Bishop, Christopher M. 2006. Pattern Recognition and Machine Learning. Springer.

Dinh, Laurent, Jascha Sohl-Dickstein, and Samy Bengio. 2017. “Density Estimation Using Real NVP.” International Conference on Learning Representations.

Geman, Stuart, and Donald Geman. 1984. “Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images.” IEEE Transactions on Pattern Analysis and Machine Intelligence 6 (6): 721–41.

Ho, Jonathan, Ajay Jain, and Pieter Abbeel. 2020. “Denoising Diffusion Probabilistic Models.” Advances in Neural Information Processing Systems 33: 6840–51. https://arxiv.org/abs/2006.11239.

Kingma, Diederik P., and Max Welling. 2014. “Auto-Encoding Variational Bayes.” International Conference on Learning Representations.

Li, Stan Z. 2009. Markov Random Field Modeling in Image Analysis. 3rd ed. Springer.

Prince, Simon J. D. 2012. Computer Vision: Models, Learning, and Inference. Cambridge University Press. https://www.computervisionmodels.com/.

Rezende, Danilo Jimenez, Shakir Mohamed, and Daan Wierstra. 2014. “Stochastic Backpropagation and Approximate Inference in Deep Generative Models.” International Conference on Machine Learning, 1278–86.

Rudin, Leonid I., Stanley Osher, and Emad Fatemi. 1992. “Nonlinear Total Variation Based Noise Removal Algorithms.” Physica D: Nonlinear Phenomena 60 (1–4): 259–68.

Rue, Håvard, and Leonhard Held. 2005. Gaussian Markov Random Fields: Theory and Applications. Chapman; Hall/CRC.

# Probabilistic Models of Images ::: {.hero-box} Classical computer vision often asks for a deterministic answer: an edge map, a segmentation, a depth map, or a reconstruction. Modern vision also asks: **How uncertain is the answer? What assumptions produced it? What distribution over images or labels is being learned?** This chapter develops probabilistic models of images for MA Applied Mathematics and MS Statistics students, emphasizing likelihoods, priors, posterior inference, MAP estimation, random fields, and connections to modern generative AI. ::: ## Learning Goals After reading this chapter, students should be able to: 1. model images, labels, and latent scenes as random variables or random fields; 2. write likelihoods for common image noise models, including Gaussian, Laplace, Bernoulli, multinomial, and Poisson models; 3. derive maximum likelihood, MAP, posterior mean, and posterior predictive estimators; 4. interpret regularization as a negative log-prior; 5. connect classical denoising, segmentation, stereo, and reconstruction to Bayesian inference; 6. explain Markov random fields, Gibbs distributions, and pairwise smoothness priors; 7. analyze uncertainty using covariance, entropy, calibration, and posterior samples; 8. recognize how modern generative models, including VAEs, normalizing flows, and diffusion models, extend probabilistic vision. ::: {.math-spine} The central mathematical idea is that a vision problem is usually an **inverse problem with uncertainty**: $$ y = A(x) + \varepsilon, $$ where $x$ is an unknown image, scene, label field, or representation; $y$ is the observed data; $A$ is a forward measurement process; and $\varepsilon$ is noise or modeling error. Probability turns this schematic equation into likelihoods, priors, posteriors, and decisions. ::: ## 18.1 Why Probability Is Needed in Computer Vision An image rarely determines a unique interpretation. The same 2D photograph can be consistent with many possible 3D scenes, illuminations, textures, segmentations, camera poses, and object identities. This is not merely a computational inconvenience. It is a mathematical feature of the inverse problem. For example: - a noisy image may come from many possible clean images; - a pixel near an object boundary may have ambiguous class membership; - a short-baseline stereo pair may produce a broad depth distribution; - an occluded object may have several plausible completions; - a low-resolution image may correspond to many high-resolution images. A deterministic algorithm returns one estimate. A probabilistic model describes a full uncertainty structure. The probabilistic viewpoint introduces four basic objects. ::: {.cv-card} **Probabilistic vision objects** | Object | Meaning in vision | |---|---| | $p(y|x)$ | likelihood: how an unknown image/scene $x$ produces observation $y$ | | $p(x)$ | prior: which images/scenes are plausible before seeing $y$ | | $p(x|y)$ | posterior: uncertainty about $x$ after observing $y$ | | $\delta(y)$ | decision rule: an estimate, prediction, or action based on $y$ | ::: Bayes' rule is the bridge: $$ p(x|y)=\frac{p(y|x)p(x)}{p(y)}, \qquad p(y)=\int p(y|x)p(x)\,dx. $$ The denominator $p(y)$ is called the evidence or marginal likelihood. For many vision problems, it is difficult to compute exactly because $x$ may be a high-dimensional image, a depth map, a segmentation mask, a set of 3D points, or a neural-network latent code. ## 18.2 Images as Random Vectors and Random Fields Let a grayscale image have height $H$ and width $W$. A deterministic image is an array $$ I \in \mathbb{R}^{H\times W}. $$ A random image is a random vector $$ X = (X_1,\ldots,X_n) \in \mathbb{R}^n, \qquad n=HW, $$ where each coordinate $X_i$ corresponds to one pixel after vectorization. A color image is a random vector in $\mathbb{R}^{HWC}$. This vector notation is useful for linear algebra and statistics, but it hides spatial structure. A more geometric view treats an image as a random field $$ \{X_s:s\in\Omega_h\}, $$ where $\Omega_h$ is the discrete pixel grid. Neighboring pixels are usually statistically dependent. A model that ignores spatial dependence may be too weak for denoising, segmentation, and texture modeling. ### Independent-pixel model The simplest model assumes $$ p(x)=\prod_{i=1}^n p_i(x_i). $$ This can be useful for histograms, thresholding, and naive Bayes models, but it cannot express smoothness or texture. ### Correlated Gaussian image model A basic correlated model is $$ X \sim \mathcal{N}(\mu,\Sigma), $$ where $\Sigma_{ij}$ describes covariance between pixels $i$ and $j$. If nearby pixels have positive covariance, the model favors smooth images. The negative log-density is, up to an additive constant, $$ -\log p(x) = \frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu) + \frac{1}{2}\log\det\Sigma. $$ The precision matrix $Q=\Sigma^{-1}$ is especially important. Sparse precision matrices encode conditional independence and lead to Gaussian Markov random fields. ### Label fields For segmentation, we often use a discrete random field $$ Z_s\in\{1,\ldots,K\}, \qquad s\in\Omega_h. $$ The observed image intensities are $Y_s$, and the unknown labels are $Z_s$. A probabilistic segmentation model seeks $$ p(z|y) \propto p(y|z)p(z). $$ The likelihood describes how each region generates intensities. The prior describes spatial coherence of labels. ## 18.3 Forward Models and Likelihoods The likelihood $p(y|x)$ encodes the measurement model. It answers the question: > If the true image, scene, or label field were $x$, how likely would the observed data $y$ be? A common forward model is $$ y=A x+\varepsilon, $$ where $A$ may represent blurring, downsampling, projection, masking, tomographic measurement, or a camera model. ### Gaussian noise If $$ \varepsilon\sim \mathcal{N}(0,\sigma^2I), $$ then $$ p(y|x) = (2\pi\sigma^2)^{-m/2} \exp\left(-\frac{1}{2\sigma^2}\|y-Ax\|_2^2\right). $$ The negative log-likelihood is $$ -\log p(y|x) = \frac{1}{2\sigma^2}\|y-Ax\|_2^2+\text{constant}. $$ Thus least squares is maximum likelihood under Gaussian noise. ### Laplace noise and robust fitting If each noise coordinate has a Laplace distribution, $$ p(\varepsilon_i)=\frac{1}{2b}\exp\left(-\frac{|\varepsilon_i|}{b}\right), $$ then the negative log-likelihood is proportional to an $\ell_1$ residual: $$ -\log p(y|x) = \frac{1}{b}\|y-Ax\|_1+\text{constant}. $$ This model is more robust to outliers than the Gaussian model. ### Poisson noise Photon counts in low-light imaging are often modeled as Poisson: $$ Y_i|x_i \sim \operatorname{Poisson}((Ax)_i), $$ so $$ p(y|x)=\prod_i \frac{(Ax)_i^{y_i}e^{-(Ax)_i}}{y_i!}. $$ Ignoring constants independent of $x$, the negative log-likelihood is $$ -\log p(y|x) = \sum_i \left((Ax)_i-y_i\log (Ax)_i\right)+\text{constant}. $$ Poisson models are natural for microscopy, astronomy, medical imaging, and low-light photography. ### Bernoulli and categorical likelihoods For a binary mask $Y_s\in\{0,1\}$ with predicted probability $\pi_s$, the Bernoulli likelihood is $$ p(y_s|\pi_s)=\pi_s^{y_s}(1-\pi_s)^{1-y_s}. $$ The negative log-likelihood is binary cross-entropy: $$ -\log p(y|\pi) = -\sum_s \left[y_s\log \pi_s+(1-y_s)\log(1-\pi_s)\right]. $$ For semantic segmentation with $K$ classes, the categorical likelihood gives multiclass cross-entropy: $$ -\log p(y|\pi) = -\sum_s \sum_{k=1}^K 1\{y_s=k\}\log \pi_{s,k}. $$ This is the probabilistic foundation of many neural segmentation losses. ## Interactive 18.1: Likelihoods for Common Noise Models The following interactive compares negative log-likelihood shapes for Gaussian, Laplace, and Huber-style robust residuals. The independent variable is the residual $r=y-Ax$. ```{=html} <div class="interactive-card"> <h3>Interactive 18.1: Negative log-likelihood and residual models</h3> <p>Change the scale and compare how Gaussian, Laplace, and Huber residual penalties treat small and large errors.</p> <label for="ch18-likelihood-scale">scale parameter: <span id="ch18-likelihood-scale-value">1.00</span></label> <input id="ch18-likelihood-scale" type="range" min="0.40" max="2.50" step="0.05" value="1.00" style="width:100%;"> <div id="ch18-likelihood-plot" class="plotly-demo"></div> </div> <script> (function(){ function loadPlotly(callback){ if(window.Plotly){ callback(); return; } var s=document.createElement('script'); s.src='https://cdn.plot.ly/plotly-2.35.2.min.js'; s.onload=callback; document.head.appendChild(s); } function range(a,b,n){ var out=[]; for(var i=0;i<n;i++){ out.push(a+(b-a)*i/(n-1)); } return out; } function draw(){ var slider=document.getElementById('ch18-likelihood-scale'); if(!slider || !window.Plotly) return; var scale=parseFloat(slider.value); document.getElementById('ch18-likelihood-scale-value').textContent=scale.toFixed(2); var r=range(-5,5,401); var gaussian=r.map(function(x){ return 0.5*Math.pow(x/scale,2); }); var laplace=r.map(function(x){ return Math.abs(x)/scale; }); var delta=scale; var huber=r.map(function(x){ var ax=Math.abs(x); return ax<=delta ? 0.5*x*x/(delta*delta) : ax/delta-0.5; }); var data=[ {x:r,y:gaussian,type:'scatter',mode:'lines',name:'Gaussian: quadratic'}, {x:r,y:laplace,type:'scatter',mode:'lines',name:'Laplace: absolute value'}, {x:r,y:huber,type:'scatter',mode:'lines',name:'Huber-style robust'} ]; var layout={margin:{t:30,r:20,b:45,l:55},xaxis:{title:'residual r'},yaxis:{title:'negative log-likelihood up to constants'},legend:{orientation:'h'}}; Plotly.react('ch18-likelihood-plot',data,layout,{responsive:true,displayModeBar:false}); } function init(){ var slider=document.getElementById('ch18-likelihood-scale'); if(slider){ slider.addEventListener('input',draw); } draw(); } if(document.readyState==='loading') document.addEventListener('DOMContentLoaded',function(){loadPlotly(init);}); else loadPlotly(init); })(); </script> ``` The graph illustrates a statistical modeling principle. The loss function is not arbitrary: it usually corresponds to an assumed noise distribution. ## 18.4 Maximum Likelihood Estimation Suppose $y$ is observed and $x$ is unknown. The maximum likelihood estimator is $$ \widehat{x}_{\operatorname{ML}} \in \arg\max_x p(y|x). $$ Equivalently, $$ \widehat{x}_{\operatorname{ML}} \in \arg\min_x -\log p(y|x). $$ For Gaussian linear observations, $$ y=Ax+\varepsilon, \qquad \varepsilon\sim\mathcal{N}(0,\sigma^2I), $$ ML becomes ordinary least squares: $$ \widehat{x}_{\operatorname{ML}} \in \arg\min_x \|y-Ax\|_2^2. $$ If $A$ has full column rank, then $$ \widehat{x}_{\operatorname{ML}} =(A^TA)^{-1}A^Ty. $$ If $A$ is ill-conditioned or rank deficient, the ML estimate may be unstable. This is exactly why priors and regularization are central in image reconstruction. ## 18.5 Priors and Regularization A prior $p(x)$ describes which images are plausible before observing $y$. The MAP estimator is $$ \widehat{x}_{\operatorname{MAP}} \in \arg\max_x p(x|y) = \arg\max_x p(y|x)p(x). $$ Equivalently, $$ \widehat{x}_{\operatorname{MAP}} \in \arg\min_x \left\{-\log p(y|x)-\log p(x)\right\}. $$ Thus the data-fidelity term comes from the likelihood, while the regularization term comes from the negative log-prior. ### Gaussian prior and Tikhonov regularization Assume $$ x\sim \mathcal{N}(0,\tau^2I) $$ and $$ y|x\sim \mathcal{N}(Ax,\sigma^2I). $$ Then $$ \widehat{x}_{\operatorname{MAP}} = \arg\min_x \frac{1}{2\sigma^2}\|y-Ax\|_2^2 + \frac{1}{2\tau^2}\|x\|_2^2. $$ Multiplying by $2\sigma^2$ gives $$ \widehat{x}_{\operatorname{MAP}} = \arg\min_x \|y-Ax\|_2^2+\lambda\|x\|_2^2, \qquad \lambda=\frac{\sigma^2}{\tau^2}. $$ This is ridge or Tikhonov regularization. The closed form is $$ \widehat{x}_{\operatorname{MAP}} =(A^TA+\lambda I)^{-1}A^Ty. $$ ### Smoothness prior Images are not usually small in Euclidean norm. A more natural prior penalizes roughness: $$ p(x)\propto \exp\left(-\frac{\alpha}{2}\|Dx\|_2^2\right), $$ where $D$ is a finite-difference gradient operator. Then MAP estimation becomes $$ \widehat{x}_{\operatorname{MAP}} = \arg\min_x \frac{1}{2\sigma^2}\|y-Ax\|_2^2 + \frac{\alpha}{2}\|Dx\|_2^2. $$ This is a probabilistic version of smooth denoising and deblurring. ### Total variation prior Edges are important in vision. A quadratic smoothness penalty tends to oversmooth edges. Total variation uses an $\ell_1$ penalty on gradients: $$ \operatorname{TV}(x) = \sum_s \|\nabla x_s\|_2. $$ The corresponding prior has the form $$ p(x)\propto \exp(-\alpha\operatorname{TV}(x)). $$ The MAP estimator is $$ \widehat{x}_{\operatorname{MAP}} = \arg\min_x \frac{1}{2\sigma^2}\|y-Ax\|_2^2 + \alpha\operatorname{TV}(x). $$ This is edge-preserving because it allows sparse large gradients. ## Interactive 18.2: MAP Denoising and the Bias-Variance Tradeoff This demo shows a one-dimensional analogy of image denoising. The noisy observation is smoothed by a MAP estimator with a quadratic smoothness prior. Increasing $\lambda$ reduces noise but also removes detail. ```{=html} <div class="interactive-card"> <h3>Interactive 18.2: MAP smoothing from a Gaussian prior</h3> <p>The estimator minimizes squared data error plus a quadratic roughness penalty.</p> <label for="ch18-map-lambda">regularization strength λ: <span id="ch18-map-lambda-value">1.0</span></label> <input id="ch18-map-lambda" type="range" min="0" max="50" step="0.5" value="1" style="width:100%;"> <div id="ch18-map-plot" class="plotly-demo"></div> </div> <script> (function(){ function loadPlotly(callback){ if(window.Plotly){ callback(); return; } var s=document.createElement('script'); s.src='https://cdn.plot.ly/plotly-2.35.2.min.js'; s.onload=callback; document.head.appendChild(s); } function makeData(){ var n=80, x=[], clean=[], noisy=[]; for(var i=0;i<n;i++){ var t=i/(n-1); x.push(t); var c=Math.sin(2*Math.PI*t)+0.45*Math.sin(6*Math.PI*t)+(t>0.55?0.7:0); clean.push(c); var deterministicNoise=0.35*Math.sin(97*t)+0.22*Math.cos(173*t)+0.12*Math.sin(311*t); noisy.push(c+deterministicNoise); } return {x:x,clean:clean,noisy:noisy}; } function smooth(y,lambda){ var n=y.length, out=y.slice(), tmp=y.slice(); var alpha=lambda/(1+lambda); var iter=120; for(var k=0;k<iter;k++){ tmp[0]=(y[0]+lambda*out[1])/(1+lambda); tmp[n-1]=(y[n-1]+lambda*out[n-2])/(1+lambda); for(var i=1;i<n-1;i++){ tmp[i]=(y[i]+lambda*(out[i-1]+out[i+1]))/(1+2*lambda); } var swap=out; out=tmp; tmp=swap; } return out; } var data0=makeData(); function draw(){ var slider=document.getElementById('ch18-map-lambda'); if(!slider || !window.Plotly) return; var lambda=parseFloat(slider.value); document.getElementById('ch18-map-lambda-value').textContent=lambda.toFixed(1); var est=smooth(data0.noisy,lambda); var data=[ {x:data0.x,y:data0.clean,type:'scatter',mode:'lines',name:'clean signal'}, {x:data0.x,y:data0.noisy,type:'scatter',mode:'markers',name:'noisy observation'}, {x:data0.x,y:est,type:'scatter',mode:'lines',name:'MAP estimate'} ]; var layout={margin:{t:30,r:20,b:45,l:55},xaxis:{title:'position'},yaxis:{title:'intensity'},legend:{orientation:'h'}}; Plotly.react('ch18-map-plot',data,layout,{responsive:true,displayModeBar:false}); } function init(){ var s=document.getElementById('ch18-map-lambda'); if(s){s.addEventListener('input',draw);} draw(); } if(document.readyState==='loading') document.addEventListener('DOMContentLoaded',function(){loadPlotly(init);}); else loadPlotly(init); })(); </script> ``` The slider is a concrete version of a general question in probabilistic vision: > How strongly should prior belief constrain the estimate relative to the observed data? ## 18.6 Posterior Mean, MAP, and Decision Theory The MAP estimate is only one Bayesian estimator. If the loss function is squared error, $$ L(x,a)=\|x-a\|_2^2, $$ then the Bayes estimator is the posterior mean: $$ \widehat{x}_{\operatorname{PM}}=\mathbb{E}[X|Y=y]. $$ If the loss is absolute error in one dimension, $$ L(x,a)=|x-a|, $$ then a posterior median is optimal. If the target is a class label and the loss is zero-one loss, the optimal classifier is $$ \widehat{z}(y)=\arg\max_k p(z=k|y). $$ This is the maximum posterior probability rule. The key lesson is that an estimator is not only a formula. It is the solution to a decision problem involving a posterior distribution and a loss function. ### MAP can be misleading For a highly asymmetric or multimodal posterior, the MAP estimate may be unrepresentative. Consider a depth posterior with two modes: one corresponding to a nearby object and one corresponding to a far object. The MAP returns only one mode. The posterior mean may fall between two physically plausible depths. A calibrated uncertainty report may be more informative than either point estimate. ## 18.7 Bayesian Linear Inverse Problems The Gaussian linear model is one of the cleanest mathematical laboratories for probabilistic vision. Assume $$ y|x\sim \mathcal{N}(Ax,\Sigma_\varepsilon), \qquad x\sim \mathcal{N}(\mu_0,\Sigma_0). $$ Then the posterior is also Gaussian: $$ x|y\sim \mathcal{N}(\mu_{\operatorname{post}},\Sigma_{\operatorname{post}}), $$ where $$ \Sigma_{\operatorname{post}} = \left(\Sigma_0^{-1}+A^T\Sigma_\varepsilon^{-1}A\right)^{-1}, $$ and $$ \mu_{\operatorname{post}} = \Sigma_{\operatorname{post}} \left(\Sigma_0^{-1}\mu_0+A^T\Sigma_\varepsilon^{-1}y\right). $$ For Gaussian posteriors, the posterior mean and MAP estimate coincide. ### Interpretation The posterior precision is $$ \Sigma_{\operatorname{post}}^{-1} = \Sigma_0^{-1}+A^T\Sigma_\varepsilon^{-1}A. $$ This formula says that information from the prior and information from the data add in precision space. A strong prior has large $\Sigma_0^{-1}$. Accurate measurements have large $\Sigma_\varepsilon^{-1}$. ### Image deblurring example If $A$ is a blur matrix, then deblurring is ill-posed because high frequencies are strongly attenuated. A prior supplies information about those unstable directions. In Fourier-like coordinates, the posterior variance is large where both the prior and the data provide little information. ## 18.8 Markov Random Fields and Gibbs Priors For label fields and images, spatial dependence is essential. A Markov random field models local interactions. Let $G=(V,E)$ be a graph whose nodes are pixels and whose edges connect neighboring pixels. A random field $X=(X_s)_{s\in V}$ is Markov with respect to $G$ if $$ p(x_s|x_{V\setminus\{s\}})=p(x_s|x_{N(s)}), $$ where $N(s)$ is the set of neighbors of $s$. A Gibbs distribution has the form $$ p(x)=\frac{1}{Z}\exp(-E(x)), $$ where $$ Z=\sum_x \exp(-E(x)) $$ or, for continuous variables, $$ Z=\int \exp(-E(x))\,dx. $$ The normalizing constant $Z$ is called the partition function. It is often computationally difficult. ### Pairwise prior for binary labels For binary labels $z_s\in\{0,1\}$, a common prior is $$ p(z)\propto \exp\left(-\beta\sum_{(s,t)\in E} 1\{z_s\ne z_t\}\right). $$ When $\beta>0$, neighboring labels prefer to be equal. This is a probabilistic version of segmentation smoothness. The MAP segmentation problem has energy $$ E(z) = \sum_s D_s(z_s) + \beta\sum_{(s,t)\in E} 1\{z_s\ne z_t\}, $$ where $D_s(k)$ is a data cost, often $-\log p(y_s|z_s=k)$. This connects directly to the graph-cut energies from Chapter 8. ### Gaussian Markov random fields For continuous images, a Gaussian Markov random field may use energy $$ E(x)=\frac{1}{2}\sum_s \frac{(y_s-x_s)^2}{\sigma^2} + \frac{\beta}{2}\sum_{(s,t)\in E}(x_s-x_t)^2. $$ The first term fits the observed data. The second term encourages neighboring pixels to be similar. In matrix form, $$ E(x)=\frac{1}{2\sigma^2}\|y-x\|_2^2 + \frac{\beta}{2}x^TLx, $$ where $L$ is the graph Laplacian. The MAP estimate solves the linear system $$ \left(\frac{1}{\sigma^2}I+\beta L\right)x = \frac{1}{\sigma^2}y. $$ ## Interactive 18.3: Pairwise MRF Prior for Binary Segmentation This demo shows how the pairwise parameter $\beta$ rewards spatial coherence. Larger $\beta$ makes isolated label changes more expensive. ```{=html} <div class="interactive-card"> <h3>Interactive 18.3: MRF pairwise smoothness</h3> <p>The displayed energy is a toy binary segmentation energy: data cost plus a penalty for neighboring label disagreements.</p> <label for="ch18-mrf-beta">pairwise strength β: <span id="ch18-mrf-beta-value">0.60</span></label> <input id="ch18-mrf-beta" type="range" min="0" max="2" step="0.05" value="0.60" style="width:100%;"> <div id="ch18-mrf-plot" class="plotly-demo"></div> </div> <script> (function(){ function loadPlotly(callback){ if(window.Plotly){ callback(); return; } var s=document.createElement('script'); s.src='https://cdn.plot.ly/plotly-2.35.2.min.js'; s.onload=callback; document.head.appendChild(s); } var base=[ [0,0,0,0,1,1,1,1], [0,0,0,1,1,1,1,1], [0,0,1,1,1,1,0,1], [0,0,1,1,1,0,0,0], [0,1,1,1,0,0,0,0], [1,1,1,0,0,0,0,0], [1,1,0,0,0,0,0,0], [1,1,1,0,0,0,0,0] ]; var unary=[ [0.2,0.2,0.3,0.4,0.6,0.7,0.8,0.9], [0.2,0.2,0.3,0.6,0.7,0.8,0.8,0.9], [0.3,0.4,0.7,0.8,0.8,0.7,0.4,0.8], [0.3,0.4,0.8,0.9,0.7,0.4,0.3,0.2], [0.4,0.7,0.8,0.8,0.4,0.3,0.2,0.2], [0.7,0.8,0.7,0.4,0.3,0.2,0.2,0.2], [0.8,0.7,0.4,0.3,0.2,0.2,0.2,0.2], [0.9,0.8,0.7,0.3,0.2,0.2,0.2,0.2] ]; function energy(beta){ var h=base.length,w=base[0].length,e=0,disc=0; for(var i=0;i<h;i++){ for(var j=0;j<w;j++){ var z=base[i][j]; e += z ? (1-unary[i][j]) : unary[i][j]; if(i+1<h && base[i][j]!==base[i+1][j]) disc++; if(j+1<w && base[i][j]!==base[i][j+1]) disc++; } } return {total:e+beta*disc,unary:e,pair:beta*disc,disc:disc}; } function boundaryMap(beta){ var h=base.length,w=base[0].length,z=[]; for(var i=0;i<h;i++){ var row=[]; for(var j=0;j<w;j++){ var b=0; if(i>0 && base[i][j]!==base[i-1][j]) b++; if(j>0 && base[i][j]!==base[i][j-1]) b++; if(i+1<h && base[i][j]!==base[i+1][j]) b++; if(j+1<w && base[i][j]!==base[i][j+1]) b++; row.push(base[i][j]+0.25*beta*b); } z.push(row); } return z; } function draw(){ var slider=document.getElementById('ch18-mrf-beta'); if(!slider || !window.Plotly) return; var beta=parseFloat(slider.value); document.getElementById('ch18-mrf-beta-value').textContent=beta.toFixed(2); var e=energy(beta); var traces=[ {z:boundaryMap(beta),type:'heatmap',showscale:false,name:'labels'}, {x:['data cost','pairwise cost','total'],y:[e.unary,e.pair,e.total],type:'bar',xaxis:'x2',yaxis:'y2',name:'energy'} ]; var layout={ margin:{t:45,r:20,b:45,l:45}, grid:{rows:1,columns:2,pattern:'independent'}, title:'neighbor disagreements = '+e.disc, xaxis:{showticklabels:false},yaxis:{showticklabels:false,scaleanchor:'x'}, xaxis2:{title:'energy components'},yaxis2:{title:'energy'} }; Plotly.react('ch18-mrf-plot',traces,layout,{responsive:true,displayModeBar:false}); } function init(){ var s=document.getElementById('ch18-mrf-beta'); if(s){s.addEventListener('input',draw);} draw(); } if(document.readyState==='loading') document.addEventListener('DOMContentLoaded',function(){loadPlotly(init);}); else loadPlotly(init); })(); </script> ``` This small example is intentionally discrete. It prepares students for probabilistic segmentation, graph cuts, belief propagation, and conditional random fields. ## 18.9 Mixture Models and Probabilistic Segmentation A useful model for intensity-based segmentation is a Gaussian mixture model. Suppose each pixel intensity $Y_s\in\mathbb{R}^d$ belongs to a latent class $Z_s\in\{1,\ldots,K\}$. Let $$ P(Z_s=k)=\pi_k, \qquad Y_s|Z_s=k\sim \mathcal{N}(\mu_k,\Sigma_k). $$ The marginal density is $$ p(y_s)=\sum_{k=1}^K \pi_k\,\mathcal{N}(y_s;\mu_k,\Sigma_k). $$ The posterior class probability is $$ \gamma_{s,k} =P(Z_s=k|Y_s=y_s) = \frac{\pi_k\mathcal{N}(y_s;\mu_k,\Sigma_k)}{\sum_{\ell=1}^K \pi_\ell\mathcal{N}(y_s;\mu_\ell,\Sigma_\ell)}. $$ These posterior probabilities are soft segmentations. ### EM algorithm For independent pixels, the log-likelihood is $$ \ell(\theta) = \sum_s \log\left(\sum_{k=1}^K \pi_k\mathcal{N}(y_s;\mu_k,\Sigma_k)\right). $$ The EM algorithm alternates: **E-step:** compute responsibilities $$ \gamma_{s,k}^{(t)} = \frac{\pi_k^{(t)}\mathcal{N}(y_s;\mu_k^{(t)},\Sigma_k^{(t)})}{\sum_{\ell=1}^K \pi_\ell^{(t)}\mathcal{N}(y_s;\mu_\ell^{(t)},\Sigma_\ell^{(t)})}. $$ **M-step:** update parameters $$ N_k^{(t)}=\sum_s \gamma_{s,k}^{(t)}, $$ $$ \pi_k^{(t+1)}=\frac{N_k^{(t)}}{n}, $$ $$ \mu_k^{(t+1)}=\frac{1}{N_k^{(t)}}\sum_s \gamma_{s,k}^{(t)}y_s, $$ $$ \Sigma_k^{(t+1)}=\frac{1}{N_k^{(t)}}\sum_s \gamma_{s,k}^{(t)}(y_s-\mu_k^{(t+1)})(y_s-\mu_k^{(t+1)})^T. $$ This model does not enforce spatial smoothness. Adding an MRF prior on labels produces a more realistic segmentation model but makes inference harder. ## Interactive 18.4: Gaussian Mixture Responsibilities The next demo shows soft class probabilities in a one-dimensional mixture model. Notice that uncertainty is highest near class overlap. ```{=html} <div class="interactive-card"> <h3>Interactive 18.4: Mixture-model posterior probabilities</h3> <p>Move the component means closer or farther apart and observe how the posterior responsibilities change.</p> <label for="ch18-gmm-sep">mean separation: <span id="ch18-gmm-sep-value">2.5</span></label> <input id="ch18-gmm-sep" type="range" min="0.4" max="5.0" step="0.1" value="2.5" style="width:100%;"> <div id="ch18-gmm-plot" class="plotly-demo"></div> </div> <script> (function(){ function loadPlotly(callback){ if(window.Plotly){ callback(); return; } var s=document.createElement('script'); s.src='https://cdn.plot.ly/plotly-2.35.2.min.js'; s.onload=callback; document.head.appendChild(s); } function range(a,b,n){ var out=[]; for(var i=0;i<n;i++){ out.push(a+(b-a)*i/(n-1)); } return out; } function normal(x,mu,sig){ return Math.exp(-0.5*Math.pow((x-mu)/sig,2))/(sig*Math.sqrt(2*Math.PI)); } function draw(){ var slider=document.getElementById('ch18-gmm-sep'); if(!slider || !window.Plotly) return; var sep=parseFloat(slider.value); document.getElementById('ch18-gmm-sep-value').textContent=sep.toFixed(1); var x=range(-6,6,501), sig=1.0, mu1=-sep/2, mu2=sep/2; var p1=[],p2=[],mix=[],r1=[],r2=[],ent=[]; for(var i=0;i<x.length;i++){ var a=0.5*normal(x[i],mu1,sig), b=0.5*normal(x[i],mu2,sig), m=a+b; p1.push(a); p2.push(b); mix.push(m); var q1=a/m, q2=b/m; r1.push(q1); r2.push(q2); ent.push(-(q1*Math.log(q1+1e-12)+q2*Math.log(q2+1e-12))/Math.log(2)); } var data=[ {x:x,y:p1,type:'scatter',mode:'lines',name:'component 1 density'}, {x:x,y:p2,type:'scatter',mode:'lines',name:'component 2 density'}, {x:x,y:mix,type:'scatter',mode:'lines',name:'mixture density'}, {x:x,y:r1,type:'scatter',mode:'lines',name:'P(class 1 | y)',xaxis:'x2',yaxis:'y2'}, {x:x,y:r2,type:'scatter',mode:'lines',name:'P(class 2 | y)',xaxis:'x2',yaxis:'y2'}, {x:x,y:ent,type:'scatter',mode:'lines',name:'posterior entropy',xaxis:'x2',yaxis:'y2'} ]; var layout={ margin:{t:30,r:20,b:45,l:50}, grid:{rows:1,columns:2,pattern:'independent'}, xaxis:{title:'intensity y'},yaxis:{title:'density'}, xaxis2:{title:'intensity y'},yaxis2:{title:'probability / entropy',range:[0,1.05]}, legend:{orientation:'h'} }; Plotly.react('ch18-gmm-plot',data,layout,{responsive:true,displayModeBar:false}); } function init(){ var s=document.getElementById('ch18-gmm-sep'); if(s){s.addEventListener('input',draw);} draw(); } if(document.readyState==='loading') document.addEventListener('DOMContentLoaded',function(){loadPlotly(init);}); else loadPlotly(init); })(); </script> ``` The soft responsibilities $\gamma_{s,k}$ are a statistically principled alternative to hard thresholding. ## 18.10 Conditional Models and Discriminative Probability The models above often begin with a generative story $p(y|x)p(x)$ or $p(y,z)$. In supervised computer vision, we often model the conditional distribution directly: $$ p_\theta(z|y). $$ For classification, a neural network may output logits $a_k(y)$ and probabilities $$ p_\theta(z=k|y) = \frac{\exp(a_k(y))}{\sum_{\ell=1}^K\exp(a_\ell(y))}. $$ Training by cross-entropy minimizes $$ \widehat{R}(\theta) = -\frac{1}{n}\sum_{i=1}^n \log p_\theta(z_i|y_i). $$ This is maximum likelihood for the conditional model. ### Calibration A classifier is calibrated if predicted probabilities match empirical frequencies. Informally, among examples for which the model predicts probability $0.8$, approximately $80\%$ should be correct. One common calibration score is expected calibration error. Partition predictions into confidence bins $B_1,\ldots,B_M$. Then $$ \operatorname{ECE} = \sum_{m=1}^M \frac{|B_m|}{n} \left|\operatorname{acc}(B_m)-\operatorname{conf}(B_m)\right|. $$ For high-stakes vision systems, such as medical imaging and autonomous driving, calibrated uncertainty may be as important as accuracy. ## Interactive 18.5: Calibration of Vision Predictions This demo compares an ideal calibrated classifier with an overconfident classifier. ```{=html} <div class="interactive-card"> <h3>Interactive 18.5: Reliability diagram</h3> <p>Overconfident models have predicted confidence larger than empirical accuracy.</p> <label for="ch18-calibration-over">overconfidence level: <span id="ch18-calibration-over-value">0.25</span></label> <input id="ch18-calibration-over" type="range" min="0" max="0.45" step="0.01" value="0.25" style="width:100%;"> <div id="ch18-calibration-plot" class="plotly-demo"></div> </div> <script> (function(){ function loadPlotly(callback){ if(window.Plotly){ callback(); return; } var s=document.createElement('script'); s.src='https://cdn.plot.ly/plotly-2.35.2.min.js'; s.onload=callback; document.head.appendChild(s); } function draw(){ var slider=document.getElementById('ch18-calibration-over'); if(!slider || !window.Plotly) return; var over=parseFloat(slider.value); document.getElementById('ch18-calibration-over-value').textContent=over.toFixed(2); var conf=[],acc=[],gap=[]; for(var i=1;i<=10;i++){ var c=(i-0.5)/10; conf.push(c); var a=Math.max(0,Math.min(1,c-over*Math.pow(c,0.65))); acc.push(a); gap.push(c-a); } var ece=0; for(var j=0;j<conf.length;j++){ ece+=Math.abs(conf[j]-acc[j])/conf.length; } var data=[ {x:conf,y:acc,type:'bar',name:'empirical accuracy'}, {x:[0,1],y:[0,1],type:'scatter',mode:'lines',name:'perfect calibration'}, {x:conf,y:gap,type:'scatter',mode:'lines+markers',name:'confidence gap',xaxis:'x2',yaxis:'y2'} ]; var layout={ margin:{t:45,r:20,b:50,l:55}, title:'approximate ECE = '+ece.toFixed(3), grid:{rows:1,columns:2,pattern:'independent'}, xaxis:{title:'confidence bin',range:[0,1]},yaxis:{title:'accuracy',range:[0,1]}, xaxis2:{title:'confidence bin',range:[0,1]},yaxis2:{title:'gap',range:[0,0.5]}, legend:{orientation:'h'} }; Plotly.react('ch18-calibration-plot',data,layout,{responsive:true,displayModeBar:false}); } function init(){ var s=document.getElementById('ch18-calibration-over'); if(s){s.addEventListener('input',draw);} draw(); } if(document.readyState==='loading') document.addEventListener('DOMContentLoaded',function(){loadPlotly(init);}); else loadPlotly(init); })(); </script> ``` A model can have high accuracy and poor calibration. This distinction becomes important when probabilities are used for downstream decisions. ## 18.11 Uncertainty in Vision Models Uncertainty can be separated into two broad types. ::: {.cv-card} **Aleatoric uncertainty** is uncertainty caused by noise or inherent ambiguity in the data. For example, a low-light image has photon noise, and an occluded object may be intrinsically ambiguous. **Epistemic uncertainty** is uncertainty caused by limited knowledge of model parameters or limited training data. It can often be reduced by collecting more relevant data. ::: A probabilistic model may represent uncertainty through: - posterior covariance matrices; - marginal posterior variances; - entropy of class probabilities; - predictive distributions; - ensembles; - Bayesian neural networks; - Monte Carlo dropout; - diffusion or generative posterior samples. ### Predictive distribution Given parameters $\theta$, a conditional model predicts $$ p(y_*|x_*,\theta). $$ In Bayesian modeling, parameters are uncertain, so the predictive distribution integrates over their posterior: $$ p(y_*|x_*,\mathcal{D}) = \int p(y_*|x_*,\theta)p(\theta|\mathcal{D})\,d\theta. $$ This integral is usually approximated. ### Entropy for segmentation If a neural segmentation model outputs probabilities $p_{s,k}$ at pixel $s$, the pixelwise entropy is $$ H_s=-\sum_{k=1}^K p_{s,k}\log p_{s,k}. $$ High entropy indicates uncertain class membership. Entropy maps are useful for quality control, active learning, and human review. ## 18.12 Probabilistic Graphical Models for Vision A probabilistic graphical model represents a joint distribution using a graph. Nodes represent random variables. Edges represent dependence. Examples in computer vision include: - hidden Markov models for video and tracking; - conditional random fields for segmentation; - Bayesian networks for object relationships; - factor graphs for SLAM and bundle adjustment; - diffusion graphical models for generative modeling. A factor graph writes the joint density as $$ p(x)\propto \prod_{a\in\mathcal{F}} \psi_a(x_a), $$ where each factor $\psi_a$ depends only on a subset $x_a$ of variables. Taking negative logarithms gives an additive energy: $$ E(x)=-\sum_{a\in\mathcal{F}}\log \psi_a(x_a)+\text{constant}. $$ This explains why so many vision objectives are sums of local terms. Local probabilistic dependencies become additive optimization objectives. ## 18.13 Sampling and Approximate Inference Exact posterior inference is often impossible in high-dimensional vision models. Common approximation methods include: 1. **Laplace approximation:** approximate the posterior near a mode by a Gaussian; 2. **variational inference:** approximate $p(x|y)$ by a simpler distribution $q_\phi(x)$; 3. **Monte Carlo methods:** draw approximate samples from the posterior; 4. **belief propagation:** pass messages on a graph; 5. **mean-field approximation:** replace a coupled distribution by a factorized one. ### Laplace approximation Suppose $$ p(x|y)\propto \exp(-E(x)). $$ Let $\widehat{x}$ be a local minimizer of $E$. The second-order Taylor approximation is $$ E(x)\approx E(\widehat{x})+rac{1}{2}(x-\widehat{x})^TH(x-\widehat{x}), $$ where $$ H=\nabla^2E(\widehat{x}). $$ Then $$ p(x|y)\approx \mathcal{N}(\widehat{x},H^{-1}). $$ This connects optimization curvature to uncertainty. ### Variational inference Variational inference chooses a tractable family $q_\phi(x)$ and minimizes $$ \operatorname{KL}(q_\phi(x)\|p(x|y)). $$ Equivalently, it maximizes an evidence lower bound. This idea is central to variational autoencoders and many modern latent-variable models. ## 18.14 Modern Generative Models as Probabilistic Image Models Deep learning did not remove probability from vision. It made probabilistic models more expressive. ### Variational autoencoders A variational autoencoder introduces a latent variable $z$: $$ z\sim p(z), \qquad x|z\sim p_\theta(x|z). $$ The marginal likelihood is $$ p_\theta(x)=\int p_\theta(x|z)p(z)\,dz. $$ Because this integral is often intractable, VAEs use an approximate posterior $q_\phi(z|x)$ and optimize the ELBO: $$ \log p_\theta(x) \ge \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)] - \operatorname{KL}(q_\phi(z|x)\|p(z)). $$ The first term is a reconstruction term. The second term regularizes the latent distribution. ### Normalizing flows A normalizing flow constructs an invertible map $$ x=f_\theta(z), \qquad z\sim p_Z. $$ By the change-of-variables formula, $$ p_X(x) = p_Z(f_\theta^{-1}(x)) \left|\det \frac{\partial f_\theta^{-1}}{\partial x}\right|. $$ Flows give exact likelihoods but require invertible architectures. ### Diffusion models Diffusion models define a forward noising process and learn a reverse denoising process. A common forward model is $$ q(x_t|x_{t-1}) = \mathcal{N}(\sqrt{1-\beta_t}x_{t-1},\beta_tI). $$ After many steps, $x_T$ is approximately noise. The learned reverse model estimates how to move from noise back toward data. Diffusion models can be understood as very high-dimensional probabilistic image models with learned denoising transitions. Chapter 29 returns to diffusion models in detail. Here the important point is conceptual: modern generative AI is built from likelihoods, latent variables, priors, scores, and approximate inference. ## 18.15 AI Components: Probabilistic Thinking for Modern Vision Systems Modern AI vision systems use probability in several ways. ### Cross-entropy training For classification and segmentation, cross-entropy is negative log-likelihood for categorical outputs. This turns supervised learning into conditional maximum likelihood. ### Diffusion and score models Denoising models estimate information about the score $$ \nabla_x \log p_t(x), $$ which points toward higher-density regions of a noised data distribution. This gives a probabilistic interpretation of learned denoisers. ### Uncertainty-aware neural networks A model may output both a mean and a variance. For a regression target $y$, suppose $$ y|x\sim\mathcal{N}(\mu_\theta(x),\sigma_\theta^2(x)). $$ The negative log-likelihood is $$ \frac{1}{2}\log \sigma_\theta^2(x) + \frac{(y-\mu_\theta(x))^2}{2\sigma_\theta^2(x)} + \text{constant}. $$ This objective lets the model learn data-dependent uncertainty. ### Segmentation confidence and active learning For segmentation, uncertainty maps can identify pixels or images for human annotation. One acquisition score is entropy: $$ a(y)=\sum_s H(p_\theta(z_s|y)). $$ Images with high entropy may be informative for retraining. ### Probabilistic prompts for AI study When using an AI assistant to study probabilistic vision, ask it to state assumptions explicitly: - What is the random variable? - What is observed and what is latent? - What is the likelihood? - What is the prior? - What loss function justifies the estimator? - Which independence assumptions are being made? - How would the answer change under Gaussian, Laplace, or Poisson noise? ## 18.16 Python Example: Noise Models and Negative Log-Likelihoods The following code compares Gaussian, Laplace, and Poisson negative log-likelihoods. It is written for independent study; execute it in a notebook after installing `numpy` and `matplotlib`. ```{python} #| label: py-ch18-noise-likelihoods #| eval: false import numpy as np import matplotlib.pyplot as plt r = np.linspace(-5, 5, 500) sigma = 1.0 b = 1.0 nll_gaussian = 0.5 * (r / sigma) ** 2 nll_laplace = np.abs(r) / b plt.figure(figsize=(7, 4)) plt.plot(r, nll_gaussian, label="Gaussian noise: quadratic loss") plt.plot(r, nll_laplace, label="Laplace noise: absolute loss") plt.xlabel("residual r") plt.ylabel("negative log-likelihood, up to constants") plt.title("Noise assumptions determine loss functions") plt.legend() plt.show() ``` For Poisson noise: ```{python} #| label: py-ch18-poisson-nll #| eval: false lam = np.linspace(0.05, 10, 500) y_obs = 3 # Negative log-likelihood up to constants independent of lambda nll_poisson = lam - y_obs * np.log(lam) plt.figure(figsize=(7, 4)) plt.plot(lam, nll_poisson) plt.axvline(y_obs, linestyle="--", label="MLE lambda = observed count") plt.xlabel("Poisson rate lambda") plt.ylabel("negative log-likelihood") plt.title("Poisson likelihood for a photon count") plt.legend() plt.show() ``` ## 18.17 Python Example: MAP Denoising with a Graph Laplacian This example performs one-dimensional MAP denoising using a smoothness prior. The same idea extends to 2D images by replacing the 1D Laplacian with a 2D grid Laplacian. ```{python} #| label: py-ch18-map-denoising-laplacian #| eval: false import numpy as np import matplotlib.pyplot as plt rng = np.random.default_rng(7243) n = 160 t = np.linspace(0, 1, n) x_true = np.sin(2 * np.pi * t) + 0.6 * (t > 0.55) y = x_true + 0.25 * rng.normal(size=n) # First-difference matrix D, shape (n - 1, n) D = np.zeros((n - 1, n)) for i in range(n - 1): D[i, i] = -1 D[i, i + 1] = 1 lam = 15.0 A = np.eye(n) + lam * D.T @ D x_map = np.linalg.solve(A, y) plt.figure(figsize=(8, 4)) plt.plot(t, x_true, label="true signal") plt.scatter(t, y, s=12, alpha=0.6, label="noisy observations") plt.plot(t, x_map, linewidth=2, label="MAP estimate") plt.xlabel("position") plt.ylabel("intensity") plt.title("MAP denoising with a quadratic smoothness prior") plt.legend() plt.show() ``` ## 18.18 Python Example: Gaussian Mixture Segmentation The next example fits a Gaussian mixture model to pixel intensities and interprets posterior responsibilities as soft labels. ```{python} #| label: py-ch18-gmm-segmentation #| eval: false import numpy as np import matplotlib.pyplot as plt from sklearn.mixture import GaussianMixture rng = np.random.default_rng(18) H, W = 120, 160 Y, X = np.mgrid[0:H, 0:W] mask = ((X - 80) ** 2 / 45 ** 2 + (Y - 60) ** 2 / 28 ** 2) < 1 image = np.where(mask, 0.75, 0.25) + 0.12 * rng.normal(size=(H, W)) image = np.clip(image, 0, 1) pixels = image.reshape(-1, 1) gmm = GaussianMixture(n_components=2, covariance_type="full", random_state=0) gmm.fit(pixels) responsibilities = gmm.predict_proba(pixels) # Choose the component with larger mean as foreground foreground_component = np.argmax(gmm.means_.ravel()) soft_mask = responsibilities[:, foreground_component].reshape(H, W) fig, axes = plt.subplots(1, 3, figsize=(11, 3.5)) axes[0].imshow(image, cmap="gray") axes[0].set_title("noisy image") axes[1].imshow(mask, cmap="gray") axes[1].set_title("true mask") axes[2].imshow(soft_mask, cmap="gray", vmin=0, vmax=1) axes[2].set_title("posterior foreground probability") for ax in axes: ax.axis("off") plt.tight_layout() plt.show() ``` This example is intentionally simple. It separates intensity modeling from spatial regularization. Adding an MRF prior would reduce isolated mistakes. ## 18.19 Python Example: Pixelwise Entropy for Segmentation Uncertainty Suppose a segmentation model outputs pixelwise class probabilities. Entropy gives a simple uncertainty map. ```{python} #| label: py-ch18-segmentation-entropy #| eval: false import numpy as np import matplotlib.pyplot as plt rng = np.random.default_rng(9) H, W, K = 80, 100, 3 logits = rng.normal(size=(H, W, K)) # Make the center more ambiguous by shrinking logits there Y, X = np.mgrid[0:H, 0:W] r2 = (X - W / 2) ** 2 + (Y - H / 2) ** 2 ambiguity = np.exp(-r2 / (2 * 18 ** 2)) logits = logits * (1 - 0.8 * ambiguity[..., None]) exp_logits = np.exp(logits - logits.max(axis=-1, keepdims=True)) probs = exp_logits / exp_logits.sum(axis=-1, keepdims=True) entropy = -np.sum(probs * np.log(probs + 1e-12), axis=-1) plt.figure(figsize=(6, 4)) plt.imshow(entropy, cmap="magma") plt.colorbar(label="entropy") plt.title("Pixelwise segmentation uncertainty") plt.axis("off") plt.show() ``` ## 18.20 TensorFlow Example: Learning Mean and Uncertainty A neural model can output both a mean and a variance. This is useful when the noise level depends on the input. ```{python} #| label: py-ch18-tensorflow-heteroscedastic #| eval: false import tensorflow as tf inputs = tf.keras.Input(shape=(64, 64, 3)) x = tf.keras.layers.Conv2D(16, 3, padding="same", activation="relu")(inputs) x = tf.keras.layers.MaxPooling2D()(x) x = tf.keras.layers.Conv2D(32, 3, padding="same", activation="relu")(x) x = tf.keras.layers.GlobalAveragePooling2D()(x) x = tf.keras.layers.Dense(64, activation="relu")(x) # Example: predict a scalar depth value and log variance mu = tf.keras.layers.Dense(1, name="mean_prediction")(x) log_var = tf.keras.layers.Dense(1, name="log_variance_prediction")(x) model = tf.keras.Model(inputs=inputs, outputs=[mu, log_var]) def gaussian_nll(y_true, outputs): mu_pred, log_var_pred = outputs return tf.reduce_mean( 0.5 * log_var_pred + 0.5 * tf.square(y_true - mu_pred) * tf.exp(-log_var_pred) ) ``` In practice, one must regularize and validate uncertainty predictions. A model can learn to inflate variance unless the objective, data, and evaluation are carefully designed. ## 18.21 AI-Assisted Mathematical Study Prompts Use these prompts to study the chapter with an AI assistant. They are designed to force explicit assumptions and derivations. ::: {.ai-box} **Prompt 1: Derive a loss from a likelihood.** I have a vision inverse problem $y=Ax+\varepsilon$. Derive the negative log-likelihood for Gaussian, Laplace, and Poisson noise. Explain which loss function each model produces and when each assumption is reasonable. ::: ::: {.ai-box} **Prompt 2: Interpret a regularizer as a prior.** Given the MAP objective $\|y-Ax\|_2^2+\lambda\|Dx\|_2^2$, identify the likelihood and prior assumptions. Explain the role of $D$ and derive the normal equations. ::: ::: {.ai-box} **Prompt 3: Compare MAP and posterior mean.** Give examples of posterior distributions for which the MAP estimate and posterior mean are close, and examples where they are very different. Explain the decision-theoretic reason. ::: ::: {.ai-box} **Prompt 4: Diagnose uncertainty.** A segmentation model produces high softmax probabilities but makes many mistakes. Explain the difference between accuracy and calibration. Suggest a reliability diagram and entropy-based diagnostics. ::: ::: {.ai-box} **Prompt 5: Connect classical and modern vision.** Explain how Markov random fields, graph cuts, cross-entropy neural networks, VAEs, and diffusion models all fit into the probabilistic modeling framework of likelihoods, priors, posteriors, and approximate inference. ::: ## 18.22 Common Pitfalls ### Pitfall 1: Treating softmax probabilities as automatically calibrated Softmax outputs sum to one, but that does not imply they are calibrated probabilities. Overconfident deep networks can produce high confidence for wrong predictions. ### Pitfall 2: Confusing a loss function with an arbitrary engineering choice Many losses correspond to negative log-likelihoods. Squared error, absolute error, cross-entropy, and Poisson loss each encode different probabilistic assumptions. ### Pitfall 3: Forgetting the prior in ill-posed inverse problems When $A$ destroys information, the data alone cannot recover $x$. Super-resolution, deblurring, inpainting, and monocular depth all rely on priors, whether explicit or learned. ### Pitfall 4: Reporting one estimate without uncertainty For ambiguous visual tasks, one point estimate may hide important alternatives. Posterior samples, entropy maps, confidence intervals, and calibrated probabilities often reveal more. ### Pitfall 5: Assuming independent pixels Independent-pixel models are mathematically convenient but often miss the spatial structure that makes images images. ## 18.23 Summary This chapter introduced probabilistic image modeling as the statistical foundation of modern computer vision. The main ideas are: - images, labels, scenes, and representations can be modeled as random variables or random fields; - likelihoods encode noise and measurement assumptions; - priors encode plausibility, smoothness, sparsity, edges, geometry, or learned structure; - MAP estimation is regularized optimization; - posterior means, MAP estimates, and classifiers correspond to different decision losses; - Markov random fields connect probability, graph models, and energy minimization; - mixture models produce soft probabilistic segmentations; - calibration and uncertainty matter in real vision systems; - VAEs, flows, and diffusion models are modern probabilistic image models. The next chapters move from general probabilistic modeling to supervised learning and representation learning for vision. ## Exercises ### Conceptual Exercises 1. Explain why image denoising is an inverse problem. Identify $x$, $y$, $A$, and $\varepsilon$ in the simplest denoising model. 2. Give one computer vision example where Gaussian noise is reasonable and one where Poisson noise is more reasonable. 3. Explain why a spatial prior is important for segmentation. 4. What is the difference between aleatoric and epistemic uncertainty? Give a vision example of each. 5. Why can MAP estimation be misleading for a multimodal posterior? ### Mathematical Exercises 1. Let $y|x\sim \mathcal{N}(x,\sigma^2I)$ and $x\sim\mathcal{N}(0,\tau^2I)$. Derive the MAP estimate. 2. Let $y=Ax+\varepsilon$ with $\varepsilon\sim\mathcal{N}(0,\sigma^2I)$ and prior $p(x)\propto\exp(-\alpha\|Dx\|_2^2/2)$. Derive the normal equations for the MAP estimator. 3. Show that the negative log-likelihood for independent Bernoulli observations is binary cross-entropy. 4. For a two-class Gaussian mixture with equal variances and equal mixing weights, derive the posterior probability $P(Z=1|Y=y)$. 5. Let $p(x)\propto\exp(-\beta x^TLx/2)$ where $L$ is a graph Laplacian. Explain why this prior favors smooth signals on the graph. 6. Derive the posterior covariance for the Gaussian linear inverse problem. 7. Show that the posterior mean minimizes posterior expected squared error. 8. For a categorical distribution $p=(p_1,\ldots,p_K)$, prove that entropy is maximized when $p_k=1/K$ for all $k$. ### Python Exercises 1. Simulate Gaussian, Laplace, and Poisson noise on the same grayscale image. Compare the empirical histograms of residuals. 2. Implement MAP denoising with a 2D finite-difference operator. 3. Fit a Gaussian mixture model to a color image using RGB features. Display posterior probability maps for each component. 4. Implement a reliability diagram for a trained classifier or a synthetic classifier. 5. Compute entropy maps from pixelwise softmax outputs. 6. Compare quadratic smoothness and total variation denoising on an image with sharp edges. ### AI and Modeling Exercises 1. Ask an AI assistant to derive the MAP objective for a denoising problem. Then check whether it clearly identifies the likelihood and prior. 2. Ask an AI assistant for a probabilistic model for semantic segmentation. Require it to distinguish $p(y|z)$, $p(z)$, and $p(z|y)$. 3. Ask an AI assistant to explain why cross-entropy is a negative log-likelihood. Verify the derivation line by line. 4. Ask an AI assistant to propose uncertainty diagnostics for a medical image classifier. Critique which diagnostics measure calibration and which measure discrimination. 5. Ask an AI assistant to compare VAEs, flows, and diffusion models as probabilistic image models. Summarize the difference in likelihood tractability. ## Notes and References The probabilistic view of computer vision is developed in @prince2012computer and connects naturally to statistical learning in @bishop2006pattern. Markov random fields and Gibbs distributions are classical tools for spatial image modeling; see @geman1984stochastic, @li2009markov, and @rue2005gaussian. Variational and Bayesian deep generative models include VAEs @kingma2013auto, @rezende2014stochastic, normalizing flows @dinh2016density, and diffusion models @ho2020denoising. Total variation regularization is historically central in image restoration; see @rudin1992nonlinear.

Object	Meaning in vision
\(p(y\|x)\)	likelihood: how an unknown image/scene \(x\) produces observation \(y\)
\(p(x)\)	prior: which images/scenes are plausible before seeing \(y\)
\(p(x\|y)\)	posterior: uncertainty about \(x\) after observing \(y\)
\(\delta(y)\)	decision rule: an estimate, prediction, or action based on \(y\)