25 Chapter 24. Semantic and Instance Segmentation

Pixelwise Labels, Masks, Dice Loss, U-Net, Mask R-CNN, Panoptic Segmentation, and Promptable Models

25.1 24.1 From boxes to masks

Object detection localizes objects by bounding boxes. Segmentation asks for a more detailed geometric output: which pixels belong to which object, region, or class? Instead of predicting only a rectangle, a segmentation model predicts a label field, a collection of masks, or a probability map.

Let an image be

\[ X\in\mathbb R^{H\times W\times C}. \]

A semantic segmentation target is a discrete function

\[ y:\{1,\dots,H\}\times \{1,\dots,W\}\to \{1,\dots,K\}, \]

where each pixel receives one of $K$ class labels. Equivalently, the target can be represented by one-hot indicator functions

\[ Y_{ijk}=\mathbf 1\{y_{ij}=k\}, \qquad Y\in\{0,1\}^{H\times W\times K}. \]

A model predicts class probabilities

\[ P_\theta(k\mid X,i,j), \qquad \sum_{k=1}^K P_\theta(k\mid X,i,j)=1. \]

Instance segmentation is different. It predicts a set of object instances

\[ \widehat{\mathcal M}(X) = \{(\widehat M_r,\widehat c_r,\widehat s_r):r=1,\dots,\widehat m\}, \]

where $\widehat M_r\subseteq \Omega$ is a binary mask, $\widehat c_r$ is an object class, $\widehat s_r$ is a confidence score, and $\widehat m$ is the predicted number of instances. The order of the masks does not matter, so instance segmentation is a set-valued structured prediction problem.

Panoptic segmentation combines both viewpoints. Each pixel is assigned a semantic class and, for countable object categories, an instance identity. In informal language:

semantic segmentation: every pixel gets a class;
instance segmentation: each object instance gets a mask;
panoptic segmentation: every pixel gets both class information and, when appropriate, instance identity.

Central idea

Segmentation is dense statistical prediction with geometric structure. The output is not a single label but a spatially organized field or a set of masks. The mathematics combines pixelwise likelihoods, set overlap, variational regularization, structured prediction, and learned representations.

25.1.1 Learning goals

After this chapter, you should be able to:

distinguish semantic, instance, and panoptic segmentation mathematically;
represent masks as indicator functions, probability fields, signed distance functions, and sets;
derive pixelwise cross-entropy from maximum likelihood;
compute IoU, Dice, boundary metrics, and panoptic quality;
explain why segmentation losses must handle class imbalance and spatial correlation;
describe fully convolutional networks, encoder-decoder architectures, skip connections, U-Net, DeepLab-style dilation, and Mask R-CNN;
formulate instance segmentation as detection plus mask prediction or as set prediction;
explain promptable segmentation and its relation to foundation models;
write Python experiments for segmentation losses and evaluation metrics;
use AI tools to check mask conventions, loss definitions, and evaluation code.

25.2 24.2 Masks as mathematical objects

A binary mask is a function

\[ M:\Omega_h\to\{0,1\}, \]

where

\[ \Omega_h=\{1,\dots,H\}\times\{1,\dots,W\} \]

is the discrete image grid. We may also view a mask as a subset

\[ A=\{(i,j)\in\Omega_h:M(i,j)=1\}. \]

The area of the mask is the counting measure

\[ |A|=\sum_{i=1}^H\sum_{j=1}^W M(i,j). \]

For soft segmentation, the mask is replaced by a probability field

\[ P:\Omega_h\to[0,1]. \]

A hard mask can be obtained by thresholding:

\[ \widehat M_\tau(i,j)=\mathbf 1\{P(i,j)\ge \tau\}. \]

The threshold $\tau$ is not merely an implementation detail. It defines a decision rule. Changing $\tau$ changes false positives, false negatives, IoU, Dice score, and boundary behavior.

25.2.1 Indicator functions and vector spaces

Although a binary mask is a set, it can also be embedded in a vector space:

\[ M\in\{0,1\}^{HW}\subset \mathbb R^{HW}. \]

This representation allows us to use inner products:

\[ \langle M,N\rangle = \sum_{i,j}M(i,j)N(i,j) =|A\cap B|. \]

Also,

\[ \|M\|_1=|A|, \qquad \|M-N\|_1=|A\triangle B|, \]

where $A\triangle B$ is the symmetric difference. Thus, many segmentation metrics can be written using linear algebra.

25.2.2 Signed distance representation

For geometric segmentation, it is useful to represent a region by a signed distance function

\[ \phi(x)= \begin{cases} -d(x,\partial A), & x\in A,\\ \phantom{-}d(x,\partial A), & x\notin A. \end{cases} \]

Then the mask is recovered by

\[ M(x)=\mathbf 1\{\phi(x)\le 0\}. \]

This representation connects segmentation to level sets, active contours, implicit neural fields, and differentiable shape optimization.

25.3 24.3 Semantic segmentation as pixelwise classification

For semantic segmentation, each pixel has a categorical label. A neural network often outputs logits

\[ z_\theta(X)_{ijk}\in\mathbb R, \]

and probabilities are obtained by the softmax

\[ p_{ijk} = \frac{\exp(z_{ijk})}{\sum_{\ell=1}^K\exp(z_{ij\ell})}. \]

If the true one-hot label is $Y_{ijk}$, the pixelwise negative log-likelihood is

\[ \ell_{ij}(\theta) = -\sum_{k=1}^K Y_{ijk}\log p_{ijk}. \]

The full image loss is usually averaged over pixels:

\[ \mathcal L_{\mathrm{CE}}(\theta) = -\frac{1}{HW}\sum_{i=1}^H\sum_{j=1}^W\sum_{k=1}^K Y_{ijk}\log p_{ijk}. \]

This is the same cross-entropy used for classification, but now repeated at every pixel. The model is trained as if each pixel contributes a supervised example. However, pixels are spatially correlated, so this is not an independent data model in a strict statistical sense. The pixelwise likelihood is best understood as a practical surrogate objective.

Statistical caution

Pixelwise cross-entropy treats pixels as loss terms, not as independent observations. A $512\times512$ mask does not provide $262{,}144$ independent labels. The effective sample size is much smaller because neighboring pixels are strongly dependent.

25.3.1 Class-weighted cross-entropy

In segmentation, background pixels often dominate. If class $k$ is rare, ordinary cross-entropy may produce a model that ignores it. A weighted loss is

\[ \mathcal L_{\mathrm{WCE}} = -\frac{1}{HW}\sum_{i,j,k}w_kY_{ijk}\log p_{ijk}, \]

where $w_k>0$ is larger for rare or clinically important classes. A common heuristic is inverse class frequency:

\[ w_k\propto \frac{1}{\widehat\pi_k}, \qquad \widehat\pi_k=\frac{1}{NHW}\sum_{n=1}^N\sum_{i,j}Y_{nijk}. \]

But extremely large weights can make optimization unstable. In practice, one often clips or smooths these weights.

25.3.2 Focal loss for dense imbalance

For binary segmentation with target $y\in\{0,1\}$ and predicted probability $p$, define

\[ p_t= \begin{cases} p, & y=1,\\ 1-p, & y=0. \end{cases} \]

The focal loss is

\[ \ell_{\mathrm{focal}} =-\alpha(1-p_t)^\gamma\log(p_t), \]

where $\gamma\ge 0$. When $p_t$ is near $1$, the sample is already easy and the factor $(1-p_t)^\gamma$ downweights it. Focal loss was introduced for dense detection, but the same idea is useful in dense segmentation.

25.4 24.4 Overlap metrics: IoU and Dice

Segmentation is usually evaluated by set overlap. For a predicted mask $\widehat A$ and a true mask $A$, the intersection-over-union is

\[ \operatorname{IoU}(\widehat A,A) = \frac{|\widehat A\cap A|}{|\widehat A\cup A|}. \]

Using indicator functions $\widehat M$ and $M$,

\[ |\widehat A\cap A|=\sum_{i,j}\widehat M_{ij}M_{ij}, \]

and

\[ |\widehat A\cup A|= \sum_{i,j}\widehat M_{ij}+ \sum_{i,j}M_{ij} - \sum_{i,j}\widehat M_{ij}M_{ij}. \]

The Dice coefficient is

\[ \operatorname{Dice}(\widehat A,A) = \frac{2|\widehat A\cap A|}{|\widehat A|+|A|}. \]

Dice and IoU are monotone transforms of each other for fixed pair of masks:

\[ \operatorname{Dice} = \frac{2\operatorname{IoU}}{1+\operatorname{IoU}}, \qquad \operatorname{IoU} = \frac{\operatorname{Dice}}{2-\operatorname{Dice}}. \]

This means Dice and IoU rank the same pairwise overlaps in the same order, although their numerical values differ.

25.4.1 Soft Dice loss

For a soft predicted mask $P_{ij}\in[0,1]$ and binary target $M_{ij}\in\{0,1\}$, the soft Dice coefficient is

\[ D(P,M)= \frac{2\sum_{i,j}P_{ij}M_{ij}+\varepsilon} {\sum_{i,j}P_{ij}+\sum_{i,j}M_{ij}+\varepsilon}, \]

where $\varepsilon>0$ prevents division by zero. A Dice loss is

\[ \mathcal L_{\mathrm{Dice}}=1-D(P,M). \]

Dice loss directly rewards overlap and is often more stable than pure cross-entropy for small objects. It is also nonseparable across pixels because the denominator couples all pixels in the mask.

25.4.2 Tversky index

The Tversky index generalizes Dice by weighting false positives and false negatives differently:

\[ T_{\alpha,\beta}(\widehat A,A) = \frac{|\widehat A\cap A|} {|\widehat A\cap A|+ \alpha|\widehat A\setminus A|+ \beta|A\setminus \widehat A|}. \]

If missing the object is more costly than including extra pixels, choose $\beta>\alpha$. This is useful in medical imaging, where false negatives may be more serious than false positives.

25.5 24.5 Interactive demo: IoU, Dice, and thresholding

The same probability map can produce different masks under different thresholds. The following demo shows how the threshold affects IoU, Dice, false positives, and false negatives.

threshold: 0.50

25.6 24.6 Boundary metrics and why overlap is not enough

Two masks can have similar IoU but different boundary quality. This matters when the object boundary is scientifically important, such as in tumor segmentation, cell segmentation, road extraction, and anatomical measurement.

Let $\partial A$ and $\partial \widehat A$ denote the boundaries of two regions. The directed Hausdorff distance is

\[ h(\partial A,\partial \widehat A) = \max_{x\in\partial A}\min_{y\in\partial\widehat A}\|x-y\|_2. \]

The symmetric Hausdorff distance is

\[ H(\partial A,\partial \widehat A) = \max\{h(\partial A,\partial \widehat A),h(\partial \widehat A,\partial A)\}. \]

The average symmetric surface distance is less sensitive to one extreme outlier:

\[ \operatorname{ASSD} = \frac{1}{|\partial A|+|\partial\widehat A|} \left( \sum_{x\in\partial A}d(x,\partial\widehat A) + \sum_{y\in\partial\widehat A}d(y,\partial A) \right). \]

Boundary-sensitive losses often use distance transforms. If $\phi_A$ is the signed distance function of the true mask, a boundary loss can penalize predicted mass according to distance from the true boundary:

\[ \mathcal L_{\mathrm{boundary}} = \sum_{i,j} P_{ij}\phi_A(i,j). \]

More generally, distance-transform losses connect mask prediction to geometric measure theory: one is no longer just classifying pixels but estimating a shape.

25.7 24.7 Fully convolutional networks

A fully convolutional network, or FCN, replaces dense classification layers with convolutional layers so that the model outputs a spatial map rather than one image-level label (Long et al. 2015).

Let an encoder produce feature maps

\[ F_\ell\in\mathbb R^{H_\ell\times W_\ell\times C_\ell}. \]

A segmentation head maps these features to logits

\[ Z\in\mathbb R^{H\times W\times K}. \]

Because repeated pooling or strided convolution reduces spatial resolution, the network must upsample. A simple approach is bilinear interpolation. A learned approach is transposed convolution. Another approach is a decoder that gradually combines low-resolution semantic features with high-resolution spatial features.

25.7.1 Why convolution is natural for segmentation

Segmentation requires the model to preserve spatial structure. A convolutional layer is translation equivariant:

\[ T_a(f*K)=(T_af)*K, \]

where $T_a$ is translation by $a$. If an object moves in the image, the segmentation mask should move with it. Thus, equivariance is a structural prior for dense prediction.

However, segmentation also requires context. A single pixel may be ambiguous without its neighborhood. Therefore, segmentation models must balance:

local detail: sharp boundaries and small objects;
global context: object identity, scene layout, and long-range dependencies.

This local-global tension explains many architecture choices: skip connections, dilation, multi-scale features, attention, and prompt-conditioned masks.

25.8 24.8 Encoder-decoder architectures and U-Net

An encoder-decoder model has two parts. The encoder compresses the image into increasingly abstract features:

\[ X\mapsto F_1\mapsto F_2\mapsto\cdots\mapsto F_L. \]

The decoder reconstructs a high-resolution prediction:

\[ F_L\mapsto G_{L-1}\mapsto\cdots\mapsto G_1\mapsto Z. \]

U-Net adds skip connections between encoder and decoder layers of matching resolution (Ronneberger et al. 2015). In a simplified notation,

\[ G_\ell = \Psi_\ell\big(\operatorname{Upsample}(G_{\ell+1}),F_\ell\big), \]

where $F_\ell$ is an encoder feature map and $G_\ell$ is a decoder feature map. The operation $\Psi_\ell$ may concatenate features followed by convolutions.

The skip connection has a mathematical interpretation: high-resolution encoder features provide local coordinate information, while low-resolution decoder features provide semantic context. Without skip connections, the decoder may recover class identity but lose boundary precision.

25.8.1 U-Net as a multiscale approximation scheme

For applied mathematics students, it is helpful to compare U-Net to multigrid or wavelet ideas. The encoder produces coarser representations, while the decoder refines resolution. Skip connections pass fine-scale information around the bottleneck.

The architecture can be interpreted as learning a nonlinear multiscale operator

\[ \mathcal S_\theta:\mathbb R^{H\times W\times C}\to [0,1]^{H\times W\times K}. \]

The model is not simply classifying pixels independently. It learns a nonlinear, multiscale map from an image function to a label function.

25.9 24.9 Interactive demo: skip connections and boundary recovery

This schematic demo shows the effect of combining coarse semantic information with high-resolution detail. The coarse map captures the object approximately; the detail map sharpens the boundary.

detail weight: 0.45

25.10 24.10 Dilated convolution and DeepLab-style context

Pooling increases receptive field but reduces resolution. Dilated convolution increases receptive field without the same reduction in resolution. In one dimension, a dilated convolution with dilation rate $r$ is

\[ (f *_r k)[i]=\sum_m k[m]f[i-rm]. \]

In two dimensions,

\[ (f *_r K)[i,j] = \sum_{u,v}K[u,v]f[i-ru,j-rv]. \]

A $3\times3$ kernel with dilation $r$ samples a grid spanning effective size

\[ 1+2r. \]

DeepLab-style models use dilated convolutions and multi-scale context aggregation to improve semantic segmentation, especially for large objects and scene layouts.

25.10.1 Atrous spatial pyramid pooling

Atrous spatial pyramid pooling applies several dilated convolutions with different dilation rates:

\[ G_r=K_r *_r F, \qquad r\in\mathcal R. \]

The outputs are concatenated:

\[ G=\operatorname{concat}\{G_r:r\in\mathcal R\}. \]

This gives a learned multiscale feature representation. Mathematically, it resembles using multiple basis functions at different spatial scales.

25.11 24.11 Interactive demo: dilation and receptive field

A larger dilation increases the receptive field without increasing the number of kernel parameters.

dilation rate: 2

25.12 24.12 Conditional random fields and structured smoothing

Pixelwise neural predictions are often noisy at boundaries. A classical way to impose structure is to use a conditional random field. Let $y_i$ denote the label at pixel $i$. A typical energy is

\[ E(y)=\sum_i \psi_i(y_i)+\sum_{i<j}\psi_{ij}(y_i,y_j), \]

where the unary term comes from the network logits:

\[ \psi_i(k)=-\log p_i(k), \]

and the pairwise term encourages nearby or visually similar pixels to share labels:

\[ \psi_{ij}(y_i,y_j) = \mathbf 1\{y_i\ne y_j\}w_{ij}. \]

A common weight has the form

\[ w_{ij} =\lambda\exp\left(-\frac{\|p_i-p_j\|^2}{2\sigma_p^2} -\frac{\|I_i-I_j\|^2}{2\sigma_I^2}\right), \]

where $p_i$ is pixel position and $I_i$ is color. This couples segmentation to both geometry and appearance.

Modern architectures often learn these smoothing effects implicitly, but the CRF viewpoint remains valuable because it makes the structured-prediction assumptions explicit.

25.13 24.13 Instance segmentation

In instance segmentation, the model must separate different objects of the same class. For example, if an image contains three people, semantic segmentation may assign all person pixels the label “person,” while instance segmentation must produce three separate masks.

A common approach is detection plus mask prediction. First predict boxes and object classes. Then predict a binary mask inside each region of interest. Mask R-CNN follows this strategy by adding a mask head to Faster R-CNN (He et al. 2017).

For a predicted region proposal $r$, the mask head produces

\[ P_r\in[0,1]^{m\times m\times K}, \]

or often one binary mask per class. The mask is trained using binary cross-entropy on the ground-truth mask cropped to the proposal.

25.13.1 RoI alignment

A key operation is to extract a fixed-size feature tensor from a variable-size proposal. RoI pooling used quantization. RoI Align avoids harsh quantization by bilinear interpolation. If $F$ is a feature map and $(x,y)$ is a noninteger coordinate, bilinear interpolation gives

\[ F(x,y)= \sum_{a\in\{\lfloor x\rfloor,\lceil x\rceil\}} \sum_{b\in\{\lfloor y\rfloor,\lceil y\rceil\}} w_{ab}(x,y)F(a,b). \]

This matters because instance masks are sensitive to spatial alignment.

25.13.2 Instance mask loss

For a proposal assigned to class $c$, the binary mask loss is

\[ \mathcal L_{\mathrm{mask}} = -\frac{1}{m^2}\sum_{u=1}^m\sum_{v=1}^m \left[ M_{uv}\log P_{uv}^{(c)}+(1-M_{uv})\log(1-P_{uv}^{(c)}) \right]. \]

The total training loss combines classification, box regression, and mask loss:

\[ \mathcal L =\mathcal L_{\mathrm{cls}}+ \mathcal L_{\mathrm{box}}+ \mathcal L_{\mathrm{mask}}. \]

25.14 24.14 Instance matching and average precision for masks

Instance segmentation evaluation uses mask IoU rather than box IoU. A predicted mask is a true positive at threshold $\tau$ if its class is correct and

\[ \operatorname{IoU}(\widehat M,M)\ge \tau. \]

Predictions are sorted by confidence. Each predicted mask can match at most one ground-truth instance, and each ground-truth instance can be matched at most once. Precision and recall are computed over the sorted list, producing an average precision curve.

The mean average precision averages AP over classes and overlap thresholds. The COCO-style metric uses multiple thresholds, which rewards both detection and precise mask boundaries.

25.14.1 Why instance segmentation is not only pixel classification

Suppose two adjacent cells touch each other. Semantic segmentation can correctly label every foreground pixel as “cell” but still fail to split the two instances. Instance segmentation requires additional structure: object centers, contours, embeddings, watershed markers, or set prediction.

Some models solve this by predicting:

object centers and offsets;
boundary maps;
instance embeddings;
masks associated with detected boxes;
transformer queries that represent object instances.

25.15 24.15 Interactive demo: instance masks and matching

The following demo illustrates mask matching by IoU. A prediction with high overlap is matched to a ground truth instance; another prediction may be counted as duplicate or false positive.

prediction shift: 6

25.16 24.16 Panoptic segmentation

Panoptic segmentation assigns every pixel to exactly one semantic class and, for object-like categories, an instance identity. Classes are often divided into:

things: countable objects such as person, car, dog;
stuff: amorphous regions such as sky, road, grass.

A panoptic output can be represented as

\[ y(i,j)=(c(i,j),q(i,j)), \]

where $c(i,j)$ is the semantic class and $q(i,j)$ is the instance ID when the class is a thing category. Stuff regions may use a special null instance ID.

Panoptic quality is often decomposed into segmentation quality and recognition quality. Let $\operatorname{TP}$ be matched segments, $\operatorname{FP}$ unmatched predictions, and $\operatorname{FN}$ unmatched ground-truth segments. Then

\[ \operatorname{PQ} = \frac{\sum_{(p,g)\in\operatorname{TP}}\operatorname{IoU}(p,g)} {|\operatorname{TP}|+\frac12|\operatorname{FP}|+\frac12|\operatorname{FN}|}. \]

This can be factored as

\[ \operatorname{PQ}=\operatorname{SQ}\cdot\operatorname{RQ}, \]

where

\[ \operatorname{SQ} = \frac{1}{|\operatorname{TP}|}\sum_{(p,g)\in\operatorname{TP}}\operatorname{IoU}(p,g), \]

and

\[ \operatorname{RQ} = \frac{|\operatorname{TP}|} {|\operatorname{TP}|+\frac12|\operatorname{FP}|+\frac12|\operatorname{FN}|}. \]

Thus PQ separates mask quality from detection/recognition quality.

25.17 24.17 Promptable segmentation and foundation models

Modern segmentation increasingly uses foundation models. A promptable segmentation model takes an image and a prompt, such as points, boxes, rough masks, or text-like guidance, and returns masks. Segment Anything is a central example of this paradigm (Kirillov et al. 2023).

Mathematically, a promptable segmentation model is a conditional map

\[ \mathcal S_\theta:(X,p)\mapsto \{(M_r,s_r):r=1,\dots,R\}, \]

where $p$ is a prompt and $s_r$ are mask confidence scores. The prompt changes the conditional distribution:

\[ P_\theta(M\mid X,p). \]

This is a significant shift from closed-set segmentation. Instead of learning one fixed mapping from image to class labels, the model learns an interface between human intention and visual masks.

25.17.1 Prompt types as constraints

A positive point prompt $a\in\Omega$ imposes the constraint

\[ M(a)=1. \]

A negative point prompt $b\in\Omega$ imposes

\[ M(b)=0. \]

A box prompt imposes support constraints:

\[ M(x)=0\quad\text{for many }x\notin B. \]

A mask prompt gives an initial estimate. From an optimization viewpoint, prompts act like constraints, priors, or conditioning variables.

25.17.2 Why promptable segmentation is useful

Promptable segmentation is useful when the desired object category is not known in advance or when the user wants one particular object among many possibilities. It supports annotation, editing, medical review, interactive analysis, and weakly supervised pipelines.

For a statistics student, the key question is uncertainty: different plausible masks may satisfy the same prompt. A good system should produce calibrated confidence or multiple candidate masks when the prompt is ambiguous.

25.18 24.18 Interactive demo: point prompts as constraints

This simplified demo shows how positive and negative point prompts can select different masks from the same image-like score field.

positive point x-coordinate: 28

25.19 24.19 Segmentation uncertainty

Segmentation models often output probabilities, but probability maps are not automatically calibrated. A pixel probability $p=0.9$ should mean that, among similar pixels assigned probability $0.9$, about $90\%$ are truly positive. This is a calibration statement, not just a confidence score.

For binary segmentation, the Brier score is

\[ \operatorname{Brier} =\frac{1}{HW}\sum_{i,j}(P_{ij}-M_{ij})^2. \]

Expected calibration error bins predicted probabilities and compares average confidence with empirical accuracy:

\[ \operatorname{ECE} =\sum_{b=1}^B \frac{|S_b|}{HW} \left|\operatorname{acc}(S_b)-\operatorname{conf}(S_b)\right|. \]

Uncertainty can arise from:

image noise;
ambiguous boundaries;
annotation disagreement;
out-of-distribution images;
model uncertainty;
prompt ambiguity.

A mathematically careful segmentation study should report not only masks but also uncertainty, calibration, and failure modes.

25.20 24.20 Segmentation as empirical risk minimization

Let $\{(X_n,Y_n)\}_{n=1}^N$ be a training set. A general segmentation objective is

\[ \widehat R(\theta) = \frac1N\sum_{n=1}^N \mathcal L(f_\theta(X_n),Y_n) +\lambda\Omega(\theta), \]

where $f_\theta$ is the segmentation model, $\mathcal L$ may combine cross-entropy, Dice, boundary, and auxiliary losses, and $\Omega$ is a regularizer.

A common combined loss for binary segmentation is

\[ \mathcal L =\lambda_{\mathrm{CE}}\mathcal L_{\mathrm{CE}} +\lambda_{\mathrm{Dice}}\mathcal L_{\mathrm{Dice}} +\lambda_{\mathrm{bdry}}\mathcal L_{\mathrm{boundary}}. \]

This is not merely engineering. It reflects competing statistical and geometric goals:

cross-entropy improves probabilistic calibration;
Dice improves overlap for imbalanced regions;
boundary loss improves contour accuracy;
regularization improves generalization and smoothness.

25.21 24.21 Python example: metrics for binary masks

The following code computes IoU, Dice, precision, recall, and a soft Dice loss. It is designed to be readable before being efficient.

import numpy as np


def binary_metrics(y_true, y_prob, threshold=0.5, eps=1e-8):
    """Compute common binary segmentation metrics."""
    y_true = np.asarray(y_true).astype(bool)
    y_pred = np.asarray(y_prob) >= threshold

    tp = np.logical_and(y_pred, y_true).sum()
    fp = np.logical_and(y_pred, ~y_true).sum()
    fn = np.logical_and(~y_pred, y_true).sum()
    tn = np.logical_and(~y_pred, ~y_true).sum()

    iou = tp / (tp + fp + fn + eps)
    dice = 2 * tp / (2 * tp + fp + fn + eps)
    precision = tp / (tp + fp + eps)
    recall = tp / (tp + fn + eps)

    return {
        "tp": int(tp), "fp": int(fp), "fn": int(fn), "tn": int(tn),
        "iou": float(iou), "dice": float(dice),
        "precision": float(precision), "recall": float(recall)
    }


def soft_dice_loss(y_true, y_prob, eps=1e-6):
    y_true = np.asarray(y_true, dtype=float)
    y_prob = np.asarray(y_prob, dtype=float)
    intersection = np.sum(y_true * y_prob)
    denominator = np.sum(y_true) + np.sum(y_prob)
    dice = (2 * intersection + eps) / (denominator + eps)
    return 1.0 - dice

# Example synthetic mask
H, W = 96, 96
y, x = np.ogrid[:H, :W]
true_mask = (x - 48) ** 2 + (y - 48) ** 2 <= 24 ** 2
prob_map = np.exp(-((x - 51) ** 2 + (y - 47) ** 2) / (2 * 20 ** 2))

print(binary_metrics(true_mask, prob_map, threshold=0.5))
print("soft Dice loss:", soft_dice_loss(true_mask, prob_map))

25.22 24.22 Python example: semantic segmentation data tensors

A common source of bugs is mixing class-index masks and one-hot masks. The following example shows both conventions.

import numpy as np

H, W, K = 128, 128, 4
class_mask = np.zeros((H, W), dtype=np.int64)

# Create simple synthetic regions.
class_mask[20:70, 20:80] = 1
class_mask[60:110, 50:120] = 2
class_mask[30:100, 95:115] = 3

# One-hot encoding: shape (H, W, K)
one_hot = np.eye(K, dtype=np.float32)[class_mask]

print("class-index mask shape:", class_mask.shape)
print("one-hot mask shape:", one_hot.shape)
print("pixel class probabilities should also have shape:", one_hot.shape)

# Recover class labels from predicted probabilities.
logits = np.random.normal(size=(H, W, K))
prob = np.exp(logits) / np.exp(logits).sum(axis=-1, keepdims=True)
pred_class = prob.argmax(axis=-1)
print("predicted class mask shape:", pred_class.shape)

25.23 24.23 Python example: a small U-Net in TensorFlow

This is a minimal educational U-Net-style architecture. For serious experiments, add proper data loading, augmentation, validation, checkpointing, metrics, and careful preprocessing.

import tensorflow as tf
from tensorflow.keras import layers, Model


def conv_block(x, filters):
    x = layers.Conv2D(filters, 3, padding="same", activation="relu")(x)
    x = layers.Conv2D(filters, 3, padding="same", activation="relu")(x)
    return x


def tiny_unet(input_shape=(128, 128, 3), num_classes=2):
    inputs = layers.Input(shape=input_shape)

    c1 = conv_block(inputs, 32)
    p1 = layers.MaxPool2D()(c1)

    c2 = conv_block(p1, 64)
    p2 = layers.MaxPool2D()(c2)

    b = conv_block(p2, 128)

    u2 = layers.UpSampling2D(interpolation="bilinear")(b)
    u2 = layers.Concatenate()([u2, c2])
    c3 = conv_block(u2, 64)

    u1 = layers.UpSampling2D(interpolation="bilinear")(c3)
    u1 = layers.Concatenate()([u1, c1])
    c4 = conv_block(u1, 32)

    logits = layers.Conv2D(num_classes, 1, padding="same")(c4)
    outputs = layers.Softmax(axis=-1)(logits)

    return Model(inputs, outputs)

model = tiny_unet(num_classes=3)
model.summary()

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

25.24 24.24 TensorFlow example: Dice loss and combined loss

import tensorflow as tf


def soft_dice_loss_tf(y_true, y_pred, eps=1e-6):
    """
    y_true and y_pred should both have shape (batch, H, W, K).
    y_true is one-hot. y_pred contains probabilities.
    """
    y_true = tf.cast(y_true, tf.float32)
    y_pred = tf.cast(y_pred, tf.float32)

    axes = [1, 2]
    intersection = tf.reduce_sum(y_true * y_pred, axis=axes)
    denominator = tf.reduce_sum(y_true + y_pred, axis=axes)
    dice = (2.0 * intersection + eps) / (denominator + eps)

    # Average over classes and batch.
    return 1.0 - tf.reduce_mean(dice)


def combined_segmentation_loss(y_true_sparse, y_pred):
    num_classes = tf.shape(y_pred)[-1]
    y_true_one_hot = tf.one_hot(tf.cast(y_true_sparse, tf.int32), num_classes)

    ce = tf.keras.losses.sparse_categorical_crossentropy(y_true_sparse, y_pred)
    ce = tf.reduce_mean(ce)

    dice = soft_dice_loss_tf(y_true_one_hot, y_pred)
    return ce + dice

25.25 24.25 AI-assisted segmentation workflow

AI tools are useful for segmentation projects, but they should be used in a mathematically controlled way. Good prompts ask the model to verify definitions, tensor shapes, and evaluation logic.

25.25.1 Example AI prompts

Metric verification
“Given this implementation of IoU and Dice for binary masks, check whether false positives and false negatives are handled correctly. Provide a small counterexample if there is a bug.”
Tensor-shape audit
“My segmentation model outputs shape (batch, H, W, K) and my target mask has shape (batch, H, W). Explain which loss functions expect sparse labels and which expect one-hot labels.”
Loss-design critique
“For a medical image segmentation problem where the positive region is less than 2% of pixels, compare cross-entropy, weighted cross-entropy, Dice loss, focal loss, and Tversky loss.”
Boundary analysis
“Explain why two masks can have high Dice score but visibly different boundaries. Suggest boundary-sensitive metrics and losses.”
Statistical interpretation
“Interpret a segmentation probability map as a calibrated posterior probability. What diagnostics would test whether the map is calibrated?”

Avoid this common AI mistake

Do not ask an AI tool only for “segmentation code.” Ask it to state the mask convention, class encoding, probability/logit convention, thresholding rule, and metric definition. Most segmentation bugs are convention bugs.

25.26 24.26 A mathematical checklist for segmentation projects

A segmentation experiment should specify the following objects.

25.26.1 Data and labels

What is the image domain $\Omega_h$?
Are labels semantic, instance, or panoptic?
Are masks binary, class-index, one-hot, or instance-ID maps?
Are unlabeled pixels ignored or treated as background?
Is there annotation uncertainty or multiple annotators?

25.26.2 Model output

Does the model output logits or probabilities?
What is the output shape?
Are predictions at full resolution or upsampled?
Are classes mutually exclusive or multilabel?
Are instance masks generated from boxes, centers, embeddings, or queries?

25.26.3 Loss and optimization

Which likelihood or surrogate loss is being optimized?
Does the loss handle class imbalance?
Does the loss care about boundaries?
Is the loss differentiable with respect to logits?
Are augmentations label-preserving?

25.26.4 Evaluation

Which metric is primary: IoU, Dice, AP, PQ, boundary distance, calibration?
Is the threshold fixed before evaluation?
Are small objects weighted equally or by area?
Are all classes averaged equally?
Is uncertainty evaluated?

25.27 24.27 Chapter summary

Segmentation extends computer vision from image-level labels and bounding boxes to dense geometric prediction. The central mathematical objects are label functions, indicator masks, probability fields, sets, and boundaries. Semantic segmentation is often trained by pixelwise cross-entropy, but overlap metrics such as IoU and Dice better reflect mask quality. Instance segmentation introduces set-valued prediction and mask matching. Panoptic segmentation combines semantic labels with instance identities. Modern segmentation architectures, including FCNs, U-Net, DeepLab-style dilation, Mask R-CNN, and promptable foundation models, can be understood as different ways of combining local detail, global context, geometric constraints, and statistical uncertainty.

25.28 24.28 Exercises

25.28.1 Conceptual exercises

Explain the difference between semantic segmentation, instance segmentation, and panoptic segmentation using mathematical notation.
Why is segmentation not just classification repeated independently at each pixel?
Give an example where pixel accuracy is high but segmentation quality is poor.
Explain why U-Net skip connections improve boundary quality.
Describe how point prompts can be interpreted as constraints on a mask.

25.28.2 Mathematical exercises

Prove that for two masks $A$ and $B$, \[ \operatorname{Dice}(A,B)=\frac{2\operatorname{IoU}(A,B)}{1+\operatorname{IoU}(A,B)}. \]
For binary segmentation, derive the gradient of the binary cross-entropy loss with respect to the logit $z$ when $p=\sigma(z)$.
Derive the soft Dice loss gradient with respect to a predicted probability $P_{ij}$.
Show that Dice loss is not a sum of independent pixel losses.
Suppose a foreground class has prevalence $\pi$. Derive inverse-frequency weights for binary weighted cross-entropy and discuss what happens when $\pi$ is very small.
Compute the Hausdorff distance between two one-dimensional intervals $A=[0,1]$ and $B=[\epsilon,1+\epsilon]$.
For panoptic quality, show how false positives and false negatives reduce recognition quality.

25.28.3 Python exercises

Implement IoU and Dice for multiclass semantic segmentation using class-index masks.
Generate a synthetic dataset of circles and rectangles and train the small U-Net from this chapter.
Compare cross-entropy, Dice loss, and combined loss on a class-imbalanced synthetic segmentation problem.
Implement boundary extraction using binary erosion and compare boundary F1 to Dice.
Write a function that converts instance-ID masks into a list of binary instance masks.
Implement mask matching using IoU and compute AP for a small synthetic instance-segmentation example.
Create reliability diagrams for segmentation probability maps.

25.28.4 AI-assisted exercises

Ask an AI tool to review your segmentation metric code. Require it to produce a $3\times3$ mask example that tests false positives and false negatives.
Ask an AI tool to explain the difference between logits, probabilities, sparse labels, and one-hot labels. Then check its answer against TensorFlow or PyTorch documentation.
Ask an AI tool to design an ablation study comparing loss functions for a rare-object segmentation problem.
Ask an AI tool to critique the use of Dice score as the only metric for a medical segmentation problem.
Ask an AI tool to propose a promptable segmentation workflow for annotating a new dataset. Identify which steps require human verification.

25.29 24.29 References and further reading

Fully convolutional networks introduced the modern dense-prediction framework for semantic segmentation (Long et al. 2015). U-Net is a central encoder-decoder architecture, especially in biomedical imaging (Ronneberger et al. 2015). Mask R-CNN extends object detection with an instance mask head (He et al. 2017). Segment Anything is a major example of promptable segmentation and segmentation foundation models (Kirillov et al. 2023). For classical segmentation, graph cuts, normalized cuts, morphology, and variational methods from earlier chapters remain important foundations.

He, Kaiming, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. “Mask r-CNN.” IEEE International Conference on Computer Vision, 2961–69. https://arxiv.org/abs/1703.06870.

Kirillov, Alexander, Eric Mintun, Nikhila Ravi, et al. 2023. “Segment Anything.” Proceedings of the IEEE/CVF International Conference on Computer Vision. https://arxiv.org/abs/2304.02643.

Long, Jonathan, Evan Shelhamer, and Trevor Darrell. 2015. “Fully Convolutional Networks for Semantic Segmentation.” IEEE Conference on Computer Vision and Pattern Recognition, 3431–40.

Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. 2015. “U-Net: Convolutional Networks for Biomedical Image Segmentation.” Medical Image Computing and Computer-Assisted Intervention.

--- title: "Chapter 24. Semantic and Instance Segmentation" subtitle: "Pixelwise Labels, Masks, Dice Loss, U-Net, Mask R-CNN, Panoptic Segmentation, and Promptable Models" format: html: toc: true toc-depth: 3 execute: warning: false message: false --- ## 24.1 From boxes to masks Object detection localizes objects by bounding boxes. Segmentation asks for a more detailed geometric output: **which pixels belong to which object, region, or class?** Instead of predicting only a rectangle, a segmentation model predicts a label field, a collection of masks, or a probability map. Let an image be $$ X\in\mathbb R^{H\times W\times C}. $$ A semantic segmentation target is a discrete function $$ y:\{1,\dots,H\}\times \{1,\dots,W\}\to \{1,\dots,K\}, $$ where each pixel receives one of $K$ class labels. Equivalently, the target can be represented by one-hot indicator functions $$ Y_{ijk}=\mathbf 1\{y_{ij}=k\}, \qquad Y\in\{0,1\}^{H\times W\times K}. $$ A model predicts class probabilities $$ P_\theta(k\mid X,i,j), \qquad \sum_{k=1}^K P_\theta(k\mid X,i,j)=1. $$ Instance segmentation is different. It predicts a set of object instances $$ \widehat{\mathcal M}(X) = \{(\widehat M_r,\widehat c_r,\widehat s_r):r=1,\dots,\widehat m\}, $$ where $\widehat M_r\subseteq \Omega$ is a binary mask, $\widehat c_r$ is an object class, $\widehat s_r$ is a confidence score, and $\widehat m$ is the predicted number of instances. The order of the masks does not matter, so instance segmentation is a set-valued structured prediction problem. Panoptic segmentation combines both viewpoints. Each pixel is assigned a semantic class and, for countable object categories, an instance identity. In informal language: - **semantic segmentation:** every pixel gets a class; - **instance segmentation:** each object instance gets a mask; - **panoptic segmentation:** every pixel gets both class information and, when appropriate, instance identity. ::: {.callout-note} ## Central idea Segmentation is dense statistical prediction with geometric structure. The output is not a single label but a spatially organized field or a set of masks. The mathematics combines pixelwise likelihoods, set overlap, variational regularization, structured prediction, and learned representations. ::: ### Learning goals After this chapter, you should be able to: 1. distinguish semantic, instance, and panoptic segmentation mathematically; 2. represent masks as indicator functions, probability fields, signed distance functions, and sets; 3. derive pixelwise cross-entropy from maximum likelihood; 4. compute IoU, Dice, boundary metrics, and panoptic quality; 5. explain why segmentation losses must handle class imbalance and spatial correlation; 6. describe fully convolutional networks, encoder-decoder architectures, skip connections, U-Net, DeepLab-style dilation, and Mask R-CNN; 7. formulate instance segmentation as detection plus mask prediction or as set prediction; 8. explain promptable segmentation and its relation to foundation models; 9. write Python experiments for segmentation losses and evaluation metrics; 10. use AI tools to check mask conventions, loss definitions, and evaluation code. ## 24.2 Masks as mathematical objects A binary mask is a function $$ M:\Omega_h\to\{0,1\}, $$ where $$ \Omega_h=\{1,\dots,H\}\times\{1,\dots,W\} $$ is the discrete image grid. We may also view a mask as a subset $$ A=\{(i,j)\in\Omega_h:M(i,j)=1\}. $$ The area of the mask is the counting measure $$ |A|=\sum_{i=1}^H\sum_{j=1}^W M(i,j). $$ For soft segmentation, the mask is replaced by a probability field $$ P:\Omega_h\to[0,1]. $$ A hard mask can be obtained by thresholding: $$ \widehat M_\tau(i,j)=\mathbf 1\{P(i,j)\ge \tau\}. $$ The threshold $\tau$ is not merely an implementation detail. It defines a decision rule. Changing $\tau$ changes false positives, false negatives, IoU, Dice score, and boundary behavior. ### Indicator functions and vector spaces Although a binary mask is a set, it can also be embedded in a vector space: $$ M\in\{0,1\}^{HW}\subset \mathbb R^{HW}. $$ This representation allows us to use inner products: $$ \langle M,N\rangle = \sum_{i,j}M(i,j)N(i,j) =|A\cap B|. $$ Also, $$ \|M\|_1=|A|, \qquad \|M-N\|_1=|A\triangle B|, $$ where $A\triangle B$ is the symmetric difference. Thus, many segmentation metrics can be written using linear algebra. ### Signed distance representation For geometric segmentation, it is useful to represent a region by a signed distance function $$ \phi(x)= \begin{cases} -d(x,\partial A), & x\in A,\\ \phantom{-}d(x,\partial A), & x\notin A. \end{cases} $$ Then the mask is recovered by $$ M(x)=\mathbf 1\{\phi(x)\le 0\}. $$ This representation connects segmentation to level sets, active contours, implicit neural fields, and differentiable shape optimization. ## 24.3 Semantic segmentation as pixelwise classification For semantic segmentation, each pixel has a categorical label. A neural network often outputs logits $$ z_\theta(X)_{ijk}\in\mathbb R, $$ and probabilities are obtained by the softmax $$ p_{ijk} = \frac{\exp(z_{ijk})}{\sum_{\ell=1}^K\exp(z_{ij\ell})}. $$ If the true one-hot label is $Y_{ijk}$, the pixelwise negative log-likelihood is $$ \ell_{ij}(\theta) = -\sum_{k=1}^K Y_{ijk}\log p_{ijk}. $$ The full image loss is usually averaged over pixels: $$ \mathcal L_{\mathrm{CE}}(\theta) = -\frac{1}{HW}\sum_{i=1}^H\sum_{j=1}^W\sum_{k=1}^K Y_{ijk}\log p_{ijk}. $$ This is the same cross-entropy used for classification, but now repeated at every pixel. The model is trained as if each pixel contributes a supervised example. However, pixels are spatially correlated, so this is not an independent data model in a strict statistical sense. The pixelwise likelihood is best understood as a practical surrogate objective. ::: {.callout-tip} ## Statistical caution Pixelwise cross-entropy treats pixels as loss terms, not as independent observations. A $512\times512$ mask does not provide $262{,}144$ independent labels. The effective sample size is much smaller because neighboring pixels are strongly dependent. ::: ### Class-weighted cross-entropy In segmentation, background pixels often dominate. If class $k$ is rare, ordinary cross-entropy may produce a model that ignores it. A weighted loss is $$ \mathcal L_{\mathrm{WCE}} = -\frac{1}{HW}\sum_{i,j,k}w_kY_{ijk}\log p_{ijk}, $$ where $w_k>0$ is larger for rare or clinically important classes. A common heuristic is inverse class frequency: $$ w_k\propto \frac{1}{\widehat\pi_k}, \qquad \widehat\pi_k=\frac{1}{NHW}\sum_{n=1}^N\sum_{i,j}Y_{nijk}. $$ But extremely large weights can make optimization unstable. In practice, one often clips or smooths these weights. ### Focal loss for dense imbalance For binary segmentation with target $y\in\{0,1\}$ and predicted probability $p$, define $$ p_t= \begin{cases} p, & y=1,\\ 1-p, & y=0. \end{cases} $$ The focal loss is $$ \ell_{\mathrm{focal}} =-\alpha(1-p_t)^\gamma\log(p_t), $$ where $\gamma\ge 0$. When $p_t$ is near $1$, the sample is already easy and the factor $(1-p_t)^\gamma$ downweights it. Focal loss was introduced for dense detection, but the same idea is useful in dense segmentation. ## 24.4 Overlap metrics: IoU and Dice Segmentation is usually evaluated by set overlap. For a predicted mask $\widehat A$ and a true mask $A$, the intersection-over-union is $$ \operatorname{IoU}(\widehat A,A) = \frac{|\widehat A\cap A|}{|\widehat A\cup A|}. $$ Using indicator functions $\widehat M$ and $M$, $$ |\widehat A\cap A|=\sum_{i,j}\widehat M_{ij}M_{ij}, $$ and $$ |\widehat A\cup A|= \sum_{i,j}\widehat M_{ij}+ \sum_{i,j}M_{ij} - \sum_{i,j}\widehat M_{ij}M_{ij}. $$ The Dice coefficient is $$ \operatorname{Dice}(\widehat A,A) = \frac{2|\widehat A\cap A|}{|\widehat A|+|A|}. $$ Dice and IoU are monotone transforms of each other for fixed pair of masks: $$ \operatorname{Dice} = \frac{2\operatorname{IoU}}{1+\operatorname{IoU}}, \qquad \operatorname{IoU} = \frac{\operatorname{Dice}}{2-\operatorname{Dice}}. $$ This means Dice and IoU rank the same pairwise overlaps in the same order, although their numerical values differ. ### Soft Dice loss For a soft predicted mask $P_{ij}\in[0,1]$ and binary target $M_{ij}\in\{0,1\}$, the soft Dice coefficient is $$ D(P,M)= \frac{2\sum_{i,j}P_{ij}M_{ij}+\varepsilon} {\sum_{i,j}P_{ij}+\sum_{i,j}M_{ij}+\varepsilon}, $$ where $\varepsilon>0$ prevents division by zero. A Dice loss is $$ \mathcal L_{\mathrm{Dice}}=1-D(P,M). $$ Dice loss directly rewards overlap and is often more stable than pure cross-entropy for small objects. It is also nonseparable across pixels because the denominator couples all pixels in the mask. ### Tversky index The Tversky index generalizes Dice by weighting false positives and false negatives differently: $$ T_{\alpha,\beta}(\widehat A,A) = \frac{|\widehat A\cap A|} {|\widehat A\cap A|+ \alpha|\widehat A\setminus A|+ \beta|A\setminus \widehat A|}. $$ If missing the object is more costly than including extra pixels, choose $\beta>\alpha$. This is useful in medical imaging, where false negatives may be more serious than false positives. ## 24.5 Interactive demo: IoU, Dice, and thresholding The same probability map can produce different masks under different thresholds. The following demo shows how the threshold affects IoU, Dice, false positives, and false negatives. ::: {.interactive-card} <div id="ch24-threshold-metrics" class="plotly-demo"></div> <p> <label for="ch24-threshold-slider">threshold:</label> <input id="ch24-threshold-slider" type="range" min="0.05" max="0.95" step="0.05" value="0.50" style="width: 55%;"> <span id="ch24-threshold-text">0.50</span> </p> ::: <script> (function(){ function sigmoid(x){ return 1/(1+Math.exp(-x)); } const n = 60; const gt = []; const prob = []; for(let i=0;i<n;i++){ gt[i] = []; prob[i] = []; for(let j=0;j<n;j++){ const x = (j-n/2)/(n/2); const y = (i-n/2)/(n/2); const r1 = Math.sqrt((x+0.20)**2 + (y+0.05)**2); const r2 = Math.sqrt((x-0.25)**2 + (y-0.15)**2); const inside = (r1 < 0.42 || (Math.abs(x-0.10)<0.25 && Math.abs(y+0.25)<0.18)); gt[i][j] = inside ? 1 : 0; const score = 6*(0.43-r1) + 2.5*Math.exp(-10*r2*r2) + 0.7*Math.sin(8*x)*Math.cos(6*y); prob[i][j] = sigmoid(score); } } function compute(th){ const pred = []; let tp=0, fp=0, fn=0; for(let i=0;i<n;i++){ pred[i] = []; for(let j=0;j<n;j++){ const p = prob[i][j] >= th ? 1 : 0; pred[i][j] = p; if(p===1 && gt[i][j]===1) tp++; if(p===1 && gt[i][j]===0) fp++; if(p===0 && gt[i][j]===1) fn++; } } const iou = tp/(tp+fp+fn+1e-9); const dice = 2*tp/(2*tp+fp+fn+1e-9); return {pred, tp, fp, fn, iou, dice}; } function draw(){ const th = parseFloat(document.getElementById('ch24-threshold-slider').value); const out = compute(th); document.getElementById('ch24-threshold-text').innerText = th.toFixed(2) + ' | IoU=' + out.iou.toFixed(3) + ' | Dice=' + out.dice.toFixed(3) + ' | FP=' + out.fp + ' | FN=' + out.fn; const data = [ {z: gt, type:'heatmap', colorscale:'Greys', showscale:false, xaxis:'x1', yaxis:'y1', name:'truth'}, {z: prob, type:'heatmap', colorscale:'Viridis', showscale:false, xaxis:'x2', yaxis:'y2', name:'probability'}, {z: out.pred, type:'heatmap', colorscale:'Greys', showscale:false, xaxis:'x3', yaxis:'y3', name:'prediction'} ]; const layout = { title:'Thresholded mask: truth, probability, prediction', grid:{rows:1, columns:3, pattern:'independent'}, margin:{t:50,l:30,r:20,b:30}, xaxis:{title:'true mask'}, yaxis:{scaleanchor:'x'}, xaxis2:{title:'probability'}, yaxis2:{scaleanchor:'x2'}, xaxis3:{title:'predicted mask'}, yaxis3:{scaleanchor:'x3'} }; Plotly.react('ch24-threshold-metrics', data, layout, {responsive:true}); } document.getElementById('ch24-threshold-slider').addEventListener('input', draw); if(window.Plotly){ draw(); } })(); </script> ## 24.6 Boundary metrics and why overlap is not enough Two masks can have similar IoU but different boundary quality. This matters when the object boundary is scientifically important, such as in tumor segmentation, cell segmentation, road extraction, and anatomical measurement. Let $\partial A$ and $\partial \widehat A$ denote the boundaries of two regions. The directed Hausdorff distance is $$ h(\partial A,\partial \widehat A) = \max_{x\in\partial A}\min_{y\in\partial\widehat A}\|x-y\|_2. $$ The symmetric Hausdorff distance is $$ H(\partial A,\partial \widehat A) = \max\{h(\partial A,\partial \widehat A),h(\partial \widehat A,\partial A)\}. $$ The average symmetric surface distance is less sensitive to one extreme outlier: $$ \operatorname{ASSD} = \frac{1}{|\partial A|+|\partial\widehat A|} \left( \sum_{x\in\partial A}d(x,\partial\widehat A) + \sum_{y\in\partial\widehat A}d(y,\partial A) \right). $$ Boundary-sensitive losses often use distance transforms. If $\phi_A$ is the signed distance function of the true mask, a boundary loss can penalize predicted mass according to distance from the true boundary: $$ \mathcal L_{\mathrm{boundary}} = \sum_{i,j} P_{ij}\phi_A(i,j). $$ More generally, distance-transform losses connect mask prediction to geometric measure theory: one is no longer just classifying pixels but estimating a shape. ## 24.7 Fully convolutional networks A fully convolutional network, or FCN, replaces dense classification layers with convolutional layers so that the model outputs a spatial map rather than one image-level label [@long2015fully]. Let an encoder produce feature maps $$ F_\ell\in\mathbb R^{H_\ell\times W_\ell\times C_\ell}. $$ A segmentation head maps these features to logits $$ Z\in\mathbb R^{H\times W\times K}. $$ Because repeated pooling or strided convolution reduces spatial resolution, the network must upsample. A simple approach is bilinear interpolation. A learned approach is transposed convolution. Another approach is a decoder that gradually combines low-resolution semantic features with high-resolution spatial features. ### Why convolution is natural for segmentation Segmentation requires the model to preserve spatial structure. A convolutional layer is translation equivariant: $$ T_a(f*K)=(T_af)*K, $$ where $T_a$ is translation by $a$. If an object moves in the image, the segmentation mask should move with it. Thus, equivariance is a structural prior for dense prediction. However, segmentation also requires context. A single pixel may be ambiguous without its neighborhood. Therefore, segmentation models must balance: 1. **local detail:** sharp boundaries and small objects; 2. **global context:** object identity, scene layout, and long-range dependencies. This local-global tension explains many architecture choices: skip connections, dilation, multi-scale features, attention, and prompt-conditioned masks. ## 24.8 Encoder-decoder architectures and U-Net An encoder-decoder model has two parts. The encoder compresses the image into increasingly abstract features: $$ X\mapsto F_1\mapsto F_2\mapsto\cdots\mapsto F_L. $$ The decoder reconstructs a high-resolution prediction: $$ F_L\mapsto G_{L-1}\mapsto\cdots\mapsto G_1\mapsto Z. $$ U-Net adds skip connections between encoder and decoder layers of matching resolution [@ronneberger2015unet]. In a simplified notation, $$ G_\ell = \Psi_\ell\big(\operatorname{Upsample}(G_{\ell+1}),F_\ell\big), $$ where $F_\ell$ is an encoder feature map and $G_\ell$ is a decoder feature map. The operation $\Psi_\ell$ may concatenate features followed by convolutions. The skip connection has a mathematical interpretation: high-resolution encoder features provide local coordinate information, while low-resolution decoder features provide semantic context. Without skip connections, the decoder may recover class identity but lose boundary precision. ### U-Net as a multiscale approximation scheme For applied mathematics students, it is helpful to compare U-Net to multigrid or wavelet ideas. The encoder produces coarser representations, while the decoder refines resolution. Skip connections pass fine-scale information around the bottleneck. The architecture can be interpreted as learning a nonlinear multiscale operator $$ \mathcal S_\theta:\mathbb R^{H\times W\times C}\to [0,1]^{H\times W\times K}. $$ The model is not simply classifying pixels independently. It learns a nonlinear, multiscale map from an image function to a label function. ## 24.9 Interactive demo: skip connections and boundary recovery This schematic demo shows the effect of combining coarse semantic information with high-resolution detail. The coarse map captures the object approximately; the detail map sharpens the boundary. ::: {.interactive-card} <div id="ch24-skip-demo" class="plotly-demo"></div> <p> <label for="ch24-skip-alpha">detail weight:</label> <input id="ch24-skip-alpha" type="range" min="0" max="1" step="0.05" value="0.45" style="width: 55%;"> <span id="ch24-skip-alpha-text">0.45</span> </p> ::: <script> (function(){ const n = 80; const coarse=[], detail=[], combined=[]; for(let i=0;i<n;i++){ coarse[i]=[]; detail[i]=[]; combined[i]=[]; for(let j=0;j<n;j++){ const x=(j-n/2)/(n/2), y=(i-n/2)/(n/2); const r=Math.sqrt((x+0.05)**2 + (y-0.02)**2); coarse[i][j]=1/(1+Math.exp(12*(r-0.50))); const edge=Math.exp(-150*(r-0.48)**2); detail[i][j]=edge + 0.20*Math.sin(20*x)*Math.sin(16*y); } } function draw(){ const a=parseFloat(document.getElementById('ch24-skip-alpha').value); document.getElementById('ch24-skip-alpha-text').innerText=a.toFixed(2); for(let i=0;i<n;i++) for(let j=0;j<n;j++) combined[i][j]=Math.max(0, Math.min(1, coarse[i][j] + a*(detail[i][j]-0.35))); const data=[ {z:coarse,type:'heatmap',colorscale:'Viridis',showscale:false,xaxis:'x1',yaxis:'y1'}, {z:detail,type:'heatmap',colorscale:'Viridis',showscale:false,xaxis:'x2',yaxis:'y2'}, {z:combined,type:'heatmap',colorscale:'Viridis',showscale:false,xaxis:'x3',yaxis:'y3'} ]; const layout={ title:'Coarse context plus high-resolution detail', grid:{rows:1,columns:3,pattern:'independent'}, margin:{t:50,l:25,r:20,b:30}, xaxis:{title:'coarse semantic map'}, yaxis:{scaleanchor:'x'}, xaxis2:{title:'detail / boundary map'}, yaxis2:{scaleanchor:'x2'}, xaxis3:{title:'combined prediction'}, yaxis3:{scaleanchor:'x3'} }; Plotly.react('ch24-skip-demo', data, layout, {responsive:true}); } document.getElementById('ch24-skip-alpha').addEventListener('input', draw); if(window.Plotly){ draw(); } })(); </script> ## 24.10 Dilated convolution and DeepLab-style context Pooling increases receptive field but reduces resolution. Dilated convolution increases receptive field without the same reduction in resolution. In one dimension, a dilated convolution with dilation rate $r$ is $$ (f *_r k)[i]=\sum_m k[m]f[i-rm]. $$ In two dimensions, $$ (f *_r K)[i,j] = \sum_{u,v}K[u,v]f[i-ru,j-rv]. $$ A $3\times3$ kernel with dilation $r$ samples a grid spanning effective size $$ 1+2r. $$ DeepLab-style models use dilated convolutions and multi-scale context aggregation to improve semantic segmentation, especially for large objects and scene layouts. ### Atrous spatial pyramid pooling Atrous spatial pyramid pooling applies several dilated convolutions with different dilation rates: $$ G_r=K_r *_r F, \qquad r\in\mathcal R. $$ The outputs are concatenated: $$ G=\operatorname{concat}\{G_r:r\in\mathcal R\}. $$ This gives a learned multiscale feature representation. Mathematically, it resembles using multiple basis functions at different spatial scales. ## 24.11 Interactive demo: dilation and receptive field A larger dilation increases the receptive field without increasing the number of kernel parameters. ::: {.interactive-card} <div id="ch24-dilation-demo" class="plotly-demo"></div> <p> <label for="ch24-dilation-rate">dilation rate:</label> <input id="ch24-dilation-rate" type="range" min="1" max="6" step="1" value="2" style="width: 55%;"> <span id="ch24-dilation-rate-text">2</span> </p> ::: <script> (function(){ function draw(){ const r=parseInt(document.getElementById('ch24-dilation-rate').value); document.getElementById('ch24-dilation-rate-text').innerText=String(r); const n=17, z=[]; for(let i=0;i<n;i++){ z[i]=[]; for(let j=0;j<n;j++) z[i][j]=0; } const c=Math.floor(n/2); for(let u=-1;u<=1;u++) for(let v=-1;v<=1;v++) z[c+r*u][c+r*v]=1; z[c][c]=2; const data=[{z:z,type:'heatmap',colorscale:[[0,'white'],[0.4,'lightgray'],[0.7,'steelblue'],[1,'darkorange']],showscale:false}]; const ticks=[...Array(n).keys()]; const layout={ title:'Sampled input positions for a 3 by 3 dilated kernel', margin:{t:50,l:40,r:20,b:40}, xaxis:{tickmode:'array',tickvals:ticks,showgrid:true}, yaxis:{tickmode:'array',tickvals:ticks,scaleanchor:'x',showgrid:true,autorange:'reversed'}, annotations:[{x:c,y:c,text:'center',showarrow:false,font:{size:12}}] }; Plotly.react('ch24-dilation-demo',data,layout,{responsive:true}); } document.getElementById('ch24-dilation-rate').addEventListener('input', draw); if(window.Plotly){ draw(); } })(); </script> ## 24.12 Conditional random fields and structured smoothing Pixelwise neural predictions are often noisy at boundaries. A classical way to impose structure is to use a conditional random field. Let $y_i$ denote the label at pixel $i$. A typical energy is $$ E(y)=\sum_i \psi_i(y_i)+\sum_{i<j}\psi_{ij}(y_i,y_j), $$ where the unary term comes from the network logits: $$ \psi_i(k)=-\log p_i(k), $$ and the pairwise term encourages nearby or visually similar pixels to share labels: $$ \psi_{ij}(y_i,y_j) = \mathbf 1\{y_i\ne y_j\}w_{ij}. $$ A common weight has the form $$ w_{ij} =\lambda\exp\left(-\frac{\|p_i-p_j\|^2}{2\sigma_p^2} -\frac{\|I_i-I_j\|^2}{2\sigma_I^2}\right), $$ where $p_i$ is pixel position and $I_i$ is color. This couples segmentation to both geometry and appearance. Modern architectures often learn these smoothing effects implicitly, but the CRF viewpoint remains valuable because it makes the structured-prediction assumptions explicit. ## 24.13 Instance segmentation In instance segmentation, the model must separate different objects of the same class. For example, if an image contains three people, semantic segmentation may assign all person pixels the label “person,” while instance segmentation must produce three separate masks. A common approach is detection plus mask prediction. First predict boxes and object classes. Then predict a binary mask inside each region of interest. Mask R-CNN follows this strategy by adding a mask head to Faster R-CNN [@he2017mask]. For a predicted region proposal $r$, the mask head produces $$ P_r\in[0,1]^{m\times m\times K}, $$ or often one binary mask per class. The mask is trained using binary cross-entropy on the ground-truth mask cropped to the proposal. ### RoI alignment A key operation is to extract a fixed-size feature tensor from a variable-size proposal. RoI pooling used quantization. RoI Align avoids harsh quantization by bilinear interpolation. If $F$ is a feature map and $(x,y)$ is a noninteger coordinate, bilinear interpolation gives $$ F(x,y)= \sum_{a\in\{\lfloor x\rfloor,\lceil x\rceil\}} \sum_{b\in\{\lfloor y\rfloor,\lceil y\rceil\}} w_{ab}(x,y)F(a,b). $$ This matters because instance masks are sensitive to spatial alignment. ### Instance mask loss For a proposal assigned to class $c$, the binary mask loss is $$ \mathcal L_{\mathrm{mask}} = -\frac{1}{m^2}\sum_{u=1}^m\sum_{v=1}^m \left[ M_{uv}\log P_{uv}^{(c)}+(1-M_{uv})\log(1-P_{uv}^{(c)}) \right]. $$ The total training loss combines classification, box regression, and mask loss: $$ \mathcal L =\mathcal L_{\mathrm{cls}}+ \mathcal L_{\mathrm{box}}+ \mathcal L_{\mathrm{mask}}. $$ ## 24.14 Instance matching and average precision for masks Instance segmentation evaluation uses mask IoU rather than box IoU. A predicted mask is a true positive at threshold $\tau$ if its class is correct and $$ \operatorname{IoU}(\widehat M,M)\ge \tau. $$ Predictions are sorted by confidence. Each predicted mask can match at most one ground-truth instance, and each ground-truth instance can be matched at most once. Precision and recall are computed over the sorted list, producing an average precision curve. The mean average precision averages AP over classes and overlap thresholds. The COCO-style metric uses multiple thresholds, which rewards both detection and precise mask boundaries. ### Why instance segmentation is not only pixel classification Suppose two adjacent cells touch each other. Semantic segmentation can correctly label every foreground pixel as “cell” but still fail to split the two instances. Instance segmentation requires additional structure: object centers, contours, embeddings, watershed markers, or set prediction. Some models solve this by predicting: 1. object centers and offsets; 2. boundary maps; 3. instance embeddings; 4. masks associated with detected boxes; 5. transformer queries that represent object instances. ## 24.15 Interactive demo: instance masks and matching The following demo illustrates mask matching by IoU. A prediction with high overlap is matched to a ground truth instance; another prediction may be counted as duplicate or false positive. ::: {.interactive-card} <div id="ch24-instance-match" class="plotly-demo"></div> <p> <label for="ch24-instance-shift">prediction shift:</label> <input id="ch24-instance-shift" type="range" min="-20" max="20" step="1" value="6" style="width: 55%;"> <span id="ch24-instance-shift-text">6</span> </p> ::: <script> (function(){ const n=80; function mask(cx,cy,rx,ry){ const z=[]; for(let i=0;i<n;i++){ z[i]=[]; for(let j=0;j<n;j++){ const val=((j-cx)/rx)**2+((i-cy)/ry)**2 <= 1 ? 1 : 0; z[i][j]=val; } } return z; } function iou(a,b){ let inter=0, uni=0; for(let i=0;i<n;i++) for(let j=0;j<n;j++){ if(a[i][j]===1 && b[i][j]===1) inter++; if(a[i][j]===1 || b[i][j]===1) uni++; } return inter/(uni+1e-9); } const gt1=mask(28,42,14,20), gt2=mask(55,35,12,16); function draw(){ const shift=parseInt(document.getElementById('ch24-instance-shift').value); const pr1=mask(28+shift,42,14,20), pr2=mask(55,35,12,16); const iou11=iou(pr1,gt1), iou12=iou(pr1,gt2), iou22=iou(pr2,gt2); document.getElementById('ch24-instance-shift-text').innerText = shift + ' | IoU(pred1,gt1)=' + iou11.toFixed(3) + ' | IoU(pred1,gt2)=' + iou12.toFixed(3) + ' | IoU(pred2,gt2)=' + iou22.toFixed(3); const truth=[], pred=[]; for(let i=0;i<n;i++){ truth[i]=[]; pred[i]=[]; for(let j=0;j<n;j++){ truth[i][j]=gt1[i][j]+2*gt2[i][j]; pred[i][j]=pr1[i][j]+2*pr2[i][j]; } } const data=[ {z:truth,type:'heatmap',colorscale:'Viridis',showscale:false,xaxis:'x1',yaxis:'y1'}, {z:pred,type:'heatmap',colorscale:'Viridis',showscale:false,xaxis:'x2',yaxis:'y2'} ]; const layout={ title:'Instance segmentation uses mask matching, not only pixel accuracy', grid:{rows:1,columns:2,pattern:'independent'}, margin:{t:50,l:30,r:20,b:30}, xaxis:{title:'ground truth instances'}, yaxis:{scaleanchor:'x'}, xaxis2:{title:'predicted instances'}, yaxis2:{scaleanchor:'x2'} }; Plotly.react('ch24-instance-match',data,layout,{responsive:true}); } document.getElementById('ch24-instance-shift').addEventListener('input',draw); if(window.Plotly){ draw(); } })(); </script> ## 24.16 Panoptic segmentation Panoptic segmentation assigns every pixel to exactly one semantic class and, for object-like categories, an instance identity. Classes are often divided into: - **things:** countable objects such as person, car, dog; - **stuff:** amorphous regions such as sky, road, grass. A panoptic output can be represented as $$ y(i,j)=(c(i,j),q(i,j)), $$ where $c(i,j)$ is the semantic class and $q(i,j)$ is the instance ID when the class is a thing category. Stuff regions may use a special null instance ID. Panoptic quality is often decomposed into segmentation quality and recognition quality. Let $\operatorname{TP}$ be matched segments, $\operatorname{FP}$ unmatched predictions, and $\operatorname{FN}$ unmatched ground-truth segments. Then $$ \operatorname{PQ} = \frac{\sum_{(p,g)\in\operatorname{TP}}\operatorname{IoU}(p,g)} {|\operatorname{TP}|+\frac12|\operatorname{FP}|+\frac12|\operatorname{FN}|}. $$ This can be factored as $$ \operatorname{PQ}=\operatorname{SQ}\cdot\operatorname{RQ}, $$ where $$ \operatorname{SQ} = \frac{1}{|\operatorname{TP}|}\sum_{(p,g)\in\operatorname{TP}}\operatorname{IoU}(p,g), $$ and $$ \operatorname{RQ} = \frac{|\operatorname{TP}|} {|\operatorname{TP}|+\frac12|\operatorname{FP}|+\frac12|\operatorname{FN}|}. $$ Thus PQ separates mask quality from detection/recognition quality. ## 24.17 Promptable segmentation and foundation models Modern segmentation increasingly uses foundation models. A promptable segmentation model takes an image and a prompt, such as points, boxes, rough masks, or text-like guidance, and returns masks. Segment Anything is a central example of this paradigm [@kirillov2023segment]. Mathematically, a promptable segmentation model is a conditional map $$ \mathcal S_\theta:(X,p)\mapsto \{(M_r,s_r):r=1,\dots,R\}, $$ where $p$ is a prompt and $s_r$ are mask confidence scores. The prompt changes the conditional distribution: $$ P_\theta(M\mid X,p). $$ This is a significant shift from closed-set segmentation. Instead of learning one fixed mapping from image to class labels, the model learns an interface between human intention and visual masks. ### Prompt types as constraints A positive point prompt $a\in\Omega$ imposes the constraint $$ M(a)=1. $$ A negative point prompt $b\in\Omega$ imposes $$ M(b)=0. $$ A box prompt imposes support constraints: $$ M(x)=0\quad\text{for many }x\notin B. $$ A mask prompt gives an initial estimate. From an optimization viewpoint, prompts act like constraints, priors, or conditioning variables. ### Why promptable segmentation is useful Promptable segmentation is useful when the desired object category is not known in advance or when the user wants one particular object among many possibilities. It supports annotation, editing, medical review, interactive analysis, and weakly supervised pipelines. For a statistics student, the key question is uncertainty: different plausible masks may satisfy the same prompt. A good system should produce calibrated confidence or multiple candidate masks when the prompt is ambiguous. ## 24.18 Interactive demo: point prompts as constraints This simplified demo shows how positive and negative point prompts can select different masks from the same image-like score field. ::: {.interactive-card} <div id="ch24-prompt-demo" class="plotly-demo"></div> <p> <label for="ch24-prompt-x">positive point x-coordinate:</label> <input id="ch24-prompt-x" type="range" min="15" max="65" step="1" value="28" style="width: 50%;"> <span id="ch24-prompt-x-text">28</span> </p> ::: <script> (function(){ const n=80; function gaussian(x,y,cx,cy,s){ return Math.exp(-((x-cx)**2+(y-cy)**2)/(2*s*s)); } function draw(){ const px=parseInt(document.getElementById('ch24-prompt-x').value); document.getElementById('ch24-prompt-x-text').innerText=String(px); const py=40; const z=[], m=[]; const targetLeft = px < 40; for(let i=0;i<n;i++){ z[i]=[]; m[i]=[]; for(let j=0;j<n;j++){ const score = gaussian(j,i,28,42,13)+0.95*gaussian(j,i,55,36,12)+0.25*gaussian(j,i,40,62,18); const promptBias = targetLeft ? gaussian(j,i,28,42,25) : gaussian(j,i,55,36,25); const selected = score + 0.70*promptBias; z[i][j]=score; m[i][j]=selected>0.55 ? 1 : 0; } } const data=[ {z:z,type:'heatmap',colorscale:'Viridis',showscale:false,xaxis:'x1',yaxis:'y1'}, {z:m,type:'heatmap',colorscale:'Greys',showscale:false,xaxis:'x2',yaxis:'y2'}, {x:[px],y:[py],mode:'markers',marker:{size:14,symbol:'x'},name:'positive prompt',xaxis:'x2',yaxis:'y2'} ]; const layout={ title:'A point prompt conditions the selected mask', grid:{rows:1,columns:2,pattern:'independent'}, margin:{t:50,l:30,r:20,b:30}, xaxis:{title:'image-like score field'}, yaxis:{scaleanchor:'x'}, xaxis2:{title:'prompt-conditioned mask'}, yaxis2:{scaleanchor:'x2'} }; Plotly.react('ch24-prompt-demo',data,layout,{responsive:true}); } document.getElementById('ch24-prompt-x').addEventListener('input',draw); if(window.Plotly){ draw(); } })(); </script> ## 24.19 Segmentation uncertainty Segmentation models often output probabilities, but probability maps are not automatically calibrated. A pixel probability $p=0.9$ should mean that, among similar pixels assigned probability $0.9$, about $90\%$ are truly positive. This is a calibration statement, not just a confidence score. For binary segmentation, the Brier score is $$ \operatorname{Brier} =\frac{1}{HW}\sum_{i,j}(P_{ij}-M_{ij})^2. $$ Expected calibration error bins predicted probabilities and compares average confidence with empirical accuracy: $$ \operatorname{ECE} =\sum_{b=1}^B \frac{|S_b|}{HW} \left|\operatorname{acc}(S_b)-\operatorname{conf}(S_b)\right|. $$ Uncertainty can arise from: 1. image noise; 2. ambiguous boundaries; 3. annotation disagreement; 4. out-of-distribution images; 5. model uncertainty; 6. prompt ambiguity. A mathematically careful segmentation study should report not only masks but also uncertainty, calibration, and failure modes. ## 24.20 Segmentation as empirical risk minimization Let $\{(X_n,Y_n)\}_{n=1}^N$ be a training set. A general segmentation objective is $$ \widehat R(\theta) = \frac1N\sum_{n=1}^N \mathcal L(f_\theta(X_n),Y_n) +\lambda\Omega(\theta), $$ where $f_\theta$ is the segmentation model, $\mathcal L$ may combine cross-entropy, Dice, boundary, and auxiliary losses, and $\Omega$ is a regularizer. A common combined loss for binary segmentation is $$ \mathcal L =\lambda_{\mathrm{CE}}\mathcal L_{\mathrm{CE}} +\lambda_{\mathrm{Dice}}\mathcal L_{\mathrm{Dice}} +\lambda_{\mathrm{bdry}}\mathcal L_{\mathrm{boundary}}. $$ This is not merely engineering. It reflects competing statistical and geometric goals: - cross-entropy improves probabilistic calibration; - Dice improves overlap for imbalanced regions; - boundary loss improves contour accuracy; - regularization improves generalization and smoothness. ## 24.21 Python example: metrics for binary masks The following code computes IoU, Dice, precision, recall, and a soft Dice loss. It is designed to be readable before being efficient. ```{python} #| label: ch24-python-mask-metrics #| eval: false import numpy as np def binary_metrics(y_true, y_prob, threshold=0.5, eps=1e-8): """Compute common binary segmentation metrics.""" y_true = np.asarray(y_true).astype(bool) y_pred = np.asarray(y_prob) >= threshold tp = np.logical_and(y_pred, y_true).sum() fp = np.logical_and(y_pred, ~y_true).sum() fn = np.logical_and(~y_pred, y_true).sum() tn = np.logical_and(~y_pred, ~y_true).sum() iou = tp / (tp + fp + fn + eps) dice = 2 * tp / (2 * tp + fp + fn + eps) precision = tp / (tp + fp + eps) recall = tp / (tp + fn + eps) return { "tp": int(tp), "fp": int(fp), "fn": int(fn), "tn": int(tn), "iou": float(iou), "dice": float(dice), "precision": float(precision), "recall": float(recall) } def soft_dice_loss(y_true, y_prob, eps=1e-6): y_true = np.asarray(y_true, dtype=float) y_prob = np.asarray(y_prob, dtype=float) intersection = np.sum(y_true * y_prob) denominator = np.sum(y_true) + np.sum(y_prob) dice = (2 * intersection + eps) / (denominator + eps) return 1.0 - dice # Example synthetic mask H, W = 96, 96 y, x = np.ogrid[:H, :W] true_mask = (x - 48) ** 2 + (y - 48) ** 2 <= 24 ** 2 prob_map = np.exp(-((x - 51) ** 2 + (y - 47) ** 2) / (2 * 20 ** 2)) print(binary_metrics(true_mask, prob_map, threshold=0.5)) print("soft Dice loss:", soft_dice_loss(true_mask, prob_map)) ``` ## 24.22 Python example: semantic segmentation data tensors A common source of bugs is mixing class-index masks and one-hot masks. The following example shows both conventions. ```{python} #| label: ch24-python-one-hot #| eval: false import numpy as np H, W, K = 128, 128, 4 class_mask = np.zeros((H, W), dtype=np.int64) # Create simple synthetic regions. class_mask[20:70, 20:80] = 1 class_mask[60:110, 50:120] = 2 class_mask[30:100, 95:115] = 3 # One-hot encoding: shape (H, W, K) one_hot = np.eye(K, dtype=np.float32)[class_mask] print("class-index mask shape:", class_mask.shape) print("one-hot mask shape:", one_hot.shape) print("pixel class probabilities should also have shape:", one_hot.shape) # Recover class labels from predicted probabilities. logits = np.random.normal(size=(H, W, K)) prob = np.exp(logits) / np.exp(logits).sum(axis=-1, keepdims=True) pred_class = prob.argmax(axis=-1) print("predicted class mask shape:", pred_class.shape) ``` ## 24.23 Python example: a small U-Net in TensorFlow This is a minimal educational U-Net-style architecture. For serious experiments, add proper data loading, augmentation, validation, checkpointing, metrics, and careful preprocessing. ```{python} #| label: ch24-python-tiny-unet #| eval: false import tensorflow as tf from tensorflow.keras import layers, Model def conv_block(x, filters): x = layers.Conv2D(filters, 3, padding="same", activation="relu")(x) x = layers.Conv2D(filters, 3, padding="same", activation="relu")(x) return x def tiny_unet(input_shape=(128, 128, 3), num_classes=2): inputs = layers.Input(shape=input_shape) c1 = conv_block(inputs, 32) p1 = layers.MaxPool2D()(c1) c2 = conv_block(p1, 64) p2 = layers.MaxPool2D()(c2) b = conv_block(p2, 128) u2 = layers.UpSampling2D(interpolation="bilinear")(b) u2 = layers.Concatenate()([u2, c2]) c3 = conv_block(u2, 64) u1 = layers.UpSampling2D(interpolation="bilinear")(c3) u1 = layers.Concatenate()([u1, c1]) c4 = conv_block(u1, 32) logits = layers.Conv2D(num_classes, 1, padding="same")(c4) outputs = layers.Softmax(axis=-1)(logits) return Model(inputs, outputs) model = tiny_unet(num_classes=3) model.summary() model.compile( optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3), loss="sparse_categorical_crossentropy", metrics=["accuracy"] ) ``` ## 24.24 TensorFlow example: Dice loss and combined loss ```{python} #| label: ch24-python-tensorflow-dice #| eval: false import tensorflow as tf def soft_dice_loss_tf(y_true, y_pred, eps=1e-6): """ y_true and y_pred should both have shape (batch, H, W, K). y_true is one-hot. y_pred contains probabilities. """ y_true = tf.cast(y_true, tf.float32) y_pred = tf.cast(y_pred, tf.float32) axes = [1, 2] intersection = tf.reduce_sum(y_true * y_pred, axis=axes) denominator = tf.reduce_sum(y_true + y_pred, axis=axes) dice = (2.0 * intersection + eps) / (denominator + eps) # Average over classes and batch. return 1.0 - tf.reduce_mean(dice) def combined_segmentation_loss(y_true_sparse, y_pred): num_classes = tf.shape(y_pred)[-1] y_true_one_hot = tf.one_hot(tf.cast(y_true_sparse, tf.int32), num_classes) ce = tf.keras.losses.sparse_categorical_crossentropy(y_true_sparse, y_pred) ce = tf.reduce_mean(ce) dice = soft_dice_loss_tf(y_true_one_hot, y_pred) return ce + dice ``` ## 24.25 AI-assisted segmentation workflow AI tools are useful for segmentation projects, but they should be used in a mathematically controlled way. Good prompts ask the model to verify definitions, tensor shapes, and evaluation logic. ### Example AI prompts 1. **Metric verification** “Given this implementation of IoU and Dice for binary masks, check whether false positives and false negatives are handled correctly. Provide a small counterexample if there is a bug.” 2. **Tensor-shape audit** “My segmentation model outputs shape `(batch, H, W, K)` and my target mask has shape `(batch, H, W)`. Explain which loss functions expect sparse labels and which expect one-hot labels.” 3. **Loss-design critique** “For a medical image segmentation problem where the positive region is less than 2% of pixels, compare cross-entropy, weighted cross-entropy, Dice loss, focal loss, and Tversky loss.” 4. **Boundary analysis** “Explain why two masks can have high Dice score but visibly different boundaries. Suggest boundary-sensitive metrics and losses.” 5. **Statistical interpretation** “Interpret a segmentation probability map as a calibrated posterior probability. What diagnostics would test whether the map is calibrated?” ::: {.callout-warning} ## Avoid this common AI mistake Do not ask an AI tool only for “segmentation code.” Ask it to state the mask convention, class encoding, probability/logit convention, thresholding rule, and metric definition. Most segmentation bugs are convention bugs. ::: ## 24.26 A mathematical checklist for segmentation projects A segmentation experiment should specify the following objects. ### Data and labels - What is the image domain $\Omega_h$? - Are labels semantic, instance, or panoptic? - Are masks binary, class-index, one-hot, or instance-ID maps? - Are unlabeled pixels ignored or treated as background? - Is there annotation uncertainty or multiple annotators? ### Model output - Does the model output logits or probabilities? - What is the output shape? - Are predictions at full resolution or upsampled? - Are classes mutually exclusive or multilabel? - Are instance masks generated from boxes, centers, embeddings, or queries? ### Loss and optimization - Which likelihood or surrogate loss is being optimized? - Does the loss handle class imbalance? - Does the loss care about boundaries? - Is the loss differentiable with respect to logits? - Are augmentations label-preserving? ### Evaluation - Which metric is primary: IoU, Dice, AP, PQ, boundary distance, calibration? - Is the threshold fixed before evaluation? - Are small objects weighted equally or by area? - Are all classes averaged equally? - Is uncertainty evaluated? ## 24.27 Chapter summary Segmentation extends computer vision from image-level labels and bounding boxes to dense geometric prediction. The central mathematical objects are label functions, indicator masks, probability fields, sets, and boundaries. Semantic segmentation is often trained by pixelwise cross-entropy, but overlap metrics such as IoU and Dice better reflect mask quality. Instance segmentation introduces set-valued prediction and mask matching. Panoptic segmentation combines semantic labels with instance identities. Modern segmentation architectures, including FCNs, U-Net, DeepLab-style dilation, Mask R-CNN, and promptable foundation models, can be understood as different ways of combining local detail, global context, geometric constraints, and statistical uncertainty. ## 24.28 Exercises ### Conceptual exercises 1. Explain the difference between semantic segmentation, instance segmentation, and panoptic segmentation using mathematical notation. 2. Why is segmentation not just classification repeated independently at each pixel? 3. Give an example where pixel accuracy is high but segmentation quality is poor. 4. Explain why U-Net skip connections improve boundary quality. 5. Describe how point prompts can be interpreted as constraints on a mask. ### Mathematical exercises 1. Prove that for two masks $A$ and $B$, $$ \operatorname{Dice}(A,B)=\frac{2\operatorname{IoU}(A,B)}{1+\operatorname{IoU}(A,B)}. $$ 2. For binary segmentation, derive the gradient of the binary cross-entropy loss with respect to the logit $z$ when $p=\sigma(z)$. 3. Derive the soft Dice loss gradient with respect to a predicted probability $P_{ij}$. 4. Show that Dice loss is not a sum of independent pixel losses. 5. Suppose a foreground class has prevalence $\pi$. Derive inverse-frequency weights for binary weighted cross-entropy and discuss what happens when $\pi$ is very small. 6. Compute the Hausdorff distance between two one-dimensional intervals $A=[0,1]$ and $B=[\epsilon,1+\epsilon]$. 7. For panoptic quality, show how false positives and false negatives reduce recognition quality. ### Python exercises 1. Implement IoU and Dice for multiclass semantic segmentation using class-index masks. 2. Generate a synthetic dataset of circles and rectangles and train the small U-Net from this chapter. 3. Compare cross-entropy, Dice loss, and combined loss on a class-imbalanced synthetic segmentation problem. 4. Implement boundary extraction using binary erosion and compare boundary F1 to Dice. 5. Write a function that converts instance-ID masks into a list of binary instance masks. 6. Implement mask matching using IoU and compute AP for a small synthetic instance-segmentation example. 7. Create reliability diagrams for segmentation probability maps. ### AI-assisted exercises 1. Ask an AI tool to review your segmentation metric code. Require it to produce a $3\times3$ mask example that tests false positives and false negatives. 2. Ask an AI tool to explain the difference between logits, probabilities, sparse labels, and one-hot labels. Then check its answer against TensorFlow or PyTorch documentation. 3. Ask an AI tool to design an ablation study comparing loss functions for a rare-object segmentation problem. 4. Ask an AI tool to critique the use of Dice score as the only metric for a medical segmentation problem. 5. Ask an AI tool to propose a promptable segmentation workflow for annotating a new dataset. Identify which steps require human verification. ## 24.29 References and further reading Fully convolutional networks introduced the modern dense-prediction framework for semantic segmentation [@long2015fully]. U-Net is a central encoder-decoder architecture, especially in biomedical imaging [@ronneberger2015unet]. Mask R-CNN extends object detection with an instance mask head [@he2017mask]. Segment Anything is a major example of promptable segmentation and segmentation foundation models [@kirillov2023segment]. For classical segmentation, graph cuts, normalized cuts, morphology, and variational methods from earlier chapters remain important foundations.