20 Classification from a Mathematical View

Classification is the simplest-looking vision task: assign an image to one of several labels. Mathematically, however, image classification brings together probability, geometry, optimization, high-dimensional statistics, and representation learning. This chapter develops classification from first principles for MA Applied Mathematics and MS Statistics students. The central question is: How should a mathematical model convert an image into a decision, and how should we evaluate the uncertainty and reliability of that decision?

20.1 Learning Goals

After reading this chapter, students should be able to:

formulate image classification as a statistical decision problem;
derive the Bayes classifier from conditional risk minimization;
distinguish generative and discriminative classifiers;
explain linear scores, logits, decision regions, margins, and separating hyperplanes;
derive binary logistic regression, softmax regression, cross-entropy, and their gradients;
connect regularization to MAP estimation and high-dimensional generalization;
interpret confusion matrices, precision, recall, specificity, F1 score, ROC curves, calibration, and decision thresholds;
explain how modern vision classifiers use learned representations, CNNs, transformers, and foundation-model embeddings;
recognize statistical issues in vision classification: class imbalance, distribution shift, overconfidence, adversarial perturbations, and out-of-distribution inputs.

The central mathematical object is a decision rule

\[ \delta: \mathcal{X}\to \{1,\ldots,K\}, \]

where $\mathcal{X}$ is the image space and $K$ is the number of classes. The ideal classifier minimizes expected loss:

\[ R(\delta)=\mathbb{E}\big[L(Y,\delta(X))\big]. \]

The modern learning problem replaces the unknown distribution by training data and solves a regularized empirical-risk problem:

\[ \widehat\theta \in \arg\min_\theta \frac{1}{n}\sum_{i=1}^n \ell\big(y_i,f_\theta(x_i)\big)+\lambda\Omega(\theta). \]

20.2 19.1 The Image Classification Problem

An image classifier receives an image and outputs a class label. In a handwritten digit problem, the classes are $0,1,\ldots,9$. In a medical image problem, the classes may be normal'',benign’‘, and ``malignant’’. In a satellite image problem, the classes may represent land-use categories.

Let

\[ X\in\mathcal{X} \]

be an image-valued random variable and let

\[ Y\in\{1,\ldots,K\} \]

be its label. For a grayscale image of size $H\times W$, one simple choice is

\[ \mathcal{X}=\mathbb{R}^{H\times W}. \]

For a color image,

\[ \mathcal{X}=\mathbb{R}^{H\times W\times 3}. \]

After vectorization, both may be viewed as high-dimensional Euclidean vectors:

\[ x\in\mathbb{R}^p, \qquad p=HWC. \]

For example, a $224\times224$ RGB image has

\[ p=224\cdot224\cdot3=150{,}528 \]

raw pixel coordinates. This dimension is already much larger than many classical statistical datasets. A key theme in vision classification is therefore:

Raw pixels are high-dimensional measurements, not automatically good statistical coordinates.

A successful classifier usually depends on a representation map

\[ \phi:\mathcal{X}\to\mathbb{R}^d \]

that turns an image into features with better geometry for classification.

The representation $\phi$ may be hand-designed, such as edges, histograms, or HOG features, or learned, such as the hidden layers of a CNN, a vision transformer, or a vision-language model.

A classifier can then be written as

\[ \delta(x)=g(\phi(x)), \]

where $g$ is the final decision function.

20.3 19.2 Classification as Statistical Decision Theory

A decision rule is a measurable map

\[ \delta:\mathcal{X}\to\{1,\ldots,K\}. \]

To decide what makes a classifier good, we need a loss function. The most common loss for classification decisions is the zero-one loss:

\[ L(y,a)=1\{y\ne a\}. \]

The risk of a classifier is

\[ R(\delta) = \mathbb{E}[1\{Y\ne \delta(X)\}] = P(Y\ne\delta(X)). \]

Thus, under zero-one loss, risk is simply the probability of misclassification.

20.3.1 Conditional risk

Condition on a particular image $X=x$. If the classifier predicts class $a$, the conditional risk is

\[ R(a|x)=\mathbb{E}[L(Y,a)|X=x]. \]

For zero-one loss,

\[ R(a|x)=P(Y\ne a|X=x)=1-P(Y=a|X=x). \]

To minimize this conditional risk, we maximize the posterior class probability. Therefore the Bayes classifier is

\[ \delta^*(x)=\arg\max_{k\in\{1,\ldots,K\}} P(Y=k|X=x). \]

This is the best possible classifier under the true data-generating distribution. Its risk is called the Bayes risk.

20.3.2 Binary case

For $Y\in\{0,1\}$, define

\[ \eta(x)=P(Y=1|X=x). \]

Then the Bayes classifier is

\[ \delta^*(x)= \begin{cases} 1, & \eta(x)\ge 1/2,\\ 0, & \eta(x)<1/2. \end{cases} \]

Thus binary classification is mathematically equivalent to estimating a conditional probability function $\eta(x)$ and thresholding it.

20.3.3 Unequal costs

In medical imaging, false negatives may be much more costly than false positives. Suppose the cost of predicting $1$ when the true class is $0$ is $c_{01}$, and the cost of predicting $0$ when the true class is $1$ is $c_{10}$. Then predicting class $1$ has conditional risk

\[ R(1|x)=c_{01}P(Y=0|X=x)=c_{01}(1-\eta(x)), \]

while predicting class $0$ has conditional risk

\[ R(0|x)=c_{10}P(Y=1|X=x)=c_{10}\eta(x). \]

Predict class $1$ when

\[ c_{01}(1-\eta(x)) \le c_{10}\eta(x), \]

or equivalently

\[ \eta(x)\ge \frac{c_{01}}{c_{01}+c_{10}}. \]

The usual $1/2$ threshold is optimal only when the two error costs are equal.

20.4 19.3 Generative and Discriminative Classifiers

There are two major routes to estimating the Bayes classifier.

20.4.1 Generative classification

A generative classifier models

\[ p(x|Y=k) \quad\text{and}\quad P(Y=k). \]

Bayes’ rule gives

\[ P(Y=k|X=x) = \frac{p(x|Y=k)P(Y=k)}{\sum_{j=1}^K p(x|Y=j)P(Y=j)}. \]

The classifier is

\[ \delta(x)=\arg\max_k \left\{\log p(x|Y=k)+\log P(Y=k)\right\}. \]

Generative classifiers are useful when a probabilistic model for each class is meaningful. Examples include Gaussian discriminant analysis, naive Bayes, mixture models, and certain Bayesian image models.

20.4.2 Discriminative classification

A discriminative classifier directly models either

\[ P(Y=k|X=x) \]

or a score function whose maximum determines the class. Logistic regression, softmax regression, support vector machines, neural networks, and transformer classifiers are discriminative classifiers.

20.4.3 Comparison

Generative versus discriminative classification

Viewpoint	Model	Strength	Weakness
Generative	$p(x\|y)p(y)$	gives a full model of data; can use priors and missing data	density modeling in high-dimensional image space is difficult
Discriminative	$p(y\|x)$ or scores $f_k(x)$	focuses on decision boundary; often strong predictive performance	may give poor uncertainty under shift

A modern vision system often combines both perspectives. A neural classifier is discriminative during prediction, but its pretraining may use generative or self-supervised objectives.

20.5 19.4 Linear Scores and Decision Geometry

A simple classifier assigns each class a linear score:

\[ s_k(x)=w_k^Tx+b_k, \qquad k=1,\ldots,K. \]

The predicted class is

\[ \delta(x)=\arg\max_k s_k(x). \]

The vector $w_k$ determines which directions in feature space increase the score for class $k$. The bias $b_k$ shifts the score.

For two classes, define

\[ s(x)=w^Tx+b. \]

The decision rule is

\[ \delta(x)=1\{s(x)\ge 0\}. \]

The decision boundary is the hyperplane

\[ \{x\in\mathbb{R}^d:w^Tx+b=0\}. \]

The signed distance from a point $x$ to this hyperplane is

\[ \frac{w^Tx+b}{\|w\|_2}. \]

This is a geometric quantity. A point far from the boundary is classified with a large margin, while a point near the boundary is fragile.

20.5.1 Multi-class decision regions

For classes $j$ and $k$, the boundary between them is

\[ s_j(x)=s_k(x), \]

\[ (w_j-w_k)^Tx+(b_j-b_k)=0. \]

Thus a linear multi-class classifier partitions feature space into polyhedral decision regions.

20.6 Interactive 19.1: Linear Classifier and Decision Boundary

This demo shows a binary linear classifier in a two-dimensional feature space. The boundary is $w_1x_1+w_2x_2+b=0$. Moving the sliders changes the normal vector and the bias.

Interactive 19.1: Linear decision boundary

Adjust the weights and bias. Points are classified by the sign of w₁x₁+w₂x₂+b.

w₁: 1.00 w₂: -1.00 b: 0.00

For images, a linear classifier in raw pixel space is usually too limited. But a linear classifier on a good feature representation can be powerful. This is the logic behind linear probing of deep features.

20.7 19.5 Binary Logistic Regression

Linear classification gives a score, but statistical classification often wants a probability. Binary logistic regression models

\[ P(Y=1|X=x)=\sigma(w^Tx+b), \]

where

\[ \sigma(t)=\frac{1}{1+e^{-t}}. \]

The log-odds are linear:

\[ \log\frac{P(Y=1|X=x)}{P(Y=0|X=x)}=w^Tx+b. \]

Thus logistic regression is a linear model for the log-odds.

20.7.1 Bernoulli likelihood

Given data $(x_i,y_i)$ with $y_i\in\{0,1\}$, define

\[ p_i=\sigma(w^Tx_i+b). \]

The conditional likelihood is

\[ L(w,b)=\prod_{i=1}^n p_i^{y_i}(1-p_i)^{1-y_i}. \]

The negative log-likelihood is

\[ \ell(w,b) = -\sum_{i=1}^n \left[y_i\log p_i+(1-y_i)\log(1-p_i)\right]. \]

This is binary cross-entropy.

If we use labels $y_i\in\{-1,+1\}$ and score $s_i=w^Tx_i+b$, the loss can be written compactly as

\[ \ell_i(w,b)=\log(1+e^{-y_is_i}). \]

The quantity $m_i=y_is_i$ is the margin. Correct and confident predictions have large positive margins.

20.7.2 Gradient and Hessian

Let $X\in\mathbb{R}^{n\times d}$ be the design matrix, let $p=(p_1,\ldots,p_n)^T$, and use an intercept column if desired. Then the gradient is

\[ \nabla_w \ell=X^T(p-y). \]

The Hessian is

\[ \nabla_w^2\ell=X^TWX, \]

where

\[ W=\operatorname{diag}(p_i(1-p_i)). \]

Because $W$ is positive semidefinite, the negative log-likelihood is convex in $w$. This convexity is a major reason logistic regression is mathematically clean.

20.7.3 Regularized logistic regression

In high-dimensional image problems, regularization is essential. Ridge-regularized logistic regression solves

\[ \min_{w,b} \frac{1}{n}\sum_{i=1}^n \log(1+e^{-y_i(w^Tx_i+b)}) + \lambda\|w\|_2^2. \]

The regularization term shrinks weights and improves stability. Bayesianly, it corresponds to a Gaussian prior

\[ w\sim \mathcal{N}(0,\tau^2I). \]

An $\ell_1$ penalty,

\[ \lambda\|w\|_1, \]

encourages sparsity and can be useful when only a small subset of features is relevant.

20.8 Interactive 19.2: Margin, Sigmoid, and Logistic Loss

The logistic loss is small for large positive margins and approximately linear for large negative margins. This behavior makes it smoother than hinge loss but still strongly penalizes confident mistakes.

Interactive 19.2: Logistic loss and margin

The margin is m = y(wᵀx+b). Positive margins are correct; negative margins are incorrect.

highlighted margin m: 0.00

20.9 19.6 Multiclass Softmax Regression

For $K$ classes, softmax regression assigns each class a logit

\[ z_k=w_k^Tx+b_k, \qquad k=1,\ldots,K. \]

The predicted probability is

\[ p_k(x)=P(Y=k|X=x) = \frac{e^{z_k}}{\sum_{j=1}^K e^{z_j}}. \]

The predicted class is

\[ \widehat y=\arg\max_k p_k(x)=\arg\max_k z_k. \]

The softmax probabilities change if all logits are scaled, but the predicted class does not. This is important for calibration.

20.9.1 Cross-entropy loss

If $y_i$ is encoded as a one-hot vector, the multiclass cross-entropy loss is

\[ \ell_i=-\sum_{k=1}^K y_{ik}\log p_{ik}. \]

Since only one component of $y_i$ equals $1$, this is

\[ \ell_i=-\log p_{i,y_i}. \]

The gradient with respect to the logit $z_{ik}$ is especially simple:

\[ \frac{\partial \ell_i}{\partial z_{ik}}=p_{ik}-y_{ik}. \]

This formula is one reason softmax cross-entropy is the standard output loss for neural classifiers.

20.9.2 Temperature scaling

For a temperature $T>0$, define

\[ p_k^{(T)}(x)= \frac{e^{z_k/T}}{\sum_{j=1}^K e^{z_j/T}}. \]

Small $T$ makes probabilities sharper. Large $T$ makes them flatter. Temperature scaling is often used for probability calibration.

20.10 Interactive 19.3: Softmax Temperature and Confidence

This demo shows how the same logits can lead to different probability confidence depending on temperature.

Interactive 19.3: Softmax temperature

The predicted class is unchanged by temperature, but the probabilities become sharper or flatter.

temperature T: 1.00

Softmax probabilities should not be interpreted blindly as calibrated probabilities. A model can be highly accurate but poorly calibrated, meaning its stated probabilities do not match empirical frequencies.

20.11 19.7 Gaussian Discriminant Analysis: LDA and QDA

Generative Gaussian classifiers are mathematically useful because they show how decision boundaries arise from probability models.

Assume

\[ X|Y=k\sim \mathcal{N}(\mu_k,\Sigma_k), \qquad P(Y=k)=\pi_k. \]

The Bayes classifier chooses the class maximizing

\[ \log \pi_k -\frac{1}{2}\log\det\Sigma_k - \frac{1}{2}(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k). \]

20.11.1 Linear discriminant analysis

If all classes share the same covariance matrix,

\[ \Sigma_k=\Sigma, \]

then terms quadratic in $x$ cancel across classes. The discriminant function is linear:

\[ \delta_k(x)=x^T\Sigma^{-1}\mu_k - \frac{1}{2}\mu_k^T\Sigma^{-1}\mu_k + \log\pi_k. \]

This is linear discriminant analysis, or LDA.

For two classes, the boundary is a hyperplane. The normal vector is proportional to

\[ \Sigma^{-1}(\mu_1-\mu_0). \]

Thus LDA does not only compare class means. It compares means after whitening by the covariance.

20.11.2 Quadratic discriminant analysis

If each class has its own covariance $\Sigma_k$, the quadratic terms do not cancel. The decision boundary is quadratic. This is quadratic discriminant analysis, or QDA.

In image classification, direct LDA or QDA on raw pixels is usually unstable because $p$ is huge and covariance estimation is difficult. But LDA/QDA ideas remain important after dimensionality reduction or feature extraction.

20.12 Interactive 19.4: LDA versus QDA Geometry

This demo compares shared-covariance and class-specific covariance models. LDA produces a linear boundary; QDA can curve.

Interactive 19.4: LDA and QDA decision geometry

Change the covariance ratio of class 1. When covariances are equal, the Bayes boundary is linear; when they differ, it becomes quadratic.

class 1 covariance scale: 1.00

20.13 19.8 Support Vector Machines and Margin-Based Classification

Support vector machines classify using margins. For labels $y_i\in\{-1,+1\}$ and scores $s_i=w^Tx_i+b$, the hard-margin SVM solves

\[ \min_{w,b}\frac{1}{2}\|w\|_2^2 \]

subject to

\[ y_i(w^Tx_i+b)\ge 1, \qquad i=1,\ldots,n. \]

The geometric margin is

\[ \frac{1}{\|w\|_2}. \]

Thus minimizing $\|w\|_2^2$ maximizes the margin.

For nonseparable data, the soft-margin SVM solves

\[ \min_{w,b} \frac{1}{n}\sum_{i=1}^n \max(0,1-y_i(w^Tx_i+b)) + \lambda\|w\|_2^2. \]

The loss

\[ \max(0,1-m) \]

is the hinge loss. It ignores points with margin at least $1$ and penalizes points inside the margin or on the wrong side of the boundary.

20.13.1 Kernel idea

A kernel classifier replaces dot products by a kernel

\[ K(x,z)=\langle \phi(x),\phi(z)\rangle. \]

This allows linear classification in a high-dimensional feature space without explicitly computing $\phi$. Kernel methods were central in classical vision before large-scale deep learning became dominant. In modern vision, neural networks learn the representation $\phi_\theta$ directly from data.

20.14 19.9 Feature Maps for Image Classification

The quality of classification depends heavily on the feature map.

20.14.1 Raw pixel features

Raw pixel vectors are simple:

\[ \phi(x)=\operatorname{vec}(x). \]

This is useful for small aligned images, such as centered handwritten digits. It is fragile when objects translate, rotate, deform, or change illumination.

20.14.2 Hand-designed features

Classical features include:

edge histograms;
color histograms;
texture features;
HOG features;
SIFT-like local descriptors;
bag-of-visual-words representations.

These features encode prior knowledge about images. For example, gradient-orientation histograms are less sensitive to uniform brightness changes than raw pixels.

20.14.3 Learned features

Modern models learn

\[ \phi_\theta(x) \]

from data. A classifier may use

\[ f_{\theta,W}(x)=W\phi_\theta(x)+b. \]

The representation $\phi_\theta$ is often a CNN backbone, a vision transformer, or a pretrained foundation model. A linear classifier on top of frozen features is called a linear probe.

20.14.4 Geometry of features

A good feature map should make same-class examples close and different-class examples separated. For class means

\[ \bar z_k=\frac{1}{n_k}\sum_{i:y_i=k}\phi(x_i), \]

a simple measure of within-class scatter is

\[ S_W=\sum_{k=1}^K\sum_{i:y_i=k}(z_i-\bar z_k)(z_i-\bar z_k)^T. \]

A measure of between-class scatter is

\[ S_B=\sum_{k=1}^K n_k(\bar z_k-\bar z)(\bar z_k-\bar z)^T. \]

A representation is more linearly classifiable when between-class variation dominates within-class variation.

20.15 19.10 Empirical Risk, Generalization, and Regularization

In practice, the true distribution of $(X,Y)$ is unknown. We observe training data

\[ \mathcal{D}_n=\{(x_i,y_i)\}_{i=1}^n. \]

The empirical risk is

\[ \widehat R_n(f)=\frac{1}{n}\sum_{i=1}^n \ell(y_i,f(x_i)). \]

Training chooses a model that approximately minimizes empirical risk plus regularization:

\[ \widehat f \in \arg\min_{f\in\mathcal{F}} \widehat R_n(f)+\lambda\Omega(f). \]

The central statistical question is not only whether $\widehat R_n$ is small, but whether the population risk

\[ R(f)=\mathbb{E}[\ell(Y,f(X))] \]

is small.

20.15.1 Bias-variance and representation

If the classifier class is too simple, it has high bias. If it is too flexible relative to data, it may overfit. In modern deep learning, this classical description is incomplete, but still useful. Regularization, augmentation, architecture, pretraining, and early stopping all influence generalization.

20.15.2 Data augmentation as invariance

Many image labels are invariant under transformations. If $g$ is a small translation, rotation, crop, or color perturbation, often

\[ Y(x)=Y(g\cdot x). \]

Data augmentation encourages the model to satisfy

\[ f_\theta(x)\approx f_\theta(g\cdot x). \]

This is a practical way to inject geometric prior knowledge into statistical learning.

20.16 19.11 Class Imbalance and Cost-Sensitive Learning

Vision datasets often have class imbalance. In medical diagnosis, positive cases may be rare. In defect detection, anomalies may be rare. In ecological or satellite data, some categories may dominate.

If class $k$ has $n_k$ examples, a weighted cross-entropy may use weights

\[ \alpha_k\propto \frac{1}{n_k}. \]

The weighted loss is

\[ \ell_i=-\alpha_{y_i}\log p_{i,y_i}. \]

This changes the training objective so that rare classes have more influence.

20.16.1 Focal loss

For difficult imbalanced detection problems, focal loss modifies cross-entropy:

\[ \ell_i=-(1-p_{i,y_i})^\gamma\log p_{i,y_i}. \]

When $p_{i,y_i}$ is large, the factor $(1-p_{i,y_i})^\gamma$ downweights easy examples. This idea is especially important in object detection, where background examples can dominate foreground examples.

20.16.2 Thresholding under imbalance

If a model estimates $P(Y=1|X=x)$ accurately, the classification threshold should be chosen according to costs and deployment goals, not automatically set to $0.5$.

20.17 19.12 Evaluation Metrics for Classification

For binary classification, predictions can be summarized by a confusion matrix.

	Predicted positive	Predicted negative
True positive	TP	FN
True negative	FP	TN

Common metrics are:

\[ \text{Accuracy}=\frac{TP+TN}{TP+FP+FN+TN}, \]

\[ \text{Precision}=\frac{TP}{TP+FP}, \]

\[ \text{Recall}=\frac{TP}{TP+FN}, \]

\[ \text{Specificity}=\frac{TN}{TN+FP}, \]

and

\[ F_1=\frac{2\cdot \text{Precision}\cdot\text{Recall}}{\text{Precision}+\text{Recall}}. \]

Accuracy can be misleading under class imbalance. For example, if only $1\%$ of images are positive, a classifier that always predicts negative has $99\%$ accuracy but zero recall.

20.17.1 ROC and precision-recall curves

A probabilistic classifier outputs a score $s(x)$ or probability $p(x)$. Varying the threshold creates different values of true-positive rate and false-positive rate.

The ROC curve plots

\[ \text{TPR}=\frac{TP}{TP+FN} \]

against

\[ \text{FPR}=\frac{FP}{FP+TN}. \]

A precision-recall curve is often more informative for rare-positive problems.

20.17.2 Calibration

A classifier is calibrated if, among examples assigned confidence $0.8$, approximately $80\%$ are correct. Calibration matters whenever probabilities are used for decision-making.

The Brier score for binary classification is

\[ \frac{1}{n}\sum_{i=1}^n (p_i-y_i)^2. \]

Expected calibration error approximates calibration by binning predictions by confidence.

20.18 Interactive 19.5: Threshold, Confusion Matrix, and Metrics

This demo shows how changing the threshold changes precision, recall, specificity, and F1 score.

Interactive 19.5: Threshold-dependent evaluation

A fixed score model can produce many classifiers, depending on the threshold.

classification threshold: 0.50

20.19 19.13 Neural Image Classifiers

A modern neural image classifier can be written as

\[ z=f_\theta(x)\in\mathbb{R}^K, \]

where $z$ is the vector of logits. The predicted probabilities are

\[ p=\operatorname{softmax}(z). \]

The network is trained by minimizing average cross-entropy:

\[ \min_\theta \frac{1}{n}\sum_{i=1}^n - \log \operatorname{softmax}(f_\theta(x_i))_{y_i}. \]

A CNN classifier uses convolutional layers to build local-to-global representations. A vision transformer classifier divides the image into patches, embeds those patches as tokens, applies self-attention, and then classifies from a pooled or special token.

20.19.1 Why deep classifiers work well for images

Images have structure that generic high-dimensional vectors do not have:

local spatial correlation;
translation and deformation structure;
compositional objects made of parts;
multi-scale patterns;
repeated visual motifs;
semantic labels that are often invariant under many transformations.

CNNs encode locality and translation equivariance. Transformers can model long-range interactions and learn global context. Pretrained foundation models learn representations from very large datasets and can be adapted to smaller classification tasks.

20.19.2 Linear head on a learned representation

Many practical vision classifiers use

\[ f_{\theta,W}(x)=W\phi_\theta(x)+b. \]

Here $\phi_\theta$ is the backbone and $W,b$ form the classification head. If $\phi_\theta$ is fixed and only $W,b$ are trained, the problem is softmax regression on learned features. If both $\theta$ and $W,b$ are trained, this is end-to-end representation learning.

20.20 19.14 Robustness, Shift, and Uncertainty

High test accuracy on a benchmark does not guarantee safe deployment. Classification models may fail under several types of shift.

20.20.1 Covariate shift

The image distribution changes:

\[ p_{\text{train}}(x)\ne p_{\text{test}}(x), \]

but the conditional label rule may remain similar.

Examples include new lighting, new camera hardware, different hospitals, different satellites, or different image preprocessing.

20.20.2 Label shift

Class priors change:

\[ P_{\text{train}}(Y=k)\ne P_{\text{test}}(Y=k). \]

This is common in medical screening and anomaly detection.

20.20.3 Concept shift

The conditional distribution changes:

\[ P_{\text{train}}(Y|X)\ne P_{\text{test}}(Y|X). \]

This is more serious. It means the meaning of visual evidence has changed.

20.20.4 Adversarial perturbations

A small perturbation $r$ may change the predicted class even when $\|r\|$ is small:

\[ \delta(x+r)\ne\delta(x). \]

This is possible because high-dimensional decision boundaries can be close to many data points. Robust classification studies objectives such as

\[ \min_\theta \frac{1}{n}\sum_{i=1}^n \max_{\|r\|\le\epsilon} \ell(y_i,f_\theta(x_i+r)). \]

20.20.5 Out-of-distribution inputs

A classifier trained on natural object images may still produce confident predictions for medical images, random noise, or drawings. This is a warning: softmax confidence is not the same as epistemic uncertainty.

20.21 19.15 Python Example: Logistic Regression on Image Features

The following example uses simple synthetic image features. It is intentionally small enough for students to understand before moving to CNNs.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report, ConfusionMatrixDisplay

rng = np.random.default_rng(7243)

# Synthetic 16 x 16 grayscale images.
# Class 0: vertical bright bar. Class 1: horizontal bright bar.
n = 600
H = W = 16
X_img = np.zeros((n, H, W))
y = np.zeros(n, dtype=int)

for i in range(n):
    img = 0.15 * rng.normal(size=(H, W))
    if i < n // 2:
        c = rng.integers(5, 11)
        img[:, c-1:c+2] += 1.0
        y[i] = 0
    else:
        r = rng.integers(5, 11)
        img[r-1:r+2, :] += 1.0
        y[i] = 1
    X_img[i] = img

X = X_img.reshape(n, -1)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0, stratify=y
)

model = Pipeline([
    ("scale", StandardScaler()),
    ("logreg", LogisticRegression(C=1.0, max_iter=2000))
])

model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print(classification_report(y_test, y_pred, target_names=["vertical", "horizontal"]))
ConfusionMatrixDisplay.from_predictions(y_test, y_pred)
plt.show()

# Inspect the learned weight image.
w = model.named_steps["logreg"].coef_[0].reshape(H, W)
plt.imshow(w, cmap="coolwarm")
plt.colorbar()
plt.title("learned logistic-regression weight image")
plt.show()

The learned weight image is mathematically meaningful. Positive pixels support one class; negative pixels support the other. This is a rare case where a linear classifier on raw pixels is interpretable because the synthetic images are aligned.

20.22 19.16 Python Example: Softmax Regression with PCA Features

For higher-dimensional images, PCA can reduce dimension before classification.

from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score

clf = Pipeline([
    ("scale", StandardScaler()),
    ("pca", PCA(n_components=20, random_state=0)),
    ("softmax", LogisticRegression(
        multi_class="multinomial",
        solver="lbfgs",
        max_iter=3000,
        C=2.0
    ))
])

clf.fit(X_train, y_train)
prob = clf.predict_proba(X_test)
pred = prob.argmax(axis=1)

print("test accuracy:", accuracy_score(y_test, pred))
print("first five probability vectors:")
print(prob[:5])

For real images, PCA features are usually weaker than CNN or transformer features, but this pipeline is mathematically transparent. It combines linear algebra, dimension reduction, likelihood-based classification, and statistical evaluation.

20.23 19.17 TensorFlow Example: A Small CNN Classifier

The following example gives a minimal CNN classifier. It is meant as a template, not as a large benchmark model.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

num_classes = 2

cnn = keras.Sequential([
    layers.Input(shape=(16, 16, 1)),
    layers.Conv2D(16, kernel_size=3, padding="same", activation="relu"),
    layers.MaxPooling2D(pool_size=2),
    layers.Conv2D(32, kernel_size=3, padding="same", activation="relu"),
    layers.GlobalAveragePooling2D(),
    layers.Dense(num_classes)  # logits, not probabilities
])

cnn.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=["accuracy"]
)

X_train_img = X_train.reshape(-1, 16, 16, 1).astype("float32")
X_test_img = X_test.reshape(-1, 16, 16, 1).astype("float32")

history = cnn.fit(
    X_train_img, y_train,
    validation_split=0.2,
    epochs=10,
    batch_size=32,
    verbose=1
)

logits = cnn.predict(X_test_img)
prob = tf.nn.softmax(logits, axis=1).numpy()
print(prob[:5])

Notice the final dense layer returns logits. The softmax is included inside the loss during training by setting from_logits=True. This is numerically more stable than applying softmax first and then taking logarithms.

20.24 19.18 AI-Assisted Study Components

AI tools can help students learn classification, but they should be used as mathematical assistants rather than answer generators.

Prompt 1: derive the Bayes classifier

Ask an AI assistant:

Derive the Bayes classifier for zero-one loss. Then repeat the derivation when false negatives cost five times as much as false positives. Explain the threshold change.

Then verify that the threshold formula agrees with the conditional-risk calculation in this chapter.

Prompt 2: debug cross-entropy confusion

Ask:

I have logits from a neural network and integer class labels. Should I apply softmax before using cross-entropy? Explain the difference between logits, probabilities, and log-probabilities.

Check whether the answer mentions numerical stability.

Prompt 3: evaluate a classifier under imbalance

Ask:

A medical image classifier has 98% accuracy, but the positive class prevalence is 1%. What other metrics should I inspect and why?

A good answer should discuss recall/sensitivity, specificity, precision, the confusion matrix, ROC/PR curves, calibration, and clinical decision thresholds.

Prompt 4: representation geometry

Ask:

Explain why a linear classifier may fail on raw pixels but work well on pretrained CNN or vision-transformer embeddings. Use the language of feature maps and decision boundaries.

20.25 19.19 Common Mistakes

Mistake 1: treating softmax confidence as truth.

A softmax probability is a model output. It may be poorly calibrated, especially under distribution shift.

Mistake 2: using accuracy alone.

Accuracy can hide poor minority-class performance. Always inspect the confusion matrix and task-relevant metrics.

Mistake 3: forgetting the feature map.

A classifier is not only its final loss. Its performance depends strongly on the representation $\phi(x)$.

Mistake 4: training-test leakage through preprocessing.

Scalers, PCA, feature selection, and threshold choice should be fitted using training data or validation data, not test data.

Mistake 5: assuming the $0.5$ threshold is always optimal.

The optimal threshold depends on class priors, costs, and deployment goals.

20.26 19.20 Chapter Summary

Classification is a decision problem under uncertainty. The Bayes classifier minimizes expected zero-one loss by choosing the most probable class given the image. Logistic regression and softmax regression model class probabilities through linear logits and are trained by cross-entropy. SVMs emphasize margins. Generative classifiers such as LDA and QDA show how probability models induce linear or quadratic boundaries.

For computer vision, the most important practical issue is representation. Raw pixels are high-dimensional and fragile; feature maps, CNNs, transformers, and foundation models aim to transform images into spaces where simple decision rules work well. Evaluation must go beyond accuracy and include threshold-dependent metrics, calibration, class imbalance, and robustness under distribution shift.

20.27 Exercises

20.27.1 Conceptual Exercises

Explain why the Bayes classifier is optimal under zero-one loss.
Give an example where accuracy is misleading for image classification.
Explain the difference between a logit, a probability, and a predicted class.
Why can a linear classifier be weak on raw pixels but strong on learned features?
Describe the difference between generative and discriminative classification.
Explain why calibration matters in medical imaging.
What is the role of data augmentation in classification?
Why is class imbalance especially important in anomaly detection?

20.27.2 Mathematical Exercises

Derive the binary Bayes classifier for zero-one loss.
Derive the optimal threshold when false negatives cost $c_{10}$ and false positives cost $c_{01}$.
Show that the logistic-regression negative log-likelihood is \[ \sum_{i=1}^n \log(1+e^{-y_i(w^Tx_i+b)}) \] when $y_i\in\{-1,+1\}$.
Derive the gradient $X^T(p-y)$ for binary logistic regression.
Derive the softmax gradient \[ \frac{\partial \ell_i}{\partial z_{ik}}=p_{ik}-y_{ik}. \]
For two Gaussian classes with common covariance $\Sigma$, derive the LDA decision boundary.
Show that if $\Sigma_0\ne\Sigma_1$, the Gaussian Bayes boundary is generally quadratic.
Prove that the signed distance from $x$ to the hyperplane $w^Tx+b=0$ is $(w^Tx+b)/\|w\|_2$.
Derive the hard-margin SVM margin $1/\|w\|_2$ under constraints $y_i(w^Tx_i+b)\ge 1$.
Show that temperature scaling changes softmax probabilities but not the argmax class.

20.27.3 Python Exercises

Implement logistic regression from scratch with gradient descent on a synthetic image dataset.
Compare raw-pixel logistic regression with PCA plus logistic regression.
Generate two synthetic image classes and visualize the learned weight image.
Compute a confusion matrix and classification report for different thresholds.
Plot ROC and precision-recall curves for an imbalanced image classification problem.
Train a small CNN on synthetic shape images and compare it with logistic regression.
Use class weights in cross-entropy and measure recall on a rare class.
Apply temperature scaling on validation logits and inspect calibration before and after scaling.

20.27.4 AI-Assisted Exercises

Ask an AI assistant to derive LDA from Gaussian class-conditional densities. Check every algebraic step.
Ask for a counterexample where high accuracy hides poor recall. Translate the example into a confusion matrix.
Ask the assistant to explain softmax cross-entropy to a statistics student. Improve the response by adding the gradient derivation.
Ask for possible distribution shifts in a medical image classifier. Categorize them as covariate shift, label shift, or concept shift.
Ask the assistant to generate a checklist for evaluating an image classifier before deployment. Add mathematical metrics to the checklist.

20.28 References and Further Reading

Good general references for statistical classification include Bishop (2006), Hastie et al. (2009), and Duda et al. (2001). For margin methods, see Cortes and Vapnik (1995) and Vapnik (1998). For deep learning and image classification, see Goodfellow et al. (2016), LeCun et al. (1998), Krizhevsky et al. (2012), and He et al. (2015). For calibration and evaluation, see Platt (1999), Guo et al. (2017), and Fawcett (2006).

Bishop, Christopher M. 2006. Pattern Recognition and Machine Learning. Springer.

Cortes, Corinna, and Vladimir Vapnik. 1995. “Support-Vector Networks.” Machine Learning 20 (3): 273–97.

Duda, Richard O., Peter E. Hart, and David G. Stork. 2001. Pattern Classification. 2nd ed. Wiley.

Fawcett, Tom. 2006. “An Introduction to ROC Analysis.” Pattern Recognition Letters 27 (8): 861–74.

Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. https://www.deeplearningbook.org/.

Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. “On Calibration of Modern Neural Networks.” International Conference on Machine Learning, 1321–30.

Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. Springer. https://hastie.su.domains/ElemStatLearn/.

He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. “Deep Residual Learning for Image Recognition.” arXiv Preprint arXiv:1512.03385. https://arxiv.org/abs/1512.03385.

Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. 2012. “ImageNet Classification with Deep Convolutional Neural Networks.” Advances in Neural Information Processing Systems.

LeCun, Yann, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. “Gradient-Based Learning Applied to Document Recognition.” Proceedings of the IEEE 86: 2278–324.

Platt, John C. 1999. “Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods.” Advances in Large Margin Classifiers, 61–74.

Vapnik, Vladimir N. 1998. Statistical Learning Theory. Wiley.

# Classification from a Mathematical View ::: {.hero-box} Classification is the simplest-looking vision task: assign an image to one of several labels. Mathematically, however, image classification brings together probability, geometry, optimization, high-dimensional statistics, and representation learning. This chapter develops classification from first principles for MA Applied Mathematics and MS Statistics students. The central question is: **How should a mathematical model convert an image into a decision, and how should we evaluate the uncertainty and reliability of that decision?** ::: ## Learning Goals After reading this chapter, students should be able to: 1. formulate image classification as a statistical decision problem; 2. derive the Bayes classifier from conditional risk minimization; 3. distinguish generative and discriminative classifiers; 4. explain linear scores, logits, decision regions, margins, and separating hyperplanes; 5. derive binary logistic regression, softmax regression, cross-entropy, and their gradients; 6. connect regularization to MAP estimation and high-dimensional generalization; 7. interpret confusion matrices, precision, recall, specificity, F1 score, ROC curves, calibration, and decision thresholds; 8. explain how modern vision classifiers use learned representations, CNNs, transformers, and foundation-model embeddings; 9. recognize statistical issues in vision classification: class imbalance, distribution shift, overconfidence, adversarial perturbations, and out-of-distribution inputs. ::: {.math-spine} The central mathematical object is a decision rule $$ \delta: \mathcal{X}\to \{1,\ldots,K\}, $$ where $\mathcal{X}$ is the image space and $K$ is the number of classes. The ideal classifier minimizes expected loss: $$ R(\delta)=\mathbb{E}\big[L(Y,\delta(X))\big]. $$ The modern learning problem replaces the unknown distribution by training data and solves a regularized empirical-risk problem: $$ \widehat\theta \in \arg\min_\theta \frac{1}{n}\sum_{i=1}^n \ell\big(y_i,f_\theta(x_i)\big)+\lambda\Omega(\theta). $$ ::: ## 19.1 The Image Classification Problem An image classifier receives an image and outputs a class label. In a handwritten digit problem, the classes are $0,1,\ldots,9$. In a medical image problem, the classes may be ``normal'', ``benign'', and ``malignant''. In a satellite image problem, the classes may represent land-use categories. Let $$ X\in\mathcal{X} $$ be an image-valued random variable and let $$ Y\in\{1,\ldots,K\} $$ be its label. For a grayscale image of size $H\times W$, one simple choice is $$ \mathcal{X}=\mathbb{R}^{H\times W}. $$ For a color image, $$ \mathcal{X}=\mathbb{R}^{H\times W\times 3}. $$ After vectorization, both may be viewed as high-dimensional Euclidean vectors: $$ x\in\mathbb{R}^p, \qquad p=HWC. $$ For example, a $224\times224$ RGB image has $$ p=224\cdot224\cdot3=150{,}528 $$ raw pixel coordinates. This dimension is already much larger than many classical statistical datasets. A key theme in vision classification is therefore: ::: {.cv-card} **Raw pixels are high-dimensional measurements, not automatically good statistical coordinates.** A successful classifier usually depends on a representation map $$ \phi:\mathcal{X}\to\mathbb{R}^d $$ that turns an image into features with better geometry for classification. ::: The representation $\phi$ may be hand-designed, such as edges, histograms, or HOG features, or learned, such as the hidden layers of a CNN, a vision transformer, or a vision-language model. A classifier can then be written as $$ \delta(x)=g(\phi(x)), $$ where $g$ is the final decision function. ## 19.2 Classification as Statistical Decision Theory A decision rule is a measurable map $$ \delta:\mathcal{X}\to\{1,\ldots,K\}. $$ To decide what makes a classifier good, we need a loss function. The most common loss for classification decisions is the zero-one loss: $$ L(y,a)=1\{y\ne a\}. $$ The risk of a classifier is $$ R(\delta) = \mathbb{E}[1\{Y\ne \delta(X)\}] = P(Y\ne\delta(X)). $$ Thus, under zero-one loss, risk is simply the probability of misclassification. ### Conditional risk Condition on a particular image $X=x$. If the classifier predicts class $a$, the conditional risk is $$ R(a|x)=\mathbb{E}[L(Y,a)|X=x]. $$ For zero-one loss, $$ R(a|x)=P(Y\ne a|X=x)=1-P(Y=a|X=x). $$ To minimize this conditional risk, we maximize the posterior class probability. Therefore the Bayes classifier is $$ \delta^*(x)=\arg\max_{k\in\{1,\ldots,K\}} P(Y=k|X=x). $$ This is the best possible classifier under the true data-generating distribution. Its risk is called the Bayes risk. ### Binary case For $Y\in\{0,1\}$, define $$ \eta(x)=P(Y=1|X=x). $$ Then the Bayes classifier is $$ \delta^*(x)= \begin{cases} 1, & \eta(x)\ge 1/2,\\ 0, & \eta(x)<1/2. \end{cases} $$ Thus binary classification is mathematically equivalent to estimating a conditional probability function $\eta(x)$ and thresholding it. ### Unequal costs In medical imaging, false negatives may be much more costly than false positives. Suppose the cost of predicting $1$ when the true class is $0$ is $c_{01}$, and the cost of predicting $0$ when the true class is $1$ is $c_{10}$. Then predicting class $1$ has conditional risk $$ R(1|x)=c_{01}P(Y=0|X=x)=c_{01}(1-\eta(x)), $$ while predicting class $0$ has conditional risk $$ R(0|x)=c_{10}P(Y=1|X=x)=c_{10}\eta(x). $$ Predict class $1$ when $$ c_{01}(1-\eta(x)) \le c_{10}\eta(x), $$ or equivalently $$ \eta(x)\ge \frac{c_{01}}{c_{01}+c_{10}}. $$ The usual $1/2$ threshold is optimal only when the two error costs are equal. ## 19.3 Generative and Discriminative Classifiers There are two major routes to estimating the Bayes classifier. ### Generative classification A generative classifier models $$ p(x|Y=k) \quad\text{and}\quad P(Y=k). $$ Bayes' rule gives $$ P(Y=k|X=x) = \frac{p(x|Y=k)P(Y=k)}{\sum_{j=1}^K p(x|Y=j)P(Y=j)}. $$ The classifier is $$ \delta(x)=\arg\max_k \left\{\log p(x|Y=k)+\log P(Y=k)\right\}. $$ Generative classifiers are useful when a probabilistic model for each class is meaningful. Examples include Gaussian discriminant analysis, naive Bayes, mixture models, and certain Bayesian image models. ### Discriminative classification A discriminative classifier directly models either $$ P(Y=k|X=x) $$ or a score function whose maximum determines the class. Logistic regression, softmax regression, support vector machines, neural networks, and transformer classifiers are discriminative classifiers. ### Comparison ::: {.cv-card} **Generative versus discriminative classification** | Viewpoint | Model | Strength | Weakness | |---|---|---|---| | Generative | $p(x|y)p(y)$ | gives a full model of data; can use priors and missing data | density modeling in high-dimensional image space is difficult | | Discriminative | $p(y|x)$ or scores $f_k(x)$ | focuses on decision boundary; often strong predictive performance | may give poor uncertainty under shift | ::: A modern vision system often combines both perspectives. A neural classifier is discriminative during prediction, but its pretraining may use generative or self-supervised objectives. ## 19.4 Linear Scores and Decision Geometry A simple classifier assigns each class a linear score: $$ s_k(x)=w_k^Tx+b_k, \qquad k=1,\ldots,K. $$ The predicted class is $$ \delta(x)=\arg\max_k s_k(x). $$ The vector $w_k$ determines which directions in feature space increase the score for class $k$. The bias $b_k$ shifts the score. For two classes, define $$ s(x)=w^Tx+b. $$ The decision rule is $$ \delta(x)=1\{s(x)\ge 0\}. $$ The decision boundary is the hyperplane $$ \{x\in\mathbb{R}^d:w^Tx+b=0\}. $$ The signed distance from a point $x$ to this hyperplane is $$ \frac{w^Tx+b}{\|w\|_2}. $$ This is a geometric quantity. A point far from the boundary is classified with a large margin, while a point near the boundary is fragile. ### Multi-class decision regions For classes $j$ and $k$, the boundary between them is $$ s_j(x)=s_k(x), $$ or $$ (w_j-w_k)^Tx+(b_j-b_k)=0. $$ Thus a linear multi-class classifier partitions feature space into polyhedral decision regions. ## Interactive 19.1: Linear Classifier and Decision Boundary This demo shows a binary linear classifier in a two-dimensional feature space. The boundary is $w_1x_1+w_2x_2+b=0$. Moving the sliders changes the normal vector and the bias. ```{=html} <div class="interactive-card"> <h3>Interactive 19.1: Linear decision boundary</h3> <p>Adjust the weights and bias. Points are classified by the sign of <code>w₁x₁+w₂x₂+b</code>.</p> <label>w₁: <span id="ch19-w1-value">1.00</span></label> <input id="ch19-w1" type="range" min="-3" max="3" step="0.05" value="1" style="width:100%;"> <label>w₂: <span id="ch19-w2-value">-1.00</span></label> <input id="ch19-w2" type="range" min="-3" max="3" step="0.05" value="-1" style="width:100%;"> <label>b: <span id="ch19-b-value">0.00</span></label> <input id="ch19-b" type="range" min="-3" max="3" step="0.05" value="0" style="width:100%;"> <div id="ch19-linear-boundary-plot" class="plotly-demo"></div> </div> <script> (function(){ function loadPlotly(callback){ if(window.Plotly){ callback(); return; } var s=document.createElement('script'); s.src='https://cdn.plot.ly/plotly-2.35.2.min.js'; s.onload=callback; document.head.appendChild(s); } var xs=[], ys=[], trueLabels=[]; for(var i=0;i<80;i++){ var t=i/80*2*Math.PI; var r=0.75+0.25*Math.sin(3*t); xs.push(-1.1+r*Math.cos(t)+0.15*Math.sin(7*t)); ys.push(-0.7+r*Math.sin(t)); trueLabels.push(0); } for(var j=0;j<80;j++){ var u=j/80*2*Math.PI; var r2=0.75+0.25*Math.cos(4*u); xs.push(1.1+r2*Math.cos(u)+0.1*Math.cos(9*u)); ys.push(0.7+r2*Math.sin(u)); trueLabels.push(1); } function get(id){return parseFloat(document.getElementById(id).value);} function draw(){ if(!window.Plotly) return; var w1=get('ch19-w1'), w2=get('ch19-w2'), b=get('ch19-b'); document.getElementById('ch19-w1-value').textContent=w1.toFixed(2); document.getElementById('ch19-w2-value').textContent=w2.toFixed(2); document.getElementById('ch19-b-value').textContent=b.toFixed(2); var x0=[], y0=[], x1=[], y1=[]; for(var i=0;i<xs.length;i++){ var score=w1*xs[i]+w2*ys[i]+b; if(score>=0){x1.push(xs[i]);y1.push(ys[i]);} else {x0.push(xs[i]);y0.push(ys[i]);} } var gridX=[], lineY=[]; for(var k=0;k<=100;k++){ var xx=-3+6*k/100; gridX.push(xx); lineY.push(Math.abs(w2)<0.05 ? null : -(w1*xx+b)/w2); } var z=[]; var gx=[]; var gy=[]; for(var a=0;a<=50;a++){gx.push(-3+6*a/50);} for(var c=0;c<=50;c++){gy.push(-3+6*c/50);} for(var yy of gy){ var row=[]; for(var xx of gx){ row.push(w1*xx+w2*yy+b); } z.push(row); } var data=[ {x:gx,y:gy,z:z,type:'contour',showscale:false,colorscale:'RdBu',opacity:0.28,contours:{start:0,end:0,size:1,showlines:false}}, {x:x0,y:y0,mode:'markers',type:'scatter',name:'predicted class 0',marker:{size:7,opacity:0.75}}, {x:x1,y:y1,mode:'markers',type:'scatter',name:'predicted class 1',marker:{size:7,opacity:0.75}}, {x:gridX,y:lineY,mode:'lines',type:'scatter',name:'boundary',line:{width:3}} ]; var layout={margin:{t:40,r:20,b:45,l:50},title:'Linear score sign in feature space',xaxis:{title:'feature x₁',range:[-3,3]},yaxis:{title:'feature x₂',range:[-3,3],scaleanchor:'x'},legend:{orientation:'h'}}; Plotly.react('ch19-linear-boundary-plot',data,layout,{responsive:true,displayModeBar:false}); } function init(){['ch19-w1','ch19-w2','ch19-b'].forEach(function(id){var e=document.getElementById(id); if(e){e.addEventListener('input',draw);}}); draw();} if(document.readyState==='loading') document.addEventListener('DOMContentLoaded',function(){loadPlotly(init);}); else loadPlotly(init); })(); </script> ``` For images, a linear classifier in raw pixel space is usually too limited. But a linear classifier on a good feature representation can be powerful. This is the logic behind linear probing of deep features. ## 19.5 Binary Logistic Regression Linear classification gives a score, but statistical classification often wants a probability. Binary logistic regression models $$ P(Y=1|X=x)=\sigma(w^Tx+b), $$ where $$ \sigma(t)=\frac{1}{1+e^{-t}}. $$ The log-odds are linear: $$ \log\frac{P(Y=1|X=x)}{P(Y=0|X=x)}=w^Tx+b. $$ Thus logistic regression is a linear model for the log-odds. ### Bernoulli likelihood Given data $(x_i,y_i)$ with $y_i\in\{0,1\}$, define $$ p_i=\sigma(w^Tx_i+b). $$ The conditional likelihood is $$ L(w,b)=\prod_{i=1}^n p_i^{y_i}(1-p_i)^{1-y_i}. $$ The negative log-likelihood is $$ \ell(w,b) = -\sum_{i=1}^n \left[y_i\log p_i+(1-y_i)\log(1-p_i)\right]. $$ This is binary cross-entropy. If we use labels $y_i\in\{-1,+1\}$ and score $s_i=w^Tx_i+b$, the loss can be written compactly as $$ \ell_i(w,b)=\log(1+e^{-y_is_i}). $$ The quantity $m_i=y_is_i$ is the margin. Correct and confident predictions have large positive margins. ### Gradient and Hessian Let $X\in\mathbb{R}^{n\times d}$ be the design matrix, let $p=(p_1,\ldots,p_n)^T$, and use an intercept column if desired. Then the gradient is $$ \nabla_w \ell=X^T(p-y). $$ The Hessian is $$ \nabla_w^2\ell=X^TWX, $$ where $$ W=\operatorname{diag}(p_i(1-p_i)). $$ Because $W$ is positive semidefinite, the negative log-likelihood is convex in $w$. This convexity is a major reason logistic regression is mathematically clean. ### Regularized logistic regression In high-dimensional image problems, regularization is essential. Ridge-regularized logistic regression solves $$ \min_{w,b} \frac{1}{n}\sum_{i=1}^n \log(1+e^{-y_i(w^Tx_i+b)}) + \lambda\|w\|_2^2. $$ The regularization term shrinks weights and improves stability. Bayesianly, it corresponds to a Gaussian prior $$ w\sim \mathcal{N}(0,\tau^2I). $$ An $\ell_1$ penalty, $$ \lambda\|w\|_1, $$ encourages sparsity and can be useful when only a small subset of features is relevant. ## Interactive 19.2: Margin, Sigmoid, and Logistic Loss The logistic loss is small for large positive margins and approximately linear for large negative margins. This behavior makes it smoother than hinge loss but still strongly penalizes confident mistakes. ```{=html} <div class="interactive-card"> <h3>Interactive 19.2: Logistic loss and margin</h3> <p>The margin is <code>m = y(wᵀx+b)</code>. Positive margins are correct; negative margins are incorrect.</p> <label>highlighted margin m: <span id="ch19-margin-value">0.00</span></label> <input id="ch19-margin" type="range" min="-6" max="6" step="0.05" value="0" style="width:100%;"> <div id="ch19-logistic-loss-plot" class="plotly-demo"></div> </div> <script> (function(){ function loadPlotly(callback){ if(window.Plotly){ callback(); return; } var s=document.createElement('script'); s.src='https://cdn.plot.ly/plotly-2.35.2.min.js'; s.onload=callback; document.head.appendChild(s); } function logistic(z){return 1/(1+Math.exp(-z));} function draw(){ if(!window.Plotly) return; var m=parseFloat(document.getElementById('ch19-margin').value); document.getElementById('ch19-margin-value').textContent=m.toFixed(2); var xs=[], logloss=[], hinge=[], prob=[]; for(var i=0;i<=300;i++){ var x=-6+12*i/300; xs.push(x); logloss.push(Math.log(1+Math.exp(-x))); hinge.push(Math.max(0,1-x)); prob.push(logistic(x)); } var data=[ {x:xs,y:logloss,type:'scatter',mode:'lines',name:'logistic loss log(1+exp(-m))'}, {x:xs,y:hinge,type:'scatter',mode:'lines',name:'hinge loss max(0,1-m)'}, {x:xs,y:prob,type:'scatter',mode:'lines',name:'P(correct)=σ(m)',yaxis:'y2'}, {x:[m],y:[Math.log(1+Math.exp(-m))],type:'scatter',mode:'markers',name:'selected margin',marker:{size:12}} ]; var layout={margin:{t:45,r:55,b:45,l:55},title:'Margin-based classification losses',xaxis:{title:'margin m'},yaxis:{title:'loss'},yaxis2:{title:'probability',overlaying:'y',side:'right',range:[0,1]},legend:{orientation:'h'}}; Plotly.react('ch19-logistic-loss-plot',data,layout,{responsive:true,displayModeBar:false}); } function init(){var s=document.getElementById('ch19-margin'); if(s){s.addEventListener('input',draw);} draw();} if(document.readyState==='loading') document.addEventListener('DOMContentLoaded',function(){loadPlotly(init);}); else loadPlotly(init); })(); </script> ``` ## 19.6 Multiclass Softmax Regression For $K$ classes, softmax regression assigns each class a logit $$ z_k=w_k^Tx+b_k, \qquad k=1,\ldots,K. $$ The predicted probability is $$ p_k(x)=P(Y=k|X=x) = \frac{e^{z_k}}{\sum_{j=1}^K e^{z_j}}. $$ The predicted class is $$ \widehat y=\arg\max_k p_k(x)=\arg\max_k z_k. $$ The softmax probabilities change if all logits are scaled, but the predicted class does not. This is important for calibration. ### Cross-entropy loss If $y_i$ is encoded as a one-hot vector, the multiclass cross-entropy loss is $$ \ell_i=-\sum_{k=1}^K y_{ik}\log p_{ik}. $$ Since only one component of $y_i$ equals $1$, this is $$ \ell_i=-\log p_{i,y_i}. $$ The gradient with respect to the logit $z_{ik}$ is especially simple: $$ \frac{\partial \ell_i}{\partial z_{ik}}=p_{ik}-y_{ik}. $$ This formula is one reason softmax cross-entropy is the standard output loss for neural classifiers. ### Temperature scaling For a temperature $T>0$, define $$ p_k^{(T)}(x)= \frac{e^{z_k/T}}{\sum_{j=1}^K e^{z_j/T}}. $$ Small $T$ makes probabilities sharper. Large $T$ makes them flatter. Temperature scaling is often used for probability calibration. ## Interactive 19.3: Softmax Temperature and Confidence This demo shows how the same logits can lead to different probability confidence depending on temperature. ```{=html} <div class="interactive-card"> <h3>Interactive 19.3: Softmax temperature</h3> <p>The predicted class is unchanged by temperature, but the probabilities become sharper or flatter.</p> <label>temperature T: <span id="ch19-temp-value">1.00</span></label> <input id="ch19-temp" type="range" min="0.2" max="5" step="0.05" value="1" style="width:100%;"> <div id="ch19-softmax-temp-plot" class="plotly-demo"></div> </div> <script> (function(){ function loadPlotly(callback){ if(window.Plotly){ callback(); return; } var s=document.createElement('script'); s.src='https://cdn.plot.ly/plotly-2.35.2.min.js'; s.onload=callback; document.head.appendChild(s); } var logits=[2.4,1.3,0.4,-0.5,-1.1]; var labels=['cat','dog','car','tree','road']; function softmax(T){ var ex=logits.map(function(z){return Math.exp(z/T);}); var s=ex.reduce(function(a,b){return a+b;},0); return ex.map(function(v){return v/s;}); } function entropy(p){return -p.reduce(function(a,b){return a+(b>0?b*Math.log(b):0);},0);} function draw(){ if(!window.Plotly) return; var T=parseFloat(document.getElementById('ch19-temp').value); document.getElementById('ch19-temp-value').textContent=T.toFixed(2); var p=softmax(T); var data=[ {x:labels,y:logits,type:'bar',name:'logits',xaxis:'x',yaxis:'y'}, {x:labels,y:p,type:'bar',name:'softmax probability',xaxis:'x2',yaxis:'y2'} ]; var layout={margin:{t:60,r:20,b:55,l:55},title:'entropy = '+entropy(p).toFixed(3),grid:{rows:1,columns:2,pattern:'independent'},yaxis:{title:'logit'},yaxis2:{title:'probability',range:[0,1]}}; Plotly.react('ch19-softmax-temp-plot',data,layout,{responsive:true,displayModeBar:false}); } function init(){var s=document.getElementById('ch19-temp'); if(s){s.addEventListener('input',draw);} draw();} if(document.readyState==='loading') document.addEventListener('DOMContentLoaded',function(){loadPlotly(init);}); else loadPlotly(init); })(); </script> ``` Softmax probabilities should not be interpreted blindly as calibrated probabilities. A model can be highly accurate but poorly calibrated, meaning its stated probabilities do not match empirical frequencies. ## 19.7 Gaussian Discriminant Analysis: LDA and QDA Generative Gaussian classifiers are mathematically useful because they show how decision boundaries arise from probability models. Assume $$ X|Y=k\sim \mathcal{N}(\mu_k,\Sigma_k), \qquad P(Y=k)=\pi_k. $$ The Bayes classifier chooses the class maximizing $$ \log \pi_k -\frac{1}{2}\log\det\Sigma_k - \frac{1}{2}(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k). $$ ### Linear discriminant analysis If all classes share the same covariance matrix, $$ \Sigma_k=\Sigma, $$ then terms quadratic in $x$ cancel across classes. The discriminant function is linear: $$ \delta_k(x)=x^T\Sigma^{-1}\mu_k - \frac{1}{2}\mu_k^T\Sigma^{-1}\mu_k + \log\pi_k. $$ This is linear discriminant analysis, or LDA. For two classes, the boundary is a hyperplane. The normal vector is proportional to $$ \Sigma^{-1}(\mu_1-\mu_0). $$ Thus LDA does not only compare class means. It compares means after whitening by the covariance. ### Quadratic discriminant analysis If each class has its own covariance $\Sigma_k$, the quadratic terms do not cancel. The decision boundary is quadratic. This is quadratic discriminant analysis, or QDA. In image classification, direct LDA or QDA on raw pixels is usually unstable because $p$ is huge and covariance estimation is difficult. But LDA/QDA ideas remain important after dimensionality reduction or feature extraction. ## Interactive 19.4: LDA versus QDA Geometry This demo compares shared-covariance and class-specific covariance models. LDA produces a linear boundary; QDA can curve. ```{=html} <div class="interactive-card"> <h3>Interactive 19.4: LDA and QDA decision geometry</h3> <p>Change the covariance ratio of class 1. When covariances are equal, the Bayes boundary is linear; when they differ, it becomes quadratic.</p> <label>class 1 covariance scale: <span id="ch19-covscale-value">1.00</span></label> <input id="ch19-covscale" type="range" min="0.4" max="2.8" step="0.05" value="1" style="width:100%;"> <div id="ch19-lda-qda-plot" class="plotly-demo"></div> </div> <script> (function(){ function loadPlotly(callback){ if(window.Plotly){ callback(); return; } var s=document.createElement('script'); s.src='https://cdn.plot.ly/plotly-2.35.2.min.js'; s.onload=callback; document.head.appendChild(s); } function quadForm(x,y,mu,cov){ var a=cov[0][0], b=cov[0][1], d=cov[1][1]; var det=a*d-b*b; var dx=x-mu[0], dy=y-mu[1]; return (d*dx*dx-2*b*dx*dy+a*dy*dy)/det; } function logdet(cov){return Math.log(cov[0][0]*cov[1][1]-cov[0][1]*cov[1][0]);} function draw(){ if(!window.Plotly) return; var scale=parseFloat(document.getElementById('ch19-covscale').value); document.getElementById('ch19-covscale-value').textContent=scale.toFixed(2); var mu0=[-1.0,-0.4], mu1=[1.0,0.5]; var S0=[[1.0,0.35],[0.35,0.8]]; var S1=[[scale*0.7,0.15*scale],[0.15*scale,scale*1.25]]; var xs=[], ys=[], zlda=[], zqda=[]; for(var i=0;i<=80;i++){xs.push(-4+8*i/80); ys.push(-4+8*i/80);} for(var j=0;j<ys.length;j++){ var rowL=[], rowQ=[]; for(var i=0;i<xs.length;i++){ var x=xs[i], y=ys[j]; var q0=quadForm(x,y,mu0,S0), q1same=quadForm(x,y,mu1,S0); var q1=quadForm(x,y,mu1,S1); rowL.push(-0.5*q1same+0.5*q0); rowQ.push(-0.5*q1-0.5*logdet(S1)+0.5*q0+0.5*logdet(S0)); } zlda.push(rowL); zqda.push(rowQ); } var theta=[], e0x=[], e0y=[], e1x=[], e1y=[]; for(var t=0;t<=120;t++){ var u=2*Math.PI*t/120; theta.push(u); e0x.push(mu0[0]+Math.cos(u)); e0y.push(mu0[1]+Math.sin(u)); e1x.push(mu1[0]+Math.sqrt(scale)*Math.cos(u)); e1y.push(mu1[1]+Math.sqrt(scale)*Math.sin(u)); } var data=[ {x:xs,y:ys,z:zlda,type:'contour',name:'LDA boundary',showscale:false,contours:{start:0,end:0,size:1,coloring:'lines'},line:{width:3}}, {x:xs,y:ys,z:zqda,type:'contour',name:'QDA boundary',showscale:false,contours:{start:0,end:0,size:1,coloring:'lines'},line:{width:3,dash:'dash'}}, {x:[mu0[0]],y:[mu0[1]],mode:'markers',type:'scatter',name:'class 0 mean',marker:{size:12}}, {x:[mu1[0]],y:[mu1[1]],mode:'markers',type:'scatter',name:'class 1 mean',marker:{size:12}}, {x:e0x,y:e0y,mode:'lines',type:'scatter',name:'class 0 scale ellipse'}, {x:e1x,y:e1y,mode:'lines',type:'scatter',name:'class 1 scale ellipse'} ]; var layout={margin:{t:45,r:20,b:45,l:50},title:'LDA boundary versus QDA boundary',xaxis:{range:[-4,4],title:'feature 1'},yaxis:{range:[-4,4],scaleanchor:'x',title:'feature 2'},legend:{orientation:'h'}}; Plotly.react('ch19-lda-qda-plot',data,layout,{responsive:true,displayModeBar:false}); } function init(){var s=document.getElementById('ch19-covscale'); if(s){s.addEventListener('input',draw);} draw();} if(document.readyState==='loading') document.addEventListener('DOMContentLoaded',function(){loadPlotly(init);}); else loadPlotly(init); })(); </script> ``` ## 19.8 Support Vector Machines and Margin-Based Classification Support vector machines classify using margins. For labels $y_i\in\{-1,+1\}$ and scores $s_i=w^Tx_i+b$, the hard-margin SVM solves $$ \min_{w,b}\frac{1}{2}\|w\|_2^2 $$ subject to $$ y_i(w^Tx_i+b)\ge 1, \qquad i=1,\ldots,n. $$ The geometric margin is $$ \frac{1}{\|w\|_2}. $$ Thus minimizing $\|w\|_2^2$ maximizes the margin. For nonseparable data, the soft-margin SVM solves $$ \min_{w,b} \frac{1}{n}\sum_{i=1}^n \max(0,1-y_i(w^Tx_i+b)) + \lambda\|w\|_2^2. $$ The loss $$ \max(0,1-m) $$ is the hinge loss. It ignores points with margin at least $1$ and penalizes points inside the margin or on the wrong side of the boundary. ### Kernel idea A kernel classifier replaces dot products by a kernel $$ K(x,z)=\langle \phi(x),\phi(z)\rangle. $$ This allows linear classification in a high-dimensional feature space without explicitly computing $\phi$. Kernel methods were central in classical vision before large-scale deep learning became dominant. In modern vision, neural networks learn the representation $\phi_\theta$ directly from data. ## 19.9 Feature Maps for Image Classification The quality of classification depends heavily on the feature map. ### Raw pixel features Raw pixel vectors are simple: $$ \phi(x)=\operatorname{vec}(x). $$ This is useful for small aligned images, such as centered handwritten digits. It is fragile when objects translate, rotate, deform, or change illumination. ### Hand-designed features Classical features include: - edge histograms; - color histograms; - texture features; - HOG features; - SIFT-like local descriptors; - bag-of-visual-words representations. These features encode prior knowledge about images. For example, gradient-orientation histograms are less sensitive to uniform brightness changes than raw pixels. ### Learned features Modern models learn $$ \phi_\theta(x) $$ from data. A classifier may use $$ f_{\theta,W}(x)=W\phi_\theta(x)+b. $$ The representation $\phi_\theta$ is often a CNN backbone, a vision transformer, or a pretrained foundation model. A linear classifier on top of frozen features is called a linear probe. ### Geometry of features A good feature map should make same-class examples close and different-class examples separated. For class means $$ \bar z_k=\frac{1}{n_k}\sum_{i:y_i=k}\phi(x_i), $$ a simple measure of within-class scatter is $$ S_W=\sum_{k=1}^K\sum_{i:y_i=k}(z_i-\bar z_k)(z_i-\bar z_k)^T. $$ A measure of between-class scatter is $$ S_B=\sum_{k=1}^K n_k(\bar z_k-\bar z)(\bar z_k-\bar z)^T. $$ A representation is more linearly classifiable when between-class variation dominates within-class variation. ## 19.10 Empirical Risk, Generalization, and Regularization In practice, the true distribution of $(X,Y)$ is unknown. We observe training data $$ \mathcal{D}_n=\{(x_i,y_i)\}_{i=1}^n. $$ The empirical risk is $$ \widehat R_n(f)=\frac{1}{n}\sum_{i=1}^n \ell(y_i,f(x_i)). $$ Training chooses a model that approximately minimizes empirical risk plus regularization: $$ \widehat f \in \arg\min_{f\in\mathcal{F}} \widehat R_n(f)+\lambda\Omega(f). $$ The central statistical question is not only whether $\widehat R_n$ is small, but whether the population risk $$ R(f)=\mathbb{E}[\ell(Y,f(X))] $$ is small. ### Bias-variance and representation If the classifier class is too simple, it has high bias. If it is too flexible relative to data, it may overfit. In modern deep learning, this classical description is incomplete, but still useful. Regularization, augmentation, architecture, pretraining, and early stopping all influence generalization. ### Data augmentation as invariance Many image labels are invariant under transformations. If $g$ is a small translation, rotation, crop, or color perturbation, often $$ Y(x)=Y(g\cdot x). $$ Data augmentation encourages the model to satisfy $$ f_\theta(x)\approx f_\theta(g\cdot x). $$ This is a practical way to inject geometric prior knowledge into statistical learning. ## 19.11 Class Imbalance and Cost-Sensitive Learning Vision datasets often have class imbalance. In medical diagnosis, positive cases may be rare. In defect detection, anomalies may be rare. In ecological or satellite data, some categories may dominate. If class $k$ has $n_k$ examples, a weighted cross-entropy may use weights $$ \alpha_k\propto \frac{1}{n_k}. $$ The weighted loss is $$ \ell_i=-\alpha_{y_i}\log p_{i,y_i}. $$ This changes the training objective so that rare classes have more influence. ### Focal loss For difficult imbalanced detection problems, focal loss modifies cross-entropy: $$ \ell_i=-(1-p_{i,y_i})^\gamma\log p_{i,y_i}. $$ When $p_{i,y_i}$ is large, the factor $(1-p_{i,y_i})^\gamma$ downweights easy examples. This idea is especially important in object detection, where background examples can dominate foreground examples. ### Thresholding under imbalance If a model estimates $P(Y=1|X=x)$ accurately, the classification threshold should be chosen according to costs and deployment goals, not automatically set to $0.5$. ## 19.12 Evaluation Metrics for Classification For binary classification, predictions can be summarized by a confusion matrix. | | Predicted positive | Predicted negative | |---|---:|---:| | True positive | TP | FN | | True negative | FP | TN | Common metrics are: $$ \text{Accuracy}=\frac{TP+TN}{TP+FP+FN+TN}, $$ $$ \text{Precision}=\frac{TP}{TP+FP}, $$ $$ \text{Recall}=\frac{TP}{TP+FN}, $$ $$ \text{Specificity}=\frac{TN}{TN+FP}, $$ and $$ F_1=\frac{2\cdot \text{Precision}\cdot\text{Recall}}{\text{Precision}+\text{Recall}}. $$ Accuracy can be misleading under class imbalance. For example, if only $1\%$ of images are positive, a classifier that always predicts negative has $99\%$ accuracy but zero recall. ### ROC and precision-recall curves A probabilistic classifier outputs a score $s(x)$ or probability $p(x)$. Varying the threshold creates different values of true-positive rate and false-positive rate. The ROC curve plots $$ \text{TPR}=\frac{TP}{TP+FN} $$ against $$ \text{FPR}=\frac{FP}{FP+TN}. $$ A precision-recall curve is often more informative for rare-positive problems. ### Calibration A classifier is calibrated if, among examples assigned confidence $0.8$, approximately $80\%$ are correct. Calibration matters whenever probabilities are used for decision-making. The Brier score for binary classification is $$ \frac{1}{n}\sum_{i=1}^n (p_i-y_i)^2. $$ Expected calibration error approximates calibration by binning predictions by confidence. ## Interactive 19.5: Threshold, Confusion Matrix, and Metrics This demo shows how changing the threshold changes precision, recall, specificity, and F1 score. ```{=html} <div class="interactive-card"> <h3>Interactive 19.5: Threshold-dependent evaluation</h3> <p>A fixed score model can produce many classifiers, depending on the threshold.</p> <label>classification threshold: <span id="ch19-threshold-value">0.50</span></label> <input id="ch19-threshold" type="range" min="0" max="1" step="0.01" value="0.5" style="width:100%;"> <div id="ch19-threshold-plot" class="plotly-demo"></div> </div> <script> (function(){ function loadPlotly(callback){ if(window.Plotly){ callback(); return; } var s=document.createElement('script'); s.src='https://cdn.plot.ly/plotly-2.35.2.min.js'; s.onload=callback; document.head.appendChild(s); } var pos=[], neg=[]; for(var i=0;i<100;i++){ var u=(i+0.5)/100; pos.push(Math.min(0.995,Math.max(0.005,0.25+0.72*Math.pow(u,0.45)+0.06*Math.sin(17*u)))); } for(var j=0;j<300;j++){ var v=(j+0.5)/300; neg.push(Math.min(0.995,Math.max(0.005,0.02+0.72*Math.pow(v,1.9)+0.05*Math.sin(21*v)))); } function metrics(th){ var TP=0,FN=0,FP=0,TN=0; pos.forEach(function(s){if(s>=th) TP++; else FN++;}); neg.forEach(function(s){if(s>=th) FP++; else TN++;}); var prec=TP+FP>0?TP/(TP+FP):0; var rec=TP+FN>0?TP/(TP+FN):0; var spec=TN+FP>0?TN/(TN+FP):0; var f1=prec+rec>0?2*prec*rec/(prec+rec):0; var acc=(TP+TN)/(TP+TN+FP+FN); return {TP:TP,FN:FN,FP:FP,TN:TN,precision:prec,recall:rec,specificity:spec,f1:f1,accuracy:acc}; } function draw(){ if(!window.Plotly) return; var th=parseFloat(document.getElementById('ch19-threshold').value); document.getElementById('ch19-threshold-value').textContent=th.toFixed(2); var m=metrics(th); var data=[ {x:neg,type:'histogram',name:'negative scores',opacity:0.65,xbins:{start:0,end:1,size:0.04},xaxis:'x',yaxis:'y'}, {x:pos,type:'histogram',name:'positive scores',opacity:0.65,xbins:{start:0,end:1,size:0.04},xaxis:'x',yaxis:'y'}, {x:[th,th],y:[0,35],mode:'lines',type:'scatter',name:'threshold',xaxis:'x',yaxis:'y'}, {x:['precision','recall','specificity','F1','accuracy'],y:[m.precision,m.recall,m.specificity,m.f1,m.accuracy],type:'bar',name:'metrics',xaxis:'x2',yaxis:'y2'}, {z:[[m.TP,m.FN],[m.FP,m.TN]],x:['pred +','pred -'],y:['true +','true -'],type:'heatmap',showscale:false,text:[[m.TP,m.FN],[m.FP,m.TN]],texttemplate:'%{text}',xaxis:'x3',yaxis:'y3',name:'confusion'} ]; var layout={margin:{t:65,r:20,b:60,l:55},title:'TP='+m.TP+', FP='+m.FP+', FN='+m.FN+', TN='+m.TN,grid:{rows:1,columns:3,pattern:'independent'},barmode:'overlay',xaxis:{title:'score'},yaxis:{title:'count'},yaxis2:{range:[0,1],title:'metric'},xaxis3:{title:'prediction'},yaxis3:{title:'truth'},legend:{orientation:'h'}}; Plotly.react('ch19-threshold-plot',data,layout,{responsive:true,displayModeBar:false}); } function init(){var s=document.getElementById('ch19-threshold'); if(s){s.addEventListener('input',draw);} draw();} if(document.readyState==='loading') document.addEventListener('DOMContentLoaded',function(){loadPlotly(init);}); else loadPlotly(init); })(); </script> ``` ## 19.13 Neural Image Classifiers A modern neural image classifier can be written as $$ z=f_\theta(x)\in\mathbb{R}^K, $$ where $z$ is the vector of logits. The predicted probabilities are $$ p=\operatorname{softmax}(z). $$ The network is trained by minimizing average cross-entropy: $$ \min_\theta \frac{1}{n}\sum_{i=1}^n - \log \operatorname{softmax}(f_\theta(x_i))_{y_i}. $$ A CNN classifier uses convolutional layers to build local-to-global representations. A vision transformer classifier divides the image into patches, embeds those patches as tokens, applies self-attention, and then classifies from a pooled or special token. ### Why deep classifiers work well for images Images have structure that generic high-dimensional vectors do not have: - local spatial correlation; - translation and deformation structure; - compositional objects made of parts; - multi-scale patterns; - repeated visual motifs; - semantic labels that are often invariant under many transformations. CNNs encode locality and translation equivariance. Transformers can model long-range interactions and learn global context. Pretrained foundation models learn representations from very large datasets and can be adapted to smaller classification tasks. ### Linear head on a learned representation Many practical vision classifiers use $$ f_{\theta,W}(x)=W\phi_\theta(x)+b. $$ Here $\phi_\theta$ is the backbone and $W,b$ form the classification head. If $\phi_\theta$ is fixed and only $W,b$ are trained, the problem is softmax regression on learned features. If both $\theta$ and $W,b$ are trained, this is end-to-end representation learning. ## 19.14 Robustness, Shift, and Uncertainty High test accuracy on a benchmark does not guarantee safe deployment. Classification models may fail under several types of shift. ### Covariate shift The image distribution changes: $$ p_{\text{train}}(x)\ne p_{\text{test}}(x), $$ but the conditional label rule may remain similar. Examples include new lighting, new camera hardware, different hospitals, different satellites, or different image preprocessing. ### Label shift Class priors change: $$ P_{\text{train}}(Y=k)\ne P_{\text{test}}(Y=k). $$ This is common in medical screening and anomaly detection. ### Concept shift The conditional distribution changes: $$ P_{\text{train}}(Y|X)\ne P_{\text{test}}(Y|X). $$ This is more serious. It means the meaning of visual evidence has changed. ### Adversarial perturbations A small perturbation $r$ may change the predicted class even when $\|r\|$ is small: $$ \delta(x+r)\ne\delta(x). $$ This is possible because high-dimensional decision boundaries can be close to many data points. Robust classification studies objectives such as $$ \min_\theta \frac{1}{n}\sum_{i=1}^n \max_{\|r\|\le\epsilon} \ell(y_i,f_\theta(x_i+r)). $$ ### Out-of-distribution inputs A classifier trained on natural object images may still produce confident predictions for medical images, random noise, or drawings. This is a warning: softmax confidence is not the same as epistemic uncertainty. ## 19.15 Python Example: Logistic Regression on Image Features The following example uses simple synthetic image features. It is intentionally small enough for students to understand before moving to CNNs. ```{python} #| label: py-ch19-logistic-image-features #| eval: false import numpy as np import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression from sklearn.pipeline import Pipeline from sklearn.metrics import classification_report, ConfusionMatrixDisplay rng = np.random.default_rng(7243) # Synthetic 16 x 16 grayscale images. # Class 0: vertical bright bar. Class 1: horizontal bright bar. n = 600 H = W = 16 X_img = np.zeros((n, H, W)) y = np.zeros(n, dtype=int) for i in range(n): img = 0.15 * rng.normal(size=(H, W)) if i < n // 2: c = rng.integers(5, 11) img[:, c-1:c+2] += 1.0 y[i] = 0 else: r = rng.integers(5, 11) img[r-1:r+2, :] += 1.0 y[i] = 1 X_img[i] = img X = X_img.reshape(n, -1) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, random_state=0, stratify=y ) model = Pipeline([ ("scale", StandardScaler()), ("logreg", LogisticRegression(C=1.0, max_iter=2000)) ]) model.fit(X_train, y_train) y_pred = model.predict(X_test) print(classification_report(y_test, y_pred, target_names=["vertical", "horizontal"])) ConfusionMatrixDisplay.from_predictions(y_test, y_pred) plt.show() # Inspect the learned weight image. w = model.named_steps["logreg"].coef_[0].reshape(H, W) plt.imshow(w, cmap="coolwarm") plt.colorbar() plt.title("learned logistic-regression weight image") plt.show() ``` The learned weight image is mathematically meaningful. Positive pixels support one class; negative pixels support the other. This is a rare case where a linear classifier on raw pixels is interpretable because the synthetic images are aligned. ## 19.16 Python Example: Softmax Regression with PCA Features For higher-dimensional images, PCA can reduce dimension before classification. ```{python} #| label: py-ch19-pca-softmax #| eval: false from sklearn.decomposition import PCA from sklearn.linear_model import LogisticRegression from sklearn.pipeline import Pipeline from sklearn.metrics import accuracy_score clf = Pipeline([ ("scale", StandardScaler()), ("pca", PCA(n_components=20, random_state=0)), ("softmax", LogisticRegression( multi_class="multinomial", solver="lbfgs", max_iter=3000, C=2.0 )) ]) clf.fit(X_train, y_train) prob = clf.predict_proba(X_test) pred = prob.argmax(axis=1) print("test accuracy:", accuracy_score(y_test, pred)) print("first five probability vectors:") print(prob[:5]) ``` For real images, PCA features are usually weaker than CNN or transformer features, but this pipeline is mathematically transparent. It combines linear algebra, dimension reduction, likelihood-based classification, and statistical evaluation. ## 19.17 TensorFlow Example: A Small CNN Classifier The following example gives a minimal CNN classifier. It is meant as a template, not as a large benchmark model. ```{python} #| label: py-ch19-tensorflow-cnn-classifier #| eval: false import tensorflow as tf from tensorflow import keras from tensorflow.keras import layers num_classes = 2 cnn = keras.Sequential([ layers.Input(shape=(16, 16, 1)), layers.Conv2D(16, kernel_size=3, padding="same", activation="relu"), layers.MaxPooling2D(pool_size=2), layers.Conv2D(32, kernel_size=3, padding="same", activation="relu"), layers.GlobalAveragePooling2D(), layers.Dense(num_classes) # logits, not probabilities ]) cnn.compile( optimizer=keras.optimizers.Adam(learning_rate=1e-3), loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=["accuracy"] ) X_train_img = X_train.reshape(-1, 16, 16, 1).astype("float32") X_test_img = X_test.reshape(-1, 16, 16, 1).astype("float32") history = cnn.fit( X_train_img, y_train, validation_split=0.2, epochs=10, batch_size=32, verbose=1 ) logits = cnn.predict(X_test_img) prob = tf.nn.softmax(logits, axis=1).numpy() print(prob[:5]) ``` Notice the final dense layer returns logits. The softmax is included inside the loss during training by setting `from_logits=True`. This is numerically more stable than applying softmax first and then taking logarithms. ## 19.18 AI-Assisted Study Components AI tools can help students learn classification, but they should be used as mathematical assistants rather than answer generators. ::: {.ai-box} **Prompt 1: derive the Bayes classifier** Ask an AI assistant: > Derive the Bayes classifier for zero-one loss. Then repeat the derivation when false negatives cost five times as much as false positives. Explain the threshold change. Then verify that the threshold formula agrees with the conditional-risk calculation in this chapter. ::: ::: {.ai-box} **Prompt 2: debug cross-entropy confusion** Ask: > I have logits from a neural network and integer class labels. Should I apply softmax before using cross-entropy? Explain the difference between logits, probabilities, and log-probabilities. Check whether the answer mentions numerical stability. ::: ::: {.ai-box} **Prompt 3: evaluate a classifier under imbalance** Ask: > A medical image classifier has 98% accuracy, but the positive class prevalence is 1%. What other metrics should I inspect and why? A good answer should discuss recall/sensitivity, specificity, precision, the confusion matrix, ROC/PR curves, calibration, and clinical decision thresholds. ::: ::: {.ai-box} **Prompt 4: representation geometry** Ask: > Explain why a linear classifier may fail on raw pixels but work well on pretrained CNN or vision-transformer embeddings. Use the language of feature maps and decision boundaries. ::: ## 19.19 Common Mistakes ::: {.warning-box} **Mistake 1: treating softmax confidence as truth.** A softmax probability is a model output. It may be poorly calibrated, especially under distribution shift. ::: ::: {.warning-box} **Mistake 2: using accuracy alone.** Accuracy can hide poor minority-class performance. Always inspect the confusion matrix and task-relevant metrics. ::: ::: {.warning-box} **Mistake 3: forgetting the feature map.** A classifier is not only its final loss. Its performance depends strongly on the representation $\phi(x)$. ::: ::: {.warning-box} **Mistake 4: training-test leakage through preprocessing.** Scalers, PCA, feature selection, and threshold choice should be fitted using training data or validation data, not test data. ::: ::: {.warning-box} **Mistake 5: assuming the $0.5$ threshold is always optimal.** The optimal threshold depends on class priors, costs, and deployment goals. ::: ## 19.20 Chapter Summary Classification is a decision problem under uncertainty. The Bayes classifier minimizes expected zero-one loss by choosing the most probable class given the image. Logistic regression and softmax regression model class probabilities through linear logits and are trained by cross-entropy. SVMs emphasize margins. Generative classifiers such as LDA and QDA show how probability models induce linear or quadratic boundaries. For computer vision, the most important practical issue is representation. Raw pixels are high-dimensional and fragile; feature maps, CNNs, transformers, and foundation models aim to transform images into spaces where simple decision rules work well. Evaluation must go beyond accuracy and include threshold-dependent metrics, calibration, class imbalance, and robustness under distribution shift. ## Exercises ### Conceptual Exercises 1. Explain why the Bayes classifier is optimal under zero-one loss. 2. Give an example where accuracy is misleading for image classification. 3. Explain the difference between a logit, a probability, and a predicted class. 4. Why can a linear classifier be weak on raw pixels but strong on learned features? 5. Describe the difference between generative and discriminative classification. 6. Explain why calibration matters in medical imaging. 7. What is the role of data augmentation in classification? 8. Why is class imbalance especially important in anomaly detection? ### Mathematical Exercises 1. Derive the binary Bayes classifier for zero-one loss. 2. Derive the optimal threshold when false negatives cost $c_{10}$ and false positives cost $c_{01}$. 3. Show that the logistic-regression negative log-likelihood is $$ \sum_{i=1}^n \log(1+e^{-y_i(w^Tx_i+b)}) $$ when $y_i\in\{-1,+1\}$. 4. Derive the gradient $X^T(p-y)$ for binary logistic regression. 5. Derive the softmax gradient $$ \frac{\partial \ell_i}{\partial z_{ik}}=p_{ik}-y_{ik}. $$ 6. For two Gaussian classes with common covariance $\Sigma$, derive the LDA decision boundary. 7. Show that if $\Sigma_0\ne\Sigma_1$, the Gaussian Bayes boundary is generally quadratic. 8. Prove that the signed distance from $x$ to the hyperplane $w^Tx+b=0$ is $(w^Tx+b)/\|w\|_2$. 9. Derive the hard-margin SVM margin $1/\|w\|_2$ under constraints $y_i(w^Tx_i+b)\ge 1$. 10. Show that temperature scaling changes softmax probabilities but not the argmax class. ### Python Exercises 1. Implement logistic regression from scratch with gradient descent on a synthetic image dataset. 2. Compare raw-pixel logistic regression with PCA plus logistic regression. 3. Generate two synthetic image classes and visualize the learned weight image. 4. Compute a confusion matrix and classification report for different thresholds. 5. Plot ROC and precision-recall curves for an imbalanced image classification problem. 6. Train a small CNN on synthetic shape images and compare it with logistic regression. 7. Use class weights in cross-entropy and measure recall on a rare class. 8. Apply temperature scaling on validation logits and inspect calibration before and after scaling. ### AI-Assisted Exercises 1. Ask an AI assistant to derive LDA from Gaussian class-conditional densities. Check every algebraic step. 2. Ask for a counterexample where high accuracy hides poor recall. Translate the example into a confusion matrix. 3. Ask the assistant to explain softmax cross-entropy to a statistics student. Improve the response by adding the gradient derivation. 4. Ask for possible distribution shifts in a medical image classifier. Categorize them as covariate shift, label shift, or concept shift. 5. Ask the assistant to generate a checklist for evaluating an image classifier before deployment. Add mathematical metrics to the checklist. ## References and Further Reading Good general references for statistical classification include @bishop2006pattern, @hastie2009elements, and @duda2001pattern. For margin methods, see @cortes1995support and @vapnik1998statistical. For deep learning and image classification, see @goodfellow2016deep, @lecun1998gradient, @krizhevsky2012imagenet, and @he2015resnet. For calibration and evaluation, see @platt1999probabilistic, @guo2017calibration, and @fawcett2006roc.