22 Chapter 21. Convolutional Neural Networks

Local Operators, Equivariance, Receptive Fields, and Learned Visual Representations

22.1 21.1 Why convolutional neural networks belong in a mathematical vision book

Computer vision has always used local image operations. In earlier chapters we studied convolution, Gaussian smoothing, derivative filters, edges, corners, optical flow, and local descriptors. A convolutional neural network, or CNN, is the modern statistical-learning version of the same idea:

\[ \text{local linear operator} \quad + \quad \text{nonlinear activation} \quad + \quad \text{composition}. \]

The mathematical importance of CNNs is not just that they work well. Their importance is that they encode several structural assumptions about images:

locality: nearby pixels usually interact more strongly than far-away pixels;
translation equivariance: if an object shifts, the feature map should shift in the same way;
parameter sharing: the same local detector can be used at every image location;
hierarchy: simple local patterns combine into larger semantic patterns;
stability: useful features should not change too much under small perturbations.

For MA Applied Math and MS Statistics students, CNNs are best understood as structured nonlinear maps

\[ f_\theta : \mathbb R^{H\times W\times C} \to \mathbb R^K \]

whose parameters are constrained by the geometry of the image lattice. Compared with a fully connected network on raw pixels, a CNN uses far fewer parameters and has a much stronger inductive bias.

Historically, convolutional networks were central in handwritten-digit recognition through LeNet-style architectures (LeCun et al. 1998). They became the dominant architecture for large-scale image recognition after AlexNet (Krizhevsky et al. 2012), and residual networks then showed that very deep CNNs could be optimized effectively (He et al. 2015). Even in the era of vision transformers, CNN ideas remain foundational: locality, equivariance, multi-scale features, pooling, residual connections, normalization, and convolutional feature maps are still used in detection, segmentation, medical imaging, robotics, and scientific vision.

22.1.1 Learning goals

After this chapter, you should be able to:

define convolutional and cross-correlation layers for single-channel and multi-channel images;
prove that convolutional layers are translation equivariant under suitable boundary conventions;
compute the number of parameters and output size of convolutional layers;
calculate receptive-field size across layers with stride, padding, and dilation;
explain pooling, local invariance, and downsampling from a mathematical viewpoint;
derive basic backpropagation formulas for convolutional kernels;
interpret CNNs as compositions of equivariant maps and pointwise nonlinearities;
connect CNNs to empirical risk minimization, regularization, sample complexity, and statistical bias;
explain residual connections, normalization layers, and modern CNN architectures;
implement small CNN examples in NumPy, scikit-learn style workflows, and TensorFlow.

Central idea

A convolutional network is a learned hierarchy of local filters:

\[ X^{(0)}=I, \qquad X^{(\ell+1)}=\sigma\!\left(K^{(\ell)} * X^{(\ell)}+b^{(\ell)}\right). \]

The mathematical power comes from the combination of linear local structure, nonlinear composition, and statistical learning from data.

22.2 21.2 Images as lattice-indexed tensors

Let a color image be represented as a tensor

\[ X\in \mathbb R^{H\times W\times C}. \]

We write

\[ X[i,j,c] \]

for the pixel value at row $i$, column $j$, and channel $c$. The set of spatial indices

\[ \Lambda = \{0,1,\ldots,H-1\}\times \{0,1,\ldots,W-1\} \]

is a finite two-dimensional lattice. A convolutional filter uses a small neighborhood around each location. For a kernel of spatial size $(2r+1)\times(2s+1)$, the local patch around $(i,j)$ is

\[ \mathcal P_{i,j}(X) = \{X[i+a,j+b,c] : -r\le a\le r,\ -s\le b\le s,\ 1\le c\le C\}. \]

A fully connected layer treats the image as an unstructured vector in $\mathbb R^{HWC}$. A convolutional layer treats the image as an object with spatial geometry.

22.2.1 Locality assumption

A local operator has the form

\[ Y[i,j] = \psi(\mathcal P_{i,j}(X)) \]

for some function $\psi$. If $\psi$ is linear, then

\[ Y[i,j] = \sum_{a=-r}^r\sum_{b=-s}^s\sum_{c=1}^C K[a,b,c]X[i+a,j+b,c]. \]

This is the basic algebraic form of a convolutional feature detector.

Statistical interpretation

Locality is a prior. It says that the response at a spatial location should depend mostly on a local neighborhood. This prior is often useful for natural images, medical images, satellite images, microscopy images, and video frames.

22.3 21.3 Convolution and cross-correlation

There is a small notational difference between mathematical convolution and the operation used in many deep learning libraries.

For one-dimensional signals, mathematical convolution is

\[ (K*X)[i]=\sum_a K[a]X[i-a]. \]

Cross-correlation is

\[ (K\star X)[i]=\sum_a K[a]X[i+a]. \]

In two dimensions,

\[ (K*X)[i,j] = \sum_a\sum_b K[a,b]X[i-a,j-b], \]

while

\[ (K\star X)[i,j] = \sum_a\sum_b K[a,b]X[i+a,j+b]. \]

Most neural-network libraries call cross-correlation convolution because the kernel is learned. If the kernel is learned, flipping the kernel is not a meaningful distinction: the network can learn either orientation.

22.3.1 Multi-channel convolution

Suppose the input has $C_{\text{in}}$ channels and the output has $C_{\text{out}}$ channels. A convolutional kernel is a four-dimensional tensor

\[ K\in \mathbb R^{k_h\times k_w\times C_{\text{in}}\times C_{\text{out}}}. \]

The output feature map is

\[ Y[i,j,q] = \sum_{a=0}^{k_h-1}\sum_{b=0}^{k_w-1}\sum_{c=1}^{C_{\text{in}}} K[a,b,c,q]X[i+a,j+b,c]+b[q]. \]

For each output channel $q$, the layer learns one bank of local filters across all input channels. The number of trainable parameters is

\[ \#\text{parameters}=k_hk_wC_{\text{in}}C_{\text{out}}+C_{\text{out}}. \]

This should be compared with a dense layer from $\mathbb R^{HWC}$ to $\mathbb R^{H'W'C_{\text{out}}}$, which would have roughly

\[ (HWC)(H'W'C_{\text{out}}) \]

weights. The reduction in parameter count is a major statistical advantage.

22.4 21.4 Translation equivariance

Let $T_\tau$ denote translation by a lattice vector $\tau=(\tau_1,\tau_2)$:

\[ (T_\tau X)[i,j] = X[i-\tau_1,j-\tau_2]. \]

An operator $F$ is translation equivariant if

\[ F(T_\tau X)=T_\tau F(X). \]

That is, translating the input translates the output.

22.4.1 Proposition 21.1: convolution is translation equivariant

Ignoring boundary effects, convolution satisfies

\[ K*(T_\tau X)=T_\tau(K*X). \]

Proof. For one-dimensional notation,

\[ \begin{aligned} (K*(T_\tau X))[i] &=\sum_a K[a](T_\tau X)[i-a] \\ &=\sum_a K[a]X[i-a-\tau] \\ &=(K*X)[i-\tau] \\ &=(T_\tau(K*X))[i]. \end{aligned} \]

The same argument works in two dimensions by replacing $i-a$ with $(i-a,j-b)$. $\square$

The proof is short, but the consequence is deep. CNNs are built from layers that respect the spatial symmetry of images.

22.4.2 Equivariance versus invariance

A feature map $F$ is equivariant if movement in the input produces corresponding movement in the output:

\[ F(T_\tau X)=T_\tau F(X). \]

A classifier $g$ is invariant if movement in the input does not change the final label:

\[ g(T_\tau X)=g(X). \]

A common CNN strategy is:

\[ \text{equivariant feature maps} \quad \longrightarrow \quad \text{pooling/global aggregation} \quad \longrightarrow \quad \text{approximately invariant prediction}. \]

22.5 21.5 Interactive demo: convolution and translation equivariance

The next demo shows a one-dimensional signal and a learned-looking local kernel. Moving the input impulse moves the convolution response. This is the simplest visualization of translation equivariance.

Convolution response shifts with the input

Move the input spike. The output response moves by the same amount.

Input shift:

22.6 21.6 A convolutional layer as an affine map plus nonlinearity

A basic convolutional layer is

\[ Z = K \star X + b, \]

followed by an activation function

\[ Y = \sigma(Z). \]

Common activation functions include

\[ \operatorname{ReLU}(t)=\max\{0,t\}, \]

\[ \sigma_{\text{sigmoid}}(t)=\frac{1}{1+e^{-t}}, \]

and

\[ \operatorname{GELU}(t)=t\Phi(t), \]

where $\Phi$ is the standard normal distribution function.

The activation is applied pointwise:

\[ Y[i,j,q]=\sigma(Z[i,j,q]). \]

22.6.1 Why nonlinearities are necessary

A composition of linear convolutional layers without nonlinearities is still a linear convolutional operator, up to boundary effects. Therefore depth alone is not enough. Nonlinearities allow the network to represent nonlinear decision boundaries, local pattern combinations, and hierarchical image features.

For example, the first layer may detect oriented edges, the next layer may detect corners or curves, and later layers may detect parts of objects.

22.7 21.7 Interactive demo: activation functions

Activation functions

Compare ReLU, sigmoid, tanh, and GELU-like nonlinearities.

22.8 21.8 Output size: padding, stride, and dilation

Let an input have height $H$ and width $W$. Consider a convolutional layer with kernel size $k_h\times k_w$, padding $p_h,p_w$, stride $s_h,s_w$, and dilation $d_h,d_w$.

The effective kernel size in height is

\[ k_h^{\text{eff}} = d_h(k_h-1)+1, \]

and similarly

\[ k_w^{\text{eff}} = d_w(k_w-1)+1. \]

The output height is

\[ H_{\text{out}} = \left\lfloor \frac{H+2p_h-k_h^{\text{eff}}}{s_h} \right\rfloor+1. \]

The output width is

\[ W_{\text{out}} = \left\lfloor \frac{W+2p_w-k_w^{\text{eff}}}{s_w} \right\rfloor+1. \]

Special cases are important:

valid convolution: no padding;
same convolution: padding chosen so output size is close to input size when stride is one;
stride greater than one: convolution and downsampling are combined;
dilation greater than one: the kernel sees a wider spatial range without increasing parameter count.

22.8.1 Boundary effects

Boundary conventions determine what happens outside the observed image. Common choices are zero padding, reflection padding, replication padding, circular padding, or cropping. These choices affect exact equivariance and may produce artifacts near the image boundary.

22.9 21.9 Receptive fields

The receptive field of a unit in a deep feature map is the set of input pixels that can affect it.

For a sequence of convolutional layers, define:

$r_\ell$: receptive-field size at layer $\ell$;
$j_\ell$: jump, or effective spacing between adjacent features in input coordinates;
$k_\ell$: kernel size at layer $\ell$;
$s_\ell$: stride at layer $\ell$;
$d_\ell$: dilation at layer $\ell$.

Initialize

\[ r_0=1, \qquad j_0=1. \]

Then

\[ j_\ell=j_{\ell-1}s_\ell, \]

and

\[ r_\ell=r_{\ell-1}+(k_\ell-1)d_\ell j_{\ell-1}. \]

This recursion is essential for understanding architecture design. Small kernels stacked deeply can produce large receptive fields while using fewer parameters and more nonlinearities.

22.9.1 Example: three $3\times 3$ convolutions

With stride one and dilation one,

\[ r_0=1, \]

\[ r_1=3, \]

\[ r_2=5, \]

\[ r_3=7. \]

Three $3\times 3$ layers have the receptive-field size of one $7\times 7$ layer but include three nonlinear transformations.

22.10 21.10 Interactive demo: receptive field growth

Receptive field growth

Change the number of layers, kernel size, stride, and dilation.

Layers: Kernel size: Stride: Dilation:

22.11 21.11 Pooling and local invariance

A pooling layer summarizes a local region. Max pooling is

\[ Y[i,j,c]=\max_{(a,b)\in\mathcal N} X[si+a,sj+b,c], \]

where $s$ is the stride. Average pooling is

\[ Y[i,j,c]=\frac{1}{|\mathcal N|}\sum_{(a,b)\in\mathcal N}X[si+a,sj+b,c]. \]

Pooling has two effects:

it reduces spatial resolution;
it introduces local stability to small translations.

Max pooling says: if a feature appears anywhere in a small neighborhood, retain its strongest response. Average pooling says: summarize local evidence by local averaging.

22.11.1 Global average pooling

Near the end of a CNN, global average pooling maps

\[ X\in\mathbb R^{H\times W\times C} \]

\[ z[c]=\frac{1}{HW}\sum_{i=1}^H\sum_{j=1}^W X[i,j,c]. \]

This produces a feature vector $z\in\mathbb R^C$ and removes explicit spatial location. It is a common bridge from equivariant feature maps to invariant classification.

22.11.2 Pooling as a statistical estimator

Average pooling estimates a local mean. Max pooling estimates an extreme response. The choice matters statistically: average pooling is smoother and more stable, while max pooling is more selective and sensitive to strong local activations.

22.14 21.14 CNNs as compositions of equivariant maps

A deep CNN can be written as

\[ f_\theta = A_L\circ \sigma \circ A_{L-1}\circ \cdots \circ \sigma \circ A_1, \]

where each $A_\ell$ is an affine convolutional map:

\[ A_\ell(X)=K^{(\ell)}\star X+b^{(\ell)}. \]

If each $A_\ell$ is translation equivariant and each activation is applied pointwise, then the composition remains translation equivariant:

\[ F(T_\tau X)=T_\tau F(X). \]

Pooling and boundary handling may weaken exact equivariance, but the network still has a strong approximate translation structure.

22.14.1 Hierarchical representation

Early layers often represent local low-level features:

\[ \text{edges},\quad \text{colors},\quad \text{textures}. \]

Middle layers represent larger motifs:

\[ \text{corners},\quad \text{parts},\quad \text{repeated patterns}. \]

Late layers represent task-level abstractions:

\[ \text{object parts},\quad \text{semantic regions},\quad \text{class evidence}. \]

This hierarchy is not guaranteed in every trained network, but it is a useful organizing model and often appears empirically.

22.15 21.15 Matrix view: convolution as a sparse linear operator

For a fixed kernel $K$, convolution is linear in the input $X$. After vectorization, we can write

\[ \operatorname{vec}(Y)=A_K\operatorname{vec}(X), \]

where $A_K$ is a sparse structured matrix. In one dimension with suitable padding, $A_K$ is Toeplitz. In two dimensions, it has block Toeplitz structure.

This view is useful for:

proving linearity and equivariance;
deriving gradients;
analyzing stability and operator norms;
connecting CNNs to numerical linear algebra;
comparing convolution to classical filters.

22.15.1 Operator norm and stability

A linear convolutional layer satisfies

\[ \|K*X-K*X'\|_2 \le \|A_K\|_2\|X-X'\|_2. \]

If the activation $\sigma$ is Lipschitz with constant $L_\sigma$, then

\[ \|\sigma(K*X)-\sigma(K*X')\|_2 \le L_\sigma\|A_K\|_2\|X-X'\|_2. \]

For ReLU, $L_\sigma=1$. Therefore the Lipschitz behavior of a CNN is controlled by the operator norms of its convolutional layers. This matters for robustness, adversarial perturbations, and stability analysis.

22.16 21.16 Backpropagation through convolution

Suppose a one-dimensional cross-correlation layer is

\[ Y[i]=\sum_a K[a]X[i+a]. \]

Let $L$ be a scalar loss and let

\[ G[i]=\frac{\partial L}{\partial Y[i]}. \]

Then the gradient with respect to the kernel is

\[ \frac{\partial L}{\partial K[a]} =\sum_i G[i]X[i+a]. \]

This is a cross-correlation between the input and the upstream gradient.

The gradient with respect to the input is

\[ \frac{\partial L}{\partial X[t]} =\sum_i G[i]K[t-i], \]

which is a convolution-like operation with the flipped kernel.

22.16.1 Multi-channel version

For

\[ Y[i,j,q] = \sum_{a,b,c}K[a,b,c,q]X[i+a,j+b,c], \]

the kernel gradient is

\[ \frac{\partial L}{\partial K[a,b,c,q]} = \sum_{i,j}G[i,j,q]X[i+a,j+b,c]. \]

The input gradient sums over all output channels:

\[ \frac{\partial L}{\partial X[u,v,c]} = \sum_{i,j,q}G[i,j,q]K[u-i,v-j,c,q], \]

with index restrictions determined by padding and boundary conventions.

Connection to classical filtering

Forward convolution extracts features. Backward convolution distributes credit to filters and pixels. Training a CNN repeatedly applies structured linear operators in both directions.

22.17 21.17 Residual connections

A residual block has the form

\[ Y=X+F_\theta(X), \]

or, when dimensions differ,

\[ Y=P(X)+F_\theta(X), \]

where $P$ is a projection or strided convolution. Residual networks made it practical to train very deep CNNs (He et al. 2015).

22.17.1 Why residual connections help

The Jacobian of a residual block is

\[ \frac{\partial Y}{\partial X} =I+\frac{\partial F_\theta}{\partial X}. \]

The identity term provides a direct path for gradients. This helps reduce optimization difficulties associated with very deep compositions.

Residual blocks also make it easier for a network to learn perturbations around the identity map. In image restoration, denoising, super-resolution, and feature refinement, learning a residual correction is often easier than learning the full map from scratch.

22.18 21.18 Interactive demo: residual connections and gradients

Residual connection intuition

Compare repeated multiplication by a small slope with repeated residual updates.

Layer slope a: Depth:

22.19 21.19 Normalization layers

Batch normalization transforms activations using mini-batch statistics:

\[ \widehat z = \frac{z-\mu_B}{\sqrt{\sigma_B^2+\epsilon}}, \]

then applies learned scale and shift:

\[ y=\gamma \widehat z+\beta. \]

Normalization changes optimization geometry. It can smooth training, reduce sensitivity to initialization, and allow larger learning rates. Common normalization layers include:

batch normalization;
layer normalization;
instance normalization;
group normalization.

22.19.1 Batch normalization as reparameterization

Batch normalization is not only a statistical standardization step. It changes the parameterization of the network. Since the batch mean and variance depend on the current mini-batch, the layer also introduces stochasticity during training.

22.19.2 Caution for small batches

For medical imaging and high-resolution segmentation, batch size may be small. Batch normalization can then be unstable. Group normalization or instance normalization may be preferable.

22.20 21.20 Dropout, weight decay, and data augmentation

CNNs are usually trained with several regularization methods.

22.20.1 Weight decay

Weight decay adds a penalty

\[ \lambda\|\theta\|_2^2 \]

to the empirical risk. For a loss $\ell$, training solves approximately

\[ \min_\theta \frac{1}{n}\sum_{i=1}^n \ell(f_\theta(x_i),y_i)+\lambda\|\theta\|_2^2. \]

22.20.2 Dropout

Dropout randomly masks activations during training. If $m$ is a random Bernoulli mask, then

\[ \widetilde h = m\odot h. \]

This discourages excessive co-adaptation of units.

22.20.3 Data augmentation

Data augmentation applies transformations such as cropping, flipping, rotation, color jitter, blur, and noise. Mathematically, augmentation replaces the empirical risk by

\[ \frac{1}{n}\sum_{i=1}^n \mathbb E_{T\sim\mathcal T} \ell(f_\theta(Tx_i),y_i). \]

This encourages invariance to transformations in $\mathcal T$.

22.21 21.21 Classical CNN architectures

This section is not a history lesson only. Each architecture introduced a mathematical or optimization idea.

22.21.1 LeNet

LeNet-style networks used convolution, pooling, and gradient-based learning for document recognition (LeCun et al. 1998). The key idea was to combine local filters with weight sharing.

22.21.2 AlexNet

AlexNet used deeper CNNs, ReLU activations, data augmentation, dropout, and GPU training to achieve strong large-scale image classification performance (Krizhevsky et al. 2012).

22.21.3 VGG

VGG-style networks emphasized repeated small $3\times3$ convolutions. The mathematical lesson is that stacked small kernels can replace larger kernels while adding depth and nonlinearity.

22.21.4 Inception

Inception-style modules use multiple filter sizes in parallel. The idea is multi-scale processing: different receptive fields capture different spatial structures.

22.21.5 ResNet

Residual networks use skip connections:

\[ X\mapsto X+F_\theta(X). \]

This changes the optimization landscape and enables very deep networks (He et al. 2015).

22.21.6 DenseNet

DenseNet-style networks concatenate features from earlier layers, encouraging feature reuse and direct gradient flow.

22.21.7 MobileNet and efficient CNNs

MobileNet-style architectures use depthwise separable convolutions. A standard convolution has parameter count

\[ k_hk_wC_{\text{in}}C_{\text{out}}. \]

A depthwise separable convolution has approximately

\[ k_hk_wC_{\text{in}} + C_{\text{in}}C_{\text{out}} \]

parameters. This is much smaller when $C_{\text{out}}$ is large.

22.22 21.22 Depthwise separable convolution

A standard convolution mixes spatial and channel information simultaneously:

\[ Y[i,j,q] =\sum_{a,b,c}K[a,b,c,q]X[i+a,j+b,c]. \]

A depthwise separable convolution separates this into two steps.

First, a depthwise spatial convolution applies one spatial filter per input channel:

\[ U[i,j,c]=\sum_{a,b}D[a,b,c]X[i+a,j+b,c]. \]

Second, a pointwise $1\times1$ convolution mixes channels:

\[ Y[i,j,q]=\sum_c P[c,q]U[i,j,c]. \]

This is a structured low-parameter factorization of a standard convolutional operator. It is central to efficient mobile and edge-device vision.

22.23 21.23 CNNs for classification, detection, and segmentation

CNNs can be adapted to many vision tasks.

22.23.1 Classification

A classification CNN maps

\[ X\mapsto z\mapsto \operatorname{softmax}(z). \]

The final vector gives class probabilities.

22.23.2 Detection

Detection requires both classification and localization:

\[ X\mapsto \{(\widehat b_m,\widehat c_m,\widehat s_m)\}_{m=1}^M, \]

where $\widehat b_m$ is a bounding box, $\widehat c_m$ is a class, and $\widehat s_m$ is a confidence score.

22.23.3 Segmentation

Segmentation produces a spatial prediction:

\[ X\mapsto \widehat Y\in \{1,\ldots,K\}^{H\times W}. \]

Fully convolutional networks and U-Net architectures use convolutional encoder-decoder structures to preserve spatial information (Ronneberger et al. 2015).

22.23.4 Feature pyramid networks

Detection and segmentation often require multi-scale features. A feature pyramid builds representations at multiple spatial resolutions, combining semantic depth with spatial precision.

22.24 21.24 Statistical learning perspective

Let the training data be

\[ \{(X_i,Y_i)\}_{i=1}^n\sim P. \]

A CNN class $\mathcal F$ contains functions $f_\theta$. Training solves empirical risk minimization:

\[ \widehat\theta \in \arg\min_\theta \frac{1}{n}\sum_{i=1}^n \ell(f_\theta(X_i),Y_i). \]

The population risk is

\[ R(\theta)=\mathbb E[\ell(f_\theta(X),Y)]. \]

The mathematical questions are:

approximation: can a CNN represent a good predictor?
optimization: can training find useful parameters?
generalization: will low training risk imply low population risk?
robustness: will the predictor remain stable under distribution shift?

22.24.1 CNN prior as a model class restriction

CNNs do not search over all functions from images to labels. They search over functions with locality, parameter sharing, equivariance, and hierarchical composition. This restriction can reduce sample complexity when the assumptions are appropriate.

22.24.2 Distribution shift

Computer vision datasets often differ from deployment data. For example:

scanner type changes in medical imaging;
lighting changes in autonomous driving;
geography changes in satellite imaging;
demographics change in face analysis;
acquisition protocols change in microscopy.

Mathematically, training uses $P_{\text{train}}$, but deployment uses $P_{\text{test}}$. Generalization across this shift is not guaranteed.

22.25 21.25 CNNs and vision transformers

CNNs and vision transformers use different inductive biases.

A CNN begins with local spatial operations:

\[ Y[i,j]=\psi(\mathcal P_{i,j}(X)). \]

A vision transformer begins by splitting an image into patches and applying attention across tokens:

\[ \operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^T}{\sqrt d}\right)V. \]

CNNs encode locality and translation structure directly. Transformers can learn long-range interactions more flexibly, but may require more data or pretraining unless locality is reintroduced through architectural choices.

Modern architectures often mix both ideas: convolutional stems, local attention, hierarchical token pyramids, patch merging, and convolutional positional encodings.

22.26 21.26 Interpretability and feature visualization

CNN interpretability methods attempt to understand what a network has learned.

22.26.1 Activation maps

Given a feature map $X^{(\ell)}$, one can visualize each channel

\[ X^{(\ell)}[:,:,c] \]

as a spatial response map. Large values indicate locations where that feature is active.

22.26.2 Saliency maps

For a class score $s_k(X)$, a saliency map uses

\[ \left|\frac{\partial s_k}{\partial X[i,j,c]}\right| \]

to estimate sensitivity to input pixels.

22.26.3 Class activation maps

If the final convolutional feature map is $A_c(i,j)$ and the classifier weight for class $k$ is $w_{k,c}$, a class activation map is

\[ M_k(i,j)=\sum_c w_{k,c}A_c(i,j). \]

This highlights spatial regions supporting class $k$.

Interpretability caution

A visualization is not a proof. Saliency and activation maps are diagnostic tools. They can suggest hypotheses, reveal artifacts, or catch failures, but they should not be treated as definitive explanations.

22.27 21.27 Robustness and adversarial perturbations

A small perturbation $\delta$ may change the prediction:

\[ f_\theta(X+\delta)\ne f_\theta(X), \qquad \|\delta\| \text{ small}. \]

An adversarial example solves approximately

\[ \max_{\|\delta\|\le \epsilon} \ell(f_\theta(X+\delta),y). \]

Using a first-order Taylor approximation,

\[ \ell(f_\theta(X+\delta),y) \approx \ell(f_\theta(X),y)+\nabla_X\ell(f_\theta(X),y)^T\delta. \]

For an $\ell_\infty$ constraint, the maximizing perturbation is approximately

\[ \delta=\epsilon\operatorname{sign}(\nabla_X\ell(f_\theta(X),y)). \]

This is the fast gradient sign idea. It shows that high-dimensional image classifiers can be sensitive to carefully aligned perturbations.

22.27.1 Robustness as mathematical stability

Robustness asks for bounds such as

\[ \|f_\theta(X)-f_\theta(X')\| \le L\|X-X'\|. \]

Small Lipschitz constants, margin-based training, data augmentation, adversarial training, and certified robustness methods all attempt to improve stability.

22.28 21.28 Python example: convolution by hand

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(7243)

# Synthetic grayscale image
H, W = 64, 64
x = np.zeros((H, W))
x[18:46, 25:39] = 1.0
x += 0.08 * rng.normal(size=(H, W))

# A Sobel-like horizontal derivative kernel
K = np.array([
    [-1, 0, 1],
    [-2, 0, 2],
    [-1, 0, 1]
], dtype=float)

def correlate2d_same(image, kernel):
    kh, kw = kernel.shape
    ph, pw = kh // 2, kw // 2
    padded = np.pad(image, ((ph, ph), (pw, pw)), mode="edge")
    out = np.zeros_like(image, dtype=float)
    for i in range(image.shape[0]):
        for j in range(image.shape[1]):
            patch = padded[i:i+kh, j:j+kw]
            out[i, j] = np.sum(kernel * patch)
    return out

y = correlate2d_same(x, K)

fig, axes = plt.subplots(1, 2, figsize=(9, 4))
axes[0].imshow(x, cmap="gray")
axes[0].set_title("input image")
axes[0].axis("off")
axes[1].imshow(y, cmap="gray")
axes[1].set_title("filter response")
axes[1].axis("off")
plt.show()

This example uses cross-correlation, which is the operation used by most deep-learning convolution layers.

22.29 21.29 Python example: parameter counting

def conv_params(kh, kw, cin, cout, bias=True):
    return kh * kw * cin * cout + (cout if bias else 0)

def dense_params(input_dim, output_dim, bias=True):
    return input_dim * output_dim + (output_dim if bias else 0)

H, W, C = 224, 224, 3
cout = 64

print("Dense parameters:", dense_params(H * W * C, cout))
print("3x3 convolution parameters:", conv_params(3, 3, C, cout))
print("7x7 convolution parameters:", conv_params(7, 7, C, cout))

Students should compare the parameter counts and explain the difference using parameter sharing.

22.30 21.30 TensorFlow example: a small CNN classifier

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

model = keras.Sequential([
    layers.Input(shape=(32, 32, 3)),
    layers.Conv2D(32, kernel_size=3, padding="same", activation="relu"),
    layers.MaxPooling2D(pool_size=2),
    layers.Conv2D(64, kernel_size=3, padding="same", activation="relu"),
    layers.MaxPooling2D(pool_size=2),
    layers.Conv2D(128, kernel_size=3, padding="same", activation="relu"),
    layers.GlobalAveragePooling2D(),
    layers.Dense(10, activation="softmax")
])

model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

model.summary()

Important mathematical observations:

convolutional layers preserve spatial structure;
pooling reduces spatial resolution;
global average pooling produces a vector representation;
the final dense layer is a multiclass logistic regression on learned features.

22.31 21.31 TensorFlow example: convolutional block with batch normalization and residual connection

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

class ResidualBlock(layers.Layer):
    def __init__(self, channels, stride=1):
        super().__init__()
        self.conv1 = layers.Conv2D(channels, 3, strides=stride, padding="same", use_bias=False)
        self.bn1 = layers.BatchNormalization()
        self.conv2 = layers.Conv2D(channels, 3, padding="same", use_bias=False)
        self.bn2 = layers.BatchNormalization()
        self.proj = None
        if stride != 1:
            self.proj = keras.Sequential([
                layers.Conv2D(channels, 1, strides=stride, padding="same", use_bias=False),
                layers.BatchNormalization()
            ])

    def call(self, x, training=False):
        shortcut = x if self.proj is None else self.proj(x, training=training)
        z = self.conv1(x)
        z = self.bn1(z, training=training)
        z = tf.nn.relu(z)
        z = self.conv2(z)
        z = self.bn2(z, training=training)
        return tf.nn.relu(z + shortcut)

inputs = keras.Input(shape=(64, 64, 3))
x = layers.Conv2D(32, 3, padding="same", activation="relu")(inputs)
x = ResidualBlock(32)(x)
x = ResidualBlock(64, stride=2)(x)
x = layers.GlobalAveragePooling2D()(x)
outputs = layers.Dense(5, activation="softmax")(x)
model = keras.Model(inputs, outputs)
model.summary()

This example shows how a residual network changes spatial resolution while maintaining identity or projected shortcut paths.

22.32 21.32 TensorFlow example: feature-map visualization

import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# Suppose model is a trained Keras CNN.
# Extract the output of the first convolutional layer.
conv_layer = model.layers[0]
feature_model = keras.Model(inputs=model.input, outputs=conv_layer.output)

# x_image should have shape (1, H, W, C).
features = feature_model(x_image, training=False).numpy()[0]

num_channels = min(8, features.shape[-1])
fig, axes = plt.subplots(1, num_channels, figsize=(2 * num_channels, 2))
for c in range(num_channels):
    axes[c].imshow(features[:, :, c], cmap="viridis")
    axes[c].set_title(f"channel {c}")
    axes[c].axis("off")
plt.show()

Feature-map visualization is useful for debugging. For example, if all feature maps are nearly zero, the model may have an optimization or preprocessing problem.

22.33 21.33 AI component: using AI systems responsibly while learning CNNs

AI tools can help students explore CNNs, but they should be used as mathematical assistants rather than answer generators.

22.33.1 Useful prompts

Equivariance check.
“Given the definition of translation $T_\tau$ and convolution $K*X$, verify the equivariance proof step by step. Point out where boundary effects enter.”
Receptive-field calculation.
“Compute the receptive field of a CNN with layers $(3,1,1)$, $(3,2,1)$, $(3,1,2)$, where each triple is kernel size, stride, dilation. Show the recursion.”
Architecture critique.
“This CNN has four max-pooling layers on a $32\times32$ image. Explain what spatial resolution remains and whether this is appropriate for segmentation.”
Debugging.
“My CNN training accuracy increases but validation accuracy decreases. List statistical explanations and architecture-level fixes.”
Model comparison.
“Compare CNNs and vision transformers for small medical imaging datasets. Focus on inductive bias, sample size, augmentation, and interpretability.”

22.33.2 Verification habit

When using AI to reason about CNNs, always verify:

tensor shapes;
parameter counts;
output sizes;
loss definitions;
whether the claimed operation is convolution or cross-correlation;
whether the architecture preserves enough spatial resolution for the task.

22.34 21.34 Common pitfalls

22.34.1 Pitfall 1: confusing convolution and cross-correlation

Most software uses cross-correlation but calls it convolution. This is usually harmless for learned kernels but matters when manually implementing known filters.

22.34.2 Pitfall 2: ignoring output shape

Stride and pooling can quickly reduce spatial resolution. For dense prediction tasks such as segmentation, too much downsampling can destroy boundary precision.

22.34.3 Pitfall 3: treating augmentation as free data

Augmentation encodes invariances. If the invariance is false, augmentation can harm performance. For example, horizontal flipping may be invalid in medical imaging or digit recognition depending on the label semantics.

22.34.4 Pitfall 4: using batch normalization with tiny batches

Mini-batch statistics can be noisy when batch size is very small. This is common in 3D medical imaging and high-resolution segmentation.

22.34.5 Pitfall 5: believing feature maps are explanations

Feature maps and saliency maps are diagnostic tools. They do not automatically provide causal explanations.

22.35 21.35 Summary

CNNs are mathematically structured neural networks for image data. Their core principles are:

\[ \text{locality}, \qquad \text{translation equivariance}, \qquad \text{parameter sharing}, \qquad \text{hierarchical composition}. \]

A convolutional layer is a learned local linear operator followed by a nonlinearity. Deep CNNs compose many such layers to build increasingly abstract representations. Pooling and global aggregation convert equivariant feature maps into approximately invariant predictions. Residual connections and normalization layers improve optimization and enable deeper architectures.

From a statistical point of view, CNNs impose a useful inductive bias on high-dimensional image learning problems. From an applied-mathematics point of view, they are nonlinear compositions of structured linear operators. From a modern AI point of view, they remain essential building blocks for classification, detection, segmentation, medical imaging, remote sensing, robotics, and hybrid CNN-transformer systems.

22.36 21.36 Exercises

22.36.1 Conceptual exercises

Explain the difference between translation equivariance and translation invariance. Give one example of a vision task where equivariance is useful and one where invariance is useful.
Why does parameter sharing reduce the effective complexity of a model? Explain from both a computational and statistical viewpoint.
Suppose a CNN uses four $2\times2$ max-pooling layers with stride two. If the input image is $128\times128$, what is the spatial size after the pooling layers?
Why might global average pooling be preferable to flattening followed by a large dense layer?
Explain why a $1\times1$ convolution is not trivial in a multi-channel feature map.

22.36.2 Mathematical exercises

Prove that a pointwise activation $\sigma$ commutes with translations:

\[ \sigma(T_\tau X)=T_\tau\sigma(X). \]

Use this to show that a convolution followed by a pointwise nonlinearity is translation equivariant.

For a convolutional layer with input channels $C_{\text{in}}=32$, output channels $C_{\text{out}}=64$, and kernel size $3\times3$, compute the number of weights and biases.
A CNN has three convolutional layers. The first has kernel size 3 and stride 1. The second has kernel size 3 and stride 2. The third has kernel size 5 and stride 1. Compute the receptive-field size and effective jump after each layer.
Derive the gradient

\[ \frac{\partial L}{\partial K[a,b,c,q]} \]

for a multi-channel cross-correlation layer.

Show that a depthwise separable convolution has fewer parameters than a standard convolution when

\[ k_hk_wC_{\text{in}}C_{\text{out}} > k_hk_wC_{\text{in}}+C_{\text{in}}C_{\text{out}}. \]

Interpret this inequality.

22.36.3 Python exercises

Implement a two-dimensional convolution or cross-correlation function using only NumPy. Compare its output with scipy.signal.correlate2d or a TensorFlow convolution.
Train a small CNN on a simple image dataset such as MNIST, Fashion-MNIST, or CIFAR-10. Record training and validation accuracy. Try at least two architectures.
Visualize first-layer filters and first-layer feature maps. Describe what patterns appear.
Change kernel size from $3\times3$ to $5\times5$. Compare parameter counts, training behavior, and validation accuracy.
Replace max pooling by average pooling. Compare performance and feature-map smoothness.

22.36.4 AI-assisted exercises

Ask an AI system to compute the receptive field of a CNN architecture you design. Then verify every step manually.
Ask an AI system to explain why a CNN is translation equivariant. Identify any missing assumptions in the explanation.
Ask an AI system to generate a Keras CNN for semantic segmentation. Check whether the output tensor has the correct spatial dimensions.
Ask an AI system to compare CNNs and ViTs for a small biomedical dataset. Rewrite the answer in mathematical terms using inductive bias, sample complexity, and regularization.
Ask an AI system to debug a CNN with poor validation performance. Separate the suggestions into optimization issues, statistical issues, data issues, and architecture issues.

22.37 21.37 Notes and further reading

Convolutional neural networks are discussed in classical and modern forms in LeCun et al. (1998), Krizhevsky et al. (2012), Goodfellow et al. (2016), and He et al. (2015). For broad computer-vision context, see Szeliski (2022) and Prince (2012). For segmentation-oriented CNNs, see Ronneberger et al. (2015) and fully convolutional networks. For the modern transition from CNNs to attention-based vision models, compare this chapter with the later chapter on vision transformers.

Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. https://www.deeplearningbook.org/.

He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. “Deep Residual Learning for Image Recognition.” arXiv Preprint arXiv:1512.03385. https://arxiv.org/abs/1512.03385.

Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. 2012. “ImageNet Classification with Deep Convolutional Neural Networks.” Advances in Neural Information Processing Systems.

LeCun, Yann, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. “Gradient-Based Learning Applied to Document Recognition.” Proceedings of the IEEE 86: 2278–324.

Prince, Simon J. D. 2012. Computer Vision: Models, Learning, and Inference. Cambridge University Press. https://www.computervisionmodels.com/.

Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. 2015. “U-Net: Convolutional Networks for Biomedical Image Segmentation.” Medical Image Computing and Computer-Assisted Intervention.

Szeliski, Richard. 2022. Computer Vision: Algorithms and Applications. 2nd ed. Springer. https://szeliski.org/Book/.

--- title: "Chapter 21. Convolutional Neural Networks" subtitle: "Local Operators, Equivariance, Receptive Fields, and Learned Visual Representations" format: html: toc: true toc-depth: 3 execute: warning: false message: false --- ## 21.1 Why convolutional neural networks belong in a mathematical vision book Computer vision has always used local image operations. In earlier chapters we studied convolution, Gaussian smoothing, derivative filters, edges, corners, optical flow, and local descriptors. A convolutional neural network, or CNN, is the modern statistical-learning version of the same idea: $$ \text{local linear operator} \quad + \quad \text{nonlinear activation} \quad + \quad \text{composition}. $$ The mathematical importance of CNNs is not just that they work well. Their importance is that they encode several structural assumptions about images: 1. **locality:** nearby pixels usually interact more strongly than far-away pixels; 2. **translation equivariance:** if an object shifts, the feature map should shift in the same way; 3. **parameter sharing:** the same local detector can be used at every image location; 4. **hierarchy:** simple local patterns combine into larger semantic patterns; 5. **stability:** useful features should not change too much under small perturbations. For MA Applied Math and MS Statistics students, CNNs are best understood as structured nonlinear maps $$ f_\theta : \mathbb R^{H\times W\times C} \to \mathbb R^K $$ whose parameters are constrained by the geometry of the image lattice. Compared with a fully connected network on raw pixels, a CNN uses far fewer parameters and has a much stronger inductive bias. Historically, convolutional networks were central in handwritten-digit recognition through LeNet-style architectures [@lecun1998gradient]. They became the dominant architecture for large-scale image recognition after AlexNet [@krizhevsky2012imagenet], and residual networks then showed that very deep CNNs could be optimized effectively [@he2015resnet]. Even in the era of vision transformers, CNN ideas remain foundational: locality, equivariance, multi-scale features, pooling, residual connections, normalization, and convolutional feature maps are still used in detection, segmentation, medical imaging, robotics, and scientific vision. ### Learning goals After this chapter, you should be able to: 1. define convolutional and cross-correlation layers for single-channel and multi-channel images; 2. prove that convolutional layers are translation equivariant under suitable boundary conventions; 3. compute the number of parameters and output size of convolutional layers; 4. calculate receptive-field size across layers with stride, padding, and dilation; 5. explain pooling, local invariance, and downsampling from a mathematical viewpoint; 6. derive basic backpropagation formulas for convolutional kernels; 7. interpret CNNs as compositions of equivariant maps and pointwise nonlinearities; 8. connect CNNs to empirical risk minimization, regularization, sample complexity, and statistical bias; 9. explain residual connections, normalization layers, and modern CNN architectures; 10. implement small CNN examples in NumPy, scikit-learn style workflows, and TensorFlow. ::: {.callout-note} ## Central idea A convolutional network is a learned hierarchy of local filters: $$ X^{(0)}=I, \qquad X^{(\ell+1)}=\sigma\!\left(K^{(\ell)} * X^{(\ell)}+b^{(\ell)}\right). $$ The mathematical power comes from the combination of **linear local structure**, **nonlinear composition**, and **statistical learning from data**. ::: ## 21.2 Images as lattice-indexed tensors Let a color image be represented as a tensor $$ X\in \mathbb R^{H\times W\times C}. $$ We write $$ X[i,j,c] $$ for the pixel value at row $i$, column $j$, and channel $c$. The set of spatial indices $$ \Lambda = \{0,1,\ldots,H-1\}\times \{0,1,\ldots,W-1\} $$ is a finite two-dimensional lattice. A convolutional filter uses a small neighborhood around each location. For a kernel of spatial size $(2r+1)\times(2s+1)$, the local patch around $(i,j)$ is $$ \mathcal P_{i,j}(X) = \{X[i+a,j+b,c] : -r\le a\le r,\ -s\le b\le s,\ 1\le c\le C\}. $$ A fully connected layer treats the image as an unstructured vector in $\mathbb R^{HWC}$. A convolutional layer treats the image as an object with spatial geometry. ### Locality assumption A local operator has the form $$ Y[i,j] = \psi(\mathcal P_{i,j}(X)) $$ for some function $\psi$. If $\psi$ is linear, then $$ Y[i,j] = \sum_{a=-r}^r\sum_{b=-s}^s\sum_{c=1}^C K[a,b,c]X[i+a,j+b,c]. $$ This is the basic algebraic form of a convolutional feature detector. ::: {.callout-tip} ## Statistical interpretation Locality is a prior. It says that the response at a spatial location should depend mostly on a local neighborhood. This prior is often useful for natural images, medical images, satellite images, microscopy images, and video frames. ::: ## 21.3 Convolution and cross-correlation There is a small notational difference between mathematical convolution and the operation used in many deep learning libraries. For one-dimensional signals, mathematical convolution is $$ (K*X)[i]=\sum_a K[a]X[i-a]. $$ Cross-correlation is $$ (K\star X)[i]=\sum_a K[a]X[i+a]. $$ In two dimensions, $$ (K*X)[i,j] = \sum_a\sum_b K[a,b]X[i-a,j-b], $$ while $$ (K\star X)[i,j] = \sum_a\sum_b K[a,b]X[i+a,j+b]. $$ Most neural-network libraries call cross-correlation **convolution** because the kernel is learned. If the kernel is learned, flipping the kernel is not a meaningful distinction: the network can learn either orientation. ### Multi-channel convolution Suppose the input has $C_{\text{in}}$ channels and the output has $C_{\text{out}}$ channels. A convolutional kernel is a four-dimensional tensor $$ K\in \mathbb R^{k_h\times k_w\times C_{\text{in}}\times C_{\text{out}}}. $$ The output feature map is $$ Y[i,j,q] = \sum_{a=0}^{k_h-1}\sum_{b=0}^{k_w-1}\sum_{c=1}^{C_{\text{in}}} K[a,b,c,q]X[i+a,j+b,c]+b[q]. $$ For each output channel $q$, the layer learns one bank of local filters across all input channels. The number of trainable parameters is $$ \#\text{parameters}=k_hk_wC_{\text{in}}C_{\text{out}}+C_{\text{out}}. $$ This should be compared with a dense layer from $\mathbb R^{HWC}$ to $\mathbb R^{H'W'C_{\text{out}}}$, which would have roughly $$ (HWC)(H'W'C_{\text{out}}) $$ weights. The reduction in parameter count is a major statistical advantage. ## 21.4 Translation equivariance Let $T_\tau$ denote translation by a lattice vector $\tau=(\tau_1,\tau_2)$: $$ (T_\tau X)[i,j] = X[i-\tau_1,j-\tau_2]. $$ An operator $F$ is **translation equivariant** if $$ F(T_\tau X)=T_\tau F(X). $$ That is, translating the input translates the output. ### Proposition 21.1: convolution is translation equivariant Ignoring boundary effects, convolution satisfies $$ K*(T_\tau X)=T_\tau(K*X). $$ **Proof.** For one-dimensional notation, $$ \begin{aligned} (K*(T_\tau X))[i] &=\sum_a K[a](T_\tau X)[i-a] \\ &=\sum_a K[a]X[i-a-\tau] \\ &=(K*X)[i-\tau] \\ &=(T_\tau(K*X))[i]. \end{aligned} $$ The same argument works in two dimensions by replacing $i-a$ with $(i-a,j-b)$. $\square$ The proof is short, but the consequence is deep. CNNs are built from layers that respect the spatial symmetry of images. ### Equivariance versus invariance A feature map $F$ is equivariant if movement in the input produces corresponding movement in the output: $$ F(T_\tau X)=T_\tau F(X). $$ A classifier $g$ is invariant if movement in the input does not change the final label: $$ g(T_\tau X)=g(X). $$ A common CNN strategy is: $$ \text{equivariant feature maps} \quad \longrightarrow \quad \text{pooling/global aggregation} \quad \longrightarrow \quad \text{approximately invariant prediction}. $$ ## 21.5 Interactive demo: convolution and translation equivariance The next demo shows a one-dimensional signal and a learned-looking local kernel. Moving the input impulse moves the convolution response. This is the simplest visualization of translation equivariance. <div class="interactive-card"> <h3>Convolution response shifts with the input</h3> <p>Move the input spike. The output response moves by the same amount.</p> <label for="ch21-shift">Input shift:</label> <input id="ch21-shift" type="range" min="2" max="38" value="20" step="1" style="width: 60%;"> <div id="ch21-equivariance-plot" class="plotly-demo"></div> </div> <script> (function(){ function convSame(x,k){ const n=x.length, m=k.length, r=Math.floor(m/2), y=Array(n).fill(0); for(let i=0;i<n;i++){ let s=0; for(let a=0;a<m;a++){ const idx=i+a-r; if(idx>=0 && idx<n){ s += k[a]*x[idx]; } } y[i]=s; } return y; } function draw(){ const div=document.getElementById('ch21-equivariance-plot'); if(!div || typeof Plotly === 'undefined'){ if(div){ div.innerHTML='<p><strong>Plotly.js was not available.</strong></p>'; } return; } const slider=document.getElementById('ch21-shift'); const shift=parseInt(slider.value); const n=41; const xs=Array.from({length:n},(_,i)=>i); const x=Array(n).fill(0); x[shift]=1; const k=[-0.25,0.4,1.0,0.4,-0.25]; const y=convSame(x,k); Plotly.newPlot(div,[ {x:xs,y:x,type:'scatter',mode:'lines+markers',name:'input signal'}, {x:xs,y:y,type:'scatter',mode:'lines+markers',name:'convolution response'} ],{ margin:{t:20}, xaxis:{title:'lattice location'}, yaxis:{title:'value',range:[-0.4,1.15]}, legend:{orientation:'h'} },{responsive:true}); } const slider=document.getElementById('ch21-shift'); if(slider){ slider.addEventListener('input',draw); } draw(); })(); </script> ## 21.6 A convolutional layer as an affine map plus nonlinearity A basic convolutional layer is $$ Z = K \star X + b, $$ followed by an activation function $$ Y = \sigma(Z). $$ Common activation functions include $$ \operatorname{ReLU}(t)=\max\{0,t\}, $$ $$ \sigma_{\text{sigmoid}}(t)=\frac{1}{1+e^{-t}}, $$ and $$ \operatorname{GELU}(t)=t\Phi(t), $$ where $\Phi$ is the standard normal distribution function. The activation is applied pointwise: $$ Y[i,j,q]=\sigma(Z[i,j,q]). $$ ### Why nonlinearities are necessary A composition of linear convolutional layers without nonlinearities is still a linear convolutional operator, up to boundary effects. Therefore depth alone is not enough. Nonlinearities allow the network to represent nonlinear decision boundaries, local pattern combinations, and hierarchical image features. For example, the first layer may detect oriented edges, the next layer may detect corners or curves, and later layers may detect parts of objects. ## 21.7 Interactive demo: activation functions <div class="interactive-card"> <h3>Activation functions</h3> <p>Compare ReLU, sigmoid, tanh, and GELU-like nonlinearities.</p> <div id="ch21-activations-plot" class="plotly-demo"></div> </div> <script> (function(){ function erfApprox(x){ const sign = x >= 0 ? 1 : -1; x = Math.abs(x); const a1=0.254829592, a2=-0.284496736, a3=1.421413741, a4=-1.453152027, a5=1.061405429, p=0.3275911; const t=1/(1+p*x); const y=1-(((((a5*t+a4)*t)+a3)*t+a2)*t+a1)*t*Math.exp(-x*x); return sign*y; } function gelu(x){ return 0.5*x*(1+erfApprox(x/Math.sqrt(2))); } function draw(){ const div=document.getElementById('ch21-activations-plot'); if(!div || typeof Plotly === 'undefined'){ if(div){ div.innerHTML='<p><strong>Plotly.js was not available.</strong></p>'; } return; } const xs=Array.from({length:301},(_,i)=>-5+10*i/300); Plotly.newPlot(div,[ {x:xs,y:xs.map(x=>Math.max(0,x)),mode:'lines',name:'ReLU'}, {x:xs,y:xs.map(x=>1/(1+Math.exp(-x))),mode:'lines',name:'sigmoid'}, {x:xs,y:xs.map(x=>Math.tanh(x)),mode:'lines',name:'tanh'}, {x:xs,y:xs.map(gelu),mode:'lines',name:'GELU'} ],{ margin:{t:20}, xaxis:{title:'pre-activation z'}, yaxis:{title:'activation σ(z)'}, legend:{orientation:'h'} },{responsive:true}); } draw(); })(); </script> ## 21.8 Output size: padding, stride, and dilation Let an input have height $H$ and width $W$. Consider a convolutional layer with kernel size $k_h\times k_w$, padding $p_h,p_w$, stride $s_h,s_w$, and dilation $d_h,d_w$. The effective kernel size in height is $$ k_h^{\text{eff}} = d_h(k_h-1)+1, $$ and similarly $$ k_w^{\text{eff}} = d_w(k_w-1)+1. $$ The output height is $$ H_{\text{out}} = \left\lfloor \frac{H+2p_h-k_h^{\text{eff}}}{s_h} \right\rfloor+1. $$ The output width is $$ W_{\text{out}} = \left\lfloor \frac{W+2p_w-k_w^{\text{eff}}}{s_w} \right\rfloor+1. $$ Special cases are important: - **valid convolution:** no padding; - **same convolution:** padding chosen so output size is close to input size when stride is one; - **stride greater than one:** convolution and downsampling are combined; - **dilation greater than one:** the kernel sees a wider spatial range without increasing parameter count. ### Boundary effects Boundary conventions determine what happens outside the observed image. Common choices are zero padding, reflection padding, replication padding, circular padding, or cropping. These choices affect exact equivariance and may produce artifacts near the image boundary. ## 21.9 Receptive fields The **receptive field** of a unit in a deep feature map is the set of input pixels that can affect it. For a sequence of convolutional layers, define: - $r_\ell$: receptive-field size at layer $\ell$; - $j_\ell$: jump, or effective spacing between adjacent features in input coordinates; - $k_\ell$: kernel size at layer $\ell$; - $s_\ell$: stride at layer $\ell$; - $d_\ell$: dilation at layer $\ell$. Initialize $$ r_0=1, \qquad j_0=1. $$ Then $$ j_\ell=j_{\ell-1}s_\ell, $$ and $$ r_\ell=r_{\ell-1}+(k_\ell-1)d_\ell j_{\ell-1}. $$ This recursion is essential for understanding architecture design. Small kernels stacked deeply can produce large receptive fields while using fewer parameters and more nonlinearities. ### Example: three $3\times 3$ convolutions With stride one and dilation one, $$ r_0=1, $$ $$ r_1=3, $$ $$ r_2=5, $$ $$ r_3=7. $$ Three $3\times 3$ layers have the receptive-field size of one $7\times 7$ layer but include three nonlinear transformations. ## 21.10 Interactive demo: receptive field growth <div class="interactive-card"> <h3>Receptive field growth</h3> <p>Change the number of layers, kernel size, stride, and dilation.</p> <label>Layers: <input id="ch21-rf-layers" type="range" min="1" max="12" value="5" step="1"></label> <label style="margin-left:1rem;">Kernel size: <input id="ch21-rf-kernel" type="range" min="1" max="9" value="3" step="2"></label> <label style="margin-left:1rem;">Stride: <input id="ch21-rf-stride" type="range" min="1" max="3" value="1" step="1"></label> <label style="margin-left:1rem;">Dilation: <input id="ch21-rf-dilation" type="range" min="1" max="4" value="1" step="1"></label> <div id="ch21-receptive-field-plot" class="plotly-demo"></div> </div> <script> (function(){ function draw(){ const div=document.getElementById('ch21-receptive-field-plot'); if(!div || typeof Plotly === 'undefined'){ if(div){ div.innerHTML='<p><strong>Plotly.js was not available.</strong></p>'; } return; } const L=parseInt(document.getElementById('ch21-rf-layers').value); const k=parseInt(document.getElementById('ch21-rf-kernel').value); const s=parseInt(document.getElementById('ch21-rf-stride').value); const d=parseInt(document.getElementById('ch21-rf-dilation').value); let r=1, j=1; const layers=[0], rf=[1], jump=[1]; for(let ell=1; ell<=L; ell++){ r = r + (k-1)*d*j; j = j*s; layers.push(ell); rf.push(r); jump.push(j); } Plotly.newPlot(div,[ {x:layers,y:rf,type:'scatter',mode:'lines+markers',name:'receptive field size'}, {x:layers,y:jump,type:'scatter',mode:'lines+markers',name:'effective jump'} ],{ margin:{t:30}, xaxis:{title:'layer'}, yaxis:{title:'input pixels'}, title:'r_ell = r_{ell-1} + (k-1)d j_{ell-1}', legend:{orientation:'h'} },{responsive:true}); } ['ch21-rf-layers','ch21-rf-kernel','ch21-rf-stride','ch21-rf-dilation'].forEach(id=>{ const el=document.getElementById(id); if(el){ el.addEventListener('input',draw); } }); draw(); })(); </script> ## 21.11 Pooling and local invariance A pooling layer summarizes a local region. Max pooling is $$ Y[i,j,c]=\max_{(a,b)\in\mathcal N} X[si+a,sj+b,c], $$ where $s$ is the stride. Average pooling is $$ Y[i,j,c]=\frac{1}{|\mathcal N|}\sum_{(a,b)\in\mathcal N}X[si+a,sj+b,c]. $$ Pooling has two effects: 1. it reduces spatial resolution; 2. it introduces local stability to small translations. Max pooling says: if a feature appears anywhere in a small neighborhood, retain its strongest response. Average pooling says: summarize local evidence by local averaging. ### Global average pooling Near the end of a CNN, global average pooling maps $$ X\in\mathbb R^{H\times W\times C} $$ to $$ z[c]=\frac{1}{HW}\sum_{i=1}^H\sum_{j=1}^W X[i,j,c]. $$ This produces a feature vector $z\in\mathbb R^C$ and removes explicit spatial location. It is a common bridge from equivariant feature maps to invariant classification. ### Pooling as a statistical estimator Average pooling estimates a local mean. Max pooling estimates an extreme response. The choice matters statistically: average pooling is smoother and more stable, while max pooling is more selective and sensitive to strong local activations. ## 21.12 Parameter sharing and statistical efficiency Consider a $224\times224\times3$ image. A dense layer with 64 output units has $$ 224\cdot224\cdot3\cdot64 \approx 9.6\times 10^6 $$ weights. A $3\times3$ convolution with 64 output channels has $$ 3\cdot3\cdot3\cdot64=1728 $$ weights, plus 64 biases. The difference is not only computational. It is statistical. A dense layer must learn unrelated weights for each pixel location. A convolutional layer assumes that the same local pattern can appear anywhere. ### Bias-variance viewpoint A convolutional architecture has higher inductive bias but lower variance. A dense architecture has lower architectural bias but many more parameters. When image data are limited, CNN structure can improve generalization. This does not mean CNNs are always better. If the task is not spatially local or not translation structured, the CNN prior may be inappropriate. Architecture is a modeling assumption. ## 21.13 Interactive demo: parameter sharing <div class="interactive-card"> <h3>Dense layer versus convolutional layer parameter count</h3> <p>Change image size and channel count to compare dense and convolutional parameter growth.</p> <label>Image size H=W: <input id="ch21-param-size" type="range" min="16" max="256" value="128" step="16"></label> <label style="margin-left:1rem;">Output channels/units: <input id="ch21-param-out" type="range" min="8" max="256" value="64" step="8"></label> <label style="margin-left:1rem;">Kernel size: <input id="ch21-param-k" type="range" min="1" max="11" value="3" step="2"></label> <div id="ch21-parameter-plot" class="plotly-demo"></div> </div> <script> (function(){ function draw(){ const div=document.getElementById('ch21-parameter-plot'); if(!div || typeof Plotly === 'undefined'){ if(div){ div.innerHTML='<p><strong>Plotly.js was not available.</strong></p>'; } return; } const H=parseInt(document.getElementById('ch21-param-size').value); const O=parseInt(document.getElementById('ch21-param-out').value); const k=parseInt(document.getElementById('ch21-param-k').value); const C=3; const dense=H*H*C*O + O; const conv=k*k*C*O + O; Plotly.newPlot(div,[{ x:['dense layer','convolution layer'], y:[dense,conv], type:'bar', text:[dense.toLocaleString(),conv.toLocaleString()], textposition:'auto' }],{ margin:{t:30}, yaxis:{title:'number of trainable parameters',type:'log'}, title:'Parameter count on a log scale' },{responsive:true}); } ['ch21-param-size','ch21-param-out','ch21-param-k'].forEach(id=>{ const el=document.getElementById(id); if(el){ el.addEventListener('input',draw); } }); draw(); })(); </script> ## 21.14 CNNs as compositions of equivariant maps A deep CNN can be written as $$ f_\theta = A_L\circ \sigma \circ A_{L-1}\circ \cdots \circ \sigma \circ A_1, $$ where each $A_\ell$ is an affine convolutional map: $$ A_\ell(X)=K^{(\ell)}\star X+b^{(\ell)}. $$ If each $A_\ell$ is translation equivariant and each activation is applied pointwise, then the composition remains translation equivariant: $$ F(T_\tau X)=T_\tau F(X). $$ Pooling and boundary handling may weaken exact equivariance, but the network still has a strong approximate translation structure. ### Hierarchical representation Early layers often represent local low-level features: $$ \text{edges},\quad \text{colors},\quad \text{textures}. $$ Middle layers represent larger motifs: $$ \text{corners},\quad \text{parts},\quad \text{repeated patterns}. $$ Late layers represent task-level abstractions: $$ \text{object parts},\quad \text{semantic regions},\quad \text{class evidence}. $$ This hierarchy is not guaranteed in every trained network, but it is a useful organizing model and often appears empirically. ## 21.15 Matrix view: convolution as a sparse linear operator For a fixed kernel $K$, convolution is linear in the input $X$. After vectorization, we can write $$ \operatorname{vec}(Y)=A_K\operatorname{vec}(X), $$ where $A_K$ is a sparse structured matrix. In one dimension with suitable padding, $A_K$ is Toeplitz. In two dimensions, it has block Toeplitz structure. This view is useful for: 1. proving linearity and equivariance; 2. deriving gradients; 3. analyzing stability and operator norms; 4. connecting CNNs to numerical linear algebra; 5. comparing convolution to classical filters. ### Operator norm and stability A linear convolutional layer satisfies $$ \|K*X-K*X'\|_2 \le \|A_K\|_2\|X-X'\|_2. $$ If the activation $\sigma$ is Lipschitz with constant $L_\sigma$, then $$ \|\sigma(K*X)-\sigma(K*X')\|_2 \le L_\sigma\|A_K\|_2\|X-X'\|_2. $$ For ReLU, $L_\sigma=1$. Therefore the Lipschitz behavior of a CNN is controlled by the operator norms of its convolutional layers. This matters for robustness, adversarial perturbations, and stability analysis. ## 21.16 Backpropagation through convolution Suppose a one-dimensional cross-correlation layer is $$ Y[i]=\sum_a K[a]X[i+a]. $$ Let $L$ be a scalar loss and let $$ G[i]=\frac{\partial L}{\partial Y[i]}. $$ Then the gradient with respect to the kernel is $$ \frac{\partial L}{\partial K[a]} =\sum_i G[i]X[i+a]. $$ This is a cross-correlation between the input and the upstream gradient. The gradient with respect to the input is $$ \frac{\partial L}{\partial X[t]} =\sum_i G[i]K[t-i], $$ which is a convolution-like operation with the flipped kernel. ### Multi-channel version For $$ Y[i,j,q] = \sum_{a,b,c}K[a,b,c,q]X[i+a,j+b,c], $$ the kernel gradient is $$ \frac{\partial L}{\partial K[a,b,c,q]} = \sum_{i,j}G[i,j,q]X[i+a,j+b,c]. $$ The input gradient sums over all output channels: $$ \frac{\partial L}{\partial X[u,v,c]} = \sum_{i,j,q}G[i,j,q]K[u-i,v-j,c,q], $$ with index restrictions determined by padding and boundary conventions. ::: {.callout-note} ## Connection to classical filtering Forward convolution extracts features. Backward convolution distributes credit to filters and pixels. Training a CNN repeatedly applies structured linear operators in both directions. ::: ## 21.17 Residual connections A residual block has the form $$ Y=X+F_\theta(X), $$ or, when dimensions differ, $$ Y=P(X)+F_\theta(X), $$ where $P$ is a projection or strided convolution. Residual networks made it practical to train very deep CNNs [@he2015resnet]. ### Why residual connections help The Jacobian of a residual block is $$ \frac{\partial Y}{\partial X} =I+\frac{\partial F_\theta}{\partial X}. $$ The identity term provides a direct path for gradients. This helps reduce optimization difficulties associated with very deep compositions. Residual blocks also make it easier for a network to learn perturbations around the identity map. In image restoration, denoising, super-resolution, and feature refinement, learning a residual correction is often easier than learning the full map from scratch. ## 21.18 Interactive demo: residual connections and gradients <div class="interactive-card"> <h3>Residual connection intuition</h3> <p>Compare repeated multiplication by a small slope with repeated residual updates.</p> <label>Layer slope a: <input id="ch21-res-a" type="range" min="0.1" max="1.5" value="0.7" step="0.05"></label> <label style="margin-left:1rem;">Depth: <input id="ch21-res-depth" type="range" min="1" max="50" value="20" step="1"></label> <div id="ch21-residual-plot" class="plotly-demo"></div> </div> <script> (function(){ function draw(){ const div=document.getElementById('ch21-residual-plot'); if(!div || typeof Plotly === 'undefined'){ if(div){ div.innerHTML='<p><strong>Plotly.js was not available.</strong></p>'; } return; } const a=parseFloat(document.getElementById('ch21-res-a').value); const depth=parseInt(document.getElementById('ch21-res-depth').value); const layers=Array.from({length:depth+1},(_,i)=>i); const plain=layers.map(l=>Math.pow(a,l)); const residual=layers.map(l=>Math.pow(1+0.05*(a-1),l)); Plotly.newPlot(div,[ {x:layers,y:plain,mode:'lines+markers',name:'plain product a^L'}, {x:layers,y:residual,mode:'lines+markers',name:'residual product (1+0.05(a-1))^L'} ],{ margin:{t:30}, xaxis:{title:'depth'}, yaxis:{title:'gradient-scale proxy'}, title:'Identity paths help preserve gradient scale', legend:{orientation:'h'} },{responsive:true}); } ['ch21-res-a','ch21-res-depth'].forEach(id=>{ const el=document.getElementById(id); if(el){ el.addEventListener('input',draw); } }); draw(); })(); </script> ## 21.19 Normalization layers Batch normalization transforms activations using mini-batch statistics: $$ \widehat z = \frac{z-\mu_B}{\sqrt{\sigma_B^2+\epsilon}}, $$ then applies learned scale and shift: $$ y=\gamma \widehat z+\beta. $$ Normalization changes optimization geometry. It can smooth training, reduce sensitivity to initialization, and allow larger learning rates. Common normalization layers include: - batch normalization; - layer normalization; - instance normalization; - group normalization. ### Batch normalization as reparameterization Batch normalization is not only a statistical standardization step. It changes the parameterization of the network. Since the batch mean and variance depend on the current mini-batch, the layer also introduces stochasticity during training. ### Caution for small batches For medical imaging and high-resolution segmentation, batch size may be small. Batch normalization can then be unstable. Group normalization or instance normalization may be preferable. ## 21.20 Dropout, weight decay, and data augmentation CNNs are usually trained with several regularization methods. ### Weight decay Weight decay adds a penalty $$ \lambda\|\theta\|_2^2 $$ to the empirical risk. For a loss $\ell$, training solves approximately $$ \min_\theta \frac{1}{n}\sum_{i=1}^n \ell(f_\theta(x_i),y_i)+\lambda\|\theta\|_2^2. $$ ### Dropout Dropout randomly masks activations during training. If $m$ is a random Bernoulli mask, then $$ \widetilde h = m\odot h. $$ This discourages excessive co-adaptation of units. ### Data augmentation Data augmentation applies transformations such as cropping, flipping, rotation, color jitter, blur, and noise. Mathematically, augmentation replaces the empirical risk by $$ \frac{1}{n}\sum_{i=1}^n \mathbb E_{T\sim\mathcal T} \ell(f_\theta(Tx_i),y_i). $$ This encourages invariance to transformations in $\mathcal T$. ## 21.21 Classical CNN architectures This section is not a history lesson only. Each architecture introduced a mathematical or optimization idea. ### LeNet LeNet-style networks used convolution, pooling, and gradient-based learning for document recognition [@lecun1998gradient]. The key idea was to combine local filters with weight sharing. ### AlexNet AlexNet used deeper CNNs, ReLU activations, data augmentation, dropout, and GPU training to achieve strong large-scale image classification performance [@krizhevsky2012imagenet]. ### VGG VGG-style networks emphasized repeated small $3\times3$ convolutions. The mathematical lesson is that stacked small kernels can replace larger kernels while adding depth and nonlinearity. ### Inception Inception-style modules use multiple filter sizes in parallel. The idea is multi-scale processing: different receptive fields capture different spatial structures. ### ResNet Residual networks use skip connections: $$ X\mapsto X+F_\theta(X). $$ This changes the optimization landscape and enables very deep networks [@he2015resnet]. ### DenseNet DenseNet-style networks concatenate features from earlier layers, encouraging feature reuse and direct gradient flow. ### MobileNet and efficient CNNs MobileNet-style architectures use depthwise separable convolutions. A standard convolution has parameter count $$ k_hk_wC_{\text{in}}C_{\text{out}}. $$ A depthwise separable convolution has approximately $$ k_hk_wC_{\text{in}} + C_{\text{in}}C_{\text{out}} $$ parameters. This is much smaller when $C_{\text{out}}$ is large. ## 21.22 Depthwise separable convolution A standard convolution mixes spatial and channel information simultaneously: $$ Y[i,j,q] =\sum_{a,b,c}K[a,b,c,q]X[i+a,j+b,c]. $$ A depthwise separable convolution separates this into two steps. First, a depthwise spatial convolution applies one spatial filter per input channel: $$ U[i,j,c]=\sum_{a,b}D[a,b,c]X[i+a,j+b,c]. $$ Second, a pointwise $1\times1$ convolution mixes channels: $$ Y[i,j,q]=\sum_c P[c,q]U[i,j,c]. $$ This is a structured low-parameter factorization of a standard convolutional operator. It is central to efficient mobile and edge-device vision. ## 21.23 CNNs for classification, detection, and segmentation CNNs can be adapted to many vision tasks. ### Classification A classification CNN maps $$ X\mapsto z\mapsto \operatorname{softmax}(z). $$ The final vector gives class probabilities. ### Detection Detection requires both classification and localization: $$ X\mapsto \{(\widehat b_m,\widehat c_m,\widehat s_m)\}_{m=1}^M, $$ where $\widehat b_m$ is a bounding box, $\widehat c_m$ is a class, and $\widehat s_m$ is a confidence score. ### Segmentation Segmentation produces a spatial prediction: $$ X\mapsto \widehat Y\in \{1,\ldots,K\}^{H\times W}. $$ Fully convolutional networks and U-Net architectures use convolutional encoder-decoder structures to preserve spatial information [@ronneberger2015unet]. ### Feature pyramid networks Detection and segmentation often require multi-scale features. A feature pyramid builds representations at multiple spatial resolutions, combining semantic depth with spatial precision. ## 21.24 Statistical learning perspective Let the training data be $$ \{(X_i,Y_i)\}_{i=1}^n\sim P. $$ A CNN class $\mathcal F$ contains functions $f_\theta$. Training solves empirical risk minimization: $$ \widehat\theta \in \arg\min_\theta \frac{1}{n}\sum_{i=1}^n \ell(f_\theta(X_i),Y_i). $$ The population risk is $$ R(\theta)=\mathbb E[\ell(f_\theta(X),Y)]. $$ The mathematical questions are: 1. **approximation:** can a CNN represent a good predictor? 2. **optimization:** can training find useful parameters? 3. **generalization:** will low training risk imply low population risk? 4. **robustness:** will the predictor remain stable under distribution shift? ### CNN prior as a model class restriction CNNs do not search over all functions from images to labels. They search over functions with locality, parameter sharing, equivariance, and hierarchical composition. This restriction can reduce sample complexity when the assumptions are appropriate. ### Distribution shift Computer vision datasets often differ from deployment data. For example: - scanner type changes in medical imaging; - lighting changes in autonomous driving; - geography changes in satellite imaging; - demographics change in face analysis; - acquisition protocols change in microscopy. Mathematically, training uses $P_{\text{train}}$, but deployment uses $P_{\text{test}}$. Generalization across this shift is not guaranteed. ## 21.25 CNNs and vision transformers CNNs and vision transformers use different inductive biases. A CNN begins with local spatial operations: $$ Y[i,j]=\psi(\mathcal P_{i,j}(X)). $$ A vision transformer begins by splitting an image into patches and applying attention across tokens: $$ \operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^T}{\sqrt d}\right)V. $$ CNNs encode locality and translation structure directly. Transformers can learn long-range interactions more flexibly, but may require more data or pretraining unless locality is reintroduced through architectural choices. Modern architectures often mix both ideas: convolutional stems, local attention, hierarchical token pyramids, patch merging, and convolutional positional encodings. ## 21.26 Interpretability and feature visualization CNN interpretability methods attempt to understand what a network has learned. ### Activation maps Given a feature map $X^{(\ell)}$, one can visualize each channel $$ X^{(\ell)}[:,:,c] $$ as a spatial response map. Large values indicate locations where that feature is active. ### Saliency maps For a class score $s_k(X)$, a saliency map uses $$ \left|\frac{\partial s_k}{\partial X[i,j,c]}\right| $$ to estimate sensitivity to input pixels. ### Class activation maps If the final convolutional feature map is $A_c(i,j)$ and the classifier weight for class $k$ is $w_{k,c}$, a class activation map is $$ M_k(i,j)=\sum_c w_{k,c}A_c(i,j). $$ This highlights spatial regions supporting class $k$. ::: {.callout-warning} ## Interpretability caution A visualization is not a proof. Saliency and activation maps are diagnostic tools. They can suggest hypotheses, reveal artifacts, or catch failures, but they should not be treated as definitive explanations. ::: ## 21.27 Robustness and adversarial perturbations A small perturbation $\delta$ may change the prediction: $$ f_\theta(X+\delta)\ne f_\theta(X), \qquad \|\delta\| \text{ small}. $$ An adversarial example solves approximately $$ \max_{\|\delta\|\le \epsilon} \ell(f_\theta(X+\delta),y). $$ Using a first-order Taylor approximation, $$ \ell(f_\theta(X+\delta),y) \approx \ell(f_\theta(X),y)+\nabla_X\ell(f_\theta(X),y)^T\delta. $$ For an $\ell_\infty$ constraint, the maximizing perturbation is approximately $$ \delta=\epsilon\operatorname{sign}(\nabla_X\ell(f_\theta(X),y)). $$ This is the fast gradient sign idea. It shows that high-dimensional image classifiers can be sensitive to carefully aligned perturbations. ### Robustness as mathematical stability Robustness asks for bounds such as $$ \|f_\theta(X)-f_\theta(X')\| \le L\|X-X'\|. $$ Small Lipschitz constants, margin-based training, data augmentation, adversarial training, and certified robustness methods all attempt to improve stability. ## 21.28 Python example: convolution by hand ```{python} #| label: ch21-numpy-convolution #| eval: false import numpy as np import matplotlib.pyplot as plt rng = np.random.default_rng(7243) # Synthetic grayscale image H, W = 64, 64 x = np.zeros((H, W)) x[18:46, 25:39] = 1.0 x += 0.08 * rng.normal(size=(H, W)) # A Sobel-like horizontal derivative kernel K = np.array([ [-1, 0, 1], [-2, 0, 2], [-1, 0, 1] ], dtype=float) def correlate2d_same(image, kernel): kh, kw = kernel.shape ph, pw = kh // 2, kw // 2 padded = np.pad(image, ((ph, ph), (pw, pw)), mode="edge") out = np.zeros_like(image, dtype=float) for i in range(image.shape[0]): for j in range(image.shape[1]): patch = padded[i:i+kh, j:j+kw] out[i, j] = np.sum(kernel * patch) return out y = correlate2d_same(x, K) fig, axes = plt.subplots(1, 2, figsize=(9, 4)) axes[0].imshow(x, cmap="gray") axes[0].set_title("input image") axes[0].axis("off") axes[1].imshow(y, cmap="gray") axes[1].set_title("filter response") axes[1].axis("off") plt.show() ``` This example uses cross-correlation, which is the operation used by most deep-learning convolution layers. ## 21.29 Python example: parameter counting ```{python} #| label: ch21-parameter-counting #| eval: false def conv_params(kh, kw, cin, cout, bias=True): return kh * kw * cin * cout + (cout if bias else 0) def dense_params(input_dim, output_dim, bias=True): return input_dim * output_dim + (output_dim if bias else 0) H, W, C = 224, 224, 3 cout = 64 print("Dense parameters:", dense_params(H * W * C, cout)) print("3x3 convolution parameters:", conv_params(3, 3, C, cout)) print("7x7 convolution parameters:", conv_params(7, 7, C, cout)) ``` Students should compare the parameter counts and explain the difference using parameter sharing. ## 21.30 TensorFlow example: a small CNN classifier ```{python} #| label: ch21-tensorflow-small-cnn #| eval: false import tensorflow as tf from tensorflow import keras from tensorflow.keras import layers model = keras.Sequential([ layers.Input(shape=(32, 32, 3)), layers.Conv2D(32, kernel_size=3, padding="same", activation="relu"), layers.MaxPooling2D(pool_size=2), layers.Conv2D(64, kernel_size=3, padding="same", activation="relu"), layers.MaxPooling2D(pool_size=2), layers.Conv2D(128, kernel_size=3, padding="same", activation="relu"), layers.GlobalAveragePooling2D(), layers.Dense(10, activation="softmax") ]) model.compile( optimizer=keras.optimizers.Adam(learning_rate=1e-3), loss="sparse_categorical_crossentropy", metrics=["accuracy"] ) model.summary() ``` Important mathematical observations: 1. convolutional layers preserve spatial structure; 2. pooling reduces spatial resolution; 3. global average pooling produces a vector representation; 4. the final dense layer is a multiclass logistic regression on learned features. ## 21.31 TensorFlow example: convolutional block with batch normalization and residual connection ```{python} #| label: ch21-tensorflow-residual-block #| eval: false import tensorflow as tf from tensorflow import keras from tensorflow.keras import layers class ResidualBlock(layers.Layer): def __init__(self, channels, stride=1): super().__init__() self.conv1 = layers.Conv2D(channels, 3, strides=stride, padding="same", use_bias=False) self.bn1 = layers.BatchNormalization() self.conv2 = layers.Conv2D(channels, 3, padding="same", use_bias=False) self.bn2 = layers.BatchNormalization() self.proj = None if stride != 1: self.proj = keras.Sequential([ layers.Conv2D(channels, 1, strides=stride, padding="same", use_bias=False), layers.BatchNormalization() ]) def call(self, x, training=False): shortcut = x if self.proj is None else self.proj(x, training=training) z = self.conv1(x) z = self.bn1(z, training=training) z = tf.nn.relu(z) z = self.conv2(z) z = self.bn2(z, training=training) return tf.nn.relu(z + shortcut) inputs = keras.Input(shape=(64, 64, 3)) x = layers.Conv2D(32, 3, padding="same", activation="relu")(inputs) x = ResidualBlock(32)(x) x = ResidualBlock(64, stride=2)(x) x = layers.GlobalAveragePooling2D()(x) outputs = layers.Dense(5, activation="softmax")(x) model = keras.Model(inputs, outputs) model.summary() ``` This example shows how a residual network changes spatial resolution while maintaining identity or projected shortcut paths. ## 21.32 TensorFlow example: feature-map visualization ```{python} #| label: ch21-feature-map-visualization #| eval: false import numpy as np import matplotlib.pyplot as plt import tensorflow as tf from tensorflow import keras from tensorflow.keras import layers # Suppose model is a trained Keras CNN. # Extract the output of the first convolutional layer. conv_layer = model.layers[0] feature_model = keras.Model(inputs=model.input, outputs=conv_layer.output) # x_image should have shape (1, H, W, C). features = feature_model(x_image, training=False).numpy()[0] num_channels = min(8, features.shape[-1]) fig, axes = plt.subplots(1, num_channels, figsize=(2 * num_channels, 2)) for c in range(num_channels): axes[c].imshow(features[:, :, c], cmap="viridis") axes[c].set_title(f"channel {c}") axes[c].axis("off") plt.show() ``` Feature-map visualization is useful for debugging. For example, if all feature maps are nearly zero, the model may have an optimization or preprocessing problem. ## 21.33 AI component: using AI systems responsibly while learning CNNs AI tools can help students explore CNNs, but they should be used as mathematical assistants rather than answer generators. ### Useful prompts 1. **Equivariance check.** "Given the definition of translation $T_\tau$ and convolution $K*X$, verify the equivariance proof step by step. Point out where boundary effects enter." 2. **Receptive-field calculation.** "Compute the receptive field of a CNN with layers $(3,1,1)$, $(3,2,1)$, $(3,1,2)$, where each triple is kernel size, stride, dilation. Show the recursion." 3. **Architecture critique.** "This CNN has four max-pooling layers on a $32\times32$ image. Explain what spatial resolution remains and whether this is appropriate for segmentation." 4. **Debugging.** "My CNN training accuracy increases but validation accuracy decreases. List statistical explanations and architecture-level fixes." 5. **Model comparison.** "Compare CNNs and vision transformers for small medical imaging datasets. Focus on inductive bias, sample size, augmentation, and interpretability." ### Verification habit When using AI to reason about CNNs, always verify: - tensor shapes; - parameter counts; - output sizes; - loss definitions; - whether the claimed operation is convolution or cross-correlation; - whether the architecture preserves enough spatial resolution for the task. ## 21.34 Common pitfalls ### Pitfall 1: confusing convolution and cross-correlation Most software uses cross-correlation but calls it convolution. This is usually harmless for learned kernels but matters when manually implementing known filters. ### Pitfall 2: ignoring output shape Stride and pooling can quickly reduce spatial resolution. For dense prediction tasks such as segmentation, too much downsampling can destroy boundary precision. ### Pitfall 3: treating augmentation as free data Augmentation encodes invariances. If the invariance is false, augmentation can harm performance. For example, horizontal flipping may be invalid in medical imaging or digit recognition depending on the label semantics. ### Pitfall 4: using batch normalization with tiny batches Mini-batch statistics can be noisy when batch size is very small. This is common in 3D medical imaging and high-resolution segmentation. ### Pitfall 5: believing feature maps are explanations Feature maps and saliency maps are diagnostic tools. They do not automatically provide causal explanations. ## 21.35 Summary CNNs are mathematically structured neural networks for image data. Their core principles are: $$ \text{locality}, \qquad \text{translation equivariance}, \qquad \text{parameter sharing}, \qquad \text{hierarchical composition}. $$ A convolutional layer is a learned local linear operator followed by a nonlinearity. Deep CNNs compose many such layers to build increasingly abstract representations. Pooling and global aggregation convert equivariant feature maps into approximately invariant predictions. Residual connections and normalization layers improve optimization and enable deeper architectures. From a statistical point of view, CNNs impose a useful inductive bias on high-dimensional image learning problems. From an applied-mathematics point of view, they are nonlinear compositions of structured linear operators. From a modern AI point of view, they remain essential building blocks for classification, detection, segmentation, medical imaging, remote sensing, robotics, and hybrid CNN-transformer systems. ## 21.36 Exercises ### Conceptual exercises 1. Explain the difference between translation equivariance and translation invariance. Give one example of a vision task where equivariance is useful and one where invariance is useful. 2. Why does parameter sharing reduce the effective complexity of a model? Explain from both a computational and statistical viewpoint. 3. Suppose a CNN uses four $2\times2$ max-pooling layers with stride two. If the input image is $128\times128$, what is the spatial size after the pooling layers? 4. Why might global average pooling be preferable to flattening followed by a large dense layer? 5. Explain why a $1\times1$ convolution is not trivial in a multi-channel feature map. ### Mathematical exercises 6. Prove that a pointwise activation $\sigma$ commutes with translations: $$ \sigma(T_\tau X)=T_\tau\sigma(X). $$ Use this to show that a convolution followed by a pointwise nonlinearity is translation equivariant. 7. For a convolutional layer with input channels $C_{\text{in}}=32$, output channels $C_{\text{out}}=64$, and kernel size $3\times3$, compute the number of weights and biases. 8. A CNN has three convolutional layers. The first has kernel size 3 and stride 1. The second has kernel size 3 and stride 2. The third has kernel size 5 and stride 1. Compute the receptive-field size and effective jump after each layer. 9. Derive the gradient $$ \frac{\partial L}{\partial K[a,b,c,q]} $$ for a multi-channel cross-correlation layer. 10. Show that a depthwise separable convolution has fewer parameters than a standard convolution when $$ k_hk_wC_{\text{in}}C_{\text{out}} > k_hk_wC_{\text{in}}+C_{\text{in}}C_{\text{out}}. $$ Interpret this inequality. ### Python exercises 11. Implement a two-dimensional convolution or cross-correlation function using only NumPy. Compare its output with `scipy.signal.correlate2d` or a TensorFlow convolution. 12. Train a small CNN on a simple image dataset such as MNIST, Fashion-MNIST, or CIFAR-10. Record training and validation accuracy. Try at least two architectures. 13. Visualize first-layer filters and first-layer feature maps. Describe what patterns appear. 14. Change kernel size from $3\times3$ to $5\times5$. Compare parameter counts, training behavior, and validation accuracy. 15. Replace max pooling by average pooling. Compare performance and feature-map smoothness. ### AI-assisted exercises 16. Ask an AI system to compute the receptive field of a CNN architecture you design. Then verify every step manually. 17. Ask an AI system to explain why a CNN is translation equivariant. Identify any missing assumptions in the explanation. 18. Ask an AI system to generate a Keras CNN for semantic segmentation. Check whether the output tensor has the correct spatial dimensions. 19. Ask an AI system to compare CNNs and ViTs for a small biomedical dataset. Rewrite the answer in mathematical terms using inductive bias, sample complexity, and regularization. 20. Ask an AI system to debug a CNN with poor validation performance. Separate the suggestions into optimization issues, statistical issues, data issues, and architecture issues. ## 21.37 Notes and further reading Convolutional neural networks are discussed in classical and modern forms in @lecun1998gradient, @krizhevsky2012imagenet, @goodfellow2016deep, and @he2015resnet. For broad computer-vision context, see @szeliski2022computer and @prince2012computer. For segmentation-oriented CNNs, see @ronneberger2015unet and fully convolutional networks. For the modern transition from CNNs to attention-based vision models, compare this chapter with the later chapter on vision transformers.

22.1 21.1 Why convolutional neural networks belong in a mathematical vision book

22.1.1 Learning goals

22.2 21.2 Images as lattice-indexed tensors

22.2.1 Locality assumption

22.3 21.3 Convolution and cross-correlation

22.3.1 Multi-channel convolution

22.4 21.4 Translation equivariance

22.4.1 Proposition 21.1: convolution is translation equivariant

22.4.2 Equivariance versus invariance

22.5 21.5 Interactive demo: convolution and translation equivariance

Convolution response shifts with the input

22.6 21.6 A convolutional layer as an affine map plus nonlinearity

22.6.1 Why nonlinearities are necessary

22.7 21.7 Interactive demo: activation functions

Activation functions

22.8 21.8 Output size: padding, stride, and dilation

22.8.1 Boundary effects

22.9 21.9 Receptive fields

22.9.1 Example: three \(3\times 3\) convolutions

22.10 21.10 Interactive demo: receptive field growth

Receptive field growth

22.11 21.11 Pooling and local invariance

22.11.1 Global average pooling

22.11.2 Pooling as a statistical estimator

22.12 21.12 Parameter sharing and statistical efficiency

22.12.1 Bias-variance viewpoint

22.13 21.13 Interactive demo: parameter sharing

Dense layer versus convolutional layer parameter count

22.14 21.14 CNNs as compositions of equivariant maps

22.14.1 Hierarchical representation

22.15 21.15 Matrix view: convolution as a sparse linear operator

22.15.1 Operator norm and stability

22.16 21.16 Backpropagation through convolution

22.16.1 Multi-channel version

22.17 21.17 Residual connections

22.17.1 Why residual connections help

22.18 21.18 Interactive demo: residual connections and gradients

Residual connection intuition

22.19 21.19 Normalization layers

22.19.1 Batch normalization as reparameterization

22.19.2 Caution for small batches

22.20 21.20 Dropout, weight decay, and data augmentation

22.20.1 Weight decay

22.20.2 Dropout

22.20.3 Data augmentation

22.21 21.21 Classical CNN architectures

22.21.1 LeNet

22.21.2 AlexNet

22.21.3 VGG

22.21.4 Inception

22.21.5 ResNet

22.21.6 DenseNet

22.21.7 MobileNet and efficient CNNs

22.22 21.22 Depthwise separable convolution

22.23 21.23 CNNs for classification, detection, and segmentation

22.23.1 Classification

22.23.2 Detection

22.23.3 Segmentation

22.23.4 Feature pyramid networks

22.24 21.24 Statistical learning perspective

22.24.1 CNN prior as a model class restriction

22.24.2 Distribution shift

22.25 21.25 CNNs and vision transformers

22.26 21.26 Interpretability and feature visualization

22.26.1 Activation maps

22.26.2 Saliency maps

22.26.3 Class activation maps

22.27 21.27 Robustness and adversarial perturbations

22.27.1 Robustness as mathematical stability

22.28 21.28 Python example: convolution by hand

22.29 21.29 Python example: parameter counting

22.30 21.30 TensorFlow example: a small CNN classifier

22.31 21.31 TensorFlow example: convolutional block with batch normalization and residual connection

22.32 21.32 TensorFlow example: feature-map visualization

22.33 21.33 AI component: using AI systems responsibly while learning CNNs

22.33.1 Useful prompts

22.33.2 Verification habit

22.34 21.34 Common pitfalls

22.34.1 Pitfall 1: confusing convolution and cross-correlation

22.34.2 Pitfall 2: ignoring output shape