2 What Is Computer Vision?

Computer vision studies how visual measurements can be transformed into useful information. The measurements may be photographs, video frames, medical images, satellite images, microscopy images, depth maps, point clouds, or image–text pairs. The output may be a label, a mask, a geometric reconstruction, a motion field, a decision, or a generated image.

From a mathematical point of view, computer vision is the study of maps involving images: maps from scenes to measurements, maps from images to geometric structures, maps from data to statistical decisions, and maps learned from examples.

2.1 Learning Goals

After reading this chapter, students should be able to:

describe an image as an array, a tensor, a sampled function, a measurement, and a random object;
explain why computer vision problems are usually inverse problems;
distinguish low-level, mid-level, high-level, geometric, and generative vision tasks;
identify the roles of linear algebra, calculus, probability, statistics, optimization, and geometry in vision;
formulate a modern vision model as a parameterized map trained by empirical risk minimization;
explain why classical mathematical vision remains useful even in the era of deep learning and foundation models.

2.2 1.1 The Central Question

A human can look at an image and immediately say something meaningful: there is a dog, the road curves left, the tumor boundary is irregular, the microscope image contains several cells, or the camera moved forward. A computer sees only numbers. Computer vision asks how useful information can be inferred from these numbers.

A first abstraction is:

\[ \text{scene} \longrightarrow \text{image} \longrightarrow \text{interpretation}. \]

Let $S$ denote a physical scene, $I$ the observed image, and $Y$ the desired output. Then a vision problem can often be written as

\[ S \xrightarrow{\text{measurement}} I \xrightarrow{\text{inference}} Y. \]

The measurement map may be a camera, a CT scanner, an MRI machine, a microscope, a satellite sensor, or a simulated renderer. The inference map may be hand-designed, estimated statistically, optimized variationally, or learned from data.

Examples of outputs include:

Task	Input	Output
Classification	image $I$	label $Y \in \{1,\ldots,K\}$
Regression	image $I$	vector $Y \in \mathbb{R}^d$
Semantic segmentation	image $I$	mask $Y \in \{1,\ldots,K\}^{H\times W}$
Object detection	image $I$	finite set of boxes and labels
Depth estimation	image $I$	depth map $D \in \mathbb{R}^{H\times W}$
Optical flow	frames $I_t,I_{t+1}$	vector field $u:\Omega\to\mathbb{R}^2$
Pose estimation	image or video	element of $SE(3)$ or joint coordinates
Image generation	text, noise, or condition	synthetic image $I$

The table already shows one reason computer vision is mathematically rich: the output space is not always a simple class label. Many outputs are structured objects: functions, sets, graphs, surfaces, transformations, probability distributions, or sequences.

A useful first slogan is:

\[ \boxed{\text{Computer vision is structured inference from visual measurements.}} \]

The word structured is important. A segmentation mask has spatial structure, a depth map has geometric structure, a video has temporal structure, and a 3D reconstruction has projective and Euclidean structure.

2.3 1.2 Computer Vision as an Inverse Problem

Many vision problems are inverse problems. A forward problem predicts the measurement from the object. An inverse problem tries to recover information about the object from the measurement.

A simple linear measurement model is

\[ I = A x + \varepsilon, \]

where:

$x$ is an unknown object or scene representation;
$A$ is a measurement operator;
$I$ is the observed image;
$\varepsilon$ is noise.

The inverse problem is to estimate $x$ or some function of $x$ from $I$. In ordinary least squares, one might solve

\[ \min_x \|Ax-I\|_2^2. \]

In vision this is usually too simple. The operator $A$ may be nonlinear, the unknown object may be high-dimensional, and the inverse problem may be ill-posed. Different scenes may produce nearly identical images. Occlusion removes information. Noise corrupts measurements. Lighting changes intensities. Perspective projection loses depth.

For example, in a pinhole camera, a 3D point $(X,Y,Z)$ projects to approximately

\[ u = f\frac{X}{Z}, \qquad v = f\frac{Y}{Z}, \]

where $f$ is the focal length. The absolute scale of $(X,Y,Z)$ is not directly visible from one image. The points $(X,Y,Z)$ and $(\lambda X,\lambda Y,\lambda Z)$ project to the same image point for every $\lambda>0$. Therefore, reconstructing depth from a single image requires additional assumptions, learned priors, multiple views, motion, shading, or other information.

For MS Statistics students, the inverse-problem viewpoint connects vision to identifiability. If two different scenes can generate the same image distribution, then no estimator can perfectly distinguish them without extra assumptions. The assumptions may be physical, geometric, probabilistic, or learned from data.

2.4 1.3 Images as Arrays and Tensors

The most direct representation of a grayscale image is a matrix

\[ I \in \mathbb{R}^{H\times W}, \]

where $H$ is the height and $W$ is the width. The entry $I_{ij}$ is the intensity at row $i$ and column $j$. A color image is often represented as a third-order tensor

\[ I \in \mathbb{R}^{H\times W\times C}, \]

where $C=3$ for red, green, and blue channels.

This representation turns an image into a finite-dimensional mathematical object. If we vectorize the image, then

\[ \operatorname{vec}(I) \in \mathbb{R}^{HWC}. \]

For a $224\times 224$ RGB image, this vector has

\[ 224\cdot 224\cdot 3 = 150{,}528 \]

coordinates. A dataset of $n$ images can therefore be viewed as a matrix

\[ X \in \mathbb{R}^{n\times d}, \qquad d=HWC, \]

where each row is one image vector. This is the standard data-matrix viewpoint used in statistics and machine learning.

However, vectorization hides spatial structure. Nearby pixels are no longer obviously nearby after vectorization. This is why convolutional and patch-based architectures are important: they use the grid structure of the image rather than treating all coordinates as exchangeable.

The notation $I_{ij}$ usually means row $i$ and column $j$, while Cartesian coordinates are often written as $(x,y)$. In image processing, the vertical image coordinate increases downward, while in standard calculus the $y$-axis increases upward. Many implementation errors come from mixing these conventions.

2.5 1.4 Images as Sampled Functions

Although a digital image is an array, it is often useful to imagine an underlying continuous function

\[ I:\Omega\subset \mathbb{R}^2 \to \mathbb{R}^C. \]

Here $\Omega$ is the image domain, such as a rectangle in the plane, and $C$ is the number of channels. The pixel array is obtained by sampling the function on a grid.

For a grayscale image, $C=1$ and

\[ I(x,y) \in \mathbb{R}. \]

For an RGB image, $C=3$ and

\[ I(x,y) = \bigl(I_R(x,y), I_G(x,y), I_B(x,y)\bigr). \]

The function viewpoint explains why calculus appears throughout vision. The gradient

\[ \nabla I(x,y) = \begin{bmatrix} I_x(x,y)\\ I_y(x,y) \end{bmatrix} \]

measures local change. Edges often occur where $\|\nabla I(x,y)\|$ is large. A second derivative or Laplacian

\[ \Delta I = I_{xx}+I_{yy} \]

can detect changes in curvature or local contrast.

For a digital image, derivatives are approximated. A simple horizontal finite difference is

\[ I_x(i,j) \approx I(i,j+1)-I(i,j), \]

and a simple vertical finite difference is

\[ I_y(i,j) \approx I(i+1,j)-I(i,j). \]

More stable derivative estimates use smoothing before differentiating. This leads naturally to convolution, Gaussian filters, and scale space.

2.6 1.5 Images as Measurements

Images are not just arrays; they are measurements of something. A photograph measures light arriving at a sensor. A medical image measures physical responses produced by a scanner. A satellite image measures reflected or emitted radiation. A microscope image measures an optical interaction between biological structures and a microscope system.

A generic observation model is

\[ I = M(S) + \varepsilon, \]

where $M$ is a measurement map and $\varepsilon$ is measurement noise. This model is intentionally broad. Different parts of computer vision study different choices of $M$.

In geometric vision, $M$ is often a camera projection. In image restoration, $M$ may include blur, downsampling, or missing pixels. In medical imaging, $M$ may be a tomographic transform or a more complicated physical acquisition process. In deep learning, $M$ is often not modeled explicitly, but the model still learns from samples produced by the real measurement process.

A simple blur model is

\[ I = K * I_0 + \varepsilon, \]

where $I_0$ is a sharp image, $K$ is a blur kernel, $*$ denotes convolution, and $\varepsilon$ is noise. Deblurring asks for $I_0$ from $I$. Since high frequencies may be suppressed by $K$, deblurring is unstable without regularization.

A regularized restoration problem may have the form

\[ \min_U \frac{1}{2}\|K*U-I\|_2^2 + \lambda R(U), \]

where $R(U)$ encodes a preference for smoothness, sparsity, piecewise constancy, or another image prior. This kind of formulation is a bridge between classical variational vision and modern learned priors.

2.7 1.6 Images as Random Objects

For statistical analysis, it is useful to treat images as random objects. A dataset

\[ \{(I_i,Y_i)\}_{i=1}^n \]

can be modeled as independent samples from an unknown distribution

\[ (I,Y) \sim P. \]

The goal is to learn a decision rule $f$ with small expected loss

\[ R(f)=\mathbb{E}_{(I,Y)\sim P}\bigl[L(f(I),Y)\bigr]. \]

Since $P$ is unknown, we minimize the empirical risk

\[ \widehat R_n(f)=\frac{1}{n}\sum_{i=1}^n L(f(I_i),Y_i). \]

This is the statistical learning formulation of vision. It applies to classification, segmentation, detection, depth estimation, and many other tasks once the output space and loss function are specified.

For example, in image classification with $K$ classes, a model may output a probability vector

\[ p_\theta(I) = \bigl(p_1(I),\ldots,p_K(I)\bigr), \qquad p_k(I)\ge 0, \qquad \sum_{k=1}^K p_k(I)=1. \]

For a one-hot class label $y$, the cross-entropy loss is

\[ L(p_\theta(I),y)=-\sum_{k=1}^K y_k\log p_k(I). \]

In segmentation, the same idea is applied at every pixel:

\[ L(p_\theta(I),Y) = -\sum_{(i,j)\in\Omega_h}\sum_{k=1}^K Y_{ijk}\log p_{ijk}(I), \]

where $\Omega_h$ is the discrete pixel grid.

This formulation makes computer vision a natural extension of statistical learning. The main differences are dimensionality, spatial dependence, structured outputs, and the need to respect geometry and invariance.

2.8 1.7 Images as Points on or Near Low-Dimensional Structure

A $224\times224$ RGB image lives in $\mathbb{R}^{150528}$ after vectorization. But natural images do not fill this whole space uniformly. They have structure: local smoothness, edges, repeated textures, object shapes, perspective geometry, and semantic regularities.

A common mathematical intuition is that natural images concentrate near a lower-dimensional set inside the ambient pixel space. This set is often informally called the natural image manifold. The word manifold should be used carefully: the actual set of natural images may have singularities, boundaries, multiple components, and complicated stratified structure. Still, the idea is useful.

If $z\in\mathbb{R}^m$ is a latent variable and $G$ is a generator, then a generative model represents images as

\[ I \approx G(z), \qquad m \ll HWC. \]

In representation learning, an encoder maps images to feature vectors

\[ \phi_\theta(I)\in\mathbb{R}^d. \]

The hope is that the feature space is more useful than raw pixel space. Images with similar semantic meaning should be close, separable, or geometrically organized in the representation space.

This viewpoint connects modern vision to linear algebra, manifold learning, metric geometry, optimal transport, and high-dimensional probability.

2.9 1.8 Three Basic Types of Vision Maps

Many computer vision algorithms can be understood as maps between spaces. Three types appear repeatedly.

2.9.1 Image-to-Image Maps

An image-to-image map transforms one image into another image:

\[ T:\mathbb{R}^{H\times W\times C}\to \mathbb{R}^{H\times W\times C'}. \]

Examples include denoising, deblurring, super-resolution, edge detection, style transfer, and image generation. Some maps are linear, such as convolution by a fixed kernel. Others are nonlinear, such as median filtering or deep neural networks.

2.9.2 Image-to-Structure Maps

An image-to-structure map produces a structured object:

\[ T(I) \in \text{structures}. \]

Examples include contours, keypoints, segmentation masks, bounding boxes, depth maps, surface normals, skeletons, and camera poses. These outputs have constraints. A rotation matrix must satisfy $R^TR=I$ and $\det R=1$. A segmentation mask has spatial consistency. A set of boxes has permutation symmetry.

2.9.3 Image-to-Representation Maps

A representation map produces a feature vector:

\[ \phi_\theta:\mathbb{R}^{H\times W\times C}\to \mathbb{R}^d. \]

The representation may be used for classification, retrieval, clustering, captioning, matching, or multimodal alignment. Modern vision-language models learn image embeddings that can be compared with text embeddings. This makes images and words live in a shared geometric space.

2.10 1.9 A First Geometric Example: Camera Projection

Geometry enters vision because images are formed by projecting three-dimensional scenes onto two-dimensional sensors. In homogeneous coordinates, a 3D point is written as

\[ X= \begin{bmatrix} X_1\\X_2\\X_3\\1 \end{bmatrix} \in\mathbb{P}^3, \]

and an image point is written as

\[ x= \begin{bmatrix} u\\v\\1 \end{bmatrix} \in\mathbb{P}^2. \]

A projective camera is a matrix

\[ P\in\mathbb{R}^{3\times4} \]

such that

\[ x \sim PX. \]

The symbol $\sim$ means equality up to nonzero scale. If

\[ PX= \begin{bmatrix} \tilde u\\ \tilde v\\ \tilde w \end{bmatrix}, \]

then the visible image coordinates are

\[ u=\frac{\tilde u}{\tilde w}, \qquad v=\frac{\tilde v}{\tilde w}. \]

This small formula is the starting point for camera calibration, stereo vision, epipolar geometry, structure from motion, and bundle adjustment. It also shows why linear algebra alone is not enough. The projective relation is linear in homogeneous coordinates, but the final image coordinates involve division by depth.

Classic multi-view geometry develops this subject systematically using projective geometry, matrix factorizations, and nonlinear optimization (Hartley and Zisserman 2004).

2.11 1.10 A First Statistical Example: Classification

Suppose we have training images and labels

\[ (I_1,y_1),\ldots,(I_n,y_n), \qquad y_i\in\{1,\ldots,K\}. \]

A classifier is a function

\[ f:\mathbb{R}^{H\times W\times C}\to\{1,\ldots,K\}. \]

A probabilistic classifier instead estimates

\[ p(y=k\mid I), \qquad k=1,\ldots,K. \]

A linear classifier after vectorization has scores

\[ s_k(I)=w_k^T\operatorname{vec}(I)+b_k. \]

The predicted class is

\[ \widehat y = \arg\max_k s_k(I). \]

A deep model replaces the linear score with a nonlinear feature map:

\[ s_k(I)=a_k^T\phi_\theta(I)+b_k. \]

The central statistical question is not only whether the model fits the training data. The central question is whether it generalizes to new images from the target population. This raises questions about sampling, distribution shift, augmentation, calibration, uncertainty, and robustness.

2.12 1.11 A First Optimization Example: Denoising

Let $I$ be a noisy grayscale image and let $U$ be the unknown clean image. A basic denoising model solves

\[ \min_U \frac{1}{2}\|U-I\|_2^2 + \frac{\lambda}{2}\|\nabla U\|_2^2. \]

The first term says that $U$ should stay close to the observed image. The second term says that $U$ should be smooth. The parameter $\lambda>0$ controls the tradeoff.

In discrete form, $\nabla U$ is built from finite differences. If $D_x$ and $D_y$ are horizontal and vertical difference operators, then

\[ \|\nabla U\|_2^2 = \|D_xU\|_2^2+\|D_yU\|_2^2. \]

After vectorizing $U$ and $I$, the problem becomes a quadratic minimization:

\[ \min_u \frac{1}{2}\|u-i\|_2^2 + \frac{\lambda}{2}\|Du\|_2^2. \]

The normal equation is

\[ (I_d+\lambda D^TD)u=i, \]

where $I_d$ is the identity matrix in pixel space. This is a concrete example of how image processing leads to linear algebra, numerical methods, and regularization.

Denoising illustrates a recurring template:

\[ \boxed{\text{data fidelity} + \text{regularization}.} \]

The data term connects the unknown to the measurement. The regularizer encodes prior knowledge about desirable images.

2.13 1.12 A First Deep Learning Example: Learned Feature Maps

Classical computer vision often begins with designed features: edges, corners, gradients, textures, and geometric correspondences. Modern deep learning learns features from data.

A neural network for vision is a composition of maps

\[ f_\theta = T_L\circ T_{L-1}\circ\cdots\circ T_1. \]

Each layer transforms an intermediate representation. In a convolutional neural network, a typical layer has the form

\[ Z_{\ell+1} = \sigma(K_\ell * Z_\ell + b_\ell), \]

where $K_\ell$ is a learned convolutional kernel, $*$ denotes convolution, $b_\ell$ is a bias, and $\sigma$ is a nonlinear activation function.

In a vision transformer, an image is split into patches, each patch is embedded as a vector, and self-attention mixes information across patches. A simplified attention layer uses

\[ \operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^T}{\sqrt d}\right)V. \]

The transformer view treats an image as a sequence of patch tokens. The convolutional view treats an image as a grid signal. Both are mathematical choices about how to represent and process visual information. Modern computer vision uses both ideas extensively (LeCun et al. 1998; Krizhevsky et al. 2012; Dosovitskiy et al. 2020).

2.14 1.13 The Main Mathematical Languages of Vision

Computer vision is not one mathematical subject. It is a meeting point of several mathematical languages.

Linear algebra

Images as vectors, filters as linear maps, least squares, PCA, SVD, low-rank approximation, embeddings, attention matrices.

Calculus and analysis

Gradients, edges, optical flow, variational energies, differential operators, smoothing, scale space.

Probability and statistics

Noise models, likelihoods, priors, Bayesian inference, empirical risk, generalization, calibration, uncertainty.

Optimization

Energy minimization, graph cuts, gradient descent, stochastic optimization, bundle adjustment, training neural networks.

Geometry

Projective geometry, cameras, rigid motions, depth, epipolar geometry, 3D reconstruction, manifolds.

Learning and representation

CNNs, transformers, contrastive learning, embeddings, foundation models, diffusion models, multimodal vision.

A clean mathematical explanation of computer vision should show how these languages interact. For example, stereo reconstruction uses geometry to define constraints, statistics to handle noise, and optimization to estimate depth. Segmentation uses probability to define pixelwise likelihoods, optimization to find masks, and deep learning to learn representations.

2.15 1.14 Low-Level, Mid-Level, High-Level, and Generative Vision

It is useful to classify vision tasks by the kind of output they produce.

2.15.1 Low-Level Vision

Low-level vision transforms images into images or local measurements:

\[ I \mapsto T(I). \]

Examples include denoising, deblurring, super-resolution, sharpening, edge detection, and optical flow. These tasks often rely on local operators, convolution, calculus, and inverse problems.

2.15.2 Mid-Level Vision

Mid-level vision extracts structures:

\[ I \mapsto \text{regions, boundaries, keypoints, depth, motion}. \]

Examples include segmentation, feature detection, stereo matching, tracking, and shape extraction. These tasks often combine geometry, graphs, and optimization.

2.15.3 High-Level Vision

High-level vision produces semantic or decision-oriented outputs:

\[ I \mapsto \text{label, box, caption, answer, action}. \]

Examples include classification, detection, recognition, captioning, visual question answering, and robotics perception. These tasks often rely on statistical learning and large datasets.

2.15.4 Generative Vision

Generative vision models the distribution of images or produces new images:

\[ z \mapsto I, \qquad \text{or} \qquad \text{text prompt} \mapsto I. \]

Diffusion models, autoregressive image models, and other generative systems learn complicated distributions over images (Ho et al. 2020). In generative vision, the output is not merely an interpretation of an image; the output may itself be an image.

2.16 1.15 Why Classical Vision Still Matters

Modern vision is strongly influenced by deep learning, but classical mathematical vision remains essential.

First, many modern architectures encode classical ideas. Convolutions encode locality and translation equivariance. Pooling encodes a form of stability. Attention computes data-dependent weighted averages. Segmentation losses formalize pixelwise classification and region overlap.

Second, geometry is still necessary whenever cameras, pose, depth, and 3D structure matter. A neural network may predict depth, but the meaning of depth is geometric.

Third, classical models reveal failure modes. A vision system that ignores illumination, occlusion, scale, or perspective may fail in predictable ways. Mathematical models help us name these problems.

Fourth, many scientific and medical applications require interpretability, uncertainty quantification, and validation. In such settings, a black-box prediction is often not enough.

Finally, mathematical understanding improves the use of AI tools. Students should not merely ask an AI system to solve a vision problem. They should be able to specify the correct mathematical formulation, the assumptions, the loss, the output space, and the evaluation metric.

2.17 1.16 A Minimal Python Preview

The following code is not intended to replace a full lab. It previews three basic ideas that will appear repeatedly: image arrays, gradients, and low-rank approximation.

import numpy as np
import matplotlib.pyplot as plt

# Create a simple synthetic grayscale image.
H, W = 80, 120
y = np.linspace(-1, 1, H)
x = np.linspace(-1, 1, W)
X, Y = np.meshgrid(x, y)

image = np.exp(-5 * (X**2 + Y**2)) + 0.25 * np.sin(12 * X)

# Finite-difference gradients.
Ix = np.zeros_like(image)
Iy = np.zeros_like(image)
Ix[:, :-1] = image[:, 1:] - image[:, :-1]
Iy[:-1, :] = image[1:, :] - image[:-1, :]
grad_mag = np.sqrt(Ix**2 + Iy**2)

# Low-rank approximation by SVD.
U, s, Vt = np.linalg.svd(image, full_matrices=False)
r = 5
image_rank_r = (U[:, :r] * s[:r]) @ Vt[:r, :]

fig, axes = plt.subplots(1, 3, figsize=(12, 3.5))
axes[0].imshow(image, cmap="gray")
axes[0].set_title("Synthetic image")
axes[1].imshow(grad_mag, cmap="gray")
axes[1].set_title("Gradient magnitude")
axes[2].imshow(image_rank_r, cmap="gray")
axes[2].set_title("Rank-5 approximation")
for ax in axes:
    ax.axis("off")
plt.tight_layout()
plt.show()

The code illustrates three mathematical views:

image is a matrix;
grad_mag approximates $\|\nabla I\|$;
image_rank_r uses the best rank-$r$ approximation from linear algebra.

2.18 1.17 How to Read This Book

This book is written for MA Applied Mathematics and MS Statistics students. The goal is not to memorize a long list of algorithms. The goal is to understand the mathematical structure underneath them.

When reading a new method, ask the following questions.

What is the input space? Is the input an image, video, point cloud, image–text pair, or measurement vector?
What is the output space? Is the output a label, vector, mask, geometry, distribution, or generated image?
What assumptions are being used? Smoothness, sparsity, low rank, projective geometry, independence, equivariance, or learned priors?
What is optimized or estimated? A least-squares objective, likelihood, posterior, empirical risk, contrastive loss, or diffusion objective?
What can go wrong? Noise, occlusion, bias, distribution shift, non-identifiability, adversarial perturbation, poor calibration, or wrong geometry?

These questions help keep the mathematics visible.

2.19 1.18 Chapter Summary

Computer vision can be understood through several compatible mathematical viewpoints:

\[ \begin{aligned} \text{image} &\equiv \text{array or tensor},\\ \text{image} &\equiv \text{sampled function},\\ \text{image} &\equiv \text{physical measurement},\\ \text{image} &\equiv \text{random object},\\ \text{vision model} &\equiv \text{map from images to structured outputs}. \end{aligned} \]

The same image can be studied with linear algebra, calculus, probability, statistics, optimization, and geometry. Classical vision explains how images relate to signals, shapes, and cameras. Modern learning explains how representations can be learned from data. Foundation models extend vision to large-scale multimodal and generative systems.

The organizing principle for the rest of the book is:

\[ \boxed{\text{vision} = \text{measurement} + \text{structure} + \text{inference} + \text{learning}.} \]

2.20 Exercises

2.20.1 Conceptual Exercises

Give three different mathematical representations of a grayscale image. What information is emphasized by each representation?
Explain why estimating depth from a single image is an inverse problem. What information is lost in perspective projection?
For each output below, identify whether it is best viewed as a scalar, vector, function, set, matrix, tensor, graph, or probability distribution.
1. class label
2. segmentation mask
3. optical flow field
4. object detections
5. camera rotation
6. image embedding
Why does vectorizing an image lose spatial structure? How do convolutional neural networks partially restore that structure?
In what sense is a segmentation problem both a statistical learning problem and a spatial optimization problem?

2.20.2 Mathematical Exercises

Let $I\in\mathbb{R}^{H\times W}$ be a grayscale image. Define the forward differences

\[ (D_xI)_{ij}=I_{i,j+1}-I_{ij}, \qquad (D_yI)_{ij}=I_{i+1,j}-I_{ij}. \]

Write an expression for the discrete gradient magnitude at pixel $(i,j)$.
Consider the regularized denoising objective

\[ J(u)=\frac{1}{2}\|u-i\|_2^2+ \frac{\lambda}{2}\|Du\|_2^2. \]

Derive the normal equation for the minimizer.
Suppose a pinhole camera maps $(X,Y,Z)$ to $(u,v)$ by

\[ u=fX/Z, \qquad v=fY/Z. \]

Show that $(X,Y,Z)$ and $(\lambda X,\lambda Y,\lambda Z)$ produce the same image point for every $\lambda>0$.
Let a classifier have scores $s_k(I)$ for $k=1,\ldots,K$. Define the softmax probabilities

\[ p_k(I)=\frac{\exp(s_k(I))}{\sum_{j=1}^K\exp(s_j(I))}. \]

Verify that $p_k(I)\ge 0$ and $\sum_k p_k(I)=1$.
Let $I\in\mathbb{R}^{H\times W}$ have singular values $\sigma_1\ge\sigma_2\ge\cdots\ge 0$. What is the rank-$r$ approximation error in Frobenius norm?

2.20.3 Computational Exercises

Create a synthetic image using NumPy. Compute finite-difference gradients and display the gradient magnitude.
Add Gaussian noise to a synthetic image. Try simple averaging filters of different sizes. Describe the tradeoff between noise removal and edge preservation.
Compute the SVD of a grayscale image. Plot the reconstruction using ranks $1,5,10,20$. What types of image structure are captured first?
Simulate the pinhole projection of points on a cube. Change the depth values and observe how the projected coordinates change.
Construct a tiny classification dataset where each image is either a vertical bar or a horizontal bar. Train logistic regression on vectorized pixels. Then explain the learned weights as an image.

2.20.4 Discussion Exercises

Choose one application area: medical imaging, autonomous driving, satellite imaging, microscopy, sports analytics, or document analysis. Identify the measurement process, the output space, the likely noise sources, and one evaluation metric.
Find an example where a vision system might be accurate on average but unreliable for a specific subgroup, imaging device, environment, or lighting condition. Explain this as a distribution-shift problem.
Explain why a foundation model for images should still be evaluated using task-specific metrics and domain-specific assumptions.

2.21 Notes and Further Reading

Szeliski’s Computer Vision: Algorithms and Applications gives a broad modern overview of classical and learning-based computer vision (Szeliski 2022). Hartley and Zisserman’s Multiple View Geometry in Computer Vision is the standard reference for projective and multi-view geometry (Hartley and Zisserman 2004). Prince’s Computer Vision: Models, Learning, and Inference emphasizes probabilistic modeling (Prince 2012). Gonzalez and Woods provide a classic treatment of image processing fundamentals (Gonzalez and Woods 2018). For statistical learning and deep learning background, see Bishop (Bishop 2006) and Goodfellow, Bengio, and Courville (Goodfellow et al. 2016). Modern vision models build on convolutional networks (LeCun et al. 1998; Krizhevsky et al. 2012), U-Net-style segmentation (Ronneberger et al. 2015), vision transformers (Dosovitskiy et al. 2020), vision-language representation learning (Radford et al. 2021), promptable segmentation (Kirillov et al. 2023), and diffusion models (Ho et al. 2020).

Bishop, Christopher M. 2006. Pattern Recognition and Machine Learning. Springer.

Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, et al. 2020. “An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.” arXiv Preprint arXiv:2010.11929. https://arxiv.org/abs/2010.11929.

Gonzalez, Rafael C., and Richard E. Woods. 2018. Digital Image Processing. 4th ed. Pearson.

Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. https://www.deeplearningbook.org/.

Hartley, Richard, and Andrew Zisserman. 2004. Multiple View Geometry in Computer Vision. 2nd ed. Cambridge University Press. https://www.robots.ox.ac.uk/~vgg/hzbook/.

Ho, Jonathan, Ajay Jain, and Pieter Abbeel. 2020. “Denoising Diffusion Probabilistic Models.” Advances in Neural Information Processing Systems 33: 6840–51. https://arxiv.org/abs/2006.11239.

Kirillov, Alexander, Eric Mintun, Nikhila Ravi, et al. 2023. “Segment Anything.” Proceedings of the IEEE/CVF International Conference on Computer Vision. https://arxiv.org/abs/2304.02643.

Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. 2012. “ImageNet Classification with Deep Convolutional Neural Networks.” Advances in Neural Information Processing Systems.

LeCun, Yann, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. “Gradient-Based Learning Applied to Document Recognition.” Proceedings of the IEEE 86: 2278–324.

Prince, Simon J. D. 2012. Computer Vision: Models, Learning, and Inference. Cambridge University Press. https://www.computervisionmodels.com/.

Radford, Alec, Jong Wook Kim, Chris Hallacy, et al. 2021. “Learning Transferable Visual Models from Natural Language Supervision.” International Conference on Machine Learning. https://arxiv.org/abs/2103.00020.

Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. 2015. “U-Net: Convolutional Networks for Biomedical Image Segmentation.” Medical Image Computing and Computer-Assisted Intervention.

Szeliski, Richard. 2022. Computer Vision: Algorithms and Applications. 2nd ed. Springer. https://szeliski.org/Book/.

# What Is Computer Vision? ::: {.hero-box} Computer vision studies how visual measurements can be transformed into useful information. The measurements may be photographs, video frames, medical images, satellite images, microscopy images, depth maps, point clouds, or image--text pairs. The output may be a label, a mask, a geometric reconstruction, a motion field, a decision, or a generated image. From a mathematical point of view, computer vision is the study of maps involving images: maps from scenes to measurements, maps from images to geometric structures, maps from data to statistical decisions, and maps learned from examples. ::: ## Learning Goals After reading this chapter, students should be able to: 1. describe an image as an array, a tensor, a sampled function, a measurement, and a random object; 2. explain why computer vision problems are usually inverse problems; 3. distinguish low-level, mid-level, high-level, geometric, and generative vision tasks; 4. identify the roles of linear algebra, calculus, probability, statistics, optimization, and geometry in vision; 5. formulate a modern vision model as a parameterized map trained by empirical risk minimization; 6. explain why classical mathematical vision remains useful even in the era of deep learning and foundation models. ## 1.1 The Central Question A human can look at an image and immediately say something meaningful: there is a dog, the road curves left, the tumor boundary is irregular, the microscope image contains several cells, or the camera moved forward. A computer sees only numbers. Computer vision asks how useful information can be inferred from these numbers. A first abstraction is: $$ \text{scene} \longrightarrow \text{image} \longrightarrow \text{interpretation}. $$ Let $S$ denote a physical scene, $I$ the observed image, and $Y$ the desired output. Then a vision problem can often be written as $$ S \xrightarrow{\text{measurement}} I \xrightarrow{\text{inference}} Y. $$ The measurement map may be a camera, a CT scanner, an MRI machine, a microscope, a satellite sensor, or a simulated renderer. The inference map may be hand-designed, estimated statistically, optimized variationally, or learned from data. Examples of outputs include: | Task | Input | Output | |---|---|---| | Classification | image $I$ | label $Y \in \{1,\ldots,K\}$ | | Regression | image $I$ | vector $Y \in \mathbb{R}^d$ | | Semantic segmentation | image $I$ | mask $Y \in \{1,\ldots,K\}^{H\times W}$ | | Object detection | image $I$ | finite set of boxes and labels | | Depth estimation | image $I$ | depth map $D \in \mathbb{R}^{H\times W}$ | | Optical flow | frames $I_t,I_{t+1}$ | vector field $u:\Omega\to\mathbb{R}^2$ | | Pose estimation | image or video | element of $SE(3)$ or joint coordinates | | Image generation | text, noise, or condition | synthetic image $I$ | The table already shows one reason computer vision is mathematically rich: the output space is not always a simple class label. Many outputs are structured objects: functions, sets, graphs, surfaces, transformations, probability distributions, or sequences. ::: {.math-spine} A useful first slogan is: $$ \boxed{\text{Computer vision is structured inference from visual measurements.}} $$ The word *structured* is important. A segmentation mask has spatial structure, a depth map has geometric structure, a video has temporal structure, and a 3D reconstruction has projective and Euclidean structure. ::: ## 1.2 Computer Vision as an Inverse Problem Many vision problems are inverse problems. A forward problem predicts the measurement from the object. An inverse problem tries to recover information about the object from the measurement. A simple linear measurement model is $$ I = A x + \varepsilon, $$ where: - $x$ is an unknown object or scene representation; - $A$ is a measurement operator; - $I$ is the observed image; - $\varepsilon$ is noise. The inverse problem is to estimate $x$ or some function of $x$ from $I$. In ordinary least squares, one might solve $$ \min_x \|Ax-I\|_2^2. $$ In vision this is usually too simple. The operator $A$ may be nonlinear, the unknown object may be high-dimensional, and the inverse problem may be ill-posed. Different scenes may produce nearly identical images. Occlusion removes information. Noise corrupts measurements. Lighting changes intensities. Perspective projection loses depth. For example, in a pinhole camera, a 3D point $(X,Y,Z)$ projects to approximately $$ u = f\frac{X}{Z}, \qquad v = f\frac{Y}{Z}, $$ where $f$ is the focal length. The absolute scale of $(X,Y,Z)$ is not directly visible from one image. The points $(X,Y,Z)$ and $(\lambda X,\lambda Y,\lambda Z)$ project to the same image point for every $\lambda>0$. Therefore, reconstructing depth from a single image requires additional assumptions, learned priors, multiple views, motion, shading, or other information. ::: {.stats-note} For MS Statistics students, the inverse-problem viewpoint connects vision to identifiability. If two different scenes can generate the same image distribution, then no estimator can perfectly distinguish them without extra assumptions. The assumptions may be physical, geometric, probabilistic, or learned from data. ::: ## 1.3 Images as Arrays and Tensors The most direct representation of a grayscale image is a matrix $$ I \in \mathbb{R}^{H\times W}, $$ where $H$ is the height and $W$ is the width. The entry $I_{ij}$ is the intensity at row $i$ and column $j$. A color image is often represented as a third-order tensor $$ I \in \mathbb{R}^{H\times W\times C}, $$ where $C=3$ for red, green, and blue channels. <div id="cv-image-matrix-demo" class="plotly-demo"></div> This representation turns an image into a finite-dimensional mathematical object. If we vectorize the image, then $$ \operatorname{vec}(I) \in \mathbb{R}^{HWC}. $$ For a $224\times 224$ RGB image, this vector has $$ 224\cdot 224\cdot 3 = 150{,}528 $$ coordinates. A dataset of $n$ images can therefore be viewed as a matrix $$ X \in \mathbb{R}^{n\times d}, \qquad d=HWC, $$ where each row is one image vector. This is the standard data-matrix viewpoint used in statistics and machine learning. However, vectorization hides spatial structure. Nearby pixels are no longer obviously nearby after vectorization. This is why convolutional and patch-based architectures are important: they use the grid structure of the image rather than treating all coordinates as exchangeable. ::: {.warning-note} The notation $I_{ij}$ usually means row $i$ and column $j$, while Cartesian coordinates are often written as $(x,y)$. In image processing, the vertical image coordinate increases downward, while in standard calculus the $y$-axis increases upward. Many implementation errors come from mixing these conventions. ::: ## 1.4 Images as Sampled Functions Although a digital image is an array, it is often useful to imagine an underlying continuous function $$ I:\Omega\subset \mathbb{R}^2 \to \mathbb{R}^C. $$ Here $\Omega$ is the image domain, such as a rectangle in the plane, and $C$ is the number of channels. The pixel array is obtained by sampling the function on a grid. For a grayscale image, $C=1$ and $$ I(x,y) \in \mathbb{R}. $$ For an RGB image, $C=3$ and $$ I(x,y) = \bigl(I_R(x,y), I_G(x,y), I_B(x,y)\bigr). $$ The function viewpoint explains why calculus appears throughout vision. The gradient $$ \nabla I(x,y) = \begin{bmatrix} I_x(x,y)\\ I_y(x,y) \end{bmatrix} $$ measures local change. Edges often occur where $\|\nabla I(x,y)\|$ is large. A second derivative or Laplacian $$ \Delta I = I_{xx}+I_{yy} $$ can detect changes in curvature or local contrast. For a digital image, derivatives are approximated. A simple horizontal finite difference is $$ I_x(i,j) \approx I(i,j+1)-I(i,j), $$ and a simple vertical finite difference is $$ I_y(i,j) \approx I(i+1,j)-I(i,j). $$ More stable derivative estimates use smoothing before differentiating. This leads naturally to convolution, Gaussian filters, and scale space. ## 1.5 Images as Measurements Images are not just arrays; they are measurements of something. A photograph measures light arriving at a sensor. A medical image measures physical responses produced by a scanner. A satellite image measures reflected or emitted radiation. A microscope image measures an optical interaction between biological structures and a microscope system. A generic observation model is $$ I = M(S) + \varepsilon, $$ where $M$ is a measurement map and $\varepsilon$ is measurement noise. This model is intentionally broad. Different parts of computer vision study different choices of $M$. In geometric vision, $M$ is often a camera projection. In image restoration, $M$ may include blur, downsampling, or missing pixels. In medical imaging, $M$ may be a tomographic transform or a more complicated physical acquisition process. In deep learning, $M$ is often not modeled explicitly, but the model still learns from samples produced by the real measurement process. A simple blur model is $$ I = K * I_0 + \varepsilon, $$ where $I_0$ is a sharp image, $K$ is a blur kernel, $*$ denotes convolution, and $\varepsilon$ is noise. Deblurring asks for $I_0$ from $I$. Since high frequencies may be suppressed by $K$, deblurring is unstable without regularization. <div id="cv-convolution-demo" class="plotly-demo"></div> A regularized restoration problem may have the form $$ \min_U \frac{1}{2}\|K*U-I\|_2^2 + \lambda R(U), $$ where $R(U)$ encodes a preference for smoothness, sparsity, piecewise constancy, or another image prior. This kind of formulation is a bridge between classical variational vision and modern learned priors. ## 1.6 Images as Random Objects For statistical analysis, it is useful to treat images as random objects. A dataset $$ \{(I_i,Y_i)\}_{i=1}^n $$ can be modeled as independent samples from an unknown distribution $$ (I,Y) \sim P. $$ The goal is to learn a decision rule $f$ with small expected loss $$ R(f)=\mathbb{E}_{(I,Y)\sim P}\bigl[L(f(I),Y)\bigr]. $$ Since $P$ is unknown, we minimize the empirical risk $$ \widehat R_n(f)=\frac{1}{n}\sum_{i=1}^n L(f(I_i),Y_i). $$ This is the statistical learning formulation of vision. It applies to classification, segmentation, detection, depth estimation, and many other tasks once the output space and loss function are specified. For example, in image classification with $K$ classes, a model may output a probability vector $$ p_\theta(I) = \bigl(p_1(I),\ldots,p_K(I)\bigr), \qquad p_k(I)\ge 0, \qquad \sum_{k=1}^K p_k(I)=1. $$ For a one-hot class label $y$, the cross-entropy loss is $$ L(p_\theta(I),y)=-\sum_{k=1}^K y_k\log p_k(I). $$ In segmentation, the same idea is applied at every pixel: $$ L(p_\theta(I),Y) = -\sum_{(i,j)\in\Omega_h}\sum_{k=1}^K Y_{ijk}\log p_{ijk}(I), $$ where $\Omega_h$ is the discrete pixel grid. ::: {.stats-note} This formulation makes computer vision a natural extension of statistical learning. The main differences are dimensionality, spatial dependence, structured outputs, and the need to respect geometry and invariance. ::: ## 1.7 Images as Points on or Near Low-Dimensional Structure A $224\times224$ RGB image lives in $\mathbb{R}^{150528}$ after vectorization. But natural images do not fill this whole space uniformly. They have structure: local smoothness, edges, repeated textures, object shapes, perspective geometry, and semantic regularities. A common mathematical intuition is that natural images concentrate near a lower-dimensional set inside the ambient pixel space. This set is often informally called the *natural image manifold*. The word manifold should be used carefully: the actual set of natural images may have singularities, boundaries, multiple components, and complicated stratified structure. Still, the idea is useful. If $z\in\mathbb{R}^m$ is a latent variable and $G$ is a generator, then a generative model represents images as $$ I \approx G(z), \qquad m \ll HWC. $$ In representation learning, an encoder maps images to feature vectors $$ \phi_\theta(I)\in\mathbb{R}^d. $$ The hope is that the feature space is more useful than raw pixel space. Images with similar semantic meaning should be close, separable, or geometrically organized in the representation space. This viewpoint connects modern vision to linear algebra, manifold learning, metric geometry, optimal transport, and high-dimensional probability. ## 1.8 Three Basic Types of Vision Maps Many computer vision algorithms can be understood as maps between spaces. Three types appear repeatedly. ### Image-to-Image Maps An image-to-image map transforms one image into another image: $$ T:\mathbb{R}^{H\times W\times C}\to \mathbb{R}^{H\times W\times C'}. $$ Examples include denoising, deblurring, super-resolution, edge detection, style transfer, and image generation. Some maps are linear, such as convolution by a fixed kernel. Others are nonlinear, such as median filtering or deep neural networks. ### Image-to-Structure Maps An image-to-structure map produces a structured object: $$ T(I) \in \text{structures}. $$ Examples include contours, keypoints, segmentation masks, bounding boxes, depth maps, surface normals, skeletons, and camera poses. These outputs have constraints. A rotation matrix must satisfy $R^TR=I$ and $\det R=1$. A segmentation mask has spatial consistency. A set of boxes has permutation symmetry. ### Image-to-Representation Maps A representation map produces a feature vector: $$ \phi_\theta:\mathbb{R}^{H\times W\times C}\to \mathbb{R}^d. $$ The representation may be used for classification, retrieval, clustering, captioning, matching, or multimodal alignment. Modern vision-language models learn image embeddings that can be compared with text embeddings. This makes images and words live in a shared geometric space. ## 1.9 A First Geometric Example: Camera Projection Geometry enters vision because images are formed by projecting three-dimensional scenes onto two-dimensional sensors. In homogeneous coordinates, a 3D point is written as $$ X= \begin{bmatrix} X_1\\X_2\\X_3\\1 \end{bmatrix} \in\mathbb{P}^3, $$ and an image point is written as $$ x= \begin{bmatrix} u\\v\\1 \end{bmatrix} \in\mathbb{P}^2. $$ A projective camera is a matrix $$ P\in\mathbb{R}^{3\times4} $$ such that $$ x \sim PX. $$ The symbol $\sim$ means equality up to nonzero scale. If $$ PX= \begin{bmatrix} \tilde u\\ \tilde v\\ \tilde w \end{bmatrix}, $$ then the visible image coordinates are $$ u=\frac{\tilde u}{\tilde w}, \qquad v=\frac{\tilde v}{\tilde w}. $$ <div id="cv-camera-projection-demo" class="plotly-demo"></div> This small formula is the starting point for camera calibration, stereo vision, epipolar geometry, structure from motion, and bundle adjustment. It also shows why linear algebra alone is not enough. The projective relation is linear in homogeneous coordinates, but the final image coordinates involve division by depth. Classic multi-view geometry develops this subject systematically using projective geometry, matrix factorizations, and nonlinear optimization [@hartley2004multiple]. ## 1.10 A First Statistical Example: Classification Suppose we have training images and labels $$ (I_1,y_1),\ldots,(I_n,y_n), \qquad y_i\in\{1,\ldots,K\}. $$ A classifier is a function $$ f:\mathbb{R}^{H\times W\times C}\to\{1,\ldots,K\}. $$ A probabilistic classifier instead estimates $$ p(y=k\mid I), \qquad k=1,\ldots,K. $$ A linear classifier after vectorization has scores $$ s_k(I)=w_k^T\operatorname{vec}(I)+b_k. $$ The predicted class is $$ \widehat y = \arg\max_k s_k(I). $$ A deep model replaces the linear score with a nonlinear feature map: $$ s_k(I)=a_k^T\phi_\theta(I)+b_k. $$ The central statistical question is not only whether the model fits the training data. The central question is whether it generalizes to new images from the target population. This raises questions about sampling, distribution shift, augmentation, calibration, uncertainty, and robustness. ## 1.11 A First Optimization Example: Denoising Let $I$ be a noisy grayscale image and let $U$ be the unknown clean image. A basic denoising model solves $$ \min_U \frac{1}{2}\|U-I\|_2^2 + \frac{\lambda}{2}\|\nabla U\|_2^2. $$ The first term says that $U$ should stay close to the observed image. The second term says that $U$ should be smooth. The parameter $\lambda>0$ controls the tradeoff. In discrete form, $\nabla U$ is built from finite differences. If $D_x$ and $D_y$ are horizontal and vertical difference operators, then $$ \|\nabla U\|_2^2 = \|D_xU\|_2^2+\|D_yU\|_2^2. $$ After vectorizing $U$ and $I$, the problem becomes a quadratic minimization: $$ \min_u \frac{1}{2}\|u-i\|_2^2 + \frac{\lambda}{2}\|Du\|_2^2. $$ The normal equation is $$ (I_d+\lambda D^TD)u=i, $$ where $I_d$ is the identity matrix in pixel space. This is a concrete example of how image processing leads to linear algebra, numerical methods, and regularization. ::: {.math-spine} Denoising illustrates a recurring template: $$ \boxed{\text{data fidelity} + \text{regularization}.} $$ The data term connects the unknown to the measurement. The regularizer encodes prior knowledge about desirable images. ::: ## 1.12 A First Deep Learning Example: Learned Feature Maps Classical computer vision often begins with designed features: edges, corners, gradients, textures, and geometric correspondences. Modern deep learning learns features from data. A neural network for vision is a composition of maps $$ f_\theta = T_L\circ T_{L-1}\circ\cdots\circ T_1. $$ Each layer transforms an intermediate representation. In a convolutional neural network, a typical layer has the form $$ Z_{\ell+1} = \sigma(K_\ell * Z_\ell + b_\ell), $$ where $K_\ell$ is a learned convolutional kernel, $*$ denotes convolution, $b_\ell$ is a bias, and $\sigma$ is a nonlinear activation function. In a vision transformer, an image is split into patches, each patch is embedded as a vector, and self-attention mixes information across patches. A simplified attention layer uses $$ \operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^T}{\sqrt d}\right)V. $$ The transformer view treats an image as a sequence of patch tokens. The convolutional view treats an image as a grid signal. Both are mathematical choices about how to represent and process visual information. Modern computer vision uses both ideas extensively [@lecun1998gradient; @krizhevsky2012imagenet; @dosovitskiy2020image]. ## 1.13 The Main Mathematical Languages of Vision Computer vision is not one mathematical subject. It is a meeting point of several mathematical languages. ::: {.cv-card-grid} ::: {.cv-card} **Linear algebra** Images as vectors, filters as linear maps, least squares, PCA, SVD, low-rank approximation, embeddings, attention matrices. ::: ::: {.cv-card} **Calculus and analysis** Gradients, edges, optical flow, variational energies, differential operators, smoothing, scale space. ::: ::: {.cv-card} **Probability and statistics** Noise models, likelihoods, priors, Bayesian inference, empirical risk, generalization, calibration, uncertainty. ::: ::: {.cv-card} **Optimization** Energy minimization, graph cuts, gradient descent, stochastic optimization, bundle adjustment, training neural networks. ::: ::: {.cv-card} **Geometry** Projective geometry, cameras, rigid motions, depth, epipolar geometry, 3D reconstruction, manifolds. ::: ::: {.cv-card} **Learning and representation** CNNs, transformers, contrastive learning, embeddings, foundation models, diffusion models, multimodal vision. ::: ::: A clean mathematical explanation of computer vision should show how these languages interact. For example, stereo reconstruction uses geometry to define constraints, statistics to handle noise, and optimization to estimate depth. Segmentation uses probability to define pixelwise likelihoods, optimization to find masks, and deep learning to learn representations. ## 1.14 Low-Level, Mid-Level, High-Level, and Generative Vision It is useful to classify vision tasks by the kind of output they produce. ### Low-Level Vision Low-level vision transforms images into images or local measurements: $$ I \mapsto T(I). $$ Examples include denoising, deblurring, super-resolution, sharpening, edge detection, and optical flow. These tasks often rely on local operators, convolution, calculus, and inverse problems. ### Mid-Level Vision Mid-level vision extracts structures: $$ I \mapsto \text{regions, boundaries, keypoints, depth, motion}. $$ Examples include segmentation, feature detection, stereo matching, tracking, and shape extraction. These tasks often combine geometry, graphs, and optimization. ### High-Level Vision High-level vision produces semantic or decision-oriented outputs: $$ I \mapsto \text{label, box, caption, answer, action}. $$ Examples include classification, detection, recognition, captioning, visual question answering, and robotics perception. These tasks often rely on statistical learning and large datasets. ### Generative Vision Generative vision models the distribution of images or produces new images: $$ z \mapsto I, \qquad \text{or} \qquad \text{text prompt} \mapsto I. $$ Diffusion models, autoregressive image models, and other generative systems learn complicated distributions over images [@ho2020denoising]. In generative vision, the output is not merely an interpretation of an image; the output may itself be an image. ## 1.15 Why Classical Vision Still Matters Modern vision is strongly influenced by deep learning, but classical mathematical vision remains essential. First, many modern architectures encode classical ideas. Convolutions encode locality and translation equivariance. Pooling encodes a form of stability. Attention computes data-dependent weighted averages. Segmentation losses formalize pixelwise classification and region overlap. Second, geometry is still necessary whenever cameras, pose, depth, and 3D structure matter. A neural network may predict depth, but the meaning of depth is geometric. Third, classical models reveal failure modes. A vision system that ignores illumination, occlusion, scale, or perspective may fail in predictable ways. Mathematical models help us name these problems. Fourth, many scientific and medical applications require interpretability, uncertainty quantification, and validation. In such settings, a black-box prediction is often not enough. Finally, mathematical understanding improves the use of AI tools. Students should not merely ask an AI system to solve a vision problem. They should be able to specify the correct mathematical formulation, the assumptions, the loss, the output space, and the evaluation metric. ## 1.16 A Minimal Python Preview The following code is not intended to replace a full lab. It previews three basic ideas that will appear repeatedly: image arrays, gradients, and low-rank approximation. ```{python} #| label: ch01-python-preview #| eval: false import numpy as np import matplotlib.pyplot as plt # Create a simple synthetic grayscale image. H, W = 80, 120 y = np.linspace(-1, 1, H) x = np.linspace(-1, 1, W) X, Y = np.meshgrid(x, y) image = np.exp(-5 * (X**2 + Y**2)) + 0.25 * np.sin(12 * X) # Finite-difference gradients. Ix = np.zeros_like(image) Iy = np.zeros_like(image) Ix[:, :-1] = image[:, 1:] - image[:, :-1] Iy[:-1, :] = image[1:, :] - image[:-1, :] grad_mag = np.sqrt(Ix**2 + Iy**2) # Low-rank approximation by SVD. U, s, Vt = np.linalg.svd(image, full_matrices=False) r = 5 image_rank_r = (U[:, :r] * s[:r]) @ Vt[:r, :] fig, axes = plt.subplots(1, 3, figsize=(12, 3.5)) axes[0].imshow(image, cmap="gray") axes[0].set_title("Synthetic image") axes[1].imshow(grad_mag, cmap="gray") axes[1].set_title("Gradient magnitude") axes[2].imshow(image_rank_r, cmap="gray") axes[2].set_title("Rank-5 approximation") for ax in axes: ax.axis("off") plt.tight_layout() plt.show() ``` The code illustrates three mathematical views: 1. `image` is a matrix; 2. `grad_mag` approximates $\|\nabla I\|$; 3. `image_rank_r` uses the best rank-$r$ approximation from linear algebra. ## 1.17 How to Read This Book This book is written for MA Applied Mathematics and MS Statistics students. The goal is not to memorize a long list of algorithms. The goal is to understand the mathematical structure underneath them. When reading a new method, ask the following questions. 1. **What is the input space?** Is the input an image, video, point cloud, image--text pair, or measurement vector? 2. **What is the output space?** Is the output a label, vector, mask, geometry, distribution, or generated image? 3. **What assumptions are being used?** Smoothness, sparsity, low rank, projective geometry, independence, equivariance, or learned priors? 4. **What is optimized or estimated?** A least-squares objective, likelihood, posterior, empirical risk, contrastive loss, or diffusion objective? 5. **What can go wrong?** Noise, occlusion, bias, distribution shift, non-identifiability, adversarial perturbation, poor calibration, or wrong geometry? These questions help keep the mathematics visible. ## 1.18 Chapter Summary Computer vision can be understood through several compatible mathematical viewpoints: $$ \begin{aligned} \text{image} &\equiv \text{array or tensor},\\ \text{image} &\equiv \text{sampled function},\\ \text{image} &\equiv \text{physical measurement},\\ \text{image} &\equiv \text{random object},\\ \text{vision model} &\equiv \text{map from images to structured outputs}. \end{aligned} $$ The same image can be studied with linear algebra, calculus, probability, statistics, optimization, and geometry. Classical vision explains how images relate to signals, shapes, and cameras. Modern learning explains how representations can be learned from data. Foundation models extend vision to large-scale multimodal and generative systems. The organizing principle for the rest of the book is: $$ \boxed{\text{vision} = \text{measurement} + \text{structure} + \text{inference} + \text{learning}.} $$ ## Exercises ### Conceptual Exercises 1. Give three different mathematical representations of a grayscale image. What information is emphasized by each representation? 2. Explain why estimating depth from a single image is an inverse problem. What information is lost in perspective projection? 3. For each output below, identify whether it is best viewed as a scalar, vector, function, set, matrix, tensor, graph, or probability distribution. a. class label b. segmentation mask c. optical flow field d. object detections e. camera rotation f. image embedding 4. Why does vectorizing an image lose spatial structure? How do convolutional neural networks partially restore that structure? 5. In what sense is a segmentation problem both a statistical learning problem and a spatial optimization problem? ### Mathematical Exercises 6. Let $I\in\mathbb{R}^{H\times W}$ be a grayscale image. Define the forward differences $$ (D_xI)_{ij}=I_{i,j+1}-I_{ij}, \qquad (D_yI)_{ij}=I_{i+1,j}-I_{ij}. $$ Write an expression for the discrete gradient magnitude at pixel $(i,j)$. 7. Consider the regularized denoising objective $$ J(u)=\frac{1}{2}\|u-i\|_2^2+ \frac{\lambda}{2}\|Du\|_2^2. $$ Derive the normal equation for the minimizer. 8. Suppose a pinhole camera maps $(X,Y,Z)$ to $(u,v)$ by $$ u=fX/Z, \qquad v=fY/Z. $$ Show that $(X,Y,Z)$ and $(\lambda X,\lambda Y,\lambda Z)$ produce the same image point for every $\lambda>0$. 9. Let a classifier have scores $s_k(I)$ for $k=1,\ldots,K$. Define the softmax probabilities $$ p_k(I)=\frac{\exp(s_k(I))}{\sum_{j=1}^K\exp(s_j(I))}. $$ Verify that $p_k(I)\ge 0$ and $\sum_k p_k(I)=1$. 10. Let $I\in\mathbb{R}^{H\times W}$ have singular values $\sigma_1\ge\sigma_2\ge\cdots\ge 0$. What is the rank-$r$ approximation error in Frobenius norm? ### Computational Exercises 11. Create a synthetic image using NumPy. Compute finite-difference gradients and display the gradient magnitude. 12. Add Gaussian noise to a synthetic image. Try simple averaging filters of different sizes. Describe the tradeoff between noise removal and edge preservation. 13. Compute the SVD of a grayscale image. Plot the reconstruction using ranks $1,5,10,20$. What types of image structure are captured first? 14. Simulate the pinhole projection of points on a cube. Change the depth values and observe how the projected coordinates change. 15. Construct a tiny classification dataset where each image is either a vertical bar or a horizontal bar. Train logistic regression on vectorized pixels. Then explain the learned weights as an image. ### Discussion Exercises 16. Choose one application area: medical imaging, autonomous driving, satellite imaging, microscopy, sports analytics, or document analysis. Identify the measurement process, the output space, the likely noise sources, and one evaluation metric. 17. Find an example where a vision system might be accurate on average but unreliable for a specific subgroup, imaging device, environment, or lighting condition. Explain this as a distribution-shift problem. 18. Explain why a foundation model for images should still be evaluated using task-specific metrics and domain-specific assumptions. ## Notes and Further Reading Szeliski's *Computer Vision: Algorithms and Applications* gives a broad modern overview of classical and learning-based computer vision [@szeliski2022computer]. Hartley and Zisserman's *Multiple View Geometry in Computer Vision* is the standard reference for projective and multi-view geometry [@hartley2004multiple]. Prince's *Computer Vision: Models, Learning, and Inference* emphasizes probabilistic modeling [@prince2012computer]. Gonzalez and Woods provide a classic treatment of image processing fundamentals [@gonzalez2018digital]. For statistical learning and deep learning background, see Bishop [@bishop2006pattern] and Goodfellow, Bengio, and Courville [@goodfellow2016deep]. Modern vision models build on convolutional networks [@lecun1998gradient; @krizhevsky2012imagenet], U-Net-style segmentation [@ronneberger2015unet], vision transformers [@dosovitskiy2020image], vision-language representation learning [@radford2021clip], promptable segmentation [@kirillov2023segment], and diffusion models [@ho2020denoising].