Introduction

Computer vision is the study of how mathematical and computational systems extract structure from visual data. This book develops that subject from first principles: pixels, coordinates, functions, linear maps, frequencies, geometry, probability, optimization, and modern learned representations.

Computer vision is often presented as a sequence of techniques: filters, features, cameras, neural networks, transformers, and generative models. This book takes a different route. It treats computer vision as a mathematical language for visual information.

At the center of the book is the question:

What mathematical structure allows a machine to see, measure, recognize, reconstruct, segment, track, or generate visual objects?

The answer changes from chapter to chapter. Sometimes the main object is a matrix of pixel values. Sometimes it is a function on a domain, a tensor, a probability distribution, a projective map, a graph of feature correspondences, a segmentation mask, a neural network, or a sequence of image tokens. Modern computer vision combines all of these viewpoints.

The Mathematical View of an Image

A digital image may first appear to be just a rectangular array of numbers. For a grayscale image, one writes

\[ I \in \mathbb{R}^{H \times W}, \]

where $H$ is the image height and $W$ is the image width. For a color image, one writes

\[ I \in \mathbb{R}^{H \times W \times C}, \]

where $C$ is the number of channels. For an RGB image, $C=3$.

But this array view is only one layer. A more mathematical view is to treat an image as a sampled function

\[ I: \Omega \to \mathbb{R}^C, \]

where $\Omega \subset \mathbb{R}^2$ is an image domain. This viewpoint immediately raises questions about sampling, interpolation, smoothing, derivatives, edges, and transformations.

A still deeper view is that an image is a measurement of a scene. A camera maps three-dimensional world points to two-dimensional image points:

\[ X \in \mathbb{R}^3 \longmapsto x \in \mathbb{R}^2. \]

In projective geometry, this becomes a homogeneous matrix map

\[ \tilde{x} \sim P \tilde{X}, \]

where $P$ is a camera matrix. This leads to calibration, epipolar geometry, stereo vision, structure from motion, and bundle adjustment.

In modern machine learning, an image is also an input to a learned map

\[ f_\theta: \text{images} \to \text{predictions, embeddings, masks, captions, or generated images}. \]

The purpose of the book is to help students move fluently among these views.

One Image, Many Mathematical Languages

The same visual object can be studied in several mathematical languages.

View of an image	Mathematical language	Typical computer vision question
Array	linear algebra	What are its norms, subspaces, and low-rank structure?
Sampled function	calculus and analysis	How do smoothing, interpolation, and differentiation behave?
Signal	Fourier analysis	Which frequencies are present, suppressed, or amplified?
Local pattern field	convolution and operators	What edges, corners, textures, or local features appear?
Mask or label field	optimization and variational methods	Which pixels belong to the same object or region?
Measurement of a scene	projective geometry	What 3D structure could have produced the image?
Random object	probability and statistics	What noise, uncertainty, or likelihood model is appropriate?
Training example	statistical learning	What decision rule or representation should be learned?
Token sequence	attention and transformers	How do image patches interact globally?
Image-text pair	multimodal representation learning	How are visual and linguistic meanings aligned?
Noisy sample	diffusion and generative modeling	How can denoising become image generation?

This table is a map of the book. Each chapter develops one or more of these languages and shows how they connect.

Three Organizing Principles

Although computer vision contains many models and algorithms, this book uses three organizing principles.

1. Representation

A representation is a way to encode visual information. Examples include pixels, color channels, gradients, Fourier coefficients, feature descriptors, PCA coordinates, convolutional feature maps, transformer tokens, and multimodal embeddings.

A representation is useful when it makes some task easier. For example, edges are easier to detect after computing image gradients. Objects may be easier to classify after applying a convolutional neural network. Images and text can be compared after mapping them into a shared embedding space.

Mathematically, a representation is often a map

\[ \Phi: \text{visual data} \to \text{feature space}. \]

A major theme of the book is to understand what properties $\Phi$ should preserve: distances, angles, labels, geometry, semantics, invariances, or uncertainty.

2. Geometry and Constraints

Vision is constrained by geometry. Camera projections, epipolar lines, stereo disparities, triangulation, and 3D reconstruction are not arbitrary numerical procedures. They follow from mathematical relationships among points, lines, planes, cameras, and transformations.

For example, corresponding points in two views satisfy an epipolar constraint

\[ \tilde{x}'^T F \tilde{x} = 0, \]

where $F$ is the fundamental matrix. This equation is simple, but it organizes a large part of two-view geometry.

Modern vision models also contain constraints, though they may look different: equivariance, invariance, normalization, attention structure, token interactions, contrastive alignment, and denoising consistency.

3. Optimization and Learning

Many vision tasks are formulated as optimization problems. Segmentation, optical flow, camera calibration, stereo matching, bundle adjustment, neural-network training, and diffusion-model denoising can all be written in terms of objectives, losses, residuals, or likelihoods.

A typical supervised learning objective has the form

\[ \min_\theta \frac{1}{n}\sum_{i=1}^n L(f_\theta(x_i), y_i), \]

where $f_\theta$ is a model, $L$ is a loss function, and $(x_i,y_i)$ are training examples. A classical reconstruction problem may instead minimize a data term plus a regularization term:

\[ \min_u D(u; I) + \lambda R(u). \]

The book repeatedly asks: What is being optimized? What assumptions justify the objective? What failure modes appear when the assumptions are violated?

From Classical Vision to Modern Vision AI

Classical computer vision and modern vision AI are sometimes taught as separate worlds. This book treats them as connected.

Classical vision contributes the language of images, filters, edges, geometry, reconstruction, and variational principles. Modern vision AI contributes learned representations, deep networks, attention, self-supervision, vision-language alignment, promptable models, and generative models. The mathematical bridge between them is built from linear algebra, analysis, probability, optimization, and geometry.

For example:

convolutional neural networks extend the idea of local filters and translation equivariance;
vision transformers reinterpret images as sequences of patches and use attention as a learned interaction operator;
self-supervised models formalize representation learning using invariance, contrastive objectives, reconstruction, or redundancy reduction;
vision-language models align image and text embeddings using geometric relationships in feature space;
diffusion models turn denoising into a generative process.

The goal is not to replace classical ideas with deep learning. The goal is to understand how modern methods inherit, generalize, or transform older mathematical principles.

Structure of the Book

The book is organized into five parts.

Part I. Images as Mathematical Objects

Chapters 1–5 introduce the basic mathematical language of images: pixels, coordinates, color spaces, functions on domains, vector spaces, inner products, SVD, PCA, Fourier analysis, and frequency-domain reasoning.

Part II. Classical Vision: Local Operators, Shape, and Motion

Chapters 6–11 study convolution, filtering, edges, corners, local features, segmentation, morphology, shape, curves, level sets, optical flow, and motion estimation.

Part III. Projective Geometry and 3D Vision

Chapters 12–17 develop cameras as maps, calibration, epipolar geometry, stereo vision, structure from motion, and bundle adjustment.

Part IV. Probability, Statistics, and Learning for Vision

Chapters 18–24 study probabilistic image models, classification, representation learning, convolutional neural networks, training, object detection, and semantic/instance segmentation.

Part V. Modern Vision: Foundation and Generative Models

Chapters 25–30 study vision transformers, self-supervised learning, vision-language models, promptable segmentation, diffusion models, and multimodal AI systems.

How the Computer Labs Support the Book

The book includes one Google Colab lab for each chapter. The labs are stored in the labs/ folder and summarized in labs-summary.qmd.

Each lab is designed for independent study. This means that students should be able to open a notebook in Colab, read the mathematical background, run the code, change the examples, and complete exercises without needing a separate software manual.

A typical lab contains:

a short conceptual overview;
mathematical definitions and notation;
synthetic data or small image examples;
step-by-step Python implementations;
visual interpretation of results;
exercises and reflection questions;
a mini-project connecting the lab to the chapter.

The labs are intentionally small and transparent. They are not meant to hide mathematics inside large software packages. Whenever possible, the code uses NumPy, SciPy, Matplotlib, and simple models before introducing higher-level machine-learning tools.

How to Read This Book

A useful rhythm for each chapter is:

identify the main mathematical objects;
write down the maps between those objects;
identify the objective function, constraint, likelihood, or invariance;
run the corresponding lab;
modify one example;
explain the result in words and formulas.

For example, when studying segmentation, ask what the unknown object is. Is it a binary mask, a multiclass label field, a probability map, or a set of instance masks? Then ask what objective is being optimized and what prior assumptions are being made about region smoothness, boundaries, or class probabilities.

When studying camera geometry, ask which coordinate systems are involved and which quantities are known or unknown. Then identify the map from 3D points to 2D image points and the residual used to measure error.

When studying neural vision models, ask what representation is being learned, what invariances are encouraged, what loss function is optimized, and how the model may fail under distribution shift.

Mathematical Conventions

The book uses the following conventions.

Symbol	Meaning
$I$	image or image function
$H,W$	image height and width
$C$	number of channels
$\Omega$	image domain
$x,y$	image coordinates
$X$	world or 3D point
$K$	camera intrinsic matrix
$R,t$	camera rotation and translation
$P$	camera projection matrix
$f_\theta$	learned model with parameters $\theta$
$L$	loss function
$\Phi$	feature or representation map

The text uses both continuous and discrete notation. This is deliberate. Many computer vision ideas are easiest to understand continuously, but all computations are eventually discrete.

For instance, an image derivative may be written formally as

\[ \nabla I(x,y) = \left(\frac{\partial I}{\partial x}, \frac{\partial I}{\partial y}\right), \]

but in a computer it is approximated using finite differences or convolution filters.

What Students Should Be Able to Do

After completing the book and labs, students should be able to:

describe images as arrays, functions, signals, tensors, random fields, and token sequences;
implement and interpret basic image operations such as filtering, interpolation, Fourier analysis, segmentation, and morphology;
formulate camera projection, calibration, stereo, structure from motion, and bundle adjustment mathematically;
connect classical feature extraction and modern representation learning;
explain CNNs and vision transformers using linear maps, nonlinearities, equivariance, attention, and optimization;
understand how self-supervised learning, vision-language models, promptable segmentation, and diffusion models fit into a unified mathematical framework;
use Python notebooks to test mathematical ideas visually and computationally;
communicate not only what an algorithm does, but why the mathematical structure matters.

A First Guiding Example

Consider a simple task: detect the boundary of an object in an image.

A classical approach computes image gradients and looks for large changes:

\[ \|\nabla I(x,y)\| \text{ is large near an edge.} \]

A variational approach may seek a curve that balances image evidence and smoothness. A probabilistic approach may ask for the posterior probability that a pixel is on a boundary. A neural approach may train a model to predict boundary labels from data. A transformer-based approach may allow distant image regions to interact through attention. A promptable model may ask the user for a point or box and return a mask.

These methods look different, but they share a common structure:

\[ \text{image} \longrightarrow \text{representation} \longrightarrow \text{decision or reconstruction}. \]

This book is about understanding that structure mathematically.

Final Perspective

Computer vision is not only about making machines recognize pictures. It is about representing visual information, imposing geometric and statistical structure, learning useful maps from data, and reasoning under uncertainty.

The mathematics of computer vision is therefore both classical and modern. It includes the geometry of cameras, the analysis of signals, the algebra of tensors, the probability of images, the optimization of losses, and the representation geometry of foundation models.

The goal of this book is to make those ideas visible.

# Introduction {.unnumbered} ::: {.hero-box} Computer vision is the study of how mathematical and computational systems extract structure from visual data. This book develops that subject from first principles: pixels, coordinates, functions, linear maps, frequencies, geometry, probability, optimization, and modern learned representations. ::: Computer vision is often presented as a sequence of techniques: filters, features, cameras, neural networks, transformers, and generative models. This book takes a different route. It treats computer vision as a mathematical language for visual information. At the center of the book is the question: > What mathematical structure allows a machine to see, measure, recognize, reconstruct, segment, track, or generate visual objects? The answer changes from chapter to chapter. Sometimes the main object is a matrix of pixel values. Sometimes it is a function on a domain, a tensor, a probability distribution, a projective map, a graph of feature correspondences, a segmentation mask, a neural network, or a sequence of image tokens. Modern computer vision combines all of these viewpoints. ## The Mathematical View of an Image {.unnumbered} A digital image may first appear to be just a rectangular array of numbers. For a grayscale image, one writes $$ I \in \mathbb{R}^{H \times W}, $$ where $H$ is the image height and $W$ is the image width. For a color image, one writes $$ I \in \mathbb{R}^{H \times W \times C}, $$ where $C$ is the number of channels. For an RGB image, $C=3$. But this array view is only one layer. A more mathematical view is to treat an image as a sampled function $$ I: \Omega \to \mathbb{R}^C, $$ where $\Omega \subset \mathbb{R}^2$ is an image domain. This viewpoint immediately raises questions about sampling, interpolation, smoothing, derivatives, edges, and transformations. A still deeper view is that an image is a measurement of a scene. A camera maps three-dimensional world points to two-dimensional image points: $$ X \in \mathbb{R}^3 \longmapsto x \in \mathbb{R}^2. $$ In projective geometry, this becomes a homogeneous matrix map $$ \tilde{x} \sim P \tilde{X}, $$ where $P$ is a camera matrix. This leads to calibration, epipolar geometry, stereo vision, structure from motion, and bundle adjustment. In modern machine learning, an image is also an input to a learned map $$ f_\theta: \text{images} \to \text{predictions, embeddings, masks, captions, or generated images}. $$ The purpose of the book is to help students move fluently among these views. ## One Image, Many Mathematical Languages {.unnumbered} The same visual object can be studied in several mathematical languages. | View of an image | Mathematical language | Typical computer vision question | |---|---|---| | Array | linear algebra | What are its norms, subspaces, and low-rank structure? | | Sampled function | calculus and analysis | How do smoothing, interpolation, and differentiation behave? | | Signal | Fourier analysis | Which frequencies are present, suppressed, or amplified? | | Local pattern field | convolution and operators | What edges, corners, textures, or local features appear? | | Mask or label field | optimization and variational methods | Which pixels belong to the same object or region? | | Measurement of a scene | projective geometry | What 3D structure could have produced the image? | | Random object | probability and statistics | What noise, uncertainty, or likelihood model is appropriate? | | Training example | statistical learning | What decision rule or representation should be learned? | | Token sequence | attention and transformers | How do image patches interact globally? | | Image-text pair | multimodal representation learning | How are visual and linguistic meanings aligned? | | Noisy sample | diffusion and generative modeling | How can denoising become image generation? | This table is a map of the book. Each chapter develops one or more of these languages and shows how they connect. ## Three Organizing Principles {.unnumbered} Although computer vision contains many models and algorithms, this book uses three organizing principles. ### 1. Representation {.unnumbered} A representation is a way to encode visual information. Examples include pixels, color channels, gradients, Fourier coefficients, feature descriptors, PCA coordinates, convolutional feature maps, transformer tokens, and multimodal embeddings. A representation is useful when it makes some task easier. For example, edges are easier to detect after computing image gradients. Objects may be easier to classify after applying a convolutional neural network. Images and text can be compared after mapping them into a shared embedding space. Mathematically, a representation is often a map $$ \Phi: \text{visual data} \to \text{feature space}. $$ A major theme of the book is to understand what properties $\Phi$ should preserve: distances, angles, labels, geometry, semantics, invariances, or uncertainty. ### 2. Geometry and Constraints {.unnumbered} Vision is constrained by geometry. Camera projections, epipolar lines, stereo disparities, triangulation, and 3D reconstruction are not arbitrary numerical procedures. They follow from mathematical relationships among points, lines, planes, cameras, and transformations. For example, corresponding points in two views satisfy an epipolar constraint $$ \tilde{x}'^T F \tilde{x} = 0, $$ where $F$ is the fundamental matrix. This equation is simple, but it organizes a large part of two-view geometry. Modern vision models also contain constraints, though they may look different: equivariance, invariance, normalization, attention structure, token interactions, contrastive alignment, and denoising consistency. ### 3. Optimization and Learning {.unnumbered} Many vision tasks are formulated as optimization problems. Segmentation, optical flow, camera calibration, stereo matching, bundle adjustment, neural-network training, and diffusion-model denoising can all be written in terms of objectives, losses, residuals, or likelihoods. A typical supervised learning objective has the form $$ \min_\theta \frac{1}{n}\sum_{i=1}^n L(f_\theta(x_i), y_i), $$ where $f_\theta$ is a model, $L$ is a loss function, and $(x_i,y_i)$ are training examples. A classical reconstruction problem may instead minimize a data term plus a regularization term: $$ \min_u D(u; I) + \lambda R(u). $$ The book repeatedly asks: What is being optimized? What assumptions justify the objective? What failure modes appear when the assumptions are violated? ## From Classical Vision to Modern Vision AI {.unnumbered} Classical computer vision and modern vision AI are sometimes taught as separate worlds. This book treats them as connected. Classical vision contributes the language of images, filters, edges, geometry, reconstruction, and variational principles. Modern vision AI contributes learned representations, deep networks, attention, self-supervision, vision-language alignment, promptable models, and generative models. The mathematical bridge between them is built from linear algebra, analysis, probability, optimization, and geometry. For example: - convolutional neural networks extend the idea of local filters and translation equivariance; - vision transformers reinterpret images as sequences of patches and use attention as a learned interaction operator; - self-supervised models formalize representation learning using invariance, contrastive objectives, reconstruction, or redundancy reduction; - vision-language models align image and text embeddings using geometric relationships in feature space; - diffusion models turn denoising into a generative process. The goal is not to replace classical ideas with deep learning. The goal is to understand how modern methods inherit, generalize, or transform older mathematical principles. ## Structure of the Book {.unnumbered} The book is organized into five parts. ### Part I. Images as Mathematical Objects {.unnumbered} Chapters 1--5 introduce the basic mathematical language of images: pixels, coordinates, color spaces, functions on domains, vector spaces, inner products, SVD, PCA, Fourier analysis, and frequency-domain reasoning. ### Part II. Classical Vision: Local Operators, Shape, and Motion {.unnumbered} Chapters 6--11 study convolution, filtering, edges, corners, local features, segmentation, morphology, shape, curves, level sets, optical flow, and motion estimation. ### Part III. Projective Geometry and 3D Vision {.unnumbered} Chapters 12--17 develop cameras as maps, calibration, epipolar geometry, stereo vision, structure from motion, and bundle adjustment. ### Part IV. Probability, Statistics, and Learning for Vision {.unnumbered} Chapters 18--24 study probabilistic image models, classification, representation learning, convolutional neural networks, training, object detection, and semantic/instance segmentation. ### Part V. Modern Vision: Foundation and Generative Models {.unnumbered} Chapters 25--30 study vision transformers, self-supervised learning, vision-language models, promptable segmentation, diffusion models, and multimodal AI systems. ## How the Computer Labs Support the Book {.unnumbered} The book includes one Google Colab lab for each chapter. The labs are stored in the `labs/` folder and summarized in `labs-summary.qmd`. Each lab is designed for independent study. This means that students should be able to open a notebook in Colab, read the mathematical background, run the code, change the examples, and complete exercises without needing a separate software manual. A typical lab contains: 1. a short conceptual overview; 2. mathematical definitions and notation; 3. synthetic data or small image examples; 4. step-by-step Python implementations; 5. visual interpretation of results; 6. exercises and reflection questions; 7. a mini-project connecting the lab to the chapter. The labs are intentionally small and transparent. They are not meant to hide mathematics inside large software packages. Whenever possible, the code uses NumPy, SciPy, Matplotlib, and simple models before introducing higher-level machine-learning tools. ## How to Read This Book {.unnumbered} A useful rhythm for each chapter is: 1. identify the main mathematical objects; 2. write down the maps between those objects; 3. identify the objective function, constraint, likelihood, or invariance; 4. run the corresponding lab; 5. modify one example; 6. explain the result in words and formulas. For example, when studying segmentation, ask what the unknown object is. Is it a binary mask, a multiclass label field, a probability map, or a set of instance masks? Then ask what objective is being optimized and what prior assumptions are being made about region smoothness, boundaries, or class probabilities. When studying camera geometry, ask which coordinate systems are involved and which quantities are known or unknown. Then identify the map from 3D points to 2D image points and the residual used to measure error. When studying neural vision models, ask what representation is being learned, what invariances are encouraged, what loss function is optimized, and how the model may fail under distribution shift. ## Mathematical Conventions {.unnumbered} The book uses the following conventions. | Symbol | Meaning | |---|---| | $I$ | image or image function | | $H,W$ | image height and width | | $C$ | number of channels | | $\Omega$ | image domain | | $x,y$ | image coordinates | | $X$ | world or 3D point | | $K$ | camera intrinsic matrix | | $R,t$ | camera rotation and translation | | $P$ | camera projection matrix | | $f_\theta$ | learned model with parameters $\theta$ | | $L$ | loss function | | $\Phi$ | feature or representation map | The text uses both continuous and discrete notation. This is deliberate. Many computer vision ideas are easiest to understand continuously, but all computations are eventually discrete. For instance, an image derivative may be written formally as $$ \nabla I(x,y) = \left(\frac{\partial I}{\partial x}, \frac{\partial I}{\partial y}\right), $$ but in a computer it is approximated using finite differences or convolution filters. ## What Students Should Be Able to Do {.unnumbered} After completing the book and labs, students should be able to: - describe images as arrays, functions, signals, tensors, random fields, and token sequences; - implement and interpret basic image operations such as filtering, interpolation, Fourier analysis, segmentation, and morphology; - formulate camera projection, calibration, stereo, structure from motion, and bundle adjustment mathematically; - connect classical feature extraction and modern representation learning; - explain CNNs and vision transformers using linear maps, nonlinearities, equivariance, attention, and optimization; - understand how self-supervised learning, vision-language models, promptable segmentation, and diffusion models fit into a unified mathematical framework; - use Python notebooks to test mathematical ideas visually and computationally; - communicate not only what an algorithm does, but why the mathematical structure matters. ## A First Guiding Example {.unnumbered} Consider a simple task: detect the boundary of an object in an image. A classical approach computes image gradients and looks for large changes: $$ \|\nabla I(x,y)\| \text{ is large near an edge.} $$ A variational approach may seek a curve that balances image evidence and smoothness. A probabilistic approach may ask for the posterior probability that a pixel is on a boundary. A neural approach may train a model to predict boundary labels from data. A transformer-based approach may allow distant image regions to interact through attention. A promptable model may ask the user for a point or box and return a mask. These methods look different, but they share a common structure: $$ \text{image} \longrightarrow \text{representation} \longrightarrow \text{decision or reconstruction}. $$ This book is about understanding that structure mathematically. ## Final Perspective {.unnumbered} Computer vision is not only about making machines recognize pictures. It is about representing visual information, imposing geometric and statistical structure, learning useful maps from data, and reasoning under uncertainty. The mathematics of computer vision is therefore both classical and modern. It includes the geometry of cameras, the analysis of signals, the algebra of tensors, the probability of images, the optimization of losses, and the representation geometry of foundation models. The goal of this book is to make those ideas visible.