The Mathematics of Modern Computer Vision

From Pixels and Projective Geometry to Transformers and Diffusion Models

Author

He Wang

Published

June 22, 2026

Preface

The Mathematics of Modern Computer Vision develops computer vision from its mathematical foundations to modern vision AI. The central goal is to help students see images, cameras, motion, shape, segmentation, neural networks, transformers, vision-language models, and diffusion models as precise mathematical objects.

Computer vision is often introduced as a collection of algorithms. This book takes a mathematical route. We begin with pixels and coordinate systems, then move through functions, linear operators, Fourier analysis, optimization, geometry, probability, and learned representations. The guiding question is:

What mathematical structure makes a visual computation possible?

An image may be a matrix, a function on a domain, a random field, a tensor, a point in a high-dimensional vector space, or a collection of tokens. A camera may be a projective map. A segmentation mask may be a label field. A neural vision model may be a composition of linear maps, nonlinearities, normalizations, attention operators, and learned embeddings.

This book is designed to connect these views into one coherent story.

Course Philosophy

The mathematical spine of the book is the idea that a vision system is a composition of maps:

\[ \text{visual data} \longrightarrow \text{features} \longrightarrow \text{geometry/statistics} \longrightarrow \text{decision, reconstruction, or generation}. \]

Each arrow has mathematical content. Sometimes it is a linear operator, such as convolution or projection. Sometimes it is a variational problem, such as segmentation or optical flow. Sometimes it is a probabilistic model, such as denoising, calibration, or uncertainty estimation. In modern vision AI, it may be a learned representation, an attention mechanism, or a generative diffusion process.

The book emphasizes three complementary viewpoints:

Viewpoint	Main mathematical objects	Computer vision examples
Images as data	arrays, tensors, functions, vector spaces, inner products	pixels, color channels, filtering, frequency analysis
Vision as geometry	projective maps, homogeneous coordinates, constraints, manifolds	cameras, calibration, epipolar geometry, stereo, structure from motion
Vision as learning	probability, optimization, loss functions, representations, attention	classification, CNNs, detection, segmentation, ViTs, self-supervised learning, diffusion

Intended Audience

This book is written for advanced undergraduate and graduate students in mathematics, applied mathematics, statistics, data science, computer science, and related fields.

It is especially suitable for students who want a rigorous bridge from mathematical coursework into modern visual AI. The material is appropriate for courses, independent study, reading groups, and project preparation in areas such as:

mathematical foundations of computer vision;
image processing and inverse problems;
projective geometry and 3D reconstruction;
statistical learning for image data;
convolutional neural networks and vision transformers;
vision-language models, promptable segmentation, and diffusion models;
scientific, medical, and multimodal imaging applications.

Prerequisites

The book assumes a working knowledge of:

vectors, matrices, projections, eigenvalues, SVD, and least squares;
multivariable calculus, gradients, Jacobians, and optimization basics;
probability, conditional probability, likelihood, and basic statistical inference;
basic Python programming with NumPy and Matplotlib.

Deep learning background is helpful but not required. CNNs, transformers, contrastive learning, vision-language models, and diffusion models are introduced from their mathematical structure before code is used.

Learning Goals

After studying the book, students should be able to:

represent images as arrays, functions, random fields, feature vectors, and token sequences;
use linear algebra and Fourier analysis to understand filtering, convolution, compression, and feature extraction;
formulate segmentation, optical flow, calibration, stereo, and bundle adjustment as optimization problems;
explain projective camera geometry using homogeneous coordinates and matrix maps;
derive and implement basic probabilistic models for images and visual predictions;
analyze CNNs, vision transformers, self-supervised models, and vision-language systems mathematically;
connect modern generative models, especially diffusion models, to denoising and probabilistic sampling;
complete computational experiments in Python and communicate the mathematical ideas behind them.

Book Structure

The book is organized into five parts.

Part I. Images as Mathematical Objects

Chapters 1–5 introduce computer vision, pixels, coordinates, color spaces, images as functions, linear algebra of images, and Fourier analysis.

This part builds the basic language:

\[ I: \Omega \to \mathbb{R}^c, \]

where $\Omega$ is an image domain and $c$ is the number of channels. Students learn to move between continuous and discrete views of an image.

Part II. Classical Vision: Geometry and Variational Thinking

Chapters 6–11 cover convolution, filtering, edges, corners, local features, segmentation, morphology, shape, curves, level sets, optical flow, and motion.

The emphasis is on local operators, variational models, and the link between image structure and optimization.

Part III. Projective Geometry and 3D Vision

Chapters 12–17 develop cameras as maps, calibration, epipolar geometry, stereo, structure from motion, and bundle adjustment.

The central mathematical tools are homogeneous coordinates, projective transformations, rank constraints, nonlinear least squares, and sparse optimization.

Part IV. Probability, Statistics, and Learning for Vision

Chapters 18–24 cover probabilistic image models, classification, representation learning, CNNs, training vision models, object detection, and semantic/instance segmentation.

This part connects classical statistical modeling with modern supervised learning.

Part V. Modern Vision: Transformers, Foundation Models, and Generative Models

Chapters 25–30 study vision transformers, self-supervised vision, vision-language models, promptable segmentation, diffusion models, and multimodal AI systems.

This part shows how mathematical ideas from attention, contrastive learning, conditional prediction, denoising, and representation geometry organize current vision AI.

Computer Labs

The book includes 30 Google Colab computer labs, one for each chapter. The labs are stored in the labs/ folder and are designed for independent study.

Each lab includes:

mathematical background before programming;
small synthetic examples that students can understand fully;
Python experiments using NumPy, SciPy, Matplotlib, and standard machine-learning tools;
visual diagnostics and interpretation questions;
AI-assisted study prompts;
exercises and a mini-project.

Students should start with the lab summary page:

Open the computer labs summary

The Colab links use the GitHub repository:

https://github.com/wanghemath/Book-MathCV

Suggested Paths Through the Book

Different readers may use the book in different ways.

Path	Recommended chapters	Purpose
Image mathematics path	Chapters 1–11	Build a foundation in images, functions, filters, segmentation, shape, and motion
3D vision path	Chapters 1–5, 12–17	Study camera geometry, stereo, structure from motion, and bundle adjustment
Statistical vision path	Chapters 1–5, 18–24	Connect image data to probability, classification, CNNs, detection, and segmentation
Modern AI vision path	Chapters 18–30	Study representation learning, transformers, self-supervision, VLMs, promptable models, and diffusion
Full course path	Chapters 1–30	Complete mathematical introduction to modern computer vision

How to Study

A productive rhythm for each chapter is:

read the chapter for mathematical structure;
identify the main objects, maps, losses, and assumptions;
run the corresponding Colab lab;
modify at least one experiment;
write a short explanation of what changed mathematically;
connect the chapter to a modern vision model or application.

The goal is not only to run code. The goal is to explain why the code is doing meaningful mathematics.

Repository Structure

The GitHub repository is organized as follows:

Book-MathCV/
├── _quarto.yml
├── index.qmd
├── introduction.qmd
├── roadmap.qmd
├── labs-summary.qmd
├── chapters/
│   ├── chapter-01-what-is-computer-vision.qmd
│   ├── chapter-02-pixels-coordinates-color.qmd
│   └── ...
├── labs/
│   ├── lab-01-what-is-computer-vision.ipynb
│   ├── lab-02-pixels-coordinates-color.ipynb
│   └── ...
├── appendices/
├── assets/
├── references.bib
└── README.md

Citation

Wang, H. The Mathematics of Modern Computer Vision: From Pixels and Projective Geometry to Transformers and Diffusion Models. Quarto book project.

Acknowledgment

This project is written for students learning computer vision as a mathematical subject. The hope is that the book helps students move confidently between formulas, algorithms, code, and modern AI systems.

# Preface {.unnumbered} ::: {.hero-box} **The Mathematics of Modern Computer Vision** develops computer vision from its mathematical foundations to modern vision AI. The central goal is to help students see images, cameras, motion, shape, segmentation, neural networks, transformers, vision-language models, and diffusion models as precise mathematical objects. ::: Computer vision is often introduced as a collection of algorithms. This book takes a mathematical route. We begin with pixels and coordinate systems, then move through functions, linear operators, Fourier analysis, optimization, geometry, probability, and learned representations. The guiding question is: > What mathematical structure makes a visual computation possible? An image may be a matrix, a function on a domain, a random field, a tensor, a point in a high-dimensional vector space, or a collection of tokens. A camera may be a projective map. A segmentation mask may be a label field. A neural vision model may be a composition of linear maps, nonlinearities, normalizations, attention operators, and learned embeddings. This book is designed to connect these views into one coherent story. ## Course Philosophy {.unnumbered} The mathematical spine of the book is the idea that a vision system is a composition of maps: $$ \text{visual data} \longrightarrow \text{features} \longrightarrow \text{geometry/statistics} \longrightarrow \text{decision, reconstruction, or generation}. $$ Each arrow has mathematical content. Sometimes it is a linear operator, such as convolution or projection. Sometimes it is a variational problem, such as segmentation or optical flow. Sometimes it is a probabilistic model, such as denoising, calibration, or uncertainty estimation. In modern vision AI, it may be a learned representation, an attention mechanism, or a generative diffusion process. The book emphasizes three complementary viewpoints: | Viewpoint | Main mathematical objects | Computer vision examples | |---|---|---| | **Images as data** | arrays, tensors, functions, vector spaces, inner products | pixels, color channels, filtering, frequency analysis | | **Vision as geometry** | projective maps, homogeneous coordinates, constraints, manifolds | cameras, calibration, epipolar geometry, stereo, structure from motion | | **Vision as learning** | probability, optimization, loss functions, representations, attention | classification, CNNs, detection, segmentation, ViTs, self-supervised learning, diffusion | ## Intended Audience {.unnumbered} This book is written for advanced undergraduate and graduate students in mathematics, applied mathematics, statistics, data science, computer science, and related fields. It is especially suitable for students who want a rigorous bridge from mathematical coursework into modern visual AI. The material is appropriate for courses, independent study, reading groups, and project preparation in areas such as: - mathematical foundations of computer vision; - image processing and inverse problems; - projective geometry and 3D reconstruction; - statistical learning for image data; - convolutional neural networks and vision transformers; - vision-language models, promptable segmentation, and diffusion models; - scientific, medical, and multimodal imaging applications. ## Prerequisites {.unnumbered} The book assumes a working knowledge of: - vectors, matrices, projections, eigenvalues, SVD, and least squares; - multivariable calculus, gradients, Jacobians, and optimization basics; - probability, conditional probability, likelihood, and basic statistical inference; - basic Python programming with NumPy and Matplotlib. Deep learning background is helpful but not required. CNNs, transformers, contrastive learning, vision-language models, and diffusion models are introduced from their mathematical structure before code is used. ## Learning Goals {.unnumbered} After studying the book, students should be able to: 1. represent images as arrays, functions, random fields, feature vectors, and token sequences; 2. use linear algebra and Fourier analysis to understand filtering, convolution, compression, and feature extraction; 3. formulate segmentation, optical flow, calibration, stereo, and bundle adjustment as optimization problems; 4. explain projective camera geometry using homogeneous coordinates and matrix maps; 5. derive and implement basic probabilistic models for images and visual predictions; 6. analyze CNNs, vision transformers, self-supervised models, and vision-language systems mathematically; 7. connect modern generative models, especially diffusion models, to denoising and probabilistic sampling; 8. complete computational experiments in Python and communicate the mathematical ideas behind them. ## Book Structure {.unnumbered} The book is organized into five parts. ### Part I. Images as Mathematical Objects {.unnumbered} Chapters 1--5 introduce computer vision, pixels, coordinates, color spaces, images as functions, linear algebra of images, and Fourier analysis. This part builds the basic language: $$ I: \Omega \to \mathbb{R}^c, $$ where $\Omega$ is an image domain and $c$ is the number of channels. Students learn to move between continuous and discrete views of an image. ### Part II. Classical Vision: Geometry and Variational Thinking {.unnumbered} Chapters 6--11 cover convolution, filtering, edges, corners, local features, segmentation, morphology, shape, curves, level sets, optical flow, and motion. The emphasis is on local operators, variational models, and the link between image structure and optimization. ### Part III. Projective Geometry and 3D Vision {.unnumbered} Chapters 12--17 develop cameras as maps, calibration, epipolar geometry, stereo, structure from motion, and bundle adjustment. The central mathematical tools are homogeneous coordinates, projective transformations, rank constraints, nonlinear least squares, and sparse optimization. ### Part IV. Probability, Statistics, and Learning for Vision {.unnumbered} Chapters 18--24 cover probabilistic image models, classification, representation learning, CNNs, training vision models, object detection, and semantic/instance segmentation. This part connects classical statistical modeling with modern supervised learning. ### Part V. Modern Vision: Transformers, Foundation Models, and Generative Models {.unnumbered} Chapters 25--30 study vision transformers, self-supervised vision, vision-language models, promptable segmentation, diffusion models, and multimodal AI systems. This part shows how mathematical ideas from attention, contrastive learning, conditional prediction, denoising, and representation geometry organize current vision AI. ## Computer Labs {.unnumbered} The book includes **30 Google Colab computer labs**, one for each chapter. The labs are stored in the `labs/` folder and are designed for independent study. Each lab includes: - mathematical background before programming; - small synthetic examples that students can understand fully; - Python experiments using NumPy, SciPy, Matplotlib, and standard machine-learning tools; - visual diagnostics and interpretation questions; - AI-assisted study prompts; - exercises and a mini-project. Students should start with the lab summary page: [Open the computer labs summary](labs-summary.qmd) The Colab links use the GitHub repository: ```text https://github.com/wanghemath/Book-MathCV ``` ## Suggested Paths Through the Book {.unnumbered} Different readers may use the book in different ways. | Path | Recommended chapters | Purpose | |---|---|---| | **Image mathematics path** | Chapters 1--11 | Build a foundation in images, functions, filters, segmentation, shape, and motion | | **3D vision path** | Chapters 1--5, 12--17 | Study camera geometry, stereo, structure from motion, and bundle adjustment | | **Statistical vision path** | Chapters 1--5, 18--24 | Connect image data to probability, classification, CNNs, detection, and segmentation | | **Modern AI vision path** | Chapters 18--30 | Study representation learning, transformers, self-supervision, VLMs, promptable models, and diffusion | | **Full course path** | Chapters 1--30 | Complete mathematical introduction to modern computer vision | ## How to Study {.unnumbered} A productive rhythm for each chapter is: 1. read the chapter for mathematical structure; 2. identify the main objects, maps, losses, and assumptions; 3. run the corresponding Colab lab; 4. modify at least one experiment; 5. write a short explanation of what changed mathematically; 6. connect the chapter to a modern vision model or application. The goal is not only to run code. The goal is to explain why the code is doing meaningful mathematics. ## Repository Structure {.unnumbered} The GitHub repository is organized as follows: ```text Book-MathCV/ ├── _quarto.yml ├── index.qmd ├── introduction.qmd ├── roadmap.qmd ├── labs-summary.qmd ├── chapters/ │ ├── chapter-01-what-is-computer-vision.qmd │ ├── chapter-02-pixels-coordinates-color.qmd │ └── ... ├── labs/ │ ├── lab-01-what-is-computer-vision.ipynb │ ├── lab-02-pixels-coordinates-color.ipynb │ └── ... ├── appendices/ ├── assets/ ├── references.bib └── README.md ``` ## Citation {.unnumbered} Wang, H. *The Mathematics of Modern Computer Vision: From Pixels and Projective Geometry to Transformers and Diffusion Models*. Quarto book project. ## Acknowledgment {.unnumbered} This project is written for students learning computer vision as a mathematical subject. The hope is that the book helps students move confidently between formulas, algorithms, code, and modern AI systems.