Roadmap

This roadmap explains how to study The Mathematics of Modern Computer Vision as a complete course, an independent-study sequence, or a focused path in image mathematics, 3D vision, statistical learning, or modern vision AI.

Computer vision has many entry points. Some students first meet it through image processing, some through projective geometry, some through machine learning, and some through modern foundation models. This book is designed so that these perspectives become one coherent mathematical story.

The central progression is:

\[ \text{pixels} \longrightarrow \text{functions and operators} \longrightarrow \text{geometry and optimization} \longrightarrow \text{probability and learning} \longrightarrow \text{modern multimodal vision AI}. \]

Each chapter has a matching Google Colab lab. The recommended rhythm is to read the chapter first, then run and modify the corresponding lab.

Big Picture

The book is organized into five parts.

Part	Chapters	Main theme	Main mathematical tools
I	1–5	Images as mathematical objects	arrays, tensors, functions, vector spaces, Fourier analysis
II	6–11	Classical vision and variational thinking	convolution, local operators, gradients, optimization, shape, motion
III	12–17	Projective geometry and 3D vision	homogeneous coordinates, camera maps, epipolar constraints, nonlinear least squares
IV	18–24	Probability, statistics, and learning for vision	likelihoods, priors, classifiers, representations, CNNs, losses, metrics
V	25–30	Modern vision AI	attention, self-supervision, vision-language alignment, promptable models, diffusion, multimodal systems

The first five chapters build the mathematical language used throughout the rest of the book. Chapters 6–17 develop the classical and geometric foundations of vision. Chapters 18–30 connect those foundations to modern statistical and neural vision systems.

Recommended Full-Course Schedule

A semester course can cover the full book by grouping related chapters together. The schedule below assumes a 14-week semester and one computer lab per week, with some labs assigned as independent work.

Week	Chapters	Labs	Main goal
1	1–2	1–2	Understand computer vision tasks, image arrays, pixels, coordinates, and color.
2	3–4	3–4	Model images as functions and vectors; use inner products, norms, SVD, and PCA.
3	5–6	5–6	Analyze frequency content and local filtering operators.
4	7–8	7–8	Detect edges/corners/features and formulate segmentation as optimization.
5	9–10	9–10	Study morphology, shape descriptors, curves, surfaces, and level sets.
6	11	11	Derive optical flow and motion estimation from brightness constancy.
7	12–13	12–13	Model cameras as maps and estimate camera parameters by calibration.
8	14–15	14–15	Study epipolar geometry, stereo, disparity, and depth.
9	16–17	16–17	Reconstruct 3D structure and refine it with bundle adjustment.
10	18–19	18–19	Introduce probabilistic image models and classification theory.
11	20–22	20–22	Learn representations, CNNs, and training principles.
12	23–24	23–24	Study detection and dense segmentation.
13	25–27	25–27	Study vision transformers, self-supervision, and vision-language models.
14	28–30	28–30	Study promptable models, diffusion, multimodal AI, and final projects.

For a shorter course, Chapters 1–18 form a classical mathematical computer vision course, while Chapters 18–30 form a modern vision AI course after a brief review of Chapters 1–6.

Chapter-by-Chapter Roadmap

Part I. Images as Mathematical Objects

Chapter	Topic	What students should learn	Matching lab
1	What Is Computer Vision?	Vision tasks, inverse problems, data-to-decision pipelines, and the mathematical viewpoint of vision.	Lab 1
2	Pixels, Coordinates, and Color Spaces	Image arrays, coordinate systems, channels, quantization, histograms, normalization, and color transforms.	Lab 2
3	Images as Functions on Domains	Continuous and discrete images, sampling, interpolation, warping, finite differences, and aliasing.	Lab 3
4	Linear Algebra of Images	Vectorization, inner products, norms, projections, SVD, PCA, and patch embeddings.	Lab 4
5	Fourier Analysis and Frequency	Fourier bases, 2D DFT, magnitude/phase, convolution theorem, filtering, compression, and aliasing.	Lab 5

Checkpoint after Part I. Students should be able to explain an image as an array, a sampled function, a vector, a signal with frequencies, and a collection of local patches.

Part II. Classical Vision: Operators, Shape, and Motion

Chapter	Topic	What students should learn	Matching lab
6	Convolution, Filtering, and Local Operators	Local filters, convolution/correlation, boundary conditions, separability, derivatives, and CNN-style filter banks.	Lab 6
7	Edges, Corners, and Local Features	Gradients, Canny-style edges, structure tensor, Harris/Shi–Tomasi corners, orientation histograms, and descriptors.	Lab 7
8	Image Segmentation as Optimization	Binary masks, thresholding, Otsu’s method, clustering, GMMs, smoothness terms, Dice/IoU, and dense prediction losses.	Lab 8
9	Morphology and Shape	Dilation, erosion, opening, closing, connected components, distance transforms, skeletons, and shape descriptors.	Lab 9
10	Curves, Surfaces, and Level Sets	Parametric curves, implicit curves, curvature, signed distance functions, active contours, and level-set surfaces.	Lab 10
11	Optical Flow and Motion	Brightness constancy, the aperture problem, Lucas–Kanade flow, Horn–Schunck regularization, warping, and flow metrics.	Lab 11

Checkpoint after Part II. Students should be able to derive local image operators, formulate simple segmentation and motion problems as optimization problems, and explain how image shape is represented computationally.

Part III. Projective Geometry and 3D Vision

Chapter	Topic	What students should learn	Matching lab
12	Cameras as Maps	Pinhole projection, homogeneous coordinates, projection matrices, back-projection rays, vanishing points, and distortion.	Lab 12
13	Camera Calibration	Checkerboards, homographies, intrinsic/extrinsic parameters, radial distortion, reprojection error, and uncertainty.	Lab 13
14	Epipolar Geometry	Fundamental and essential matrices, epipolar lines, normalized eight-point estimation, Sampson error, RANSAC, and pose ambiguity.	Lab 14
15	Stereo Vision and Depth	Rectified stereo, disparity, depth formula, uncertainty, matching costs, cost volumes, and photometric loss.	Lab 15
16	Structure from Motion	Multi-view tracks, triangulation, PnP, gauge ambiguity, affine factorization, and simplified bundle adjustment.	Lab 16
17	Bundle Adjustment	Reprojection residuals, Gauss–Newton and LM methods, sparse Jacobians, Schur complement, robust loss, and covariance.	Lab 17

Checkpoint after Part III. Students should be able to move between 3D points and 2D image measurements, derive epipolar constraints, triangulate depth, and understand why 3D reconstruction requires optimization.

Part IV. Probability, Statistics, and Learning for Vision

Chapter	Topic	What students should learn	Matching lab
18	Probabilistic Models of Images	Random fields, likelihoods, priors, MAP denoising, Wiener filtering, mixture models, entropy, and uncertainty.	Lab 18
19	Classification from a Mathematical View	Bayes rules, logistic and softmax regression, LDA/QDA, metrics, calibration, class imbalance, and robustness.	Lab 19
20	Representation Learning	PCA features, embedding geometry, retrieval, clustering, invariance, autoencoders, metric learning, and masked modeling.	Lab 20
21	Convolutional Neural Networks	Multichannel convolution, equivariance, receptive fields, pooling, sparse linear operators, residual blocks, and CNN features.	Lab 21
22	Training Vision Models	Empirical risk, cross-entropy, backpropagation, SGD/Adam-style updates, regularization, augmentation, and calibration.	Lab 22
23	Object Detection	Bounding boxes, IoU, anchors, target assignment, NMS, regression losses, focal loss, AP, and set prediction.	Lab 23
24	Semantic and Instance Segmentation	Pixel labels, one-hot tensors, Dice/IoU, connected components, mask matching, panoptic quality, and promptable segmentation.	Lab 24

Checkpoint after Part IV. Students should be able to connect image data with statistical decision theory, learned representations, neural network architectures, losses, metrics, and uncertainty.

Part V. Modern Vision: Transformers, Foundation Models, and Generative Models

Chapter	Topic	What students should learn	Matching lab
25	Vision Transformers	Patch tokens, positional encodings, scaled dot-product attention, multi-head attention, transformer blocks, and dense token prediction.	Lab 25
26	Self-Supervised Vision	Contrastive pairs, InfoNCE, alignment/uniformity, collapse diagnostics, Barlow Twins, VICReg, BYOL-style ideas, and masked image modeling.	Lab 26
27	Vision-Language Models	Paired image-text data, two-tower encoders, CLIP-style loss, zero-shot classification, prompt ensembling, retrieval, and cross-attention.	Lab 27
28	Promptable Segmentation and Foundation Models	Point/box/mask prompts, prompt-conditioned prediction, mask ranking, prompt efficiency, positional encoding, and calibration.	Lab 28
29	Diffusion Models for Images	Gaussian noising, denoising objectives, DDPM/DDIM sampling, parameterizations, classifier-free guidance, latent diffusion, and inpainting.	Lab 29
30	Vision in the Age of Multimodal AI	Multimodal spaces, fusion, retrieval, zero-shot prediction, VQA, captioning, video/3D tokens, calibration, and hallucination-risk auditing.	Lab 30

Checkpoint after Part V. Students should be able to describe modern vision AI systems using tokens, embeddings, attention, contrastive objectives, conditional prediction, denoising, and multimodal reasoning.

Focused Study Paths

Path A. Image Mathematics and Signal Processing

Recommended chapters: 1–7.

Recommended labs: 1–7.

This path is best for students who want the mathematical foundations of images, linear operators, filtering, Fourier analysis, edges, and local features. It prepares students for image processing, inverse problems, and scientific imaging.

Path B. Geometry and 3D Vision

Recommended chapters: 1–5, 11–17.

Recommended labs: 1–5, 11–17.

This path is best for students interested in cameras, robotics, 3D reconstruction, autonomous systems, augmented reality, and geometric computer vision. The key tools are projective geometry, rank constraints, triangulation, nonlinear least squares, and sparse optimization.

Path C. Statistical Vision and Supervised Learning

Recommended chapters: 1–6, 18–24.

Recommended labs: 1–6, 18–24.

This path is best for students who want to connect probability, classification, CNNs, detection, and segmentation. It prepares students for applied machine learning projects involving image classification, medical imaging, object localization, and dense prediction.

Path D. Modern Foundation Models for Vision

Recommended chapters: 20–30.

Recommended labs: 20–30.

This path is best for students who already know basic machine learning and want to study modern vision AI. It emphasizes representation learning, CNNs, training, detection, segmentation, transformers, self-supervision, vision-language models, promptable models, diffusion, and multimodal systems.

Path E. Independent Study Path

Recommended chapters: 1–30 in order, with one lab per chapter.

For independent study, students should maintain a notebook journal with four entries per chapter:

1. the main mathematical object of the chapter;
2. the main map, loss, constraint, or probability model;
3. one experiment they modified in the lab;
4. one connection to a modern computer vision application.

Prerequisite Review Map

Students may enter the book with different backgrounds. The following map shows where to review key mathematical ideas.

Needed background	Where it appears first	Why it matters
Vectors, matrices, norms, inner products	Chapters 2–4	Images become vectors, tensors, and data matrices.
Linear least squares and SVD	Chapter 4	Projection, compression, PCA, calibration, and reconstruction use least squares.
Fourier series and transforms	Chapter 5	Filtering and convolution can be understood in frequency space.
Gradients and Jacobians	Chapters 6–8	Edges, optimization, training, and geometric estimation require derivatives.
Probability and likelihood	Chapter 18	Denoising, classification, uncertainty, and diffusion all use probabilistic models.
Optimization	Chapters 8, 11, 17, 22	Segmentation, optical flow, bundle adjustment, and neural training are optimization problems.
Homogeneous coordinates	Chapter 12	Camera models and projective geometry require scale-equivalence classes.
Neural networks	Chapters 20–22	CNNs, transformers, and foundation models are learned compositions of maps.
Attention and embeddings	Chapter 25	Vision transformers and multimodal models use token embeddings and attention.

How to Use the Labs

The labs are stored in the labs/ folder and are opened through Google Colab. Each notebook is designed for independent study and contains:

• background explanation before programming;
• small synthetic examples that can be understood without external datasets;
• visualizations and diagnostics;
• exercises and interpretation questions;
• AI-assisted study prompts;
• a mini-project that can be expanded into a course project.

The general Colab link pattern is:

https://colab.research.google.com/github/wanghemath/Book-MathCV/blob/main/labs/lab-XX-name.ipynb

Students should begin with the full lab index:

Open the labs summary

Suggested Assessment Structure

For a course based on this book, the following assessment structure works well:

Component	Suggested weight	Description
Lab notebooks	35%	Students complete selected labs and submit short mathematical reflections.
Problem sets	25%	Derivations, proofs, calculations, and conceptual questions from the chapters.
Midterm project	15%	A focused project on classical vision, geometry, or probabilistic vision.
Final project	25%	A larger project connecting mathematical theory, code, and a modern vision application.

A strong final project should include a mathematical formulation, a computational experiment, diagnostic plots, error analysis, and a written explanation of what the model or algorithm is doing.

Final Project Themes

Good final projects can be built from many parts of the book. Examples include:

• edge and feature detection under noise and blur;
• segmentation using energy functions and learned pixelwise classifiers;
• camera calibration and uncertainty analysis;
• stereo depth reconstruction with confidence diagnostics;
• structure from motion on synthetic or small real datasets;
• probabilistic denoising and uncertainty maps;
• CNN versus transformer feature comparisons;
• self-supervised representation learning on image transformations;
• zero-shot classification with vision-language embeddings;
• prompt sensitivity in segmentation systems;
• diffusion-based inpainting and denoising;
• hallucination-risk auditing for multimodal vision systems.

Completion Milestones

Students can use the following milestones to track progress.

Milestone	After chapters	Student can now…
Image literacy	1–5	Represent images as arrays, functions, vectors, and frequency signals.
Local vision	6–11	Build local operators, detect structure, segment images, describe shape, and estimate motion.
Geometric vision	12–17	Model cameras, derive two-view constraints, recover depth, and optimize 3D reconstructions.
Statistical vision	18–24	Build probabilistic and learned models for classification, detection, and segmentation.
Modern vision AI	25–30	Explain transformers, self-supervised learning, VLMs, promptable models, diffusion, and multimodal systems.

Closing Perspective

The goal of the roadmap is not to turn computer vision into a list of disconnected techniques. The goal is to show that modern computer vision is organized by a small number of mathematical ideas: maps, invariances, constraints, losses, probabilities, representations, and optimization.

When students can identify these structures, they can move more confidently from classical algorithms to modern vision AI systems.