13 Cameras as Maps

Chapter goal. This chapter introduces cameras as mathematical maps from three-dimensional scene coordinates to two-dimensional image coordinates. The central model is the projective camera

\[ x \sim PX, \]

where $X\in \mathbb{P}^3$ is a homogeneous 3D point, $x\in \mathbb{P}^2$ is a homogeneous image point, and $P$ is a $3\times 4$ camera matrix. The goal is to understand what this equation means geometrically, algebraically, statistically, and computationally.

Computer vision becomes geometric as soon as we ask where an image came from. A pixel is not merely an intensity value. It is a measurement produced by light from a direction in space, integrated over a small sensor cell, and indexed by a coordinate system. A camera is therefore both a physical device and a mathematical map.

In the first two parts of the book, we mostly treated images as functions, arrays, tensors, and random fields. In this part, we add the missing physical structure: images are formed by projecting 3D scenes onto 2D sensors. The foundational question is:

How does a 3D point $X=(X,Y,Z)$ become a 2D image point $(u,v)$?

The simplest answer is the pinhole camera model:

\[ (u,v)=\left(f\frac{X}{Z}, f\frac{Y}{Z}\right), \]

where $f$ is focal length and $Z$ is depth. This formula is short, but it carries the main difficulty of 3D vision: division by depth. Nearby objects appear larger, parallel 3D lines may meet at vanishing points, and depth is lost when projecting from 3D to 2D.

For MA Applied Mathematics and MS Statistics students, cameras are an ideal place to see several mathematical themes work together:

linear algebra through homogeneous coordinates and matrix factorization;
geometry through projective spaces, lines, planes, and rigid motion;
calculus through the projection Jacobian and uncertainty propagation;
statistics through noisy image measurements and parameter estimation;
optimization through camera estimation and differentiable projection layers;
AI through geometric inductive bias in 3D-aware neural models.

13.1 Learning objectives

After studying this chapter, you should be able to:

describe a camera as a mapping from 3D scene coordinates to 2D image coordinates;
derive the pinhole projection equations;
explain homogeneous coordinates and why $x\sim PX$ is a projective equation;
distinguish world, camera, normalized image, sensor, and pixel coordinates;
decompose the camera matrix into intrinsic and extrinsic components;
interpret the intrinsic calibration matrix $K$;
interpret the extrinsic parameters $R$ and $t$ as a rigid transformation;
compute image projections of 3D points;
identify the camera center and viewing rays from a camera matrix;
explain vanishing points, perspective effects, and loss of depth;
compute and interpret the projection Jacobian;
formulate camera estimation as a least-squares problem;
connect projective camera models to differentiable computer vision and modern AI.

13.2 12.1 The first map: from space to image

Let a 3D scene point have coordinates

\[ X_c=(X_c,Y_c,Z_c)^T \]

in the camera coordinate system, where the camera center is the origin and the optical axis points along the positive $Z_c$-axis. The ideal pinhole camera maps this point to normalized image coordinates

\[ x_n = \frac{X_c}{Z_c}, \qquad y_n = \frac{Y_c}{Z_c}. \]

Multiplying by focal length $f$ gives metric image-plane coordinates

\[ x_s = f\frac{X_c}{Z_c}, \qquad y_s = f\frac{Y_c}{Z_c}. \]

The division by $Z_c$ is the mathematical signature of perspective.

Core idea. A camera does not preserve distances, angles, or areas in general. It preserves incidence relations: points on a 3D line project to points on a 2D line, except for degenerate cases. This is why projective geometry is the natural language of cameras.

13.2.1 Perspective projection is nonlinear in Euclidean coordinates

The map

\[ \pi:\mathbb{R}^3\setminus\{Z=0\}\to \mathbb{R}^2, \qquad \pi(X,Y,Z)=\left(\frac{X}{Z},\frac{Y}{Z}\right) \]

is nonlinear because of division by depth. However, it becomes linear in homogeneous coordinates. This is the key algebraic trick of camera geometry.

13.2.2 Depth controls scale

If two 3D points lie on the same viewing ray,

\[ X_2=\lambda X_1, \qquad \lambda>0, \]

then

\[ \pi(X_2)=\pi(X_1). \]

Indeed,

\[ \pi(\lambda X,\lambda Y,\lambda Z) = \left(\frac{\lambda X}{\lambda Z},\frac{\lambda Y}{\lambda Z}\right) = \left(\frac{X}{Z},\frac{Y}{Z}\right). \]

Thus, a single image point corresponds not to one 3D point, but to a ray of possible 3D points. This is why monocular depth estimation is intrinsically ambiguous.

13.3 12.2 Coordinate systems in camera geometry

A camera model is a composition of coordinate transformations. Confusion often comes from mixing coordinate systems. We will use the following levels.

Coordinate system	Typical symbol	Meaning
world coordinates	$X_w$	coordinates in a scene or object reference frame
camera coordinates	$X_c$	coordinates relative to the camera center and optical axis
normalized image coordinates	$(x_n,y_n)$	coordinates after dividing by depth
sensor/image-plane coordinates	$(x_s,y_s)$	metric coordinates on the image plane
pixel coordinates	$(u,v)$	row-column or column-row image indices

A common pipeline is

\[ X_w \xrightarrow{\text{rigid transform}} X_c \xrightarrow{\text{perspective projection}} (x_n,y_n) \xrightarrow{\text{intrinsics}} (u,v). \]

The rigid transform places the scene relative to the camera. The perspective projection divides by depth. The intrinsics convert normalized coordinates to pixel coordinates.

13.3.1 World to camera coordinates

Let

\[ X_w=(X_w,Y_w,Z_w)^T \]

be a world-coordinate point. A camera pose is described by a rotation matrix $R\in SO(3)$ and a translation vector $t\in\mathbb{R}^3$. The world-to-camera transformation is

\[ X_c = R X_w + t. \]

Here

\[ SO(3)=\{R\in\mathbb{R}^{3\times 3}:R^TR=I,\det(R)=1\}. \]

The vector $t$ is not always the camera center in world coordinates. If $C$ is the camera center in world coordinates, then

\[ 0 = R C + t, \]

\[ C=-R^Tt. \]

Common mistake. The translation vector $t$ in $X_c=RX_w+t$ is not the world-coordinate camera position. The camera center is $C=-R^Tt$.

13.3.2 Normalized image coordinates

Once $X_c=(X_c,Y_c,Z_c)^T$ is known, normalized image coordinates are

\[ \begin{pmatrix}x_n\\y_n\\1\end{pmatrix} \sim \begin{pmatrix}X_c\\Y_c\\Z_c\end{pmatrix}, \qquad x_n=\frac{X_c}{Z_c}, \qquad y_n=\frac{Y_c}{Z_c}. \]

The word normalized means that focal length, pixel size, and principal point have not yet been applied.

13.3.3 Pixel coordinates

Pixel coordinates are obtained through an intrinsic calibration matrix $K$:

\[ \begin{pmatrix}u\\v\\1\end{pmatrix} \sim K \begin{pmatrix}x_n\\y_n\\1\end{pmatrix}. \]

A standard intrinsic matrix is

\[ K=\begin{pmatrix} f_x & s & c_x\\ 0 & f_y & c_y\\ 0 & 0 & 1 \end{pmatrix}. \]

Here $f_x$ and $f_y$ are focal lengths measured in pixels, $s$ is skew, and $(c_x,c_y)$ is the principal point.

13.4 12.3 Homogeneous coordinates

The equation $x\sim PX$ uses homogeneous coordinates. This notation deserves careful explanation.

A 2D Euclidean point $(u,v)$ is represented by a nonzero vector in $\mathbb{R}^3$:

\[ \tilde x = (\tilde u,\tilde v,\tilde w)^T, \qquad \tilde w\neq 0, \]

with Euclidean coordinates

\[ u = \frac{\tilde u}{\tilde w}, \qquad v = \frac{\tilde v}{\tilde w}. \]

All nonzero scalar multiples represent the same point:

\[ (\tilde u,\tilde v,\tilde w)^T \sim \lambda(\tilde u,\tilde v,\tilde w)^T, \qquad \lambda\neq 0. \]

The set of equivalence classes is the projective plane $\mathbb{P}^2$.

Similarly, a 3D Euclidean point $(X,Y,Z)$ is represented by

\[ \tilde X=(\tilde X,\tilde Y,\tilde Z,\tilde W)^T\in\mathbb{R}^4\setminus\{0\}, \]

with Euclidean coordinates

\[ X=\frac{\tilde X}{\tilde W}, \qquad Y=\frac{\tilde Y}{\tilde W}, \qquad Z=\frac{\tilde Z}{\tilde W}, \]

when $\tilde W\neq 0$.

13.4.1 Points at infinity

If $\tilde w=0$ in $\mathbb{P}^2$, the point has no finite Euclidean coordinate. It is a point at infinity. Points at infinity are not an artificial trick; they encode directions. Parallel Euclidean lines meet at a point at infinity in projective geometry.

In images, this gives vanishing points. Parallel 3D lines can project to 2D lines that meet at a finite image point.

13.4.2 The equivalence symbol

When we write

\[ x\sim PX, \]

we mean there exists a nonzero scalar $\lambda$ such that

\[ x=\lambda PX. \]

The equation is not equality of Euclidean coordinate vectors. It is equality of projective points.

13.5 12.4 The projective camera matrix

A projective camera is a linear map in homogeneous coordinates:

\[ P:\mathbb{P}^3\to\mathbb{P}^2, \qquad x\sim PX, \]

where

\[ P\in\mathbb{R}^{3\times 4}. \]

The matrix $P$ is defined up to nonzero scale: $P$ and $\lambda P$ represent the same projective camera.

For a calibrated pinhole camera with no lens distortion, the standard factorization is

\[ P=K[R\mid t]. \]

Here:

$K$ is the $3\times 3$ intrinsic matrix;
$R$ is a $3\times 3$ rotation matrix;
$t$ is a $3\times 1$ translation vector;
$[R\mid t]$ is a $3\times 4$ matrix.

Thus, for a homogeneous world point

\[ X_w^h=\begin{pmatrix}X_w\\Y_w\\Z_w\\1\end{pmatrix}, \]

we have

\[ \tilde x = P X_w^h =K[R\mid t]X_w^h. \]

The pixel coordinates are obtained by dehomogenization:

\[ u=\frac{\tilde u}{\tilde w}, \qquad v=\frac{\tilde v}{\tilde w}. \]

Camera projection algorithm. Given $K$, $R$, $t$, and a 3D point $X_w$:

compute $X_c=RX_w+t$;
compute normalized image coordinates $(X_c/Z_c,Y_c/Z_c)$;
apply the intrinsic matrix $K$;
divide by the final homogeneous coordinate.

13.6 12.5 Deriving $P=K[R\mid t]$

Let a world point in homogeneous coordinates be

\[ X_w^h=\begin{pmatrix}X_w\\Y_w\\Z_w\\1\end{pmatrix}. \]

The extrinsic transformation gives

\[ X_c=RX_w+t=[R\mid t]X_w^h. \]

In homogeneous form, this gives the camera-coordinate point

\[ \begin{pmatrix}X_c\\Y_c\\Z_c\end{pmatrix} = [R\mid t] \begin{pmatrix}X_w\\Y_w\\Z_w\\1\end{pmatrix}. \]

The normalized image point is

\[ \begin{pmatrix}x_n\\y_n\\1\end{pmatrix} \sim \begin{pmatrix}X_c\\Y_c\\Z_c\end{pmatrix}. \]

Applying intrinsics gives

\[ \tilde x =K \begin{pmatrix}X_c\\Y_c\\Z_c\end{pmatrix} =K[R\mid t]X_w^h. \]

Therefore,

\[ P=K[R\mid t]. \]

This expression is the bridge from physical camera parameters to a single projective matrix.

13.6.1 Special case: canonical camera

The simplest camera has

\[ K=I, \qquad R=I, \qquad t=0. \]

Then

\[ P=[I\mid 0] = \begin{pmatrix} 1&0&0&0\\ 0&1&0&0\\ 0&0&1&0 \end{pmatrix}. \]

For $X^h=(X,Y,Z,1)^T$,

\[ PX^h=(X,Y,Z)^T, \]

and dehomogenization gives

\[ (u,v)=\left(\frac{X}{Z},\frac{Y}{Z}\right). \]

13.7 12.6 Intrinsic parameters

The intrinsic matrix

\[ K=\begin{pmatrix} f_x&s&c_x\\ 0&f_y&c_y\\ 0&0&1 \end{pmatrix} \]

maps normalized image coordinates to pixel coordinates:

\[ \begin{pmatrix} u\\v\\1\end{pmatrix} \sim \begin{pmatrix} f_x&s&c_x\\ 0&f_y&c_y\\ 0&0&1 \end{pmatrix} \begin{pmatrix}x_n\\y_n\\1\end{pmatrix}. \]

Therefore,

\[ u=f_xx_n+sy_n+c_x, \qquad v=f_yy_n+c_y. \]

13.7.1 Focal length in pixels

The physical focal length may be measured in millimeters, while pixels have physical size. If the physical focal length is $f$ and pixel widths are $m_x$ and $m_y$, then approximately

\[ f_x=\frac{f}{m_x}, \qquad f_y=\frac{f}{m_y}. \]

Thus $f_x$ and $f_y$ are measured in pixels.

13.7.2 Principal point

The principal point $(c_x,c_y)$ is where the optical axis intersects the image plane, expressed in pixel coordinates. It is often near the image center but should not be assumed to be exactly the center.

13.7.3 Skew

The skew parameter $s$ describes non-orthogonality between pixel axes. In most modern cameras, $s$ is close to zero, and many models set $s=0$.

13.7.4 Aspect ratio

If pixels are square and axes are orthogonal, then $f_x\approx f_y$. If not, the aspect ratio contributes to different values of $f_x$ and $f_y$.

Statistical note. Camera calibration estimates $K$ from noisy measurements. Treating $c_x$ as the exact image center or $f_x=f_y$ as exact may reduce variance but introduce bias. Chapter 13 studies this estimation problem in detail.

13.8 12.7 Extrinsic parameters and rigid motion

The extrinsic parameters describe the camera pose relative to the world:

\[ X_c=RX_w+t. \]

Here $R\in SO(3)$ and $t\in\mathbb{R}^3$.

13.8.1 Rotation matrices

A rotation matrix satisfies

\[ R^TR=I, \qquad \det(R)=1. \]

The columns of $R$ form an orthonormal basis. Rotations preserve lengths and angles:

\[ \|Ra-Rb\|=\|a-b\|, \qquad (Ra)^T(Rb)=a^Tb. \]

13.8.2 Rigid transformations

A rigid transformation in 3D has the form

\[ T(X)=RX+t. \]

In homogeneous coordinates, it is represented by a $4\times 4$ matrix

\[ T= \begin{pmatrix} R&t\\ 0&1 \end{pmatrix}. \]

The inverse transformation is

\[ T^{-1}=\begin{pmatrix} R^T&-R^Tt\\ 0&1 \end{pmatrix}. \]

The camera center in world coordinates is

\[ C=-R^Tt. \]

13.8.3 Pose versus projection

Extrinsics describe where the camera is. Intrinsics describe the internal coordinate conversion. Projection combines both. In symbols:

\[ \text{camera pose} + \text{intrinsics} \longrightarrow \text{camera matrix}. \]

But the camera matrix $P$ loses some structural information unless one enforces calibration constraints. A general $3\times 4$ matrix has 11 degrees of freedom up to scale, while a calibrated camera has fewer physical degrees of freedom.

13.9 12.8 Degrees of freedom

Counting parameters is useful for understanding identifiability.

A general $3\times 4$ projective camera matrix has 12 entries. Because scale is irrelevant, it has

\[ 12-1=11 \]

degrees of freedom.

A calibrated pinhole camera with known $K$ has:

3 degrees of freedom for rotation;
3 degrees of freedom for translation.

Thus it has 6 extrinsic degrees of freedom.

If $K$ is unknown with parameters $(f_x,f_y,s,c_x,c_y)$, then there are 5 additional intrinsic degrees of freedom, giving 11 total. This matches the projective camera count.

Why degree counting matters. Each 3D–2D point correspondence gives two scalar equations after dehomogenization. Since a general camera has 11 degrees of freedom, at least 6 point correspondences are needed for linear estimation in general position. In practice, more points are used because measurements are noisy.

13.10 12.9 Lines, planes, and incidence

Projective geometry treats points and lines algebraically.

A 2D image line is represented by a homogeneous vector

\[ \ell=(a,b,c)^T, \]

corresponding to the equation

\[ \ell^Tx=0. \]

For Euclidean coordinates $x=(u,v,1)^T$, this becomes

\[ au+bv+c=0. \]

A 3D plane is represented by

\[ \Pi=(\pi_1,\pi_2,\pi_3,\pi_4)^T, \]

with equation

\[ \Pi^TX=0. \]

13.10.1 Line through two points

Given two image points $x_1,x_2\in\mathbb{P}^2$, the line through them is

\[ \ell=x_1\times x_2. \]

Indeed,

\[ \ell^Tx_1=0, \qquad \ell^Tx_2=0. \]

13.10.2 Intersection of two lines

Given two image lines $\ell_1,\ell_2$, their intersection point is

\[ x=\ell_1\times \ell_2. \]

This formula includes parallel Euclidean lines: if they are parallel, their intersection has homogeneous coordinate $w=0$ and is therefore a point at infinity.

13.10.3 Projection of a 3D line

A 3D line generally projects to a 2D line. If two 3D points $X_1$ and $X_2$ lie on the 3D line, their image points are

\[ x_1\sim PX_1, \qquad x_2\sim PX_2. \]

The projected image line is

\[ \ell=x_1\times x_2. \]

This incidence preservation is why projective geometry is the natural language of pinhole cameras.

13.11 12.10 Vanishing points

Consider a 3D line parameterized by

\[ X(\lambda)=A+\lambda d, \]

where $d$ is a 3D direction. In homogeneous coordinates,

\[ X(\lambda)= \begin{pmatrix}A+\lambda d\\1\end{pmatrix}. \]

As $\lambda\to\infty$, the direction dominates. The corresponding point at infinity is

\[ X_\infty=\begin{pmatrix}d\\0\end{pmatrix}. \]

The image of this point is

\[ x_\infty \sim P X_\infty. \]

This image point is the vanishing point of all 3D lines with direction $d$.

13.11.1 Parallel lines can meet in the image

In Euclidean geometry, parallel lines do not meet. In projective geometry, parallel lines meet at a point at infinity. Under camera projection, that point at infinity may map to a finite image location. This is why railroad tracks appear to converge.

13.11.2 Orthogonal directions and calibration

If the camera is calibrated, vanishing points from orthogonal 3D directions satisfy useful constraints. Let $v_1$ and $v_2$ be image homogeneous coordinates of vanishing points corresponding to orthogonal 3D directions. Then, under appropriate assumptions,

\[ v_1^T K^{-T}K^{-1}v_2=0. \]

This equation is one reason vanishing points can be used for camera calibration.

13.12 12.11 Viewing rays and back-projection

Projection maps 3D points to image points. Back-projection maps an image point to a 3D ray.

For a calibrated camera, let a pixel point be

\[ x=(u,v,1)^T. \]

The corresponding normalized direction in camera coordinates is

\[ r_c \sim K^{-1}x. \]

In world coordinates, the ray has camera center $C$ and direction

\[ r_w=R^T r_c. \]

Thus the 3D ray is

\[ X(\lambda)=C+\lambda r_w, \qquad \lambda>0. \]

A single calibrated image point determines a 3D ray, not a 3D point. Depth must come from additional information: stereo, motion, known object size, shading, focus, learned priors, or scene constraints.

13.12.1 Ray-plane intersection

If a 3D point is known to lie on a plane

\[ n^T X + d =0, \]

then its depth along a ray can be recovered. Substitute

\[ X(\lambda)=C+\lambda r \]

into the plane equation:

\[ n^T(C+\lambda r)+d=0. \]

Solving gives

\[ \lambda=-\frac{n^TC+d}{n^Tr}, \]

provided $n^Tr\neq 0$.

This calculation is used in augmented reality, planar tracking, robotics, and homography-based vision.

13.13 12.12 The projection Jacobian

Projection is nonlinear, so calculus matters. For normalized projection,

\[ \pi(X,Y,Z)=\left(\frac{X}{Z},\frac{Y}{Z}\right), \]

the Jacobian is

\[ D\pi(X,Y,Z)= \begin{pmatrix} \frac{1}{Z} & 0 & -\frac{X}{Z^2}\\ 0 & \frac{1}{Z} & -\frac{Y}{Z^2} \end{pmatrix}. \]

With focal lengths $f_x$ and $f_y$, the pixel-coordinate Jacobian is approximately

\[ J_\pi= \begin{pmatrix} \frac{f_x}{Z} & 0 & -\frac{f_xX}{Z^2}\\ 0 & \frac{f_y}{Z} & -\frac{f_yY}{Z^2} \end{pmatrix}. \]

13.13.1 Interpretation

The entries show that sensitivity grows when $Z$ is small. Nearby points move more in the image when perturbed in 3D. This is why close objects produce large optical flow and why triangulation accuracy depends strongly on depth and baseline.

13.13.2 Uncertainty propagation

Suppose a 3D point estimate has covariance matrix $\Sigma_X$. A first-order approximation to the image covariance is

\[ \Sigma_x \approx J_\pi \Sigma_X J_\pi^T. \]

Conversely, if image measurements have covariance $\Sigma_x$, then estimating 3D quantities from images requires solving an inverse problem whose uncertainty is governed by Jacobians and conditioning.

Statistical viewpoint. Camera geometry is not just deterministic geometry. In real data, pixel locations, feature matches, calibration targets, and 3D points are noisy. The Jacobian is the local bridge between geometric perturbations and statistical uncertainty.

13.14 12.13 Estimating a camera matrix from correspondences

Chapter 13 will study camera calibration in more detail. Here we introduce the simplest algebraic estimation problem.

Suppose we observe correspondences

\[ X_i \leftrightarrow x_i, \qquad i=1,\dots,n, \]

where $X_i\in\mathbb{P}^3$ are known 3D points and $x_i\in\mathbb{P}^2$ are image points. We want to estimate $P$ such that

\[ x_i\sim PX_i. \]

The cross product condition

\[ x_i\times PX_i=0 \]

gives linear equations in the entries of $P$.

Let

\[ x_i=(u_i,v_i,1)^T, \qquad P=\begin{pmatrix}p_1^T\\p_2^T\\p_3^T\end{pmatrix}. \]

Then

\[ u_i=\frac{p_1^TX_i}{p_3^TX_i}, \qquad v_i=\frac{p_2^TX_i}{p_3^TX_i}. \]

Rearranging gives

\[ p_1^TX_i-u_i p_3^TX_i=0, \]

\[ p_2^TX_i-v_i p_3^TX_i=0. \]

Each correspondence contributes two linear equations. Stacking them produces

\[ A p=0, \]

where $p\in\mathbb{R}^{12}$ contains the entries of $P$.

Since $p$ is defined up to scale, the direct linear transform solves

\[ \min_{\|p\|=1}\|Ap\|^2. \]

The solution is the right singular vector of $A$ corresponding to the smallest singular value.

13.14.1 Why normalization matters

The homogeneous least-squares problem can be numerically ill-conditioned if coordinate values are large or unevenly scaled. A standard practical step is to normalize image and 3D coordinates before building $A$, solve the normalized problem, and then transform the camera matrix back.

Important. Algebraic error $\|Ap\|^2$ is not the same as geometric reprojection error. It is useful for initialization, but final estimation often minimizes reprojection error.

13.15 12.14 Reprojection error

Given an estimated camera $P$, the predicted image point for $X_i$ is

\[ \hat x_i=\operatorname{dehom}(PX_i). \]

The reprojection error is

\[ r_i=x_i-\hat x_i\in\mathbb{R}^2. \]

A common objective is

\[ \min_P \sum_{i=1}^n \|x_i-\operatorname{dehom}(PX_i)\|^2. \]

If image measurement noise is modeled as independent Gaussian noise,

\[ x_i=\operatorname{dehom}(PX_i)+\varepsilon_i, \qquad \varepsilon_i\sim N(0,\sigma^2I_2), \]

then least-squares reprojection error is the negative log-likelihood up to constants.

For outlier-contaminated data, one may use robust losses:

\[ \min_P\sum_{i=1}^n \rho\left(\|x_i-\operatorname{dehom}(PX_i)\|\right), \]

where $\rho$ grows more slowly than a quadratic.

13.16 12.15 Camera models and lens distortion

The pinhole model is mathematically clean, but real lenses introduce distortion. A common radial distortion model writes normalized coordinates $(x,y)$ and radius

\[ r^2=x^2+y^2. \]

The distorted coordinates are

\[ x_d=x(1+k_1r^2+k_2r^4+k_3r^6), \]

\[ y_d=y(1+k_1r^2+k_2r^4+k_3r^6). \]

Tangential distortion may be modeled by additional parameters. This chapter focuses on the ideal camera, while Chapter 13 studies calibration and distortion estimation.

13.16.1 Why distortion matters

If lens distortion is ignored, straight 3D lines may project to curved image traces. This violates the projective model and can bias estimates of camera parameters, epipolar geometry, and 3D structure.

13.17 12.16 Orthographic and weak-perspective cameras

Not all useful camera models are fully perspective.

13.17.1 Orthographic projection

Orthographic projection ignores depth variation:

\[ (u,v)=(X,Y). \]

It is a good approximation when object depth variation is small compared with distance from the camera.

13.17.2 Scaled orthographic projection

A weak-perspective model uses a single average depth $\bar Z$:

\[ (u,v)=\left(sX,sY\right), \qquad s=\frac{f}{\bar Z}. \]

This model is simpler and often useful for small objects far from the camera.

13.17.3 Perspective versus orthographic

Perspective projection changes scale with depth. Orthographic projection does not. The choice affects pose estimation, shape reconstruction, and learned 3D models.

13.18 12.17 Differentiable projection and modern AI

Modern computer vision often uses camera projection inside learning systems. Examples include:

differentiable rendering;
neural radiance fields and neural scene representations;
3D-aware generative models;
pose estimation networks;
monocular depth networks;
visual SLAM with learned features;
self-supervised depth and ego-motion from video.

The key computational operation is still the same:

\[ X\mapsto \operatorname{dehom}(K[R\mid t]X). \]

Because this map is differentiable away from $Z=0$, it can be used inside gradient-based learning.

13.18.1 Photometric self-supervision

Suppose a network predicts depth $D_t$ for an image $I_t$ and camera motion $T_{t\to s}$ from time $t$ to a source image $I_s$. A pixel $x_t$ is back-projected to 3D, transformed, and reprojected:

\[ x_s \sim K T_{t\to s} D_t(x_t) K^{-1}x_t. \]

The source image is sampled at $x_s$ to reconstruct $I_t$:

\[ \hat I_t(x_t)=I_s(x_s). \]

A photometric loss is

\[ L=\sum_{x_t}\rho\left(I_t(x_t)-\hat I_t(x_t)\right). \]

This is a modern AI objective, but its mathematical core is projective geometry plus differentiable interpolation.

Classical camera geometry has not disappeared in modern AI. It has become a differentiable layer, a loss function, a constraint, or a prior inside larger learned systems.

13.19 12.18 Worked example: projecting a cube

Consider the canonical camera with focal length $f=800$ pixels and principal point $(c_x,c_y)=(320,240)$. Let

\[ K=\begin{pmatrix} 800&0&320\\ 0&800&240\\ 0&0&1 \end{pmatrix}, \qquad R=I, \qquad t=0. \]

A 3D point $X=(1,2,5)^T$ projects as follows. First compute normalized coordinates:

\[ x_n=\frac{1}{5}=0.2, \qquad y_n=\frac{2}{5}=0.4. \]

Then convert to pixels:

\[ u=800(0.2)+320=480, \]

\[ v=800(0.4)+240=560. \]

Thus the image point is $(480,560)$.

If the same 3D direction is doubled to $(2,4,10)^T$, the normalized coordinates remain

\[ \left(\frac{2}{10},\frac{4}{10}\right)=(0.2,0.4), \]

so the image point is unchanged. This illustrates depth ambiguity along a viewing ray.

13.20 12.19 Python example: projection with NumPy

import numpy as np


def project_points(X_world, K, R=None, t=None):
    """Project 3D world points to pixel coordinates.

    Parameters
    ----------
    X_world : array, shape (n, 3)
        3D world points.
    K : array, shape (3, 3)
        Intrinsic calibration matrix.
    R : array, shape (3, 3)
        World-to-camera rotation. Defaults to identity.
    t : array, shape (3,)
        World-to-camera translation. Defaults to zero.

    Returns
    -------
    uv : array, shape (n, 2)
        Pixel coordinates.
    depth : array, shape (n,)
        Camera-coordinate depths.
    """
    X_world = np.asarray(X_world, dtype=float)
    K = np.asarray(K, dtype=float)

    if R is None:
        R = np.eye(3)
    if t is None:
        t = np.zeros(3)

    R = np.asarray(R, dtype=float)
    t = np.asarray(t, dtype=float).reshape(1, 3)

    X_cam = X_world @ R.T + t
    depth = X_cam[:, 2]

    x_h = X_cam @ K.T
    uv = x_h[:, :2] / x_h[:, 2:3]
    return uv, depth


K = np.array([
    [800.0, 0.0, 320.0],
    [0.0, 800.0, 240.0],
    [0.0, 0.0, 1.0]
])

cube = np.array([
    [-1, -1, 4],
    [ 1, -1, 4],
    [ 1,  1, 4],
    [-1,  1, 4],
    [-1, -1, 6],
    [ 1, -1, 6],
    [ 1,  1, 6],
    [-1,  1, 6],
])

uv, depth = project_points(cube, K)
print(np.round(uv, 2))
print(depth)

13.21 12.20 Python example: direct linear transform

The following code estimates a projective camera matrix from 3D–2D correspondences using the basic homogeneous least-squares method. For a real calibration pipeline, one should normalize coordinates and refine by minimizing reprojection error.

import numpy as np


def estimate_camera_dlt(X, uv):
    """Estimate a 3 x 4 camera matrix from 3D--2D correspondences.

    X : array, shape (n, 3)
        Euclidean 3D points.
    uv : array, shape (n, 2)
        Euclidean image points.
    """
    X = np.asarray(X, dtype=float)
    uv = np.asarray(uv, dtype=float)
    n = X.shape[0]

    Xh = np.column_stack([X, np.ones(n)])
    A = []
    for Xi, (u, v) in zip(Xh, uv):
        zeros = np.zeros(4)
        A.append(np.r_[Xi, zeros, -u * Xi])
        A.append(np.r_[zeros, Xi, -v * Xi])
    A = np.vstack(A)

    _, _, Vt = np.linalg.svd(A)
    p = Vt[-1]
    P = p.reshape(3, 4)
    return P / P[-1, -1]


def dehomogeneous(xh):
    return xh[..., :2] / xh[..., 2:3]


# Synthetic test
rng = np.random.default_rng(12)
X = rng.uniform(-1.0, 1.0, size=(20, 3))
X[:, 2] += 4.0

K = np.array([[700, 0, 320], [0, 700, 240], [0, 0, 1]], dtype=float)
P_true = K @ np.hstack([np.eye(3), np.zeros((3, 1))])
Xh = np.column_stack([X, np.ones(len(X))])
uv = dehomogeneous(Xh @ P_true.T)
uv_noisy = uv + rng.normal(0, 0.5, size=uv.shape)

P_hat = estimate_camera_dlt(X, uv_noisy)
uv_hat = dehomogeneous(Xh @ P_hat.T)
rmse = np.sqrt(np.mean(np.sum((uv_noisy - uv_hat) ** 2, axis=1)))
print("Reprojection RMSE:", rmse)
print(P_hat)

13.22 12.21 Python example: projection Jacobian

import numpy as np


def projection_jacobian(X, fx=1.0, fy=1.0):
    """Jacobian of perspective projection at camera-coordinate point X."""
    Xc, Yc, Zc = X
    return np.array([
        [fx / Zc, 0.0, -fx * Xc / (Zc ** 2)],
        [0.0, fy / Zc, -fy * Yc / (Zc ** 2)]
    ])


Sigma_X = np.diag([0.01, 0.01, 0.04])
for Z in [2.0, 4.0, 8.0]:
    J = projection_jacobian(np.array([0.5, 0.25, Z]), fx=800, fy=800)
    Sigma_uv = J @ Sigma_X @ J.T
    print("Z =", Z)
    print(Sigma_uv)

13.23 12.22 TensorFlow example: differentiable projection layer

The following example shows how camera projection can be written as a differentiable operation. This is the core operation behind many neural 3D systems.

import tensorflow as tf


def project_points_tf(X_world, K, R, t, eps=1e-6):
    """Differentiable 3D-to-2D projection.

    X_world: tensor, shape (n, 3)
    K: tensor, shape (3, 3)
    R: tensor, shape (3, 3)
    t: tensor, shape (3,)
    """
    X_cam = tf.linalg.matmul(X_world, R, transpose_b=True) + t[None, :]
    x_h = tf.linalg.matmul(X_cam, K, transpose_b=True)
    uv = x_h[:, :2] / tf.maximum(x_h[:, 2:3], eps)
    return uv, X_cam[:, 2]


K = tf.constant([[800.0, 0.0, 320.0],
                 [0.0, 800.0, 240.0],
                 [0.0, 0.0, 1.0]])
R = tf.eye(3)
t = tf.Variable([0.0, 0.0, 0.0])
X = tf.constant([[0.2, 0.1, 3.0],
                 [0.5, -0.2, 4.0],
                 [-0.3, 0.4, 5.0]])
target_uv = tf.constant([[370.0, 270.0],
                         [420.0, 200.0],
                         [275.0, 305.0]])

with tf.GradientTape() as tape:
    uv, depth = project_points_tf(X, K, R, t)
    loss = tf.reduce_mean(tf.reduce_sum((uv - target_uv) ** 2, axis=1))

grad_t = tape.gradient(loss, t)
print("loss:", loss.numpy())
print("gradient with respect to translation:", grad_t.numpy())

13.24 12.23 AI-assisted study prompts

Use an AI assistant as a mathematical critic, not as a replacement for derivation.

Prompt 1: explain the map.

I am studying pinhole cameras in projective geometry. Explain carefully why the Euclidean map $(X,Y,Z)\mapsto (X/Z,Y/Z)$ becomes linear in homogeneous coordinates. Include a simple numerical example and identify where dehomogenization occurs.

Prompt 2: debug notation.

Given $P=K[R\mid t]$, explain the difference between $t$ and the camera center $C$. Derive $C=-R^Tt$ and give a numerical check.

Prompt 3: statistical thinking.

Suppose 2D image measurements have independent Gaussian noise with standard deviation 1 pixel. Explain why minimizing squared reprojection error is a maximum likelihood estimator, and discuss when this assumption may fail.

When using AI for camera geometry, ask it to specify coordinate systems. Many wrong answers come from silently switching between world coordinates, camera coordinates, normalized coordinates, and pixel coordinates.

13.25 12.24 Summary

This chapter introduced the mathematical model of a camera as a projective map. The central equation is

\[ x\sim PX. \]

The main ideas are:

pinhole projection divides by depth;
homogeneous coordinates turn perspective projection into a linear projective map;
a physical camera matrix can be written as $P=K[R\mid t]$;
intrinsics describe internal pixel geometry;
extrinsics describe camera pose;
an image point back-projects to a 3D ray;
projective geometry preserves incidence but not Euclidean distances;
vanishing points arise from projected directions;
the projection Jacobian controls local sensitivity and uncertainty;
camera estimation can be formulated as least squares;
modern AI systems often use camera projection as a differentiable geometric layer.

The next chapter studies camera calibration: how to estimate $K$, distortion parameters, and pose from data.

13.26 Exercises

13.26.1 Conceptual exercises

Explain why a single image point corresponds to a ray in 3D rather than a unique 3D point.
What is the difference between normalized image coordinates and pixel coordinates?
Why is the equation $x\sim PX$ projective rather than ordinary vector equality?
Explain the roles of $K$, $R$, and $t$ in $P=K[R\mid t]$.
Why can parallel 3D lines meet at a finite point in an image?
What information is lost under perspective projection?
Why is the camera center not equal to the vector $t$ in $X_c=RX_w+t$?
Describe one situation where orthographic projection is a good approximation.

13.26.2 Mathematical exercises

Let $X=(2,3,6)^T$. Compute its normalized image coordinates under the canonical camera.
Let $K=\begin{pmatrix}600&0&320\\0&600&240\\0&0&1\end{pmatrix}$. Project $X=(1,-1,4)^T$ with $R=I$ and $t=0$.
Prove that all points on the ray $\lambda(X,Y,Z)^T$, $\lambda>0$, project to the same normalized image point.
Derive the Jacobian of $\pi(X,Y,Z)=(X/Z,Y/Z)$.
Given $X_c=RX_w+t$, derive the world-coordinate camera center $C=-R^Tt$.
Let $x_1=(1,0,1)^T$ and $x_2=(0,1,1)^T$. Compute the line $\ell=x_1\times x_2$.
Let two image lines be $\ell_1=(1,0,-2)^T$ and $\ell_2=(0,1,-3)^T$. Compute their intersection.
Show that a general $3\times 4$ projective camera matrix has 11 degrees of freedom.
Derive the two linear DLT equations from $u=(p_1^TX)/(p_3^TX)$ and $v=(p_2^TX)/(p_3^TX)$.
Suppose $\Sigma_X=\sigma^2I_3$. Write the first-order projected covariance $\Sigma_x$ in terms of the projection Jacobian.

13.26.3 Python exercises

Write a function that projects a set of 3D points using $K$, $R$, and $t$.
Generate a cube in 3D and plot both the cube and its projected 2D image.
Vary the depth of a square and plot its projected size as a function of depth.
Implement the projection Jacobian and compare finite-difference approximations with the analytic formula.
Simulate noisy 2D projections of known 3D points and estimate $P$ using DLT.
Compare algebraic error and reprojection error for the estimated camera matrix.
Add radial distortion to projected points and visualize the effect.
Implement back-projection from pixel coordinates to 3D rays.

13.26.4 AI and modeling exercises

Ask an AI assistant to derive $P=K[R\mid t]$. Then identify every coordinate system used in the derivation.
Ask an AI assistant to explain vanishing points using homogeneous coordinates. Check whether it correctly distinguishes points at infinity from far-away finite points.
Use an AI assistant to generate synthetic calibration data, then inspect whether the code correctly handles homogeneous scaling.
Design a self-supervised depth-learning loss based on differentiable projection. Specify all inputs, outputs, and assumptions.
Explain how a vision transformer could use camera intrinsics as metadata for 3D-aware image understanding.
Discuss why learned monocular depth models can predict plausible depth despite the geometric ambiguity of a single image.

13.27 References

Classical treatments of projective cameras and multi-view geometry can be found in Hartley and Zisserman (2004) and Szeliski (2022). Broader computer vision and image formation background appears in Prince (2012) and Gonzalez and Woods (2018). The connection between camera geometry, learning, and modern differentiable systems will be developed further in later chapters.

Gonzalez, Rafael C., and Richard E. Woods. 2018. Digital Image Processing. 4th ed. Pearson.

Hartley, Richard, and Andrew Zisserman. 2004. Multiple View Geometry in Computer Vision. 2nd ed. Cambridge University Press. https://www.robots.ox.ac.uk/~vgg/hzbook/.

Prince, Simon J. D. 2012. Computer Vision: Models, Learning, and Inference. Cambridge University Press. https://www.computervisionmodels.com/.

Szeliski, Richard. 2022. Computer Vision: Algorithms and Applications. 2nd ed. Springer. https://szeliski.org/Book/.

# Cameras as Maps {#sec-cameras-as-maps} > **Chapter goal.** This chapter introduces cameras as mathematical maps from three-dimensional scene coordinates to two-dimensional image coordinates. The central model is the projective camera > > $$ > x \sim PX, > $$ > > where $X\in \mathbb{P}^3$ is a homogeneous 3D point, $x\in \mathbb{P}^2$ is a homogeneous image point, and $P$ is a $3\times 4$ camera matrix. The goal is to understand what this equation means geometrically, algebraically, statistically, and computationally. Computer vision becomes geometric as soon as we ask where an image came from. A pixel is not merely an intensity value. It is a measurement produced by light from a direction in space, integrated over a small sensor cell, and indexed by a coordinate system. A camera is therefore both a physical device and a mathematical map. In the first two parts of the book, we mostly treated images as functions, arrays, tensors, and random fields. In this part, we add the missing physical structure: images are formed by projecting 3D scenes onto 2D sensors. The foundational question is: ::: {.math-spine} How does a 3D point $X=(X,Y,Z)$ become a 2D image point $(u,v)$? ::: The simplest answer is the **pinhole camera model**: $$ (u,v)=\left(f\frac{X}{Z}, f\frac{Y}{Z}\right), $$ where $f$ is focal length and $Z$ is depth. This formula is short, but it carries the main difficulty of 3D vision: division by depth. Nearby objects appear larger, parallel 3D lines may meet at vanishing points, and depth is lost when projecting from 3D to 2D. For MA Applied Mathematics and MS Statistics students, cameras are an ideal place to see several mathematical themes work together: - linear algebra through homogeneous coordinates and matrix factorization; - geometry through projective spaces, lines, planes, and rigid motion; - calculus through the projection Jacobian and uncertainty propagation; - statistics through noisy image measurements and parameter estimation; - optimization through camera estimation and differentiable projection layers; - AI through geometric inductive bias in 3D-aware neural models. ## Learning objectives After studying this chapter, you should be able to: 1. describe a camera as a mapping from 3D scene coordinates to 2D image coordinates; 2. derive the pinhole projection equations; 3. explain homogeneous coordinates and why $x\sim PX$ is a projective equation; 4. distinguish world, camera, normalized image, sensor, and pixel coordinates; 5. decompose the camera matrix into intrinsic and extrinsic components; 6. interpret the intrinsic calibration matrix $K$; 7. interpret the extrinsic parameters $R$ and $t$ as a rigid transformation; 8. compute image projections of 3D points; 9. identify the camera center and viewing rays from a camera matrix; 10. explain vanishing points, perspective effects, and loss of depth; 11. compute and interpret the projection Jacobian; 12. formulate camera estimation as a least-squares problem; 13. connect projective camera models to differentiable computer vision and modern AI. ## 12.1 The first map: from space to image Let a 3D scene point have coordinates $$ X_c=(X_c,Y_c,Z_c)^T $$ in the **camera coordinate system**, where the camera center is the origin and the optical axis points along the positive $Z_c$-axis. The ideal pinhole camera maps this point to normalized image coordinates $$ x_n = \frac{X_c}{Z_c}, \qquad y_n = \frac{Y_c}{Z_c}. $$ Multiplying by focal length $f$ gives metric image-plane coordinates $$ x_s = f\frac{X_c}{Z_c}, \qquad y_s = f\frac{Y_c}{Z_c}. $$ The division by $Z_c$ is the mathematical signature of perspective. ::: {.cv-card} **Core idea.** A camera does not preserve distances, angles, or areas in general. It preserves incidence relations: points on a 3D line project to points on a 2D line, except for degenerate cases. This is why projective geometry is the natural language of cameras. ::: ### Perspective projection is nonlinear in Euclidean coordinates The map $$ \pi:\mathbb{R}^3\setminus\{Z=0\}\to \mathbb{R}^2, \qquad \pi(X,Y,Z)=\left(\frac{X}{Z},\frac{Y}{Z}\right) $$ is nonlinear because of division by depth. However, it becomes linear in homogeneous coordinates. This is the key algebraic trick of camera geometry. <div id="cv-ch12-pinhole-depth-demo" class="plotly-demo"></div> ### Depth controls scale If two 3D points lie on the same viewing ray, $$ X_2=\lambda X_1, \qquad \lambda>0, $$ then $$ \pi(X_2)=\pi(X_1). $$ Indeed, $$ \pi(\lambda X,\lambda Y,\lambda Z) = \left(\frac{\lambda X}{\lambda Z},\frac{\lambda Y}{\lambda Z}\right) = \left(\frac{X}{Z},\frac{Y}{Z}\right). $$ Thus, a single image point corresponds not to one 3D point, but to a **ray** of possible 3D points. This is why monocular depth estimation is intrinsically ambiguous. ## 12.2 Coordinate systems in camera geometry A camera model is a composition of coordinate transformations. Confusion often comes from mixing coordinate systems. We will use the following levels. | Coordinate system | Typical symbol | Meaning | |---|---:|---| | world coordinates | $X_w$ | coordinates in a scene or object reference frame | | camera coordinates | $X_c$ | coordinates relative to the camera center and optical axis | | normalized image coordinates | $(x_n,y_n)$ | coordinates after dividing by depth | | sensor/image-plane coordinates | $(x_s,y_s)$ | metric coordinates on the image plane | | pixel coordinates | $(u,v)$ | row-column or column-row image indices | A common pipeline is $$ X_w \xrightarrow{\text{rigid transform}} X_c \xrightarrow{\text{perspective projection}} (x_n,y_n) \xrightarrow{\text{intrinsics}} (u,v). $$ The rigid transform places the scene relative to the camera. The perspective projection divides by depth. The intrinsics convert normalized coordinates to pixel coordinates. ### World to camera coordinates Let $$ X_w=(X_w,Y_w,Z_w)^T $$ be a world-coordinate point. A camera pose is described by a rotation matrix $R\in SO(3)$ and a translation vector $t\in\mathbb{R}^3$. The world-to-camera transformation is $$ X_c = R X_w + t. $$ Here $$ SO(3)=\{R\in\mathbb{R}^{3\times 3}:R^TR=I,\det(R)=1\}. $$ The vector $t$ is not always the camera center in world coordinates. If $C$ is the camera center in world coordinates, then $$ 0 = R C + t, $$ so $$ C=-R^Tt. $$ ::: {.warning-note} **Common mistake.** The translation vector $t$ in $X_c=RX_w+t$ is not the world-coordinate camera position. The camera center is $C=-R^Tt$. ::: ### Normalized image coordinates Once $X_c=(X_c,Y_c,Z_c)^T$ is known, normalized image coordinates are $$ \begin{pmatrix}x_n\\y_n\\1\end{pmatrix} \sim \begin{pmatrix}X_c\\Y_c\\Z_c\end{pmatrix}, \qquad x_n=\frac{X_c}{Z_c}, \qquad y_n=\frac{Y_c}{Z_c}. $$ The word **normalized** means that focal length, pixel size, and principal point have not yet been applied. ### Pixel coordinates Pixel coordinates are obtained through an intrinsic calibration matrix $K$: $$ \begin{pmatrix}u\\v\\1\end{pmatrix} \sim K \begin{pmatrix}x_n\\y_n\\1\end{pmatrix}. $$ A standard intrinsic matrix is $$ K=\begin{pmatrix} f_x & s & c_x\\ 0 & f_y & c_y\\ 0 & 0 & 1 \end{pmatrix}. $$ Here $f_x$ and $f_y$ are focal lengths measured in pixels, $s$ is skew, and $(c_x,c_y)$ is the principal point. <div id="cv-ch12-intrinsic-matrix-demo" class="plotly-demo"></div> ## 12.3 Homogeneous coordinates The equation $x\sim PX$ uses homogeneous coordinates. This notation deserves careful explanation. A 2D Euclidean point $(u,v)$ is represented by a nonzero vector in $\mathbb{R}^3$: $$ \tilde x = (\tilde u,\tilde v,\tilde w)^T, \qquad \tilde w\neq 0, $$ with Euclidean coordinates $$ u = \frac{\tilde u}{\tilde w}, \qquad v = \frac{\tilde v}{\tilde w}. $$ All nonzero scalar multiples represent the same point: $$ (\tilde u,\tilde v,\tilde w)^T \sim \lambda(\tilde u,\tilde v,\tilde w)^T, \qquad \lambda\neq 0. $$ The set of equivalence classes is the projective plane $\mathbb{P}^2$. Similarly, a 3D Euclidean point $(X,Y,Z)$ is represented by $$ \tilde X=(\tilde X,\tilde Y,\tilde Z,\tilde W)^T\in\mathbb{R}^4\setminus\{0\}, $$ with Euclidean coordinates $$ X=\frac{\tilde X}{\tilde W}, \qquad Y=\frac{\tilde Y}{\tilde W}, \qquad Z=\frac{\tilde Z}{\tilde W}, $$ when $\tilde W\neq 0$. ### Points at infinity If $\tilde w=0$ in $\mathbb{P}^2$, the point has no finite Euclidean coordinate. It is a **point at infinity**. Points at infinity are not an artificial trick; they encode directions. Parallel Euclidean lines meet at a point at infinity in projective geometry. In images, this gives vanishing points. Parallel 3D lines can project to 2D lines that meet at a finite image point. ### The equivalence symbol When we write $$ x\sim PX, $$ we mean there exists a nonzero scalar $\lambda$ such that $$ x=\lambda PX. $$ The equation is not equality of Euclidean coordinate vectors. It is equality of projective points. ## 12.4 The projective camera matrix A **projective camera** is a linear map in homogeneous coordinates: $$ P:\mathbb{P}^3\to\mathbb{P}^2, \qquad x\sim PX, $$ where $$ P\in\mathbb{R}^{3\times 4}. $$ The matrix $P$ is defined up to nonzero scale: $P$ and $\lambda P$ represent the same projective camera. For a calibrated pinhole camera with no lens distortion, the standard factorization is $$ P=K[R\mid t]. $$ Here: - $K$ is the $3\times 3$ intrinsic matrix; - $R$ is a $3\times 3$ rotation matrix; - $t$ is a $3\times 1$ translation vector; - $[R\mid t]$ is a $3\times 4$ matrix. Thus, for a homogeneous world point $$ X_w^h=\begin{pmatrix}X_w\\Y_w\\Z_w\\1\end{pmatrix}, $$ we have $$ \tilde x = P X_w^h =K[R\mid t]X_w^h. $$ The pixel coordinates are obtained by dehomogenization: $$ u=\frac{\tilde u}{\tilde w}, \qquad v=\frac{\tilde v}{\tilde w}. $$ ::: {.cv-card} **Camera projection algorithm.** Given $K$, $R$, $t$, and a 3D point $X_w$: 1. compute $X_c=RX_w+t$; 2. compute normalized image coordinates $(X_c/Z_c,Y_c/Z_c)$; 3. apply the intrinsic matrix $K$; 4. divide by the final homogeneous coordinate. ::: ## 12.5 Deriving $P=K[R\mid t]$ Let a world point in homogeneous coordinates be $$ X_w^h=\begin{pmatrix}X_w\\Y_w\\Z_w\\1\end{pmatrix}. $$ The extrinsic transformation gives $$ X_c=RX_w+t=[R\mid t]X_w^h. $$ In homogeneous form, this gives the camera-coordinate point $$ \begin{pmatrix}X_c\\Y_c\\Z_c\end{pmatrix} = [R\mid t] \begin{pmatrix}X_w\\Y_w\\Z_w\\1\end{pmatrix}. $$ The normalized image point is $$ \begin{pmatrix}x_n\\y_n\\1\end{pmatrix} \sim \begin{pmatrix}X_c\\Y_c\\Z_c\end{pmatrix}. $$ Applying intrinsics gives $$ \tilde x =K \begin{pmatrix}X_c\\Y_c\\Z_c\end{pmatrix} =K[R\mid t]X_w^h. $$ Therefore, $$ P=K[R\mid t]. $$ This expression is the bridge from physical camera parameters to a single projective matrix. ### Special case: canonical camera The simplest camera has $$ K=I, \qquad R=I, \qquad t=0. $$ Then $$ P=[I\mid 0] = \begin{pmatrix} 1&0&0&0\\ 0&1&0&0\\ 0&0&1&0 \end{pmatrix}. $$ For $X^h=(X,Y,Z,1)^T$, $$ PX^h=(X,Y,Z)^T, $$ and dehomogenization gives $$ (u,v)=\left(\frac{X}{Z},\frac{Y}{Z}\right). $$ ## 12.6 Intrinsic parameters The intrinsic matrix $$ K=\begin{pmatrix} f_x&s&c_x\\ 0&f_y&c_y\\ 0&0&1 \end{pmatrix} $$ maps normalized image coordinates to pixel coordinates: $$ \begin{pmatrix} u\\v\\1\end{pmatrix} \sim \begin{pmatrix} f_x&s&c_x\\ 0&f_y&c_y\\ 0&0&1 \end{pmatrix} \begin{pmatrix}x_n\\y_n\\1\end{pmatrix}. $$ Therefore, $$ u=f_xx_n+sy_n+c_x, \qquad v=f_yy_n+c_y. $$ ### Focal length in pixels The physical focal length may be measured in millimeters, while pixels have physical size. If the physical focal length is $f$ and pixel widths are $m_x$ and $m_y$, then approximately $$ f_x=\frac{f}{m_x}, \qquad f_y=\frac{f}{m_y}. $$ Thus $f_x$ and $f_y$ are measured in pixels. ### Principal point The principal point $(c_x,c_y)$ is where the optical axis intersects the image plane, expressed in pixel coordinates. It is often near the image center but should not be assumed to be exactly the center. ### Skew The skew parameter $s$ describes non-orthogonality between pixel axes. In most modern cameras, $s$ is close to zero, and many models set $s=0$. ### Aspect ratio If pixels are square and axes are orthogonal, then $f_x\approx f_y$. If not, the aspect ratio contributes to different values of $f_x$ and $f_y$. ::: {.stats-note} **Statistical note.** Camera calibration estimates $K$ from noisy measurements. Treating $c_x$ as the exact image center or $f_x=f_y$ as exact may reduce variance but introduce bias. Chapter 13 studies this estimation problem in detail. ::: ## 12.7 Extrinsic parameters and rigid motion The extrinsic parameters describe the camera pose relative to the world: $$ X_c=RX_w+t. $$ Here $R\in SO(3)$ and $t\in\mathbb{R}^3$. ### Rotation matrices A rotation matrix satisfies $$ R^TR=I, \qquad \det(R)=1. $$ The columns of $R$ form an orthonormal basis. Rotations preserve lengths and angles: $$ \|Ra-Rb\|=\|a-b\|, \qquad (Ra)^T(Rb)=a^Tb. $$ ### Rigid transformations A rigid transformation in 3D has the form $$ T(X)=RX+t. $$ In homogeneous coordinates, it is represented by a $4\times 4$ matrix $$ T= \begin{pmatrix} R&t\\ 0&1 \end{pmatrix}. $$ The inverse transformation is $$ T^{-1}=\begin{pmatrix} R^T&-R^Tt\\ 0&1 \end{pmatrix}. $$ The camera center in world coordinates is $$ C=-R^Tt. $$ ### Pose versus projection Extrinsics describe where the camera is. Intrinsics describe the internal coordinate conversion. Projection combines both. In symbols: $$ \text{camera pose} + \text{intrinsics} \longrightarrow \text{camera matrix}. $$ But the camera matrix $P$ loses some structural information unless one enforces calibration constraints. A general $3\times 4$ matrix has 11 degrees of freedom up to scale, while a calibrated camera has fewer physical degrees of freedom. ## 12.8 Degrees of freedom Counting parameters is useful for understanding identifiability. A general $3\times 4$ projective camera matrix has 12 entries. Because scale is irrelevant, it has $$ 12-1=11 $$ degrees of freedom. A calibrated pinhole camera with known $K$ has: - 3 degrees of freedom for rotation; - 3 degrees of freedom for translation. Thus it has 6 extrinsic degrees of freedom. If $K$ is unknown with parameters $(f_x,f_y,s,c_x,c_y)$, then there are 5 additional intrinsic degrees of freedom, giving 11 total. This matches the projective camera count. ::: {.cv-card} **Why degree counting matters.** Each 3D--2D point correspondence gives two scalar equations after dehomogenization. Since a general camera has 11 degrees of freedom, at least 6 point correspondences are needed for linear estimation in general position. In practice, more points are used because measurements are noisy. ::: ## 12.9 Lines, planes, and incidence Projective geometry treats points and lines algebraically. A 2D image line is represented by a homogeneous vector $$ \ell=(a,b,c)^T, $$ corresponding to the equation $$ \ell^Tx=0. $$ For Euclidean coordinates $x=(u,v,1)^T$, this becomes $$ au+bv+c=0. $$ A 3D plane is represented by $$ \Pi=(\pi_1,\pi_2,\pi_3,\pi_4)^T, $$ with equation $$ \Pi^TX=0. $$ ### Line through two points Given two image points $x_1,x_2\in\mathbb{P}^2$, the line through them is $$ \ell=x_1\times x_2. $$ Indeed, $$ \ell^Tx_1=0, \qquad \ell^Tx_2=0. $$ ### Intersection of two lines Given two image lines $\ell_1,\ell_2$, their intersection point is $$ x=\ell_1\times \ell_2. $$ This formula includes parallel Euclidean lines: if they are parallel, their intersection has homogeneous coordinate $w=0$ and is therefore a point at infinity. ### Projection of a 3D line A 3D line generally projects to a 2D line. If two 3D points $X_1$ and $X_2$ lie on the 3D line, their image points are $$ x_1\sim PX_1, \qquad x_2\sim PX_2. $$ The projected image line is $$ \ell=x_1\times x_2. $$ This incidence preservation is why projective geometry is the natural language of pinhole cameras. ## 12.10 Vanishing points Consider a 3D line parameterized by $$ X(\lambda)=A+\lambda d, $$ where $d$ is a 3D direction. In homogeneous coordinates, $$ X(\lambda)= \begin{pmatrix}A+\lambda d\\1\end{pmatrix}. $$ As $\lambda\to\infty$, the direction dominates. The corresponding point at infinity is $$ X_\infty=\begin{pmatrix}d\\0\end{pmatrix}. $$ The image of this point is $$ x_\infty \sim P X_\infty. $$ This image point is the **vanishing point** of all 3D lines with direction $d$. <div id="cv-ch12-vanishing-point-demo" class="plotly-demo"></div> ### Parallel lines can meet in the image In Euclidean geometry, parallel lines do not meet. In projective geometry, parallel lines meet at a point at infinity. Under camera projection, that point at infinity may map to a finite image location. This is why railroad tracks appear to converge. ### Orthogonal directions and calibration If the camera is calibrated, vanishing points from orthogonal 3D directions satisfy useful constraints. Let $v_1$ and $v_2$ be image homogeneous coordinates of vanishing points corresponding to orthogonal 3D directions. Then, under appropriate assumptions, $$ v_1^T K^{-T}K^{-1}v_2=0. $$ This equation is one reason vanishing points can be used for camera calibration. ## 12.11 Viewing rays and back-projection Projection maps 3D points to image points. Back-projection maps an image point to a 3D ray. For a calibrated camera, let a pixel point be $$ x=(u,v,1)^T. $$ The corresponding normalized direction in camera coordinates is $$ r_c \sim K^{-1}x. $$ In world coordinates, the ray has camera center $C$ and direction $$ r_w=R^T r_c. $$ Thus the 3D ray is $$ X(\lambda)=C+\lambda r_w, \qquad \lambda>0. $$ ::: {.math-spine} A single calibrated image point determines a 3D ray, not a 3D point. Depth must come from additional information: stereo, motion, known object size, shading, focus, learned priors, or scene constraints. ::: ### Ray-plane intersection If a 3D point is known to lie on a plane $$ n^T X + d =0, $$ then its depth along a ray can be recovered. Substitute $$ X(\lambda)=C+\lambda r $$ into the plane equation: $$ n^T(C+\lambda r)+d=0. $$ Solving gives $$ \lambda=-\frac{n^TC+d}{n^Tr}, $$ provided $n^Tr\neq 0$. This calculation is used in augmented reality, planar tracking, robotics, and homography-based vision. ## 12.12 The projection Jacobian Projection is nonlinear, so calculus matters. For normalized projection, $$ \pi(X,Y,Z)=\left(\frac{X}{Z},\frac{Y}{Z}\right), $$ the Jacobian is $$ D\pi(X,Y,Z)= \begin{pmatrix} \frac{1}{Z} & 0 & -\frac{X}{Z^2}\\ 0 & \frac{1}{Z} & -\frac{Y}{Z^2} \end{pmatrix}. $$ With focal lengths $f_x$ and $f_y$, the pixel-coordinate Jacobian is approximately $$ J_\pi= \begin{pmatrix} \frac{f_x}{Z} & 0 & -\frac{f_xX}{Z^2}\\ 0 & \frac{f_y}{Z} & -\frac{f_yY}{Z^2} \end{pmatrix}. $$ <div id="cv-ch12-projection-jacobian-demo" class="plotly-demo"></div> ### Interpretation The entries show that sensitivity grows when $Z$ is small. Nearby points move more in the image when perturbed in 3D. This is why close objects produce large optical flow and why triangulation accuracy depends strongly on depth and baseline. ### Uncertainty propagation Suppose a 3D point estimate has covariance matrix $\Sigma_X$. A first-order approximation to the image covariance is $$ \Sigma_x \approx J_\pi \Sigma_X J_\pi^T. $$ Conversely, if image measurements have covariance $\Sigma_x$, then estimating 3D quantities from images requires solving an inverse problem whose uncertainty is governed by Jacobians and conditioning. ::: {.stats-note} **Statistical viewpoint.** Camera geometry is not just deterministic geometry. In real data, pixel locations, feature matches, calibration targets, and 3D points are noisy. The Jacobian is the local bridge between geometric perturbations and statistical uncertainty. ::: ## 12.13 Estimating a camera matrix from correspondences Chapter 13 will study camera calibration in more detail. Here we introduce the simplest algebraic estimation problem. Suppose we observe correspondences $$ X_i \leftrightarrow x_i, \qquad i=1,\dots,n, $$ where $X_i\in\mathbb{P}^3$ are known 3D points and $x_i\in\mathbb{P}^2$ are image points. We want to estimate $P$ such that $$ x_i\sim PX_i. $$ The cross product condition $$ x_i\times PX_i=0 $$ gives linear equations in the entries of $P$. Let $$ x_i=(u_i,v_i,1)^T, \qquad P=\begin{pmatrix}p_1^T\\p_2^T\\p_3^T\end{pmatrix}. $$ Then $$ u_i=\frac{p_1^TX_i}{p_3^TX_i}, \qquad v_i=\frac{p_2^TX_i}{p_3^TX_i}. $$ Rearranging gives $$ p_1^TX_i-u_i p_3^TX_i=0, $$ $$ p_2^TX_i-v_i p_3^TX_i=0. $$ Each correspondence contributes two linear equations. Stacking them produces $$ A p=0, $$ where $p\in\mathbb{R}^{12}$ contains the entries of $P$. Since $p$ is defined up to scale, the direct linear transform solves $$ \min_{\|p\|=1}\|Ap\|^2. $$ The solution is the right singular vector of $A$ corresponding to the smallest singular value. ### Why normalization matters The homogeneous least-squares problem can be numerically ill-conditioned if coordinate values are large or unevenly scaled. A standard practical step is to normalize image and 3D coordinates before building $A$, solve the normalized problem, and then transform the camera matrix back. ::: {.warning-note} **Important.** Algebraic error $\|Ap\|^2$ is not the same as geometric reprojection error. It is useful for initialization, but final estimation often minimizes reprojection error. ::: ## 12.14 Reprojection error Given an estimated camera $P$, the predicted image point for $X_i$ is $$ \hat x_i=\operatorname{dehom}(PX_i). $$ The reprojection error is $$ r_i=x_i-\hat x_i\in\mathbb{R}^2. $$ A common objective is $$ \min_P \sum_{i=1}^n \|x_i-\operatorname{dehom}(PX_i)\|^2. $$ If image measurement noise is modeled as independent Gaussian noise, $$ x_i=\operatorname{dehom}(PX_i)+\varepsilon_i, \qquad \varepsilon_i\sim N(0,\sigma^2I_2), $$ then least-squares reprojection error is the negative log-likelihood up to constants. For outlier-contaminated data, one may use robust losses: $$ \min_P\sum_{i=1}^n \rho\left(\|x_i-\operatorname{dehom}(PX_i)\|\right), $$ where $\rho$ grows more slowly than a quadratic. ## 12.15 Camera models and lens distortion The pinhole model is mathematically clean, but real lenses introduce distortion. A common radial distortion model writes normalized coordinates $(x,y)$ and radius $$ r^2=x^2+y^2. $$ The distorted coordinates are $$ x_d=x(1+k_1r^2+k_2r^4+k_3r^6), $$ $$ y_d=y(1+k_1r^2+k_2r^4+k_3r^6). $$ Tangential distortion may be modeled by additional parameters. This chapter focuses on the ideal camera, while Chapter 13 studies calibration and distortion estimation. ### Why distortion matters If lens distortion is ignored, straight 3D lines may project to curved image traces. This violates the projective model and can bias estimates of camera parameters, epipolar geometry, and 3D structure. ## 12.16 Orthographic and weak-perspective cameras Not all useful camera models are fully perspective. ### Orthographic projection Orthographic projection ignores depth variation: $$ (u,v)=(X,Y). $$ It is a good approximation when object depth variation is small compared with distance from the camera. ### Scaled orthographic projection A weak-perspective model uses a single average depth $\bar Z$: $$ (u,v)=\left(sX,sY\right), \qquad s=\frac{f}{\bar Z}. $$ This model is simpler and often useful for small objects far from the camera. ### Perspective versus orthographic Perspective projection changes scale with depth. Orthographic projection does not. The choice affects pose estimation, shape reconstruction, and learned 3D models. ## 12.17 Differentiable projection and modern AI Modern computer vision often uses camera projection inside learning systems. Examples include: - differentiable rendering; - neural radiance fields and neural scene representations; - 3D-aware generative models; - pose estimation networks; - monocular depth networks; - visual SLAM with learned features; - self-supervised depth and ego-motion from video. The key computational operation is still the same: $$ X\mapsto \operatorname{dehom}(K[R\mid t]X). $$ Because this map is differentiable away from $Z=0$, it can be used inside gradient-based learning. ### Photometric self-supervision Suppose a network predicts depth $D_t$ for an image $I_t$ and camera motion $T_{t\to s}$ from time $t$ to a source image $I_s$. A pixel $x_t$ is back-projected to 3D, transformed, and reprojected: $$ x_s \sim K T_{t\to s} D_t(x_t) K^{-1}x_t. $$ The source image is sampled at $x_s$ to reconstruct $I_t$: $$ \hat I_t(x_t)=I_s(x_s). $$ A photometric loss is $$ L=\sum_{x_t}\rho\left(I_t(x_t)-\hat I_t(x_t)\right). $$ This is a modern AI objective, but its mathematical core is projective geometry plus differentiable interpolation. ::: {.math-spine} Classical camera geometry has not disappeared in modern AI. It has become a differentiable layer, a loss function, a constraint, or a prior inside larger learned systems. ::: ## 12.18 Worked example: projecting a cube Consider the canonical camera with focal length $f=800$ pixels and principal point $(c_x,c_y)=(320,240)$. Let $$ K=\begin{pmatrix} 800&0&320\\ 0&800&240\\ 0&0&1 \end{pmatrix}, \qquad R=I, \qquad t=0. $$ A 3D point $X=(1,2,5)^T$ projects as follows. First compute normalized coordinates: $$ x_n=\frac{1}{5}=0.2, \qquad y_n=\frac{2}{5}=0.4. $$ Then convert to pixels: $$ u=800(0.2)+320=480, $$ $$ v=800(0.4)+240=560. $$ Thus the image point is $(480,560)$. If the same 3D direction is doubled to $(2,4,10)^T$, the normalized coordinates remain $$ \left(\frac{2}{10},\frac{4}{10}\right)=(0.2,0.4), $$ so the image point is unchanged. This illustrates depth ambiguity along a viewing ray. ## 12.19 Python example: projection with NumPy ```{python} #| label: ch12-project-points-numpy #| eval: false #| code-fold: false import numpy as np def project_points(X_world, K, R=None, t=None): """Project 3D world points to pixel coordinates. Parameters ---------- X_world : array, shape (n, 3) 3D world points. K : array, shape (3, 3) Intrinsic calibration matrix. R : array, shape (3, 3) World-to-camera rotation. Defaults to identity. t : array, shape (3,) World-to-camera translation. Defaults to zero. Returns ------- uv : array, shape (n, 2) Pixel coordinates. depth : array, shape (n,) Camera-coordinate depths. """ X_world = np.asarray(X_world, dtype=float) K = np.asarray(K, dtype=float) if R is None: R = np.eye(3) if t is None: t = np.zeros(3) R = np.asarray(R, dtype=float) t = np.asarray(t, dtype=float).reshape(1, 3) X_cam = X_world @ R.T + t depth = X_cam[:, 2] x_h = X_cam @ K.T uv = x_h[:, :2] / x_h[:, 2:3] return uv, depth K = np.array([ [800.0, 0.0, 320.0], [0.0, 800.0, 240.0], [0.0, 0.0, 1.0] ]) cube = np.array([ [-1, -1, 4], [ 1, -1, 4], [ 1, 1, 4], [-1, 1, 4], [-1, -1, 6], [ 1, -1, 6], [ 1, 1, 6], [-1, 1, 6], ]) uv, depth = project_points(cube, K) print(np.round(uv, 2)) print(depth) ``` ## 12.20 Python example: direct linear transform The following code estimates a projective camera matrix from 3D--2D correspondences using the basic homogeneous least-squares method. For a real calibration pipeline, one should normalize coordinates and refine by minimizing reprojection error. ```{python} #| label: ch12-dlt-camera-estimation #| eval: false #| code-fold: false import numpy as np def estimate_camera_dlt(X, uv): """Estimate a 3 x 4 camera matrix from 3D--2D correspondences. X : array, shape (n, 3) Euclidean 3D points. uv : array, shape (n, 2) Euclidean image points. """ X = np.asarray(X, dtype=float) uv = np.asarray(uv, dtype=float) n = X.shape[0] Xh = np.column_stack([X, np.ones(n)]) A = [] for Xi, (u, v) in zip(Xh, uv): zeros = np.zeros(4) A.append(np.r_[Xi, zeros, -u * Xi]) A.append(np.r_[zeros, Xi, -v * Xi]) A = np.vstack(A) _, _, Vt = np.linalg.svd(A) p = Vt[-1] P = p.reshape(3, 4) return P / P[-1, -1] def dehomogeneous(xh): return xh[..., :2] / xh[..., 2:3] # Synthetic test rng = np.random.default_rng(12) X = rng.uniform(-1.0, 1.0, size=(20, 3)) X[:, 2] += 4.0 K = np.array([[700, 0, 320], [0, 700, 240], [0, 0, 1]], dtype=float) P_true = K @ np.hstack([np.eye(3), np.zeros((3, 1))]) Xh = np.column_stack([X, np.ones(len(X))]) uv = dehomogeneous(Xh @ P_true.T) uv_noisy = uv + rng.normal(0, 0.5, size=uv.shape) P_hat = estimate_camera_dlt(X, uv_noisy) uv_hat = dehomogeneous(Xh @ P_hat.T) rmse = np.sqrt(np.mean(np.sum((uv_noisy - uv_hat) ** 2, axis=1))) print("Reprojection RMSE:", rmse) print(P_hat) ``` ## 12.21 Python example: projection Jacobian ```{python} #| label: ch12-projection-jacobian #| eval: false #| code-fold: false import numpy as np def projection_jacobian(X, fx=1.0, fy=1.0): """Jacobian of perspective projection at camera-coordinate point X.""" Xc, Yc, Zc = X return np.array([ [fx / Zc, 0.0, -fx * Xc / (Zc ** 2)], [0.0, fy / Zc, -fy * Yc / (Zc ** 2)] ]) Sigma_X = np.diag([0.01, 0.01, 0.04]) for Z in [2.0, 4.0, 8.0]: J = projection_jacobian(np.array([0.5, 0.25, Z]), fx=800, fy=800) Sigma_uv = J @ Sigma_X @ J.T print("Z =", Z) print(Sigma_uv) ``` ## 12.22 TensorFlow example: differentiable projection layer The following example shows how camera projection can be written as a differentiable operation. This is the core operation behind many neural 3D systems. ```{python} #| label: ch12-tensorflow-differentiable-projection #| eval: false #| code-fold: false import tensorflow as tf def project_points_tf(X_world, K, R, t, eps=1e-6): """Differentiable 3D-to-2D projection. X_world: tensor, shape (n, 3) K: tensor, shape (3, 3) R: tensor, shape (3, 3) t: tensor, shape (3,) """ X_cam = tf.linalg.matmul(X_world, R, transpose_b=True) + t[None, :] x_h = tf.linalg.matmul(X_cam, K, transpose_b=True) uv = x_h[:, :2] / tf.maximum(x_h[:, 2:3], eps) return uv, X_cam[:, 2] K = tf.constant([[800.0, 0.0, 320.0], [0.0, 800.0, 240.0], [0.0, 0.0, 1.0]]) R = tf.eye(3) t = tf.Variable([0.0, 0.0, 0.0]) X = tf.constant([[0.2, 0.1, 3.0], [0.5, -0.2, 4.0], [-0.3, 0.4, 5.0]]) target_uv = tf.constant([[370.0, 270.0], [420.0, 200.0], [275.0, 305.0]]) with tf.GradientTape() as tape: uv, depth = project_points_tf(X, K, R, t) loss = tf.reduce_mean(tf.reduce_sum((uv - target_uv) ** 2, axis=1)) grad_t = tape.gradient(loss, t) print("loss:", loss.numpy()) print("gradient with respect to translation:", grad_t.numpy()) ``` ## 12.23 AI-assisted study prompts Use an AI assistant as a mathematical critic, not as a replacement for derivation. ::: {.cv-card} **Prompt 1: explain the map.** I am studying pinhole cameras in projective geometry. Explain carefully why the Euclidean map $(X,Y,Z)\mapsto (X/Z,Y/Z)$ becomes linear in homogeneous coordinates. Include a simple numerical example and identify where dehomogenization occurs. ::: ::: {.cv-card} **Prompt 2: debug notation.** Given $P=K[R\mid t]$, explain the difference between $t$ and the camera center $C$. Derive $C=-R^Tt$ and give a numerical check. ::: ::: {.cv-card} **Prompt 3: statistical thinking.** Suppose 2D image measurements have independent Gaussian noise with standard deviation 1 pixel. Explain why minimizing squared reprojection error is a maximum likelihood estimator, and discuss when this assumption may fail. ::: ::: {.warning-note} When using AI for camera geometry, ask it to specify coordinate systems. Many wrong answers come from silently switching between world coordinates, camera coordinates, normalized coordinates, and pixel coordinates. ::: ## 12.24 Summary This chapter introduced the mathematical model of a camera as a projective map. The central equation is $$ x\sim PX. $$ The main ideas are: - pinhole projection divides by depth; - homogeneous coordinates turn perspective projection into a linear projective map; - a physical camera matrix can be written as $P=K[R\mid t]$; - intrinsics describe internal pixel geometry; - extrinsics describe camera pose; - an image point back-projects to a 3D ray; - projective geometry preserves incidence but not Euclidean distances; - vanishing points arise from projected directions; - the projection Jacobian controls local sensitivity and uncertainty; - camera estimation can be formulated as least squares; - modern AI systems often use camera projection as a differentiable geometric layer. The next chapter studies camera calibration: how to estimate $K$, distortion parameters, and pose from data. ## Exercises ### Conceptual exercises 1. Explain why a single image point corresponds to a ray in 3D rather than a unique 3D point. 2. What is the difference between normalized image coordinates and pixel coordinates? 3. Why is the equation $x\sim PX$ projective rather than ordinary vector equality? 4. Explain the roles of $K$, $R$, and $t$ in $P=K[R\mid t]$. 5. Why can parallel 3D lines meet at a finite point in an image? 6. What information is lost under perspective projection? 7. Why is the camera center not equal to the vector $t$ in $X_c=RX_w+t$? 8. Describe one situation where orthographic projection is a good approximation. ### Mathematical exercises 1. Let $X=(2,3,6)^T$. Compute its normalized image coordinates under the canonical camera. 2. Let $K=\begin{pmatrix}600&0&320\\0&600&240\\0&0&1\end{pmatrix}$. Project $X=(1,-1,4)^T$ with $R=I$ and $t=0$. 3. Prove that all points on the ray $\lambda(X,Y,Z)^T$, $\lambda>0$, project to the same normalized image point. 4. Derive the Jacobian of $\pi(X,Y,Z)=(X/Z,Y/Z)$. 5. Given $X_c=RX_w+t$, derive the world-coordinate camera center $C=-R^Tt$. 6. Let $x_1=(1,0,1)^T$ and $x_2=(0,1,1)^T$. Compute the line $\ell=x_1\times x_2$. 7. Let two image lines be $\ell_1=(1,0,-2)^T$ and $\ell_2=(0,1,-3)^T$. Compute their intersection. 8. Show that a general $3\times 4$ projective camera matrix has 11 degrees of freedom. 9. Derive the two linear DLT equations from $u=(p_1^TX)/(p_3^TX)$ and $v=(p_2^TX)/(p_3^TX)$. 10. Suppose $\Sigma_X=\sigma^2I_3$. Write the first-order projected covariance $\Sigma_x$ in terms of the projection Jacobian. ### Python exercises 1. Write a function that projects a set of 3D points using $K$, $R$, and $t$. 2. Generate a cube in 3D and plot both the cube and its projected 2D image. 3. Vary the depth of a square and plot its projected size as a function of depth. 4. Implement the projection Jacobian and compare finite-difference approximations with the analytic formula. 5. Simulate noisy 2D projections of known 3D points and estimate $P$ using DLT. 6. Compare algebraic error and reprojection error for the estimated camera matrix. 7. Add radial distortion to projected points and visualize the effect. 8. Implement back-projection from pixel coordinates to 3D rays. ### AI and modeling exercises 1. Ask an AI assistant to derive $P=K[R\mid t]$. Then identify every coordinate system used in the derivation. 2. Ask an AI assistant to explain vanishing points using homogeneous coordinates. Check whether it correctly distinguishes points at infinity from far-away finite points. 3. Use an AI assistant to generate synthetic calibration data, then inspect whether the code correctly handles homogeneous scaling. 4. Design a self-supervised depth-learning loss based on differentiable projection. Specify all inputs, outputs, and assumptions. 5. Explain how a vision transformer could use camera intrinsics as metadata for 3D-aware image understanding. 6. Discuss why learned monocular depth models can predict plausible depth despite the geometric ambiguity of a single image. ## References Classical treatments of projective cameras and multi-view geometry can be found in @hartley2004multiple and @szeliski2022computer. Broader computer vision and image formation background appears in @prince2012computer and @gonzalez2018digital. The connection between camera geometry, learning, and modern differentiable systems will be developed further in later chapters.