14  Camera Calibration

Camera calibration is the mathematical problem of estimating the parameters that turn 3D geometry into pixel measurements. This chapter develops calibration as a problem in projective geometry, numerical linear algebra, nonlinear least squares, and statistical estimation. The target audience is MA Applied Mathematics and MS Statistics students, so the emphasis is on clean models, identifiable parameters, conditioning, and uncertainty rather than software recipes.

14.1 Learning goals

After reading this chapter, students should be able to:

1. distinguish intrinsic parameters, extrinsic parameters, and lens-distortion parameters;
2. write the calibrated camera model from world coordinates to distorted pixels;
3. derive the planar homography relation used in checkerboard calibration;
4. explain how multiple homographies impose linear constraints on \(K\);
5. formulate calibration as a nonlinear least-squares problem using reprojection error;
6. identify common degeneracies and conditioning problems;
7. interpret calibration statistically through likelihood, Jacobians, covariance, and residual diagnostics;
8. connect classical calibration to modern AI systems such as differentiable cameras, self-supervised depth, neural rendering, and visual SLAM.

14.2 13.1 Why calibration matters

Many computer vision tasks require a quantitative relation between 3D coordinates and image coordinates. Suppose a world point is

\[ X_w=(X_w,Y_w,Z_w)^T \in \mathbb{R}^3 \]

and its measured image location is

\[ z=(u,v)^T \in \mathbb{R}^2. \]

A camera model predicts \(z\) from \(X_w\) by a map

\[ z \approx \Pi_\theta(X_w), \]

where \(\theta\) contains camera parameters. Calibration estimates \(\theta\) from known 3D points and observed 2D image points.

Important examples include:

• 3D reconstruction: depth and size estimates require focal length and camera pose;
• robotics: a robot must convert pixels into rays and rays into world coordinates;
• medical imaging: geometric measurements must be metrically meaningful;
• augmented reality: virtual objects must align with the camera image;
• self-supervised learning: differentiable camera models define image-warping losses;
• neural rendering: camera intrinsics and poses determine how 3D fields generate images.

Camera calibration is not only a preprocessing step. It is the mathematical bridge between pixels and metric geometry.

14.3 13.2 The calibrated camera model

From Chapter 12, the ideal projective camera has

\[ \tilde x \sim P X_w^h, \qquad P=K[R\mid t]. \]

Here

\[ X_w^h=\begin{pmatrix}X_w\\Y_w\\Z_w\\1\end{pmatrix} \]

is a homogeneous world point,

\[ R\in SO(3), \qquad t\in \mathbb{R}^3, \]

are extrinsic parameters, and

\[ K=\begin{pmatrix} f_x & s & c_x\\ 0 & f_y & c_y\\ 0 & 0 & 1 \end{pmatrix} \]

is the intrinsic matrix.

The physical interpretation is:

• \(f_x,f_y\): focal lengths in horizontal and vertical pixel units;
• \(s\): skew parameter, usually close to \(0\) for modern cameras;
• \((c_x,c_y)\): principal point, often near the image center.

For a world point \(X_w\), first transform into camera coordinates:

\[ X_c=RX_w+t = \begin{pmatrix}X_c\\Y_c\\Z_c\end{pmatrix}. \]

Normalize by depth:

\[ x_n=\frac{X_c}{Z_c}, \qquad y_n=\frac{Y_c}{Z_c}. \]

Then map normalized coordinates to pixels:

\[ \begin{pmatrix}u\\v\\1\end{pmatrix} = K \begin{pmatrix}x_n\\y_n\\1\end{pmatrix} = \begin{pmatrix} f_x x_n + s y_n + c_x\\ f_y y_n + c_y\\ 1 \end{pmatrix}. \]

If \(s=0\), this reduces to

\[ u=f_x x_n+c_x, \qquad v=f_y y_n+c_y. \]

Calibration problem. Given known 3D calibration points \(X_j\) and measured image points \(z_{ij}\) in image \(i\), estimate

\[ K, \quad \{R_i,t_i\}_{i=1}^m, \quad \text{and possibly distortion parameters}. \]

14.4 13.3 Intrinsic and extrinsic parameters

Calibration separates two kinds of unknowns.

Intrinsics describe the camera itself:

\[ \theta_{\text{int}}=(f_x,f_y,s,c_x,c_y,\text{distortion parameters}). \]

These usually remain fixed across images if the same camera and zoom setting are used.

Extrinsics describe the camera pose relative to the calibration object:

\[ \theta_{\text{ext},i}=(R_i,t_i). \]

These change from image to image.

This separation is crucial. In a calibration sequence with \(m\) images and \(n\) observed points per image, the number of observations is \(2mn\). A typical model has:

\[ 5 \text{ intrinsic parameters} + d \text{ distortion parameters} + 6m \text{ extrinsic parameters}. \]

For a well-posed problem, \(2mn\) must be much larger than the number of unknowns. But counting equations is not enough; the equations must also be informative and well conditioned.

Interactive 13.1. Intrinsic parameters and pixel projection. Move conceptually through the plot: increasing focal length expands projected image coordinates away from the principal point; changing the principal point translates the projected image.

14.5 13.4 Lens distortion

The ideal pinhole model is projective. Real lenses often deviate from this model. The most common correction is to apply distortion in normalized image coordinates before mapping to pixels.

Let

\[ r^2=x_n^2+y_n^2. \]

A common radial distortion model is

\[ x_r=x_n(1+k_1r^2+k_2r^4+k_3r^6), \]

\[ y_r=y_n(1+k_1r^2+k_2r^4+k_3r^6). \]

Tangential distortion can be modeled by

\[ x_d=x_r+2p_1x_ny_n+p_2(r^2+2x_n^2), \]

\[ y_d=y_r+p_1(r^2+2y_n^2)+2p_2x_ny_n. \]

The final pixel coordinates are

\[ u=f_xx_d+s y_d+c_x, \qquad v=f_yy_d+c_y. \]

Barrel and pincushion distortion. If \(k_1<0\), points often move inward near the boundary, producing barrel distortion. If \(k_1>0\), points move outward, producing pincushion distortion. The sign convention can vary across software packages, so always check the exact model.

Interactive 13.2. Radial distortion. The grid shows how a straight calibration pattern is warped by radial distortion. The parameter \(k_1\) controls whether points move inward or outward as their radius increases.

14.6 13.5 Calibration data

A calibration object provides known 3D points. Common choices include:

• a checkerboard pattern;
• a circle grid;
• a Charuco or AprilTag board;
• a known 3D calibration rig.

For a planar checkerboard, it is natural to choose object coordinates with \(Z=0\):

\[ X_j=(X_j,Y_j,0)^T. \]

If the square size is \(a\), then grid coordinates might be

\[ X_j=(a q_j, a r_j, 0)^T, \]

where \(q_j,r_j\) are integer grid indices.

The observed data are image measurements

\[ z_{ij}=(u_{ij},v_{ij})^T, \]

where \(i\) indexes the image and \(j\) indexes the calibration point.

A statistical model is

\[ z_{ij}=\Pi_\theta(X_j;i)+\varepsilon_{ij}, \qquad \varepsilon_{ij}\sim N(0,\sigma^2 I_2), \]

where \(\Pi_\theta(X_j;i)\) is the predicted pixel location in image \(i\).

Under this Gaussian model, maximum likelihood estimation is equivalent to minimizing sum of squared reprojection errors.

14.7 13.6 Planar homographies

The key to many practical calibration methods is the planar homography. Suppose all calibration points lie in the plane \(Z=0\). Then

\[ X_w^h=\begin{pmatrix}X\\Y\\0\\1\end{pmatrix}. \]

Let

\[ R=\begin{pmatrix}r_1&r_2&r_3\end{pmatrix}, \]

where \(r_1,r_2,r_3\) are the columns of \(R\). Then

\[ [R\mid t] \begin{pmatrix}X\\Y\\0\\1\end{pmatrix} = r_1X+r_2Y+t. \]

Therefore

\[ \tilde x \sim K\begin{pmatrix}r_1&r_2&t\end{pmatrix} \begin{pmatrix}X\\Y\\1\end{pmatrix}. \]

Define the homography

\[ H=K\begin{pmatrix}r_1&r_2&t\end{pmatrix}. \]

Then the planar calibration pattern and its image are related by

\[ \tilde x \sim H \tilde X_p, \]

where

\[ \tilde X_p=\begin{pmatrix}X\\Y\\1\end{pmatrix}. \]

Thus, each calibration image gives a homography from the plane to the image.

Interactive 13.3. A checkerboard plane mapped by a homography. A homography maps lines to lines and preserves incidence, but it does not preserve lengths, angles, or parallelism in general.

14.8 13.7 Estimating a homography by DLT

A homography is a \(3\times 3\) matrix

\[ H=\begin{pmatrix} h_{11}&h_{12}&h_{13}\\ h_{21}&h_{22}&h_{23}\\ h_{31}&h_{32}&h_{33} \end{pmatrix}. \]

It is defined only up to scale, so it has \(8\) degrees of freedom.

Given a planar point \((X,Y)\) and an image point \((u,v)\), write

\[ \lambda \begin{pmatrix}u\\v\\1\end{pmatrix} = H \begin{pmatrix}X\\Y\\1\end{pmatrix}. \]

Equivalently,

\[ \tilde x \times H\tilde X_p=0. \]

This gives two independent linear equations in the entries of \(H\). A common form is

\[ \begin{pmatrix} -X&-Y&-1&0&0&0&uX&uY&u\\ 0&0&0&-X&-Y&-1&vX&vY&v \end{pmatrix} h=0, \]

where

\[ h=(h_{11},h_{12},h_{13},h_{21},h_{22},h_{23},h_{31},h_{32},h_{33})^T. \]

Stacking equations from at least four point correspondences gives

\[ Ah=0. \]

The DLT estimate is the right singular vector of \(A\) associated with the smallest singular value.

Normalization matters. Before applying DLT, coordinates should be translated and scaled so that their centroid is near \(0\) and their typical distance from the origin is near \(\sqrt{2}\). This reduces numerical conditioning problems and usually improves the estimate.

14.9 13.8 From homographies to the intrinsic matrix

For the \(i\)th image, let

\[ H_i=\begin{pmatrix}h_{i1}&h_{i2}&h_{i3}\end{pmatrix}, \]

where \(h_{i1},h_{i2},h_{i3}\) are the columns of \(H_i\). Since

\[ H_i=K\begin{pmatrix}r_{i1}&r_{i2}&t_i\end{pmatrix}, \]

we have

\[ K^{-1}h_{i1}=\lambda_i r_{i1}, \qquad K^{-1}h_{i2}=\lambda_i r_{i2}, \]

for some unknown scale \(\lambda_i\).

Because rotation columns are orthonormal,

\[ r_{i1}^T r_{i2}=0, \qquad \|r_{i1}\|^2=\|r_{i2}\|^2. \]

Define the symmetric matrix

\[ B=K^{-T}K^{-1}. \]

Then the orthogonality condition becomes

\[ h_{i1}^T B h_{i2}=0, \]

and the equal-norm condition becomes

\[ h_{i1}^T B h_{i1}=h_{i2}^T B h_{i2}. \]

Each image provides two linear constraints on the six independent entries of the symmetric matrix \(B\).

Let

\[ B=\begin{pmatrix} B_{11}&B_{12}&B_{13}\\ B_{12}&B_{22}&B_{23}\\ B_{13}&B_{23}&B_{33} \end{pmatrix}. \]

For two vectors \(a=(a_1,a_2,a_3)^T\) and \(b=(b_1,b_2,b_3)^T\), define

\[ v(a,b)= \begin{pmatrix} a_1b_1\\ a_1b_2+a_2b_1\\ a_2b_2\\ a_3b_1+a_1b_3\\ a_3b_2+a_2b_3\\ a_3b_3 \end{pmatrix}. \]

Then

\[ a^T B b=v(a,b)^T b_K, \]

where

\[ b_K=(B_{11},B_{12},B_{22},B_{13},B_{23},B_{33})^T. \]

For image \(i\), the two constraints are

\[ v(h_{i1},h_{i2})^T b_K=0, \]

and

\[ \left(v(h_{i1},h_{i1})-v(h_{i2},h_{i2})\right)^T b_K=0. \]

Stacking all images gives

\[ V b_K=0. \]

The solution is again obtained by an SVD: \(b_K\) is the right singular vector corresponding to the smallest singular value of \(V\).

After estimating \(B\), one recovers \(K\) from

\[ B=K^{-T}K^{-1}. \]

Equivalently,

\[ B^{-1}=KK^T. \]

This can be done by Cholesky-type factorization, followed by enforcing the standard upper-triangular form of \(K\) with positive diagonal entries.

Planar calibration converts many image correspondences into homographies, then converts many homographies into linear constraints on \(B=K^{-T}K^{-1}\).

14.10 13.9 Recovering extrinsic parameters

Once \(K\) is known, each homography gives the corresponding pose.

Since

\[ H_i=K\begin{pmatrix}r_{i1}&r_{i2}&t_i\end{pmatrix}, \]

we compute

\[ K^{-1}H_i=\begin{pmatrix}\tilde r_{i1}&\tilde r_{i2}&\tilde t_i\end{pmatrix}. \]

Because \(r_{i1}\) and \(r_{i2}\) should be unit vectors, define

\[ \lambda_i=\frac{1}{\|K^{-1}h_{i1}\|}. \]

Then

\[ r_{i1}=\lambda_i K^{-1}h_{i1}, \qquad r_{i2}=\lambda_i K^{-1}h_{i2}, \qquad t_i=\lambda_i K^{-1}h_{i3}. \]

The third rotation column is

\[ r_{i3}=r_{i1}\times r_{i2}. \]

Due to noise, the matrix

\[ \widehat R_i=\begin{pmatrix}r_{i1}&r_{i2}&r_{i3}\end{pmatrix} \]

may not lie exactly in \(SO(3)\). A common projection onto \(SO(3)\) is:

1. compute the SVD \(\widehat R_i=U\Sigma V^T\);
2. set \(R_i=UV^T\);
3. if \(\det(R_i)<0\), correct the sign of one column.

This is the closest rotation matrix in Frobenius norm.

14.11 13.10 Nonlinear refinement by reprojection error

The linear calibration steps give an initial estimate. Practical calibration usually refines all parameters by minimizing reprojection error.

Let

\[ \theta=(K,\{R_i,t_i\}_{i=1}^m,d), \]

where \(d\) denotes distortion parameters. The reprojection residual is

\[ r_{ij}(\theta)=z_{ij}-\Pi_\theta(X_j;i). \]

The nonlinear least-squares problem is

\[ \min_\theta \sum_{i=1}^m\sum_{j=1}^n \|r_{ij}(\theta)\|_2^2. \]

If some detected points are unreliable, a robust loss may be used:

\[ \min_\theta \sum_{i=1}^m\sum_{j=1}^n \rho\left(\|r_{ij}(\theta)\|_2^2\right), \]

where \(\rho\) grows more slowly than the identity for large residuals.

14.11.1 Rotation parameterization

Optimization should not treat the nine entries of \(R\) as free variables. A rotation has only three degrees of freedom. Common parameterizations include:

• axis-angle vector \(\omega\in\mathbb{R}^3\) with Rodrigues formula;
• unit quaternion \(q\in S^3\);
• Lie algebra update \(R \leftarrow \exp(\delta^\wedge)R\).

For small applied mathematics and statistics projects, the axis-angle representation is often the easiest to implement.

Interactive 13.4. Reprojection-error landscape. Calibration is nonlinear. A good initialization keeps the optimizer in the basin of the correct focal length and distortion parameters.

14.12 13.11 Statistical interpretation

Assume independent Gaussian measurement noise:

\[ z_{ij}=\Pi_\theta(X_j;i)+\varepsilon_{ij}, \qquad \varepsilon_{ij}\sim N(0,\sigma^2 I_2). \]

The negative log-likelihood is, up to constants,

\[ \frac{1}{2\sigma^2} \sum_{i,j}\|z_{ij}-\Pi_\theta(X_j;i)\|_2^2. \]

Thus least squares is maximum likelihood under isotropic Gaussian image noise.

Let \(r(\theta)\) be the stacked residual vector and let

\[ J=\frac{\partial r}{\partial \theta} \]

be the residual Jacobian at the optimum. A local approximation gives

\[ r(\theta+\delta) \approx r(\theta)+J\delta. \]

If the model is correct and the noise variance is \(\sigma^2\), the approximate parameter covariance is

\[ \operatorname{Cov}(\widehat\theta) \approx \sigma^2(J^TJ)^{-1}. \]

This formula makes conditioning concrete: if \(J^TJ\) has small eigenvalues, some parameter directions are poorly determined.

Interactive 13.5. Parameter uncertainty ellipse. The ellipse shows a local covariance approximation for two calibration parameters. Narrow ellipses indicate strong information; long ellipses indicate weakly identifiable parameter combinations.

14.13 13.12 Conditioning and degenerate configurations

Calibration can fail even when many points are detected. The issue is not just the number of equations, but their geometry.

Common problems include:

• all views have nearly the same pose;
• the calibration pattern is always centered and fronto-parallel;
• the image points cover only a small central region;
• focus or zoom changes during acquisition;
• corner detections are biased or blurred;
• the distortion model is too flexible for the data;
• the wrong square size or coordinate convention is used.

A good calibration sequence should include:

• multiple tilts of the planar board;
• coverage near the image boundaries;
• different depths and translations;
• sharp, well-exposed images;
• enough points per image and enough images;
• residual checks after optimization.

Interactive 13.6. Pose diversity and conditioning. A calibration set with many nearly identical views can be worse than a smaller set with diverse poses.

14.14 13.13 Residual diagnostics

After calibration, the most important diagnostic is not only the average reprojection error. One should also inspect the residual field

\[ r_{ij}=\begin{pmatrix}u_{ij}-\widehat u_{ij}\\v_{ij}-\widehat v_{ij}\end{pmatrix}. \]

Useful checks include:

• mean reprojection error;
• median reprojection error;
• maximum residual;
• residual vectors plotted across the image;
• residuals versus radius from the principal point;
• residuals by image;
• residuals by board location;
• whether residuals show systematic patterns.

If residuals form radial patterns, the distortion model may be wrong. If residuals are much larger in some images, those images may have bad detections or motion blur. If residuals are anisotropic, the pixel-noise model may be too simple.

14.15 13.14 Calibration and AI systems

Modern vision models often contain camera geometry explicitly or implicitly.

14.15.1 Differentiable cameras

In self-supervised depth, a pixel in one view is projected into another view using predicted depth and known or estimated camera pose. A photometric loss may have the form

\[ \sum_p \left\| I_1(p)-I_2\left(\pi(KT D(p)K^{-1}\tilde p)\right) \right\|. \]

Here \(D(p)\) is predicted depth, \(T\) is relative pose, and \(K\) is the intrinsic matrix. Incorrect calibration can produce a biased learning signal.

14.15.2 Neural rendering

Neural radiance field methods and Gaussian splatting systems use cameras to cast rays through pixels. For a pixel \(p=(u,v)\), the normalized ray direction is

\[ \begin{pmatrix}x_n\\y_n\\1\end{pmatrix} =K^{-1}\begin{pmatrix}u\\v\\1\end{pmatrix}. \]

Thus intrinsics determine ray directions. Pose errors create inconsistent multi-view supervision.

14.15.3 Learned calibration

Some AI systems estimate intrinsics, focal length, distortion, or camera pose from images. These methods can be useful when metadata is missing, but they should still be evaluated against geometric constraints. A learned estimate is not a replacement for understanding identifiability and residual error.

AI caution. A model can learn correlations that look like camera geometry without satisfying camera geometry. For applications requiring measurement, reconstruction, or safety, the projection equations and residual diagnostics remain essential.

14.16 13.15 Python example: projection with distortion

The following code simulates a calibrated camera and radial distortion.

import numpy as np

K = np.array([
    [800.0, 0.0, 320.0],
    [0.0, 820.0, 240.0],
    [0.0, 0.0, 1.0]
])

k1, k2 = -0.15, 0.03

# 3D points in camera coordinates
Xc = np.array([
    [-0.5, -0.3, 2.0],
    [ 0.0,  0.0, 2.0],
    [ 0.5,  0.4, 2.0],
    [ 0.8, -0.4, 3.0]
])

x = Xc[:, 0] / Xc[:, 2]
y = Xc[:, 1] / Xc[:, 2]
r2 = x**2 + y**2
scale = 1 + k1 * r2 + k2 * r2**2
xd = x * scale
yd = y * scale

hom = np.column_stack([xd, yd, np.ones_like(xd)])
pixels = (K @ hom.T).T
pixels = pixels[:, :2] / pixels[:, 2:3]

print(pixels)

14.17 13.16 Python example: homography by DLT

This example estimates a homography from planar correspondences.

import numpy as np


def estimate_homography_dlt(world_xy, image_uv):
    A = []
    for (X, Y), (u, v) in zip(world_xy, image_uv):
        A.append([-X, -Y, -1, 0, 0, 0, u*X, u*Y, u])
        A.append([0, 0, 0, -X, -Y, -1, v*X, v*Y, v])
    A = np.asarray(A, dtype=float)
    _, _, Vt = np.linalg.svd(A)
    h = Vt[-1]
    H = h.reshape(3, 3)
    return H / H[2, 2]

# Synthetic square mapped to a quadrilateral
world_xy = np.array([
    [0, 0], [1, 0], [1, 1], [0, 1],
    [0.5, 0], [1, 0.5], [0.5, 1], [0, 0.5]
], dtype=float)

image_uv = np.array([
    [120, 100], [420, 130], [390, 360], [100, 340],
    [270, 116], [405, 245], [245, 350], [110, 220]
], dtype=float)

H = estimate_homography_dlt(world_xy, image_uv)
print(H)

For real data, add coordinate normalization before constructing \(A\).

14.18 13.17 Python example: nonlinear calibration refinement

The next example sketches a focal-length and radial-distortion refinement. It is intentionally small so students can see the objective clearly.

import numpy as np
from scipy.optimize import least_squares

# Suppose Xc are known camera-coordinate points for this toy example.
# In full calibration, each image also has unknown rotation and translation.
Xc = np.array([
    [-0.6, -0.4, 2.0], [-0.2, 0.3, 2.2], [0.4, -0.2, 2.1],
    [0.7, 0.5, 2.5], [0.0, 0.0, 2.0]
])

observed = np.array([
    [80.2, 73.1], [245.5, 351.4], [472.3, 160.7],
    [545.1, 405.6], [320.3, 240.1]
])

cx, cy = 320.0, 240.0


def residual(params):
    fx, fy, k1 = params
    x = Xc[:, 0] / Xc[:, 2]
    y = Xc[:, 1] / Xc[:, 2]
    r2 = x*x + y*y
    scale = 1 + k1 * r2
    u = fx * x * scale + cx
    v = fy * y * scale + cy
    pred = np.column_stack([u, v])
    return (observed - pred).ravel()

result = least_squares(residual, x0=np.array([750.0, 750.0, 0.0]))
print(result.x)
print("RMS reprojection error:", np.sqrt(np.mean(result.fun**2)))

14.19 13.18 Optional OpenCV workflow

Many practical projects use OpenCV for corner detection and calibration. The mathematical objective is still the same reprojection-error minimization described above.

import cv2
import numpy as np

pattern_size = (9, 6)       # interior corners per checkerboard row and column
square_size = 0.025         # meters, for example

objp = np.zeros((pattern_size[0] * pattern_size[1], 3), np.float32)
objp[:, :2] = np.mgrid[0:pattern_size[0], 0:pattern_size[1]].T.reshape(-1, 2)
objp *= square_size

object_points = []
image_points = []

for filename in image_filenames:
    img = cv2.imread(filename)
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    ok, corners = cv2.findChessboardCorners(gray, pattern_size)
    if ok:
        corners = cv2.cornerSubPix(
            gray, corners, winSize=(11, 11), zeroZone=(-1, -1),
            criteria=(cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-6)
        )
        object_points.append(objp)
        image_points.append(corners)

rms, K, dist, rvecs, tvecs = cv2.calibrateCamera(
    object_points, image_points, gray.shape[::-1], None, None
)

print("RMS reprojection error:", rms)
print("K =\n", K)
print("distortion =", dist.ravel())

14.20 13.19 TensorFlow example: differentiable projection layer

A differentiable camera layer is useful in self-supervised depth, pose learning, neural rendering, and differentiable bundle adjustment.

import tensorflow as tf


def project_points_tf(Xc, fx, fy, cx, cy, k1):
    x = Xc[:, 0] / Xc[:, 2]
    y = Xc[:, 1] / Xc[:, 2]
    r2 = x*x + y*y
    scale = 1.0 + k1 * r2
    u = fx * x * scale + cx
    v = fy * y * scale + cy
    return tf.stack([u, v], axis=1)

Xc = tf.constant([
    [-0.5, -0.3, 2.0],
    [ 0.2,  0.1, 2.5],
    [ 0.6, -0.2, 3.0]
], dtype=tf.float32)

z_obs = tf.constant([
    [120.0, 116.0],
    [384.0, 273.0],
    [477.0, 186.0]
], dtype=tf.float32)

fx = tf.Variable(750.0)
fy = tf.Variable(750.0)
k1 = tf.Variable(0.0)
cx = tf.constant(320.0)
cy = tf.constant(240.0)

with tf.GradientTape() as tape:
    z_pred = project_points_tf(Xc, fx, fy, cx, cy, k1)
    loss = tf.reduce_mean(tf.square(z_obs - z_pred))

grads = tape.gradient(loss, [fx, fy, k1])
print("loss:", loss.numpy())
print("gradients:", [g.numpy() for g in grads])

14.21 13.20 AI-assisted study prompts

Use an AI assistant as a mathematical critic, not as an authority. Good prompts include:

1. Derivation check. “Starting from \(H=K[r_1\ r_2\ t]\), derive the two constraints on \(B=K^{-T}K^{-1}\). Show every algebra step and explain where rotation orthogonality is used.”
2. Failure-mode analysis. “Explain why a set of only fronto-parallel checkerboard images gives poor calibration. Relate your explanation to the singular values of the calibration design matrix.”
3. Code review. “Review this homography DLT implementation. Check coordinate normalization, point ordering, homogeneous scaling, and residual computation.”
4. Statistical interpretation. “Given residual vectors after calibration, suggest diagnostic plots that would reveal radial distortion, outlier images, and biased corner detections.”
5. Model comparison. “Compare a pinhole-only model with a radial-distortion model using validation reprojection error. What overfitting risks should I watch for?”

AI verification rule. Ask the assistant to produce a numerical check. For example, after estimating \(K,R,t\), verify that \(R^TR\approx I\), \(\det(R)\approx 1\), and residuals are small in pixels.

14.22 13.21 Summary

Camera calibration estimates the parameters of the map

\[ X_w \mapsto z. \]

The ideal projective model is

\[ \tilde x\sim K[R\mid t]X_w^h, \]

while practical models add lens distortion and measurement noise.

For planar calibration objects, each image gives a homography

\[ H_i=K[r_{i1}\ r_{i2}\ t_i]. \]

The rotation constraints imply linear equations in

\[ B=K^{-T}K^{-1}. \]

This gives an initial estimate of the intrinsic matrix. Nonlinear optimization then refines intrinsics, extrinsics, and distortion parameters by minimizing reprojection error.

From a statistics viewpoint, calibration is a maximum likelihood problem under a pixel-noise model. From an applied mathematics viewpoint, it is a structured nonlinear least-squares problem whose success depends on conditioning, parameterization, and geometric diversity.

14.23 Exercises

14.23.1 Conceptual exercises

1. Explain the difference between intrinsic and extrinsic parameters.
2. Why is the camera matrix \(P\) defined only up to scale?
3. Why does a planar checkerboard image determine a homography rather than a full 3D camera matrix?
4. What does reprojection error measure? Why is it not the same as 3D reconstruction error?
5. Describe a calibration dataset that is likely to be poorly conditioned.

14.23.2 Mathematical exercises

6. Starting from \(P=K[R\mid t]\), derive the pixel-coordinate equations \(u=f_xX_c/Z_c+c_x\) and \(v=f_yY_c/Z_c+c_y\) for \(s=0\).
7. Prove that a homography has \(8\) degrees of freedom.
8. Derive the two DLT equations for one point correspondence \((X,Y)\leftrightarrow(u,v)\).
9. Prove that \(h_1^T B h_2=0\) follows from \(r_1^Tr_2=0\).
10. Prove that \(h_1^T B h_1=h_2^T B h_2\) follows from \(\|r_1\|=\|r_2\|=1\).
11. Show that projecting a noisy \(3\times 3\) matrix \(M\) to the closest rotation matrix in Frobenius norm can be solved using the SVD of \(M\).
12. Derive the Jacobian of \(u=f_xX/Z+c_x\) and \(v=f_yY/Z+c_y\) with respect to \((X,Y,Z)\).
13. Suppose \(z_j=\Pi_\theta(X_j)+\varepsilon_j\) with \(\varepsilon_j\sim N(0,\sigma^2I_2)\). Show that maximum likelihood is equivalent to least squares.

14.23.3 Python exercises

14. Generate synthetic checkerboard points, choose \(K,R,t\), project them, add Gaussian noise, and estimate a homography.
15. Compare homography estimation with and without coordinate normalization.
16. Simulate radial distortion with several values of \(k_1\). Plot the original grid and distorted grid.
17. Implement reprojection error for a known \(K,R,t\) and compute RMS error.
18. Use scipy.optimize.least_squares to estimate \(f_x,f_y,c_x,c_y\) from synthetic data with known extrinsics.
19. Implement a residual diagnostic plot: residual vectors overlaid on the image plane.
20. Use OpenCV on your own checkerboard images and report \(K\), distortion coefficients, and residual diagnostics.

14.23.4 AI and modeling exercises

21. Ask an AI assistant to explain the Zhang calibration method. Then identify any missing assumptions or unclear scale conventions.
22. Ask an AI assistant to generate a homography-estimation implementation. Test it on synthetic data and write a short critique.
23. Design a validation experiment comparing a pinhole-only model against a model with radial distortion.
24. Explain how an incorrect focal length affects self-supervised depth learning.
25. Discuss when learned single-image calibration might be acceptable and when explicit calibration is necessary.

14.24 Notes and references

The projective geometry foundation follows Hartley and Zisserman (2004) and Faugeras (1993). Planar calibration with flexible patterns is classically associated with Zhang (2000). Lens distortion models build on long-standing photogrammetry and camera-modeling literature, including Brown (1971). The OpenCV-style calibration workflow is a practical implementation of the same mathematical ideas.

Brown, Duane C. 1971. “Close-Range Camera Calibration.” Photogrammetric Engineering 37 (8): 855–66.

Faugeras, Olivier. 1993. Three-Dimensional Computer Vision: A Geometric Viewpoint. MIT Press.

Hartley, Richard, and Andrew Zisserman. 2004. Multiple View Geometry in Computer Vision. 2nd ed. Cambridge University Press. https://www.robots.ox.ac.uk/~vgg/hzbook/.

Zhang, Zhengyou. 2000. “A Flexible New Technique for Camera Calibration.” IEEE Transactions on Pattern Analysis and Machine Intelligence 22 (11): 1330–34.