16 Stereo Vision and Depth

Stereo vision estimates scene depth from two or more images. Mathematically, stereo combines epipolar geometry, projective cameras, discrete optimization, statistical estimation, and modern learned matching. This chapter develops rectified stereo, disparity, triangulation, matching costs, regularization, uncertainty, and neural stereo/depth systems from a clean mathematical point of view.

16.1 Learning goals

After reading this chapter, students should be able to:

explain why rectified stereo reduces correspondence search from two dimensions to one dimension;
derive the depth-disparity relationship \[ Z = \frac{fb}{d}; \]
formulate triangulation as ray intersection, linear least squares, and reprojection-error minimization;
distinguish depth, inverse depth, disparity, and range;
construct local stereo matching costs using SSD, SAD, normalized cross-correlation, Census-style transforms, and learned features;
explain why matching ambiguity occurs in textureless, repeated, occluded, and reflective regions;
write stereo as an energy minimization problem over a disparity field;
describe dynamic programming, graph cuts, belief propagation, and semi-global matching at a mathematical level;
propagate disparity uncertainty into depth uncertainty;
connect classical stereo to learned cost volumes, self-supervised depth, and differentiable rendering losses.

16.2 15.1 From epipolar geometry to stereo depth

Chapter 14 showed that corresponding points in two views must satisfy the epipolar constraint

\[ x_2^T F x_1 = 0. \]

This constraint says that once a point $x_1$ is selected in image 1, the possible match $x_2$ in image 2 must lie on an epipolar line. General two-view matching is still difficult because epipolar lines may have arbitrary slopes and positions. Stereo vision becomes much simpler after rectification.

A stereo pair is rectified when corresponding epipolar lines are horizontal scanlines. Then a point

\[ x_L=(u_L,v,1)^T \]

in the left image can match only points of the form

\[ x_R=(u_R,v,1)^T \]

in the right image. The correspondence search becomes one-dimensional along the same row.

The horizontal displacement

\[ d = u_L-u_R \]

is called disparity. For a standard rectified stereo pair with focal length $f$ and baseline $b$, disparity determines depth by

\[ Z = \frac{fb}{d}. \]

This simple equation is the central formula of metric stereo vision.

Stereo depth is projective geometry plus correspondence. Epipolar geometry gives the search constraint; matching gives disparity; camera geometry converts disparity into depth.

Interactive 15.1. Disparity and depth. Increase the disparity and observe how the estimated depth decreases. Small disparity errors at large depth create large depth errors.

16.3 15.2 Rectified camera model

Consider two pinhole cameras with parallel optical axes. Let the left camera center be

\[ C_L=(0,0,0)^T \]

and the right camera center be

\[ C_R=(b,0,0)^T. \]

Let the common focal length be $f$, and suppose image coordinates are centered at the principal point. A 3D point

\[ X=(X,Y,Z)^T \]

projects to the left image as

\[ u_L = f\frac{X}{Z}, \qquad v_L = f\frac{Y}{Z}. \]

The same point, expressed relative to the right camera center, has coordinate $(X-b,Y,Z)^T$. Hence it projects to the right image as

\[ u_R = f\frac{X-b}{Z}, \qquad v_R = f\frac{Y}{Z}. \]

Therefore

\[ v_L=v_R, \]

so corresponding points lie on the same horizontal scanline, and

\[ d = u_L-u_R = f\frac{X}{Z}-f\frac{X-b}{Z} = \frac{fb}{Z}. \]

Solving for $Z$ gives

\[ Z=\frac{fb}{d}. \]

This derivation assumes a rectified stereo pair, known baseline, known focal length, and no lens distortion after calibration and undistortion.

16.3.1 Pixel coordinates and principal point

If the principal point is $(c_x,c_y)$, then

\[ u_L = f\frac{X}{Z}+c_x, \qquad u_R = f\frac{X-b}{Z}+c_x, \]

and the principal point cancels in the disparity:

\[ d = u_L-u_R=\frac{fb}{Z}. \]

If left and right images have different intrinsic matrices, rectification usually maps both images into a common virtual camera coordinate system. After rectification, the same disparity-depth relation holds with the focal length and baseline of that virtual stereo system.

16.3.2 Depth, range, and inverse depth

Depth $Z$ is the coordinate along the camera optical axis. Range is Euclidean distance from the camera center:

\[ r=\|X\|_2=\sqrt{X^2+Y^2+Z^2}. \]

For small fields of view, depth and range may be similar. For wide-angle cameras, they can differ substantially.

Many algorithms use inverse depth

\[ \rho=\frac{1}{Z} \]

because disparity is proportional to inverse depth:

\[ d = fb\rho. \]

Thus uniform disparity quantization corresponds to uniform inverse-depth quantization, not uniform depth quantization.

16.4 15.3 Geometry of triangulation

Stereo depth is one form of triangulation. In general, suppose two calibrated cameras observe normalized image points

\[ \hat x_1=(x_1,y_1,1)^T, \qquad \hat x_2=(x_2,y_2,1)^T. \]

The corresponding 3D point lies on two rays:

\[ X = C_1 + \lambda_1 r_1, \qquad X = C_2 + \lambda_2 r_2, \]

where $r_1$ and $r_2$ are unit or non-unit direction vectors determined by the image coordinates.

In exact noiseless geometry, the two rays intersect. With image noise, they usually become skew lines in 3D. Then triangulation becomes an estimation problem: choose $X$ that best explains both image measurements.

Interactive 15.2. Triangulation angle. A larger baseline or wider intersection angle gives better depth accuracy. A small intersection angle makes triangulation ill-conditioned.

16.4.1 Closest point between two rays

Let

\[ L_1(\lambda_1)=C_1+\lambda_1 r_1, \qquad L_2(\lambda_2)=C_2+\lambda_2 r_2. \]

A natural triangulated point is the midpoint of the shortest segment connecting the two rays. The parameters solve

\[ \min_{\lambda_1,\lambda_2} \left\|C_1+\lambda_1r_1-C_2-\lambda_2r_2\right\|_2^2. \]

Let

\[ A = \begin{pmatrix} r_1 & -r_2 \end{pmatrix}, \qquad c = C_2-C_1. \]

Then

\[ A \begin{pmatrix}\lambda_1\\ \lambda_2\end{pmatrix} \approx c \]

in least squares. The normal equations are

\[ A^T A \begin{pmatrix}\lambda_1\\ \lambda_2\end{pmatrix} = A^T c. \]

The conditioning of $A^TA$ depends on the angle between the rays. If the rays are nearly parallel, then $A^TA$ is nearly singular and depth becomes unstable.

16.4.2 Linear triangulation with camera matrices

For projective cameras

\[ x_i \sim P_i X, \]

where $X\in\mathbb{P}^3$, the cross-product constraint

\[ x_i\times P_iX=0 \]

gives two independent linear equations per view. For two views, we stack equations to obtain

\[ A X = 0. \]

The homogeneous least-squares solution is the right singular vector of $A$ associated with the smallest singular value. This is linear triangulation.

Linear triangulation is easy and fast, but it minimizes an algebraic error rather than a statistically meaningful image-plane error.

16.4.3 Reprojection-error triangulation

A more geometrically meaningful estimate solves

\[ \min_X \sum_{i=1}^2 \left\|x_i - \pi(P_iX)\right\|_2^2, \]

where $\pi$ denotes perspective division. This is nonlinear least squares. Starting from the linear triangulation solution, one can refine $X$ using Gauss–Newton or Levenberg–Marquardt.

If image noise is approximately Gaussian with covariance $\sigma^2I$, reprojection-error minimization is the maximum likelihood estimator.

16.5 15.4 Disparity maps as functions

A dense stereo algorithm estimates a disparity value at each pixel:

\[ d:\Omega\to \mathbb{R}_{\ge 0}, \]

where $\Omega$ is the left-image grid. The depth map is then

\[ Z(i,j)=\frac{fb}{d(i,j)}. \]

Because disparity may be undefined in occluded regions, reflective regions, or textureless regions, it is often useful to represent stereo output as a pair

\[ (d(i,j),\; q(i,j)), \]

where $q(i,j)$ is a confidence score or uncertainty estimate.

A stereo method must solve three coupled problems:

matching: which point in the right image corresponds to a left pixel?
regularization: how should neighboring disparities relate?
visibility: which points are visible in one image but not the other?

This is why stereo is both a geometry problem and a statistical inference problem.

16.5.1 Disparity sign convention

This book uses

\[ d=u_L-u_R. \]

For a standard stereo rig with the right camera translated to the right, points in front have positive disparity. Some software libraries use other conventions, especially after image rectification or coordinate-system changes. Always check the sign convention before converting disparity to depth.

16.6 15.5 Local matching costs

The simplest stereo strategy compares local patches. Let $I_L$ and $I_R$ be rectified grayscale images. For a pixel $p=(i,j)$ in the left image and disparity $d$, compare the left patch around $(i,j)$ with the right patch around $(i,j-d)$.

Let $W$ be a local window. The sum of squared differences cost is

\[ C_{\mathrm{SSD}}(p,d) = \sum_{q\in W(p)} \left(I_L(q)-I_R(q-(0,d))\right)^2. \]

The sum of absolute differences cost is

\[ C_{\mathrm{SAD}}(p,d) = \sum_{q\in W(p)} \left|I_L(q)-I_R(q-(0,d))\right|. \]

Here $q-(0,d)$ means shifting the column coordinate by $d$ in the right image.

16.6.1 Normalized cross-correlation

SSD and SAD are sensitive to brightness changes. Normalized cross-correlation compares centered and normalized patches:

\[ \operatorname{NCC}(a,b) = \frac{\sum_{k=1}^m (a_k-\bar a)(b_k-\bar b)} {\sqrt{\sum_{k=1}^m(a_k-\bar a)^2}\sqrt{\sum_{k=1}^m(b_k-\bar b)^2}}. \]

Since high NCC means better match, a cost can be defined as

\[ C_{\mathrm{NCC}}=1-\operatorname{NCC}. \]

NCC is invariant to affine intensity changes of the form

\[ b\approx \alpha a+\beta, \qquad \alpha>0. \]

16.6.2 Census-style transforms

The Census transform replaces raw intensities with relative comparisons inside a local window. For a center pixel $p$, define a binary descriptor

\[ \tau_p(q)=\mathbf{1}\{I(q)<I(p)\}, \qquad q\in W(p)\setminus\{p\}. \]

Two descriptors are compared by Hamming distance. This is robust to monotone intensity transformations because it uses local order rather than absolute intensity.

16.6.3 Learned matching features

Modern stereo systems often replace raw patches with learned features:

\[ \phi_\theta(I_L,p), \qquad \phi_\theta(I_R,p-d). \]

A simple learned feature cost is

\[ C_\theta(p,d)= \left\|\phi_\theta(I_L,p)-\phi_\theta(I_R,p-d)\right\|_2^2. \]

If features are normalized, cosine similarity is also common:

\[ C_\theta(p,d)=1- \left\langle \frac{\phi_\theta(I_L,p)}{\|\phi_\theta(I_L,p)\|_2}, \frac{\phi_\theta(I_R,p-d)}{\|\phi_\theta(I_R,p-d)\|_2} \right\rangle. \]

Interactive 15.3. Matching cost curve. A good local match gives a sharp minimum over disparity. Repeated texture or low texture creates ambiguous cost curves.

16.7 15.6 Winner-take-all stereo

Given a cost volume

\[ C(p,d), \qquad p\in\Omega, \quad d\in\{0,1,\dots,D_{\max}\}, \]

the simplest disparity estimate is

\[ \hat d(p)=\arg\min_d C(p,d). \]

This is called winner-take-all stereo. It is local because each pixel is estimated independently.

Winner-take-all is easy to implement and useful as a baseline. However, it fails in many cases:

textureless regions have flat cost curves;
repeated textures have multiple local minima;
object boundaries mix foreground and background inside a patch;
occluded points have no true match;
specular surfaces violate brightness constancy;
slanted surfaces may not be well represented by fronto-parallel windows.

Mathematically, local stereo is a collection of independent one-dimensional estimation problems. Real scenes require spatial structure.

16.8 15.7 Stereo as energy minimization

A more global view estimates the entire disparity field by minimizing an energy

\[ E(d)= \sum_{p\in\Omega} C(p,d_p) + \lambda \sum_{(p,q)\in\mathcal{N}} V(d_p,d_q). \]

The first term is a data term from matching. The second term is a smoothness or regularization term over neighboring pixels.

Common smoothness penalties include:

\[ V(d_p,d_q)=|d_p-d_q|, \]

\[ V(d_p,d_q)=\min(|d_p-d_q|,\tau), \]

and the Potts penalty

\[ V(d_p,d_q)=\mathbf{1}\{d_p\ne d_q\}. \]

The truncated penalty permits depth discontinuities because it stops increasing after a threshold. This is important because disparity is smooth on surfaces but discontinuous across object boundaries.

16.8.1 MAP interpretation

The energy can be interpreted as negative log posterior:

\[ E(d) = -\log p(d\mid I_L,I_R)+\text{constant}. \]

For example, suppose

\[ p(I_L,I_R\mid d) \propto \exp\left(-\sum_p C(p,d_p)\right) \]

and

\[ p(d) \propto \exp\left(-\lambda\sum_{(p,q)\in\mathcal{N}}V(d_p,d_q)\right). \]

Then minimizing $E(d)$ is equivalent to maximum a posteriori estimation.

Interactive 15.4. Data term versus smoothness. Increasing regularization removes noise but can over-smooth true depth boundaries.

16.9 15.8 Dynamic programming along scanlines

If regularization is only along one scanline, stereo can be solved by dynamic programming. For one row, let

\[ d_1,d_2,\dots,d_n \]

be disparities along the row. Consider

\[ E(d_1,\dots,d_n)= \sum_{j=1}^n C(j,d_j) + \lambda\sum_{j=2}^n V(d_j,d_{j-1}). \]

Define the value function

\[ M_j(d)= C(j,d)+ \min_{d'}\{M_{j-1}(d')+\lambda V(d,d')\}. \]

The optimal energy ending at disparity $d$ at location $j$ is $M_j(d)$. After computing all $M_j$, backtracking recovers the optimal disparity sequence.

This is exact for a chain graph. The full image grid contains loops, so exact dynamic programming is generally not available for the full two-dimensional problem.

16.10 15.9 Semi-global matching

Semi-global matching approximates two-dimensional regularization by aggregating dynamic-programming costs along many one-dimensional paths. A common recurrence has the schematic form

\[ L_r(p,d)=C(p,d)+ \min\left\{ \begin{array}{l} L_r(p-r,d),\\ L_r(p-r,d-1)+P_1,\\ L_r(p-r,d+1)+P_1,\\ \min_k L_r(p-r,k)+P_2 \end{array} \right\} - \min_k L_r(p-r,k), \]

where $r$ is a path direction, $P_1$ penalizes small disparity changes, and $P_2$ penalizes larger jumps. Costs from multiple directions are summed:

\[ S(p,d)=\sum_r L_r(p,d). \]

The final disparity is

\[ \hat d(p)=\arg\min_d S(p,d). \]

Semi-global matching is attractive because it combines strong performance with manageable computational cost.

16.11 15.10 Graph cuts and belief propagation

Stereo energy minimization can also be studied using graph-based methods. The disparity field is a labeling problem on an image grid. If each pixel is a node and neighboring pixels are connected by edges, the energy has the form

\[ E(d)=\sum_p D_p(d_p)+\sum_{(p,q)}V_{pq}(d_p,d_q). \]

For binary labels and submodular pairwise terms, graph cuts can minimize the energy exactly. For multiple disparity labels, algorithms such as alpha-expansion and alpha-beta swap give approximate solutions under appropriate metric or semi-metric assumptions.

Belief propagation treats the labeling problem as inference in a Markov random field. Messages have the form

\[ m_{p\to q}(d_q) = \min_{d_p} \left[ D_p(d_p)+V_{pq}(d_p,d_q)+ \sum_{s\in N(p)\setminus q}m_{s\to p}(d_p) \right]. \]

On trees, min-sum belief propagation is exact. On image grids, loopy belief propagation is approximate but often useful.

16.12 15.11 Subpixel disparity

Disparity is not inherently integer-valued. Integer disparity search is a discretization. Subpixel refinement improves depth accuracy.

Suppose the matching cost near the best integer disparity $d_0$ has values

\[ C_- = C(d_0-1), \qquad C_0 = C(d_0), \qquad C_+ = C(d_0+1). \]

Fit a quadratic

\[ q(t)=at^2+bt+c \]

through $t=-1,0,1$. The minimizer is

\[ \Delta = \frac{C_- - C_+}{2(C_- -2C_0 + C_+)}, \]

so the subpixel disparity estimate is

\[ \hat d = d_0 + \Delta. \]

This formula is meaningful only if

\[ C_- -2C_0 + C_+>0, \]

which means the local quadratic is convex.

16.13 15.12 Uncertainty propagation

Depth depends nonlinearly on disparity:

\[ Z=\frac{fb}{d}. \]

Differentiate with respect to $d$:

\[ \frac{\partial Z}{\partial d} = -\frac{fb}{d^2}. \]

If disparity has standard deviation $\sigma_d$, then by first-order error propagation,

\[ \sigma_Z \approx \left|\frac{\partial Z}{\partial d}\right|\sigma_d = \frac{fb}{d^2}\sigma_d. \]

Since $d=fb/Z$, this can also be written as

\[ \sigma_Z \approx \frac{Z^2}{fb}\sigma_d. \]

Thus depth uncertainty grows quadratically with depth. This is one reason stereo depth is accurate for nearby objects but unreliable far away.

Interactive 15.5. Depth uncertainty. Depth error grows approximately quadratically with distance when disparity noise is constant.

16.13.1 Confidence from the cost curve

A sharp minimum in $C(p,d)$ suggests high confidence. A flat or multimodal cost curve suggests ambiguity. Common confidence indicators include:

\[ C_{2\mathrm{nd}}-C_{1\mathrm{st}}, \]

where $C_{1\mathrm{st}}$ and $C_{2\mathrm{nd}}$ are the best and second-best matching costs; the curvature of the cost curve near the minimum; and left-right consistency.

16.13.2 Left-right consistency

Estimate disparities both from left to right and from right to left. A left pixel $p$ with disparity $d_L(p)$ maps to a right pixel $p' = p-(0,d_L(p))$. The estimate is consistent if

\[ |d_L(p)-d_R(p')|\le \epsilon. \]

Large inconsistency suggests occlusion, mismatch, or unreliable estimation.

16.14 15.13 Occlusions and visibility

A point visible in the left image may be hidden in the right image. Such a point has no valid correspondence. This violates the basic matching assumption.

Occlusions are common near depth discontinuities. If foreground objects are closer to the cameras than the background, the foreground shifts more between views because it has larger disparity. This can reveal or hide background regions.

A stereo method can handle occlusions using:

explicit occlusion labels;
left-right consistency checks;
robust penalties that tolerate unmatched pixels;
confidence maps;
ordering constraints along scanlines;
learned occlusion prediction.

Visibility turns stereo from a pure correspondence problem into a scene-geometry problem.

16.15 15.14 Plane sweep stereo

Plane sweep stereo generalizes disparity search using hypothetical depth planes. For each candidate depth $Z_k$ or plane $\Pi_k$, warp the right image into the left image coordinate system and measure photometric agreement.

For a fronto-parallel plane at depth $Z$, rectified stereo reduces to a horizontal shift. For a general plane, the warp is a homography:

\[ x_R \sim H_\Pi x_L. \]

A cost volume is formed by stacking matching costs over candidate depths:

\[ C(x,Z_k) = \rho\left(I_L(x)-I_R(w_k(x))\right), \]

where $w_k$ is the warp induced by the $k$th depth hypothesis. The selected depth is often

\[ \hat Z(x)=\arg\min_{Z_k} C(x,Z_k). \]

Plane sweep is especially important for multi-view stereo and differentiable depth estimation.

Interactive 15.6. Plane sweep intuition. A correct depth plane aligns image content; an incorrect plane leaves residual misalignment.

16.16 15.15 Multi-view stereo

Two-view stereo uses one baseline. Multi-view stereo uses more images. If a point is visible in several cameras, its depth can be estimated more reliably.

Given cameras $P_1,\dots,P_m$ and image observations $x_1, \dots,x_m$, multi-view triangulation solves

\[ \min_X \sum_{i=1}^m \left\|x_i-\pi(P_iX)\right\|_2^2. \]

Dense multi-view stereo estimates a surface or depth map consistent across many images. Methods include:

plane sweep and cost-volume fusion;
patch-based multi-view stereo;
volumetric reconstruction;
depth-map fusion;
neural implicit surfaces and differentiable rendering.

The mathematical tradeoff is the same as in two-view stereo: data fidelity, regularization, visibility, and uncertainty.

16.17 15.16 Statistical model of stereo matching

Assume a rectified stereo pair and a true disparity field $d^*$. A simplified observation model is

\[ I_L(p)=S(p)+\varepsilon_L(p), \]

\[ I_R(p-(0,d^*(p)))=S(p)+\varepsilon_R(p), \]

where $S(p)$ is an unknown scene radiance pattern and $\varepsilon_L,\varepsilon_R$ are noise terms.

If the noise is independent Gaussian with variance $\sigma^2$, then the negative log likelihood of a patch match is proportional to SSD:

\[ -\log p(\text{patches}\mid d) \propto \sum_{q\in W(p)} \left(I_L(q)-I_R(q-(0,d))\right)^2. \]

If the noise is Laplace, SAD is more appropriate:

\[ -\log p(\text{patches}\mid d) \propto \sum_{q\in W(p)} \left|I_L(q)-I_R(q-(0,d))\right|. \]

Robust losses such as Huber or Charbonnier losses are used when there are outliers:

\[ \rho_\epsilon(r)=\sqrt{r^2+\epsilon^2}. \]

This statistical viewpoint helps students understand why different matching costs behave differently.

16.18 15.17 Learned stereo and cost volumes

Modern stereo networks usually build a learned cost volume. Let

\[ F_L = \Phi_\theta(I_L), \qquad F_R = \Phi_\theta(I_R) \]

be learned feature maps. A cost volume may be built by correlation:

\[ C(p,d)= \left\langle F_L(p), F_R(p-(0,d))\right\rangle, \]

or by concatenation:

\[ V(p,d)= \left[F_L(p),F_R(p-(0,d))\right]. \]

A neural network then regularizes and decodes this cost volume:

\[ \hat d = G_\theta(C). \]

This is a learned version of classical stereo. The architecture learns the feature representation, the matching cost, and the regularization procedure.

16.18.1 Soft argmin

Many networks convert a cost volume into a probability distribution over disparity:

\[ p(d\mid p)= \frac{\exp(-C(p,d))}{\sum_k \exp(-C(p,k))}. \]

The predicted disparity is the expectation

\[ \hat d(p)=\sum_d d\,p(d\mid p). \]

This is called soft argmin. It is differentiable, unlike the hard argmin. However, if the distribution is multimodal, the expectation may lie between two plausible disparities and may not correspond to a valid match.

Interactive 15.7. Soft argmin and ambiguity. A unimodal cost distribution gives a meaningful expectation. A bimodal distribution can produce a disparity that is not one of the plausible modes.

16.19 15.18 Self-supervised stereo and photometric loss

If ground-truth depth is unavailable, stereo models can be trained using image reconstruction. Given a predicted disparity $\hat d$, warp the right image to reconstruct the left image:

\[ \hat I_L(p)=I_R(p-(0,\hat d(p))). \]

The photometric loss is

\[ \mathcal{L}_{\mathrm{photo}} = \sum_p \rho\left(I_L(p)-\hat I_L(p)\right). \]

A smoothness penalty is often added:

\[ \mathcal{L}_{\mathrm{smooth}} = \sum_p \left| \nabla \hat d(p) \right|e^{-\alpha \|\nabla I_L(p)\|}. \]

The exponential term allows disparity discontinuities at image edges. The total objective is

\[ \mathcal{L} = \mathcal{L}_{\mathrm{photo}} + \lambda\mathcal{L}_{\mathrm{smooth}} + \mu\mathcal{L}_{\mathrm{consistency}}. \]

This connects stereo to differentiable rendering, inverse problems, and self-supervised learning.

16.19.1 Important caveat

Photometric self-supervision assumes that corresponding points have similar appearance. This assumption fails for occlusions, specular reflections, transparent objects, non-Lambertian surfaces, motion, exposure changes, and rolling-shutter effects. Modern systems therefore use robust losses, masks, multi-scale training, left-right consistency, and learned priors.

16.20 15.19 Monocular depth as learned inverse problem

Stereo uses two images and known geometry. Monocular depth estimates depth from a single image:

\[ I \mapsto Z. \]

This problem is ill-posed because a single image does not determine metric scale. A small object nearby and a large object far away can have the same image size.

A monocular depth network learns statistical regularities from data:

\[ \hat Z = f_\theta(I). \]

The model may use cues such as perspective, texture gradients, object size, shading, focus, occlusion, and semantic context. Unlike stereo, monocular depth depends heavily on learned priors.

For MA Applied Math and MS Statistics students, the key distinction is:

Stereo depth is primarily geometry constrained by correspondence. Monocular depth is primarily statistical inference constrained by learned priors.

16.21 15.20 Evaluation metrics

For ground-truth depth $Z_i$ and prediction $\hat Z_i$, common metrics include:

\[ \operatorname{AbsRel} =\frac{1}{N}\sum_i \frac{|Z_i-\hat Z_i|}{Z_i}, \]

\[ \operatorname{RMSE} =\sqrt{\frac{1}{N}\sum_i(Z_i-\hat Z_i)^2}, \]

and log RMSE

\[ \operatorname{RMSE}_{\log} =\sqrt{\frac{1}{N}\sum_i(\log Z_i-\log \hat Z_i)^2}. \]

For disparity,

\[ \operatorname{EPE} =\frac{1}{N}\sum_i |d_i-\hat d_i| \]

is the endpoint error in one dimension. Another common metric is the percentage of pixels with disparity error larger than a threshold:

\[ \frac{1}{N}\sum_i \mathbf{1}\{|d_i-\hat d_i|>\tau\}. \]

Metrics should be interpreted with care. A method can have good average error while failing at thin structures, depth boundaries, reflective surfaces, or far regions.

16.22 15.21 Python example: disparity to depth

The following example converts a synthetic disparity map into depth and visualizes uncertainty. It is intentionally small enough for independent study.

import numpy as np
import matplotlib.pyplot as plt

H, W = 120, 180
u = np.linspace(-1, 1, W)
v = np.linspace(-1, 1, H)
U, V = np.meshgrid(u, v)

# Synthetic disparity: foreground bump plus slanted background
D = 8 + 4 * (U + 1) + 12 * np.exp(-8 * (U**2 + V**2))

f = 700.0      # pixels
b = 0.12       # meters
Z = f * b / D

sigma_d = 0.25
sigma_Z = (f * b / D**2) * sigma_d

fig, ax = plt.subplots(1, 3, figsize=(12, 3.5))
im0 = ax[0].imshow(D, cmap="magma")
ax[0].set_title("disparity")
plt.colorbar(im0, ax=ax[0], fraction=0.046)

im1 = ax[1].imshow(Z, cmap="viridis")
ax[1].set_title("depth Z")
plt.colorbar(im1, ax=ax[1], fraction=0.046)

im2 = ax[2].imshow(sigma_Z, cmap="inferno")
ax[2].set_title("depth uncertainty")
plt.colorbar(im2, ax=ax[2], fraction=0.046)

for a in ax:
    a.axis("off")
plt.tight_layout()
plt.show()

16.23 15.22 Python example: winner-take-all block matching

This example implements a simple block-matching stereo baseline. It is not meant to be state of the art; it is meant to expose the mathematical structure.

import numpy as np
import matplotlib.pyplot as plt
from scipy.ndimage import uniform_filter

rng = np.random.default_rng(15)
H, W = 96, 160
x = np.linspace(0, 1, W)
y = np.linspace(0, 1, H)
X, Y = np.meshgrid(x, y)

left = (
    0.5 + 0.25 * np.sin(18 * X) +
    0.20 * np.cos(15 * Y) +
    0.25 * np.exp(-60 * ((X - 0.45)**2 + (Y - 0.55)**2))
)
left += 0.03 * rng.normal(size=(H, W))
left = np.clip(left, 0, 1)

true_d = 6 + 10 * np.exp(-40 * ((X - 0.45)**2 + (Y - 0.55)**2))
true_d_int = np.rint(true_d).astype(int)

right = np.zeros_like(left)
for i in range(H):
    for j in range(W):
        jr = j - true_d_int[i, j]
        if 0 <= jr < W:
            right[i, jr] = left[i, j]
right += 0.03 * rng.normal(size=(H, W))
right = np.clip(right, 0, 1)

Dmax = 24
window = 7
cost_volume = np.empty((H, W, Dmax + 1))

for d in range(Dmax + 1):
    shifted = np.zeros_like(right)
    shifted[:, d:] = right[:, :W - d]
    sq = (left - shifted) ** 2
    cost_volume[:, :, d] = uniform_filter(sq, size=window)

est_d = np.argmin(cost_volume, axis=2)

plt.figure(figsize=(10, 3.2))
plt.subplot(1, 3, 1)
plt.imshow(left, cmap="gray")
plt.title("left image")
plt.axis("off")

plt.subplot(1, 3, 2)
plt.imshow(true_d, cmap="magma")
plt.title("true disparity")
plt.axis("off")

plt.subplot(1, 3, 3)
plt.imshow(est_d, cmap="magma", vmin=0, vmax=Dmax)
plt.title("estimated disparity")
plt.axis("off")
plt.tight_layout()
plt.show()

16.24 15.23 Python example: soft argmin

import numpy as np

D = np.arange(0, 32)
true_d = 12.4
cost = (D - true_d)**2 / 8
prob = np.exp(-cost)
prob = prob / prob.sum()
soft = np.sum(D * prob)
hard = D[np.argmin(cost)]

print("hard argmin:", hard)
print("soft argmin:", soft)

For a unimodal cost curve, the soft argmin can give a useful subpixel estimate. For a bimodal curve, inspect the distribution rather than trusting the expectation blindly.

16.25 15.24 TensorFlow example: differentiable photometric stereo loss

This example sketches the structure of a self-supervised stereo loss. It uses one-dimensional horizontal warping by interpolation.

import tensorflow as tf


def horizontal_warp(right, disparity):
    """Warp right image toward the left image using horizontal disparity.

    right:     tensor of shape (B, H, W, C)
    disparity: tensor of shape (B, H, W, 1), measured in pixels
    """
    B = tf.shape(right)[0]
    H = tf.shape(right)[1]
    W = tf.shape(right)[2]

    x = tf.cast(tf.range(W), tf.float32)
    x = tf.reshape(x, (1, 1, W, 1))
    x = tf.tile(x, (B, H, 1, 1))

    # For left reconstruction, sample right at x - disparity.
    x_src = x - disparity
    x0 = tf.floor(x_src)
    x1 = x0 + 1.0
    w1 = x_src - x0
    w0 = 1.0 - w1

    x0i = tf.cast(tf.clip_by_value(x0, 0.0, tf.cast(W - 1, tf.float32)), tf.int32)
    x1i = tf.cast(tf.clip_by_value(x1, 0.0, tf.cast(W - 1, tf.float32)), tf.int32)

    batch = tf.reshape(tf.range(B), (B, 1, 1, 1))
    batch = tf.tile(batch, (1, H, W, 1))
    yy = tf.reshape(tf.range(H), (1, H, 1, 1))
    yy = tf.tile(yy, (B, 1, W, 1))

    idx0 = tf.concat([batch, yy, x0i], axis=-1)
    idx1 = tf.concat([batch, yy, x1i], axis=-1)

    val0 = tf.gather_nd(right, idx0)
    val1 = tf.gather_nd(right, idx1)

    return w0 * val0 + w1 * val1


def charbonnier(x, eps=1e-3):
    return tf.sqrt(tf.square(x) + eps**2)


def photometric_loss(left, right, disparity):
    recon = horizontal_warp(right, disparity)
    return tf.reduce_mean(charbonnier(left - recon))

16.26 15.25 AI-assisted study components

16.26.1 AI component 1: derive and audit

Ask an AI assistant:

Derive the stereo formula $Z=fb/d$ from two rectified pinhole cameras. Explain all coordinate assumptions.

Then audit the answer. It should explicitly mention:

two cameras have parallel optical axes after rectification;
the baseline is known;
image coordinates are in pixels or normalized units consistently;
the disparity convention is fixed;
the formula becomes unstable as $d\to 0$.

16.26.2 AI component 2: compare models

Ask an AI assistant to compare SSD, NCC, Census, and learned-feature matching. Require it to discuss invariance to brightness changes, robustness to noise, computational cost, and failure modes.

16.26.3 AI component 3: debug a stereo pipeline

Give an AI assistant a stereo pipeline with poor depth results and ask it to generate possible causes. A good diagnostic list should include calibration error, rectification error, wrong baseline units, wrong disparity sign, insufficient texture, specular surfaces, occlusions, and incorrect depth conversion.

16.26.4 AI component 4: design a differentiable stereo loss

Ask for a loss function combining photometric reconstruction, smoothness, left-right consistency, and occlusion masks. Then identify which parts are geometry-based and which parts are learned priors.

16.27 15.26 Summary

Stereo vision turns two images into depth by combining geometry and matching. Rectification reduces correspondence search to horizontal scanlines. Disparity is inversely proportional to depth:

\[ Z=\frac{fb}{d}. \]

The main practical challenge is not the depth formula but the correspondence problem. Local matching creates a cost volume, global methods regularize the disparity field, and modern neural methods learn features, costs, and regularizers. Statistical thinking clarifies noise models, uncertainty propagation, confidence, and failure modes.

16.28 Exercises

16.28.1 Conceptual exercises

Explain why a single image point determines a ray, while a stereo correspondence determines a 3D point.
Why does rectification make stereo matching easier?
What happens to depth uncertainty as disparity approaches zero?
Why do repeated textures create ambiguous disparity estimates?
Explain why occlusions violate the basic patch-matching assumption.
Compare stereo depth and monocular depth in terms of geometry and statistical priors.

16.28.2 Mathematical exercises

Derive $Z=fb/d$ from the rectified stereo camera model.
Show that disparity is proportional to inverse depth.
Let $Z=fb/d$. Derive the first-order uncertainty formula \[ \sigma_Z\approx \frac{Z^2}{fb}\sigma_d. \]
Derive the least-squares system for the closest point between two rays.
Write the dynamic-programming recurrence for scanline stereo with \[ V(d,d')=|d-d'|. \]
Derive the subpixel quadratic interpolation formula for the disparity correction $\Delta$.
Suppose $C(p,d)$ is a negative log likelihood and $V$ is a negative log prior. Show that stereo energy minimization is MAP estimation.
Explain why soft argmin can be problematic for a bimodal probability distribution.

16.28.3 Python exercises

Implement disparity-to-depth conversion and plot depth uncertainty for several values of $f$, $b$, and $\sigma_d$.
Implement winner-take-all block matching using SSD.
Replace SSD with SAD and NCC. Compare results on synthetic images.
Add a left-right consistency check to a block-matching pipeline.
Implement subpixel disparity refinement using a quadratic fit.
Build a small cost volume and visualize $C(p,d)$ for easy, ambiguous, and occluded pixels.
Implement soft argmin and test it on unimodal and bimodal cost distributions.
Use OpenCV stereo functions on a rectified stereo pair and compare with your own simple baseline.

16.28.4 AI/modeling exercises

Ask an AI assistant to generate a block-matching stereo implementation. Check whether the shift direction and disparity sign are correct.
Ask an AI assistant to explain left-right consistency. Test whether it correctly handles occlusions.
Ask for a learned stereo architecture using a cost volume. Identify where classical stereo ideas appear inside the network.
Use an AI assistant to propose robustness improvements for reflective or transparent surfaces.
Design a self-supervised stereo loss and identify all assumptions behind its photometric term.
Ask an AI assistant to explain why monocular depth is scale ambiguous. Check whether the answer distinguishes projective ambiguity from statistical priors.

16.29 Notes and further reading

The mathematical geometry of stereo follows the projective-camera framework in Hartley and Zisserman (Hartley and Zisserman 2004), Faugeras (Faugeras 1993), and Ma et al. (Ma et al. 2004). Scharstein and Szeliski’s taxonomy (Scharstein and Szeliski 2002) is a classic reference for dense two-frame stereo methods. Semi-global matching was introduced by Hirschmuller (Hirschmuller 2008). For modern learned stereo, see GC-Net (Kendall et al. 2017), PSMNet (Chang and Chen 2018), and related cost-volume methods. For self-supervised monocular and stereo depth learning, Eigen et al. (Eigen et al. 2014) and Godard et al. (Godard et al. 2017) are useful starting points.

Chang, Jia-Ren, and Yong-Sheng Chen. 2018. “Pyramid Stereo Matching Network.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5410–18.

Eigen, David, Christian Puhrsch, and Rob Fergus. 2014. “Depth Map Prediction from a Single Image Using a Multi-Scale Deep Network.” Advances in Neural Information Processing Systems 27.

Faugeras, Olivier. 1993. Three-Dimensional Computer Vision: A Geometric Viewpoint. MIT Press.

Godard, Clément, Oisin Mac Aodha, and Gabriel J. Brostow. 2017. “Unsupervised Monocular Depth Estimation with Left-Right Consistency.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 270–79.

Hartley, Richard, and Andrew Zisserman. 2004. Multiple View Geometry in Computer Vision. 2nd ed. Cambridge University Press. https://www.robots.ox.ac.uk/~vgg/hzbook/.

Hirschmuller, Heiko. 2008. “Stereo Processing by Semiglobal Matching and Mutual Information.” IEEE Transactions on Pattern Analysis and Machine Intelligence 30 (2): 328–41.

Kendall, Alex, Hayk Martirosyan, Saumitro Dasgupta, et al. 2017. “End-to-End Learning of Geometry and Context for Deep Stereo Regression.” Proceedings of the IEEE International Conference on Computer Vision, 66–75.

Ma, Yi, Stefano Soatto, Jana Kosecka, and Shankar S. Sastry. 2004. An Invitation to 3-d Vision: From Images to Geometric Models. Springer.

Scharstein, Daniel, and Richard Szeliski. 2002. “A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms.” International Journal of Computer Vision 47 (1–3): 7–42.

# Stereo Vision and Depth ::: {.hero-box} Stereo vision estimates scene depth from two or more images. Mathematically, stereo combines epipolar geometry, projective cameras, discrete optimization, statistical estimation, and modern learned matching. This chapter develops rectified stereo, disparity, triangulation, matching costs, regularization, uncertainty, and neural stereo/depth systems from a clean mathematical point of view. ::: ## Learning goals After reading this chapter, students should be able to: 1. explain why rectified stereo reduces correspondence search from two dimensions to one dimension; 2. derive the depth-disparity relationship $$ Z = \frac{fb}{d}; $$ 3. formulate triangulation as ray intersection, linear least squares, and reprojection-error minimization; 4. distinguish depth, inverse depth, disparity, and range; 5. construct local stereo matching costs using SSD, SAD, normalized cross-correlation, Census-style transforms, and learned features; 6. explain why matching ambiguity occurs in textureless, repeated, occluded, and reflective regions; 7. write stereo as an energy minimization problem over a disparity field; 8. describe dynamic programming, graph cuts, belief propagation, and semi-global matching at a mathematical level; 9. propagate disparity uncertainty into depth uncertainty; 10. connect classical stereo to learned cost volumes, self-supervised depth, and differentiable rendering losses. ## 15.1 From epipolar geometry to stereo depth Chapter 14 showed that corresponding points in two views must satisfy the epipolar constraint $$ x_2^T F x_1 = 0. $$ This constraint says that once a point $x_1$ is selected in image 1, the possible match $x_2$ in image 2 must lie on an epipolar line. General two-view matching is still difficult because epipolar lines may have arbitrary slopes and positions. Stereo vision becomes much simpler after **rectification**. A stereo pair is **rectified** when corresponding epipolar lines are horizontal scanlines. Then a point $$ x_L=(u_L,v,1)^T $$ in the left image can match only points of the form $$ x_R=(u_R,v,1)^T $$ in the right image. The correspondence search becomes one-dimensional along the same row. The horizontal displacement $$ d = u_L-u_R $$ is called **disparity**. For a standard rectified stereo pair with focal length $f$ and baseline $b$, disparity determines depth by $$ Z = \frac{fb}{d}. $$ This simple equation is the central formula of metric stereo vision. ::: {.math-spine} Stereo depth is projective geometry plus correspondence. Epipolar geometry gives the search constraint; matching gives disparity; camera geometry converts disparity into depth. ::: ::: {.interactive-card} **Interactive 15.1. Disparity and depth.** Increase the disparity and observe how the estimated depth decreases. Small disparity errors at large depth create large depth errors. <div id="cv-ch15-disparity-depth-demo" class="plotly-demo"></div> ::: ## 15.2 Rectified camera model Consider two pinhole cameras with parallel optical axes. Let the left camera center be $$ C_L=(0,0,0)^T $$ and the right camera center be $$ C_R=(b,0,0)^T. $$ Let the common focal length be $f$, and suppose image coordinates are centered at the principal point. A 3D point $$ X=(X,Y,Z)^T $$ projects to the left image as $$ u_L = f\frac{X}{Z}, \qquad v_L = f\frac{Y}{Z}. $$ The same point, expressed relative to the right camera center, has coordinate $(X-b,Y,Z)^T$. Hence it projects to the right image as $$ u_R = f\frac{X-b}{Z}, \qquad v_R = f\frac{Y}{Z}. $$ Therefore $$ v_L=v_R, $$ so corresponding points lie on the same horizontal scanline, and $$ d = u_L-u_R = f\frac{X}{Z}-f\frac{X-b}{Z} = \frac{fb}{Z}. $$ Solving for $Z$ gives $$ Z=\frac{fb}{d}. $$ This derivation assumes a rectified stereo pair, known baseline, known focal length, and no lens distortion after calibration and undistortion. ### Pixel coordinates and principal point If the principal point is $(c_x,c_y)$, then $$ u_L = f\frac{X}{Z}+c_x, \qquad u_R = f\frac{X-b}{Z}+c_x, $$ and the principal point cancels in the disparity: $$ d = u_L-u_R=\frac{fb}{Z}. $$ If left and right images have different intrinsic matrices, rectification usually maps both images into a common virtual camera coordinate system. After rectification, the same disparity-depth relation holds with the focal length and baseline of that virtual stereo system. ### Depth, range, and inverse depth Depth $Z$ is the coordinate along the camera optical axis. Range is Euclidean distance from the camera center: $$ r=\|X\|_2=\sqrt{X^2+Y^2+Z^2}. $$ For small fields of view, depth and range may be similar. For wide-angle cameras, they can differ substantially. Many algorithms use **inverse depth** $$ \rho=\frac{1}{Z} $$ because disparity is proportional to inverse depth: $$ d = fb\rho. $$ Thus uniform disparity quantization corresponds to uniform inverse-depth quantization, not uniform depth quantization. ## 15.3 Geometry of triangulation Stereo depth is one form of triangulation. In general, suppose two calibrated cameras observe normalized image points $$ \hat x_1=(x_1,y_1,1)^T, \qquad \hat x_2=(x_2,y_2,1)^T. $$ The corresponding 3D point lies on two rays: $$ X = C_1 + \lambda_1 r_1, \qquad X = C_2 + \lambda_2 r_2, $$ where $r_1$ and $r_2$ are unit or non-unit direction vectors determined by the image coordinates. In exact noiseless geometry, the two rays intersect. With image noise, they usually become skew lines in 3D. Then triangulation becomes an estimation problem: choose $X$ that best explains both image measurements. ::: {.interactive-card} **Interactive 15.2. Triangulation angle.** A larger baseline or wider intersection angle gives better depth accuracy. A small intersection angle makes triangulation ill-conditioned. <div id="cv-ch15-triangulation-angle-demo" class="plotly-demo"></div> ::: ### Closest point between two rays Let $$ L_1(\lambda_1)=C_1+\lambda_1 r_1, \qquad L_2(\lambda_2)=C_2+\lambda_2 r_2. $$ A natural triangulated point is the midpoint of the shortest segment connecting the two rays. The parameters solve $$ \min_{\lambda_1,\lambda_2} \left\|C_1+\lambda_1r_1-C_2-\lambda_2r_2\right\|_2^2. $$ Let $$ A = \begin{pmatrix} r_1 & -r_2 \end{pmatrix}, \qquad c = C_2-C_1. $$ Then $$ A \begin{pmatrix}\lambda_1\\ \lambda_2\end{pmatrix} \approx c $$ in least squares. The normal equations are $$ A^T A \begin{pmatrix}\lambda_1\\ \lambda_2\end{pmatrix} = A^T c. $$ The conditioning of $A^TA$ depends on the angle between the rays. If the rays are nearly parallel, then $A^TA$ is nearly singular and depth becomes unstable. ### Linear triangulation with camera matrices For projective cameras $$ x_i \sim P_i X, $$ where $X\in\mathbb{P}^3$, the cross-product constraint $$ x_i\times P_iX=0 $$ gives two independent linear equations per view. For two views, we stack equations to obtain $$ A X = 0. $$ The homogeneous least-squares solution is the right singular vector of $A$ associated with the smallest singular value. This is **linear triangulation**. Linear triangulation is easy and fast, but it minimizes an algebraic error rather than a statistically meaningful image-plane error. ### Reprojection-error triangulation A more geometrically meaningful estimate solves $$ \min_X \sum_{i=1}^2 \left\|x_i - \pi(P_iX)\right\|_2^2, $$ where $\pi$ denotes perspective division. This is nonlinear least squares. Starting from the linear triangulation solution, one can refine $X$ using Gauss--Newton or Levenberg--Marquardt. If image noise is approximately Gaussian with covariance $\sigma^2I$, reprojection-error minimization is the maximum likelihood estimator. ## 15.4 Disparity maps as functions A dense stereo algorithm estimates a disparity value at each pixel: $$ d:\Omega\to \mathbb{R}_{\ge 0}, $$ where $\Omega$ is the left-image grid. The depth map is then $$ Z(i,j)=\frac{fb}{d(i,j)}. $$ Because disparity may be undefined in occluded regions, reflective regions, or textureless regions, it is often useful to represent stereo output as a pair $$ (d(i,j),\; q(i,j)), $$ where $q(i,j)$ is a confidence score or uncertainty estimate. A stereo method must solve three coupled problems: 1. **matching:** which point in the right image corresponds to a left pixel? 2. **regularization:** how should neighboring disparities relate? 3. **visibility:** which points are visible in one image but not the other? This is why stereo is both a geometry problem and a statistical inference problem. ### Disparity sign convention This book uses $$ d=u_L-u_R. $$ For a standard stereo rig with the right camera translated to the right, points in front have positive disparity. Some software libraries use other conventions, especially after image rectification or coordinate-system changes. Always check the sign convention before converting disparity to depth. ## 15.5 Local matching costs The simplest stereo strategy compares local patches. Let $I_L$ and $I_R$ be rectified grayscale images. For a pixel $p=(i,j)$ in the left image and disparity $d$, compare the left patch around $(i,j)$ with the right patch around $(i,j-d)$. Let $W$ be a local window. The **sum of squared differences** cost is $$ C_{\mathrm{SSD}}(p,d) = \sum_{q\in W(p)} \left(I_L(q)-I_R(q-(0,d))\right)^2. $$ The **sum of absolute differences** cost is $$ C_{\mathrm{SAD}}(p,d) = \sum_{q\in W(p)} \left|I_L(q)-I_R(q-(0,d))\right|. $$ Here $q-(0,d)$ means shifting the column coordinate by $d$ in the right image. ### Normalized cross-correlation SSD and SAD are sensitive to brightness changes. Normalized cross-correlation compares centered and normalized patches: $$ \operatorname{NCC}(a,b) = \frac{\sum_{k=1}^m (a_k-\bar a)(b_k-\bar b)} {\sqrt{\sum_{k=1}^m(a_k-\bar a)^2}\sqrt{\sum_{k=1}^m(b_k-\bar b)^2}}. $$ Since high NCC means better match, a cost can be defined as $$ C_{\mathrm{NCC}}=1-\operatorname{NCC}. $$ NCC is invariant to affine intensity changes of the form $$ b\approx \alpha a+\beta, \qquad \alpha>0. $$ ### Census-style transforms The Census transform replaces raw intensities with relative comparisons inside a local window. For a center pixel $p$, define a binary descriptor $$ \tau_p(q)=\mathbf{1}\{I(q)<I(p)\}, \qquad q\in W(p)\setminus\{p\}. $$ Two descriptors are compared by Hamming distance. This is robust to monotone intensity transformations because it uses local order rather than absolute intensity. ### Learned matching features Modern stereo systems often replace raw patches with learned features: $$ \phi_\theta(I_L,p), \qquad \phi_\theta(I_R,p-d). $$ A simple learned feature cost is $$ C_\theta(p,d)= \left\|\phi_\theta(I_L,p)-\phi_\theta(I_R,p-d)\right\|_2^2. $$ If features are normalized, cosine similarity is also common: $$ C_\theta(p,d)=1- \left\langle \frac{\phi_\theta(I_L,p)}{\|\phi_\theta(I_L,p)\|_2}, \frac{\phi_\theta(I_R,p-d)}{\|\phi_\theta(I_R,p-d)\|_2} \right\rangle. $$ ::: {.interactive-card} **Interactive 15.3. Matching cost curve.** A good local match gives a sharp minimum over disparity. Repeated texture or low texture creates ambiguous cost curves. <div id="cv-ch15-matching-cost-demo" class="plotly-demo"></div> ::: ## 15.6 Winner-take-all stereo Given a cost volume $$ C(p,d), \qquad p\in\Omega, \quad d\in\{0,1,\dots,D_{\max}\}, $$ the simplest disparity estimate is $$ \hat d(p)=\arg\min_d C(p,d). $$ This is called **winner-take-all** stereo. It is local because each pixel is estimated independently. Winner-take-all is easy to implement and useful as a baseline. However, it fails in many cases: - textureless regions have flat cost curves; - repeated textures have multiple local minima; - object boundaries mix foreground and background inside a patch; - occluded points have no true match; - specular surfaces violate brightness constancy; - slanted surfaces may not be well represented by fronto-parallel windows. Mathematically, local stereo is a collection of independent one-dimensional estimation problems. Real scenes require spatial structure. ## 15.7 Stereo as energy minimization A more global view estimates the entire disparity field by minimizing an energy $$ E(d)= \sum_{p\in\Omega} C(p,d_p) + \lambda \sum_{(p,q)\in\mathcal{N}} V(d_p,d_q). $$ The first term is a data term from matching. The second term is a smoothness or regularization term over neighboring pixels. Common smoothness penalties include: $$ V(d_p,d_q)=|d_p-d_q|, $$ $$ V(d_p,d_q)=\min(|d_p-d_q|,\tau), $$ and the Potts penalty $$ V(d_p,d_q)=\mathbf{1}\{d_p\ne d_q\}. $$ The truncated penalty permits depth discontinuities because it stops increasing after a threshold. This is important because disparity is smooth on surfaces but discontinuous across object boundaries. ### MAP interpretation The energy can be interpreted as negative log posterior: $$ E(d) = -\log p(d\mid I_L,I_R)+\text{constant}. $$ For example, suppose $$ p(I_L,I_R\mid d) \propto \exp\left(-\sum_p C(p,d_p)\right) $$ and $$ p(d) \propto \exp\left(-\lambda\sum_{(p,q)\in\mathcal{N}}V(d_p,d_q)\right). $$ Then minimizing $E(d)$ is equivalent to maximum a posteriori estimation. ::: {.interactive-card} **Interactive 15.4. Data term versus smoothness.** Increasing regularization removes noise but can over-smooth true depth boundaries. <div id="cv-ch15-smoothness-demo" class="plotly-demo"></div> ::: ## 15.8 Dynamic programming along scanlines If regularization is only along one scanline, stereo can be solved by dynamic programming. For one row, let $$ d_1,d_2,\dots,d_n $$ be disparities along the row. Consider $$ E(d_1,\dots,d_n)= \sum_{j=1}^n C(j,d_j) + \lambda\sum_{j=2}^n V(d_j,d_{j-1}). $$ Define the value function $$ M_j(d)= C(j,d)+ \min_{d'}\{M_{j-1}(d')+\lambda V(d,d')\}. $$ The optimal energy ending at disparity $d$ at location $j$ is $M_j(d)$. After computing all $M_j$, backtracking recovers the optimal disparity sequence. This is exact for a chain graph. The full image grid contains loops, so exact dynamic programming is generally not available for the full two-dimensional problem. ## 15.9 Semi-global matching Semi-global matching approximates two-dimensional regularization by aggregating dynamic-programming costs along many one-dimensional paths. A common recurrence has the schematic form $$ L_r(p,d)=C(p,d)+ \min\left\{ \begin{array}{l} L_r(p-r,d),\\ L_r(p-r,d-1)+P_1,\\ L_r(p-r,d+1)+P_1,\\ \min_k L_r(p-r,k)+P_2 \end{array} \right\} - \min_k L_r(p-r,k), $$ where $r$ is a path direction, $P_1$ penalizes small disparity changes, and $P_2$ penalizes larger jumps. Costs from multiple directions are summed: $$ S(p,d)=\sum_r L_r(p,d). $$ The final disparity is $$ \hat d(p)=\arg\min_d S(p,d). $$ Semi-global matching is attractive because it combines strong performance with manageable computational cost. ## 15.10 Graph cuts and belief propagation Stereo energy minimization can also be studied using graph-based methods. The disparity field is a labeling problem on an image grid. If each pixel is a node and neighboring pixels are connected by edges, the energy has the form $$ E(d)=\sum_p D_p(d_p)+\sum_{(p,q)}V_{pq}(d_p,d_q). $$ For binary labels and submodular pairwise terms, graph cuts can minimize the energy exactly. For multiple disparity labels, algorithms such as alpha-expansion and alpha-beta swap give approximate solutions under appropriate metric or semi-metric assumptions. Belief propagation treats the labeling problem as inference in a Markov random field. Messages have the form $$ m_{p\to q}(d_q) = \min_{d_p} \left[ D_p(d_p)+V_{pq}(d_p,d_q)+ \sum_{s\in N(p)\setminus q}m_{s\to p}(d_p) \right]. $$ On trees, min-sum belief propagation is exact. On image grids, loopy belief propagation is approximate but often useful. ## 15.11 Subpixel disparity Disparity is not inherently integer-valued. Integer disparity search is a discretization. Subpixel refinement improves depth accuracy. Suppose the matching cost near the best integer disparity $d_0$ has values $$ C_- = C(d_0-1), \qquad C_0 = C(d_0), \qquad C_+ = C(d_0+1). $$ Fit a quadratic $$ q(t)=at^2+bt+c $$ through $t=-1,0,1$. The minimizer is $$ \Delta = \frac{C_- - C_+}{2(C_- -2C_0 + C_+)}, $$ so the subpixel disparity estimate is $$ \hat d = d_0 + \Delta. $$ This formula is meaningful only if $$ C_- -2C_0 + C_+>0, $$ which means the local quadratic is convex. ## 15.12 Uncertainty propagation Depth depends nonlinearly on disparity: $$ Z=\frac{fb}{d}. $$ Differentiate with respect to $d$: $$ \frac{\partial Z}{\partial d} = -\frac{fb}{d^2}. $$ If disparity has standard deviation $\sigma_d$, then by first-order error propagation, $$ \sigma_Z \approx \left|\frac{\partial Z}{\partial d}\right|\sigma_d = \frac{fb}{d^2}\sigma_d. $$ Since $d=fb/Z$, this can also be written as $$ \sigma_Z \approx \frac{Z^2}{fb}\sigma_d. $$ Thus depth uncertainty grows quadratically with depth. This is one reason stereo depth is accurate for nearby objects but unreliable far away. ::: {.interactive-card} **Interactive 15.5. Depth uncertainty.** Depth error grows approximately quadratically with distance when disparity noise is constant. <div id="cv-ch15-uncertainty-demo" class="plotly-demo"></div> ::: ### Confidence from the cost curve A sharp minimum in $C(p,d)$ suggests high confidence. A flat or multimodal cost curve suggests ambiguity. Common confidence indicators include: $$ C_{2\mathrm{nd}}-C_{1\mathrm{st}}, $$ where $C_{1\mathrm{st}}$ and $C_{2\mathrm{nd}}$ are the best and second-best matching costs; the curvature of the cost curve near the minimum; and left-right consistency. ### Left-right consistency Estimate disparities both from left to right and from right to left. A left pixel $p$ with disparity $d_L(p)$ maps to a right pixel $p' = p-(0,d_L(p))$. The estimate is consistent if $$ |d_L(p)-d_R(p')|\le \epsilon. $$ Large inconsistency suggests occlusion, mismatch, or unreliable estimation. ## 15.13 Occlusions and visibility A point visible in the left image may be hidden in the right image. Such a point has no valid correspondence. This violates the basic matching assumption. Occlusions are common near depth discontinuities. If foreground objects are closer to the cameras than the background, the foreground shifts more between views because it has larger disparity. This can reveal or hide background regions. A stereo method can handle occlusions using: 1. explicit occlusion labels; 2. left-right consistency checks; 3. robust penalties that tolerate unmatched pixels; 4. confidence maps; 5. ordering constraints along scanlines; 6. learned occlusion prediction. Visibility turns stereo from a pure correspondence problem into a scene-geometry problem. ## 15.14 Plane sweep stereo Plane sweep stereo generalizes disparity search using hypothetical depth planes. For each candidate depth $Z_k$ or plane $\Pi_k$, warp the right image into the left image coordinate system and measure photometric agreement. For a fronto-parallel plane at depth $Z$, rectified stereo reduces to a horizontal shift. For a general plane, the warp is a homography: $$ x_R \sim H_\Pi x_L. $$ A cost volume is formed by stacking matching costs over candidate depths: $$ C(x,Z_k) = \rho\left(I_L(x)-I_R(w_k(x))\right), $$ where $w_k$ is the warp induced by the $k$th depth hypothesis. The selected depth is often $$ \hat Z(x)=\arg\min_{Z_k} C(x,Z_k). $$ Plane sweep is especially important for multi-view stereo and differentiable depth estimation. ::: {.interactive-card} **Interactive 15.6. Plane sweep intuition.** A correct depth plane aligns image content; an incorrect plane leaves residual misalignment. <div id="cv-ch15-plane-sweep-demo" class="plotly-demo"></div> ::: ## 15.15 Multi-view stereo Two-view stereo uses one baseline. Multi-view stereo uses more images. If a point is visible in several cameras, its depth can be estimated more reliably. Given cameras $P_1,\dots,P_m$ and image observations $x_1, \dots,x_m$, multi-view triangulation solves $$ \min_X \sum_{i=1}^m \left\|x_i-\pi(P_iX)\right\|_2^2. $$ Dense multi-view stereo estimates a surface or depth map consistent across many images. Methods include: - plane sweep and cost-volume fusion; - patch-based multi-view stereo; - volumetric reconstruction; - depth-map fusion; - neural implicit surfaces and differentiable rendering. The mathematical tradeoff is the same as in two-view stereo: data fidelity, regularization, visibility, and uncertainty. ## 15.16 Statistical model of stereo matching Assume a rectified stereo pair and a true disparity field $d^*$. A simplified observation model is $$ I_L(p)=S(p)+\varepsilon_L(p), $$ $$ I_R(p-(0,d^*(p)))=S(p)+\varepsilon_R(p), $$ where $S(p)$ is an unknown scene radiance pattern and $\varepsilon_L,\varepsilon_R$ are noise terms. If the noise is independent Gaussian with variance $\sigma^2$, then the negative log likelihood of a patch match is proportional to SSD: $$ -\log p(\text{patches}\mid d) \propto \sum_{q\in W(p)} \left(I_L(q)-I_R(q-(0,d))\right)^2. $$ If the noise is Laplace, SAD is more appropriate: $$ -\log p(\text{patches}\mid d) \propto \sum_{q\in W(p)} \left|I_L(q)-I_R(q-(0,d))\right|. $$ Robust losses such as Huber or Charbonnier losses are used when there are outliers: $$ \rho_\epsilon(r)=\sqrt{r^2+\epsilon^2}. $$ This statistical viewpoint helps students understand why different matching costs behave differently. ## 15.17 Learned stereo and cost volumes Modern stereo networks usually build a learned cost volume. Let $$ F_L = \Phi_\theta(I_L), \qquad F_R = \Phi_\theta(I_R) $$ be learned feature maps. A cost volume may be built by correlation: $$ C(p,d)= \left\langle F_L(p), F_R(p-(0,d))\right\rangle, $$ or by concatenation: $$ V(p,d)= \left[F_L(p),F_R(p-(0,d))\right]. $$ A neural network then regularizes and decodes this cost volume: $$ \hat d = G_\theta(C). $$ This is a learned version of classical stereo. The architecture learns the feature representation, the matching cost, and the regularization procedure. ### Soft argmin Many networks convert a cost volume into a probability distribution over disparity: $$ p(d\mid p)= \frac{\exp(-C(p,d))}{\sum_k \exp(-C(p,k))}. $$ The predicted disparity is the expectation $$ \hat d(p)=\sum_d d\,p(d\mid p). $$ This is called **soft argmin**. It is differentiable, unlike the hard argmin. However, if the distribution is multimodal, the expectation may lie between two plausible disparities and may not correspond to a valid match. ::: {.interactive-card} **Interactive 15.7. Soft argmin and ambiguity.** A unimodal cost distribution gives a meaningful expectation. A bimodal distribution can produce a disparity that is not one of the plausible modes. <div id="cv-ch15-softargmin-demo" class="plotly-demo"></div> ::: ## 15.18 Self-supervised stereo and photometric loss If ground-truth depth is unavailable, stereo models can be trained using image reconstruction. Given a predicted disparity $\hat d$, warp the right image to reconstruct the left image: $$ \hat I_L(p)=I_R(p-(0,\hat d(p))). $$ The photometric loss is $$ \mathcal{L}_{\mathrm{photo}} = \sum_p \rho\left(I_L(p)-\hat I_L(p)\right). $$ A smoothness penalty is often added: $$ \mathcal{L}_{\mathrm{smooth}} = \sum_p \left| \nabla \hat d(p) \right|e^{-\alpha \|\nabla I_L(p)\|}. $$ The exponential term allows disparity discontinuities at image edges. The total objective is $$ \mathcal{L} = \mathcal{L}_{\mathrm{photo}} + \lambda\mathcal{L}_{\mathrm{smooth}} + \mu\mathcal{L}_{\mathrm{consistency}}. $$ This connects stereo to differentiable rendering, inverse problems, and self-supervised learning. ### Important caveat Photometric self-supervision assumes that corresponding points have similar appearance. This assumption fails for occlusions, specular reflections, transparent objects, non-Lambertian surfaces, motion, exposure changes, and rolling-shutter effects. Modern systems therefore use robust losses, masks, multi-scale training, left-right consistency, and learned priors. ## 15.19 Monocular depth as learned inverse problem Stereo uses two images and known geometry. Monocular depth estimates depth from a single image: $$ I \mapsto Z. $$ This problem is ill-posed because a single image does not determine metric scale. A small object nearby and a large object far away can have the same image size. A monocular depth network learns statistical regularities from data: $$ \hat Z = f_\theta(I). $$ The model may use cues such as perspective, texture gradients, object size, shading, focus, occlusion, and semantic context. Unlike stereo, monocular depth depends heavily on learned priors. For MA Applied Math and MS Statistics students, the key distinction is: ::: {.cv-card} Stereo depth is primarily geometry constrained by correspondence. Monocular depth is primarily statistical inference constrained by learned priors. ::: ## 15.20 Evaluation metrics For ground-truth depth $Z_i$ and prediction $\hat Z_i$, common metrics include: $$ \operatorname{AbsRel} =\frac{1}{N}\sum_i \frac{|Z_i-\hat Z_i|}{Z_i}, $$ $$ \operatorname{RMSE} =\sqrt{\frac{1}{N}\sum_i(Z_i-\hat Z_i)^2}, $$ and log RMSE $$ \operatorname{RMSE}_{\log} =\sqrt{\frac{1}{N}\sum_i(\log Z_i-\log \hat Z_i)^2}. $$ For disparity, $$ \operatorname{EPE} =\frac{1}{N}\sum_i |d_i-\hat d_i| $$ is the endpoint error in one dimension. Another common metric is the percentage of pixels with disparity error larger than a threshold: $$ \frac{1}{N}\sum_i \mathbf{1}\{|d_i-\hat d_i|>\tau\}. $$ Metrics should be interpreted with care. A method can have good average error while failing at thin structures, depth boundaries, reflective surfaces, or far regions. ## 15.21 Python example: disparity to depth The following example converts a synthetic disparity map into depth and visualizes uncertainty. It is intentionally small enough for independent study. ```{python} #| label: py-ch15-disparity-depth #| eval: false import numpy as np import matplotlib.pyplot as plt H, W = 120, 180 u = np.linspace(-1, 1, W) v = np.linspace(-1, 1, H) U, V = np.meshgrid(u, v) # Synthetic disparity: foreground bump plus slanted background D = 8 + 4 * (U + 1) + 12 * np.exp(-8 * (U**2 + V**2)) f = 700.0 # pixels b = 0.12 # meters Z = f * b / D sigma_d = 0.25 sigma_Z = (f * b / D**2) * sigma_d fig, ax = plt.subplots(1, 3, figsize=(12, 3.5)) im0 = ax[0].imshow(D, cmap="magma") ax[0].set_title("disparity") plt.colorbar(im0, ax=ax[0], fraction=0.046) im1 = ax[1].imshow(Z, cmap="viridis") ax[1].set_title("depth Z") plt.colorbar(im1, ax=ax[1], fraction=0.046) im2 = ax[2].imshow(sigma_Z, cmap="inferno") ax[2].set_title("depth uncertainty") plt.colorbar(im2, ax=ax[2], fraction=0.046) for a in ax: a.axis("off") plt.tight_layout() plt.show() ``` ## 15.22 Python example: winner-take-all block matching This example implements a simple block-matching stereo baseline. It is not meant to be state of the art; it is meant to expose the mathematical structure. ```{python} #| label: py-ch15-block-matching #| eval: false import numpy as np import matplotlib.pyplot as plt from scipy.ndimage import uniform_filter rng = np.random.default_rng(15) H, W = 96, 160 x = np.linspace(0, 1, W) y = np.linspace(0, 1, H) X, Y = np.meshgrid(x, y) left = ( 0.5 + 0.25 * np.sin(18 * X) + 0.20 * np.cos(15 * Y) + 0.25 * np.exp(-60 * ((X - 0.45)**2 + (Y - 0.55)**2)) ) left += 0.03 * rng.normal(size=(H, W)) left = np.clip(left, 0, 1) true_d = 6 + 10 * np.exp(-40 * ((X - 0.45)**2 + (Y - 0.55)**2)) true_d_int = np.rint(true_d).astype(int) right = np.zeros_like(left) for i in range(H): for j in range(W): jr = j - true_d_int[i, j] if 0 <= jr < W: right[i, jr] = left[i, j] right += 0.03 * rng.normal(size=(H, W)) right = np.clip(right, 0, 1) Dmax = 24 window = 7 cost_volume = np.empty((H, W, Dmax + 1)) for d in range(Dmax + 1): shifted = np.zeros_like(right) shifted[:, d:] = right[:, :W - d] sq = (left - shifted) ** 2 cost_volume[:, :, d] = uniform_filter(sq, size=window) est_d = np.argmin(cost_volume, axis=2) plt.figure(figsize=(10, 3.2)) plt.subplot(1, 3, 1) plt.imshow(left, cmap="gray") plt.title("left image") plt.axis("off") plt.subplot(1, 3, 2) plt.imshow(true_d, cmap="magma") plt.title("true disparity") plt.axis("off") plt.subplot(1, 3, 3) plt.imshow(est_d, cmap="magma", vmin=0, vmax=Dmax) plt.title("estimated disparity") plt.axis("off") plt.tight_layout() plt.show() ``` ## 15.23 Python example: soft argmin ```{python} #| label: py-ch15-soft-argmin #| eval: false import numpy as np D = np.arange(0, 32) true_d = 12.4 cost = (D - true_d)**2 / 8 prob = np.exp(-cost) prob = prob / prob.sum() soft = np.sum(D * prob) hard = D[np.argmin(cost)] print("hard argmin:", hard) print("soft argmin:", soft) ``` For a unimodal cost curve, the soft argmin can give a useful subpixel estimate. For a bimodal curve, inspect the distribution rather than trusting the expectation blindly. ## 15.24 TensorFlow example: differentiable photometric stereo loss This example sketches the structure of a self-supervised stereo loss. It uses one-dimensional horizontal warping by interpolation. ```{python} #| label: tf-ch15-photometric-loss #| eval: false import tensorflow as tf def horizontal_warp(right, disparity): """Warp right image toward the left image using horizontal disparity. right: tensor of shape (B, H, W, C) disparity: tensor of shape (B, H, W, 1), measured in pixels """ B = tf.shape(right)[0] H = tf.shape(right)[1] W = tf.shape(right)[2] x = tf.cast(tf.range(W), tf.float32) x = tf.reshape(x, (1, 1, W, 1)) x = tf.tile(x, (B, H, 1, 1)) # For left reconstruction, sample right at x - disparity. x_src = x - disparity x0 = tf.floor(x_src) x1 = x0 + 1.0 w1 = x_src - x0 w0 = 1.0 - w1 x0i = tf.cast(tf.clip_by_value(x0, 0.0, tf.cast(W - 1, tf.float32)), tf.int32) x1i = tf.cast(tf.clip_by_value(x1, 0.0, tf.cast(W - 1, tf.float32)), tf.int32) batch = tf.reshape(tf.range(B), (B, 1, 1, 1)) batch = tf.tile(batch, (1, H, W, 1)) yy = tf.reshape(tf.range(H), (1, H, 1, 1)) yy = tf.tile(yy, (B, 1, W, 1)) idx0 = tf.concat([batch, yy, x0i], axis=-1) idx1 = tf.concat([batch, yy, x1i], axis=-1) val0 = tf.gather_nd(right, idx0) val1 = tf.gather_nd(right, idx1) return w0 * val0 + w1 * val1 def charbonnier(x, eps=1e-3): return tf.sqrt(tf.square(x) + eps**2) def photometric_loss(left, right, disparity): recon = horizontal_warp(right, disparity) return tf.reduce_mean(charbonnier(left - recon)) ``` ## 15.25 AI-assisted study components ### AI component 1: derive and audit Ask an AI assistant: > Derive the stereo formula $Z=fb/d$ from two rectified pinhole cameras. Explain all coordinate assumptions. Then audit the answer. It should explicitly mention: 1. two cameras have parallel optical axes after rectification; 2. the baseline is known; 3. image coordinates are in pixels or normalized units consistently; 4. the disparity convention is fixed; 5. the formula becomes unstable as $d\to 0$. ### AI component 2: compare models Ask an AI assistant to compare SSD, NCC, Census, and learned-feature matching. Require it to discuss invariance to brightness changes, robustness to noise, computational cost, and failure modes. ### AI component 3: debug a stereo pipeline Give an AI assistant a stereo pipeline with poor depth results and ask it to generate possible causes. A good diagnostic list should include calibration error, rectification error, wrong baseline units, wrong disparity sign, insufficient texture, specular surfaces, occlusions, and incorrect depth conversion. ### AI component 4: design a differentiable stereo loss Ask for a loss function combining photometric reconstruction, smoothness, left-right consistency, and occlusion masks. Then identify which parts are geometry-based and which parts are learned priors. ## 15.26 Summary Stereo vision turns two images into depth by combining geometry and matching. Rectification reduces correspondence search to horizontal scanlines. Disparity is inversely proportional to depth: $$ Z=\frac{fb}{d}. $$ The main practical challenge is not the depth formula but the correspondence problem. Local matching creates a cost volume, global methods regularize the disparity field, and modern neural methods learn features, costs, and regularizers. Statistical thinking clarifies noise models, uncertainty propagation, confidence, and failure modes. ## Exercises ### Conceptual exercises 1. Explain why a single image point determines a ray, while a stereo correspondence determines a 3D point. 2. Why does rectification make stereo matching easier? 3. What happens to depth uncertainty as disparity approaches zero? 4. Why do repeated textures create ambiguous disparity estimates? 5. Explain why occlusions violate the basic patch-matching assumption. 6. Compare stereo depth and monocular depth in terms of geometry and statistical priors. ### Mathematical exercises 1. Derive $Z=fb/d$ from the rectified stereo camera model. 2. Show that disparity is proportional to inverse depth. 3. Let $Z=fb/d$. Derive the first-order uncertainty formula $$ \sigma_Z\approx \frac{Z^2}{fb}\sigma_d. $$ 4. Derive the least-squares system for the closest point between two rays. 5. Write the dynamic-programming recurrence for scanline stereo with $$ V(d,d')=|d-d'|. $$ 6. Derive the subpixel quadratic interpolation formula for the disparity correction $\Delta$. 7. Suppose $C(p,d)$ is a negative log likelihood and $V$ is a negative log prior. Show that stereo energy minimization is MAP estimation. 8. Explain why soft argmin can be problematic for a bimodal probability distribution. ### Python exercises 1. Implement disparity-to-depth conversion and plot depth uncertainty for several values of $f$, $b$, and $\sigma_d$. 2. Implement winner-take-all block matching using SSD. 3. Replace SSD with SAD and NCC. Compare results on synthetic images. 4. Add a left-right consistency check to a block-matching pipeline. 5. Implement subpixel disparity refinement using a quadratic fit. 6. Build a small cost volume and visualize $C(p,d)$ for easy, ambiguous, and occluded pixels. 7. Implement soft argmin and test it on unimodal and bimodal cost distributions. 8. Use OpenCV stereo functions on a rectified stereo pair and compare with your own simple baseline. ### AI/modeling exercises 1. Ask an AI assistant to generate a block-matching stereo implementation. Check whether the shift direction and disparity sign are correct. 2. Ask an AI assistant to explain left-right consistency. Test whether it correctly handles occlusions. 3. Ask for a learned stereo architecture using a cost volume. Identify where classical stereo ideas appear inside the network. 4. Use an AI assistant to propose robustness improvements for reflective or transparent surfaces. 5. Design a self-supervised stereo loss and identify all assumptions behind its photometric term. 6. Ask an AI assistant to explain why monocular depth is scale ambiguous. Check whether the answer distinguishes projective ambiguity from statistical priors. ## Notes and further reading The mathematical geometry of stereo follows the projective-camera framework in Hartley and Zisserman [@hartley2004multiple], Faugeras [@faugeras1993three], and Ma et al. [@ma2004invitation]. Scharstein and Szeliski's taxonomy [@scharstein2002taxonomy] is a classic reference for dense two-frame stereo methods. Semi-global matching was introduced by Hirschmuller [@hirschmuller2008stereo]. For modern learned stereo, see GC-Net [@kendall2017end], PSMNet [@chang2018pyramid], and related cost-volume methods. For self-supervised monocular and stereo depth learning, Eigen et al. [@eigen2014depth] and Godard et al. [@godard2017unsupervised] are useful starting points. <script> function cvCh15Range(start, stop, n) { const out = []; for (let i = 0; i < n; i++) out.push(start + (stop - start) * i / (n - 1)); return out; } function cvRenderCh15DisparityDepthDemo() { const el = document.getElementById('cv-ch15-disparity-depth-demo'); if (!el || typeof Plotly === 'undefined') return; const f = 700; const b = 0.12; const d = cvCh15Range(1, 64, 180); const Z = d.map(x => f * b / x); const invZ = d.map(x => x / (f * b)); Plotly.newPlot(el, [ {x: d, y: Z, mode: 'lines', name: 'depth Z = fb/d'}, {x: d, y: invZ, mode: 'lines', name: 'inverse depth d/(fb)', yaxis: 'y2'} ], { title: 'Stereo depth is inverse to disparity', margin: {t: 55, r: 65, b: 50, l: 65}, xaxis: {title: 'disparity d in pixels'}, yaxis: {title: 'depth Z in meters'}, yaxis2: {title: 'inverse depth', overlaying: 'y', side: 'right'}, legend: {orientation: 'h', y: -0.22} }, {responsive: true, displaylogo: false}); } function cvRenderCh15TriangulationAngleDemo() { const el = document.getElementById('cv-ch15-triangulation-angle-demo'); if (!el || typeof Plotly === 'undefined') return; const baselines = [0.2, 0.6, 1.0, 1.6]; const frames = []; const steps = []; const X = [0.25, 0, 4.5]; baselines.forEach((b, k) => { const C1 = [-b/2, 0, 0]; const C2 = [ b/2, 0, 0]; const trace = [ {x: [C1[0], X[0]], y: [C1[1], X[1]], z: [C1[2], X[2]], mode: 'lines+markers', type: 'scatter3d', name: 'ray from left'}, {x: [C2[0], X[0]], y: [C2[1], X[1]], z: [C2[2], X[2]], mode: 'lines+markers', type: 'scatter3d', name: 'ray from right'}, {x: [X[0]], y: [X[1]], z: [X[2]], mode: 'markers+text', type: 'scatter3d', text: ['3D point'], textposition: 'top center', name: 'point'} ]; frames.push({name: String(k), data: trace}); const angle = 2 * Math.atan((b/2) / X[2]) * 180 / Math.PI; steps.push({label: 'baseline ' + b + ', angle ' + angle.toFixed(1) + '°', method: 'animate', args: [[String(k)], {mode: 'immediate', frame: {duration: 300, redraw: true}, transition: {duration: 150}}]}); }); Plotly.newPlot(el, frames[0].data, { title: 'Triangulation improves when rays intersect at a wider angle', margin: {t: 55, r: 20, b: 70, l: 20}, scene: {xaxis: {title: 'X'}, yaxis: {title: 'Y'}, zaxis: {title: 'Z'}, camera: {eye: {x: 1.6, y: 1.1, z: 0.8}}}, sliders: [{active: 0, steps: steps}], legend: {orientation: 'h', y: -0.12} }, {responsive: true, displaylogo: false}).then(() => Plotly.addFrames(el, frames)); } function cvRenderCh15MatchingCostDemo() { const el = document.getElementById('cv-ch15-matching-cost-demo'); if (!el || typeof Plotly === 'undefined') return; const D = cvCh15Range(0, 40, 81); const sharp = D.map(d => 0.03 * Math.pow(d - 18, 2) + 0.2); const flat = D.map(d => 1.6 + 0.003 * Math.pow(d - 18, 2) + 0.05 * Math.sin(d)); const repeated = D.map(d => 0.35 + 0.018 * Math.pow(Math.sin((d - 5) * Math.PI / 9), 2) + 0.004 * Math.pow(d - 18, 2)); Plotly.newPlot(el, [ {x: D, y: sharp, mode: 'lines', name: 'textured patch: sharp minimum'}, {x: D, y: flat, mode: 'lines', name: 'textureless patch: flat cost'}, {x: D, y: repeated, mode: 'lines', name: 'repeated texture: multiple minima'} ], { title: 'Matching ambiguity appears in the shape of the cost curve', margin: {t: 55, r: 20, b: 50, l: 60}, xaxis: {title: 'candidate disparity'}, yaxis: {title: 'matching cost'}, legend: {orientation: 'h', y: -0.22} }, {responsive: true, displaylogo: false}); } function cvRenderCh15SmoothnessDemo() { const el = document.getElementById('cv-ch15-smoothness-demo'); if (!el || typeof Plotly === 'undefined') return; const xs = cvCh15Range(0, 1, 120); const trueD = xs.map(x => x < 0.48 ? 8 + 2*x : 20 - 2*x); const noisy = trueD.map((y, i) => y + 1.8 * Math.sin(73 * xs[i]) + 0.8 * Math.sin(19 * xs[i])); function smooth(alpha) { let out = noisy.slice(); for (let it = 0; it < alpha; it++) { const tmp = out.slice(); for (let i = 1; i < out.length - 1; i++) tmp[i] = 0.25*out[i-1] + 0.5*out[i] + 0.25*out[i+1]; out = tmp; } return out; } Plotly.newPlot(el, [ {x: xs, y: trueD, mode: 'lines', name: 'true piecewise-smooth disparity'}, {x: xs, y: noisy, mode: 'lines', name: 'winner-take-all noisy estimate'}, {x: xs, y: smooth(4), mode: 'lines', name: 'moderate smoothing'}, {x: xs, y: smooth(25), mode: 'lines', name: 'too much smoothing'} ], { title: 'Smoothness regularization reduces noise but can blur boundaries', margin: {t: 55, r: 20, b: 50, l: 60}, xaxis: {title: 'scanline position'}, yaxis: {title: 'disparity'}, legend: {orientation: 'h', y: -0.25} }, {responsive: true, displaylogo: false}); } function cvRenderCh15UncertaintyDemo() { const el = document.getElementById('cv-ch15-uncertainty-demo'); if (!el || typeof Plotly === 'undefined') return; const f = 700, b = 0.12; const Z = cvCh15Range(0.5, 30, 160); const s1 = Z.map(z => z*z/(f*b)*0.10); const s2 = Z.map(z => z*z/(f*b)*0.25); const s3 = Z.map(z => z*z/(f*b)*0.50); Plotly.newPlot(el, [ {x: Z, y: s1, mode: 'lines', name: 'σd = 0.10 px'}, {x: Z, y: s2, mode: 'lines', name: 'σd = 0.25 px'}, {x: Z, y: s3, mode: 'lines', name: 'σd = 0.50 px'} ], { title: 'Depth uncertainty grows quadratically with distance', margin: {t: 55, r: 20, b: 50, l: 65}, xaxis: {title: 'true depth Z'}, yaxis: {title: 'approximate σZ'}, legend: {orientation: 'h', y: -0.2} }, {responsive: true, displaylogo: false}); } function cvRenderCh15PlaneSweepDemo() { const el = document.getElementById('cv-ch15-plane-sweep-demo'); if (!el || typeof Plotly === 'undefined') return; const x = cvCh15Range(0, 1, 160); const left = x.map(t => 0.5 + 0.3*Math.sin(18*t) + 0.18*Math.exp(-120*Math.pow(t - 0.55, 2))); function shifted(s) { return x.map(t => 0.5 + 0.3*Math.sin(18*(t-s)) + 0.18*Math.exp(-120*Math.pow(t-s - 0.55, 2))); } const good = shifted(0.08); const bad = shifted(0.18); const residualGood = left.map((y, i) => Math.abs(y - good[i])); const residualBad = left.map((y, i) => Math.abs(y - bad[i])); Plotly.newPlot(el, [ {x: x, y: left, mode: 'lines', name: 'left scanline'}, {x: x, y: good, mode: 'lines', name: 'right warped with near-correct depth'}, {x: x, y: bad, mode: 'lines', name: 'right warped with wrong depth'}, {x: x, y: residualGood, mode: 'lines', name: 'small residual', yaxis: 'y2'}, {x: x, y: residualBad, mode: 'lines', name: 'large residual', yaxis: 'y2'} ], { title: 'Plane sweep: correct depth aligns image content', margin: {t: 55, r: 60, b: 60, l: 60}, xaxis: {title: 'scanline coordinate'}, yaxis: {title: 'intensity'}, yaxis2: {title: 'absolute residual', overlaying: 'y', side: 'right'}, legend: {orientation: 'h', y: -0.28} }, {responsive: true, displaylogo: false}); } function cvRenderCh15SoftargminDemo() { const el = document.getElementById('cv-ch15-softargmin-demo'); if (!el || typeof Plotly === 'undefined') return; const d = cvCh15Range(0, 40, 81); function prob(cost) { const ex = cost.map(c => Math.exp(-c)); const s = ex.reduce((a,b)=>a+b, 0); return ex.map(v => v/s); } const costUni = d.map(x => 0.06*Math.pow(x - 16.5, 2)); const costBi = d.map(x => Math.min(0.08*Math.pow(x - 10, 2), 0.08*Math.pow(x - 29, 2)) + 0.15); const pUni = prob(costUni); const pBi = prob(costBi); const softUni = d.reduce((a, x, i) => a + x*pUni[i], 0); const softBi = d.reduce((a, x, i) => a + x*pBi[i], 0); Plotly.newPlot(el, [ {x: d, y: pUni, mode: 'lines', name: 'unimodal posterior'}, {x: d, y: pBi, mode: 'lines', name: 'bimodal posterior'}, {x: [softUni, softUni], y: [0, Math.max(...pUni)], mode: 'lines', name: 'soft argmin: unimodal'}, {x: [softBi, softBi], y: [0, Math.max(...pBi)], mode: 'lines', name: 'soft argmin: bimodal'} ], { title: 'Soft argmin may average between two plausible disparities', margin: {t: 55, r: 20, b: 60, l: 60}, xaxis: {title: 'disparity'}, yaxis: {title: 'probability'}, legend: {orientation: 'h', y: -0.28} }, {responsive: true, displaylogo: false}); } document.addEventListener('DOMContentLoaded', () => { cvRenderCh15DisparityDepthDemo(); cvRenderCh15TriangulationAngleDemo(); cvRenderCh15MatchingCostDemo(); cvRenderCh15SmoothnessDemo(); cvRenderCh15UncertaintyDemo(); cvRenderCh15PlaneSweepDemo(); cvRenderCh15SoftargminDemo(); }); </script>