31 Vision in the Age of Multimodal AI

Chapter 30 studies computer vision in the age of multimodal AI. Earlier chapters treated images, geometry, probability, optimization, CNNs, transformers, segmentation foundation models, and diffusion models. This final chapter puts these ideas together. Modern vision systems increasingly process not only images, but also text, video, audio, depth, 3D scenes, actions, sensor streams, and scientific measurements.

A classical computer vision pipeline often had a narrowly defined input and output:

\[ \text{image} \longmapsto \text{edges, keypoints, objects, masks, or depth}. \]

A modern multimodal vision system is better described as a family of maps among several spaces:

\[ \mathcal X_{\text{image}},\quad \mathcal X_{\text{video}},\quad \mathcal X_{\text{text}},\quad \mathcal X_{\text{3D}},\quad \mathcal X_{\text{action}},\quad \mathcal Y_{\text{task}}. \]

For example, a single model may support image classification, image-text retrieval, visual question answering, segmentation, video understanding, captioning, robotic planning, and image generation. The mathematical question is not only whether a model is accurate on one benchmark. The deeper question is:

\[ \text{How can visual information be represented so that it transfers across tasks, modalities, domains, and decisions?} \]

This chapter is deliberately synthetic. It does not introduce one new algorithm. Instead, it gives a mathematical framework for understanding the modern landscape of multimodal computer vision.

31.1 Learning Goals

After reading this chapter, students should be able to:

define multimodal vision as learning maps among image, text, video, 3D, and action spaces;
explain early fusion, late fusion, cross-attention, and shared-embedding fusion mathematically;
formulate image-text, video-text, and vision-action models as conditional probabilistic models;
connect foundation models to representation learning, transfer learning, and task conditioning;
describe the mathematical differences among image, video, 3D, and embodied vision;
formulate visual question answering, captioning, retrieval, grounding, and instruction-following as structured prediction problems;
analyze evaluation metrics for multimodal vision, including calibration, robustness, retrieval, hallucination, and distribution shift;
explain why medical, scientific, and robotic vision require domain-specific uncertainty and validation;
design small experiments to test modality alignment, cross-attention, zero-shot transfer, and robustness;
identify open mathematical questions in multimodal vision.

31.2 30.1 From Image Models to Multimodal Systems

In the first chapters, an image was modeled as a function or tensor:

\[ I:\Omega\subset \mathbb R^2\to \mathbb R^C, \qquad X\in \mathbb R^{H\times W\times C}. \]

Classical computer vision transformed this image into a structured output:

\[ f:X\mapsto Y. \]

The output $Y$ might be a label, mask, depth map, 3D reconstruction, optical flow field, or set of bounding boxes. In multimodal AI, the input itself may have several components:

\[ Z=(X,T,V,D,A,S), \]

where:

$X$ is an image;
$T$ is text;
$V$ is video;
$D$ is depth, LiDAR, or another spatial sensor;
$A$ is action, control, or user instruction;
$S$ is scientific or clinical metadata.

A general multimodal vision model is a map

\[ F_\theta: \mathcal Z \to \mathcal Y, \]

where $\mathcal Z$ is a product or union of modality spaces. The model may also be probabilistic:

\[ p_\theta(y\mid z). \]

The conditional formulation is especially important. It covers classification, captioning, segmentation, retrieval, question answering, detection, and generation.

A useful mathematical abstraction is that multimodal vision learns a family of conditional maps

\[ F_\theta(\cdot\mid c), \]

where $c$ is a condition such as a text prompt, point prompt, task description, previous frame, robot state, or scientific context.

31.3 30.2 Modalities as Mathematical Spaces

Different modalities have different mathematical structures.

31.3.1 Images

An image is naturally spatial:

\[ X\in\mathbb R^{H\times W\times C}. \]

It has local neighborhoods, translations, edges, textures, and approximate scale structure.

31.3.2 Text

Text is naturally sequential:

\[ T=(w_1,\dots,w_L), \]

where $w_i$ are tokens. After embedding,

\[ E_T(T)\in\mathbb R^{L\times d}. \]

Text has syntax, semantics, ambiguity, and compositional structure.

31.3.3 Video

A video is spatio-temporal:

\[ V\in\mathbb R^{T\times H\times W\times C}. \]

It has motion, temporal causality, occlusion, events, and long-range dependency.

31.3.4 3D Data

A 3D scene may be represented as:

a point cloud $P=\{p_i\}_{i=1}^n\subset\mathbb R^3$;
a mesh $(\mathcal V,\mathcal E,\mathcal F)$;
an implicit surface $\{x:f_\theta(x)=0\}$;
a voxel grid $X\in\mathbb R^{N_x\times N_y\times N_z\times C}$;
a radiance field $F_\theta(x,d)=(\sigma,c)$.

The central mathematical structure is geometry in $\mathbb R^3$.

31.3.5 Action and Control

In robotics and embodied AI, the output may be an action:

\[ a_t\in\mathcal A. \]

A perception-action system is often modeled as a policy:

\[ \pi_\theta(a_t\mid o_{1:t},g), \]

where $o_{1:t}$ denotes observations and $g$ denotes a goal or instruction.

31.4 30.3 Encoders, Decoders, and Fusion

Most modern multimodal systems can be decomposed into encoders, fusion modules, and decoders.

Suppose we have an image $x$ and a text input $t$. Let

\[ z_x=f_\theta(x),\qquad z_t=g_\phi(t). \]

The fusion module combines $z_x$ and $z_t$:

\[ h=\mathcal F_\psi(z_x,z_t). \]

A task decoder then produces the output:

\[ \widehat y=d_\omega(h). \]

This modular view is useful because different architectures mainly differ in how they define $f_\theta$, $g_\phi$, $\mathcal F_\psi$, and $d_\omega$.

31.4.1 Early Fusion

Early fusion combines modalities before deep processing:

\[ h_0=[e_x;e_t]. \]

A single model then processes the concatenated representation. Early fusion can model strong interactions but may be computationally expensive and less modular.

31.4.2 Late Fusion

Late fusion processes modalities separately and combines decisions:

\[ s = \alpha s_x + (1-\alpha)s_t. \]

This is simple and robust but may miss fine-grained alignment.

31.4.3 Shared Embedding Fusion

A CLIP-style model maps images and text into a shared embedding space:

\[ a=\frac{f_\theta(x)}{\|f_\theta(x)\|}, \qquad b=\frac{g_\phi(t)}{\|g_\phi(t)\|}. \]

Similarity is measured by

\[ s(x,t)=a^Tb. \]

This supports retrieval and zero-shot classification.

31.4.4 Cross-Attention Fusion

Cross-attention uses one modality as queries and another as keys and values:

\[ \operatorname{CrossAttn}(Q_x,K_t,V_t) = \operatorname{softmax}\left(\frac{Q_xK_t^T}{\sqrt d}\right)V_t. \]

The result is a visual representation conditioned on text, or a textual representation conditioned on visual evidence.

Note

For MA Applied Mathematics and MS Statistics students, the key point is that fusion is not just an engineering choice. It changes the conditional independence assumptions, computational complexity, sample complexity, and failure modes of the model.

31.5 30.4 Interactive Demo: Fusion Strategies

The following demo compares three schematic fusion strategies. The point is not to simulate a real network, but to visualize how information can be combined.

Interactive: Early, late, and shared-embedding fusion

Move the slider to change the agreement between visual and text evidence.

Modality agreement: 0.70

31.6 30.5 Alignment as a Statistical Problem

Multimodal learning often requires alignment. For an image $x_i$ and its paired text $t_i$, a dual encoder produces normalized embeddings

\[ a_i=f_\theta(x_i),\qquad b_i=g_\phi(t_i), \qquad \|a_i\|=\|b_i\|=1. \]

The similarity matrix is

\[ S_{ij}=\frac{a_i^Tb_j}{\tau}, \]

where $\tau>0$ is the temperature. A common symmetric contrastive loss is

\[ \mathcal L = \frac{1}{2n}\sum_{i=1}^n \left[ -\log\frac{\exp(S_{ii})}{\sum_{j=1}^n \exp(S_{ij})} - \log\frac{\exp(S_{ii})}{\sum_{j=1}^n \exp(S_{ji})} \right]. \]

This loss asks each image to identify its paired text and each text to identify its paired image.

From a statistical point of view, the paired dataset is a sample from a joint distribution:

\[ (X,T)\sim P_{XT}. \]

The goal is to learn representations that preserve information about dependence between $X$ and $T$ while discarding nuisance variation. This is not the same as learning all visual detail or all language detail.

31.7 30.6 Retrieval, Grounding, and Open-Vocabulary Prediction

A shared embedding space supports retrieval. Given an image query $x$, retrieve text items by ranking

\[ t_j \mapsto s(x,t_j)=f_\theta(x)^Tg_\phi(t_j). \]

Given a text query $t$, retrieve images by ranking

\[ x_i \mapsto s(x_i,t). \]

Common retrieval metrics include:

\[ \operatorname{Recall@K}=\frac{1}{n}\sum_{i=1}^n \mathbf 1\{\text{correct item appears in top }K\}, \]

and mean reciprocal rank:

\[ \operatorname{MRR} =\frac{1}{n}\sum_{i=1}^n \frac{1}{r_i}, \]

where $r_i$ is the rank of the correct item.

Open-vocabulary classification can be formulated by constructing text prototypes $t_1,\dots,t_K$ and choosing

\[ \widehat y(x)=\arg\max_k f_\theta(x)^T g_\phi(t_k). \]

Open-vocabulary detection and segmentation extend this idea from whole images to regions or pixels.

31.8 30.7 Visual Question Answering and Captioning

Visual question answering is a conditional prediction problem:

\[ \widehat a=\arg\max_a p_\theta(a\mid x,q), \]

where $x$ is an image and $q$ is a question. Image captioning is conditional language modeling:

\[ p_\theta(w_{1:L}\mid x) =\prod_{\ell=1}^L p_\theta(w_\ell\mid w_{<\ell},x). \]

The output space is no longer a small finite label set. It is a structured language space. This creates several mathematical and statistical issues:

many answers can be semantically equivalent;
a generated caption can be fluent but visually unsupported;
likelihood does not necessarily measure factual grounding;
evaluation depends on reference captions, human judgments, or external models;
the model may exploit dataset priors rather than image evidence.

A useful decomposition is

\[ p_\theta(a\mid x,q) = \int p_\theta(a\mid h,q)p_\theta(h\mid x,q)\,dh, \]

where $h$ is an internal multimodal representation. If $h$ ignores the image, the system becomes a language prior rather than a vision-language model.

31.9 30.8 Interactive Demo: Cross-Attention as Visual Grounding

Cross-attention can be interpreted as a soft alignment between words and visual tokens. The following demo shows a schematic attention distribution from one text token to image patches.

Interactive: Cross-attention from a word to image patches

Move the slider to change attention sharpness. Sharper attention is easier to interpret but can become brittle.

Attention sharpness: 4.00

31.10 30.9 Video Understanding

Images are spatial. Video is spatio-temporal. A video can be modeled as

\[ V:\Omega\times \{1,\dots,T\}\to \mathbb R^C. \]

Discrete video data are tensors

\[ V\in\mathbb R^{T\times H\times W\times C}. \]

Video tasks include:

action recognition;
temporal localization;
event detection;
tracking;
video captioning;
video question answering;
prediction of future frames or states.

A simple video classifier has the form

\[ \widehat y=f_\theta(V_{1:T}). \]

A temporal localization model predicts intervals:

\[ \{([s_j,e_j],c_j)\}_{j=1}^m. \]

Video introduces several new mathematical issues.

31.10.1 Temporal Dependence

Frames are not independent. A common probabilistic factorization is

\[ p(V_{1:T})=\prod_{t=1}^T p(V_t\mid V_{<t}). \]

For supervised video prediction:

\[ p(y\mid V_{1:T}). \]

31.10.2 Motion and Equivariance

Motion creates transformations over time:

\[ V_{t+1}(x)\approx V_t(x-u_t(x)), \]

where $u_t$ is optical flow. A good video representation should capture both appearance and motion.

31.10.3 Long-Range Attention

If a video has $N=THW$ tokens, full self-attention costs

\[ O(N^2d)=O(T^2H^2W^2d). \]

This motivates factorized space-time attention, temporal pooling, sparse attention, memory modules, and token compression.

31.11 30.10 3D Vision and Scene Representations

Chapter 12 through Chapter 17 studied projective geometry and 3D reconstruction. Modern multimodal vision adds learned 3D representations and language-conditioned 3D reasoning.

31.11.1 Point Clouds

A point cloud is a finite set

\[ P=\{p_i\in\mathbb R^3\}_{i=1}^n. \]

Since it is a set, a point-cloud representation should be permutation invariant or permutation equivariant:

\[ f(\{p_1,\dots,p_n\})=f(\{p_{\sigma(1)},\dots,p_{\sigma(n)}\}). \]

A common form is

\[ f(P)=\rho\left(\sum_{i=1}^n \phi(p_i)\right), \]

where $\phi$ processes points and $\rho$ aggregates them.

31.11.2 Implicit Surfaces

An implicit surface is represented as a level set

\[ \mathcal S=\{x\in\mathbb R^3:f_\theta(x)=0\}. \]

Signed distance functions satisfy approximately

\[ \|\nabla f_\theta(x)\|=1. \]

31.11.3 Neural Radiance Fields

A radiance field maps position and viewing direction to density and color:

\[ F_\theta(x,d)=(\sigma(x),c(x,d)). \]

An image pixel is generated by volume rendering along a camera ray $r(s)=o+sd$:

\[ C(r)=\int_{s_n}^{s_f} T(s)\sigma(r(s))c(r(s),d)\,ds, \]

where

\[ T(s)=\exp\left(-\int_{s_n}^s \sigma(r(u))\,du\right) \]

is transmittance. This connects 3D geometry, differentiable rendering, and inverse problems.

31.11.4 Gaussian Scene Representations

A modern explicit alternative represents a scene by many 3D primitives:

\[ \{(\mu_i,\Sigma_i,c_i,\alpha_i)\}_{i=1}^n, \]

where each primitive has position, covariance, color, and opacity. Rendering becomes differentiable projection and alpha compositing.

31.12 30.11 Interactive Demo: Volume Rendering Along a Ray

The following toy visualization shows how density along a ray contributes to a rendered pixel. High density early in the ray blocks later color by reducing transmittance.

Interactive: Volume-rendering weights along a ray

Adjust density strength. The rendered color is a weighted integral along the ray.

Density strength: 2.00

31.13 30.12 Robotics and Embodied Vision

In robotics, vision is not only recognition. Vision supports action. The system observes the world, chooses actions, and receives feedback.

A partially observed decision process can be written as

\[ s_t\sim \text{latent state},\qquad o_t\sim p(o_t\mid s_t),\qquad a_t\sim \pi_\theta(a_t\mid o_{1:t},g). \]

The goal may be to maximize expected return:

\[ J(\theta)=\mathbb E_{\pi_\theta}\left[\sum_{t=0}^{T}\gamma^t r(s_t,a_t)\right]. \]

Vision contributes to estimating the state, recognizing objects, localizing the robot, predicting affordances, and grounding language instructions.

31.13.1 Vision-Language-Action Models

A language-conditioned robotic policy may have the form

\[ \pi_\theta(a_t\mid I_t, q, h_t), \]

where $I_t$ is an image, $q$ is an instruction, and $h_t$ is a history representation. A more structured model separates perception, planning, and control:

\[ I_t \mapsto z_t, \qquad (z_t,q)\mapsto \text{plan}, \qquad \text{plan}\mapsto a_t. \]

The end-to-end formulation is flexible, but the modular formulation can be easier to validate and debug.

31.13.2 Uncertainty in Embodied Vision

Errors in embodied systems have consequences. If $b_t(s)$ is a belief distribution over states, then action selection should depend on uncertainty:

\[ a_t = \arg\max_a \mathbb E_{s\sim b_t}\left[Q(s,a)\right]. \]

This connects robotics to Bayesian filtering, decision theory, and risk-sensitive control.

31.14 30.13 Medical and Scientific Vision

Medical and scientific vision are especially important for applied mathematics and statistics students. They combine high-dimensional imaging with domain-specific physical and biological constraints.

Examples include:

CT, MRI, ultrasound, pathology, dermatology, and ophthalmology;
microscopy, astronomy, climate imagery, remote sensing, and materials science;
segmentation, registration, detection, reconstruction, denoising, and measurement.

A medical imaging problem often has the form

\[ y=\mathcal A(x)+\epsilon, \]

where $x$ is the latent anatomical or physiological object, $\mathcal A$ is an imaging operator, and $y$ is observed data. Deep learning can enter as a prior, a reconstruction map, a segmentation map, or a decision support model.

31.14.1 Why Medical and Scientific Vision Are Different

Several issues become central:

ground truth may be noisy or subjective;
label prevalence may be low;
images may come from different scanners, protocols, or institutions;
small errors may have high cost;
uncertainty and calibration matter more than benchmark accuracy;
external validation is essential;
interpretability may be legally, ethically, or scientifically required.

For a diagnostic classifier that outputs $\widehat p(x)$, calibration means

\[ \mathbb P(Y=1\mid \widehat p(X)=p)\approx p. \]

For segmentation, uncertainty may be represented by a distribution over masks:

\[ p_\theta(M\mid X). \]

In scientific vision, the output should often be a measurement with uncertainty, not only a label.

31.15 30.14 Multimodal Foundation Models

A foundation model is trained on broad data and adapted to many downstream tasks. Mathematically, we may write a pretraining objective

\[ \min_\theta \mathbb E_{Z\sim P_{\text{pre}}}\left[\ell_{\text{pre}}(F_\theta,Z)\right], \]

followed by adaptation

\[ \min_{\theta'} \mathbb E_{(z,y)\sim P_{\text{task}}}\left[\ell_{\text{task}}(F_{\theta'},z,y)\right]. \]

Adaptation may be done by:

zero-shot prompting;
linear probing;
fine-tuning;
low-rank adaptation;
adapters;
prompt tuning;
retrieval augmentation;
tool use and external modules.

31.15.1 What Makes a Vision Foundation Model Useful?

A vision foundation model is useful if it provides transferable representations or conditional generation mechanisms. It should support:

many input formats;
flexible task specifications;
robust transfer;
calibrated uncertainty;
efficient adaptation;
interpretable or inspectable behavior;
evaluation beyond one benchmark.

31.15.2 Statistical Caveat

A large pretraining distribution is not the same as the target distribution. If

\[ P_{\text{pre}}\ne P_{\text{task}}, \]

then transfer can fail. The relevant question is not whether the foundation model is powerful in general, but whether the learned representation is appropriate for the target population, task, and decision.

31.16 30.15 Evaluation Beyond Accuracy

Modern multimodal vision systems require multidimensional evaluation.

31.16.1 Classification and Recognition

For classification, one may use accuracy, balanced accuracy, precision, recall, specificity, F1, AUROC, AUPRC, Brier score, ECE, and confusion matrices.

31.16.2 Retrieval

For retrieval, use Recall@K, MRR, median rank, and normalized discounted cumulative gain.

31.16.3 Detection and Segmentation

For detection, use AP and mAP. For segmentation, use IoU, Dice, boundary F-score, panoptic quality, and calibration of probability maps.

31.16.4 Generation

For generation, evaluation is harder. Metrics may measure sample quality, diversity, alignment with prompts, fidelity to input constraints, or human preference. No single scalar metric fully captures generative performance.

31.16.5 Hallucination and Grounding

A vision-language system can produce a fluent answer that is not supported by the image. A simple abstraction is to define a grounding variable $G$:

\[ G=1 \quad \text{if the output is visually supported}, \]

and evaluate

\[ \mathbb P(G=1\mid \text{model answer}). \]

This is difficult because visual support can be ambiguous. Still, hallucination should be treated as a statistical error mode, not merely as a language problem.

31.16.6 Robustness and Distribution Shift

Let $P_{\text{train}}$ and $P_{\text{test}}$ be training and test distributions. Under shift,

\[ P_{\text{train}}(X,Y)\ne P_{\text{test}}(X,Y). \]

Robust evaluation asks how performance changes under:

changes in lighting, camera, viewpoint, and resolution;
corruption and noise;
demographic and domain shift;
sensor differences;
adversarial perturbations;
compositional changes in objects and relations.

31.17 30.16 Interactive Demo: Calibration Under Domain Shift

This demo shows how a classifier can remain confident while accuracy changes under distribution shift.

Interactive: Confidence, accuracy, and distribution shift

Increase domain shift to see calibration degrade.

Domain shift: 0.30

31.18 30.17 Safety, Bias, and Responsible Deployment

Mathematics cannot solve every social problem, but it can clarify assumptions and errors. For a deployed vision model, we should ask:

What population was used for training?
What population will be affected by predictions?
What are false-positive and false-negative costs?
Is uncertainty reported?
Are subgroups evaluated separately?
Can users challenge or correct predictions?
What happens when the model is outside its domain?

Suppose groups are indexed by $G\in\{1,\dots,m\}$. A basic subgroup performance analysis reports

\[ R_g(f)=\mathbb E[\ell(f(X),Y)\mid G=g]. \]

Large variation across $g$ indicates that average performance may hide important failures.

For a high-stakes setting, a model should usually have a reject option:

\[ \widehat y(x)= \begin{cases} \arg\max_k p_k(x), & \max_k p_k(x)\ge \eta,\\ \text{defer}, & \max_k p_k(x)<\eta. \end{cases} \]

The threshold $\eta$ controls the trade-off between automation and human review.

31.19 30.18 Python Example: Multimodal Embedding Geometry

The following example simulates image and text embeddings. It illustrates retrieval and zero-shot classification without using a large external model.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(7243)

n = 80
d = 32
num_classes = 4

# Class prototypes in a shared embedding space
prototypes = rng.normal(size=(num_classes, d))
prototypes /= np.linalg.norm(prototypes, axis=1, keepdims=True)

labels = rng.integers(0, num_classes, size=n)
image_emb = prototypes[labels] + 0.25 * rng.normal(size=(n, d))
text_emb = prototypes[labels] + 0.25 * rng.normal(size=(n, d))

image_emb /= np.linalg.norm(image_emb, axis=1, keepdims=True)
text_emb /= np.linalg.norm(text_emb, axis=1, keepdims=True)

S = image_emb @ text_emb.T
ranks = []
for i in range(n):
    order = np.argsort(-S[i])
    rank = np.where(order == i)[0][0] + 1
    ranks.append(rank)

recall_at_1 = np.mean(np.array(ranks) <= 1)
recall_at_5 = np.mean(np.array(ranks) <= 5)
mrr = np.mean(1 / np.array(ranks))

print("Recall@1:", recall_at_1)
print("Recall@5:", recall_at_5)
print("MRR:", mrr)

plt.figure(figsize=(5, 4))
plt.imshow(S[:20, :20], aspect="auto")
plt.colorbar(label="cosine similarity")
plt.xlabel("text index")
plt.ylabel("image index")
plt.title("Image-text similarity matrix")
plt.show()

31.20 30.19 Python Example: Simple Cross-Attention Layer

This example implements cross-attention between visual tokens and text tokens.

import tensorflow as tf

class CrossAttention(tf.keras.layers.Layer):
    def __init__(self, dim, num_heads=4):
        super().__init__()
        self.mha = tf.keras.layers.MultiHeadAttention(
            num_heads=num_heads,
            key_dim=dim // num_heads
        )
        self.norm = tf.keras.layers.LayerNormalization()
        self.ffn = tf.keras.Sequential([
            tf.keras.layers.Dense(4 * dim, activation="gelu"),
            tf.keras.layers.Dense(dim)
        ])

    def call(self, visual_tokens, text_tokens):
        # Visual tokens query text tokens.
        attended = self.mha(
            query=visual_tokens,
            key=text_tokens,
            value=text_tokens
        )
        h = self.norm(visual_tokens + attended)
        return self.norm(h + self.ffn(h))

batch = 2
num_patches = 64
num_words = 12
dim = 128

visual = tf.random.normal((batch, num_patches, dim))
text = tf.random.normal((batch, num_words, dim))

layer = CrossAttention(dim=dim, num_heads=4)
out = layer(visual, text)
print(out.shape)  # (batch, num_patches, dim)

31.21 30.20 Python Example: Domain Shift and Calibration

This example simulates a classifier whose confidence is too high on a shifted domain.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(2026)
n = 5000

# In-domain scores
z_in = rng.normal(loc=1.2, scale=1.0, size=n)
p_in = 1 / (1 + np.exp(-z_in))
y_in = rng.binomial(1, p_in)

# Shifted-domain labels are less aligned with confidence.
z_shift = rng.normal(loc=1.2, scale=1.0, size=n)
p_shift = 1 / (1 + np.exp(-z_shift))
true_p_shift = 0.25 + 0.55 * p_shift
y_shift = rng.binomial(1, true_p_shift)

def reliability(p, y, bins=10):
    edges = np.linspace(0, 1, bins + 1)
    centers, accs, confs = [], [], []
    for a, b in zip(edges[:-1], edges[1:]):
        mask = (p >= a) & (p < b)
        if mask.sum() > 0:
            centers.append((a + b) / 2)
            confs.append(p[mask].mean())
            accs.append(y[mask].mean())
    return np.array(confs), np.array(accs)

c_in, a_in = reliability(p_in, y_in)
c_sh, a_sh = reliability(p_shift, y_shift)

plt.figure(figsize=(5, 5))
plt.plot([0, 1], [0, 1], linestyle="--", label="perfect")
plt.plot(c_in, a_in, marker="o", label="in-domain")
plt.plot(c_sh, a_sh, marker="o", label="shifted")
plt.xlabel("mean confidence")
plt.ylabel("empirical accuracy")
plt.title("Reliability under distribution shift")
plt.legend()
plt.show()

31.22 30.21 TensorFlow Example: Tiny Vision-Language Classifier

The following toy model shows the pattern of using an image encoder and a text encoder in a shared classifier. It is not a production model; it is a readable template.

import tensorflow as tf

image_input = tf.keras.Input(shape=(64, 64, 3), name="image")
text_input = tf.keras.Input(shape=(20,), dtype="int32", name="tokens")

# Image encoder
x = tf.keras.layers.Conv2D(32, 3, padding="same", activation="relu")(image_input)
x = tf.keras.layers.MaxPooling2D()(x)
x = tf.keras.layers.Conv2D(64, 3, padding="same", activation="relu")(x)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dense(128)(x)

# Text encoder
vocab_size = 5000
t = tf.keras.layers.Embedding(vocab_size, 64)(text_input)
t = tf.keras.layers.Bidirectional(tf.keras.layers.GRU(64))(t)
t = tf.keras.layers.Dense(128)(t)

# Normalize and fuse
x = tf.math.l2_normalize(x, axis=-1)
t = tf.math.l2_normalize(t, axis=-1)
h = tf.keras.layers.Concatenate()([x, t, x * t])
h = tf.keras.layers.Dense(128, activation="relu")(h)
out = tf.keras.layers.Dense(10, activation="softmax")(h)

model = tf.keras.Model(inputs={"image": image_input, "tokens": text_input}, outputs=out)
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"])
model.summary()

31.23 30.22 AI-Assisted Study Prompts

Use AI tools to test mathematical understanding, not to replace it.

Fusion assumptions. Ask an AI system to compare early fusion, late fusion, shared embeddings, and cross-attention. Then rewrite the answer using conditional independence assumptions.
Retrieval metrics. Ask for a small example where Recall@1 is low but Recall@5 is high. Verify the ranks by hand.
Video complexity. Ask for the attention complexity of full space-time attention for a $T\times H\times W$ token grid. Then derive it yourself.
3D rendering. Ask for an explanation of transmittance in volume rendering. Then connect it to survival functions in probability.
Calibration. Ask for a reliability diagram example. Then compute ECE manually for five bins.
Medical vision. Ask for failure modes of a medical segmentation model under scanner shift. Separate statistical, physical, and clinical causes.
Robotics. Ask why a visually accurate representation may still be bad for control. Connect the answer to decision loss.

31.24 30.23 Mini-Project: Auditing a Multimodal Vision Model

Choose one multimodal vision task:

image-text retrieval;
open-vocabulary classification;
visual question answering;
video captioning;
promptable segmentation;
medical-image report assistance;
robot instruction following.

Write a short audit with the following components.

31.24.1 Mathematical Formulation

Define:

\[ \mathcal X,\quad \mathcal C,\quad \mathcal Y,\quad F_\theta:\mathcal X\times\mathcal C\to\mathcal Y. \]

State whether the model is deterministic or probabilistic.

31.24.2 Data Distribution

Identify training, validation, deployment, and shifted distributions:

\[ P_{\text{train}},\quad P_{\text{val}},\quad P_{\text{deploy}},\quad P_{\text{shift}}. \]

31.24.3 Loss and Metrics

State the training loss and at least three evaluation metrics. Include one metric related to uncertainty, calibration, or robustness.

31.24.4 Failure Modes

Identify at least five failure modes, including at least one related to each of:

modality misalignment;
distribution shift;
ambiguous labels or prompts;
overconfidence;
task-specific cost.

31.24.5 Mathematical Experiment

Design a small synthetic experiment that illustrates one failure mode.

31.25 30.24 Exercises

31.25.1 Conceptual Exercises

Explain why multimodal vision is not simply image classification with more data.
Give one example where early fusion is preferable to late fusion, and one example where late fusion is preferable to early fusion.
Explain why a shared image-text embedding can support zero-shot classification.
Describe a situation where zero-shot classification may fail even if image-text retrieval works well.
Explain the difference between visual grounding and language fluency.
Give one example of a multimodal hallucination.
Explain why video understanding requires more than framewise image classification.
Explain why robotics requires decision-aware visual representations.
Explain why medical vision requires uncertainty estimates.
Explain why benchmark performance may not predict deployment performance.

31.25.2 Mathematical Exercises

Let $a,b\in\mathbb R^d$ be normalized embeddings. Show that maximizing $a^Tb$ is equivalent to minimizing $\|a-b\|^2$.
Derive the gradient of the image-to-text contrastive loss with respect to the similarity scores $S_{ij}$.
Suppose a video has $T$ frames and each frame has $N$ patch tokens. Compare the complexity of full attention on $TN$ tokens with factorized temporal-then-spatial attention.
Let $P=\{p_i\}_{i=1}^n$ be a point cloud. Prove that $f(P)=\rho(\sum_i \phi(p_i))$ is permutation invariant.
In volume rendering, show that $T'(s)=-\sigma(r(s))T(s)$.
Suppose a classifier has group risks $R_1,\dots,R_m$. Compare average risk $\sum_g \pi_gR_g$ with worst-group risk $\max_g R_g$.
Derive ECE for a classifier using $B$ confidence bins.
For a reject option with threshold $\eta$, describe how increasing $\eta$ changes coverage and error among accepted predictions.
Formulate visual question answering as empirical risk minimization when answers are drawn from a finite vocabulary.
Define a loss for text-conditioned segmentation that combines pixelwise cross-entropy and text-image alignment.

31.25.3 Python Exercises

Simulate image and text embeddings for four classes. Compute Recall@1, Recall@5, and MRR.
Create a synthetic reliability diagram for an overconfident classifier.
Implement a small cross-attention layer in TensorFlow or PyTorch.
Simulate a domain shift by rotating a two-dimensional feature distribution and measure classification accuracy.
Simulate a point cloud and verify numerically that a sum-pooling model is permutation invariant.
Implement the volume-rendering weights $T(s)\sigma(s)$ for a one-dimensional density function.
Simulate zero-shot classification using noisy class prototypes.
Generate a confusion matrix separately for two subgroups and compare subgroup risks.

31.25.4 AI-Assisted Exercises

Ask an AI system to generate a table of multimodal vision tasks. Add a mathematical domain, codomain, loss, and evaluation metric to each row.
Ask an AI system for a proof that contrastive learning aligns paired embeddings. Critique what assumptions are missing.
Ask an AI system for examples of distribution shift in medical imaging. Classify each as covariate shift, label shift, concept shift, or annotation shift.
Ask an AI system to design a VQA benchmark. Identify at least three ways the benchmark could be gamed.
Ask an AI system to compare CLIP-style dual encoders and cross-attention VQA models. Rewrite the answer in terms of computational complexity and conditional modeling.

31.26 30.25 Chapter Summary

Modern computer vision is no longer only the study of image-to-label maps. It is the study of visual information inside larger systems that include language, time, geometry, action, scientific context, and human decision making.

The main mathematical lessons are:

multimodal systems are conditional maps among structured spaces;
alignment is a statistical problem involving joint distributions;
fusion choices encode assumptions about interactions among modalities;
video introduces temporal dependence and computational scaling issues;
3D vision introduces geometric structure and differentiable rendering;
robotics introduces actions, uncertainty, and decision loss;
medical and scientific vision require domain-specific validation and calibrated uncertainty;
foundation models are powerful but depend on distributional assumptions;
evaluation must include robustness, calibration, grounding, and subgroup behavior.

This book began with pixels and ended with multimodal systems. The mathematical thread is continuous: images are functions, measurements, random variables, tensors, geometric projections, optimization variables, learned representations, and finally components of decision-making systems.

31.27 Notes and References

This chapter synthesizes ideas from image-text contrastive learning (Radford et al. 2021), visual transformers (Dosovitskiy et al. 2020; Vaswani et al. 2017), promptable segmentation (Kirillov et al. 2023), diffusion models (Ho et al. 2020; Rombach et al. 2022), and two-view/3D geometry (Hartley and Zisserman 2004). Video, 3D representation learning, neural rendering, and embodied vision are large active areas; useful entry points include TimeSformer (Bertasius et al. 2021), PointNet (Qi et al. 2017), NeRF (Mildenhall et al. 2020), 3D Gaussian splatting (Kerbl et al. 2023), Flamingo-style visual language modeling (Alayrac et al. 2022), and robotics foundation-model work (Brohan et al. 2023). For medical and scientific vision, the central statistical issue is not only prediction accuracy but also calibration, uncertainty, external validation, and decision-aware evaluation.

Alayrac, Jean-Baptiste, Jeff Donahue, Pauline Luc, et al. 2022. “Flamingo: A Visual Language Model for Few-Shot Learning.” Advances in Neural Information Processing Systems 35: 23716–36. https://arxiv.org/abs/2204.14198.

Bertasius, Gedas, Heng Wang, and Lorenzo Torresani. 2021. “Is Space-Time Attention All You Need for Video Understanding?” International Conference on Machine Learning, 813–24. https://arxiv.org/abs/2102.05095.

Brohan, Anthony, Noah Brown, Justice Carbajal, et al. 2023. “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control.” arXiv Preprint arXiv:2307.15818. https://arxiv.org/abs/2307.15818.

Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, et al. 2020. “An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.” arXiv Preprint arXiv:2010.11929. https://arxiv.org/abs/2010.11929.

Hartley, Richard, and Andrew Zisserman. 2004. Multiple View Geometry in Computer Vision. 2nd ed. Cambridge University Press. https://www.robots.ox.ac.uk/~vgg/hzbook/.

Ho, Jonathan, Ajay Jain, and Pieter Abbeel. 2020. “Denoising Diffusion Probabilistic Models.” Advances in Neural Information Processing Systems 33: 6840–51. https://arxiv.org/abs/2006.11239.

Kerbl, Bernhard, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023. “3D Gaussian Splatting for Real-Time Radiance Field Rendering.” ACM Transactions on Graphics 42 (4): 1–14. https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/.

Kirillov, Alexander, Eric Mintun, Nikhila Ravi, et al. 2023. “Segment Anything.” Proceedings of the IEEE/CVF International Conference on Computer Vision. https://arxiv.org/abs/2304.02643.

Mildenhall, Ben, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.” European Conference on Computer Vision, 405–21. https://arxiv.org/abs/2003.08934.

Qi, Charles R., Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2017. “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation.” IEEE Conference on Computer Vision and Pattern Recognition, 652–60. https://arxiv.org/abs/1612.00593.

Radford, Alec, Jong Wook Kim, Chris Hallacy, et al. 2021. “Learning Transferable Visual Models from Natural Language Supervision.” International Conference on Machine Learning. https://arxiv.org/abs/2103.00020.

Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. “High-Resolution Image Synthesis with Latent Diffusion Models.” IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10684–95. https://arxiv.org/abs/2112.10752.

Vaswani, Ashish, Noam Shazeer, Niki Parmar, et al. 2017. “Attention Is All You Need.” Advances in Neural Information Processing Systems 30. https://arxiv.org/abs/1706.03762.

# Vision in the Age of Multimodal AI ::: {.hero-box} Chapter 30 studies **computer vision in the age of multimodal AI**. Earlier chapters treated images, geometry, probability, optimization, CNNs, transformers, segmentation foundation models, and diffusion models. This final chapter puts these ideas together. Modern vision systems increasingly process not only images, but also text, video, audio, depth, 3D scenes, actions, sensor streams, and scientific measurements. ::: A classical computer vision pipeline often had a narrowly defined input and output: $$ \text{image} \longmapsto \text{edges, keypoints, objects, masks, or depth}. $$ A modern multimodal vision system is better described as a family of maps among several spaces: $$ \mathcal X_{\text{image}},\quad \mathcal X_{\text{video}},\quad \mathcal X_{\text{text}},\quad \mathcal X_{\text{3D}},\quad \mathcal X_{\text{action}},\quad \mathcal Y_{\text{task}}. $$ For example, a single model may support image classification, image-text retrieval, visual question answering, segmentation, video understanding, captioning, robotic planning, and image generation. The mathematical question is not only whether a model is accurate on one benchmark. The deeper question is: $$ \text{How can visual information be represented so that it transfers across tasks, modalities, domains, and decisions?} $$ This chapter is deliberately synthetic. It does not introduce one new algorithm. Instead, it gives a mathematical framework for understanding the modern landscape of multimodal computer vision. ## Learning Goals After reading this chapter, students should be able to: 1. define multimodal vision as learning maps among image, text, video, 3D, and action spaces; 2. explain early fusion, late fusion, cross-attention, and shared-embedding fusion mathematically; 3. formulate image-text, video-text, and vision-action models as conditional probabilistic models; 4. connect foundation models to representation learning, transfer learning, and task conditioning; 5. describe the mathematical differences among image, video, 3D, and embodied vision; 6. formulate visual question answering, captioning, retrieval, grounding, and instruction-following as structured prediction problems; 7. analyze evaluation metrics for multimodal vision, including calibration, robustness, retrieval, hallucination, and distribution shift; 8. explain why medical, scientific, and robotic vision require domain-specific uncertainty and validation; 9. design small experiments to test modality alignment, cross-attention, zero-shot transfer, and robustness; 10. identify open mathematical questions in multimodal vision. ## 30.1 From Image Models to Multimodal Systems In the first chapters, an image was modeled as a function or tensor: $$ I:\Omega\subset \mathbb R^2\to \mathbb R^C, \qquad X\in \mathbb R^{H\times W\times C}. $$ Classical computer vision transformed this image into a structured output: $$ f:X\mapsto Y. $$ The output $Y$ might be a label, mask, depth map, 3D reconstruction, optical flow field, or set of bounding boxes. In multimodal AI, the input itself may have several components: $$ Z=(X,T,V,D,A,S), $$ where: - $X$ is an image; - $T$ is text; - $V$ is video; - $D$ is depth, LiDAR, or another spatial sensor; - $A$ is action, control, or user instruction; - $S$ is scientific or clinical metadata. A general multimodal vision model is a map $$ F_\theta: \mathcal Z \to \mathcal Y, $$ where $\mathcal Z$ is a product or union of modality spaces. The model may also be probabilistic: $$ p_\theta(y\mid z). $$ The conditional formulation is especially important. It covers classification, captioning, segmentation, retrieval, question answering, detection, and generation. ::: {.math-spine} A useful mathematical abstraction is that multimodal vision learns a family of conditional maps $$ F_\theta(\cdot\mid c), $$ where $c$ is a condition such as a text prompt, point prompt, task description, previous frame, robot state, or scientific context. ::: ## 30.2 Modalities as Mathematical Spaces Different modalities have different mathematical structures. ### Images An image is naturally spatial: $$ X\in\mathbb R^{H\times W\times C}. $$ It has local neighborhoods, translations, edges, textures, and approximate scale structure. ### Text Text is naturally sequential: $$ T=(w_1,\dots,w_L), $$ where $w_i$ are tokens. After embedding, $$ E_T(T)\in\mathbb R^{L\times d}. $$ Text has syntax, semantics, ambiguity, and compositional structure. ### Video A video is spatio-temporal: $$ V\in\mathbb R^{T\times H\times W\times C}. $$ It has motion, temporal causality, occlusion, events, and long-range dependency. ### 3D Data A 3D scene may be represented as: - a point cloud $P=\{p_i\}_{i=1}^n\subset\mathbb R^3$; - a mesh $(\mathcal V,\mathcal E,\mathcal F)$; - an implicit surface $\{x:f_\theta(x)=0\}$; - a voxel grid $X\in\mathbb R^{N_x\times N_y\times N_z\times C}$; - a radiance field $F_\theta(x,d)=(\sigma,c)$. The central mathematical structure is geometry in $\mathbb R^3$. ### Action and Control In robotics and embodied AI, the output may be an action: $$ a_t\in\mathcal A. $$ A perception-action system is often modeled as a policy: $$ \pi_\theta(a_t\mid o_{1:t},g), $$ where $o_{1:t}$ denotes observations and $g$ denotes a goal or instruction. ## 30.3 Encoders, Decoders, and Fusion Most modern multimodal systems can be decomposed into encoders, fusion modules, and decoders. Suppose we have an image $x$ and a text input $t$. Let $$ z_x=f_\theta(x),\qquad z_t=g_\phi(t). $$ The fusion module combines $z_x$ and $z_t$: $$ h=\mathcal F_\psi(z_x,z_t). $$ A task decoder then produces the output: $$ \widehat y=d_\omega(h). $$ This modular view is useful because different architectures mainly differ in how they define $f_\theta$, $g_\phi$, $\mathcal F_\psi$, and $d_\omega$. ### Early Fusion Early fusion combines modalities before deep processing: $$ h_0=[e_x;e_t]. $$ A single model then processes the concatenated representation. Early fusion can model strong interactions but may be computationally expensive and less modular. ### Late Fusion Late fusion processes modalities separately and combines decisions: $$ s = \alpha s_x + (1-\alpha)s_t. $$ This is simple and robust but may miss fine-grained alignment. ### Shared Embedding Fusion A CLIP-style model maps images and text into a shared embedding space: $$ a=\frac{f_\theta(x)}{\|f_\theta(x)\|}, \qquad b=\frac{g_\phi(t)}{\|g_\phi(t)\|}. $$ Similarity is measured by $$ s(x,t)=a^Tb. $$ This supports retrieval and zero-shot classification. ### Cross-Attention Fusion Cross-attention uses one modality as queries and another as keys and values: $$ \operatorname{CrossAttn}(Q_x,K_t,V_t) = \operatorname{softmax}\left(\frac{Q_xK_t^T}{\sqrt d}\right)V_t. $$ The result is a visual representation conditioned on text, or a textual representation conditioned on visual evidence. ::: {.callout-note} For MA Applied Mathematics and MS Statistics students, the key point is that fusion is not just an engineering choice. It changes the conditional independence assumptions, computational complexity, sample complexity, and failure modes of the model. ::: ## 30.4 Interactive Demo: Fusion Strategies The following demo compares three schematic fusion strategies. The point is not to simulate a real network, but to visualize how information can be combined. ```{=html} <div class="interactive-card"> <h4>Interactive: Early, late, and shared-embedding fusion</h4> <p>Move the slider to change the agreement between visual and text evidence.</p> <label>Modality agreement: <span id="fusion-agreement-label">0.70</span></label> <input id="fusion-agreement" type="range" min="0" max="1" step="0.01" value="0.70" style="width:100%;"> <div id="plot-ch30-fusion" class="plotly-demo"></div> </div> <script> (function(){ const slider = document.getElementById('fusion-agreement'); const label = document.getElementById('fusion-agreement-label'); const div = document.getElementById('plot-ch30-fusion'); function draw(){ const a = parseFloat(slider.value); label.textContent = a.toFixed(2); const theta = Array.from({length:80}, (_,i)=>2*Math.PI*i/79); const img = theta.map(t => [Math.cos(t), Math.sin(t)]); const txt = theta.map(t => [Math.cos(t+a*Math.PI/3), Math.sin(t+a*Math.PI/3)]); const early = img.map((p,i)=>[(p[0]+txt[i][0])/2, (p[1]+txt[i][1])/2]); const lateScore = 0.55 + 0.35*a; const traces = [ {x: img.map(p=>p[0]), y: img.map(p=>p[1]), mode:'lines', name:'image representation'}, {x: txt.map(p=>p[0]), y: txt.map(p=>p[1]), mode:'lines', name:'text representation'}, {x: early.map(p=>p[0]), y: early.map(p=>p[1]), mode:'lines', name:'early fused representation'}, {x:[-1.2,1.2], y:[lateScore,lateScore], mode:'lines', name:'late fused score'} ]; Plotly.newPlot(div, traces, { margin:{t:30}, xaxis:{range:[-1.4,1.4], zeroline:true}, yaxis:{range:[-1.4,1.4], zeroline:true}, title:'Schematic modality fusion' }, {displayModeBar:false}); } slider.addEventListener('input', draw); draw(); })(); </script> ``` ## 30.5 Alignment as a Statistical Problem Multimodal learning often requires alignment. For an image $x_i$ and its paired text $t_i$, a dual encoder produces normalized embeddings $$ a_i=f_\theta(x_i),\qquad b_i=g_\phi(t_i), \qquad \|a_i\|=\|b_i\|=1. $$ The similarity matrix is $$ S_{ij}=\frac{a_i^Tb_j}{\tau}, $$ where $\tau>0$ is the temperature. A common symmetric contrastive loss is $$ \mathcal L = \frac{1}{2n}\sum_{i=1}^n \left[ -\log\frac{\exp(S_{ii})}{\sum_{j=1}^n \exp(S_{ij})} - \log\frac{\exp(S_{ii})}{\sum_{j=1}^n \exp(S_{ji})} \right]. $$ This loss asks each image to identify its paired text and each text to identify its paired image. From a statistical point of view, the paired dataset is a sample from a joint distribution: $$ (X,T)\sim P_{XT}. $$ The goal is to learn representations that preserve information about dependence between $X$ and $T$ while discarding nuisance variation. This is not the same as learning all visual detail or all language detail. ## 30.6 Retrieval, Grounding, and Open-Vocabulary Prediction A shared embedding space supports retrieval. Given an image query $x$, retrieve text items by ranking $$ t_j \mapsto s(x,t_j)=f_\theta(x)^Tg_\phi(t_j). $$ Given a text query $t$, retrieve images by ranking $$ x_i \mapsto s(x_i,t). $$ Common retrieval metrics include: $$ \operatorname{Recall@K}=\frac{1}{n}\sum_{i=1}^n \mathbf 1\{\text{correct item appears in top }K\}, $$ and mean reciprocal rank: $$ \operatorname{MRR} =\frac{1}{n}\sum_{i=1}^n \frac{1}{r_i}, $$ where $r_i$ is the rank of the correct item. Open-vocabulary classification can be formulated by constructing text prototypes $t_1,\dots,t_K$ and choosing $$ \widehat y(x)=\arg\max_k f_\theta(x)^T g_\phi(t_k). $$ Open-vocabulary detection and segmentation extend this idea from whole images to regions or pixels. ## 30.7 Visual Question Answering and Captioning Visual question answering is a conditional prediction problem: $$ \widehat a=\arg\max_a p_\theta(a\mid x,q), $$ where $x$ is an image and $q$ is a question. Image captioning is conditional language modeling: $$ p_\theta(w_{1:L}\mid x) =\prod_{\ell=1}^L p_\theta(w_\ell\mid w_{<\ell},x). $$ The output space is no longer a small finite label set. It is a structured language space. This creates several mathematical and statistical issues: - many answers can be semantically equivalent; - a generated caption can be fluent but visually unsupported; - likelihood does not necessarily measure factual grounding; - evaluation depends on reference captions, human judgments, or external models; - the model may exploit dataset priors rather than image evidence. A useful decomposition is $$ p_\theta(a\mid x,q) = \int p_\theta(a\mid h,q)p_\theta(h\mid x,q)\,dh, $$ where $h$ is an internal multimodal representation. If $h$ ignores the image, the system becomes a language prior rather than a vision-language model. ## 30.8 Interactive Demo: Cross-Attention as Visual Grounding Cross-attention can be interpreted as a soft alignment between words and visual tokens. The following demo shows a schematic attention distribution from one text token to image patches. ```{=html} <div class="interactive-card"> <h4>Interactive: Cross-attention from a word to image patches</h4> <p>Move the slider to change attention sharpness. Sharper attention is easier to interpret but can become brittle.</p> <label>Attention sharpness: <span id="attn-sharp-label">4.00</span></label> <input id="attn-sharp" type="range" min="0.5" max="12" step="0.1" value="4" style="width:100%;"> <div id="plot-ch30-attention" class="plotly-demo"></div> </div> <script> (function(){ const slider = document.getElementById('attn-sharp'); const label = document.getElementById('attn-sharp-label'); const div = document.getElementById('plot-ch30-attention'); const n=12; function draw(){ const sharp = parseFloat(slider.value); label.textContent = sharp.toFixed(2); const z=[]; for(let i=0;i<n;i++){ const row=[]; for(let j=0;j<n;j++){ const d1=(i-4)*(i-4)+(j-7)*(j-7); const d2=(i-8)*(i-8)+(j-4)*(j-4); row.push(Math.exp(-d1/sharp)+0.65*Math.exp(-d2/(1.4*sharp))); } z.push(row); } const total = z.flat().reduce((a,b)=>a+b,0); const zn = z.map(row=>row.map(v=>v/total)); Plotly.newPlot(div, [{z:zn, type:'heatmap'}], { margin:{t:30}, title:'Attention weights over image patches', xaxis:{title:'patch column'}, yaxis:{title:'patch row', autorange:'reversed'} }, {displayModeBar:false}); } slider.addEventListener('input', draw); draw(); })(); </script> ``` ## 30.9 Video Understanding Images are spatial. Video is spatio-temporal. A video can be modeled as $$ V:\Omega\times \{1,\dots,T\}\to \mathbb R^C. $$ Discrete video data are tensors $$ V\in\mathbb R^{T\times H\times W\times C}. $$ Video tasks include: - action recognition; - temporal localization; - event detection; - tracking; - video captioning; - video question answering; - prediction of future frames or states. A simple video classifier has the form $$ \widehat y=f_\theta(V_{1:T}). $$ A temporal localization model predicts intervals: $$ \{([s_j,e_j],c_j)\}_{j=1}^m. $$ Video introduces several new mathematical issues. ### Temporal Dependence Frames are not independent. A common probabilistic factorization is $$ p(V_{1:T})=\prod_{t=1}^T p(V_t\mid V_{<t}). $$ For supervised video prediction: $$ p(y\mid V_{1:T}). $$ ### Motion and Equivariance Motion creates transformations over time: $$ V_{t+1}(x)\approx V_t(x-u_t(x)), $$ where $u_t$ is optical flow. A good video representation should capture both appearance and motion. ### Long-Range Attention If a video has $N=THW$ tokens, full self-attention costs $$ O(N^2d)=O(T^2H^2W^2d). $$ This motivates factorized space-time attention, temporal pooling, sparse attention, memory modules, and token compression. ## 30.10 3D Vision and Scene Representations Chapter 12 through Chapter 17 studied projective geometry and 3D reconstruction. Modern multimodal vision adds learned 3D representations and language-conditioned 3D reasoning. ### Point Clouds A point cloud is a finite set $$ P=\{p_i\in\mathbb R^3\}_{i=1}^n. $$ Since it is a set, a point-cloud representation should be permutation invariant or permutation equivariant: $$ f(\{p_1,\dots,p_n\})=f(\{p_{\sigma(1)},\dots,p_{\sigma(n)}\}). $$ A common form is $$ f(P)=\rho\left(\sum_{i=1}^n \phi(p_i)\right), $$ where $\phi$ processes points and $\rho$ aggregates them. ### Implicit Surfaces An implicit surface is represented as a level set $$ \mathcal S=\{x\in\mathbb R^3:f_\theta(x)=0\}. $$ Signed distance functions satisfy approximately $$ \|\nabla f_\theta(x)\|=1. $$ ### Neural Radiance Fields A radiance field maps position and viewing direction to density and color: $$ F_\theta(x,d)=(\sigma(x),c(x,d)). $$ An image pixel is generated by volume rendering along a camera ray $r(s)=o+sd$: $$ C(r)=\int_{s_n}^{s_f} T(s)\sigma(r(s))c(r(s),d)\,ds, $$ where $$ T(s)=\exp\left(-\int_{s_n}^s \sigma(r(u))\,du\right) $$ is transmittance. This connects 3D geometry, differentiable rendering, and inverse problems. ### Gaussian Scene Representations A modern explicit alternative represents a scene by many 3D primitives: $$ \{(\mu_i,\Sigma_i,c_i,\alpha_i)\}_{i=1}^n, $$ where each primitive has position, covariance, color, and opacity. Rendering becomes differentiable projection and alpha compositing. ## 30.11 Interactive Demo: Volume Rendering Along a Ray The following toy visualization shows how density along a ray contributes to a rendered pixel. High density early in the ray blocks later color by reducing transmittance. ```{=html} <div class="interactive-card"> <h4>Interactive: Volume-rendering weights along a ray</h4> <p>Adjust density strength. The rendered color is a weighted integral along the ray.</p> <label>Density strength: <span id="density-label">2.00</span></label> <input id="density-strength" type="range" min="0" max="6" step="0.05" value="2" style="width:100%;"> <div id="plot-ch30-volume" class="plotly-demo"></div> </div> <script> (function(){ const slider = document.getElementById('density-strength'); const label = document.getElementById('density-label'); const div = document.getElementById('plot-ch30-volume'); function draw(){ const amp = parseFloat(slider.value); label.textContent = amp.toFixed(2); const n=160; const s=[]; const sigma=[]; const T=[]; const w=[]; let accum=0; const ds=1/(n-1); for(let i=0;i<n;i++){ const x=i*ds; const sig=amp*(Math.exp(-Math.pow((x-0.35)/0.08,2))+0.6*Math.exp(-Math.pow((x-0.7)/0.1,2))); s.push(x); sigma.push(sig); const trans=Math.exp(-accum); T.push(trans); w.push(trans*sig); accum += sig*ds; } Plotly.newPlot(div, [ {x:s,y:sigma,mode:'lines',name:'density sigma(s)'}, {x:s,y:T,mode:'lines',name:'transmittance T(s)'}, {x:s,y:w,mode:'lines',name:'rendering weight T(s)sigma(s)'} ], { margin:{t:30}, title:'Differentiable volume rendering along one ray', xaxis:{title:'ray coordinate s'}, yaxis:{title:'value'} }, {displayModeBar:false}); } slider.addEventListener('input', draw); draw(); })(); </script> ``` ## 30.12 Robotics and Embodied Vision In robotics, vision is not only recognition. Vision supports action. The system observes the world, chooses actions, and receives feedback. A partially observed decision process can be written as $$ s_t\sim \text{latent state},\qquad o_t\sim p(o_t\mid s_t),\qquad a_t\sim \pi_\theta(a_t\mid o_{1:t},g). $$ The goal may be to maximize expected return: $$ J(\theta)=\mathbb E_{\pi_\theta}\left[\sum_{t=0}^{T}\gamma^t r(s_t,a_t)\right]. $$ Vision contributes to estimating the state, recognizing objects, localizing the robot, predicting affordances, and grounding language instructions. ### Vision-Language-Action Models A language-conditioned robotic policy may have the form $$ \pi_\theta(a_t\mid I_t, q, h_t), $$ where $I_t$ is an image, $q$ is an instruction, and $h_t$ is a history representation. A more structured model separates perception, planning, and control: $$ I_t \mapsto z_t, \qquad (z_t,q)\mapsto \text{plan}, \qquad \text{plan}\mapsto a_t. $$ The end-to-end formulation is flexible, but the modular formulation can be easier to validate and debug. ### Uncertainty in Embodied Vision Errors in embodied systems have consequences. If $b_t(s)$ is a belief distribution over states, then action selection should depend on uncertainty: $$ a_t = \arg\max_a \mathbb E_{s\sim b_t}\left[Q(s,a)\right]. $$ This connects robotics to Bayesian filtering, decision theory, and risk-sensitive control. ## 30.13 Medical and Scientific Vision Medical and scientific vision are especially important for applied mathematics and statistics students. They combine high-dimensional imaging with domain-specific physical and biological constraints. Examples include: - CT, MRI, ultrasound, pathology, dermatology, and ophthalmology; - microscopy, astronomy, climate imagery, remote sensing, and materials science; - segmentation, registration, detection, reconstruction, denoising, and measurement. A medical imaging problem often has the form $$ y=\mathcal A(x)+\epsilon, $$ where $x$ is the latent anatomical or physiological object, $\mathcal A$ is an imaging operator, and $y$ is observed data. Deep learning can enter as a prior, a reconstruction map, a segmentation map, or a decision support model. ### Why Medical and Scientific Vision Are Different Several issues become central: - ground truth may be noisy or subjective; - label prevalence may be low; - images may come from different scanners, protocols, or institutions; - small errors may have high cost; - uncertainty and calibration matter more than benchmark accuracy; - external validation is essential; - interpretability may be legally, ethically, or scientifically required. For a diagnostic classifier that outputs $\widehat p(x)$, calibration means $$ \mathbb P(Y=1\mid \widehat p(X)=p)\approx p. $$ For segmentation, uncertainty may be represented by a distribution over masks: $$ p_\theta(M\mid X). $$ In scientific vision, the output should often be a measurement with uncertainty, not only a label. ## 30.14 Multimodal Foundation Models A foundation model is trained on broad data and adapted to many downstream tasks. Mathematically, we may write a pretraining objective $$ \min_\theta \mathbb E_{Z\sim P_{\text{pre}}}\left[\ell_{\text{pre}}(F_\theta,Z)\right], $$ followed by adaptation $$ \min_{\theta'} \mathbb E_{(z,y)\sim P_{\text{task}}}\left[\ell_{\text{task}}(F_{\theta'},z,y)\right]. $$ Adaptation may be done by: - zero-shot prompting; - linear probing; - fine-tuning; - low-rank adaptation; - adapters; - prompt tuning; - retrieval augmentation; - tool use and external modules. ### What Makes a Vision Foundation Model Useful? A vision foundation model is useful if it provides transferable representations or conditional generation mechanisms. It should support: - many input formats; - flexible task specifications; - robust transfer; - calibrated uncertainty; - efficient adaptation; - interpretable or inspectable behavior; - evaluation beyond one benchmark. ### Statistical Caveat A large pretraining distribution is not the same as the target distribution. If $$ P_{\text{pre}}\ne P_{\text{task}}, $$ then transfer can fail. The relevant question is not whether the foundation model is powerful in general, but whether the learned representation is appropriate for the target population, task, and decision. ## 30.15 Evaluation Beyond Accuracy Modern multimodal vision systems require multidimensional evaluation. ### Classification and Recognition For classification, one may use accuracy, balanced accuracy, precision, recall, specificity, F1, AUROC, AUPRC, Brier score, ECE, and confusion matrices. ### Retrieval For retrieval, use `Recall@K`, MRR, median rank, and normalized discounted cumulative gain. ### Detection and Segmentation For detection, use AP and mAP. For segmentation, use IoU, Dice, boundary F-score, panoptic quality, and calibration of probability maps. ### Generation For generation, evaluation is harder. Metrics may measure sample quality, diversity, alignment with prompts, fidelity to input constraints, or human preference. No single scalar metric fully captures generative performance. ### Hallucination and Grounding A vision-language system can produce a fluent answer that is not supported by the image. A simple abstraction is to define a grounding variable $G$: $$ G=1 \quad \text{if the output is visually supported}, $$ and evaluate $$ \mathbb P(G=1\mid \text{model answer}). $$ This is difficult because visual support can be ambiguous. Still, hallucination should be treated as a statistical error mode, not merely as a language problem. ### Robustness and Distribution Shift Let $P_{\text{train}}$ and $P_{\text{test}}$ be training and test distributions. Under shift, $$ P_{\text{train}}(X,Y)\ne P_{\text{test}}(X,Y). $$ Robust evaluation asks how performance changes under: - changes in lighting, camera, viewpoint, and resolution; - corruption and noise; - demographic and domain shift; - sensor differences; - adversarial perturbations; - compositional changes in objects and relations. ## 30.16 Interactive Demo: Calibration Under Domain Shift This demo shows how a classifier can remain confident while accuracy changes under distribution shift. ```{=html} <div class="interactive-card"> <h4>Interactive: Confidence, accuracy, and distribution shift</h4> <p>Increase domain shift to see calibration degrade.</p> <label>Domain shift: <span id="shift-label">0.30</span></label> <input id="domain-shift" type="range" min="0" max="1" step="0.01" value="0.30" style="width:100%;"> <div id="plot-ch30-calibration" class="plotly-demo"></div> </div> <script> (function(){ const slider = document.getElementById('domain-shift'); const label = document.getElementById('shift-label'); const div = document.getElementById('plot-ch30-calibration'); function draw(){ const s = parseFloat(slider.value); label.textContent = s.toFixed(2); const conf=[]; const acc=[]; const ideal=[]; for(let i=1;i<=10;i++){ const c=(i-0.5)/10; conf.push(c); ideal.push(c); const a=Math.max(0, Math.min(1, c - s*(0.1+0.5*c) + 0.04*Math.sin(8*c))); acc.push(a); } Plotly.newPlot(div, [ {x:conf,y:ideal,mode:'lines',name:'perfect calibration'}, {x:conf,y:acc,mode:'lines+markers',name:'observed accuracy'} ], { margin:{t:30}, title:'Reliability diagram under domain shift', xaxis:{title:'predicted confidence', range:[0,1]}, yaxis:{title:'empirical accuracy', range:[0,1]} }, {displayModeBar:false}); } slider.addEventListener('input', draw); draw(); })(); </script> ``` ## 30.17 Safety, Bias, and Responsible Deployment Mathematics cannot solve every social problem, but it can clarify assumptions and errors. For a deployed vision model, we should ask: - What population was used for training? - What population will be affected by predictions? - What are false-positive and false-negative costs? - Is uncertainty reported? - Are subgroups evaluated separately? - Can users challenge or correct predictions? - What happens when the model is outside its domain? Suppose groups are indexed by $G\in\{1,\dots,m\}$. A basic subgroup performance analysis reports $$ R_g(f)=\mathbb E[\ell(f(X),Y)\mid G=g]. $$ Large variation across $g$ indicates that average performance may hide important failures. For a high-stakes setting, a model should usually have a reject option: $$ \widehat y(x)= \begin{cases} \arg\max_k p_k(x), & \max_k p_k(x)\ge \eta,\\ \text{defer}, & \max_k p_k(x)<\eta. \end{cases} $$ The threshold $\eta$ controls the trade-off between automation and human review. ## 30.18 Python Example: Multimodal Embedding Geometry The following example simulates image and text embeddings. It illustrates retrieval and zero-shot classification without using a large external model. ```{python} #| label: py-ch30-embedding-retrieval #| eval: false import numpy as np import matplotlib.pyplot as plt rng = np.random.default_rng(7243) n = 80 d = 32 num_classes = 4 # Class prototypes in a shared embedding space prototypes = rng.normal(size=(num_classes, d)) prototypes /= np.linalg.norm(prototypes, axis=1, keepdims=True) labels = rng.integers(0, num_classes, size=n) image_emb = prototypes[labels] + 0.25 * rng.normal(size=(n, d)) text_emb = prototypes[labels] + 0.25 * rng.normal(size=(n, d)) image_emb /= np.linalg.norm(image_emb, axis=1, keepdims=True) text_emb /= np.linalg.norm(text_emb, axis=1, keepdims=True) S = image_emb @ text_emb.T ranks = [] for i in range(n): order = np.argsort(-S[i]) rank = np.where(order == i)[0][0] + 1 ranks.append(rank) recall_at_1 = np.mean(np.array(ranks) <= 1) recall_at_5 = np.mean(np.array(ranks) <= 5) mrr = np.mean(1 / np.array(ranks)) print("Recall@1:", recall_at_1) print("Recall@5:", recall_at_5) print("MRR:", mrr) plt.figure(figsize=(5, 4)) plt.imshow(S[:20, :20], aspect="auto") plt.colorbar(label="cosine similarity") plt.xlabel("text index") plt.ylabel("image index") plt.title("Image-text similarity matrix") plt.show() ``` ## 30.19 Python Example: Simple Cross-Attention Layer This example implements cross-attention between visual tokens and text tokens. ```{python} #| label: py-ch30-cross-attention #| eval: false import tensorflow as tf class CrossAttention(tf.keras.layers.Layer): def __init__(self, dim, num_heads=4): super().__init__() self.mha = tf.keras.layers.MultiHeadAttention( num_heads=num_heads, key_dim=dim // num_heads ) self.norm = tf.keras.layers.LayerNormalization() self.ffn = tf.keras.Sequential([ tf.keras.layers.Dense(4 * dim, activation="gelu"), tf.keras.layers.Dense(dim) ]) def call(self, visual_tokens, text_tokens): # Visual tokens query text tokens. attended = self.mha( query=visual_tokens, key=text_tokens, value=text_tokens ) h = self.norm(visual_tokens + attended) return self.norm(h + self.ffn(h)) batch = 2 num_patches = 64 num_words = 12 dim = 128 visual = tf.random.normal((batch, num_patches, dim)) text = tf.random.normal((batch, num_words, dim)) layer = CrossAttention(dim=dim, num_heads=4) out = layer(visual, text) print(out.shape) # (batch, num_patches, dim) ``` ## 30.20 Python Example: Domain Shift and Calibration This example simulates a classifier whose confidence is too high on a shifted domain. ```{python} #| label: py-ch30-calibration-simulation #| eval: false import numpy as np import matplotlib.pyplot as plt rng = np.random.default_rng(2026) n = 5000 # In-domain scores z_in = rng.normal(loc=1.2, scale=1.0, size=n) p_in = 1 / (1 + np.exp(-z_in)) y_in = rng.binomial(1, p_in) # Shifted-domain labels are less aligned with confidence. z_shift = rng.normal(loc=1.2, scale=1.0, size=n) p_shift = 1 / (1 + np.exp(-z_shift)) true_p_shift = 0.25 + 0.55 * p_shift y_shift = rng.binomial(1, true_p_shift) def reliability(p, y, bins=10): edges = np.linspace(0, 1, bins + 1) centers, accs, confs = [], [], [] for a, b in zip(edges[:-1], edges[1:]): mask = (p >= a) & (p < b) if mask.sum() > 0: centers.append((a + b) / 2) confs.append(p[mask].mean()) accs.append(y[mask].mean()) return np.array(confs), np.array(accs) c_in, a_in = reliability(p_in, y_in) c_sh, a_sh = reliability(p_shift, y_shift) plt.figure(figsize=(5, 5)) plt.plot([0, 1], [0, 1], linestyle="--", label="perfect") plt.plot(c_in, a_in, marker="o", label="in-domain") plt.plot(c_sh, a_sh, marker="o", label="shifted") plt.xlabel("mean confidence") plt.ylabel("empirical accuracy") plt.title("Reliability under distribution shift") plt.legend() plt.show() ``` ## 30.21 TensorFlow Example: Tiny Vision-Language Classifier The following toy model shows the pattern of using an image encoder and a text encoder in a shared classifier. It is not a production model; it is a readable template. ```{python} #| label: py-ch30-tensorflow-vl-classifier #| eval: false import tensorflow as tf image_input = tf.keras.Input(shape=(64, 64, 3), name="image") text_input = tf.keras.Input(shape=(20,), dtype="int32", name="tokens") # Image encoder x = tf.keras.layers.Conv2D(32, 3, padding="same", activation="relu")(image_input) x = tf.keras.layers.MaxPooling2D()(x) x = tf.keras.layers.Conv2D(64, 3, padding="same", activation="relu")(x) x = tf.keras.layers.GlobalAveragePooling2D()(x) x = tf.keras.layers.Dense(128)(x) # Text encoder vocab_size = 5000 t = tf.keras.layers.Embedding(vocab_size, 64)(text_input) t = tf.keras.layers.Bidirectional(tf.keras.layers.GRU(64))(t) t = tf.keras.layers.Dense(128)(t) # Normalize and fuse x = tf.math.l2_normalize(x, axis=-1) t = tf.math.l2_normalize(t, axis=-1) h = tf.keras.layers.Concatenate()([x, t, x * t]) h = tf.keras.layers.Dense(128, activation="relu")(h) out = tf.keras.layers.Dense(10, activation="softmax")(h) model = tf.keras.Model(inputs={"image": image_input, "tokens": text_input}, outputs=out) model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"]) model.summary() ``` ## 30.22 AI-Assisted Study Prompts Use AI tools to test mathematical understanding, not to replace it. 1. **Fusion assumptions.** Ask an AI system to compare early fusion, late fusion, shared embeddings, and cross-attention. Then rewrite the answer using conditional independence assumptions. 2. **Retrieval metrics.** Ask for a small example where `Recall@1` is low but `Recall@5` is high. Verify the ranks by hand. 3. **Video complexity.** Ask for the attention complexity of full space-time attention for a $T\times H\times W$ token grid. Then derive it yourself. 4. **3D rendering.** Ask for an explanation of transmittance in volume rendering. Then connect it to survival functions in probability. 5. **Calibration.** Ask for a reliability diagram example. Then compute ECE manually for five bins. 6. **Medical vision.** Ask for failure modes of a medical segmentation model under scanner shift. Separate statistical, physical, and clinical causes. 7. **Robotics.** Ask why a visually accurate representation may still be bad for control. Connect the answer to decision loss. ## 30.23 Mini-Project: Auditing a Multimodal Vision Model Choose one multimodal vision task: - image-text retrieval; - open-vocabulary classification; - visual question answering; - video captioning; - promptable segmentation; - medical-image report assistance; - robot instruction following. Write a short audit with the following components. ### Mathematical Formulation Define: $$ \mathcal X,\quad \mathcal C,\quad \mathcal Y,\quad F_\theta:\mathcal X\times\mathcal C\to\mathcal Y. $$ State whether the model is deterministic or probabilistic. ### Data Distribution Identify training, validation, deployment, and shifted distributions: $$ P_{\text{train}},\quad P_{\text{val}},\quad P_{\text{deploy}},\quad P_{\text{shift}}. $$ ### Loss and Metrics State the training loss and at least three evaluation metrics. Include one metric related to uncertainty, calibration, or robustness. ### Failure Modes Identify at least five failure modes, including at least one related to each of: - modality misalignment; - distribution shift; - ambiguous labels or prompts; - overconfidence; - task-specific cost. ### Mathematical Experiment Design a small synthetic experiment that illustrates one failure mode. ## 30.24 Exercises ### Conceptual Exercises 1. Explain why multimodal vision is not simply image classification with more data. 2. Give one example where early fusion is preferable to late fusion, and one example where late fusion is preferable to early fusion. 3. Explain why a shared image-text embedding can support zero-shot classification. 4. Describe a situation where zero-shot classification may fail even if image-text retrieval works well. 5. Explain the difference between visual grounding and language fluency. 6. Give one example of a multimodal hallucination. 7. Explain why video understanding requires more than framewise image classification. 8. Explain why robotics requires decision-aware visual representations. 9. Explain why medical vision requires uncertainty estimates. 10. Explain why benchmark performance may not predict deployment performance. ### Mathematical Exercises 1. Let $a,b\in\mathbb R^d$ be normalized embeddings. Show that maximizing $a^Tb$ is equivalent to minimizing $\|a-b\|^2$. 2. Derive the gradient of the image-to-text contrastive loss with respect to the similarity scores $S_{ij}$. 3. Suppose a video has $T$ frames and each frame has $N$ patch tokens. Compare the complexity of full attention on $TN$ tokens with factorized temporal-then-spatial attention. 4. Let $P=\{p_i\}_{i=1}^n$ be a point cloud. Prove that $f(P)=\rho(\sum_i \phi(p_i))$ is permutation invariant. 5. In volume rendering, show that $T'(s)=-\sigma(r(s))T(s)$. 6. Suppose a classifier has group risks $R_1,\dots,R_m$. Compare average risk $\sum_g \pi_gR_g$ with worst-group risk $\max_g R_g$. 7. Derive ECE for a classifier using $B$ confidence bins. 8. For a reject option with threshold $\eta$, describe how increasing $\eta$ changes coverage and error among accepted predictions. 9. Formulate visual question answering as empirical risk minimization when answers are drawn from a finite vocabulary. 10. Define a loss for text-conditioned segmentation that combines pixelwise cross-entropy and text-image alignment. ### Python Exercises 1. Simulate image and text embeddings for four classes. Compute `Recall@1`, `Recall@5`, and MRR. 2. Create a synthetic reliability diagram for an overconfident classifier. 3. Implement a small cross-attention layer in TensorFlow or PyTorch. 4. Simulate a domain shift by rotating a two-dimensional feature distribution and measure classification accuracy. 5. Simulate a point cloud and verify numerically that a sum-pooling model is permutation invariant. 6. Implement the volume-rendering weights $T(s)\sigma(s)$ for a one-dimensional density function. 7. Simulate zero-shot classification using noisy class prototypes. 8. Generate a confusion matrix separately for two subgroups and compare subgroup risks. ### AI-Assisted Exercises 1. Ask an AI system to generate a table of multimodal vision tasks. Add a mathematical domain, codomain, loss, and evaluation metric to each row. 2. Ask an AI system for a proof that contrastive learning aligns paired embeddings. Critique what assumptions are missing. 3. Ask an AI system for examples of distribution shift in medical imaging. Classify each as covariate shift, label shift, concept shift, or annotation shift. 4. Ask an AI system to design a VQA benchmark. Identify at least three ways the benchmark could be gamed. 5. Ask an AI system to compare CLIP-style dual encoders and cross-attention VQA models. Rewrite the answer in terms of computational complexity and conditional modeling. ## 30.25 Chapter Summary Modern computer vision is no longer only the study of image-to-label maps. It is the study of visual information inside larger systems that include language, time, geometry, action, scientific context, and human decision making. The main mathematical lessons are: - multimodal systems are conditional maps among structured spaces; - alignment is a statistical problem involving joint distributions; - fusion choices encode assumptions about interactions among modalities; - video introduces temporal dependence and computational scaling issues; - 3D vision introduces geometric structure and differentiable rendering; - robotics introduces actions, uncertainty, and decision loss; - medical and scientific vision require domain-specific validation and calibrated uncertainty; - foundation models are powerful but depend on distributional assumptions; - evaluation must include robustness, calibration, grounding, and subgroup behavior. This book began with pixels and ended with multimodal systems. The mathematical thread is continuous: images are functions, measurements, random variables, tensors, geometric projections, optimization variables, learned representations, and finally components of decision-making systems. ## Notes and References This chapter synthesizes ideas from image-text contrastive learning [@radford2021clip], visual transformers [@dosovitskiy2020image; @vaswani2017attention], promptable segmentation [@kirillov2023segment], diffusion models [@ho2020denoising; @rombach2022latent], and two-view/3D geometry [@hartley2004multiple]. Video, 3D representation learning, neural rendering, and embodied vision are large active areas; useful entry points include TimeSformer [@bertasius2021timesformer], PointNet [@qi2017pointnet], NeRF [@mildenhall2020nerf], 3D Gaussian splatting [@kerbl20233dgaussian], Flamingo-style visual language modeling [@alayrac2022flamingo], and robotics foundation-model work [@brohan2023rt2]. For medical and scientific vision, the central statistical issue is not only prediction accuracy but also calibration, uncertainty, external validation, and decision-aware evaluation.