21 Chapter 20. Representation Learning

Feature Maps, Embeddings, Metric Geometry, and Self-Supervised Vision

21.1 20.1 Why representation learning belongs in a mathematical vision book

Classical computer vision often begins with a hand-designed transformation:

\[ I \longmapsto \text{edges}, \qquad I \longmapsto \text{corners}, \qquad I \longmapsto \text{SIFT descriptors}, \qquad I \longmapsto \text{histograms}. \]

Modern computer vision replaces many hand-designed transformations by a learned map

\[ \phi_\theta : \mathcal X \to \mathbb R^d, \]

where $\mathcal X$ is an image space and $\phi_\theta(x)$ is a vector representation, embedding, feature vector, descriptor, or latent code. The downstream task then uses a simpler decision rule:

\[ \widehat y = h_w(\phi_\theta(x)). \]

The central mathematical question is not merely how to train a large neural network. The central question is:

What properties should a representation preserve, discard, linearize, separate, and stabilize?

For MA Applied Math and MS Statistics students, representation learning is best understood as the meeting point of linear algebra, statistics, optimization, geometry, information theory, and approximation theory. It is the modern version of feature engineering, but with the feature map learned from data.

21.1.1 Learning goals

After this chapter, you should be able to:

formulate a representation as a measurable or differentiable map $\phi_\theta : \mathcal X \to \mathbb R^d$;
distinguish task-relevant variation from nuisance variation;
explain invariance, equivariance, sufficiency, separation, stability, and compression;
connect PCA, autoencoders, metric learning, and contrastive learning;
derive common representation-learning losses, including reconstruction, contrastive, triplet, InfoNCE, redundancy-reduction, and variance-regularized losses;
explain linear probing, transfer learning, domain shift, and feature evaluation;
implement small representation-learning examples in Python and TensorFlow;
use AI systems to test mathematical intuition without outsourcing the reasoning.

A useful mental model

A good representation $z=\phi_\theta(x)$ should be simple for the task and stable to irrelevant changes:

\[ \text{task information retained}, \qquad \text{nuisance information suppressed}, \qquad \text{geometry made useful}. \]

21.2 20.2 Images, features, and latent variables

Let an image be represented as a vector

\[ x \in \mathcal X \subseteq \mathbb R^p, \]

where $p=HWC$ for height $H$, width $W$, and channels $C$. In many vision problems, $p$ is large and the raw pixel coordinates are not statistically or geometrically convenient. A representation map produces

\[ z = \phi_\theta(x) \in \mathbb R^d. \]

Usually $d$ is much smaller than $p$, but this is not required. Feature maps inside convolutional neural networks may have the form

\[ \Phi_\theta(x) \in \mathbb R^{H' \times W' \times C'}, \]

and a global embedding may be produced later by pooling or by using a special token:

\[ z = \operatorname{pool}(\Phi_\theta(x)). \]

The representation is useful because a complicated decision boundary in pixel space may become simple in feature space.

For example, a classifier in pixel space might require a nonlinear rule

\[ y \approx g(x), \]

while the same task in representation space may be approximated by a linear classifier

\[ P(Y=k \mid X=x) = \frac{\exp(w_k^T\phi_\theta(x)+b_k)} {\sum_{\ell=1}^K \exp(w_\ell^T\phi_\theta(x)+b_\ell)}. \]

This is one reason linear probing is so common: if a frozen representation supports a strong linear classifier, then the representation has already organized much of the task structure.

21.2.1 Representation as a statistic

From a statistical viewpoint, $Z=\phi(X)$ is a statistic of the image $X$. If the task label is $Y$, then an ideal representation loses little information about $Y$:

\[ P(Y\mid X) \approx P(Y\mid Z). \]

A strong formal condition is sufficiency:

\[ Y \perp X \mid Z. \]

This says that once $Z$ is known, the remaining details of $X$ add no further information about $Y$. In practice, learned representations are approximate task-specific summaries rather than exact sufficient statistics.

21.2.2 Representation as geometry

A representation also defines a geometry on images. For two images $x_i,x_j$, define

\[ d_\phi(x_i,x_j)=\|\phi(x_i)-\phi(x_j)\|_2. \]

This distance may be very different from raw pixel distance. Two images of the same object under different lighting conditions may be far apart in pixel space but close in representation space. Two visually different categories may be pushed far apart.

The representation induces a kernel

\[ k_\phi(x_i,x_j)=\phi(x_i)^T\phi(x_j), \]

or a cosine similarity

\[ s_\phi(x_i,x_j) = \frac{\phi(x_i)^T\phi(x_j)}{\|\phi(x_i)\|_2\|\phi(x_j)\|_2}. \]

Thus representation learning can be viewed as learning a useful kernel or metric.

21.3 20.3 What should a representation do?

There is no single universal answer. The desired representation depends on the task. For classification, the representation should separate classes. For detection, it should also retain spatial information. For segmentation, it must preserve pixel-level structure. For 3D reconstruction, it should be compatible with geometry. For retrieval, distances in embedding space should correspond to semantic similarity.

Still, several mathematical properties appear repeatedly.

21.3.1 Separation

For classification, embeddings from different classes should be separated. If $Z_k$ denotes the feature distribution for class $k$, then a simple goal is that the class means

\[ \mu_k=\mathbb E[Z\mid Y=k] \]

are far apart relative to within-class variation

\[ \Sigma_k=\operatorname{Cov}(Z\mid Y=k). \]

A rough measure is the ratio

\[ \frac{\|\mu_a-\mu_b\|_2^2}{\operatorname{tr}(\Sigma_a)+\operatorname{tr}(\Sigma_b)}. \]

Large ratios suggest that linear or nearest-neighbor classifiers may work well.

21.3.2 Invariance

Suppose $g$ is a transformation that changes the image but not the label, such as a small crop, color jitter, or mild rotation. A representation is invariant to $g$ if

\[ \phi(gx)=\phi(x). \]

Exact invariance is often too strong. A more realistic condition is approximate invariance:

\[ \|\phi(gx)-\phi(x)\|_2 \leq \epsilon. \]

Augmentation-based self-supervised learning often begins from this idea. If two views $t_1(x)$ and $t_2(x)$ are generated by random augmentations, they should have nearby embeddings.

21.3.3 Equivariance

For localization tasks, we should not discard all geometric information. If an image shifts, the feature map should shift in a corresponding way. A map $\Phi$ is equivariant to a transformation $g$ if

\[ \Phi(gx)=\rho(g)\Phi(x), \]

where $\rho(g)$ is the corresponding action on feature space. Convolution is translation equivariant. Pooling may increase invariance while reducing spatial precision.

21.3.4 Stability

A representation should not amplify small perturbations too much. A Lipschitz condition is

\[ \|\phi(x)-\phi(x')\|_2 \leq L\|x-x'\|_2. \]

Smaller $L$ means greater stability. Stability is important for noise robustness, adversarial robustness, domain shift, and numerical conditioning.

21.3.5 Compression

A representation often compresses the image:

\[ \mathbb R^p \to \mathbb R^d, \qquad d \ll p. \]

Compression can reduce variance and improve generalization, but excessive compression may discard task-relevant information. This is the statistical bias-variance tradeoff in representation form.

21.3.6 Linearization

A representation can turn a nonlinear task into a nearly linear task. If the Bayes decision boundary is complicated in $x$ but simple in $z=\phi(x)$, then representation learning has reorganized the geometry of the problem.

Classification is not the only target

A representation that is excellent for classifying cats and dogs may be poor for estimating pose, detecting small objects, or segmenting boundaries. Representation quality is always relative to the task and to the distribution shift expected at deployment.

21.4 20.4 Interactive: feature maps and linear separability

The following interactive demo shows a simple version of why representations matter. In raw coordinates, two classes may be nonlinearly arranged. A learned map can bend the space so that a linear separator becomes effective.

Interactive: Raw space versus representation space

Use the slider to increase the strength of a nonlinear feature map. The classes become easier to separate by a line.

Representation strength:

21.5 20.5 Hand-designed features and learned features

Before deep learning, many successful vision systems used hand-designed features. Examples include:

local gradients and edge responses;
Harris and SIFT-like descriptors;
histograms of oriented gradients;
color histograms;
texture descriptors;
shape moments;
frequency-domain features.

A hand-designed feature map has the form

\[ \phi(x)= \begin{bmatrix} \phi_1(x) \\ \phi_2(x) \\ \vdots \\ \phi_d(x) \end{bmatrix}. \]

A learned feature map has the same mathematical role but its coordinates depend on parameters estimated from data:

\[ \phi_\theta(x)= \begin{bmatrix} \phi_{\theta,1}(x) \\ \phi_{\theta,2}(x) \\ \vdots \\ \phi_{\theta,d}(x) \end{bmatrix}. \]

The distinction is not that one is mathematical and the other is not. Both are functions. The distinction is how the function is chosen.

21.5.1 A useful decomposition

For supervised learning, write

\[ \widehat y = h_w(\phi_\theta(x)). \]

Then $\phi_\theta$ is the encoder or backbone, while $h_w$ is the head. In image classification, $h_w$ may be a linear softmax classifier. In detection, the head predicts boxes and classes. In segmentation, the head predicts a label for each pixel or patch.

This decomposition appears in many modern systems:

\[ \text{image} \to \text{backbone features} \to \text{task head}. \]

21.5.2 Linear probing

A common evaluation protocol freezes $\phi_\theta$ and trains only a linear classifier:

\[ \min_{W,b} \frac{1}{n}\sum_{i=1}^n \ell(W\phi_\theta(x_i)+b,y_i). \]

If this works well, the representation makes the classes linearly accessible. Fine-tuning allows both $\theta$ and $(W,b)$ to change:

\[ \min_{\theta,W,b} \frac{1}{n}\sum_{i=1}^n \ell(W\phi_\theta(x_i)+b,y_i). \]

Linear probing tests representation quality. Fine-tuning tests adaptability.

21.6 20.6 PCA as the first representation-learning model

PCA is a linear representation-learning method. Given centered data vectors $x_i\in\mathbb R^p$, PCA seeks an orthonormal matrix

\[ U_d=[u_1,\dots,u_d]\in\mathbb R^{p\times d} \]

and represents each image by

\[ z_i=U_d^T x_i\in\mathbb R^d. \]

The reconstruction is

\[ \widehat x_i=U_d z_i=U_dU_d^T x_i. \]

PCA solves two equivalent problems:

\[ \max_{U^TU=I_d}\operatorname{tr}(U^T S U), \]

where $S$ is the sample covariance, and

\[ \min_{U^TU=I_d}\sum_{i=1}^n \|x_i-UU^Tx_i\|_2^2. \]

Thus PCA learns a representation by preserving variance and minimizing squared reconstruction error.

21.6.1 PCA is not necessarily semantic

PCA is mathematically clean, but it is unsupervised and linear. High-variance directions need not be task-relevant. For instance, lighting variation may dominate object identity. This motivates nonlinear and task-aware representations.

21.7 20.7 Autoencoders as nonlinear PCA

An autoencoder consists of an encoder and decoder:

\[ z=f_\theta(x), \qquad \widehat x=g_\psi(z). \]

It is trained by minimizing reconstruction loss:

\[ \min_{\theta,\psi} \frac{1}{n}\sum_{i=1}^n \|x_i-g_\psi(f_\theta(x_i))\|_2^2. \]

If $f$ and $g$ are linear maps and the bottleneck has dimension $d$, the optimal subspace is closely related to PCA. With nonlinear neural networks, the autoencoder can learn curved low-dimensional structure.

21.7.1 Denoising autoencoders

A denoising autoencoder corrupts the input and reconstructs the clean image:

\[ \widetilde x = x + \epsilon, \qquad \min_{\theta,\psi} \mathbb E\|x-g_\psi(f_\theta(\widetilde x))\|_2^2. \]

This encourages the representation to ignore nuisance noise.

21.7.2 Variational autoencoders

A variational autoencoder treats the latent variable probabilistically:

\[ z \sim p(z), \qquad x\sim p_\psi(x\mid z), \]

with an approximate posterior $q_\theta(z\mid x)$. The objective is the evidence lower bound:

\[ \mathcal L(\theta,\psi) = \mathbb E_{q_\theta(z\mid x)}[\log p_\psi(x\mid z)] - D_{\mathrm{KL}}(q_\theta(z\mid x)\|p(z)). \]

The VAE latent representation is shaped both by reconstruction and by a prior distribution.

21.8 20.8 Metric learning

Some vision tasks are not naturally expressed as fixed-label classification. For retrieval, face recognition, product matching, landmark recognition, and image deduplication, the output is often a similarity score. We want embeddings such that similar images are close and dissimilar images are far apart.

Let

\[ z_i=\phi_\theta(x_i). \]

A metric-learning objective uses pairs or triplets.

21.8.1 Contrastive pair loss

For a pair $(x_i,x_j)$ with label $s_{ij}=1$ if the pair is similar and $s_{ij}=0$ otherwise, define $d_{ij}=\|z_i-z_j\|_2$. A classical contrastive loss is

\[ \mathcal L_{ij} = s_{ij}d_{ij}^2 + (1-s_{ij})\max(0,m-d_{ij})^2, \]

where $m>0$ is a margin. Positive pairs are pulled together; negative pairs are pushed at least margin $m$ apart.

21.8.2 Triplet loss

A triplet consists of an anchor $a$, positive example $p$, and negative example $n$. The triplet loss is

\[ \mathcal L = \max\bigl(0, \|z_a-z_p\|_2^2 - \|z_a-z_n\|_2^2 +m \bigr). \]

The loss is zero when the positive is closer to the anchor than the negative by at least margin $m$.

21.8.3 Normalized embeddings and angular geometry

Many modern retrieval systems use normalized embeddings:

\[ \bar z = \frac{z}{\|z\|_2}. \]

Then Euclidean distance and cosine similarity are directly related:

\[ \|\bar z_i-\bar z_j\|_2^2 =2-2\bar z_i^T\bar z_j. \]

Thus learning on the unit sphere is learning angular separation.

21.9 20.9 Interactive: triplet loss geometry

Interactive: Triplet loss margin

The anchor and positive are fixed. Move the negative point and change the margin. The triplet loss is positive when the negative is not sufficiently farther away.

Margin: Negative x-coordinate:

21.10 20.10 Contrastive learning from augmentations

Self-supervised contrastive learning removes the need for explicit labels. Instead, it creates positive pairs from different augmented views of the same image.

Let $t_1,t_2$ be random augmentations. For an image $x_i$, define

\[ x_i^{(1)}=t_1(x_i), \qquad x_i^{(2)}=t_2(x_i). \]

The two views form a positive pair. Views from different images are usually treated as negatives.

Let the encoder and projection head produce normalized vectors

\[ q_i = \frac{g(\phi(x_i^{(1)}))}{\|g(\phi(x_i^{(1)}))\|_2}, \qquad k_j = \frac{g(\phi(x_j^{(2)}))}{\|g(\phi(x_j^{(2)}))\|_2}. \]

A common loss for query $q_i$ is the InfoNCE loss:

\[ \mathcal L_i = - \log \frac{\exp(q_i^T k_i/\tau)} {\sum_{j=1}^B \exp(q_i^T k_j/\tau)}. \]

Here $B$ is the batch size and $\tau>0$ is the temperature.

21.10.1 Temperature

The temperature controls how sharply the softmax emphasizes the most similar examples. Small $\tau$ produces a sharper distribution and larger gradients for hard negatives. Large $\tau$ smooths the distribution.

21.10.2 Alignment and uniformity

Contrastive learning balances two forces:

Alignment: positive pairs should be close.
Uniformity: embeddings should spread out on the representation sphere.

A simplified alignment term is

\[ \mathbb E\|z-z^+\|_2^2. \]

A simplified uniformity term is

\[ \log \mathbb E_{i\neq j}\exp(-2\|z_i-z_j\|_2^2). \]

Good representations are not produced by collapse, where all embeddings are identical. Contrastive negatives help prevent collapse.

21.11 20.11 Interactive: InfoNCE temperature

Interactive: temperature in contrastive learning

The positive similarity is fixed. The negative similarities vary. Smaller temperature makes the softmax distribution sharper.

Temperature $\tau$:

21.12 20.12 Collapse and non-contrastive self-supervision

Contrastive learning uses negative examples to avoid collapse. Other methods avoid collapse without explicit negatives by architectural or statistical constraints.

21.12.1 Predictor asymmetry

In BYOL-style methods, two networks produce views of the same image. One branch includes a predictor. The method encourages one representation to predict the other:

\[ \mathcal L = \left\| \frac{q_\theta(z_1)}{\|q_\theta(z_1)\|_2} - \frac{z_2'}{\|z_2'\|_2} \right\|_2^2, \]

where the target branch is often updated by exponential moving average rather than direct gradient descent. The asymmetry helps avoid trivial collapse.

21.12.2 Redundancy reduction

Barlow Twins-style methods use a cross-correlation matrix between two augmented views. Suppose $Z^A,Z^B\in\mathbb R^{B\times d}$ are normalized feature matrices. The empirical cross-correlation is

\[ C_{ij}=\frac{1}{B}\sum_{b=1}^B Z^A_{b i}Z^B_{b j}. \]

The objective encourages

\[ C \approx I_d. \]

A typical loss is

\[ \mathcal L = \sum_i (1-C_{ii})^2 + \lambda\sum_{i\neq j} C_{ij}^2. \]

The diagonal term aligns the two views coordinatewise. The off-diagonal term discourages redundant coordinates.

21.12.3 Variance-invariance-covariance regularization

VICReg-style methods combine three terms:

\[ \mathcal L = \lambda\mathcal L_{\text{inv}} + \mu\mathcal L_{\text{var}} + \nu\mathcal L_{\text{cov}}. \]

The invariance term aligns positive pairs. The variance term prevents collapse by requiring each coordinate to have nontrivial standard deviation. The covariance term discourages redundant dimensions.

21.13 20.13 Masked image modeling

Another self-supervised strategy hides part of the input and asks the model to reconstruct it. Let $M$ denote a random mask and $x_M$ the visible part of the image. A masked modeling objective has the form

\[ \min_{\theta,\psi} \mathbb E_{x,M} \left[ \ell\left(D_\psi(E_\theta(x_M)), x_{\bar M}\right)\right], \]

where $E_\theta$ is an encoder, $D_\psi$ is a decoder, and $x_{\bar M}$ is the hidden part.

This objective is natural for vision transformers because an image can be divided into patches. A large fraction of patches can be hidden, and the model must infer missing visual structure from context.

21.13.1 Reconstruction target

The target may be pixels, normalized patches, discrete visual tokens, low-level features, or high-level features. Pixel reconstruction encourages local detail; feature reconstruction may encourage semantic abstraction.

21.13.2 Statistical interpretation

Masked modeling estimates conditional structure:

\[ p(x_{\bar M}\mid x_M). \]

Thus the representation must encode information that predicts missing content.

21.14 20.14 Representations and nuisance variables

Many visual changes are not relevant to a given task. Let a generative description be

\[ X = G(Y, N), \]

where $Y$ is the task-relevant variable and $N$ is a nuisance variable such as lighting, viewpoint, background, pose, or sensor noise. A representation for predicting $Y$ should preserve information about $Y$ and suppress information about $N$.

A useful idealization is

\[ \phi(G(Y,N_1)) \approx \phi(G(Y,N_2)) \]

for nuisance values $N_1,N_2$ that do not change the task label.

21.14.1 Data augmentation as nuisance sampling

If augmentations mimic nuisance variation, then training with augmentations encourages invariance:

\[ \phi(t(x)) \approx \phi(x). \]

However, augmentations must be chosen carefully. A horizontal flip may preserve labels for natural-image classification but may change labels in medical imaging, traffic signs, or text recognition.

Augmentation is a modeling assumption

Every augmentation says: “This transformation should not change the meaning needed for the task.” This is a statistical assumption, not a free trick.

21.15 20.15 Interactive: invariance versus information loss

Interactive: invariance can help or hurt

The representation suppresses a nuisance direction. If the suppressed direction is truly irrelevant, classification improves. If it contains label information, the representation fails.

Strength of suppression: Task direction:

21.16 20.16 The geometry of embedding spaces

Embedding spaces are often high-dimensional. Some geometric facts are important.

21.16.1 Concentration of distances

If random vectors are uniformly distributed on a high-dimensional sphere, pairwise angles tend to concentrate near $90^\circ$. This means nearest-neighbor intuition from two-dimensional geometry can be misleading.

For normalized vectors $z_i,z_j\in S^{d-1}$,

\[ \mathbb E[z_i^Tz_j]=0, \qquad \operatorname{Var}(z_i^Tz_j)=\frac{1}{d}. \]

As $d$ increases, random dot products concentrate near zero. Therefore a cosine similarity of $0.3$ may be highly meaningful in a large embedding dimension.

21.16.2 Clustering and manifolds

A representation may organize data near lower-dimensional manifolds. A class may occupy a cluster, a curved submanifold, or multiple modes. For example, images of the same object under different viewpoints may trace a viewpoint manifold.

21.16.3 Dimensionality reduction for visualization

Methods such as PCA, t-SNE, and UMAP are often used to visualize embeddings. These tools are useful but can mislead.

PCA preserves global linear variance. t-SNE emphasizes local neighborhoods and often exaggerates cluster separation. UMAP builds a neighborhood graph and tries to preserve fuzzy topological structure. A two-dimensional plot is an exploratory diagnostic, not proof of class separation.

21.17 20.17 Interactive: PCA view of learned embeddings

Interactive: projection can hide or reveal structure

A high-dimensional representation may have class information in a direction that is not the largest-variance direction. Rotate the projection to see how visualization depends on the chosen coordinates.

Projection angle:

21.18 20.18 Supervised representation learning

In supervised learning, the representation is optimized directly for a task loss:

\[ \min_{\theta,w} \frac{1}{n}\sum_{i=1}^n \ell(h_w(\phi_\theta(x_i)),y_i) + \lambda R(\theta,w). \]

The representation is learned to support the supervised head.

21.18.1 Penultimate-layer embeddings

In a neural classifier, the vector before the final linear layer is often used as an embedding:

\[ z=\phi_\theta(x), \qquad \text{logits}=Wz+b. \]

If the classifier is trained with cross-entropy, embeddings tend to arrange themselves so that class-specific linear scores separate the data.

21.18.2 Center loss and class compactness

One can explicitly encourage embeddings of the same class to be close. Let $c_y$ be a learned center for class $y$. Center loss adds

\[ \mathcal L_{\text{center}} = \sum_{i=1}^n \|z_i-c_{y_i}\|_2^2. \]

This reduces within-class scatter but must be balanced with class separation.

21.18.3 Angular margin losses

For normalized features and weights, classification can be based on angles. If

\[ \|z\|_2=1, \qquad \|w_k\|_2=1, \]

then the logit is proportional to

\[ w_k^Tz=\cos\theta_k. \]

Angular-margin losses improve separation by requiring a margin in angle, useful in face recognition and retrieval.

21.19 20.19 Transfer learning and domain shift

A representation learned on a large source dataset may transfer to a smaller target dataset. Let

\[ P_S(X,Y) \]

be the source distribution and

\[ P_T(X,Y) \]

be the target distribution. Transfer works when the learned features capture structure useful under $P_T$.

21.19.1 Frozen features

The simplest transfer method freezes the encoder and trains a new head:

\[ \min_w \frac{1}{n_T}\sum_{i=1}^{n_T} \ell(h_w(\phi_{\theta_S}(x_i^T)),y_i^T). \]

This is statistically attractive when the target dataset is small.

21.19.2 Fine-tuning

Fine-tuning updates the encoder:

\[ \min_{\theta,w} \frac{1}{n_T}\sum_{i=1}^{n_T} \ell(h_w(\phi_\theta(x_i^T)),y_i^T). \]

Fine-tuning can adapt to the target domain but risks overfitting and catastrophic forgetting.

21.19.3 Domain shift

Representation transfer can fail when source and target differ strongly:

\[ P_S(X)\neq P_T(X), \qquad P_S(Y\mid X)\neq P_T(Y\mid X). \]

Medical images, satellite images, industrial images, and scientific images often differ from natural-image pretraining data. For statistics students, this is a covariate-shift and model-misspecification problem, not just a software problem.

21.20 20.20 Foundation representations

Modern vision often uses very large pretrained encoders. These may be trained by supervised classification, image-text contrastive learning, masked image modeling, or combinations of many objectives. The resulting representation may be reused for many downstream tasks.

21.20.1 Image-text representations

A vision-language model learns image and text encoders

\[ z_I=\phi_I(x), \qquad z_T=\phi_T(t), \]

and aligns paired image-text examples through a contrastive objective:

\[ \mathcal L_i = - \log \frac{\exp(z_{I,i}^Tz_{T,i}/\tau)} {\sum_j \exp(z_{I,i}^Tz_{T,j}/\tau)}. \]

This makes text prompts usable as classifier weights or retrieval queries.

21.20.2 Representation as interface

A foundation representation is an interface between data and tasks. The same embedding can support retrieval, classification, captioning, detection, segmentation, or visual question answering if paired with suitable heads and prompts.

21.21 20.21 Statistical evaluation of representations

A representation should be evaluated with clear statistical protocols.

21.21.1 Linear probing

Train a linear classifier on frozen embeddings. This asks: is the target information linearly accessible?

21.21.2 $k$-nearest-neighbor evaluation

Classify by nearest neighbors in embedding space:

\[ \widehat y(x)=\operatorname{mode}\{y_i: i\in N_k(\phi(x))\}. \]

This tests whether local geometry is meaningful.

21.21.3 Retrieval metrics

For retrieval, common metrics include precision at $k$, recall at $k$, mean average precision, and normalized discounted cumulative gain.

21.21.4 Clustering metrics

Unsupervised representation quality may be evaluated by clustering and then comparing clusters to labels using adjusted Rand index, normalized mutual information, or purity. These metrics must be interpreted cautiously because clustering assumptions may not match the representation geometry.

21.21.5 Calibration and uncertainty

If a representation supports probabilistic predictions, evaluate calibration, Brier score, negative log-likelihood, and reliability diagrams. A strong embedding with an overconfident head can still be statistically unsafe.

21.22 20.22 Python example: PCA and logistic regression on image features

The following example shows the classical representation-learning workflow: turn images into vectors, learn a lower-dimensional representation by PCA, and train a simple classifier.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score, confusion_matrix

# Small grayscale image dataset: 8 x 8 digit images.
digits = load_digits()
X = digits.data / 16.0
y = digits.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=7, stratify=y
)

model = Pipeline([
    ("pca", PCA(n_components=20, whiten=True, random_state=7)),
    ("clf", LogisticRegression(max_iter=3000, multi_class="auto"))
])

model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print("Test accuracy:", accuracy_score(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))

pca = model.named_steps["pca"]
plt.plot(np.cumsum(pca.explained_variance_ratio_), marker="o")
plt.xlabel("number of PCA components")
plt.ylabel("cumulative explained variance")
plt.title("PCA representation dimension")
plt.grid(True)
plt.show()

The important statistical point is that PCA is unsupervised. It may preserve high-variance directions that are not the best directions for classification.

21.23 20.23 Python example: visualizing embeddings

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)
X = StandardScaler().fit_transform(X)

Z_pca = PCA(n_components=2, random_state=7).fit_transform(X)

# t-SNE is useful for visualization but not for proving cluster structure.
Z_tsne = TSNE(
    n_components=2,
    perplexity=30,
    learning_rate="auto",
    init="pca",
    random_state=7
).fit_transform(X)

plt.figure(figsize=(6, 5))
plt.scatter(Z_pca[:, 0], Z_pca[:, 1], c=y, s=12, cmap="tab10")
plt.title("PCA visualization of raw image vectors")
plt.colorbar()
plt.show()

plt.figure(figsize=(6, 5))
plt.scatter(Z_tsne[:, 0], Z_tsne[:, 1], c=y, s=12, cmap="tab10")
plt.title("t-SNE visualization of raw image vectors")
plt.colorbar()
plt.show()

When using such plots, ask: which neighborhoods are trustworthy, which distances are distorted, and which structures are artifacts of the visualization algorithm?

21.24 20.24 TensorFlow example: a small supervised representation encoder

import tensorflow as tf
from tensorflow.keras import layers, models

# Example input: 32 x 32 RGB image.
inputs = layers.Input(shape=(32, 32, 3))

x = layers.Conv2D(32, 3, padding="same", activation="relu")(inputs)
x = layers.MaxPooling2D()(x)
x = layers.Conv2D(64, 3, padding="same", activation="relu")(x)
x = layers.MaxPooling2D()(x)
x = layers.Conv2D(128, 3, padding="same", activation="relu")(x)
x = layers.GlobalAveragePooling2D()(x)

# This is the learned representation.
embedding = layers.Dense(128, activation=None, name="embedding")(x)
embedding_norm = tf.nn.l2_normalize(embedding, axis=1)

outputs = layers.Dense(10, activation="softmax")(embedding_norm)
model = models.Model(inputs, outputs)

model.compile(
    optimizer=tf.keras.optimizers.Adam(1e-3),
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

encoder = models.Model(inputs, embedding_norm)
model.summary()

The encoder model maps images to normalized vectors. These vectors can be used for nearest-neighbor retrieval, clustering, linear probing, or transfer learning.

21.25 20.25 TensorFlow example: InfoNCE loss

The following code sketches the core of an image contrastive loss. A real implementation also needs strong data augmentation and careful batching.

import tensorflow as tf


def l2_normalize(x, axis=1, eps=1e-8):
    return x / (tf.norm(x, axis=axis, keepdims=True) + eps)


def info_nce_loss(z1, z2, temperature=0.2):
    """
    z1 and z2 are embeddings of two augmented views of the same batch.
    Shape: (batch_size, embedding_dim).
    Positive pairs are (z1[i], z2[i]).
    Other examples in the batch serve as negatives.
    """
    z1 = l2_normalize(z1)
    z2 = l2_normalize(z2)

    logits = tf.matmul(z1, z2, transpose_b=True) / temperature
    labels = tf.range(tf.shape(z1)[0])

    loss_12 = tf.keras.losses.sparse_categorical_crossentropy(
        labels, logits, from_logits=True
    )
    loss_21 = tf.keras.losses.sparse_categorical_crossentropy(
        labels, tf.transpose(logits), from_logits=True
    )
    return tf.reduce_mean((loss_12 + loss_21) / 2.0)

# Example usage with dummy embeddings.
z1 = tf.random.normal((64, 128))
z2 = tf.random.normal((64, 128))
print(info_nce_loss(z1, z2, temperature=0.2))

21.26 20.26 AI-assisted study components

Use an AI assistant as a mathematical critic and experimental partner, not as a replacement for derivation.

21.26.1 Prompt 1: representation properties

I am studying representation learning for computer vision. Given a task such as image classification, segmentation, or retrieval, list the desired mathematical properties of the representation. For each property, state whether it is an invariance, equivariance, stability, separation, or compression property.

21.26.2 Prompt 2: critique an augmentation

I plan to use random crops, horizontal flips, color jitter, and Gaussian blur for a medical imaging dataset. Analyze which augmentations may violate label preservation. Explain the statistical assumption behind each augmentation.

21.26.3 Prompt 3: compare losses

Compare cross-entropy, triplet loss, InfoNCE, Barlow Twins loss, and masked reconstruction loss. For each, identify the positive signal, the anti-collapse mechanism, and the geometry encouraged in embedding space.

21.26.4 Prompt 4: derivation checker

Verify the relationship $\|u-v\|_2^2=2-2u^Tv$ for unit vectors. Then explain why cosine similarity and Euclidean distance are equivalent for normalized embeddings.

21.26.5 Prompt 5: experimental design

Design an experiment to compare frozen features, linear probing, fine-tuning, and nearest-neighbor retrieval for a small image dataset. Include train/test splits, metrics, and possible sources of leakage.

21.27 20.27 Common pitfalls

21.27.1 Pitfall 1: treating embeddings as automatically meaningful

A vector representation is not meaningful merely because it was produced by a neural network. Its meaning is defined by training data, objective, architecture, and evaluation protocol.

21.27.2 Pitfall 2: overinterpreting two-dimensional visualizations

PCA, t-SNE, and UMAP plots are diagnostic tools. They do not prove that a representation is statistically reliable.

21.27.3 Pitfall 3: using augmentations without checking label preservation

Augmentation is a mathematical assumption about invariance. Incorrect augmentations can teach the model to ignore information needed for the task.

21.27.4 Pitfall 4: confusing linear separability with causal understanding

A representation may support strong classification while relying on shortcuts such as background, watermark, scanner type, or dataset artifacts.

21.27.5 Pitfall 5: ignoring calibration

Good representations can still produce badly calibrated probabilities when paired with overconfident heads.

21.28 20.28 Summary

Representation learning studies maps

\[ \phi_\theta : \mathcal X \to \mathbb R^d \]

that transform images into useful coordinates. Good representations preserve task-relevant information, suppress nuisance variation, stabilize noise, organize geometry, and make downstream tasks simpler.

The mathematical viewpoints are complementary:

Linear algebra: representations are coordinate systems, projections, embeddings, and kernels.
Statistics: representations are statistics of data, with sufficiency, bias-variance tradeoff, and domain-shift concerns.
Optimization: representations are learned by minimizing empirical losses.
Geometry: representations define distances, angles, manifolds, and clusters.
Information theory: representations balance compression and task information.
AI: representations are the reusable core of modern vision foundation models.

The next chapters study the architectures and training principles that produce many of these representations in practice.

21.29 Exercises

21.29.1 Conceptual exercises

Explain why raw pixel distance is often a poor measure of semantic similarity.
Give three examples of nuisance variation in image classification. For each, state an augmentation that might encourage invariance.
Explain the difference between invariance and equivariance in the context of object detection.
Why is linear probing a useful but incomplete evaluation of a representation?
Explain why a representation learned for classification may not be ideal for segmentation.

21.29.2 Mathematical exercises

Let $u,v\in\mathbb R^d$ satisfy $\|u\|_2=\|v\|_2=1$. Prove that $\|u-v\|_2^2=2-2u^Tv$.
Derive the gradient of the pairwise contrastive loss with respect to $z_i$ for a positive pair.
For the triplet loss, identify the region in which the gradient is zero.
Suppose $Z\mid Y=k\sim N(\mu_k,\Sigma)$ with common covariance. Show that the Bayes classifier is linear in $Z$.
Let $C$ be the cross-correlation matrix in a redundancy-reduction loss. Explain why forcing $C\approx I$ discourages both collapse and redundancy.
Show that PCA minimizes squared reconstruction error among all $d$-dimensional linear subspaces.
If $z_i,z_j$ are independent uniform unit vectors in $\mathbb R^d$, compute $\mathbb E[z_i^Tz_j]$ and explain why similarities concentrate near zero as $d$ grows.

21.29.3 Python exercises

Use the digits dataset to compare logistic regression on raw pixels, PCA features, and HOG-style features.
Train a small autoencoder on grayscale images and visualize the two-dimensional latent space.
Implement triplet loss for synthetic two-dimensional data and visualize the learned embedding.
Implement the InfoNCE loss and test the effect of temperature on gradient magnitudes.
Extract embeddings from a pretrained TensorFlow or PyTorch image model and evaluate $k$-nearest-neighbor classification.
Use PCA and t-SNE to visualize the same embedding. Write a short note explaining what each plot preserves and distorts.

21.29.4 AI-assisted exercises

Ask an AI assistant to propose augmentations for a dataset. Then critique each augmentation mathematically as a label-preservation assumption.
Ask an AI assistant to explain InfoNCE. Identify any missing assumptions about negatives, temperature, normalization, or collapse.
Ask an AI assistant to compare PCA, autoencoders, and contrastive learning. Rewrite the answer using precise notation.
Ask an AI assistant to design a representation-evaluation experiment. Add missing statistical safeguards such as train/test splits, calibration, and confidence intervals.

21.30 Notes and references

Representation learning is a broad field connected to classical feature extraction, statistical learning, neural networks, and modern self-supervised learning. General representation-learning perspectives appear in Bengio et al. (2013). Metric learning and Siamese-style ideas are represented by Hadsell et al. (2006), Schroff et al. (2015), and Weinberger and Saul (2009). Contrastive and self-supervised learning are represented by Oord et al. (2018), Chen et al. (2020), He et al. (2020), Grill et al. (2020), Zbontar et al. (2021), Bardes et al. (2022), and Caron et al. (2021). Masked image modeling is represented by He et al. (2022). Classical dimensionality-reduction references include Pearson (1901), Hotelling (1933), Maaten and Hinton (2008), and McInnes et al. (2018).

Bardes, Adrien, Jean Ponce, and Yann LeCun. 2022. “VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning.” International Conference on Learning Representations.

Bengio, Yoshua, Aaron Courville, and Pascal Vincent. 2013. “Representation Learning: A Review and New Perspectives.” IEEE Transactions on Pattern Analysis and Machine Intelligence 35 (8): 1798–828.

Caron, Mathilde, Hugo Touvron, Ishan Misra, et al. 2021. “Emerging Properties in Self-Supervised Vision Transformers.” IEEE/CVF International Conference on Computer Vision, 9650–60.

Chen, Ting, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. “A Simple Framework for Contrastive Learning of Visual Representations.” International Conference on Machine Learning, 1597–607.

Grill, Jean-Bastien, Florian Strub, Florent Altché, et al. 2020. “Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning.” Advances in Neural Information Processing Systems 33: 21271–84.

Hadsell, Raia, Sumit Chopra, and Yann LeCun. 2006. “Dimensionality Reduction by Learning an Invariant Mapping.” IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1735–42.

He, Kaiming, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. “Masked Autoencoders Are Scalable Vision Learners.” IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16000–16009.

He, Kaiming, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. “Momentum Contrast for Unsupervised Visual Representation Learning.” IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9729–38.

Hotelling, Harold. 1933. “Analysis of a Complex of Statistical Variables into Principal Components.” Journal of Educational Psychology 24 (6): 417–41.

Maaten, Laurens van der, and Geoffrey Hinton. 2008. “Visualizing Data Using t-SNE.” Journal of Machine Learning Research 9: 2579–605.

McInnes, Leland, John Healy, and James Melville. 2018. “UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction.” arXiv Preprint arXiv:1802.03426.

Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. 2018. “Representation Learning with Contrastive Predictive Coding.” arXiv Preprint arXiv:1807.03748.

Pearson, Karl. 1901. “On Lines and Planes of Closest Fit to Systems of Points in Space.” Philosophical Magazine 2 (11): 559–72.

Schroff, Florian, Dmitry Kalenichenko, and James Philbin. 2015. “FaceNet: A Unified Embedding for Face Recognition and Clustering.” IEEE Conference on Computer Vision and Pattern Recognition, 815–23.

Weinberger, Kilian Q., and Lawrence K. Saul. 2009. “Distance Metric Learning for Large Margin Nearest Neighbor Classification.” Journal of Machine Learning Research 10: 207–44.

Zbontar, Jure, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. 2021. “Barlow Twins: Self-Supervised Learning via Redundancy Reduction.” International Conference on Machine Learning, 12310–20.

--- title: "Chapter 20. Representation Learning" subtitle: "Feature Maps, Embeddings, Metric Geometry, and Self-Supervised Vision" format: html: toc: true toc-depth: 3 execute: warning: false message: false --- ## 20.1 Why representation learning belongs in a mathematical vision book Classical computer vision often begins with a hand-designed transformation: $$ I \longmapsto \text{edges}, \qquad I \longmapsto \text{corners}, \qquad I \longmapsto \text{SIFT descriptors}, \qquad I \longmapsto \text{histograms}. $$ Modern computer vision replaces many hand-designed transformations by a learned map $$ \phi_\theta : \mathcal X \to \mathbb R^d, $$ where $\mathcal X$ is an image space and $\phi_\theta(x)$ is a vector representation, embedding, feature vector, descriptor, or latent code. The downstream task then uses a simpler decision rule: $$ \widehat y = h_w(\phi_\theta(x)). $$ The central mathematical question is not merely how to train a large neural network. The central question is: > What properties should a representation preserve, discard, linearize, separate, and stabilize? For MA Applied Math and MS Statistics students, representation learning is best understood as the meeting point of linear algebra, statistics, optimization, geometry, information theory, and approximation theory. It is the modern version of feature engineering, but with the feature map learned from data. ### Learning goals After this chapter, you should be able to: 1. formulate a representation as a measurable or differentiable map $\phi_\theta : \mathcal X \to \mathbb R^d$; 2. distinguish task-relevant variation from nuisance variation; 3. explain invariance, equivariance, sufficiency, separation, stability, and compression; 4. connect PCA, autoencoders, metric learning, and contrastive learning; 5. derive common representation-learning losses, including reconstruction, contrastive, triplet, InfoNCE, redundancy-reduction, and variance-regularized losses; 6. explain linear probing, transfer learning, domain shift, and feature evaluation; 7. implement small representation-learning examples in Python and TensorFlow; 8. use AI systems to test mathematical intuition without outsourcing the reasoning. ::: {.callout-note} ## A useful mental model A good representation $z=\phi_\theta(x)$ should be **simple for the task** and **stable to irrelevant changes**: $$ \text{task information retained}, \qquad \text{nuisance information suppressed}, \qquad \text{geometry made useful}. $$ ::: ## 20.2 Images, features, and latent variables Let an image be represented as a vector $$ x \in \mathcal X \subseteq \mathbb R^p, $$ where $p=HWC$ for height $H$, width $W$, and channels $C$. In many vision problems, $p$ is large and the raw pixel coordinates are not statistically or geometrically convenient. A representation map produces $$ z = \phi_\theta(x) \in \mathbb R^d. $$ Usually $d$ is much smaller than $p$, but this is not required. Feature maps inside convolutional neural networks may have the form $$ \Phi_\theta(x) \in \mathbb R^{H' \times W' \times C'}, $$ and a global embedding may be produced later by pooling or by using a special token: $$ z = \operatorname{pool}(\Phi_\theta(x)). $$ The representation is useful because a complicated decision boundary in pixel space may become simple in feature space. For example, a classifier in pixel space might require a nonlinear rule $$ y \approx g(x), $$ while the same task in representation space may be approximated by a linear classifier $$ P(Y=k \mid X=x) = \frac{\exp(w_k^T\phi_\theta(x)+b_k)} {\sum_{\ell=1}^K \exp(w_\ell^T\phi_\theta(x)+b_\ell)}. $$ This is one reason linear probing is so common: if a frozen representation supports a strong linear classifier, then the representation has already organized much of the task structure. ### Representation as a statistic From a statistical viewpoint, $Z=\phi(X)$ is a statistic of the image $X$. If the task label is $Y$, then an ideal representation loses little information about $Y$: $$ P(Y\mid X) \approx P(Y\mid Z). $$ A strong formal condition is **sufficiency**: $$ Y \perp X \mid Z. $$ This says that once $Z$ is known, the remaining details of $X$ add no further information about $Y$. In practice, learned representations are approximate task-specific summaries rather than exact sufficient statistics. ### Representation as geometry A representation also defines a geometry on images. For two images $x_i,x_j$, define $$ d_\phi(x_i,x_j)=\|\phi(x_i)-\phi(x_j)\|_2. $$ This distance may be very different from raw pixel distance. Two images of the same object under different lighting conditions may be far apart in pixel space but close in representation space. Two visually different categories may be pushed far apart. The representation induces a kernel $$ k_\phi(x_i,x_j)=\phi(x_i)^T\phi(x_j), $$ or a cosine similarity $$ s_\phi(x_i,x_j) = \frac{\phi(x_i)^T\phi(x_j)}{\|\phi(x_i)\|_2\|\phi(x_j)\|_2}. $$ Thus representation learning can be viewed as learning a useful kernel or metric. ## 20.3 What should a representation do? There is no single universal answer. The desired representation depends on the task. For classification, the representation should separate classes. For detection, it should also retain spatial information. For segmentation, it must preserve pixel-level structure. For 3D reconstruction, it should be compatible with geometry. For retrieval, distances in embedding space should correspond to semantic similarity. Still, several mathematical properties appear repeatedly. ### Separation For classification, embeddings from different classes should be separated. If $Z_k$ denotes the feature distribution for class $k$, then a simple goal is that the class means $$ \mu_k=\mathbb E[Z\mid Y=k] $$ are far apart relative to within-class variation $$ \Sigma_k=\operatorname{Cov}(Z\mid Y=k). $$ A rough measure is the ratio $$ \frac{\|\mu_a-\mu_b\|_2^2}{\operatorname{tr}(\Sigma_a)+\operatorname{tr}(\Sigma_b)}. $$ Large ratios suggest that linear or nearest-neighbor classifiers may work well. ### Invariance Suppose $g$ is a transformation that changes the image but not the label, such as a small crop, color jitter, or mild rotation. A representation is invariant to $g$ if $$ \phi(gx)=\phi(x). $$ Exact invariance is often too strong. A more realistic condition is approximate invariance: $$ \|\phi(gx)-\phi(x)\|_2 \leq \epsilon. $$ Augmentation-based self-supervised learning often begins from this idea. If two views $t_1(x)$ and $t_2(x)$ are generated by random augmentations, they should have nearby embeddings. ### Equivariance For localization tasks, we should not discard all geometric information. If an image shifts, the feature map should shift in a corresponding way. A map $\Phi$ is equivariant to a transformation $g$ if $$ \Phi(gx)=\rho(g)\Phi(x), $$ where $\rho(g)$ is the corresponding action on feature space. Convolution is translation equivariant. Pooling may increase invariance while reducing spatial precision. ### Stability A representation should not amplify small perturbations too much. A Lipschitz condition is $$ \|\phi(x)-\phi(x')\|_2 \leq L\|x-x'\|_2. $$ Smaller $L$ means greater stability. Stability is important for noise robustness, adversarial robustness, domain shift, and numerical conditioning. ### Compression A representation often compresses the image: $$ \mathbb R^p \to \mathbb R^d, \qquad d \ll p. $$ Compression can reduce variance and improve generalization, but excessive compression may discard task-relevant information. This is the statistical bias-variance tradeoff in representation form. ### Linearization A representation can turn a nonlinear task into a nearly linear task. If the Bayes decision boundary is complicated in $x$ but simple in $z=\phi(x)$, then representation learning has reorganized the geometry of the problem. ::: {.callout-important} ## Classification is not the only target A representation that is excellent for classifying cats and dogs may be poor for estimating pose, detecting small objects, or segmenting boundaries. Representation quality is always relative to the task and to the distribution shift expected at deployment. ::: ## 20.4 Interactive: feature maps and linear separability The following interactive demo shows a simple version of why representations matter. In raw coordinates, two classes may be nonlinearly arranged. A learned map can bend the space so that a linear separator becomes effective. <div class="interactive-card"> <h3>Interactive: Raw space versus representation space</h3> <p>Use the slider to increase the strength of a nonlinear feature map. The classes become easier to separate by a line.</p> <label for="rep-warp">Representation strength:</label> <input id="rep-warp" type="range" min="0" max="1" step="0.05" value="0.65" style="width: 100%;"> <div id="rep-map-demo" class="plotly-demo"></div> </div> <script> (function(){ function ready(fn){ if(document.readyState !== 'loading') fn(); else document.addEventListener('DOMContentLoaded', fn); } ready(function(){ const div = document.getElementById('rep-map-demo'); const slider = document.getElementById('rep-warp'); if(!div || !slider) return; if(typeof Plotly === 'undefined'){ div.innerHTML = '<p><strong>Plotly.js was not available, so the interactive figure could not be drawn.</strong></p>'; return; } const n = 160; const pts = []; for(let i=0;i<n;i++){ const t = 2*Math.PI*i/n; const r = i < n/2 ? 1.0 + 0.08*Math.sin(7*t) : 1.9 + 0.08*Math.cos(5*t); pts.push({x:r*Math.cos(t), y:r*Math.sin(t), cls:i<n/2?0:1}); } function draw(){ const a = Number(slider.value); const inner = {x:[], y:[], mode:'markers', type:'scatter', name:'Class 0', marker:{size:7}}; const outer = {x:[], y:[], mode:'markers', type:'scatter', name:'Class 1', marker:{size:7}}; pts.forEach(p => { const r2 = p.x*p.x + p.y*p.y; const z1 = p.x; const z2 = (1-a)*p.y + a*(r2 - 2.2); if(p.cls === 0){ inner.x.push(z1); inner.y.push(z2); } else { outer.x.push(z1); outer.y.push(z2); } }); const line = {x:[-2.2,2.2], y:[0,0], mode:'lines', name:'linear separator'}; Plotly.newPlot(div, [inner, outer, line], { margin:{t:30,r:20,b:45,l:50}, xaxis:{title:'feature coordinate 1'}, yaxis:{title:'feature coordinate 2'}, title:'Representation can make a nonlinear pattern nearly linearly separable' }, {displayModeBar:false, responsive:true}); } slider.addEventListener('input', draw); draw(); }); })(); </script> ## 20.5 Hand-designed features and learned features Before deep learning, many successful vision systems used hand-designed features. Examples include: - local gradients and edge responses; - Harris and SIFT-like descriptors; - histograms of oriented gradients; - color histograms; - texture descriptors; - shape moments; - frequency-domain features. A hand-designed feature map has the form $$ \phi(x)= \begin{bmatrix} \phi_1(x) \\ \phi_2(x) \\ \vdots \\ \phi_d(x) \end{bmatrix}. $$ A learned feature map has the same mathematical role but its coordinates depend on parameters estimated from data: $$ \phi_\theta(x)= \begin{bmatrix} \phi_{\theta,1}(x) \\ \phi_{\theta,2}(x) \\ \vdots \\ \phi_{\theta,d}(x) \end{bmatrix}. $$ The distinction is not that one is mathematical and the other is not. Both are functions. The distinction is how the function is chosen. ### A useful decomposition For supervised learning, write $$ \widehat y = h_w(\phi_\theta(x)). $$ Then $\phi_\theta$ is the encoder or backbone, while $h_w$ is the head. In image classification, $h_w$ may be a linear softmax classifier. In detection, the head predicts boxes and classes. In segmentation, the head predicts a label for each pixel or patch. This decomposition appears in many modern systems: $$ \text{image} \to \text{backbone features} \to \text{task head}. $$ ### Linear probing A common evaluation protocol freezes $\phi_\theta$ and trains only a linear classifier: $$ \min_{W,b} \frac{1}{n}\sum_{i=1}^n \ell(W\phi_\theta(x_i)+b,y_i). $$ If this works well, the representation makes the classes linearly accessible. Fine-tuning allows both $\theta$ and $(W,b)$ to change: $$ \min_{\theta,W,b} \frac{1}{n}\sum_{i=1}^n \ell(W\phi_\theta(x_i)+b,y_i). $$ Linear probing tests representation quality. Fine-tuning tests adaptability. ## 20.6 PCA as the first representation-learning model PCA is a linear representation-learning method. Given centered data vectors $x_i\in\mathbb R^p$, PCA seeks an orthonormal matrix $$ U_d=[u_1,\dots,u_d]\in\mathbb R^{p\times d} $$ and represents each image by $$ z_i=U_d^T x_i\in\mathbb R^d. $$ The reconstruction is $$ \widehat x_i=U_d z_i=U_dU_d^T x_i. $$ PCA solves two equivalent problems: $$ \max_{U^TU=I_d}\operatorname{tr}(U^T S U), $$ where $S$ is the sample covariance, and $$ \min_{U^TU=I_d}\sum_{i=1}^n \|x_i-UU^Tx_i\|_2^2. $$ Thus PCA learns a representation by preserving variance and minimizing squared reconstruction error. ### PCA is not necessarily semantic PCA is mathematically clean, but it is unsupervised and linear. High-variance directions need not be task-relevant. For instance, lighting variation may dominate object identity. This motivates nonlinear and task-aware representations. ## 20.7 Autoencoders as nonlinear PCA An autoencoder consists of an encoder and decoder: $$ z=f_\theta(x), \qquad \widehat x=g_\psi(z). $$ It is trained by minimizing reconstruction loss: $$ \min_{\theta,\psi} \frac{1}{n}\sum_{i=1}^n \|x_i-g_\psi(f_\theta(x_i))\|_2^2. $$ If $f$ and $g$ are linear maps and the bottleneck has dimension $d$, the optimal subspace is closely related to PCA. With nonlinear neural networks, the autoencoder can learn curved low-dimensional structure. ### Denoising autoencoders A denoising autoencoder corrupts the input and reconstructs the clean image: $$ \widetilde x = x + \epsilon, \qquad \min_{\theta,\psi} \mathbb E\|x-g_\psi(f_\theta(\widetilde x))\|_2^2. $$ This encourages the representation to ignore nuisance noise. ### Variational autoencoders A variational autoencoder treats the latent variable probabilistically: $$ z \sim p(z), \qquad x\sim p_\psi(x\mid z), $$ with an approximate posterior $q_\theta(z\mid x)$. The objective is the evidence lower bound: $$ \mathcal L(\theta,\psi) = \mathbb E_{q_\theta(z\mid x)}[\log p_\psi(x\mid z)] - D_{\mathrm{KL}}(q_\theta(z\mid x)\|p(z)). $$ The VAE latent representation is shaped both by reconstruction and by a prior distribution. ## 20.8 Metric learning Some vision tasks are not naturally expressed as fixed-label classification. For retrieval, face recognition, product matching, landmark recognition, and image deduplication, the output is often a similarity score. We want embeddings such that similar images are close and dissimilar images are far apart. Let $$ z_i=\phi_\theta(x_i). $$ A metric-learning objective uses pairs or triplets. ### Contrastive pair loss For a pair $(x_i,x_j)$ with label $s_{ij}=1$ if the pair is similar and $s_{ij}=0$ otherwise, define $d_{ij}=\|z_i-z_j\|_2$. A classical contrastive loss is $$ \mathcal L_{ij} = s_{ij}d_{ij}^2 + (1-s_{ij})\max(0,m-d_{ij})^2, $$ where $m>0$ is a margin. Positive pairs are pulled together; negative pairs are pushed at least margin $m$ apart. ### Triplet loss A triplet consists of an anchor $a$, positive example $p$, and negative example $n$. The triplet loss is $$ \mathcal L = \max\bigl(0, \|z_a-z_p\|_2^2 - \|z_a-z_n\|_2^2 +m \bigr). $$ The loss is zero when the positive is closer to the anchor than the negative by at least margin $m$. ### Normalized embeddings and angular geometry Many modern retrieval systems use normalized embeddings: $$ \bar z = \frac{z}{\|z\|_2}. $$ Then Euclidean distance and cosine similarity are directly related: $$ \|\bar z_i-\bar z_j\|_2^2 =2-2\bar z_i^T\bar z_j. $$ Thus learning on the unit sphere is learning angular separation. ## 20.9 Interactive: triplet loss geometry <div class="interactive-card"> <h3>Interactive: Triplet loss margin</h3> <p>The anchor and positive are fixed. Move the negative point and change the margin. The triplet loss is positive when the negative is not sufficiently farther away.</p> <label for="triplet-margin">Margin:</label> <input id="triplet-margin" type="range" min="0" max="3" step="0.1" value="1.0" style="width:100%;"> <label for="triplet-neg-x">Negative x-coordinate:</label> <input id="triplet-neg-x" type="range" min="-3" max="4" step="0.1" value="2.0" style="width:100%;"> <div id="triplet-demo" class="plotly-demo"></div> </div> <script> (function(){ function ready(fn){ if(document.readyState !== 'loading') fn(); else document.addEventListener('DOMContentLoaded', fn); } ready(function(){ const div = document.getElementById('triplet-demo'); const marginSlider = document.getElementById('triplet-margin'); const negSlider = document.getElementById('triplet-neg-x'); if(!div || !marginSlider || !negSlider) return; if(typeof Plotly === 'undefined'){ div.innerHTML = '<p><strong>Plotly.js was not available, so the interactive figure could not be drawn.</strong></p>'; return; } function dist2(a,b){ return (a[0]-b[0])*(a[0]-b[0]) + (a[1]-b[1])*(a[1]-b[1]); } function draw(){ const a = [0,0]; const p = [0.9,0.45]; const n = [Number(negSlider.value), -0.8]; const m = Number(marginSlider.value); const dap = dist2(a,p); const dan = dist2(a,n); const loss = Math.max(0, dap - dan + m); const pts = { x:[a[0],p[0],n[0]], y:[a[1],p[1],n[1]], mode:'markers+text', type:'scatter', text:['anchor','positive','negative'], textposition:'top center', marker:{size:[14,14,14]}, name:'triplet' }; const lines = { x:[a[0],p[0],null,a[0],n[0]], y:[a[1],p[1],null,a[1],n[1]], mode:'lines', type:'scatter', name:'distances' }; Plotly.newPlot(div, [pts, lines], { margin:{t:60,r:20,b:45,l:50}, xaxis:{range:[-3.2,4.2], zeroline:true, title:'embedding coordinate 1'}, yaxis:{range:[-2.5,2.5], zeroline:true, title:'embedding coordinate 2'}, title:'Loss = max(0, d(a,p)^2 - d(a,n)^2 + margin) = ' + loss.toFixed(3) }, {displayModeBar:false, responsive:true}); } marginSlider.addEventListener('input', draw); negSlider.addEventListener('input', draw); draw(); }); })(); </script> ## 20.10 Contrastive learning from augmentations Self-supervised contrastive learning removes the need for explicit labels. Instead, it creates positive pairs from different augmented views of the same image. Let $t_1,t_2$ be random augmentations. For an image $x_i$, define $$ x_i^{(1)}=t_1(x_i), \qquad x_i^{(2)}=t_2(x_i). $$ The two views form a positive pair. Views from different images are usually treated as negatives. Let the encoder and projection head produce normalized vectors $$ q_i = \frac{g(\phi(x_i^{(1)}))}{\|g(\phi(x_i^{(1)}))\|_2}, \qquad k_j = \frac{g(\phi(x_j^{(2)}))}{\|g(\phi(x_j^{(2)}))\|_2}. $$ A common loss for query $q_i$ is the InfoNCE loss: $$ \mathcal L_i = - \log \frac{\exp(q_i^T k_i/\tau)} {\sum_{j=1}^B \exp(q_i^T k_j/\tau)}. $$ Here $B$ is the batch size and $\tau>0$ is the temperature. ### Temperature The temperature controls how sharply the softmax emphasizes the most similar examples. Small $\tau$ produces a sharper distribution and larger gradients for hard negatives. Large $\tau$ smooths the distribution. ### Alignment and uniformity Contrastive learning balances two forces: 1. **Alignment:** positive pairs should be close. 2. **Uniformity:** embeddings should spread out on the representation sphere. A simplified alignment term is $$ \mathbb E\|z-z^+\|_2^2. $$ A simplified uniformity term is $$ \log \mathbb E_{i\neq j}\exp(-2\|z_i-z_j\|_2^2). $$ Good representations are not produced by collapse, where all embeddings are identical. Contrastive negatives help prevent collapse. ## 20.11 Interactive: InfoNCE temperature <div class="interactive-card"> <h3>Interactive: temperature in contrastive learning</h3> <p>The positive similarity is fixed. The negative similarities vary. Smaller temperature makes the softmax distribution sharper.</p> <label for="temp-slider">Temperature $\tau$:</label> <input id="temp-slider" type="range" min="0.05" max="1.5" step="0.05" value="0.3" style="width:100%;"> <div id="temperature-demo" class="plotly-demo"></div> </div> <script> (function(){ function ready(fn){ if(document.readyState !== 'loading') fn(); else document.addEventListener('DOMContentLoaded', fn); } ready(function(){ const div = document.getElementById('temperature-demo'); const slider = document.getElementById('temp-slider'); if(!div || !slider) return; if(typeof Plotly === 'undefined'){ div.innerHTML = '<p><strong>Plotly.js was not available, so the interactive figure could not be drawn.</strong></p>'; return; } const sims = [0.82,0.68,0.42,0.25,0.1,-0.05,-0.18,-0.35]; const labels = ['positive','neg 1','neg 2','neg 3','neg 4','neg 5','neg 6','neg 7']; function draw(){ const tau = Number(slider.value); const logits = sims.map(s => Math.exp(s/tau)); const Z = logits.reduce((a,b)=>a+b,0); const probs = logits.map(v => v/Z); const loss = -Math.log(probs[0]); Plotly.newPlot(div, [{x:labels, y:probs, type:'bar', name:'softmax probability'}], { margin:{t:60,r:20,b:45,l:50}, yaxis:{range:[0,1], title:'probability'}, title:'InfoNCE loss = ' + loss.toFixed(3) + ' at temperature tau = ' + tau.toFixed(2) }, {displayModeBar:false, responsive:true}); } slider.addEventListener('input', draw); draw(); }); })(); </script> ## 20.12 Collapse and non-contrastive self-supervision Contrastive learning uses negative examples to avoid collapse. Other methods avoid collapse without explicit negatives by architectural or statistical constraints. ### Predictor asymmetry In BYOL-style methods, two networks produce views of the same image. One branch includes a predictor. The method encourages one representation to predict the other: $$ \mathcal L = \left\| \frac{q_\theta(z_1)}{\|q_\theta(z_1)\|_2} - \frac{z_2'}{\|z_2'\|_2} \right\|_2^2, $$ where the target branch is often updated by exponential moving average rather than direct gradient descent. The asymmetry helps avoid trivial collapse. ### Redundancy reduction Barlow Twins-style methods use a cross-correlation matrix between two augmented views. Suppose $Z^A,Z^B\in\mathbb R^{B\times d}$ are normalized feature matrices. The empirical cross-correlation is $$ C_{ij}=\frac{1}{B}\sum_{b=1}^B Z^A_{b i}Z^B_{b j}. $$ The objective encourages $$ C \approx I_d. $$ A typical loss is $$ \mathcal L = \sum_i (1-C_{ii})^2 + \lambda\sum_{i\neq j} C_{ij}^2. $$ The diagonal term aligns the two views coordinatewise. The off-diagonal term discourages redundant coordinates. ### Variance-invariance-covariance regularization VICReg-style methods combine three terms: $$ \mathcal L = \lambda\mathcal L_{\text{inv}} + \mu\mathcal L_{\text{var}} + \nu\mathcal L_{\text{cov}}. $$ The invariance term aligns positive pairs. The variance term prevents collapse by requiring each coordinate to have nontrivial standard deviation. The covariance term discourages redundant dimensions. ## 20.13 Masked image modeling Another self-supervised strategy hides part of the input and asks the model to reconstruct it. Let $M$ denote a random mask and $x_M$ the visible part of the image. A masked modeling objective has the form $$ \min_{\theta,\psi} \mathbb E_{x,M} \left[ \ell\left(D_\psi(E_\theta(x_M)), x_{\bar M}\right)\right], $$ where $E_\theta$ is an encoder, $D_\psi$ is a decoder, and $x_{\bar M}$ is the hidden part. This objective is natural for vision transformers because an image can be divided into patches. A large fraction of patches can be hidden, and the model must infer missing visual structure from context. ### Reconstruction target The target may be pixels, normalized patches, discrete visual tokens, low-level features, or high-level features. Pixel reconstruction encourages local detail; feature reconstruction may encourage semantic abstraction. ### Statistical interpretation Masked modeling estimates conditional structure: $$ p(x_{\bar M}\mid x_M). $$ Thus the representation must encode information that predicts missing content. ## 20.14 Representations and nuisance variables Many visual changes are not relevant to a given task. Let a generative description be $$ X = G(Y, N), $$ where $Y$ is the task-relevant variable and $N$ is a nuisance variable such as lighting, viewpoint, background, pose, or sensor noise. A representation for predicting $Y$ should preserve information about $Y$ and suppress information about $N$. A useful idealization is $$ \phi(G(Y,N_1)) \approx \phi(G(Y,N_2)) $$ for nuisance values $N_1,N_2$ that do not change the task label. ### Data augmentation as nuisance sampling If augmentations mimic nuisance variation, then training with augmentations encourages invariance: $$ \phi(t(x)) \approx \phi(x). $$ However, augmentations must be chosen carefully. A horizontal flip may preserve labels for natural-image classification but may change labels in medical imaging, traffic signs, or text recognition. ::: {.callout-warning} ## Augmentation is a modeling assumption Every augmentation says: “This transformation should not change the meaning needed for the task.” This is a statistical assumption, not a free trick. ::: ## 20.15 Interactive: invariance versus information loss <div class="interactive-card"> <h3>Interactive: invariance can help or hurt</h3> <p>The representation suppresses a nuisance direction. If the suppressed direction is truly irrelevant, classification improves. If it contains label information, the representation fails.</p> <label for="inv-suppress">Strength of suppression:</label> <input id="inv-suppress" type="range" min="0" max="1" step="0.05" value="0.5" style="width:100%;"> <label for="inv-task">Task direction:</label> <select id="inv-task"> <option value="x">label depends mostly on horizontal coordinate</option> <option value="y">label depends mostly on vertical coordinate</option> </select> <div id="invariance-demo" class="plotly-demo"></div> </div> <script> (function(){ function ready(fn){ if(document.readyState !== 'loading') fn(); else document.addEventListener('DOMContentLoaded', fn); } ready(function(){ const div = document.getElementById('invariance-demo'); const slider = document.getElementById('inv-suppress'); const select = document.getElementById('inv-task'); if(!div || !slider || !select) return; if(typeof Plotly === 'undefined'){ div.innerHTML = '<p><strong>Plotly.js was not available, so the interactive figure could not be drawn.</strong></p>'; return; } const pts=[]; for(let i=0;i<220;i++){ const u = -2 + 4*((i*37)%220)/219; const v = Math.sin(i*1.7)*1.2 + 0.4*Math.cos(i*0.31); pts.push({u:u, v:v}); } function draw(){ const s = Number(slider.value); const task = select.value; const c0={x:[],y:[],mode:'markers',type:'scatter',name:'class 0',marker:{size:7}}; const c1={x:[],y:[],mode:'markers',type:'scatter',name:'class 1',marker:{size:7}}; let errors=0; pts.forEach(p => { const ytrue = task === 'x' ? (p.u>0?1:0) : (p.v>0?1:0); const z1 = p.u; const z2 = (1-s)*p.v; const ypred = task === 'x' ? (z1>0?1:0) : (z2>0?1:0); if(ytrue !== ypred) errors += 1; if(ytrue===0){c0.x.push(z1); c0.y.push(z2);} else {c1.x.push(z1); c1.y.push(z2);} }); const acc = 1 - errors/pts.length; Plotly.newPlot(div, [c0,c1], { margin:{t:60,r:20,b:45,l:50}, xaxis:{title:'retained coordinate'}, yaxis:{title:'suppressed coordinate', range:[-1.7,1.7]}, title:'Approximate linear classification accuracy after suppression: ' + acc.toFixed(3) }, {displayModeBar:false, responsive:true}); } slider.addEventListener('input', draw); select.addEventListener('change', draw); draw(); }); })(); </script> ## 20.16 The geometry of embedding spaces Embedding spaces are often high-dimensional. Some geometric facts are important. ### Concentration of distances If random vectors are uniformly distributed on a high-dimensional sphere, pairwise angles tend to concentrate near $90^\circ$. This means nearest-neighbor intuition from two-dimensional geometry can be misleading. For normalized vectors $z_i,z_j\in S^{d-1}$, $$ \mathbb E[z_i^Tz_j]=0, \qquad \operatorname{Var}(z_i^Tz_j)=\frac{1}{d}. $$ As $d$ increases, random dot products concentrate near zero. Therefore a cosine similarity of $0.3$ may be highly meaningful in a large embedding dimension. ### Clustering and manifolds A representation may organize data near lower-dimensional manifolds. A class may occupy a cluster, a curved submanifold, or multiple modes. For example, images of the same object under different viewpoints may trace a viewpoint manifold. ### Dimensionality reduction for visualization Methods such as PCA, t-SNE, and UMAP are often used to visualize embeddings. These tools are useful but can mislead. PCA preserves global linear variance. t-SNE emphasizes local neighborhoods and often exaggerates cluster separation. UMAP builds a neighborhood graph and tries to preserve fuzzy topological structure. A two-dimensional plot is an exploratory diagnostic, not proof of class separation. ## 20.17 Interactive: PCA view of learned embeddings <div class="interactive-card"> <h3>Interactive: projection can hide or reveal structure</h3> <p>A high-dimensional representation may have class information in a direction that is not the largest-variance direction. Rotate the projection to see how visualization depends on the chosen coordinates.</p> <label for="pca-angle">Projection angle:</label> <input id="pca-angle" type="range" min="0" max="180" step="2" value="25" style="width:100%;"> <div id="pca-embedding-demo" class="plotly-demo"></div> </div> <script> (function(){ function ready(fn){ if(document.readyState !== 'loading') fn(); else document.addEventListener('DOMContentLoaded', fn); } ready(function(){ const div = document.getElementById('pca-embedding-demo'); const slider = document.getElementById('pca-angle'); if(!div || !slider) return; if(typeof Plotly === 'undefined'){ div.innerHTML = '<p><strong>Plotly.js was not available, so the interactive figure could not be drawn.</strong></p>'; return; } const pts=[]; for(let i=0;i<240;i++){ const cls = i<120?0:1; const t = i*0.17; const major = 2*Math.sin(t) + 0.2*Math.cos(3*t); const semantic = (cls===0?-0.65:0.65) + 0.18*Math.sin(5*t); const nuisance = 0.5*Math.cos(2*t); pts.push({a:major, b:semantic, c:nuisance, cls:cls}); } function draw(){ const ang = Number(slider.value)*Math.PI/180; const c = Math.cos(ang), s = Math.sin(ang); const d0={x:[],y:[],mode:'markers',type:'scatter',name:'class 0',marker:{size:7}}; const d1={x:[],y:[],mode:'markers',type:'scatter',name:'class 1',marker:{size:7}}; pts.forEach(p => { const x = c*p.a + s*p.b; const y = -s*p.a + c*p.b + 0.25*p.c; if(p.cls===0){d0.x.push(x); d0.y.push(y);} else {d1.x.push(x); d1.y.push(y);} }); Plotly.newPlot(div, [d0,d1], { margin:{t:55,r:20,b:45,l:50}, xaxis:{title:'display coordinate 1'}, yaxis:{title:'display coordinate 2'}, title:'A 2D visualization depends strongly on projection choice' }, {displayModeBar:false, responsive:true}); } slider.addEventListener('input', draw); draw(); }); })(); </script> ## 20.18 Supervised representation learning In supervised learning, the representation is optimized directly for a task loss: $$ \min_{\theta,w} \frac{1}{n}\sum_{i=1}^n \ell(h_w(\phi_\theta(x_i)),y_i) + \lambda R(\theta,w). $$ The representation is learned to support the supervised head. ### Penultimate-layer embeddings In a neural classifier, the vector before the final linear layer is often used as an embedding: $$ z=\phi_\theta(x), \qquad \text{logits}=Wz+b. $$ If the classifier is trained with cross-entropy, embeddings tend to arrange themselves so that class-specific linear scores separate the data. ### Center loss and class compactness One can explicitly encourage embeddings of the same class to be close. Let $c_y$ be a learned center for class $y$. Center loss adds $$ \mathcal L_{\text{center}} = \sum_{i=1}^n \|z_i-c_{y_i}\|_2^2. $$ This reduces within-class scatter but must be balanced with class separation. ### Angular margin losses For normalized features and weights, classification can be based on angles. If $$ \|z\|_2=1, \qquad \|w_k\|_2=1, $$ then the logit is proportional to $$ w_k^Tz=\cos\theta_k. $$ Angular-margin losses improve separation by requiring a margin in angle, useful in face recognition and retrieval. ## 20.19 Transfer learning and domain shift A representation learned on a large source dataset may transfer to a smaller target dataset. Let $$ P_S(X,Y) $$ be the source distribution and $$ P_T(X,Y) $$ be the target distribution. Transfer works when the learned features capture structure useful under $P_T$. ### Frozen features The simplest transfer method freezes the encoder and trains a new head: $$ \min_w \frac{1}{n_T}\sum_{i=1}^{n_T} \ell(h_w(\phi_{\theta_S}(x_i^T)),y_i^T). $$ This is statistically attractive when the target dataset is small. ### Fine-tuning Fine-tuning updates the encoder: $$ \min_{\theta,w} \frac{1}{n_T}\sum_{i=1}^{n_T} \ell(h_w(\phi_\theta(x_i^T)),y_i^T). $$ Fine-tuning can adapt to the target domain but risks overfitting and catastrophic forgetting. ### Domain shift Representation transfer can fail when source and target differ strongly: $$ P_S(X)\neq P_T(X), \qquad P_S(Y\mid X)\neq P_T(Y\mid X). $$ Medical images, satellite images, industrial images, and scientific images often differ from natural-image pretraining data. For statistics students, this is a covariate-shift and model-misspecification problem, not just a software problem. ## 20.20 Foundation representations Modern vision often uses very large pretrained encoders. These may be trained by supervised classification, image-text contrastive learning, masked image modeling, or combinations of many objectives. The resulting representation may be reused for many downstream tasks. ### Image-text representations A vision-language model learns image and text encoders $$ z_I=\phi_I(x), \qquad z_T=\phi_T(t), $$ and aligns paired image-text examples through a contrastive objective: $$ \mathcal L_i = - \log \frac{\exp(z_{I,i}^Tz_{T,i}/\tau)} {\sum_j \exp(z_{I,i}^Tz_{T,j}/\tau)}. $$ This makes text prompts usable as classifier weights or retrieval queries. ### Representation as interface A foundation representation is an interface between data and tasks. The same embedding can support retrieval, classification, captioning, detection, segmentation, or visual question answering if paired with suitable heads and prompts. ## 20.21 Statistical evaluation of representations A representation should be evaluated with clear statistical protocols. ### Linear probing Train a linear classifier on frozen embeddings. This asks: is the target information linearly accessible? ### $k$-nearest-neighbor evaluation Classify by nearest neighbors in embedding space: $$ \widehat y(x)=\operatorname{mode}\{y_i: i\in N_k(\phi(x))\}. $$ This tests whether local geometry is meaningful. ### Retrieval metrics For retrieval, common metrics include precision at $k$, recall at $k$, mean average precision, and normalized discounted cumulative gain. ### Clustering metrics Unsupervised representation quality may be evaluated by clustering and then comparing clusters to labels using adjusted Rand index, normalized mutual information, or purity. These metrics must be interpreted cautiously because clustering assumptions may not match the representation geometry. ### Calibration and uncertainty If a representation supports probabilistic predictions, evaluate calibration, Brier score, negative log-likelihood, and reliability diagrams. A strong embedding with an overconfident head can still be statistically unsafe. ## 20.22 Python example: PCA and logistic regression on image features The following example shows the classical representation-learning workflow: turn images into vectors, learn a lower-dimensional representation by PCA, and train a simple classifier. ```{python} #| label: py-ch20-pca-representation #| eval: false import numpy as np import matplotlib.pyplot as plt from sklearn.datasets import load_digits from sklearn.model_selection import train_test_split from sklearn.decomposition import PCA from sklearn.linear_model import LogisticRegression from sklearn.pipeline import Pipeline from sklearn.metrics import accuracy_score, confusion_matrix # Small grayscale image dataset: 8 x 8 digit images. digits = load_digits() X = digits.data / 16.0 y = digits.target X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, random_state=7, stratify=y ) model = Pipeline([ ("pca", PCA(n_components=20, whiten=True, random_state=7)), ("clf", LogisticRegression(max_iter=3000, multi_class="auto")) ]) model.fit(X_train, y_train) y_pred = model.predict(X_test) print("Test accuracy:", accuracy_score(y_test, y_pred)) print(confusion_matrix(y_test, y_pred)) pca = model.named_steps["pca"] plt.plot(np.cumsum(pca.explained_variance_ratio_), marker="o") plt.xlabel("number of PCA components") plt.ylabel("cumulative explained variance") plt.title("PCA representation dimension") plt.grid(True) plt.show() ``` The important statistical point is that PCA is unsupervised. It may preserve high-variance directions that are not the best directions for classification. ## 20.23 Python example: visualizing embeddings ```{python} #| label: py-ch20-embedding-visualization #| eval: false import numpy as np import matplotlib.pyplot as plt from sklearn.datasets import load_digits from sklearn.decomposition import PCA from sklearn.manifold import TSNE from sklearn.preprocessing import StandardScaler X, y = load_digits(return_X_y=True) X = StandardScaler().fit_transform(X) Z_pca = PCA(n_components=2, random_state=7).fit_transform(X) # t-SNE is useful for visualization but not for proving cluster structure. Z_tsne = TSNE( n_components=2, perplexity=30, learning_rate="auto", init="pca", random_state=7 ).fit_transform(X) plt.figure(figsize=(6, 5)) plt.scatter(Z_pca[:, 0], Z_pca[:, 1], c=y, s=12, cmap="tab10") plt.title("PCA visualization of raw image vectors") plt.colorbar() plt.show() plt.figure(figsize=(6, 5)) plt.scatter(Z_tsne[:, 0], Z_tsne[:, 1], c=y, s=12, cmap="tab10") plt.title("t-SNE visualization of raw image vectors") plt.colorbar() plt.show() ``` When using such plots, ask: which neighborhoods are trustworthy, which distances are distorted, and which structures are artifacts of the visualization algorithm? ## 20.24 TensorFlow example: a small supervised representation encoder ```{python} #| label: py-ch20-tensorflow-supervised-encoder #| eval: false import tensorflow as tf from tensorflow.keras import layers, models # Example input: 32 x 32 RGB image. inputs = layers.Input(shape=(32, 32, 3)) x = layers.Conv2D(32, 3, padding="same", activation="relu")(inputs) x = layers.MaxPooling2D()(x) x = layers.Conv2D(64, 3, padding="same", activation="relu")(x) x = layers.MaxPooling2D()(x) x = layers.Conv2D(128, 3, padding="same", activation="relu")(x) x = layers.GlobalAveragePooling2D()(x) # This is the learned representation. embedding = layers.Dense(128, activation=None, name="embedding")(x) embedding_norm = tf.nn.l2_normalize(embedding, axis=1) outputs = layers.Dense(10, activation="softmax")(embedding_norm) model = models.Model(inputs, outputs) model.compile( optimizer=tf.keras.optimizers.Adam(1e-3), loss="sparse_categorical_crossentropy", metrics=["accuracy"] ) encoder = models.Model(inputs, embedding_norm) model.summary() ``` The `encoder` model maps images to normalized vectors. These vectors can be used for nearest-neighbor retrieval, clustering, linear probing, or transfer learning. ## 20.25 TensorFlow example: InfoNCE loss The following code sketches the core of an image contrastive loss. A real implementation also needs strong data augmentation and careful batching. ```{python} #| label: py-ch20-tensorflow-infonce #| eval: false import tensorflow as tf def l2_normalize(x, axis=1, eps=1e-8): return x / (tf.norm(x, axis=axis, keepdims=True) + eps) def info_nce_loss(z1, z2, temperature=0.2): """ z1 and z2 are embeddings of two augmented views of the same batch. Shape: (batch_size, embedding_dim). Positive pairs are (z1[i], z2[i]). Other examples in the batch serve as negatives. """ z1 = l2_normalize(z1) z2 = l2_normalize(z2) logits = tf.matmul(z1, z2, transpose_b=True) / temperature labels = tf.range(tf.shape(z1)[0]) loss_12 = tf.keras.losses.sparse_categorical_crossentropy( labels, logits, from_logits=True ) loss_21 = tf.keras.losses.sparse_categorical_crossentropy( labels, tf.transpose(logits), from_logits=True ) return tf.reduce_mean((loss_12 + loss_21) / 2.0) # Example usage with dummy embeddings. z1 = tf.random.normal((64, 128)) z2 = tf.random.normal((64, 128)) print(info_nce_loss(z1, z2, temperature=0.2)) ``` ## 20.26 AI-assisted study components Use an AI assistant as a mathematical critic and experimental partner, not as a replacement for derivation. ### Prompt 1: representation properties > I am studying representation learning for computer vision. Given a task such as image classification, segmentation, or retrieval, list the desired mathematical properties of the representation. For each property, state whether it is an invariance, equivariance, stability, separation, or compression property. ### Prompt 2: critique an augmentation > I plan to use random crops, horizontal flips, color jitter, and Gaussian blur for a medical imaging dataset. Analyze which augmentations may violate label preservation. Explain the statistical assumption behind each augmentation. ### Prompt 3: compare losses > Compare cross-entropy, triplet loss, InfoNCE, Barlow Twins loss, and masked reconstruction loss. For each, identify the positive signal, the anti-collapse mechanism, and the geometry encouraged in embedding space. ### Prompt 4: derivation checker > Verify the relationship $\|u-v\|_2^2=2-2u^Tv$ for unit vectors. Then explain why cosine similarity and Euclidean distance are equivalent for normalized embeddings. ### Prompt 5: experimental design > Design an experiment to compare frozen features, linear probing, fine-tuning, and nearest-neighbor retrieval for a small image dataset. Include train/test splits, metrics, and possible sources of leakage. ## 20.27 Common pitfalls ### Pitfall 1: treating embeddings as automatically meaningful A vector representation is not meaningful merely because it was produced by a neural network. Its meaning is defined by training data, objective, architecture, and evaluation protocol. ### Pitfall 2: overinterpreting two-dimensional visualizations PCA, t-SNE, and UMAP plots are diagnostic tools. They do not prove that a representation is statistically reliable. ### Pitfall 3: using augmentations without checking label preservation Augmentation is a mathematical assumption about invariance. Incorrect augmentations can teach the model to ignore information needed for the task. ### Pitfall 4: confusing linear separability with causal understanding A representation may support strong classification while relying on shortcuts such as background, watermark, scanner type, or dataset artifacts. ### Pitfall 5: ignoring calibration Good representations can still produce badly calibrated probabilities when paired with overconfident heads. ## 20.28 Summary Representation learning studies maps $$ \phi_\theta : \mathcal X \to \mathbb R^d $$ that transform images into useful coordinates. Good representations preserve task-relevant information, suppress nuisance variation, stabilize noise, organize geometry, and make downstream tasks simpler. The mathematical viewpoints are complementary: - **Linear algebra:** representations are coordinate systems, projections, embeddings, and kernels. - **Statistics:** representations are statistics of data, with sufficiency, bias-variance tradeoff, and domain-shift concerns. - **Optimization:** representations are learned by minimizing empirical losses. - **Geometry:** representations define distances, angles, manifolds, and clusters. - **Information theory:** representations balance compression and task information. - **AI:** representations are the reusable core of modern vision foundation models. The next chapters study the architectures and training principles that produce many of these representations in practice. ## Exercises ### Conceptual exercises 1. Explain why raw pixel distance is often a poor measure of semantic similarity. 2. Give three examples of nuisance variation in image classification. For each, state an augmentation that might encourage invariance. 3. Explain the difference between invariance and equivariance in the context of object detection. 4. Why is linear probing a useful but incomplete evaluation of a representation? 5. Explain why a representation learned for classification may not be ideal for segmentation. ### Mathematical exercises 6. Let $u,v\in\mathbb R^d$ satisfy $\|u\|_2=\|v\|_2=1$. Prove that $\|u-v\|_2^2=2-2u^Tv$. 7. Derive the gradient of the pairwise contrastive loss with respect to $z_i$ for a positive pair. 8. For the triplet loss, identify the region in which the gradient is zero. 9. Suppose $Z\mid Y=k\sim N(\mu_k,\Sigma)$ with common covariance. Show that the Bayes classifier is linear in $Z$. 10. Let $C$ be the cross-correlation matrix in a redundancy-reduction loss. Explain why forcing $C\approx I$ discourages both collapse and redundancy. 11. Show that PCA minimizes squared reconstruction error among all $d$-dimensional linear subspaces. 12. If $z_i,z_j$ are independent uniform unit vectors in $\mathbb R^d$, compute $\mathbb E[z_i^Tz_j]$ and explain why similarities concentrate near zero as $d$ grows. ### Python exercises 13. Use the digits dataset to compare logistic regression on raw pixels, PCA features, and HOG-style features. 14. Train a small autoencoder on grayscale images and visualize the two-dimensional latent space. 15. Implement triplet loss for synthetic two-dimensional data and visualize the learned embedding. 16. Implement the InfoNCE loss and test the effect of temperature on gradient magnitudes. 17. Extract embeddings from a pretrained TensorFlow or PyTorch image model and evaluate $k$-nearest-neighbor classification. 18. Use PCA and t-SNE to visualize the same embedding. Write a short note explaining what each plot preserves and distorts. ### AI-assisted exercises 19. Ask an AI assistant to propose augmentations for a dataset. Then critique each augmentation mathematically as a label-preservation assumption. 20. Ask an AI assistant to explain InfoNCE. Identify any missing assumptions about negatives, temperature, normalization, or collapse. 21. Ask an AI assistant to compare PCA, autoencoders, and contrastive learning. Rewrite the answer using precise notation. 22. Ask an AI assistant to design a representation-evaluation experiment. Add missing statistical safeguards such as train/test splits, calibration, and confidence intervals. ## Notes and references Representation learning is a broad field connected to classical feature extraction, statistical learning, neural networks, and modern self-supervised learning. General representation-learning perspectives appear in @bengio2013representation. Metric learning and Siamese-style ideas are represented by @hadsell2006dimensionality, @schroff2015facenet, and @weinberger2009distance. Contrastive and self-supervised learning are represented by @oord2018representation, @chen2020simclr, @he2020moco, @grill2020byol, @zbontar2021barlow, @bardes2022vicreg, and @caron2021dino. Masked image modeling is represented by @he2022mae. Classical dimensionality-reduction references include @pearson1901pca, @hotelling1933analysis, @vandermaaten2008tsne, and @mcinnes2018umap.