27  Chapter 26. Self-Supervised Vision

Contrastive Learning, Masked Image Modeling, Invariance, Collapse, and Representation Evaluation

27.1 26.1 Why self-supervised vision?

In supervised image classification, every training example has a human-provided label. A model learns from pairs

\[ (x_i,y_i), \qquad x_i\in \mathcal X, \quad y_i\in \mathcal Y. \]

Modern computer vision, however, often begins with much more data than humans can label. We may have millions of images, video frames, medical scans, satellite images, microscope images, or web images, but only a small fraction of them have careful labels. Self-supervised learning uses the structure of the data itself to construct a learning signal.

The basic goal is to learn a representation map

\[ \phi_\theta:\mathcal X\to \mathbb R^d \]

without using task labels during pretraining. After pretraining, the representation is reused for classification, retrieval, detection, segmentation, depth estimation, tracking, or fine-tuning.

For MA Applied Mathematics and MS Statistics students, self-supervised vision is especially important because it exposes several deep mathematical themes:

  1. invariance: which transformations should not change the semantic representation?
  2. equivariance: which transformations should change the representation in a predictable way?
  3. identifiability: when does a self-supervised objective recover useful latent factors?
  4. regularization: how can we avoid trivial constant representations?
  5. geometry: what should the representation space look like?
  6. statistics: how do negative samples, augmentations, masking, and batch size affect estimators?
  7. optimization: why do large networks, normalization, momentum encoders, and stop-gradient operations matter?
NoteCentral idea

Self-supervised vision creates labels from the data itself. A model may be asked to make two augmented views of the same image close, to distinguish different images, to reconstruct missing patches, to predict hidden tokens, or to match a slowly moving teacher representation.

Self-supervised learning is not one algorithm. It is a family of mathematical strategies for replacing human annotation with automatically generated prediction tasks. Contrastive methods such as SimCLR and MoCo learn by comparing positive and negative pairs Chen et al. (2020); He et al. (2020). Non-contrastive methods such as BYOL, Barlow Twins, VICReg, and DINO learn from agreement, redundancy reduction, variance constraints, or teacher-student targets Grill et al. (2020); Zbontar et al. (2021); Bardes et al. (2022); Caron et al. (2021). Masked image modeling methods such as MAE learn by reconstructing hidden patches He et al. (2022).

27.1.1 Learning goals

After this chapter, you should be able to:

  1. describe self-supervised learning as representation learning without manual labels;
  2. formulate image augmentations as random transformations;
  3. distinguish invariance, equivariance, and collapse;
  4. derive the InfoNCE contrastive loss;
  5. explain the role of temperature and negative samples;
  6. analyze alignment and uniformity of embeddings;
  7. describe momentum encoders and memory queues;
  8. explain why non-contrastive methods need anti-collapse mechanisms;
  9. formulate Barlow Twins and VICReg objectives;
  10. explain masked image modeling for vision transformers;
  11. connect self-supervised pretraining to downstream tasks;
  12. implement small contrastive and masked-image experiments in Python;
  13. use AI tools to debug tensor shapes, loss functions, and representation-evaluation protocols.

27.2 26.2 The supervised-learning baseline

Let

\[ D_n=\{(x_i,y_i)\}_{i=1}^n \]

be labeled training data. A supervised model usually has the form

\[ f_\theta(x)=h_\theta(\phi_\theta(x)), \]

where \(\phi_\theta\) is a feature extractor and \(h_\theta\) is a prediction head. The empirical risk is

\[ \widehat R_n(\theta) = \frac1n\sum_{i=1}^n \ell(h_\theta(\phi_\theta(x_i)),y_i). \]

The labels \(y_i\) define the target. In image classification, \(y_i\) might be an object category. In semantic segmentation, \(y_i\) might be a mask. In depth estimation, \(y_i\) might be a depth map. In object detection, \(y_i\) might be a set of boxes and classes.

Self-supervised learning removes the manual target \(y_i\) during pretraining. Instead of using labels, it constructs a training signal from image transformations, patch masking, temporal consistency, geometry, or cross-modal pairing.

The pretraining dataset is usually

\[ U_N=\{x_i\}_{i=1}^N, \]

where \(N\) may be much larger than the number of labeled examples available for a downstream task. A self-supervised objective has the form

\[ \widehat L_N(\theta) = \frac1N\sum_{i=1}^N L_{\mathrm{ssl}}(x_i;\theta), \]

or, for pairwise/batch losses,

\[ \widehat L_N(\theta) = \frac1B\sum_{i=1}^B L_{\mathrm{ssl}}(x_i,x_1,\dots,x_B;\theta). \]

The representation \(\phi_\theta\) is then evaluated by linear probing, fine-tuning, retrieval, clustering, dense prediction, or transfer to another dataset.

27.3 26.3 Views, augmentations, and positive pairs

A central object in modern self-supervised vision is the view of an image. Let \(T\) be a random image transformation drawn from a distribution \(\mathcal T\). For an image \(x\), a view is

\[ \tilde x=T(x), \qquad T\sim \mathcal T. \]

Typical transformations include random crop, resize, horizontal flip, color jitter, grayscale conversion, blur, solarization, cutout, masking, and local patch dropout. Two independently sampled views of the same image are

\[ \tilde x_i^{(1)}=T_1(x_i), \qquad \tilde x_i^{(2)}=T_2(x_i), \qquad T_1,T_2\sim \mathcal T. \]

The pair \((\tilde x_i^{(1)},\tilde x_i^{(2)})\) is called a positive pair. Views from different images are often treated as negative pairs in contrastive learning.

WarningThe augmentation assumption

Self-supervised learning depends heavily on the assumption that the chosen transformations preserve the information needed for downstream tasks. A random crop may preserve object identity in ImageNet-style classification, but it may destroy disease location in medical imaging or remove a small object in detection.

Mathematically, an augmentation distribution encodes an invariance assumption. If \(T\) is a label-preserving transformation, then for a downstream label \(Y\) we hope

\[ Y(TX)=Y(X) \]

or, in probabilistic form,

\[ P(Y\mid X=x) \approx P(Y\mid X=T(x)). \]

The representation is encouraged to satisfy

\[ \phi_\theta(Tx)\approx \phi_\theta(x). \]

This is useful only when the transformation removes nuisance variation rather than semantic information.

27.3.1 Example: when invariance is good

For object-category classification, small translations, color jitter, and moderate crops often preserve the class label. A representation invariant to those changes can generalize better.

27.3.2 Example: when invariance is dangerous

For dermatology, color may carry diagnostic information. Aggressive color jitter may remove the signal. For satellite imagery, rotation may or may not preserve meaning depending on whether orientation matters. For digit recognition, a \(180^\circ\) rotation can change a digit.

Thus self-supervised vision is not merely an architecture. It is also a mathematical modeling decision about the transformation group or transformation family that should be treated as nuisance.

27.4 26.4 Invariance, equivariance, and representation geometry

Let \(G\) be a group of transformations acting on images. A representation \(\phi\) is invariant to \(G\) if

\[ \phi(gx)=\phi(x) \qquad \text{for all }g\in G. \]

It is equivariant if there exists a representation \(\rho(g)\) acting on the feature space such that

\[ \phi(gx)=\rho(g)\phi(x). \]

Invariance discards transformation information. Equivariance preserves it in a structured form.

For image classification, invariance is often useful. For detection, segmentation, pose estimation, and depth estimation, full invariance may be harmful. If an object shifts in the image, a segmentation mask should shift. Therefore dense vision tasks need representations that are at least partly equivariant.

A useful mental model is:

\[ \text{semantic factors} + \text{nuisance factors} \longrightarrow \text{observed image}. \]

Self-supervised learning tries to keep semantic factors while reducing nuisance factors. If the augmentation distribution is too weak, the representation may overfit to low-level texture. If it is too strong, it may discard task-relevant information.

27.5 26.5 Contrastive learning

Contrastive learning constructs a representation space where positive pairs are close and negative pairs are far apart.

Let

\[ z_i^{(1)}=g_\theta(\tilde x_i^{(1)}), \qquad z_i^{(2)}=g_\theta(\tilde x_i^{(2)}), \]

where \(g_\theta\) is usually an encoder followed by a projection head. The vectors are often normalized:

\[ \bar z=\frac{z}{\|z\|_2}. \]

The cosine similarity between two normalized embeddings is

\[ s(z,z')=\bar z^T\bar z'. \]

For an anchor \(z_i^{(1)}\), its positive is \(z_i^{(2)}\). The other images in the batch serve as negatives. The contrastive probability of selecting the correct positive among candidates is

\[ p(i^{(2)}\mid i^{(1)}) = \frac{ \exp(s(z_i^{(1)},z_i^{(2)})/\tau) }{ \sum_{k=1}^B \exp(s(z_i^{(1)},z_k^{(2)})/\tau) }, \]

where \(\tau>0\) is a temperature parameter. The loss is the negative log-likelihood:

\[ L_i = - \log \frac{ \exp(s(z_i^{(1)},z_i^{(2)})/\tau) }{ \sum_{k=1}^B \exp(s(z_i^{(1)},z_k^{(2)})/\tau) }. \]

This is one version of the InfoNCE loss. In SimCLR, both views are used symmetrically, producing a contrastive classification problem over \(2B\) augmented samples Chen et al. (2020).

TipInterpretation as softmax classification

For each anchor, contrastive learning asks: among all candidate views in the batch, which one came from the same original image? The logits are similarity scores divided by temperature.

27.5.1 Derivation as cross-entropy

Define logits

\[ a_{ik}=\frac{s(z_i^{(1)},z_k^{(2)})}{\tau}. \]

For anchor \(i\), the target class is \(k=i\). The cross-entropy loss is

\[ L_i = - \log \frac{e^{a_{ii}}}{\sum_{k=1}^B e^{a_{ik}}}. \]

Thus contrastive learning can be viewed as supervised learning where the labels are automatically created by image identity.

27.5.2 Gradient intuition

For normalized embeddings, the loss gradient pulls the anchor toward the positive embedding and pushes it away from negative embeddings. If

\[ p_{ik}=\frac{e^{a_{ik}}}{\sum_j e^{a_{ij}}}, \]

then, ignoring normalization details,

\[ \frac{\partial L_i}{\partial z_i^{(1)}} \approx \frac1\tau \left( \sum_k p_{ik}z_k^{(2)} - z_i^{(2)} \right). \]

The first term is a probability-weighted average of candidate vectors. The second term is the positive vector. The anchor moves toward the positive and away from confusing negatives.

27.6 26.6 Temperature, batch size, and negatives

The temperature \(\tau\) controls the sharpness of the softmax. Smaller \(\tau\) makes the model focus more strongly on the hardest negatives; larger \(\tau\) makes the distribution smoother.

If \(\tau\) is too large, the contrastive task may be weak. If \(\tau\) is too small, the optimization may become unstable and overly sensitive to false negatives.

A false negative occurs when a negative example is semantically similar to the anchor. For example, two different images of the same dog breed may be treated as negatives even though their downstream class is the same. Contrastive learning at the instance level does not know the class labels, so it may push semantically related images apart.

Large batch sizes help because they provide many negatives. MoCo addresses the same problem by maintaining a queue of negative embeddings and a momentum encoder He et al. (2020).

Let \(q\) be an anchor embedding, \(k^+\) its positive key, and \(\{k_j^-\}\) a queue of negative keys. A MoCo-style loss is

\[ L(q,k^+,\{k_j^-\}) = - \log \frac{ \exp(q^Tk^+/\tau) }{ \exp(q^Tk^+/\tau)+\sum_j\exp(q^Tk_j^-/\tau) }. \]

The key encoder parameters are updated by exponential moving average:

\[ \theta_k \leftarrow m\theta_k+(1-m)\theta_q, \qquad 0<m<1. \]

This stabilizes the dictionary of keys.

27.7 26.7 Alignment and uniformity

A useful geometric decomposition of contrastive representation quality is alignment and uniformity.

Alignment asks whether positive pairs are close:

\[ L_{\mathrm{align}} = \mathbb E_{(x,x^+)} \|\phi(x)-\phi(x^+)\|_2^2. \]

Uniformity asks whether embeddings are spread over the representation space. For normalized embeddings on the sphere, one measure is

\[ L_{\mathrm{uniform}} = \log \mathbb E_{x,y} \exp(-2\|\phi(x)-\phi(y)\|_2^2). \]

Alignment alone is not enough. The constant representation

\[ \phi(x)=c \qquad \text{for all }x \]

has perfect alignment, because all views are identical in representation space. But it is useless. Contrastive learning avoids this by using negative samples, which promote uniformity.

NoteRepresentation geometry

Good contrastive representations make positive pairs close while keeping the overall embedding distribution sufficiently spread out. This prevents collapse and improves downstream separability.

27.8 26.8 Collapse and why it happens

A representation collapses when many inputs map to the same or nearly the same vector. The most extreme case is

\[ \phi(x_1)=\phi(x_2)=\cdots=\phi(x_N)=c. \]

Collapse is a trivial solution for objectives that only reward agreement between views. For example, the loss

\[ L_{\mathrm{agree}} = \mathbb E \|\phi(T_1X)-\phi(T_2X)\|_2^2 \]

is minimized by any constant representation.

Anti-collapse mechanisms include:

  1. negative samples;
  2. variance constraints;
  3. covariance decorrelation;
  4. predictor networks;
  5. stop-gradient operations;
  6. teacher-student asymmetry;
  7. clustering with balancing constraints;
  8. reconstruction of masked content.

The mathematical question is: what prevents the representation covariance from degenerating?

Let \(Z\in\mathbb R^{B\times d}\) be a batch of embeddings. Collapse can be detected by the empirical covariance

\[ \widehat \Sigma = \frac1{B-1}(Z-\bar Z)^T(Z-\bar Z). \]

If many eigenvalues of \(\widehat\Sigma\) are near zero, the representation uses only a low-dimensional subspace. Complete collapse means \(\widehat\Sigma=0\).

27.9 26.9 Non-contrastive learning: BYOL-style objectives

BYOL removes explicit negative samples but uses two networks: an online network and a target network Grill et al. (2020). Let

\[ y_1=f_{\theta}(T_1x), \qquad y_2=f_{\xi}(T_2x), \]

where \(\theta\) are online parameters and \(\xi\) are target parameters. The online network also has a predictor \(q_\theta\). The loss is

\[ L = \left\| \frac{q_\theta(y_1)}{\|q_\theta(y_1)\|_2} - \operatorname{sg}\left(\frac{y_2}{\|y_2\|_2}\right) \right\|_2^2, \]

where \(\operatorname{sg}\) denotes stop-gradient. The target network is updated by exponential moving average:

\[ \xi\leftarrow m\xi+(1-m)\theta. \]

The stop-gradient and predictor introduce asymmetry. They help avoid collapse in practice, although the full theory is subtle.

27.9.1 Teacher-student interpretation

The target network acts as a slowly changing teacher. The online network learns to predict the teacher’s representation under another view. Because the teacher changes slowly, the optimization target is more stable than if both sides moved freely at the same time.

27.10 26.10 Redundancy reduction: Barlow Twins

Barlow Twins avoids collapse by encouraging two augmented views to have highly correlated corresponding dimensions and decorrelated different dimensions Zbontar et al. (2021).

Let \(Z^A,Z^B\in\mathbb R^{B\times d}\) be normalized batch embeddings from two views. Define the empirical cross-correlation matrix

\[ C_{ij} = \frac{\sum_{b=1}^B Z^A_{b i} Z^B_{b j}} {\sqrt{\sum_b (Z^A_{b i})^2}\sqrt{\sum_b (Z^B_{b j})^2}}. \]

The Barlow Twins objective is

\[ L_{\mathrm{BT}} = \sum_i(1-C_{ii})^2 + \lambda\sum_{i\ne j}C_{ij}^2. \]

The diagonal term encourages corresponding dimensions to agree. The off-diagonal term discourages redundant dimensions. If all dimensions collapse to a constant after normalization, the covariance and cross-correlation structure becomes degenerate.

This objective is attractive mathematically because it directly controls second-order statistics of the representation.

27.11 26.11 Variance-invariance-covariance regularization: VICReg

VICReg decomposes the self-supervised objective into three terms Bardes et al. (2022):

\[ L = \lambda L_{\mathrm{inv}} + \mu L_{\mathrm{var}} + \nu L_{\mathrm{cov}}. \]

The invariance term aligns positive pairs:

\[ L_{\mathrm{inv}} = \frac1B\sum_{b=1}^B \|z_b^A-z_b^B\|_2^2. \]

The variance term prevents each coordinate from collapsing:

\[ L_{\mathrm{var}} = \frac1d\sum_{j=1}^d \max(0,\gamma-\operatorname{Std}(Z_{:j})) \]

applied to both views. The covariance term reduces redundancy:

\[ L_{\mathrm{cov}} = \frac1d\sum_{i\ne j} \left[\operatorname{Cov}(Z)_{ij}\right]^2. \]

VICReg is useful pedagogically because it makes the anti-collapse logic explicit:

  1. make positive pairs similar;
  2. keep every coordinate variable;
  3. prevent coordinates from copying each other.

27.12 26.12 Clustering and teacher-student assignments

Another approach is to learn representations by assigning images to prototypes or clusters. Let

\[ C=\{c_1,\dots,c_K\}\subset \mathbb R^d \]

be prototype vectors. An image embedding \(z\) produces assignment probabilities

\[ p_k(z) = \frac{\exp(z^Tc_k/\tau)}{\sum_{j=1}^K\exp(z^Tc_j/\tau)}. \]

Clustering-based self-supervised learning tries to make different views of the same image have consistent assignments. Some methods also impose balance conditions so that all prototypes are used.

DINO uses a student-teacher framework with centering and sharpening to learn visual representations without labels Caron et al. (2021). In ViT models, the learned attention maps often show meaningful object regions even without segmentation labels.

The mathematical issue is again anti-collapse. If every image is assigned to the same prototype, the representation is useless. Balancing, centering, sharpening, teacher-student asymmetry, and entropy regularization all influence this problem.

27.13 26.13 Masked image modeling

Masked image modeling is especially natural for vision transformers. The image is divided into patches. A subset of patches is visible, and the model must predict the missing information.

Let

\[ x=(x_1,x_2,\dots,x_N) \]

be patch tokens. Let \(M\subset \{1,\dots,N\}\) be a random mask set. The visible patches are

\[ x_{\mathrm{vis}}=\{x_i:i\notin M\}, \]

and the hidden patches are

\[ x_{\mathrm{mask}}=\{x_i:i\in M\}. \]

A masked autoencoder learns an encoder-decoder pair

\[ \widehat x_M = D_\psi(E_\theta(x_{\mathrm{vis}}),M) \]

by minimizing a reconstruction loss such as

\[ L_{\mathrm{MAE}} = \frac1{|M|}\sum_{i\in M} \|\widehat x_i-x_i\|_2^2. \]

MAE uses a high masking ratio and reconstructs pixel patches He et al. (2022). The key idea is that the encoder only processes visible patches, so pretraining is computationally efficient.

27.13.1 Predicting pixels versus predicting tokens

There are several choices for the prediction target:

  1. raw pixel values;
  2. normalized patch pixels;
  3. discrete visual tokens;
  4. features from another model;
  5. latent representations.

Pixel reconstruction is easy to define but may emphasize low-level texture. Feature or token prediction can encourage more semantic representations, but the target construction becomes more complex.

27.13.2 Why masking can work

Images have spatial redundancy and semantic structure. If many patches are missing, the model must infer global context. This encourages representations that capture shape, object parts, layout, and texture regularities.

Masked image modeling is not simply denoising. The mask distribution defines which conditional prediction problem is learned:

\[ P(x_M\mid x_{\mathrm{vis}}). \]

A model trained this way learns features useful for downstream tasks if the conditional prediction task shares structure with those tasks.

27.14 26.14 Contrastive learning versus masked modeling

Contrastive learning and masked image modeling use different information sources.

Aspect Contrastive learning Masked image modeling
Main signal agreement across augmented views prediction of hidden patches
Positive relation two views of same image visible and hidden parts of same image
Anti-collapse mechanism negatives or distribution constraints reconstruction/prediction target
Strong assumption augmentations preserve semantics missing patches are predictable from context
Natural architecture CNNs or ViTs especially ViTs
Risk false negatives, augmentation bias low-level reconstruction bias

For classification, contrastive learning often gives strong global features. For dense prediction, masked modeling can preserve spatial information. In practice, modern systems often combine ideas from contrastive learning, masked modeling, distillation, and image-text supervision.

27.15 26.15 Self-supervised learning as latent-variable modeling

A statistical way to understand self-supervised vision is through latent variables. Suppose an image is generated from latent semantic and nuisance variables:

\[ X=G(S,N), \]

where \(S\) denotes semantic content and \(N\) denotes nuisance variation such as lighting, viewpoint, color shift, or background.

An augmentation \(T\) changes nuisance variables while preserving \(S\) approximately:

\[ T(G(S,N))\approx G(S,N'). \]

A useful invariant representation aims to estimate something like

\[ \phi(X)\approx r(S). \]

This is a modeling ideal, not a theorem in general. The success of self-supervised learning depends on whether the training objective makes \(S\) identifiable from the chosen prediction task.

27.15.1 Mutual information viewpoint

Contrastive losses are often related to mutual information between two views. Let

\[ V_1=T_1(X), \qquad V_2=T_2(X). \]

A contrastive objective encourages representations of \(V_1\) and \(V_2\) to retain shared information. Informally, it seeks large dependence between

\[ Z_1=\phi(V_1), \qquad Z_2=\phi(V_2), \]

while avoiding collapse. However, maximizing shared information is not the whole story. Some shared information may be low-level texture or imaging artifacts, and some task-relevant information may not be shared under aggressive augmentations.

27.16 26.16 Downstream evaluation

The quality of a self-supervised representation is not measured only by the pretraining loss. A low pretraining loss can still produce poor transfer.

Common evaluation protocols include:

  1. linear probing: freeze \(\phi_\theta\) and train a linear classifier;
  2. fine-tuning: initialize with pretrained weights and train on labeled data;
  3. few-shot evaluation: train with a small labeled dataset;
  4. nearest-neighbor classification: classify using similarity in feature space;
  5. retrieval: evaluate whether similar images are near each other;
  6. clustering: evaluate unsupervised cluster quality;
  7. dense transfer: use features for detection, segmentation, or depth;
  8. robustness: test under domain shift, corruption, or adversarial perturbation.

Linear probing is popular because it isolates whether the representation makes the classes linearly separable. But it is not the only meaningful evaluation. A representation may be excellent for segmentation and mediocre for global classification, or vice versa.

27.16.1 Linear probe objective

Given frozen features

\[ z_i=\phi_\theta(x_i), \]

train a linear classifier

\[ h_W(z)=Wz+b \]

by minimizing

\[ \min_{W,b} \frac1n\sum_{i=1}^n \operatorname{CE}(\operatorname{softmax}(Wz_i+b),y_i) + \lambda\|W\|_F^2. \]

If linear probe accuracy is high, the label information is accessible through a simple linear decision rule.

27.17 26.17 Practical design choices

Self-supervised vision methods require many design choices.

27.17.1 Encoder

The encoder may be a CNN, a vision transformer, or a hybrid architecture. CNNs have strong locality and translation bias. ViTs scale well with large data and are natural for patch masking.

27.17.2 Projection head

Contrastive and non-contrastive methods usually train a projection head

\[ g_\eta:\mathbb R^d\to \mathbb R^m \]

on top of the representation. The pretraining loss is applied to \(g_\eta(\phi_\theta(x))\), but downstream tasks often use \(\phi_\theta(x)\) before the projection head.

The projection head can absorb loss-specific distortions, leaving the backbone representation more transferable.

27.17.3 Normalization

Embeddings are often \(\ell_2\)-normalized for cosine similarity. Batch normalization, layer normalization, and centering also strongly affect representation statistics.

27.17.4 Batch size and memory

Contrastive learning benefits from many negative samples. Large batches, memory banks, and momentum queues are ways to approximate a large comparison set.

27.17.5 Augmentation strength

Too little augmentation may produce representations that memorize image-specific texture. Too much augmentation may destroy semantic information. The optimal augmentation distribution is task- and domain-dependent.

27.17.6 Masking ratio

In masked image modeling, the masking ratio controls difficulty. A low masking ratio may make reconstruction too easy. A high masking ratio forces global reasoning but may make the task unstable if little information remains.

27.18 26.18 Python example: augmentations and positive pairs

The following code creates two random views of the same image. It is intentionally simple so that the mathematical structure is visible.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(7243)

# Synthetic grayscale image with a bright object.
H, W = 96, 96
y, x = np.mgrid[0:H, 0:W]
image = np.exp(-((x - 48)**2 + (y - 48)**2) / (2 * 12**2))
image += 0.25 * np.exp(-((x - 25)**2 + (y - 65)**2) / (2 * 8**2))
image = image / image.max()

def random_crop_resize(img, crop_size=72, out_size=96):
    H, W = img.shape
    top = rng.integers(0, H - crop_size + 1)
    left = rng.integers(0, W - crop_size + 1)
    crop = img[top:top + crop_size, left:left + crop_size]

    # Simple nearest-neighbor resize for teaching.
    yy = (np.linspace(0, crop_size - 1, out_size)).astype(int)
    xx = (np.linspace(0, crop_size - 1, out_size)).astype(int)
    return crop[np.ix_(yy, xx)]

def color_jitter_gray(img, strength=0.25):
    scale = rng.uniform(1 - strength, 1 + strength)
    shift = rng.uniform(-strength, strength)
    return np.clip(scale * img + shift, 0, 1)

def make_view(img):
    view = random_crop_resize(img)
    view = color_jitter_gray(view)
    if rng.random() < 0.5:
        view = view[:, ::-1]
    return view

view1 = make_view(image)
view2 = make_view(image)

fig, axes = plt.subplots(1, 3, figsize=(9, 3))
for ax, arr, title in zip(axes, [image, view1, view2], ["original", "view 1", "view 2"]):
    ax.imshow(arr, cmap="gray", vmin=0, vmax=1)
    ax.set_title(title)
    ax.axis("off")
plt.show()

The two views are different arrays, but they are intended to represent the same underlying semantic object. The self-supervised loss will treat them as a positive pair.

27.19 26.19 Python example: InfoNCE from scratch

This code computes a small contrastive loss from two batches of embeddings.

import numpy as np

rng = np.random.default_rng(7243)
B, d = 8, 16

# Two noisy views of the same latent vectors.
latent = rng.normal(size=(B, d))
z1 = latent + 0.25 * rng.normal(size=(B, d))
z2 = latent + 0.25 * rng.normal(size=(B, d))

def normalize_rows(Z):
    return Z / np.linalg.norm(Z, axis=1, keepdims=True)

z1 = normalize_rows(z1)
z2 = normalize_rows(z2)

tau = 0.2
logits = (z1 @ z2.T) / tau

# Stable log-softmax.
logits_shifted = logits - logits.max(axis=1, keepdims=True)
log_probs = logits_shifted - np.log(np.exp(logits_shifted).sum(axis=1, keepdims=True))
loss = -np.mean(np.diag(log_probs))

print("Similarity matrix:")
print(np.round(z1 @ z2.T, 3))
print("InfoNCE loss:", loss)

The diagonal entries correspond to positive pairs. The off-diagonal entries correspond to negatives.

27.20 26.20 TensorFlow example: a small contrastive objective

The following minimal TensorFlow example shows the structure of a contrastive training step. It is not meant to be a state-of-the-art implementation.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

IMG_SIZE = 64
EMBED_DIM = 64

def make_encoder():
    inputs = keras.Input(shape=(IMG_SIZE, IMG_SIZE, 3))
    x = layers.Conv2D(32, 3, padding="same", activation="relu")(inputs)
    x = layers.MaxPooling2D()(x)
    x = layers.Conv2D(64, 3, padding="same", activation="relu")(x)
    x = layers.GlobalAveragePooling2D()(x)
    x = layers.Dense(128, activation="relu")(x)
    outputs = layers.Dense(EMBED_DIM)(x)
    return keras.Model(inputs, outputs)

encoder = make_encoder()
projection = keras.Sequential([
    layers.Dense(128, activation="relu"),
    layers.Dense(EMBED_DIM)
])

optimizer = keras.optimizers.Adam(1e-3)

def random_augment(x):
    x = tf.image.random_flip_left_right(x)
    x = tf.image.random_brightness(x, max_delta=0.2)
    x = tf.image.random_contrast(x, lower=0.8, upper=1.2)
    return tf.clip_by_value(x, 0.0, 1.0)

def contrastive_loss(z1, z2, tau=0.2):
    z1 = tf.math.l2_normalize(z1, axis=1)
    z2 = tf.math.l2_normalize(z2, axis=1)
    logits = tf.matmul(z1, z2, transpose_b=True) / tau
    labels = tf.range(tf.shape(z1)[0])
    loss_a = keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)
    loss_b = keras.losses.sparse_categorical_crossentropy(labels, tf.transpose(logits), from_logits=True)
    return 0.5 * (tf.reduce_mean(loss_a) + tf.reduce_mean(loss_b))

@tf.function
def train_step(images):
    with tf.GradientTape() as tape:
        v1 = random_augment(images)
        v2 = random_augment(images)
        z1 = projection(encoder(v1, training=True), training=True)
        z2 = projection(encoder(v2, training=True), training=True)
        loss = contrastive_loss(z1, z2)
    variables = encoder.trainable_variables + projection.trainable_variables
    grads = tape.gradient(loss, variables)
    optimizer.apply_gradients(zip(grads, variables))
    return loss

Important implementation checks:

  1. the labels are image identities within the batch;
  2. the diagonal of the similarity matrix contains positives;
  3. embeddings are normalized before cosine similarity;
  4. the loss is symmetrized over both views;
  5. augmentations should preserve the intended downstream semantics.

27.21 26.21 Python example: masked patches

The next example constructs a mask over image patches.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(7243)
H, W = 96, 96
P = 12
mask_ratio = 0.65

image = rng.random((H, W))
# Add coarse structure so masking is visually meaningful.
y, x = np.mgrid[0:H, 0:W]
image = 0.3 * image + 0.7 * np.exp(-((x - 48)**2 + (y - 48)**2) / (2 * 22**2))
image = image / image.max()

n_h, n_w = H // P, W // P
N = n_h * n_w
num_mask = int(mask_ratio * N)
mask_ids = rng.choice(N, size=num_mask, replace=False)

masked = image.copy()
for idx in mask_ids:
    r = idx // n_w
    c = idx % n_w
    masked[r*P:(r+1)*P, c*P:(c+1)*P] = 0.0

fig, axes = plt.subplots(1, 2, figsize=(6, 3))
axes[0].imshow(image, cmap="gray")
axes[0].set_title("original")
axes[0].axis("off")
axes[1].imshow(masked, cmap="gray", vmin=0, vmax=1)
axes[1].set_title(f"masked {mask_ratio:.0%}")
axes[1].axis("off")
plt.show()

In a masked autoencoder, the model sees only the visible patches and is trained to reconstruct the masked patches.

27.22 26.22 AI-assisted study: using language models responsibly

AI tools are useful for studying self-supervised learning because the tensor shapes and losses are easy to misread. Use them as mathematical assistants, not as authorities.

27.22.1 Prompt 1: check the InfoNCE formula

I have two batches of normalized embeddings z1 and z2, each with shape (B,d). I compute logits = z1 @ z2.T / tau and use labels 0,1,...,B-1. Explain why the diagonal entries are positives and the off-diagonal entries are negatives. Then derive the cross-entropy loss in mathematical notation.

27.22.2 Prompt 2: find collapse in a representation

Given a matrix of embeddings Z with shape (B,d), explain how to detect representation collapse using singular values, coordinate standard deviations, covariance eigenvalues, and pairwise cosine similarities. Provide Python code.

27.22.3 Prompt 3: analyze augmentation assumptions

For a medical image classification task, evaluate whether random crop, color jitter, Gaussian blur, rotation, and horizontal flip are likely to preserve the target label. State possible failure modes.

27.22.4 Prompt 4: compare contrastive and masked-image objectives

Compare InfoNCE and masked autoencoding as self-supervised objectives for satellite images. Discuss invariance assumptions, spatial information, dense prediction transfer, and possible shortcut solutions.

WarningVerification habit

When using AI tools, always verify tensor shapes, positive-pair indexing, stop-gradient placement, normalization, and whether augmentations preserve the scientific meaning of the image.

27.23 26.23 Interactive demo: temperature in contrastive learning

Move the temperature slider mentally by editing the value in the code if you place this in a standalone HTML page. Lower temperatures make the contrastive softmax sharper.

Contrastive softmax temperature

This demo compares the probability assigned to the positive pair as the temperature changes.

0.20

27.24 26.24 Interactive demo: alignment and uniformity

This demo shows two positive pairs in a two-dimensional representation space. Increasing alignment pulls positives together. Increasing uniformity pushes different image identities apart.

Alignment versus uniformity

0.70
0.70

27.25 26.25 Interactive demo: collapse and covariance eigenvalues

A healthy representation has nontrivial variance across many dimensions. A collapsed representation has near-zero covariance eigenvalues.

Representation collapse

0.20

27.26 26.26 Interactive demo: masking ratio

Masked image modeling becomes harder as the masking ratio increases. Too little masking gives an easy local interpolation task; too much masking may leave insufficient information.

Patch masking ratio

0.65

27.27 26.27 Summary

Self-supervised vision is representation learning from structure rather than manual labels. The main mathematical idea is to design a pretraining objective whose solution captures useful visual factors.

Contrastive methods use positive and negative pairs to align views and spread embeddings. Non-contrastive methods remove explicit negatives but must prevent collapse through asymmetry, variance control, covariance decorrelation, or teacher-student mechanisms. Masked image modeling trains a model to infer hidden patches from visible context, making it especially natural for vision transformers.

The most important modeling question is not only which loss is used. It is whether the induced prediction task preserves the information needed for the downstream scientific or engineering goal.

27.28 26.28 Exercises

27.28.1 Conceptual exercises

  1. Explain the difference between supervised, unsupervised, and self-supervised learning.
  2. Give three image augmentations that are reasonable for natural-image classification. Give one task where each might fail.
  3. Explain why the constant representation is a trivial minimizer of a pure positive-pair agreement loss.
  4. Compare invariance and equivariance for image classification, object detection, and semantic segmentation.
  5. Explain why linear probing is an imperfect but useful evaluation protocol.

27.28.2 Mathematical exercises

  1. Derive the gradient of the one-way InfoNCE loss with respect to the anchor embedding, assuming all embeddings are normalized and treating the candidate embeddings as constants.
  2. Show that if \(\phi(x)=c\) for all \(x\), then the alignment loss is zero.
  3. Let \(Z\in\mathbb R^{B\times d}\) be centered. Show that complete collapse implies \(Z^TZ=0\).
  4. For the Barlow Twins objective, explain why the identity cross-correlation matrix is desirable.
  5. For VICReg, explain how the variance term prevents coordinate-wise collapse.
  6. Suppose a masked image model uses squared reconstruction loss. Interpret this as maximum likelihood under a Gaussian conditional model.
  7. Let \(Z\) be row-normalized. Explain why cosine similarity equals the dot product.
  8. Show how the temperature parameter changes the softmax probabilities in a two-candidate contrastive problem.

27.28.3 Python exercises

  1. Implement random crop, color jitter, blur, and horizontal flip for a batch of images.
  2. Implement the SimCLR-style \(2B\times 2B\) contrastive loss.
  3. Generate synthetic embeddings and plot the eigenvalues of their empirical covariance as collapse increases.
  4. Implement Barlow Twins loss for two batches of embeddings.
  5. Implement VICReg’s invariance, variance, and covariance terms.
  6. Create a masked-patch visualization for a grayscale image.
  7. Train a tiny self-supervised encoder on synthetic shapes, then evaluate with a linear classifier.

27.28.4 AI-assisted exercises

  1. Ask an AI tool to explain the difference between InfoNCE, BYOL, Barlow Twins, VICReg, DINO, and MAE. Then verify the answer by writing each objective mathematically.
  2. Ask an AI tool to debug an incorrect contrastive loss where positives are not on the diagonal. Explain the bug in your own words.
  3. Ask an AI tool to suggest augmentations for satellite images, medical images, and handwritten digits. Critique the suggestions mathematically.
  4. Ask an AI tool to generate TensorFlow code for a masked autoencoder. Identify the encoder input shape, decoder input shape, mask shape, and reconstruction target.

27.29 Notes and references

Contrastive self-supervised vision is represented by SimCLR Chen et al. (2020) and MoCo He et al. (2020). Non-contrastive and redundancy-reduction approaches include BYOL Grill et al. (2020), Barlow Twins Zbontar et al. (2021), VICReg Bardes et al. (2022), and DINO Caron et al. (2021). Masked image modeling for vision transformers is represented by MAE He et al. (2022). This chapter also connects back to representation learning in Chapter 20, CNNs in Chapter 21, training methods in Chapter 22, and vision transformers in Chapter 25.

Bardes, Adrien, Jean Ponce, and Yann LeCun. 2022. “VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning.” International Conference on Learning Representations.
Caron, Mathilde, Hugo Touvron, Ishan Misra, et al. 2021. “Emerging Properties in Self-Supervised Vision Transformers.” IEEE/CVF International Conference on Computer Vision, 9650–60.
Chen, Ting, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. “A Simple Framework for Contrastive Learning of Visual Representations.” International Conference on Machine Learning, 1597–607.
Grill, Jean-Bastien, Florian Strub, Florent Altché, et al. 2020. “Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning.” Advances in Neural Information Processing Systems 33: 21271–84.
He, Kaiming, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. “Masked Autoencoders Are Scalable Vision Learners.” IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16000–16009.
He, Kaiming, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. “Momentum Contrast for Unsupervised Visual Representation Learning.” IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9729–38.
Zbontar, Jure, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. 2021. “Barlow Twins: Self-Supervised Learning via Redundancy Reduction.” International Conference on Machine Learning, 12310–20.