28 Vision-Language Models

Chapter 27 studies vision-language models from a mathematical point of view. The central problem is to align two different data types—images and text—inside a common representation space so that classification, retrieval, captioning, visual question answering, and multimodal reasoning become geometric or probabilistic operations.

Vision-language learning is one of the clearest examples of modern computer vision as applied mathematics and statistics. An image is not only an array of pixels; it can be paired with a sentence, a category name, a report, a caption, a prompt, or an answer. The model must learn a map from images and a map from text so that semantically related inputs become close.

The chapter emphasizes three mathematical themes.

Geometry: image and text embeddings live in a shared vector space, often on a high-dimensional unit sphere.
Probability: contrastive learning defines a conditional distribution over matching text for each image and over matching images for each text.
Optimization: training is large-scale empirical risk minimization with a temperature-scaled softmax loss.

Classic computer vision asks:

\[ \text{What object is in this image?} \]

Vision-language modeling asks a broader question:

\[ \text{How should visual evidence and linguistic meaning be represented together?} \]

28.1 Learning Goals

After reading this chapter, students should be able to:

describe image-text data as paired observations $(X_i,T_i)$;
define image encoders, text encoders, normalized embeddings, and similarity matrices;
derive the symmetric contrastive image-text loss used in CLIP-style models;
explain zero-shot image classification as nearest-text-prototype classification;
interpret retrieval, captioning, and visual question answering as different probabilistic or geometric tasks;
analyze the role of temperature, normalization, batch size, and negative examples;
connect vision-language models to representation learning, transformers, foundation models, and promptable vision systems;
identify statistical risks such as dataset bias, spurious text-image associations, leakage, poor calibration, and distribution shift.

28.2 27.1 From Image Classification to Image-Text Alignment

In a standard supervised image-classification problem, one observes pairs

\[ (X_i,Y_i), \qquad i=1,\dots,n, \]

where $X_i$ is an image and $Y_i \in \{1,\dots,K\}$ is a class label. The target is a classifier

\[ h_\theta : \mathcal X \to \{1,\dots,K\}. \]

This formulation is powerful but restrictive. Labels are fixed, finite, and usually manually defined. A model trained on cats, dogs, cars, and airplanes cannot automatically recognize a new class such as dermatology image, Romanesque arch, or microscopic yeast colony unless that class was included in the training label set.

Vision-language learning replaces the finite label $Y_i$ by a text object $T_i$:

\[ (X_i,T_i), \qquad X_i \in \mathcal X, \quad T_i \in \mathcal T. \]

Here $T_i$ may be a caption, a class name, an alt-text string, a medical report, or a natural-language prompt. Instead of learning a map from images to a fixed label set, the goal is to learn two encoders:

\[ f_\theta : \mathcal X \to \mathbb R^d, \qquad g_\phi : \mathcal T \to \mathbb R^d. \]

The image encoder and text encoder produce vectors in the same representation space. For an image $x$ and text $t$, define

\[ z_x = f_\theta(x), \qquad z_t = g_\phi(t). \]

Usually the embeddings are normalized:

\[ \bar z_x = \frac{z_x}{\|z_x\|_2}, \qquad \bar z_t = \frac{z_t}{\|z_t\|_2}. \]

Then similarity is often measured by cosine similarity:

\[ s(x,t)=\langle \bar z_x,\bar z_t\rangle. \]

If $s(x,t)$ is large, the image and text are considered semantically compatible.

A vision-language model turns semantic matching into geometry:

\[ \text{image matches text} \quad \Longleftrightarrow \quad \langle \bar z_x,\bar z_t\rangle \text{ is large}. \]

CLIP-style contrastive learning is a central example of this idea (Radford et al. 2021).

28.3 27.2 Mathematical Objects

The main mathematical objects in this chapter are:

Object	Meaning
$X$	image-valued random variable
$T$	text-valued random variable
$f_\theta$	image encoder
$g_\phi$	text encoder
$z_x,z_t \in \mathbb R^d$	image and text embeddings
$S \in \mathbb R^{m\times m}$	image-text similarity matrix inside a minibatch
$\tau>0$	temperature parameter
$p_\theta(t_j\mid x_i)$	model probability that text $t_j$ matches image $x_i$
$\ell_{IT},\ell_{TI}$	image-to-text and text-to-image contrastive losses
$\mathcal P$	prompt set or class-name set

The basic unit of training is a minibatch of matched pairs

\[ \mathcal B=\{(x_1,t_1),\dots,(x_m,t_m)\}. \]

The diagonal pairs $(x_i,t_i)$ are treated as positives. The off-diagonal pairs $(x_i,t_j)$, $j\neq i$, are treated as negatives within the minibatch.

28.4 27.3 Two-Tower Architecture

A common architecture is the two-tower model:

\[ \begin{aligned} Z_X &= f_\theta(X),\\ Z_T &= g_\phi(T),\\ S_{ij} &= \frac{1}{\tau}\langle \bar z_{x_i},\bar z_{t_j}\rangle. \end{aligned} \]

The image tower may be a CNN or a vision transformer. The text tower is usually a transformer language encoder. The two towers interact only through the similarity matrix during contrastive training.

This architecture has an important computational advantage. Since images and texts are encoded separately, one can precompute text embeddings for candidate prompts and image embeddings for databases. Retrieval can then be performed by nearest-neighbor search.

28.4.1 Matrix Form

Let

\[ A=\begin{bmatrix} \bar z_{x_1}^T\\ \vdots\\ \bar z_{x_m}^T \end{bmatrix}\in \mathbb R^{m\times d}, \qquad B=\begin{bmatrix} \bar z_{t_1}^T\\ \vdots\\ \bar z_{t_m}^T \end{bmatrix}\in \mathbb R^{m\times d}. \]

The minibatch similarity matrix is

\[ S=\frac{1}{\tau}AB^T. \]

The diagonal entry $S_{ii}$ measures the similarity of a true image-text pair. The off-diagonal entry $S_{ij}$ measures the similarity between image $i$ and a mismatched text $j$.

This is useful for applied mathematics students because the whole training objective can be read from one matrix.

28.4.2 Interactive Demo 27.1: Image-Text Embedding Geometry

The demo below shows image and text embeddings on the unit circle. A good vision-language representation places matched image and text points close together and mismatched points far apart.

28.5 27.4 Contrastive Image-Text Learning

Given a minibatch of $m$ matched pairs, define the similarity score

\[ S_{ij}=\frac{1}{\tau}\langle \bar z_{x_i},\bar z_{t_j}\rangle. \]

For each image $x_i$, the model defines a probability distribution over candidate texts in the minibatch:

\[ p_\theta(t_j\mid x_i) = \frac{\exp(S_{ij})}{\sum_{k=1}^m \exp(S_{ik})}. \]

The correct text is $t_i$. Therefore the image-to-text loss is

\[ \ell_{IT}(i) = -\log p_\theta(t_i\mid x_i) = -S_{ii}+\log\left(\sum_{k=1}^m \exp(S_{ik})\right). \]

Similarly, for each text $t_i$, define a probability distribution over candidate images:

\[ p_\theta(x_j\mid t_i) = \frac{\exp(S_{ji})}{\sum_{k=1}^m \exp(S_{ki})}. \]

The text-to-image loss is

\[ \ell_{TI}(i) = -\log p_\theta(x_i\mid t_i) = -S_{ii}+\log\left(\sum_{k=1}^m \exp(S_{ki})\right). \]

The symmetric contrastive loss is

\[ L = \frac{1}{2m}\sum_{i=1}^m \left[\ell_{IT}(i)+\ell_{TI}(i)\right]. \]

The loss is simply cross-entropy applied twice: once row-wise and once column-wise to the similarity matrix $S$.

28.5.1 Why the Loss Works

The term $-S_{ii}$ encourages the matched image-text similarity to increase. The log-sum-exp term penalizes high similarity between mismatched pairs. Thus the loss encourages

\[ S_{ii} > S_{ij}, \quad j\neq i, \qquad S_{ii} > S_{ji}, \quad j\neq i. \]

Geometrically, each matched pair should be close, and each mismatched pair should be separated by angular distance.

28.5.2 Gradient Intuition

For the image-to-text loss, the derivative with respect to a similarity score is

\[ \frac{\partial \ell_{IT}(i)}{\partial S_{ij}} = p_\theta(t_j\mid x_i)-1_{j=i}. \]

Thus:

if $j=i$, the gradient is $p_\theta(t_i\mid x_i)-1$, usually negative, so optimization increases $S_{ii}$;
if $j\neq i$, the gradient is $p_\theta(t_j\mid x_i)$, positive, so optimization decreases confusing mismatched similarities.

This is the same softmax-cross-entropy geometry used in classification, but the classes are texts inside the minibatch.

28.5.3 Interactive Demo 27.2: Temperature in Contrastive Softmax

The temperature $\tau$ controls how sharp the softmax distribution is. Small $\tau$ makes the model focus on the hardest negatives; large $\tau$ spreads probability more evenly.

28.6 27.5 The Role of Normalization and the Unit Sphere

Most CLIP-style models normalize embeddings before computing similarity. This means

\[ \|\bar z_x\|_2=\|\bar z_t\|_2=1. \]

Therefore

\[ \langle \bar z_x,\bar z_t\rangle=\cos(\alpha), \]

where $\alpha$ is the angle between the image and text embeddings.

The representation space is effectively the unit sphere

\[ S^{d-1}=\{z\in\mathbb R^d: \|z\|_2=1\}. \]

This has several consequences.

Similarity depends on angle, not vector length.
The model cannot trivially reduce the loss by increasing embedding norms.
Retrieval can use maximum inner product or nearest-neighbor search on normalized vectors.
The temperature parameter becomes especially important because cosine similarities lie in $[-1,1]$.

In high dimension, random unit vectors are nearly orthogonal. If $u,v$ are independently sampled from the unit sphere, then

\[ E\langle u,v\rangle =0, \qquad \operatorname{Var}(\langle u,v\rangle)=\frac{1}{d}. \]

Thus a high-dimensional embedding space provides many nearly orthogonal directions. Contrastive learning tries to use this geometry to separate many semantic concepts.

28.7 27.6 Zero-Shot Classification

A remarkable consequence of image-text alignment is zero-shot classification. Suppose the class names are

\[ \mathcal C=\{c_1,\dots,c_K\}. \]

For each class, construct a text prompt such as

\[ t_k = \text{``a photo of a } c_k \text{''}. \]

Compute text prototypes

\[ u_k = \frac{g_\phi(t_k)}{\|g_\phi(t_k)\|_2}. \]

Given a new image $x$, compute

\[ z=\frac{f_\theta(x)}{\|f_\theta(x)\|_2}. \]

The zero-shot classifier is

\[ \widehat y(x)=\arg\max_{1\le k\le K}\langle z,u_k\rangle. \]

This is nearest-prototype classification in a text-defined semantic space. No image examples from the new classes are required at inference time.

28.7.1 Softmax Version

One may also define class probabilities:

\[ p(y=k\mid x) = \frac{\exp(\langle z,u_k\rangle/\tau)} {\sum_{j=1}^K \exp(\langle z,u_j\rangle/\tau)}. \]

These probabilities are often useful for ranking classes, but they should not automatically be trusted as calibrated probabilities. Calibration depends on the training data, prompts, temperature, and distribution shift.

28.7.2 Interactive Demo 27.3: Zero-Shot Classification by Text Prototypes

The image embedding is classified by whichever text prototype has largest cosine similarity.

28.8 27.7 Prompt Engineering and Prompt Ensembling

A class name alone may be ambiguous. For example, the text string “crane” may refer to a bird or a machine. Vision-language systems often use prompts:

\[ \text{``a photo of a crane''}, \qquad \text{``a photo of a construction crane''}, \qquad \text{``a photo of a crane bird''}. \]

For a class $c_k$, let there be $r$ prompt templates

\[ t_{k1},\dots,t_{kr}. \]

Prompt ensembling uses the average text embedding

\[ u_k = \frac{\sum_{a=1}^r \bar g_\phi(t_{ka})}{\left\|\sum_{a=1}^r \bar g_\phi(t_{ka})\right\|_2}. \]

The classifier becomes

\[ \widehat y(x)=\arg\max_k \langle \bar f_\theta(x),u_k\rangle. \]

Mathematically, prompt ensembling reduces sensitivity to a single text description. Statistically, it is a kind of model averaging over linguistic descriptions.

28.8.1 Interactive Demo 27.4: Prompt Ensembling

Different prompts for the same class can produce slightly different text embeddings. Averaging them gives a more stable class prototype.

28.9 27.8 Image-Text Retrieval

Vision-language models naturally support retrieval.

28.9.1 Text-to-Image Retrieval

Given a text query $t$, rank images $x_1,\dots,x_N$ by

\[ s(x_i,t)=\langle \bar f_\theta(x_i),\bar g_\phi(t)\rangle. \]

The top-ranked images are

\[ \operatorname{TopK}\{s(x_i,t): i=1,\dots,N\}. \]

28.9.2 Image-to-Text Retrieval

Given an image $x$, rank texts $t_1,\dots,t_M$ by

\[ s(x,t_j)=\langle \bar f_\theta(x),\bar g_\phi(t_j)\rangle. \]

This may retrieve captions, labels, reports, or natural-language descriptions.

28.9.3 Evaluation Metrics

Common retrieval metrics include:

\[ \operatorname{Recall@K} = \frac{1}{n}\sum_{i=1}^n 1\{\text{true match for query }i\text{ appears in top }K\}, \]

and mean reciprocal rank:

\[ \operatorname{MRR} = \frac{1}{n}\sum_{i=1}^n \frac{1}{\operatorname{rank}_i}. \]

Retrieval is sensitive to the geometry of the embedding space. A small number of “hub” images or texts may appear near many queries. This is a high-dimensional nearest-neighbor phenomenon and should be checked empirically.

28.9.4 Interactive Demo 27.5: Retrieval Rankings

The bars show similarity scores between one text query and a collection of images. Retrieval quality depends on whether the correct match is ranked near the top.

28.10 27.9 Captioning, VQA, and Cross-Attention

The two-tower contrastive model is not the only form of vision-language learning. For tasks requiring generated text, the model needs more interaction between visual tokens and language tokens.

28.10.1 Image Captioning

Image captioning models the conditional distribution

\[ p(w_1,\dots,w_L\mid x) = \prod_{\ell=1}^L p(w_\ell\mid w_1,\dots,w_{\ell-1},x), \]

where $w_1,\dots,w_L$ are caption tokens. A vision encoder provides visual features, and a language decoder generates text autoregressively.

28.10.2 Visual Question Answering

Visual question answering models

\[ p(a\mid x,q), \]

where $x$ is an image, $q$ is a question, and $a$ is an answer. The model must combine visual evidence and linguistic context.

28.10.3 Cross-Attention

Suppose visual tokens are

\[ V=[v_1,\dots,v_m]^T \in \mathbb R^{m\times d}, \]

and text tokens are

\[ W=[w_1,\dots,w_L]^T \in \mathbb R^{L\times d}. \]

A cross-attention layer may let text tokens attend to visual tokens:

\[ \operatorname{CrossAttn}(W,V) = \operatorname{softmax}\left(\frac{(WQ)(VK)^T}{\sqrt{d_k}}\right)(VV). \]

Here the query comes from language and the keys/values come from vision. Cross-attention is one mathematical mechanism for fusing modalities.

28.10.4 Interactive Demo 27.6: Cross-Attention Map

Rows represent text tokens and columns represent image patches. Brighter entries correspond to stronger attention from a word to a patch.

28.11 27.10 Generative Vision-Language Models

A contrastive model scores image-text compatibility. A generative vision-language model can produce text from images, images from text, or decisions from multimodal context.

Examples of mathematical tasks include:

28.11.1 Text Given Image

\[ p_\theta(t\mid x). \]

This includes captioning, report generation, and image explanation.

28.11.2 Image Given Text

\[ p_\theta(x\mid t). \]

This includes text-to-image generation and controlled image editing.

28.11.3 Answer Given Image and Question

\[ p_\theta(a\mid x,q). \]

This includes visual question answering and multimodal assistant behavior.

28.11.4 Joint Modeling

A joint model tries to represent

\[ p_\theta(x,t) = p_\theta(t\mid x)p_\theta(x) = p_\theta(x\mid t)p_\theta(t). \]

In practice, different models approximate different pieces of this joint distribution. CLIP-style models learn a shared embedding and compatibility score. Captioning models learn $p(t\mid x)$. Diffusion models conditioned on text learn a generative approximation to $p(x\mid t)$. Multimodal language models combine visual encoders with language models to answer questions, reason about images, and produce text.

28.12 27.11 Statistical Interpretation

Vision-language learning can be described as statistical estimation from paired multimodal samples:

\[ (X_i,T_i) \sim P_{XT}, \qquad i=1,\dots,n. \]

The training objective encourages high similarity for samples from the joint distribution $P_{XT}$ and low similarity for mismatched samples that approximate the product distribution $P_XP_T$.

Informally, contrastive learning tries to distinguish

\[ (X,T)\sim P_{XT} \]

from

\[ (X,T)\sim P_XP_T. \]

This gives a connection to density-ratio estimation. A good compatibility score behaves like

\[ s(x,t) \approx \log\frac{p(x,t)}{p(x)p(t)} + \text{constant}. \]

This interpretation is useful because it shows that contrastive learning is not only a classification trick. It is a way to estimate statistical dependence between image and text.

28.13 27.12 Bias, Calibration, and Distribution Shift

Vision-language models are trained on large image-text corpora. These corpora may contain noise, bias, spurious correlations, duplicated examples, and ambiguous captions.

Important statistical issues include:

Noisy positives. A caption may not describe the image accurately.
False negatives. Two different captions in a minibatch may both describe the same image concept.
Prompt sensitivity. Predictions can change when the wording changes.
Class imbalance. Some visual concepts appear far more often than others.
Shortcut learning. The model may rely on background, style, watermark, or text artifacts.
Poor calibration. Softmax scores over prompts may not be reliable probabilities.
Distribution shift. Performance may degrade on medical, scientific, historical, satellite, or low-resource visual domains.

For MA Applied Math and MS Statistics students, these are not merely ethical concerns. They are statistical modeling issues about sampling, measurement, identifiability, and external validity.

28.14 27.13 Mathematical Connection to Earlier Chapters

Vision-language models combine many ideas from earlier chapters.

Earlier topic	Role in vision-language models
Linear algebra	embeddings, dot products, similarity matrices
Probability	conditional distributions, contrastive estimation, calibration
Optimization	large-scale softmax loss, SGD/Adam, temperature learning
CNNs and ViTs	image encoders
Transformers	text encoders and multimodal fusion
Representation learning	semantic embedding spaces
Classification	zero-shot nearest-prototype classification
Retrieval	nearest-neighbor search in embedding space
Segmentation/foundation models	text and point prompts for masks
Diffusion	text-conditioned image generation

This makes vision-language modeling a natural capstone topic for the modern part of the book.

28.15 27.14 Python Example: A Tiny Synthetic Contrastive Batch

The following code constructs a minibatch similarity matrix and computes the symmetric contrastive loss. It is intentionally small so that students can inspect every object.

import numpy as np

rng = np.random.default_rng(7243)

m = 6       # minibatch size
D = 4       # embedding dimension
tau = 0.07

# Simulate matched image/text embeddings with shared latent semantic directions.
latent = rng.normal(size=(m, D))
image = latent + 0.15 * rng.normal(size=(m, D))
text = latent + 0.15 * rng.normal(size=(m, D))

def normalize(Z):
    return Z / np.linalg.norm(Z, axis=1, keepdims=True)

A = normalize(image)
B = normalize(text)

S = (A @ B.T) / tau

def row_cross_entropy(S):
    # Correct label for row i is column i.
    S_shift = S - S.max(axis=1, keepdims=True)
    log_probs = S_shift - np.log(np.exp(S_shift).sum(axis=1, keepdims=True))
    return -np.diag(log_probs).mean()

loss_image_to_text = row_cross_entropy(S)
loss_text_to_image = row_cross_entropy(S.T)
loss = 0.5 * (loss_image_to_text + loss_text_to_image)

print("similarity matrix")
print(np.round(S, 2))
print("image-to-text loss:", loss_image_to_text)
print("text-to-image loss:", loss_text_to_image)
print("symmetric loss:", loss)

The diagonal entries of the similarity matrix should be relatively large when the paired embeddings are close.

28.16 27.15 Python Example: Zero-Shot Classification with Text Prototypes

import numpy as np

rng = np.random.default_rng(5010)

classes = ["cat", "dog", "car", "flower"]
d = 8

# Pretend these are normalized text embeddings from prompts.
text_proto = rng.normal(size=(len(classes), d))
text_proto = text_proto / np.linalg.norm(text_proto, axis=1, keepdims=True)

# Create an image embedding close to the dog prototype.
true_class = 1
image_emb = text_proto[true_class] + 0.25 * rng.normal(size=d)
image_emb = image_emb / np.linalg.norm(image_emb)

scores = text_proto @ image_emb
pred = int(np.argmax(scores))

for c, s in zip(classes, scores):
    print(f"{c:>8s}: {s:.3f}")

print("predicted class:", classes[pred])

This example shows the mathematical essence of zero-shot classification. In practice, the embeddings come from a trained image encoder and text encoder.

28.17 27.16 Python Example: Retrieval Metrics

import numpy as np

rng = np.random.default_rng(7339)

n = 10
d = 16
latent = rng.normal(size=(n, d))
image = latent + 0.4 * rng.normal(size=(n, d))
text = latent + 0.4 * rng.normal(size=(n, d))

def normalize(Z):
    return Z / np.linalg.norm(Z, axis=1, keepdims=True)

image = normalize(image)
text = normalize(text)

S = text @ image.T    # row = text query, column = image candidate
ranks = []
for i in range(n):
    ranking = np.argsort(-S[i])
    rank = int(np.where(ranking == i)[0][0]) + 1
    ranks.append(rank)

ranks = np.array(ranks)
recall_at_1 = np.mean(ranks <= 1)
recall_at_5 = np.mean(ranks <= 5)
mrr = np.mean(1 / ranks)

print("ranks:", ranks)
print("Recall@1:", recall_at_1)
print("Recall@5:", recall_at_5)
print("MRR:", mrr)

28.18 27.17 TensorFlow Example: Minimal Dual Encoder

The code below gives a minimal dual-encoder skeleton. It is not meant to be a production CLIP model; it is a transparent implementation of the mathematical loss.

import tensorflow as tf

class TinyDualEncoder(tf.keras.Model):
    def __init__(self, image_dim, text_dim, embed_dim=128):
        super().__init__()
        self.image_encoder = tf.keras.Sequential([
            tf.keras.layers.Input(shape=(image_dim,)),
            tf.keras.layers.Dense(256, activation="relu"),
            tf.keras.layers.Dense(embed_dim),
        ])
        self.text_encoder = tf.keras.Sequential([
            tf.keras.layers.Input(shape=(text_dim,)),
            tf.keras.layers.Dense(256, activation="relu"),
            tf.keras.layers.Dense(embed_dim),
        ])
        self.logit_scale = tf.Variable(tf.math.log(1 / 0.07), trainable=True)

    def call(self, image_features, text_features):
        zi = self.image_encoder(image_features)
        zt = self.text_encoder(text_features)
        zi = tf.math.l2_normalize(zi, axis=1)
        zt = tf.math.l2_normalize(zt, axis=1)
        logits = tf.exp(self.logit_scale) * tf.matmul(zi, zt, transpose_b=True)
        return logits


def symmetric_contrastive_loss(logits):
    batch_size = tf.shape(logits)[0]
    labels = tf.range(batch_size)
    loss_i2t = tf.keras.losses.sparse_categorical_crossentropy(
        labels, logits, from_logits=True
    )
    loss_t2i = tf.keras.losses.sparse_categorical_crossentropy(
        labels, tf.transpose(logits), from_logits=True
    )
    return 0.5 * (tf.reduce_mean(loss_i2t) + tf.reduce_mean(loss_t2i))

# Example usage with precomputed image/text feature vectors.
model = TinyDualEncoder(image_dim=512, text_dim=300, embed_dim=128)
optimizer = tf.keras.optimizers.AdamW(learning_rate=1e-4, weight_decay=1e-4)

image_batch = tf.random.normal((32, 512))
text_batch = tf.random.normal((32, 300))

with tf.GradientTape() as tape:
    logits = model(image_batch, text_batch)
    loss = symmetric_contrastive_loss(logits)

grads = tape.gradient(loss, model.trainable_variables)
optimizer.apply_gradients(zip(grads, model.trainable_variables))
print(float(loss))

28.19 27.18 AI-Assisted Study Prompts

Use AI tools carefully as mathematical assistants, not as substitutes for derivation.

28.19.1 Prompt 1: Deriving the Contrastive Loss

Explain why the CLIP-style image-to-text loss is a cross-entropy loss over rows of the similarity matrix. Then derive the gradient with respect to $S_{ij}$.

28.19.2 Prompt 2: Checking Dimensions

Given image embeddings $A\in\mathbb R^{m\times d}$ and text embeddings $B\in\mathbb R^{m\times d}$, check all dimensions in the formula $S=AB^T/\tau$ and explain why row-wise and column-wise softmax are both needed.

28.19.3 Prompt 3: Statistical Failure Modes

Give three examples where image-text contrastive learning can fail because of biased or noisy training data. For each, describe a statistical diagnostic that could reveal the problem.

28.19.4 Prompt 4: Zero-Shot Classification

Compare zero-shot classification by text prototypes with ordinary softmax classification trained on fixed labels. Explain the advantage and the statistical risk.

28.20 27.19 Exercises

28.20.1 Conceptual Exercises

Explain why a finite-label classifier cannot directly solve open-vocabulary recognition.
Why are image and text embeddings normalized before computing cosine similarity?
What is the role of the temperature parameter $\tau$?
Explain the difference between image-text retrieval and image classification.
Why can prompt wording affect zero-shot classification results?
Explain why captioning requires a different architecture from pure contrastive retrieval.
Give one example of a false negative in image-text contrastive learning.

28.20.2 Mathematical Exercises

Let $S\in\mathbb R^{m\times m}$ be a similarity matrix. Derive the row-wise contrastive loss and show that its gradient with respect to $S_{ij}$ is $p_{ij}-1_{j=i}$.
Suppose $u,v\in S^{d-1}$. Show that $\|u-v\|_2^2=2-2\langle u,v\rangle$. Conclude that maximizing cosine similarity is equivalent to minimizing Euclidean distance on the unit sphere.
For class prompts $u_1,\dots,u_K$, derive the zero-shot softmax probability $p(y=k\mid x)$ and discuss how the temperature changes the entropy of the distribution.
Let $A,B\in\mathbb R^{m\times d}$ have unit-norm rows. Show that the similarity matrix satisfies $S_{ij}\in[-1/\tau,1/\tau]$.
Derive Recall@K and MRR for a retrieval system with ranks $r_1,\dots,r_n$.
Show that averaging several normalized prompt embeddings and then renormalizing gives a unit vector. Explain why this may reduce prompt variance.

28.20.3 Python Exercises

Implement the symmetric contrastive loss using only NumPy.
Simulate image and text embeddings with increasing noise. Plot Recall@1 as a function of noise level.
Compare zero-shot classification using a single prompt per class versus five prompt templates per class.
Compute a similarity matrix and visualize it as a heatmap. Explain what a good diagonal pattern means.
Implement temperature scaling for zero-shot class probabilities and plot entropy as a function of temperature.

28.20.4 AI and Modeling Exercises

Design a small vision-language experiment for a scientific image dataset. What would the image inputs be? What would the text descriptions be?
Suppose a model classifies an X-ray using prompts written for natural photos. Explain why this is a domain-shift problem.
Propose a diagnostic for detecting whether a vision-language model is using background context rather than object evidence.
Compare CLIP-style contrastive learning with caption generation. Which is more appropriate for image search? Which is more appropriate for medical report drafting?
Explain how a text prompt can be used to guide segmentation or image generation.

28.21 27.20 Chapter Summary

Vision-language models connect images and text through a shared representation space. The central mathematical construction is a pair of encoders

\[ f_\theta:\mathcal X\to\mathbb R^d, \qquad g_\phi:\mathcal T\to\mathbb R^d, \]

together with normalized embeddings and cosine similarity. CLIP-style training forms a minibatch similarity matrix

\[ S=\frac{1}{\tau}AB^T \]

and applies cross-entropy row-wise and column-wise. Once trained, the same embedding space supports zero-shot classification, image-text retrieval, prompt-based recognition, and multimodal transfer.

The most important mathematical lesson is that natural language can define visual categories. Instead of learning only a fixed classifier, a vision-language model learns a geometry in which images and text descriptions can be compared directly.

28.22 Notes and References

CLIP-style image-text contrastive learning was introduced as a scalable approach to transferable visual representations in Radford et al. (2021). The transformer architecture underlying many text encoders and multimodal fusion modules is discussed in Vaswani et al. (2017), and vision transformers were introduced for image recognition in Dosovitskiy et al. (2020). The broader representation-learning ideas connect to contrastive learning and self-supervision from Chapter 26. Vision-language modeling also provides the conceptual bridge from image recognition to promptable segmentation and multimodal foundation models in Chapter 28.

Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, et al. 2020. “An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.” arXiv Preprint arXiv:2010.11929. https://arxiv.org/abs/2010.11929.

Radford, Alec, Jong Wook Kim, Chris Hallacy, et al. 2021. “Learning Transferable Visual Models from Natural Language Supervision.” International Conference on Machine Learning. https://arxiv.org/abs/2103.00020.

Vaswani, Ashish, Noam Shazeer, Niki Parmar, et al. 2017. “Attention Is All You Need.” Advances in Neural Information Processing Systems 30. https://arxiv.org/abs/1706.03762.

# Vision-Language Models ::: {.hero-box} Chapter 27 studies **vision-language models** from a mathematical point of view. The central problem is to align two different data types---images and text---inside a common representation space so that classification, retrieval, captioning, visual question answering, and multimodal reasoning become geometric or probabilistic operations. ::: Vision-language learning is one of the clearest examples of modern computer vision as applied mathematics and statistics. An image is not only an array of pixels; it can be paired with a sentence, a category name, a report, a caption, a prompt, or an answer. The model must learn a map from images and a map from text so that semantically related inputs become close. The chapter emphasizes three mathematical themes. 1. **Geometry:** image and text embeddings live in a shared vector space, often on a high-dimensional unit sphere. 2. **Probability:** contrastive learning defines a conditional distribution over matching text for each image and over matching images for each text. 3. **Optimization:** training is large-scale empirical risk minimization with a temperature-scaled softmax loss. Classic computer vision asks: $$ \text{What object is in this image?} $$ Vision-language modeling asks a broader question: $$ \text{How should visual evidence and linguistic meaning be represented together?} $$ ## Learning Goals After reading this chapter, students should be able to: 1. describe image-text data as paired observations $(X_i,T_i)$; 2. define image encoders, text encoders, normalized embeddings, and similarity matrices; 3. derive the symmetric contrastive image-text loss used in CLIP-style models; 4. explain zero-shot image classification as nearest-text-prototype classification; 5. interpret retrieval, captioning, and visual question answering as different probabilistic or geometric tasks; 6. analyze the role of temperature, normalization, batch size, and negative examples; 7. connect vision-language models to representation learning, transformers, foundation models, and promptable vision systems; 8. identify statistical risks such as dataset bias, spurious text-image associations, leakage, poor calibration, and distribution shift. ## 27.1 From Image Classification to Image-Text Alignment In a standard supervised image-classification problem, one observes pairs $$ (X_i,Y_i), \qquad i=1,\dots,n, $$ where $X_i$ is an image and $Y_i \in \{1,\dots,K\}$ is a class label. The target is a classifier $$ h_\theta : \mathcal X \to \{1,\dots,K\}. $$ This formulation is powerful but restrictive. Labels are fixed, finite, and usually manually defined. A model trained on cats, dogs, cars, and airplanes cannot automatically recognize a new class such as *dermatology image*, *Romanesque arch*, or *microscopic yeast colony* unless that class was included in the training label set. Vision-language learning replaces the finite label $Y_i$ by a text object $T_i$: $$ (X_i,T_i), \qquad X_i \in \mathcal X, \quad T_i \in \mathcal T. $$ Here $T_i$ may be a caption, a class name, an alt-text string, a medical report, or a natural-language prompt. Instead of learning a map from images to a fixed label set, the goal is to learn two encoders: $$ f_\theta : \mathcal X \to \mathbb R^d, \qquad g_\phi : \mathcal T \to \mathbb R^d. $$ The image encoder and text encoder produce vectors in the same representation space. For an image $x$ and text $t$, define $$ z_x = f_\theta(x), \qquad z_t = g_\phi(t). $$ Usually the embeddings are normalized: $$ \bar z_x = \frac{z_x}{\|z_x\|_2}, \qquad \bar z_t = \frac{z_t}{\|z_t\|_2}. $$ Then similarity is often measured by cosine similarity: $$ s(x,t)=\langle \bar z_x,\bar z_t\rangle. $$ If $s(x,t)$ is large, the image and text are considered semantically compatible. ::: {.math-spine} A vision-language model turns semantic matching into geometry: $$ \text{image matches text} \quad \Longleftrightarrow \quad \langle \bar z_x,\bar z_t\rangle \text{ is large}. $$ ::: CLIP-style contrastive learning is a central example of this idea [@radford2021clip]. ## 27.2 Mathematical Objects The main mathematical objects in this chapter are: | Object | Meaning | |---|---| | $X$ | image-valued random variable | | $T$ | text-valued random variable | | $f_\theta$ | image encoder | | $g_\phi$ | text encoder | | $z_x,z_t \in \mathbb R^d$ | image and text embeddings | | $S \in \mathbb R^{m\times m}$ | image-text similarity matrix inside a minibatch | | $\tau>0$ | temperature parameter | | $p_\theta(t_j\mid x_i)$ | model probability that text $t_j$ matches image $x_i$ | | $\ell_{IT},\ell_{TI}$ | image-to-text and text-to-image contrastive losses | | $\mathcal P$ | prompt set or class-name set | The basic unit of training is a minibatch of matched pairs $$ \mathcal B=\{(x_1,t_1),\dots,(x_m,t_m)\}. $$ The diagonal pairs $(x_i,t_i)$ are treated as positives. The off-diagonal pairs $(x_i,t_j)$, $j\neq i$, are treated as negatives within the minibatch. ## 27.3 Two-Tower Architecture A common architecture is the **two-tower model**: $$ \begin{aligned} Z_X &= f_\theta(X),\\ Z_T &= g_\phi(T),\\ S_{ij} &= \frac{1}{\tau}\langle \bar z_{x_i},\bar z_{t_j}\rangle. \end{aligned} $$ The image tower may be a CNN or a vision transformer. The text tower is usually a transformer language encoder. The two towers interact only through the similarity matrix during contrastive training. This architecture has an important computational advantage. Since images and texts are encoded separately, one can precompute text embeddings for candidate prompts and image embeddings for databases. Retrieval can then be performed by nearest-neighbor search. ### Matrix Form Let $$ A=\begin{bmatrix} \bar z_{x_1}^T\\ \vdots\\ \bar z_{x_m}^T \end{bmatrix}\in \mathbb R^{m\times d}, \qquad B=\begin{bmatrix} \bar z_{t_1}^T\\ \vdots\\ \bar z_{t_m}^T \end{bmatrix}\in \mathbb R^{m\times d}. $$ The minibatch similarity matrix is $$ S=\frac{1}{\tau}AB^T. $$ The diagonal entry $S_{ii}$ measures the similarity of a true image-text pair. The off-diagonal entry $S_{ij}$ measures the similarity between image $i$ and a mismatched text $j$. This is useful for applied mathematics students because the whole training objective can be read from one matrix. ::: {.interactive-card} ### Interactive Demo 27.1: Image-Text Embedding Geometry The demo below shows image and text embeddings on the unit circle. A good vision-language representation places matched image and text points close together and mismatched points far apart. <div id="ch27-embedding-geometry" class="plotly-demo"></div> <script> (function(){ const div = document.getElementById('ch27-embedding-geometry'); if (!div || typeof Plotly === 'undefined') return; const names = ['cat','dog','car','tree','ship','bird']; const base = [0.2, 1.2, 2.4, 3.3, 4.4, 5.3]; const imgX = base.map(a => Math.cos(a)); const imgY = base.map(a => Math.sin(a)); const txtX = base.map((a,i) => Math.cos(a + 0.18*Math.sin(i+1))); const txtY = base.map((a,i) => Math.sin(a + 0.18*Math.sin(i+1))); const traces = [ {x: imgX, y: imgY, mode:'markers+text', text:names.map(n=>'image: '+n), textposition:'top center', name:'image embeddings', marker:{size:12}}, {x: txtX, y: txtY, mode:'markers+text', text:names.map(n=>'text: '+n), textposition:'bottom center', name:'text embeddings', marker:{size:12, symbol:'diamond'}}, ]; for (let i=0; i<names.length; i++) { traces.push({x:[imgX[i], txtX[i]], y:[imgY[i], txtY[i]], mode:'lines', showlegend:false, line:{dash:'dot'}}); } Plotly.newPlot(div, traces, { title:'Matched image and text embeddings should be close', xaxis:{title:'embedding coordinate 1', scaleanchor:'y', range:[-1.25,1.25]}, yaxis:{title:'embedding coordinate 2', range:[-1.25,1.25]}, margin:{t:50} }, {responsive:true}); })(); </script> ::: ## 27.4 Contrastive Image-Text Learning Given a minibatch of $m$ matched pairs, define the similarity score $$ S_{ij}=\frac{1}{\tau}\langle \bar z_{x_i},\bar z_{t_j}\rangle. $$ For each image $x_i$, the model defines a probability distribution over candidate texts in the minibatch: $$ p_\theta(t_j\mid x_i) = \frac{\exp(S_{ij})}{\sum_{k=1}^m \exp(S_{ik})}. $$ The correct text is $t_i$. Therefore the image-to-text loss is $$ \ell_{IT}(i) = -\log p_\theta(t_i\mid x_i) = -S_{ii}+\log\left(\sum_{k=1}^m \exp(S_{ik})\right). $$ Similarly, for each text $t_i$, define a probability distribution over candidate images: $$ p_\theta(x_j\mid t_i) = \frac{\exp(S_{ji})}{\sum_{k=1}^m \exp(S_{ki})}. $$ The text-to-image loss is $$ \ell_{TI}(i) = -\log p_\theta(x_i\mid t_i) = -S_{ii}+\log\left(\sum_{k=1}^m \exp(S_{ki})\right). $$ The symmetric contrastive loss is $$ L = \frac{1}{2m}\sum_{i=1}^m \left[\ell_{IT}(i)+\ell_{TI}(i)\right]. $$ ::: {.math-spine} The loss is simply cross-entropy applied twice: once row-wise and once column-wise to the similarity matrix $S$. ::: ### Why the Loss Works The term $-S_{ii}$ encourages the matched image-text similarity to increase. The log-sum-exp term penalizes high similarity between mismatched pairs. Thus the loss encourages $$ S_{ii} > S_{ij}, \quad j\neq i, \qquad S_{ii} > S_{ji}, \quad j\neq i. $$ Geometrically, each matched pair should be close, and each mismatched pair should be separated by angular distance. ### Gradient Intuition For the image-to-text loss, the derivative with respect to a similarity score is $$ \frac{\partial \ell_{IT}(i)}{\partial S_{ij}} = p_\theta(t_j\mid x_i)-1_{j=i}. $$ Thus: - if $j=i$, the gradient is $p_\theta(t_i\mid x_i)-1$, usually negative, so optimization increases $S_{ii}$; - if $j\neq i$, the gradient is $p_\theta(t_j\mid x_i)$, positive, so optimization decreases confusing mismatched similarities. This is the same softmax-cross-entropy geometry used in classification, but the classes are texts inside the minibatch. ::: {.interactive-card} ### Interactive Demo 27.2: Temperature in Contrastive Softmax The temperature $\tau$ controls how sharp the softmax distribution is. Small $\tau$ makes the model focus on the hardest negatives; large $\tau$ spreads probability more evenly. <div id="ch27-temperature" class="plotly-demo"></div> <script> (function(){ const div = document.getElementById('ch27-temperature'); if (!div || typeof Plotly === 'undefined') return; const sims = [-0.1, 0.2, 0.35, 0.55, 0.72, 0.9]; const labels = ['text 1','text 2','text 3','text 4','text 5','true text']; const temps = [0.03, 0.07, 0.2, 0.7]; const traces = temps.map(tau => { const exps = sims.map(s => Math.exp(s/tau)); const Z = exps.reduce((a,b)=>a+b,0); return {x: labels, y: exps.map(e=>e/Z), type:'bar', name:'tau = '+tau}; }); Plotly.newPlot(div, traces, { title:'Softmax probabilities for different temperatures', yaxis:{title:'probability', range:[0,1]}, xaxis:{title:'candidate text'}, barmode:'group', margin:{t:50} }, {responsive:true}); })(); </script> ::: ## 27.5 The Role of Normalization and the Unit Sphere Most CLIP-style models normalize embeddings before computing similarity. This means $$ \|\bar z_x\|_2=\|\bar z_t\|_2=1. $$ Therefore $$ \langle \bar z_x,\bar z_t\rangle=\cos(\alpha), $$ where $\alpha$ is the angle between the image and text embeddings. The representation space is effectively the unit sphere $$ S^{d-1}=\{z\in\mathbb R^d: \|z\|_2=1\}. $$ This has several consequences. 1. Similarity depends on angle, not vector length. 2. The model cannot trivially reduce the loss by increasing embedding norms. 3. Retrieval can use maximum inner product or nearest-neighbor search on normalized vectors. 4. The temperature parameter becomes especially important because cosine similarities lie in $[-1,1]$. In high dimension, random unit vectors are nearly orthogonal. If $u,v$ are independently sampled from the unit sphere, then $$ E\langle u,v\rangle =0, \qquad \operatorname{Var}(\langle u,v\rangle)=\frac{1}{d}. $$ Thus a high-dimensional embedding space provides many nearly orthogonal directions. Contrastive learning tries to use this geometry to separate many semantic concepts. ## 27.6 Zero-Shot Classification A remarkable consequence of image-text alignment is **zero-shot classification**. Suppose the class names are $$ \mathcal C=\{c_1,\dots,c_K\}. $$ For each class, construct a text prompt such as $$ t_k = \text{``a photo of a } c_k \text{''}. $$ Compute text prototypes $$ u_k = \frac{g_\phi(t_k)}{\|g_\phi(t_k)\|_2}. $$ Given a new image $x$, compute $$ z=\frac{f_\theta(x)}{\|f_\theta(x)\|_2}. $$ The zero-shot classifier is $$ \widehat y(x)=\arg\max_{1\le k\le K}\langle z,u_k\rangle. $$ This is nearest-prototype classification in a text-defined semantic space. No image examples from the new classes are required at inference time. ### Softmax Version One may also define class probabilities: $$ p(y=k\mid x) = \frac{\exp(\langle z,u_k\rangle/\tau)} {\sum_{j=1}^K \exp(\langle z,u_j\rangle/\tau)}. $$ These probabilities are often useful for ranking classes, but they should not automatically be trusted as calibrated probabilities. Calibration depends on the training data, prompts, temperature, and distribution shift. ::: {.interactive-card} ### Interactive Demo 27.3: Zero-Shot Classification by Text Prototypes The image embedding is classified by whichever text prototype has largest cosine similarity. <div id="ch27-zero-shot" class="plotly-demo"></div> <script> (function(){ const div = document.getElementById('ch27-zero-shot'); if (!div || typeof Plotly === 'undefined') return; const classes = ['cat','dog','truck','flower']; const theta = [0.3, 1.5, 3.2, 5.1]; const px = theta.map(a => Math.cos(a)); const py = theta.map(a => Math.sin(a)); const imgA = 1.25; const ix = Math.cos(imgA), iy = Math.sin(imgA); const sims = theta.map(a => Math.cos(imgA-a)); const traces = [ {x:px, y:py, mode:'markers+text', text:classes.map(c=>'prompt: '+c), textposition:'top center', name:'text prototypes', marker:{size:13}}, {x:[ix], y:[iy], mode:'markers+text', text:['new image'], textposition:'bottom center', name:'image', marker:{size:16, symbol:'star'}}, {x:classes, y:sims, type:'bar', name:'cosine similarity', xaxis:'x2', yaxis:'y2'} ]; Plotly.newPlot(div, traces, { title:'Zero-shot classification as nearest text prototype', grid:{rows:1, columns:2, pattern:'independent'}, xaxis:{scaleanchor:'y', range:[-1.25,1.25], title:'embedding 1'}, yaxis:{range:[-1.25,1.25], title:'embedding 2'}, xaxis2:{title:'class prompt'}, yaxis2:{title:'similarity', range:[-1,1]}, margin:{t:50} }, {responsive:true}); })(); </script> ::: ## 27.7 Prompt Engineering and Prompt Ensembling A class name alone may be ambiguous. For example, the text string "crane" may refer to a bird or a machine. Vision-language systems often use prompts: $$ \text{``a photo of a crane''}, \qquad \text{``a photo of a construction crane''}, \qquad \text{``a photo of a crane bird''}. $$ For a class $c_k$, let there be $r$ prompt templates $$ t_{k1},\dots,t_{kr}. $$ Prompt ensembling uses the average text embedding $$ u_k = \frac{\sum_{a=1}^r \bar g_\phi(t_{ka})}{\left\|\sum_{a=1}^r \bar g_\phi(t_{ka})\right\|_2}. $$ The classifier becomes $$ \widehat y(x)=\arg\max_k \langle \bar f_\theta(x),u_k\rangle. $$ Mathematically, prompt ensembling reduces sensitivity to a single text description. Statistically, it is a kind of model averaging over linguistic descriptions. ::: {.interactive-card} ### Interactive Demo 27.4: Prompt Ensembling Different prompts for the same class can produce slightly different text embeddings. Averaging them gives a more stable class prototype. <div id="ch27-prompt-ensemble" class="plotly-demo"></div> <script> (function(){ const div = document.getElementById('ch27-prompt-ensemble'); if (!div || typeof Plotly === 'undefined') return; const centers = {cat:0.5, dog:1.7, car:3.4}; let traces = []; Object.keys(centers).forEach((name, idx) => { const a0 = centers[name]; const angles = [-0.18,-0.08,0.09,0.19].map(d=>a0+d); const xs = angles.map(a=>Math.cos(a)); const ys = angles.map(a=>Math.sin(a)); const mx = xs.reduce((a,b)=>a+b,0)/xs.length; const my = ys.reduce((a,b)=>a+b,0)/ys.length; const norm = Math.hypot(mx,my); traces.push({x:xs, y:ys, mode:'markers', name:name+' prompts', marker:{size:9}}); traces.push({x:[mx/norm], y:[my/norm], mode:'markers+text', text:[name+' ensemble'], textposition:'top center', name:name+' mean', marker:{size:16, symbol:'diamond'}}); }); Plotly.newPlot(div, traces, { title:'Class prototypes from several prompts', xaxis:{title:'embedding coordinate 1', scaleanchor:'y', range:[-1.2,1.2]}, yaxis:{title:'embedding coordinate 2', range:[-1.2,1.2]}, margin:{t:50} }, {responsive:true}); })(); </script> ::: ## 27.8 Image-Text Retrieval Vision-language models naturally support retrieval. ### Text-to-Image Retrieval Given a text query $t$, rank images $x_1,\dots,x_N$ by $$ s(x_i,t)=\langle \bar f_\theta(x_i),\bar g_\phi(t)\rangle. $$ The top-ranked images are $$ \operatorname{TopK}\{s(x_i,t): i=1,\dots,N\}. $$ ### Image-to-Text Retrieval Given an image $x$, rank texts $t_1,\dots,t_M$ by $$ s(x,t_j)=\langle \bar f_\theta(x),\bar g_\phi(t_j)\rangle. $$ This may retrieve captions, labels, reports, or natural-language descriptions. ### Evaluation Metrics Common retrieval metrics include: $$ \operatorname{Recall@K} = \frac{1}{n}\sum_{i=1}^n 1\{\text{true match for query }i\text{ appears in top }K\}, $$ and mean reciprocal rank: $$ \operatorname{MRR} = \frac{1}{n}\sum_{i=1}^n \frac{1}{\operatorname{rank}_i}. $$ Retrieval is sensitive to the geometry of the embedding space. A small number of "hub" images or texts may appear near many queries. This is a high-dimensional nearest-neighbor phenomenon and should be checked empirically. ::: {.interactive-card} ### Interactive Demo 27.5: Retrieval Rankings The bars show similarity scores between one text query and a collection of images. Retrieval quality depends on whether the correct match is ranked near the top. <div id="ch27-retrieval" class="plotly-demo"></div> <script> (function(){ const div = document.getElementById('ch27-retrieval'); if (!div || typeof Plotly === 'undefined') return; const images = ['img A','img B','img C','img D','img E','img F','img G','img H']; const scores = [0.22,0.81,0.45,0.39,0.67,0.18,0.51,0.74]; const order = scores.map((s,i)=>[s,i]).sort((a,b)=>b[0]-a[0]); const sortedImgs = order.map(p=>images[p[1]]); const sortedScores = order.map(p=>p[0]); const ranks = sortedImgs.map((name,i)=>name==='img B' ? 'true match, rank '+(i+1) : 'negative'); Plotly.newPlot(div, [ {x:sortedImgs, y:sortedScores, type:'bar', text:ranks, textposition:'auto', name:'similarity'} ], { title:'Text-to-image retrieval ranking', xaxis:{title:'candidate images sorted by similarity'}, yaxis:{title:'cosine similarity', range:[0,1]}, margin:{t:50} }, {responsive:true}); })(); </script> ::: ## 27.9 Captioning, VQA, and Cross-Attention The two-tower contrastive model is not the only form of vision-language learning. For tasks requiring generated text, the model needs more interaction between visual tokens and language tokens. ### Image Captioning Image captioning models the conditional distribution $$ p(w_1,\dots,w_L\mid x) = \prod_{\ell=1}^L p(w_\ell\mid w_1,\dots,w_{\ell-1},x), $$ where $w_1,\dots,w_L$ are caption tokens. A vision encoder provides visual features, and a language decoder generates text autoregressively. ### Visual Question Answering Visual question answering models $$ p(a\mid x,q), $$ where $x$ is an image, $q$ is a question, and $a$ is an answer. The model must combine visual evidence and linguistic context. ### Cross-Attention Suppose visual tokens are $$ V=[v_1,\dots,v_m]^T \in \mathbb R^{m\times d}, $$ and text tokens are $$ W=[w_1,\dots,w_L]^T \in \mathbb R^{L\times d}. $$ A cross-attention layer may let text tokens attend to visual tokens: $$ \operatorname{CrossAttn}(W,V) = \operatorname{softmax}\left(\frac{(WQ)(VK)^T}{\sqrt{d_k}}\right)(VV). $$ Here the query comes from language and the keys/values come from vision. Cross-attention is one mathematical mechanism for fusing modalities. ::: {.interactive-card} ### Interactive Demo 27.6: Cross-Attention Map Rows represent text tokens and columns represent image patches. Brighter entries correspond to stronger attention from a word to a patch. <div id="ch27-cross-attention" class="plotly-demo"></div> <script> (function(){ const div = document.getElementById('ch27-cross-attention'); if (!div || typeof Plotly === 'undefined') return; const words = ['a','red','car','on','street']; const patches = ['p1','p2','p3','p4','p5','p6','p7','p8']; const Z = [ [0.10,0.10,0.12,0.11,0.14,0.14,0.15,0.14], [0.05,0.08,0.30,0.32,0.08,0.06,0.06,0.05], [0.04,0.06,0.34,0.37,0.07,0.05,0.04,0.03], [0.08,0.12,0.13,0.12,0.18,0.16,0.12,0.09], [0.04,0.05,0.08,0.07,0.20,0.24,0.19,0.13] ]; Plotly.newPlot(div, [{z:Z, x:patches, y:words, type:'heatmap', colorbar:{title:'attention'}}], { title:'Text-to-vision cross-attention', xaxis:{title:'image patches'}, yaxis:{title:'text tokens'}, margin:{t:50} }, {responsive:true}); })(); </script> ::: ## 27.10 Generative Vision-Language Models A contrastive model scores image-text compatibility. A generative vision-language model can produce text from images, images from text, or decisions from multimodal context. Examples of mathematical tasks include: ### Text Given Image $$ p_\theta(t\mid x). $$ This includes captioning, report generation, and image explanation. ### Image Given Text $$ p_\theta(x\mid t). $$ This includes text-to-image generation and controlled image editing. ### Answer Given Image and Question $$ p_\theta(a\mid x,q). $$ This includes visual question answering and multimodal assistant behavior. ### Joint Modeling A joint model tries to represent $$ p_\theta(x,t) = p_\theta(t\mid x)p_\theta(x) = p_\theta(x\mid t)p_\theta(t). $$ In practice, different models approximate different pieces of this joint distribution. CLIP-style models learn a shared embedding and compatibility score. Captioning models learn $p(t\mid x)$. Diffusion models conditioned on text learn a generative approximation to $p(x\mid t)$. Multimodal language models combine visual encoders with language models to answer questions, reason about images, and produce text. ## 27.11 Statistical Interpretation Vision-language learning can be described as statistical estimation from paired multimodal samples: $$ (X_i,T_i) \sim P_{XT}, \qquad i=1,\dots,n. $$ The training objective encourages high similarity for samples from the joint distribution $P_{XT}$ and low similarity for mismatched samples that approximate the product distribution $P_XP_T$. Informally, contrastive learning tries to distinguish $$ (X,T)\sim P_{XT} $$ from $$ (X,T)\sim P_XP_T. $$ This gives a connection to density-ratio estimation. A good compatibility score behaves like $$ s(x,t) \approx \log\frac{p(x,t)}{p(x)p(t)} + \text{constant}. $$ This interpretation is useful because it shows that contrastive learning is not only a classification trick. It is a way to estimate statistical dependence between image and text. ## 27.12 Bias, Calibration, and Distribution Shift Vision-language models are trained on large image-text corpora. These corpora may contain noise, bias, spurious correlations, duplicated examples, and ambiguous captions. Important statistical issues include: 1. **Noisy positives.** A caption may not describe the image accurately. 2. **False negatives.** Two different captions in a minibatch may both describe the same image concept. 3. **Prompt sensitivity.** Predictions can change when the wording changes. 4. **Class imbalance.** Some visual concepts appear far more often than others. 5. **Shortcut learning.** The model may rely on background, style, watermark, or text artifacts. 6. **Poor calibration.** Softmax scores over prompts may not be reliable probabilities. 7. **Distribution shift.** Performance may degrade on medical, scientific, historical, satellite, or low-resource visual domains. For MA Applied Math and MS Statistics students, these are not merely ethical concerns. They are statistical modeling issues about sampling, measurement, identifiability, and external validity. ## 27.13 Mathematical Connection to Earlier Chapters Vision-language models combine many ideas from earlier chapters. | Earlier topic | Role in vision-language models | |---|---| | Linear algebra | embeddings, dot products, similarity matrices | | Probability | conditional distributions, contrastive estimation, calibration | | Optimization | large-scale softmax loss, SGD/Adam, temperature learning | | CNNs and ViTs | image encoders | | Transformers | text encoders and multimodal fusion | | Representation learning | semantic embedding spaces | | Classification | zero-shot nearest-prototype classification | | Retrieval | nearest-neighbor search in embedding space | | Segmentation/foundation models | text and point prompts for masks | | Diffusion | text-conditioned image generation | This makes vision-language modeling a natural capstone topic for the modern part of the book. ## 27.14 Python Example: A Tiny Synthetic Contrastive Batch The following code constructs a minibatch similarity matrix and computes the symmetric contrastive loss. It is intentionally small so that students can inspect every object. ```{python} #| label: ch27-synthetic-contrastive-loss #| eval: false import numpy as np rng = np.random.default_rng(7243) m = 6 # minibatch size D = 4 # embedding dimension tau = 0.07 # Simulate matched image/text embeddings with shared latent semantic directions. latent = rng.normal(size=(m, D)) image = latent + 0.15 * rng.normal(size=(m, D)) text = latent + 0.15 * rng.normal(size=(m, D)) def normalize(Z): return Z / np.linalg.norm(Z, axis=1, keepdims=True) A = normalize(image) B = normalize(text) S = (A @ B.T) / tau def row_cross_entropy(S): # Correct label for row i is column i. S_shift = S - S.max(axis=1, keepdims=True) log_probs = S_shift - np.log(np.exp(S_shift).sum(axis=1, keepdims=True)) return -np.diag(log_probs).mean() loss_image_to_text = row_cross_entropy(S) loss_text_to_image = row_cross_entropy(S.T) loss = 0.5 * (loss_image_to_text + loss_text_to_image) print("similarity matrix") print(np.round(S, 2)) print("image-to-text loss:", loss_image_to_text) print("text-to-image loss:", loss_text_to_image) print("symmetric loss:", loss) ``` The diagonal entries of the similarity matrix should be relatively large when the paired embeddings are close. ## 27.15 Python Example: Zero-Shot Classification with Text Prototypes ```{python} #| label: ch27-zero-shot-classification-example #| eval: false import numpy as np rng = np.random.default_rng(5010) classes = ["cat", "dog", "car", "flower"] d = 8 # Pretend these are normalized text embeddings from prompts. text_proto = rng.normal(size=(len(classes), d)) text_proto = text_proto / np.linalg.norm(text_proto, axis=1, keepdims=True) # Create an image embedding close to the dog prototype. true_class = 1 image_emb = text_proto[true_class] + 0.25 * rng.normal(size=d) image_emb = image_emb / np.linalg.norm(image_emb) scores = text_proto @ image_emb pred = int(np.argmax(scores)) for c, s in zip(classes, scores): print(f"{c:>8s}: {s:.3f}") print("predicted class:", classes[pred]) ``` This example shows the mathematical essence of zero-shot classification. In practice, the embeddings come from a trained image encoder and text encoder. ## 27.16 Python Example: Retrieval Metrics ```{python} #| label: ch27-retrieval-metrics-example #| eval: false import numpy as np rng = np.random.default_rng(7339) n = 10 d = 16 latent = rng.normal(size=(n, d)) image = latent + 0.4 * rng.normal(size=(n, d)) text = latent + 0.4 * rng.normal(size=(n, d)) def normalize(Z): return Z / np.linalg.norm(Z, axis=1, keepdims=True) image = normalize(image) text = normalize(text) S = text @ image.T # row = text query, column = image candidate ranks = [] for i in range(n): ranking = np.argsort(-S[i]) rank = int(np.where(ranking == i)[0][0]) + 1 ranks.append(rank) ranks = np.array(ranks) recall_at_1 = np.mean(ranks <= 1) recall_at_5 = np.mean(ranks <= 5) mrr = np.mean(1 / ranks) print("ranks:", ranks) print("Recall@1:", recall_at_1) print("Recall@5:", recall_at_5) print("MRR:", mrr) ``` ## 27.17 TensorFlow Example: Minimal Dual Encoder The code below gives a minimal dual-encoder skeleton. It is not meant to be a production CLIP model; it is a transparent implementation of the mathematical loss. ```{python} #| label: ch27-tensorflow-dual-encoder #| eval: false import tensorflow as tf class TinyDualEncoder(tf.keras.Model): def __init__(self, image_dim, text_dim, embed_dim=128): super().__init__() self.image_encoder = tf.keras.Sequential([ tf.keras.layers.Input(shape=(image_dim,)), tf.keras.layers.Dense(256, activation="relu"), tf.keras.layers.Dense(embed_dim), ]) self.text_encoder = tf.keras.Sequential([ tf.keras.layers.Input(shape=(text_dim,)), tf.keras.layers.Dense(256, activation="relu"), tf.keras.layers.Dense(embed_dim), ]) self.logit_scale = tf.Variable(tf.math.log(1 / 0.07), trainable=True) def call(self, image_features, text_features): zi = self.image_encoder(image_features) zt = self.text_encoder(text_features) zi = tf.math.l2_normalize(zi, axis=1) zt = tf.math.l2_normalize(zt, axis=1) logits = tf.exp(self.logit_scale) * tf.matmul(zi, zt, transpose_b=True) return logits def symmetric_contrastive_loss(logits): batch_size = tf.shape(logits)[0] labels = tf.range(batch_size) loss_i2t = tf.keras.losses.sparse_categorical_crossentropy( labels, logits, from_logits=True ) loss_t2i = tf.keras.losses.sparse_categorical_crossentropy( labels, tf.transpose(logits), from_logits=True ) return 0.5 * (tf.reduce_mean(loss_i2t) + tf.reduce_mean(loss_t2i)) # Example usage with precomputed image/text feature vectors. model = TinyDualEncoder(image_dim=512, text_dim=300, embed_dim=128) optimizer = tf.keras.optimizers.AdamW(learning_rate=1e-4, weight_decay=1e-4) image_batch = tf.random.normal((32, 512)) text_batch = tf.random.normal((32, 300)) with tf.GradientTape() as tape: logits = model(image_batch, text_batch) loss = symmetric_contrastive_loss(logits) grads = tape.gradient(loss, model.trainable_variables) optimizer.apply_gradients(zip(grads, model.trainable_variables)) print(float(loss)) ``` ## 27.18 AI-Assisted Study Prompts Use AI tools carefully as mathematical assistants, not as substitutes for derivation. ### Prompt 1: Deriving the Contrastive Loss > Explain why the CLIP-style image-to-text loss is a cross-entropy loss over rows of the similarity matrix. Then derive the gradient with respect to $S_{ij}$. ### Prompt 2: Checking Dimensions > Given image embeddings $A\in\mathbb R^{m\times d}$ and text embeddings $B\in\mathbb R^{m\times d}$, check all dimensions in the formula $S=AB^T/\tau$ and explain why row-wise and column-wise softmax are both needed. ### Prompt 3: Statistical Failure Modes > Give three examples where image-text contrastive learning can fail because of biased or noisy training data. For each, describe a statistical diagnostic that could reveal the problem. ### Prompt 4: Zero-Shot Classification > Compare zero-shot classification by text prototypes with ordinary softmax classification trained on fixed labels. Explain the advantage and the statistical risk. ## 27.19 Exercises ### Conceptual Exercises 1. Explain why a finite-label classifier cannot directly solve open-vocabulary recognition. 2. Why are image and text embeddings normalized before computing cosine similarity? 3. What is the role of the temperature parameter $\tau$? 4. Explain the difference between image-text retrieval and image classification. 5. Why can prompt wording affect zero-shot classification results? 6. Explain why captioning requires a different architecture from pure contrastive retrieval. 7. Give one example of a false negative in image-text contrastive learning. ### Mathematical Exercises 1. Let $S\in\mathbb R^{m\times m}$ be a similarity matrix. Derive the row-wise contrastive loss and show that its gradient with respect to $S_{ij}$ is $p_{ij}-1_{j=i}$. 2. Suppose $u,v\in S^{d-1}$. Show that $\|u-v\|_2^2=2-2\langle u,v\rangle$. Conclude that maximizing cosine similarity is equivalent to minimizing Euclidean distance on the unit sphere. 3. For class prompts $u_1,\dots,u_K$, derive the zero-shot softmax probability $p(y=k\mid x)$ and discuss how the temperature changes the entropy of the distribution. 4. Let $A,B\in\mathbb R^{m\times d}$ have unit-norm rows. Show that the similarity matrix satisfies $S_{ij}\in[-1/\tau,1/\tau]$. 5. Derive Recall@K and MRR for a retrieval system with ranks $r_1,\dots,r_n$. 6. Show that averaging several normalized prompt embeddings and then renormalizing gives a unit vector. Explain why this may reduce prompt variance. ### Python Exercises 1. Implement the symmetric contrastive loss using only NumPy. 2. Simulate image and text embeddings with increasing noise. Plot Recall@1 as a function of noise level. 3. Compare zero-shot classification using a single prompt per class versus five prompt templates per class. 4. Compute a similarity matrix and visualize it as a heatmap. Explain what a good diagonal pattern means. 5. Implement temperature scaling for zero-shot class probabilities and plot entropy as a function of temperature. ### AI and Modeling Exercises 1. Design a small vision-language experiment for a scientific image dataset. What would the image inputs be? What would the text descriptions be? 2. Suppose a model classifies an X-ray using prompts written for natural photos. Explain why this is a domain-shift problem. 3. Propose a diagnostic for detecting whether a vision-language model is using background context rather than object evidence. 4. Compare CLIP-style contrastive learning with caption generation. Which is more appropriate for image search? Which is more appropriate for medical report drafting? 5. Explain how a text prompt can be used to guide segmentation or image generation. ## 27.20 Chapter Summary Vision-language models connect images and text through a shared representation space. The central mathematical construction is a pair of encoders $$ f_\theta:\mathcal X\to\mathbb R^d, \qquad g_\phi:\mathcal T\to\mathbb R^d, $$ together with normalized embeddings and cosine similarity. CLIP-style training forms a minibatch similarity matrix $$ S=\frac{1}{\tau}AB^T $$ and applies cross-entropy row-wise and column-wise. Once trained, the same embedding space supports zero-shot classification, image-text retrieval, prompt-based recognition, and multimodal transfer. The most important mathematical lesson is that natural language can define visual categories. Instead of learning only a fixed classifier, a vision-language model learns a geometry in which images and text descriptions can be compared directly. ## Notes and References CLIP-style image-text contrastive learning was introduced as a scalable approach to transferable visual representations in @radford2021clip. The transformer architecture underlying many text encoders and multimodal fusion modules is discussed in @vaswani2017attention, and vision transformers were introduced for image recognition in @dosovitskiy2020image. The broader representation-learning ideas connect to contrastive learning and self-supervision from Chapter 26. Vision-language modeling also provides the conceptual bridge from image recognition to promptable segmentation and multimodal foundation models in Chapter 28.