29  Promptable Segmentation and Foundation Models

Chapter 28 studies promptable segmentation and foundation models from a mathematical point of view. Instead of training a separate segmentation model for every fixed label set, a promptable model accepts an image together with extra information—points, boxes, masks, text, or prior predictions—and returns candidate masks. The central mathematical idea is to turn segmentation into conditional structured prediction:

\[ \text{image} + \text{prompt} \longmapsto \text{mask distribution or ranked mask set}. \]

The previous chapters developed the pieces needed for this chapter. Chapter 8 formulated segmentation as optimization. Chapter 24 studied semantic and instance segmentation with neural mask models. Chapter 25 developed vision transformers. Chapter 26 studied self-supervised representation learning. Chapter 27 studied vision-language models. This chapter combines these ideas into a modern foundation-model viewpoint.

A promptable segmentation model is not merely a segmentation network. It is an interface between a visual representation and a user, algorithm, or downstream model. The prompt acts as a small amount of information that selects one mask from many plausible masks in an image. In mathematical language, the prompt changes the conditional distribution.

For a fixed image \(x\), there may be many valid masks:

\[ y_1, y_2, \dots, y_m \subset \Omega. \]

A point prompt might ask for the object containing a given point. A box prompt might ask for the object mostly inside a rectangle. A text prompt might ask for an object described by language. Thus the model should not approximate only \(p(y\mid x)\); it should approximate

\[ p(y\mid x,p), \]

where \(p\) denotes the prompt. This simple change is responsible for much of the flexibility of foundation models for segmentation.

29.1 Learning Goals

After reading this chapter, students should be able to:

1. define promptable segmentation as conditional structured prediction;
2. distinguish semantic, instance, panoptic, interactive, and promptable segmentation;
3. describe point, box, mask, text, and embedding prompts as mathematical objects;
4. explain the image-encoder, prompt-encoder, and mask-decoder architecture used by SAM-style models;
5. formulate mask generation as scoring candidate subsets of the image domain;
6. connect promptable segmentation to conditional probability, optimization, attention, and representation learning;
7. analyze ambiguity, uncertainty, calibration, ranking, and failure modes for mask prediction;
8. implement small mathematical experiments for prompt encoding, mask scoring, IoU ranking, and active annotation;
9. understand how foundation segmentation models interact with vision-language models, object detectors, robotics, medical imaging, and scientific image analysis.

29.2 28.1 Why Promptable Segmentation?

Classical segmentation problems usually specify the output type in advance.

Semantic segmentation asks for a label map

\[ y : \Omega \to \{1,\dots,K\}. \]

Instance segmentation asks for a finite set of object masks

\[ \{(M_j,c_j,s_j)\}_{j=1}^N, \]

where \(M_j \subset \Omega\) is a mask, \(c_j\) is a category, and \(s_j\) is a confidence score. Panoptic segmentation combines semantic and instance information.

Promptable segmentation changes the problem. The user or another model supplies a prompt \(p\), and the segmentation model returns one or more masks compatible with that prompt:

\[ F_\theta : (x,p) \mapsto \{(\widehat M_k, s_k)\}_{k=1}^K. \]

The prompt may be:

• a positive point \(q^+ \in \Omega\);
• a negative point \(q^- \in \Omega\);
• a bounding box \(b=(u_{\min},v_{\min},u_{\max},v_{\max})\);
• a coarse mask \(m_0\);
• a text phrase \(t\);
• a feature vector or memory token from another model.

The central statistical difference is this:

\[ \text{ordinary segmentation: } p(y\mid x), \qquad \text{promptable segmentation: } p(y\mid x,p). \]

The prompt reduces ambiguity. Consider an image containing a person, a backpack, a bicycle, a helmet, and a shadow. A single image \(x\) has many reasonable masks. A point inside the backpack selects one object. A box around the bicycle selects another. A phrase such as “front wheel” may select a part. The prompt changes the posterior distribution over valid masks.

This is why promptable segmentation is important for applied mathematics and statistics: it is a conditional inference problem in which the conditioning variable is often small, noisy, and heterogeneous.

29.3 28.2 Images, Prompts, and Masks as Mathematical Objects

Let

\[ \Omega = \{1,\dots,H\}\times \{1,\dots,W\} \]

be the discrete image grid. A color image is

\[ x \in [0,1]^{H\times W\times C}. \]

A binary mask is an indicator function

\[ m : \Omega \to \{0,1\}, \]

or equivalently a subset

\[ M=\{\omega\in\Omega: m(\omega)=1\}. \]

A soft mask is a probability field

\[ \pi : \Omega \to [0,1], \qquad \pi(\omega)\approx P(m(\omega)=1\mid x,p). \]

A prompt belongs to a prompt space \(P\). Promptable segmentation is a map

\[ F_\theta : X\times P \to [0,1]^{H\times W}\times \mathbb R. \]

The second output is often a predicted quality score, for example a predicted IoU value.

A single image-prompt pair may produce several candidate masks:

\[ F_\theta(x,p)=\{(\pi_1,s_1),\dots,(\pi_K,s_K)\}. \]

This multi-mask output is important because prompts may be ambiguous. A point on an animal’s body might refer to the whole animal, the torso, or a visible part. A good model should expose this ambiguity instead of pretending that there is a unique answer.

NoteMathematical distinction

A standard classifier returns a point estimate in a fixed label space. A promptable segmentation model returns a structured object conditioned on a heterogeneous input. It is closer to conditional density estimation or conditional structured prediction than to ordinary classification.

29.4 28.3 Prompt Types

29.4.1 Point prompts

A positive point prompt is a coordinate

\[ q^+=(u,v)\in\Omega. \]

It means approximately:

\[ q^+ \in M. \]

A negative point prompt \(q^-\) means

\[ q^- \notin M. \]

A collection of point prompts can be written as

\[ p_{\mathrm{pts}}=\{(q_i,a_i)\}_{i=1}^r, \qquad q_i\in\Omega, \quad a_i\in\{+1,-1\}. \]

The signs distinguish positive and negative evidence. From a probabilistic point of view, point prompts add constraints to the mask posterior:

\[ P(M\mid x,p_{\mathrm{pts}}) \propto P(M\mid x) \prod_{i:a_i=+1}\mathbf 1\{q_i\in M\} \prod_{i:a_i=-1}\mathbf 1\{q_i\notin M\}, \]

where the indicator constraints are idealized. In practice, the constraints are softened because prompts may be imprecise.

29.4.2 Box prompts

A box prompt is a rectangle

\[ b=(u_1,v_1,u_2,v_2), \qquad u_1<u_2, \quad v_1<v_2. \]

A box suggests that the desired object is spatially concentrated inside the rectangle. One approximate prior is

\[ P(M\mid b) \propto \exp\left[-\lambda_{\mathrm{out}} |M\setminus B| - \lambda_{\mathrm{miss}} |B\setminus M|\right], \]

where \(B\subset\Omega\) is the set of pixels inside the box. This prior is intentionally imperfect: boxes usually include background and may cut off parts of an object.

29.4.3 Mask prompts

A mask prompt is a coarse mask

\[ m_0:\Omega\to[0,1]. \]

It may come from a previous model, a low-resolution prediction, a user scribble, or a previous round of interaction. A mask prompt can be interpreted as an initial condition for refinement:

\[ \widehat m = R_\theta(x,m_0). \]

In classical variational segmentation, a similar idea appears as iterative energy minimization. In modern foundation models, the refinement is learned.

29.4.4 Text prompts

A text prompt is a phrase or sentence

\[ t\in T. \]

For example:

\[ t=\text{"the left front tire"}. \]

A text prompt usually requires a text encoder

\[ g_\phi:T\to \mathbb R^d, \]

and an alignment mechanism between image features and text features. Text-prompted segmentation connects this chapter to vision-language models.

29.4.5 Embedding prompts

A prompt may also be a learned vector

\[ p\in\mathbb R^d. \]

This is common when a detector, tracker, language model, or memory module sends an internal representation to a mask decoder. In this case, promptable segmentation becomes a general conditional decoding problem.

29.5 28.4 A Conditional Probability View

Let \(Y\) be a random mask and \(P\) be a random prompt. The model tries to approximate

\[ P(Y\mid X=x,P=p). \]

For a soft mask \(\pi_\theta(x,p)\), a simple independent-pixel approximation is

\[ P_\theta(Y=y\mid x,p) = \prod_{\omega\in\Omega} \pi_\theta(\omega)^{y(\omega)} (1-\pi_\theta(\omega))^{1-y(\omega)}. \]

This yields the pixelwise binary cross-entropy loss

\[ \ell_{\mathrm{BCE}}(y,\pi) = - \sum_{\omega\in\Omega} \left[ y(\omega)\log \pi(\omega) + (1-y(\omega))\log(1-\pi(\omega))\right]. \]

However, segmentation masks have spatial structure. Pixels are not independent. Objects have connectedness, smoothness, texture, topology, and part-whole structure. More realistic models use a structured distribution

\[ P_\theta(Y=y\mid x,p) \propto \exp\{-E_\theta(y;x,p)\}, \]

where \(E_\theta\) is an energy. Classical segmentation designs \(E\) explicitly. A foundation model learns a large part of the scoring rule from data.

A useful abstraction is

\[ \widehat y = \arg\min_y E_\theta(y;x,p), \]

or, for a neural model,

\[ \widehat \pi = D_\theta(\Phi_\theta(x),\Psi_\theta(p)). \]

Here \(\Phi_\theta\) is an image encoder, \(\Psi_\theta\) is a prompt encoder, and \(D_\theta\) is a mask decoder.

29.6 28.5 Architecture: Image Encoder, Prompt Encoder, Mask Decoder

A SAM-style promptable segmentation model can be summarized by three maps:

\[ z_x = \Phi_\theta(x), \qquad z_p = \Psi_\theta(p), \qquad (\pi_1,\dots,\pi_K,s_1, \dots,s_K) = D_\theta(z_x,z_p). \]

The image encoder computes a dense image representation. For a vision transformer encoder, the image is first divided into patches. The encoder returns a feature grid

\[ z_x \in \mathbb R^{h\times w\times d}. \]

The prompt encoder converts points, boxes, and masks into tokens or spatial feature maps. Point prompts may be represented using positional encodings. A coordinate \(q=(u,v)\) can be mapped to a vector

\[ \gamma(q) =\left( \sin(\omega_1 u),\cos(\omega_1 u), \sin(\omega_1 v),\cos(\omega_1 v),\dots \right). \]

The sign of a point can be encoded by adding a learned vector for positive or negative prompts:

\[ e(q,a)=\gamma(q)+r_a, \qquad a\in\{+1,-1\}. \]

A box can be represented by the embeddings of its two corner points:

\[ e(b)=\big(e((u_1,v_1),\mathrm{corner}_1), e((u_2,v_2),\mathrm{corner}_2)\big). \]

The mask decoder combines image tokens and prompt tokens, often using attention. A generic attention block has the form

\[ \operatorname{Attn}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^T}{\sqrt d}\right)V. \]

In promptable segmentation, the prompt tokens ask questions of the image tokens. The mask decoder then converts the resulting representation into spatial mask logits

\[ \ell_k:\Omega\to\mathbb R, \]

and soft masks

\[ \pi_k(\omega)=\sigma(\ell_k(\omega)). \]

The model may also predict a quality score \(s_k\), often trained to approximate the IoU between the predicted mask and the ground-truth mask.

29.7 28.6 Interactive Demo: Point Prompt and Conditional Mask Prior

The following simple model is not a foundation model. It is a mathematical toy model showing why a point prompt changes the mask posterior. The image contains two synthetic objects. A positive point changes which object receives high probability.

29.8 28.7 Mask Quality: IoU, Dice, and Ranking

For a predicted mask \(\widehat M\) and ground-truth mask \(M\), the intersection-over-union is

\[ \operatorname{IoU}(\widehat M,M) = \frac{|\widehat M\cap M|}{|\widehat M\cup M|}. \]

The Dice coefficient is

\[ \operatorname{Dice}(\widehat M,M) = \frac{2|\widehat M\cap M|}{|\widehat M|+|M|}. \]

For soft masks \(\pi\) and binary masks \(y\), a common soft Dice score is

\[ \operatorname{Dice}_{\mathrm{soft}}(\pi,y) = \frac{2\sum_\omega \pi(\omega)y(\omega)+\varepsilon} {\sum_\omega \pi(\omega)+\sum_\omega y(\omega)+\varepsilon}. \]

Promptable models often return several masks and a predicted quality score:

\[ (\widehat M_1,s_1),\dots,(\widehat M_K,s_K). \]

The score \(s_k\) should approximate a true quality measure such as

\[ s_k \approx \operatorname{IoU}(\widehat M_k,M). \]

This turns mask generation into both a segmentation problem and a calibration/ranking problem. A model can produce a visually plausible mask but rank it incorrectly. For interactive use, ranking matters because the user sees only a small number of options.

A natural training objective is

\[ \ell(\theta)= \ell_{\mathrm{mask}}(\widehat M_k,M) + \lambda \left(s_k-\operatorname{IoU}(\widehat M_k,M)\right)^2. \]

The first term trains the mask; the second term trains quality prediction.

29.9 28.8 Interactive Demo: IoU Ranking of Candidate Masks

A promptable model may output multiple candidate masks. This demo compares three synthetic candidate masks against one ground-truth mask and ranks them by IoU and Dice.

29.10 28.9 Prompt Encoding as Coordinate Geometry

A point prompt is geometrically small but mathematically rich. It contains location, sign, and sometimes scale. The model must know where the point lies relative to image features. A raw coordinate \((u,v)\) is usually not enough for an attention model, so prompts are encoded into higher-dimensional vectors.

One common idea is positional encoding. Normalize coordinates to \([-1,1]^2\):

\[ \bar u = 2u/(W-1)-1, \qquad \bar v = 2v/(H-1)-1. \]

Then define a Fourier-style encoding

\[ \gamma(\bar u,\bar v) = \big( \sin(2^0\pi \bar u), \cos(2^0\pi \bar u), \sin(2^0\pi \bar v), \cos(2^0\pi \bar v), \dots, \sin(2^{L-1}\pi \bar u), \cos(2^{L-1}\pi \bar u), \sin(2^{L-1}\pi \bar v), \cos(2^{L-1}\pi \bar v) \big). \]

This map has two useful properties.

First, nearby points have related encodings. Second, high-frequency components help the model distinguish fine spatial locations. The dot product

\[ \langle \gamma(q),\gamma(q')\rangle \]

acts like a kernel measuring coordinate similarity across several length scales.

For MA Applied Mathematics and MS Statistics students, this is a familiar theme: a nonlinear feature map turns a simple object into a richer representation. Prompt encoding is kernel-feature engineering inside a neural architecture.

29.11 28.10 Interactive Demo: Positional Encoding Similarity

This demo shows how Fourier positional encodings create a coordinate similarity kernel. As the number of frequencies increases, the kernel becomes sharper around the prompt location.

29.12 28.11 Mask Decoding as Conditional Attention

Suppose the image encoder returns image tokens

\[ Z = \begin{bmatrix}z_1^T\\ \vdots\\ z_N^T\end{bmatrix} \in\mathbb R^{N\times d}, \]

where each \(z_i\) represents a patch or spatial location. Suppose the prompt encoder returns prompt tokens

\[ P = \begin{bmatrix}p_1^T\\ \vdots\\ p_R^T\end{bmatrix} \in\mathbb R^{R\times d}. \]

A simple cross-attention layer uses prompt tokens as queries and image tokens as keys and values:

\[ A= \operatorname{softmax}\left(\frac{(PW_Q)(ZW_K)^T}{\sqrt d}\right), \qquad U=A(ZW_V). \]

The output \(U\) summarizes image content relevant to the prompt. A mask token can then generate a spatial mask by comparing a decoded mask embedding \(a\) with dense image features:

\[ \ell_i = \langle a, z_i\rangle + b, \qquad \pi_i=\sigma(\ell_i). \]

This formula is simple but useful. It says that mask decoding can be interpreted as a prompt-conditioned linear classifier applied at every image location. The classifier vector \(a\) is not fixed; it is produced by the prompt and image context.

A more abstract form is

\[ \pi(\omega) =\sigma\left(\langle a_\theta(x,p), z_x(\omega)\rangle + b_\theta(x,p)\right). \]

Thus promptable segmentation is related to hypernetworks: the model produces the parameters of a mask predictor conditioned on the prompt.

29.13 28.12 Loss Functions for Promptable Segmentation

Let \((x_i,p_i,y_i)\) be training examples where \(p_i\) is a prompt and \(y_i\) is a target mask. A basic training objective is

\[ \min_\theta \frac{1}{n}\sum_{i=1}^n \ell\left(F_\theta(x_i,p_i),y_i\right). \]

For a single mask prediction \(\pi_i\), common losses include:

\[ \ell_{\mathrm{BCE}}(\pi,y), \qquad \ell_{\mathrm{Dice}}(\pi,y)=1-\operatorname{Dice}_{\mathrm{soft}}(\pi,y), \]

and combined losses:

\[ \ell_{\mathrm{mask}}(\pi,y) =\ell_{\mathrm{BCE}}(\pi,y)+\lambda \ell_{\mathrm{Dice}}(\pi,y). \]

For multiple candidate masks, the model may use a best-of-\(K\) objective:

\[ \ell_{\mathrm{best}}(\theta) = \min_{k=1, \dots,K} \ell_{\mathrm{mask}}(\pi_k,y). \]

This objective acknowledges ambiguity. If several masks are plausible for a prompt, the model is not forced to make all candidates identical. The quality score can be trained separately:

\[ \ell_{\mathrm{quality}} = \sum_{k=1}^K \left(s_k-\operatorname{IoU}(\widehat M_k,M)\right)^2. \]

A full objective may be

\[ \ell = \ell_{\mathrm{best}} +eta \ell_{\mathrm{quality}} + ho \ell_{\mathrm{stability}}. \]

The stability term penalizes masks whose thresholded outputs change too much under small perturbations of the threshold or prompt.

29.14 28.13 Prompt Sampling and Training Data

Promptable segmentation requires prompt-mask pairs. If a dataset contains ground-truth masks \(M_i\), prompts can be synthetically sampled from the masks.

For a positive point, sample

\[ q^+ \sim \operatorname{Uniform}(M). \]

For a negative point, sample

\[ q^- \sim \operatorname{Uniform}(\Omega\setminus M). \]

For a box prompt, use the tight bounding rectangle of \(M\) and then perturb it:

\[ b = b(M)+\varepsilon, \]

where \(\varepsilon\) is random noise in the box coordinates. For a mask prompt, downsample or corrupt the ground-truth mask:

\[ m_0 = C(M), \]

where \(C\) is a corruption operator such as blur, erosion, dilation, dropout, or low-resolution resizing.

This synthetic prompt sampling is a form of data augmentation. The training distribution is not only over images and masks, but over prompts:

\[ (x,M)\sim P_{\mathrm{data}}, \qquad p\sim Q(\cdot\mid M,x). \]

The learned risk is

\[ R(\theta)= \mathbb E_{(X,M)} \mathbb E_{P\sim Q(\cdot\mid M,X)} \left[\ell(F_\theta(X,P),M)\right]. \]

This formula is important. It says that performance depends not only on the image distribution, but also on the prompt distribution. A model trained mostly with center points may behave differently from one trained with boundary points, boxes, or ambiguous prompts.

29.15 28.14 Interactive Demo: Prompt Sampling Bias

Different prompt-sampling rules teach different behaviors. This toy demo compares positive points sampled near the center of a mask versus near the boundary. Boundary prompts are harder because they are closer to neighboring regions and uncertainty.

29.16 28.15 Foundation Models in Vision

A foundation model is a large model trained on broad data so that it can be adapted to many downstream tasks. For segmentation, broad training means the model has seen many object types, imaging conditions, domains, shapes, textures, and annotation styles.

Mathematically, a foundation model attempts to learn a representation

\[ \Phi_\theta:X\to Z \]

that is useful for a wide family of tasks

\[ \{T_\alpha\}_{\alpha\in A}. \]

For segmentation, the tasks may include:

• segmenting common objects;
• segmenting object parts;
• separating foreground from background;
• refining a coarse annotation;
• producing candidate masks for an unknown object;
• helping annotate new datasets;
• segmenting scientific or medical structures with additional prompts.

The representation must be both general and spatially precise. This creates a tension. High-level semantic features are good for recognizing objects but may be spatially coarse. Low-level features are spatially precise but less semantic. Modern mask decoders combine both kinds of information.

A useful abstraction is task conditioning:

\[ F_\theta(x,c) \]

where \(c\) describes the task. In promptable segmentation, the prompt \(p\) is a form of task conditioning.

29.17 28.16 Zero-Shot, Few-Shot, and Transfer Behavior

A segmentation model has strong zero-shot behavior if it performs well on images, object categories, or domains not explicitly labeled during downstream training. Mathematically, this is a distribution-shift problem.

Training data follow one distribution:

\[ (X,P,Y)\sim P_{\mathrm{train}}. \]

Deployment data follow another:

\[ (X,P,Y)\sim P_{\mathrm{test}}. \]

The zero-shot risk is

\[ R_{\mathrm{test}}(\theta) = \mathbb E_{(X,P,Y)\sim P_{\mathrm{test}}} \ell(F_\theta(X,P),Y). \]

The question is whether low training risk under broad data implies low test risk under new but related data. There is no universal guarantee. Performance depends on the support of the training distribution, the richness of the representation, the prompt type, and the evaluation task.

Few-shot adaptation uses a small labeled set from the new domain:

\[ \{(x_i,p_i,y_i)\}_{i=1}^m, \qquad m\ll n. \]

Adaptation may tune:

• only prompt parameters;
• only a small adapter module;
• only the mask decoder;
• the full model.

For statistics students, this is a bias-variance tradeoff. Full fine-tuning has high flexibility but may overfit. Prompt tuning or adapter tuning has lower variance but may be biased if the base representation is not suitable.

29.18 28.17 Ambiguity and Multi-Mask Output

Prompts can be ambiguous. A single positive point inside a shirt could refer to:

1. the visible shirt region;
2. the whole person;
3. the torso;
4. a logo on the shirt;
5. a foreground region containing the shirt and nearby objects.

A deterministic single-mask output

\[ \widehat M=F_\theta(x,p) \]

may hide this ambiguity. A multi-mask output estimates several modes of the conditional distribution:

\[ \widehat M_1,\dots,\widehat M_K \approx \text{major modes of } P(M\mid x,p). \]

This is analogous to multimodal prediction in statistics. A conditional mean may be a poor summary if the posterior is multimodal. For masks, averaging two plausible masks can produce an implausible soft blob. Returning candidate masks is often more useful.

The quality predictor then ranks candidates:

\[ \widehat k= \arg\max_k s_k. \]

But the highest-IoU mask may not be the most semantically intended mask if the prompt is underspecified. This is why interactive segmentation often relies on additional user feedback.

29.19 28.18 Interactive Segmentation as Sequential Decision Making

In interactive segmentation, the model and user form a sequential system. At time \(t\), the model has image \(x\), previous prompts \(p_{1:t}\), and current prediction \(\widehat M_t\). The user adds a new prompt \(p_{t+1}\) based on the error.

A simplified update is

\[ \widehat M_{t+1}=F_\theta(x,p_{1:t+1}). \]

The goal is to reduce error quickly:

\[ \operatorname{IoU}(\widehat M_t,M) \to 1 \quad\text{as } t \text{ increases}. \]

The user’s next click can be viewed as an action chosen to maximize improvement:

\[ p_{t+1} =\arg\max_{p}\nabla_{p}\, \operatorname{ExpectedImprovement}(p). \]

In practice, users often click the largest false-negative or false-positive region. This gives a heuristic active-learning strategy:

• click a missed object region as a positive point;
• click an incorrectly included background region as a negative point;
• repeat until the mask is acceptable.

This links promptable segmentation to active learning, sequential experimental design, and human-in-the-loop optimization.

29.20 28.19 Interactive Demo: Sequential Click Correction

This toy demo starts with an imperfect mask. A positive click in a missed region adds mass; a negative click in a false-positive region removes mass. The formula is a simplified radial update, not a neural model.

29.21 28.20 Text-Guided and Open-Vocabulary Segmentation

Promptable segmentation with points and boxes does not by itself identify semantic categories. A mask may be correct geometrically but unlabeled. Text-guided segmentation adds language.

A common modular pipeline is:

\[ \text{text prompt} \xrightarrow{\text{open-vocabulary detector}} \text{box proposals} \xrightarrow{\text{promptable mask model}} \text{masks}. \]

Another approach aligns pixel or region embeddings with text embeddings. Let

\[ z(\omega)\in\mathbb R^d \]

be a dense image feature at pixel or patch \(\omega\), and let

\[ e(t)\in\mathbb R^d \]

be a text embedding. A text-conditioned mask score may be

\[ \ell_t(\omega)=\frac{\langle z(\omega),e(t)\rangle}{\tau}. \]

The soft mask is

\[ \pi_t(\omega)=\sigma(\ell_t(\omega)). \]

This equation is the dense analogue of CLIP-style image-text similarity. Instead of comparing a whole image with a sentence, it compares each spatial location with a phrase.

Open-vocabulary segmentation faces several statistical difficulties.

First, category names are not always visually grounded. The phrase “vehicle” is broad, while “red left rear taillight” is specific. Second, text may refer to material, part, function, affordance, location, or relation. Third, negative evidence is hard: if a model does not segment “microscope slide artifact,” does that mean the object is absent or the phrase is outside the model’s vocabulary?

29.22 28.21 Foundation Models as Annotation Engines

One major use of promptable segmentation is dataset construction. Suppose a human annotator wants masks \(M_1,\dots,M_n\). Without assistance, annotation cost may be high. With a promptable model, the annotator supplies prompts and corrects masks.

Let \(C_{\mathrm{manual}}(i)\) be the cost of manually labeling image \(i\), and let \(C_{\mathrm{prompt}}(i)\) be the cost of prompting and correcting. A model is useful if

\[ \mathbb E C_{\mathrm{prompt}} < \mathbb E C_{\mathrm{manual}} \]

while maintaining acceptable label quality.

But annotation acceleration can introduce bias. The model may make some object types easier to annotate than others. It may encourage annotators to accept plausible but systematically wrong boundaries. It may also propagate training-set biases into new datasets.

A statistical audit should examine:

• error rate by object size;
• error rate by boundary complexity;
• error rate by image domain;
• false positive and false negative patterns;
• inter-annotator agreement with and without model assistance;
• calibration of predicted mask quality.

29.23 28.22 Evaluation Beyond Average IoU

Average IoU is useful but incomplete. Promptable segmentation should be evaluated along several axes.

29.23.1 Accuracy

For each prompt type, measure IoU, Dice, boundary F-score, and failure rate.

29.23.2 Prompt efficiency

In interactive segmentation, measure clicks required to reach a target IoU:

\[ T_\alpha = \min\{t: \operatorname{IoU}(\widehat M_t,M)\ge \alpha\}. \]

Then report

\[ \mathbb E[T_\alpha] \]

or the distribution of \(T_\alpha\).

29.23.3 Calibration

If \(s\) is predicted IoU, calibration asks whether

\[ \mathbb E[\operatorname{IoU}(\widehat M,M)\mid s\approx r] \approx r. \]

A reliability curve can compare predicted quality to empirical quality.

29.23.4 Robustness

A robust promptable model should not change drastically under small prompt perturbations. For a prompt \(p\) and perturbation \(\delta\), one can measure

\[ D(F_\theta(x,p),F_\theta(x,p+\delta)), \]

where \(D\) may be \(1-\operatorname{IoU}\).

29.23.5 Domain generalization

Evaluate on new domains: medical, satellite, microscopy, robotics, historical documents, low-light images, sketches, thermal images, and scientific imagery.

29.24 28.23 Interactive Demo: Quality Calibration

A promptable model may predict mask quality. This toy reliability diagram compares predicted IoU bins with empirical IoU values.

29.25 28.24 Failure Modes

Promptable segmentation models can fail in ways that are mathematically understandable.

29.25.1 Boundary ambiguity

If image gradients are weak or boundaries are occluded, many masks have similar evidence. The posterior \(P(M\mid x,p)\) is broad.

29.25.2 Scale ambiguity

A point on a wheel may refer to the wheel, the bicycle, or the scene object group. This is a multimodal posterior problem.

29.25.3 Semantic ambiguity

A text prompt may be vague, culturally dependent, or not visually grounded.

29.25.4 Domain shift

Images from microscopes, scans, thermal cameras, or scientific instruments may lie far from the training distribution.

29.25.5 Small objects and thin structures

Patch-based encoders may lose fine structures if the spatial resolution is too low. This is a discretization and representation problem.

29.25.6 Calibration failure

The predicted quality score may be high even when the mask is wrong. This is dangerous in annotation pipelines and high-stakes applications.

29.25.7 Prompt sensitivity

Small changes in prompt location may cause large changes in the predicted mask. Mathematically, the map

\[ p\mapsto F_\theta(x,p) \]

may have high local Lipschitz constant.

A robustness diagnostic is

\[ L_{\mathrm{local}}(x,p) =\sup_{\|\delta\|\le \epsilon} \frac{D(F_\theta(x,p),F_\theta(x,p+\delta))}{\|\delta\|}. \]

Large values indicate prompt instability.

29.26 28.25 Python Example: Prompt Encoding and Toy Mask Decoder

The following example implements a toy prompt-conditioned mask model. It does not train a neural network; it illustrates how point prompts can be converted into soft spatial masks.

import numpy as np
import matplotlib.pyplot as plt

H, W = 96, 128
yy, xx = np.mgrid[0:H, 0:W]

# Synthetic image-like objectness field with two objects.
obj1 = np.exp(-(((xx - 45) / 18) ** 2 + ((yy - 50) / 24) ** 2) / 2)
obj2 = np.exp(-(((xx - 88) / 14) ** 2 + ((yy - 45) / 16) ** 2) / 2)
image_score = 0.7 * obj1 + 0.9 * obj2

positive_points = [(42, 52)]
negative_points = [(89, 45)]

def radial_prompt_field(points, sign=1.0, sigma=12.0):
    field = np.zeros((H, W), dtype=float)
    for u, v in points:
        field += sign * np.exp(-((xx - u) ** 2 + (yy - v) ** 2) / (2 * sigma ** 2))
    return field

logits = (
    3.0 * image_score
    + 2.5 * radial_prompt_field(positive_points, sign=1.0)
    + 2.5 * radial_prompt_field(negative_points, sign=-1.0)
    - 1.2
)
soft_mask = 1 / (1 + np.exp(-logits))

plt.figure(figsize=(10, 3))
plt.subplot(1, 3, 1)
plt.imshow(image_score, cmap="gray")
plt.title("synthetic image score")
plt.axis("off")

plt.subplot(1, 3, 2)
plt.imshow(soft_mask, cmap="viridis", vmin=0, vmax=1)
plt.scatter([42], [52], c="lime", marker="o", label="positive")
plt.scatter([89], [45], c="red", marker="x", label="negative")
plt.title("prompt-conditioned soft mask")
plt.axis("off")

plt.subplot(1, 3, 3)
plt.imshow(soft_mask > 0.5, cmap="gray")
plt.title("thresholded mask")
plt.axis("off")
plt.show()

29.27 28.26 Python Example: IoU, Dice, and Best-of-K Selection

import numpy as np

rng = np.random.default_rng(7243)
H, W = 64, 64
yy, xx = np.mgrid[0:H, 0:W]

def ellipse(cx, cy, rx, ry):
    return (((xx - cx) / rx) ** 2 + ((yy - cy) / ry) ** 2 <= 1)

y_true = ellipse(32, 31, 14, 10)

candidates = [
    ellipse(32, 31, 18, 13),  # too large
    ellipse(36, 31, 14, 10),  # shifted
    ellipse(32, 31, 13, 9),   # close
]

def iou(a, b):
    inter = np.logical_and(a, b).sum()
    union = np.logical_or(a, b).sum()
    return inter / union

def dice(a, b):
    inter = np.logical_and(a, b).sum()
    return 2 * inter / (a.sum() + b.sum())

scores = []
for k, y_hat in enumerate(candidates):
    scores.append((k, iou(y_hat, y_true), dice(y_hat, y_true)))

for k, s_iou, s_dice in scores:
    print(f"candidate {k}: IoU={s_iou:.3f}, Dice={s_dice:.3f}")

best = max(scores, key=lambda row: row[1])
print("Best candidate by IoU:", best[0])

29.28 28.27 TensorFlow Example: Tiny Prompt-Conditioned Decoder

The following example sketches a small prompt-conditioned mask decoder. It is intentionally simplified. The key idea is that the prompt embedding conditions the mask logits.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

H, W, C = 128, 128, 3
D = 64

image = keras.Input(shape=(H, W, C), name="image")
point = keras.Input(shape=(2,), name="point_xy_normalized")
point_sign = keras.Input(shape=(1,), name="point_sign")

# A tiny image encoder.
z = layers.Conv2D(32, 3, padding="same", activation="relu")(image)
z = layers.MaxPool2D()(z)
z = layers.Conv2D(D, 3, padding="same", activation="relu")(z)
z = layers.MaxPool2D()(z)  # H/4 by W/4 by D

# A tiny prompt encoder.
p = layers.Concatenate()([point, point_sign])
p = layers.Dense(D, activation="relu")(p)
p = layers.Dense(D, activation="relu")(p)

# Broadcast prompt embedding over spatial locations.
p_map = layers.Reshape((1, 1, D))(p)
p_map = tf.tile(p_map, [1, H // 4, W // 4, 1])

# Prompt-conditioned feature fusion.
fused = layers.Concatenate()([z, p_map])
fused = layers.Conv2D(D, 3, padding="same", activation="relu")(fused)
logits = layers.Conv2D(1, 1, padding="same")(fused)
logits = layers.UpSampling2D(size=(4, 4), interpolation="bilinear")(logits)
mask_prob = layers.Activation("sigmoid", name="mask_probability")(logits)

model = keras.Model(inputs=[image, point, point_sign], outputs=mask_prob)
model.compile(
    optimizer=keras.optimizers.Adam(1e-3),
    loss=keras.losses.BinaryCrossentropy()
)
model.summary()

This is not a foundation model, but it illustrates the computational pattern:

\[ \text{image features} + \text{prompt embedding} \longrightarrow \text{mask logits}. \]

29.29 28.28 AI Component: Promptable Segmentation Study Assistant

Use an AI assistant as a mathematical critic rather than as a replacement for reasoning. Good prompts include:

1. Define the variables.
   “In promptable segmentation, identify the domain and codomain of the image encoder, prompt encoder, and mask decoder.”

2. Check the probabilistic interpretation.
   “Explain why promptable segmentation approximates \(P(Y\mid X,P)\) rather than \(P(Y\mid X)\). Give one example where this distinction matters.”

3. Debug a loss function.
   “I used pixel accuracy for small-object segmentation. Explain why IoU or Dice may be more appropriate.”

4. Analyze a failure mode.
   “A point prompt inside a bicycle wheel returns the whole bicycle. Is this necessarily an error? Explain in terms of ambiguity and posterior modes.”

5. Design an experiment.
   “Propose an experiment to compare center-point prompts, boundary-point prompts, and box prompts for interactive segmentation.”

6. Connect to statistics.
   “Explain prompt efficiency as a stopping-time random variable \(T_\alpha\). What distributional summaries should be reported?”

29.30 28.29 Mini-Project: Prompt Efficiency Curves

A useful independent-study project is to simulate interactive segmentation and produce prompt-efficiency curves.

29.30.1 Data

Use a public segmentation dataset or generate synthetic masks.

29.30.2 Model

Use a simple baseline first:

• initial mask from thresholding or a pretrained model;
• positive click in the largest false-negative component;
• negative click in the largest false-positive component;
• update by a simple radial correction or a learned lightweight model.

29.30.3 Metrics

For each image, record

\[ T_{0.85}=\min\{t: \operatorname{IoU}(\widehat M_t,M)\ge 0.85\}. \]

Report mean, median, quantiles, and failure rate.

29.30.4 Questions

• Which objects need many clicks?
• Are boundary prompts better than center prompts?
• Does object size predict prompt difficulty?
• Does the update rule over-correct?
• How does uncertainty change with clicks?

29.31 28.30 Connections to Other Areas

29.31.1 Medical imaging

Promptable segmentation can help radiologists, pathologists, and biomedical researchers annotate structures. But domain shift, calibration, and expert validation are crucial.

29.31.2 Scientific imaging

Microscopy, materials science, astronomy, and geoscience often require segmenting structures that are not common everyday objects. Foundation models may help, but they must be audited carefully.

29.31.3 Robotics

A robot may prompt a segmentation model using points, boxes, language, or task goals. The mask can support grasping, navigation, and manipulation.

29.31.4 Video segmentation

Prompts in one frame can propagate through time. The mathematical problem becomes spatiotemporal:

\[ F_\theta(x_{1:T},p)\to M_{1:T}. \]

Temporal consistency and tracking become important.

29.31.5 3D and neural fields

Masks can supervise 3D reconstruction, NeRF-style models, Gaussian splatting, or object-centric scene representations. A 2D prompt may correspond to a 3D object region.

29.32 28.31 Summary

Promptable segmentation is a modern synthesis of ideas from the entire book.

• From classical segmentation, it inherits masks, boundaries, region energies, and overlap metrics.
• From projective and geometric vision, it inherits coordinate systems and spatial reasoning.
• From probability, it inherits conditional inference and uncertainty.
• From optimization, it inherits loss functions, ranking, calibration, and sequential improvement.
• From transformers, it inherits attention-based interaction between image tokens and prompt tokens.
• From representation learning, it inherits large pretrained encoders.
• From vision-language models, it inherits open-vocabulary and text-guided interfaces.

The mathematical core is concise:

\[ F_\theta(x,p)=D_\theta(\Phi_\theta(x),\Psi_\theta(p)). \]

The meaning of this equation is broad. The image encoder builds a reusable representation. The prompt encoder describes the user’s or downstream model’s intent. The mask decoder turns their interaction into candidate spatial structures. In modern computer vision, this is one of the clearest examples of an algorithm becoming an interactive mathematical system.

29.33 Exercises

29.33.1 Conceptual Exercises

1. Explain the difference between semantic segmentation, instance segmentation, and promptable segmentation.
2. Give three examples where a single point prompt is ambiguous.
3. Why is \(P(Y\mid X,P)\) a better mathematical model than \(P(Y\mid X)\) for promptable segmentation?
4. Explain why a model may return multiple masks for one prompt.
5. What is the difference between a quality score and a probability that a mask is correct?
6. Describe a domain-shift failure mode for a segmentation foundation model.
7. Explain why promptable segmentation can accelerate annotation but also introduce annotation bias.

29.33.2 Mathematical Exercises

1. Let \(M\) and \(\widehat M\) be finite masks. Prove that \[ \operatorname{Dice}(M,\widehat M) =\frac{2\operatorname{IoU}(M,\widehat M)}{1+\operatorname{IoU}(M,\widehat M)}. \] whenever both masks are nonempty.

2. Suppose \(q^+\) is a positive point prompt and \(M\) is a candidate mask. Write a soft penalty that encourages \(q^+\in M\) when the mask is represented by a probability field \(\pi\).

3. For the soft Dice loss, compute the derivative with respect to a predicted probability \(\pi_j\).

4. Let \(s\) be a predicted IoU score. Define a calibration error using bins \(B_1,\dots,B_K\) and empirical IoU values.

5. Suppose an interactive method reaches IoU at least \(\alpha\) after a random number \(T_\alpha\) of prompts. Propose estimators for \(E[T_\alpha]\) and \(P(T_\alpha>m)\).

6. Let \(F(x,p)\) be a mask-valued model and define \(D(A,B)=1-\operatorname{IoU}(A,B)\). Explain how to estimate a local prompt-sensitivity score empirically.

29.33.3 Python Exercises

1. Implement IoU, Dice, boundary F-score, and soft Dice loss for binary masks.
2. Generate synthetic ellipses and sample positive/negative point prompts from them.
3. Build a toy prompt-conditioned mask predictor using radial basis functions centered at prompts.
4. Simulate interactive correction by adding positive clicks in false-negative regions and negative clicks in false-positive regions.
5. Produce a prompt-efficiency curve showing mean IoU as a function of number of clicks.
6. Train the tiny TensorFlow prompt decoder on synthetic shapes.
7. Compare center-point prompts, random positive prompts, and boundary prompts.
8. Build a reliability diagram for predicted mask quality scores.

29.33.4 AI-Assisted Exercises

1. Ask an AI assistant to explain SAM-style promptable segmentation. Then identify which parts are architecture, which parts are probability, and which parts are optimization.
2. Ask for a derivation of the soft Dice gradient. Verify the derivation by finite differences.
3. Ask for three evaluation protocols for interactive segmentation. Critique each protocol statistically.
4. Ask for possible failure modes in medical image segmentation. Classify them into domain shift, ambiguity, calibration, and resolution failure.
5. Ask the assistant to design a mini-project using promptable segmentation for a scientific-image dataset. Rewrite the proposed project as a precise mathematical problem.

29.34 Notes and References

Promptable segmentation is strongly associated with the Segment Anything line of work (Kirillov et al. 2023). The chapter also connects to earlier segmentation and dense-prediction ideas including U-Net (Ronneberger et al. 2015), Mask R-CNN (He et al. 2017), DeepLab-style segmentation (Chen et al. 2018), DETR-style set prediction (Carion et al. 2020), and vision transformers (Dosovitskiy et al. 2020; Vaswani et al. 2017). Vision-language prompting and open-vocabulary segmentation build on contrastive image-text representation learning such as CLIP (Radford et al. 2021). Self-supervised representation learning and large visual encoders are discussed in connection with DINOv2 and related methods (Oquab et al. 2023).

Carion, Nicolas, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. “End-to-End Object Detection with Transformers.” European Conference on Computer Vision, 213–29. https://arxiv.org/abs/2005.12872.

Chen, Liang-Chieh, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018. “Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation.” European Conference on Computer Vision, 801–18. https://arxiv.org/abs/1802.02611.

Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, et al. 2020. “An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.” arXiv Preprint arXiv:2010.11929. https://arxiv.org/abs/2010.11929.

He, Kaiming, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. “Mask r-CNN.” IEEE International Conference on Computer Vision, 2961–69. https://arxiv.org/abs/1703.06870.

Kirillov, Alexander, Eric Mintun, Nikhila Ravi, et al. 2023. “Segment Anything.” Proceedings of the IEEE/CVF International Conference on Computer Vision. https://arxiv.org/abs/2304.02643.

Oquab, Maxime, Timothée Darcet, Théo Moutakanni, et al. 2023. “DINOv2: Learning Robust Visual Features Without Supervision.” arXiv Preprint arXiv:2304.07193. https://arxiv.org/abs/2304.07193.

Radford, Alec, Jong Wook Kim, Chris Hallacy, et al. 2021. “Learning Transferable Visual Models from Natural Language Supervision.” International Conference on Machine Learning. https://arxiv.org/abs/2103.00020.

Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. 2015. “U-Net: Convolutional Networks for Biomedical Image Segmentation.” Medical Image Computing and Computer-Assisted Intervention.

Vaswani, Ashish, Noam Shazeer, Niki Parmar, et al. 2017. “Attention Is All You Need.” Advances in Neural Information Processing Systems 30. https://arxiv.org/abs/1706.03762.