30  Offline Reinforcement Learning

Core idea: offline reinforcement learning learns a decision rule from a fixed dataset of past interactions. Unlike online RL, the learner cannot collect new corrective data. The central mathematical difficulty is distribution shift: the learned policy may choose actions that were rarely or never observed in the dataset, so Bellman backups can extrapolate beyond statistical support.

30.1 Learning goals

After reading this chapter, students should be able to:

• distinguish online RL, imitation learning, off-policy evaluation, and offline RL;
• describe the role of the behavior policy in generating a fixed dataset;
• state support and coverage assumptions for offline policy evaluation and optimization;
• derive trajectory importance sampling and explain why its variance can be large;
• explain fitted Q evaluation and fitted Q iteration as supervised-learning versions of Bellman equations;
• identify extrapolation error caused by out-of-distribution actions;
• explain pessimism, conservatism, and policy constraints as remedies for offline RL;
• derive a basic conservative Q-learning penalty;
• use Python simulation to study coverage, importance weights, fitted Q evaluation, and conservative learning;
• use AI tools responsibly to audit assumptions, debug implementations, and design offline-RL experiments.

30.2 27.1 From online RL to offline RL

In online reinforcement learning, an agent repeatedly interacts with an environment. At time \(t\), it observes \(S_t\), chooses \(A_t\), receives \(R_{t+1}\), and observes \(S_{t+1}\). The learner may improve its policy and collect new data under the improved policy.

In offline reinforcement learning, also called batch reinforcement learning, the learner receives a fixed dataset

\[ D=igl\{(S_i,A_i,R_i,S_i')\bigr\}_{i=1}^n. \]

The learner cannot ask for additional transitions. The data are usually generated by some unknown behavior policy \(\pi_b\):

\[ A_i \sim \pi_b(\cdot \mid S_i), \qquad S_i' \sim P(\cdot \mid S_i,A_i). \]

The goal is to construct a new policy \(\widehat \pi\) with large value \(V^{\widehat \pi}\), using only \(D\).

This setting is important in applications where online exploration is expensive or unsafe:

• medical treatment recommendations;
• education interventions;
• finance and portfolio allocation;
• robotics from logged demonstrations;
• recommendation systems from historical interaction logs;
• operations and inventory systems using past operating data.

The offline constraint. In online RL, a bad estimate can sometimes be corrected by new exploration. In offline RL, a bad estimate may become a bad policy with no opportunity for correction before deployment.

30.3 27.2 The statistical object: a logged dataset

Assume a discounted finite MDP with state space \(S\), action space \(A\), transition kernel \(P\), reward function \(r\), and discount factor \(\gamma\in(0,1)\). The offline dataset is drawn from a data-generating distribution

\[ \mu_b(s,a,s') = \nu_b(s)\pi_b(a\mid s)P(s'\mid s,a), \]

where \(\nu_b\) is the state distribution induced by the logging process. In a long trajectory, \(\nu_b\) is often close to a stationary or discounted occupancy distribution under \(\pi_b\).

For a candidate target policy \(\pi\), the value depends on its own occupancy distribution. A discounted state-action occupancy measure is

\[ d^\pi_\rho(s,a) = (1-\gamma) \sum_{t=0}^{\infty}\gamma^t P_\pi(S_t=s,A_t=a\mid S_0\sim \rho). \]

The dataset is informative for \(\pi\) only if the important state-action pairs for \(\pi\) are represented in the data.

A basic support condition is

\[ d^\pi_\rho(s,a)>0 \quad \Longrightarrow \quad \mu_b(s,a)>0. \]

In words, the target policy should not rely on state-action pairs absent from the dataset.

30.3.1 Interactive: dataset coverage and target-policy support

The target policy may place probability mass on actions that the behavior policy rarely used. The overlap between the two policies determines how difficult offline evaluation and optimization become.

30.4 27.3 Coverage and concentrability

The support condition prevents complete ignorance, but it is not enough for stable learning. If the behavior probability \(\pi_b(a\mid s)\) is extremely small, then estimates involving that pair have high variance.

A common theoretical quantity is a concentrability coefficient:

\[ C(\pi,\pi_b) = \sup_{s,a} \frac{d^\pi_\rho(s,a)}{\mu_b(s,a)}. \]

When \(C(\pi,\pi_b)\) is small, the offline data cover the target policy well. When it is large, target-policy behavior is poorly covered. If the denominator is zero while the numerator is positive, then \(C(\pi,\pi_b)=\infty\).

This ratio is the RL analogue of covariate shift in supervised learning. But in RL, a policy change affects not only the action distribution at a fixed state, but also the future state distribution.

Offline RL principle. A policy should be trusted only in regions of the state-action space where the dataset has enough support. Offline RL algorithms are often designed to keep the learned policy close to the data or to be pessimistic outside the data.

30.5 27.4 Offline policy evaluation

Before optimizing a new policy, we often want to estimate the value of a fixed target policy \(\pi\) from data generated by another policy \(\pi_b\). This is off-policy evaluation.

For an episodic problem with horizon \(T\), the return is

\[ G_0 = \sum_{t=0}^{T-1}\gamma^t R_{t+1}. \]

If trajectories are generated by \(\pi_b\), the likelihood ratio between \(\pi\) and \(\pi_b\) is

\[ W_{0:T-1} = \prod_{t=0}^{T-1} \frac{\pi(A_t\mid S_t)}{\pi_b(A_t\mid S_t)}. \]

The ordinary importance sampling estimator is

\[ \widehat J_{\mathrm{IS}}(\pi) = \frac{1}{m}\sum_{i=1}^m W^{(i)}_{0:T-1}G^{(i)}_0. \]

It is unbiased under correct behavior-policy probabilities and sufficient support, but its variance can be very large. The product of ratios across time is the main source of instability.

A weighted importance sampling estimator is

\[ \widehat J_{\mathrm{WIS}}(\pi) = \frac{\sum_{i=1}^m W^{(i)}_{0:T-1}G^{(i)}_0} {\sum_{i=1}^m W^{(i)}_{0:T-1}}. \]

It is usually biased but often has smaller variance.

30.5.1 Interactive: importance-weight variance

Small mismatch at each step can produce very large trajectory weights over a long horizon. This is one reason offline RL is harder than ordinary supervised learning.

import numpy as np

rng = np.random.default_rng(7243)

m = 5000
T = 12
gamma = 0.95

# Behavior and target probabilities of action 1 in a two-action problem.
p_b = 0.45
p_t = 0.65

returns = []
weights = []

for _ in range(m):
    w = 1.0
    G = 0.0
    for t in range(T):
        a = rng.binomial(1, p_b)
        reward = 1.0 if a == 1 else 0.2
        ratio = (p_t if a == 1 else 1 - p_t) / (p_b if a == 1 else 1 - p_b)
        w *= ratio
        G += (gamma ** t) * reward
    returns.append(G)
    weights.append(w)

returns = np.array(returns)
weights = np.array(weights)
ordinary_is = np.mean(weights * returns)
weighted_is = np.sum(weights * returns) / np.sum(weights)
behavior_value = np.mean(returns)

print("Behavior-policy sample return:", round(behavior_value, 3))
print("Ordinary IS estimate:", round(ordinary_is, 3))
print("Weighted IS estimate:", round(weighted_is, 3))
print("Mean importance weight:", round(weights.mean(), 3))
print("Std of importance weights:", round(weights.std(), 3))
print("Max importance weight:", round(weights.max(), 3))

Behavior-policy sample return: 5.13
Ordinary IS estimate: 6.227
Weighted IS estimate: 6.568
Mean importance weight: 0.948
Std of importance weights: 2.081
Max importance weight: 36.342

The estimator may be noisy even in this very small problem. In long-horizon RL, the product of likelihood ratios can be much more unstable.

30.6 27.5 Per-decision importance sampling

Trajectory-level importance sampling applies one large weight to the entire return. A more refined estimator weights each reward only by the ratios needed up to that reward:

\[ \widehat J_{\mathrm{PDIS}}(\pi) = \frac{1}{m}\sum_{i=1}^m \sum_{t=0}^{T-1} \gamma^t \left( \prod_{k=0}^{t} \frac{\pi(A_k^{(i)}\mid S_k^{(i)})}{\pi_b(A_k^{(i)}\mid S_k^{(i)})} \right) R_{t+1}^{(i)}. \]

Per-decision importance sampling often reduces variance because early rewards are not multiplied by future action ratios.

Still, the variance can remain large when policies differ substantially or the horizon is long.

30.7 27.6 Model-based offline evaluation

Another approach is to estimate a model from the dataset:

\[ \widehat r(s,a), \qquad \widehat P(s'\mid s,a). \]

For a fixed policy \(\pi\), form

\[ \widehat r_\pi(s)=\sum_a\pi(a\mid s)\widehat r(s,a), \]

and

\[ \widehat P_\pi(s,s')=\sum_a\pi(a\mid s)\widehat P(s'\mid s,a). \]

The plug-in value estimate is

\[ \widehat V^\pi = (I-\gamma \widehat P_\pi)^{-1}\widehat r_\pi. \]

This estimator can be efficient if the model is accurate. But it can be biased if the model extrapolates poorly for rare actions. In offline RL, rare actions are precisely the dangerous ones.

import numpy as np

rng = np.random.default_rng(2026)
S = 3
A = 2
gamma = 0.90

P_true = np.array([
    [[0.80, 0.20, 0.00], [0.10, 0.60, 0.30]],
    [[0.10, 0.80, 0.10], [0.00, 0.20, 0.80]],
    [[0.30, 0.10, 0.60], [0.05, 0.15, 0.80]],
])
r_true = np.array([
    [0.2, 0.4],
    [0.1, 1.0],
    [0.3, 0.7],
])

pi_b = np.array([
    [0.85, 0.15],
    [0.80, 0.20],
    [0.75, 0.25],
])
pi_target = np.array([
    [0.20, 0.80],
    [0.30, 0.70],
    [0.25, 0.75],
])

n = 4000
s = 0
counts_sa = np.zeros((S, A))
counts_sas = np.zeros((S, A, S))
reward_sums = np.zeros((S, A))

for _ in range(n):
    a = rng.choice(A, p=pi_b[s])
    r = r_true[s, a] + 0.05 * rng.normal()
    sp = rng.choice(S, p=P_true[s, a])
    counts_sa[s, a] += 1
    counts_sas[s, a, sp] += 1
    reward_sums[s, a] += r
    s = sp

# Add small smoothing to avoid zero rows.
P_hat = (counts_sas + 1.0) / (counts_sa[:, :, None] + S)
r_hat = np.divide(reward_sums, np.maximum(counts_sa, 1))

P_pi_hat = np.einsum("sa,sak->sk", pi_target, P_hat)
r_pi_hat = np.sum(pi_target * r_hat, axis=1)
V_hat = np.linalg.solve(np.eye(S) - gamma * P_pi_hat, r_pi_hat)

P_pi_true = np.einsum("sa,sak->sk", pi_target, P_true)
r_pi_true = np.sum(pi_target * r_true, axis=1)
V_true = np.linalg.solve(np.eye(S) - gamma * P_pi_true, r_pi_true)

print("state-action visit counts")
print(counts_sa.astype(int))
print("plug-in V estimate:", np.round(V_hat, 3))
print("true V:", np.round(V_true, 3))
print("absolute error:", np.round(np.abs(V_hat - V_true), 3))

state-action visit counts
[[1155  209]
 [1132  305]
 [ 935  264]]
plug-in V estimate: [5.808 6.2   6.016]
true V: [5.829 6.213 6.028]
absolute error: [0.02  0.013 0.011]

The visit-count table is as important as the value estimate. A value estimate is not credible if it depends heavily on state-action pairs that were rarely observed.

30.8 27.7 Fitted Q evaluation

Fitted Q evaluation, or FQE, estimates \(Q^\pi\) by repeatedly solving supervised regression problems. The Bellman equation for a fixed target policy is

\[ Q^\pi(s,a) = r(s,a)+\gamma E_{S'\sim P(\cdot\mid s,a)} \left[\sum_{a'}\pi(a'\mid S')Q^\pi(S',a')\right]. \]

Given a dataset \(D\), define the target at iteration \(k\) as

\[ y_i^{(k)} = R_i+ \gamma\sum_{a'}\pi(a'\mid S_i')Q_k(S_i',a'). \]

Then fit \(Q_{k+1}\) by minimizing

\[ \sum_{i=1}^n \left(Q(S_i,A_i)-y_i^{(k)}\right)^2. \]

In a tabular setting, this is equivalent to averaging targets over observed state-action pairs. With function approximation, this becomes ordinary supervised learning with Bellman targets.

30.8.1 Interactive: fitted Q evaluation convergence

FQE repeatedly alternates between Bellman target construction and supervised regression.

import numpy as np

rng = np.random.default_rng(5010)
S = 4
A = 2
gamma = 0.92

P = np.array([
    [[0.80, 0.20, 0.00, 0.00], [0.15, 0.65, 0.20, 0.00]],
    [[0.05, 0.80, 0.15, 0.00], [0.00, 0.20, 0.60, 0.20]],
    [[0.00, 0.10, 0.80, 0.10], [0.00, 0.00, 0.25, 0.75]],
    [[0.15, 0.00, 0.10, 0.75], [0.05, 0.05, 0.10, 0.80]],
])
r = np.array([
    [0.1, 0.4],
    [0.0, 0.8],
    [0.2, 1.0],
    [0.3, 0.5],
])

pi_b = np.array([[0.85, 0.15], [0.80, 0.20], [0.75, 0.25], [0.70, 0.30]])
pi = np.array([[0.25, 0.75], [0.20, 0.80], [0.15, 0.85], [0.30, 0.70]])

# Generate an offline transition dataset.
n = 6000
s = 0
data = []
for _ in range(n):
    a = rng.choice(A, p=pi_b[s])
    reward = r[s, a] + 0.03 * rng.normal()
    sp = rng.choice(S, p=P[s, a])
    data.append((s, a, reward, sp))
    s = sp

Q = np.zeros((S, A))
errors = []

# True Q for comparison.
P_pi = np.einsum("sa,sak->sk", pi, P)
r_pi = np.sum(pi * r, axis=1)
V_true = np.linalg.solve(np.eye(S) - gamma * P_pi, r_pi)
Q_true = r + gamma * np.einsum("sak,k->sa", P, V_true)

for k in range(50):
    target_sums = np.zeros((S, A))
    counts = np.zeros((S, A))
    for s, a, reward, sp in data:
        target = reward + gamma * np.dot(pi[sp], Q[sp])
        target_sums[s, a] += target
        counts[s, a] += 1
    mask = counts > 0
    Q_new = Q.copy()
    Q_new[mask] = target_sums[mask] / counts[mask]
    Q = Q_new
    errors.append(np.max(np.abs(Q - Q_true)))

print("final fitted Q")
print(np.round(Q, 3))
print("true Q")
print(np.round(Q_true, 3))
print("final max error:", round(errors[-1], 4))

final fitted Q
[[6.201 6.703]
 [6.333 7.059]
 [6.488 6.965]
 [6.233 6.434]]
true Q
[[6.317 6.816]
 [6.446 7.167]
 [6.616 7.1  ]
 [6.353 6.56 ]]
final max error: 0.1343

FQE is not a control algorithm by itself. It evaluates a chosen target policy. It becomes a building block for offline policy selection and offline policy improvement.

30.9 27.8 Extrapolation error

In fitted Q iteration, the control target uses a maximum:

\[ y_i^{(k)} = R_i+ \gamma\max_{a'}Q_k(S_i',a'). \]

The maximum may select an action \(a'\) that is poorly represented in the dataset at state \(S_i'\). If the learned \(Q_k(S_i',a')\) is inaccurate because of extrapolation, the error can be amplified by the Bellman backup.

This is called extrapolation error or out-of-distribution action error.

A simple decomposition is

\[ Q_{\mathrm{learned}}(s,a) = Q_{\mathrm{true}}(s,a)+\varepsilon(s,a). \]

Then a greedy backup uses

\[ \max_a Q_{\mathrm{learned}}(s,a) = \max_a \bigl(Q_{\mathrm{true}}(s,a)+\varepsilon(s,a)\bigr). \]

Even if \(E[\varepsilon(s,a)]=0\), the maximum tends to select positive errors. This is the same statistical mechanism behind maximization bias, now intensified by distribution shift.

30.9.1 Interactive: extrapolation error under action maximization

When one action is poorly covered by the dataset, its estimated value may be noisy. A max backup can repeatedly select that noisy estimate.

30.10 27.9 Offline policy optimization

Offline policy optimization tries to find a policy using only the dataset. A naive objective is

\[ \widehat \pi \in \arg\max_\pi \widehat J(\pi), \]

where \(\widehat J(\pi)\) is an off-policy value estimate. This can fail badly because the optimization procedure searches over policies and may exploit errors in \(\widehat J\).

The offline-RL problem is therefore not merely evaluation. It is optimization under statistical uncertainty. A policy that looks best under a noisy estimator may be exactly the policy that exploits estimation error.

One way to state the challenge is:

\[ \max_\pi J(\pi) \quad \text{subject to} \quad \pi \text{ remains supported by the dataset.} \]

Practical algorithms impose this idea in different ways:

• constrain the policy close to the behavior policy;
• penalize actions with low data density;
• learn a conservative value function;
• use lower-confidence value estimates;
• generate pessimistic models for unsupported transitions.

30.11 27.10 Behavior cloning and policy constraints

The simplest offline policy is behavior cloning:

\[ \widehat \pi_{\mathrm{BC}} \in \arg\max_\pi \sum_{i=1}^n \log \pi(A_i\mid S_i). \]

Behavior cloning is supervised learning from states to actions. It avoids out-of-distribution actions by imitating the dataset. However, it cannot improve beyond the behavior policy unless the data contain enough information to distinguish good from bad behavior.

A policy-constrained offline RL objective has the form

\[ \max_\pi \widehat J(\pi) \quad \text{subject to} \quad D(\pi(\cdot\mid s),\widehat\pi_b(\cdot\mid s))\leq \varepsilon \]

for a divergence \(D\), such as total variation or KL divergence.

Equivalently, one may solve a regularized problem:

\[ \max_\pi \widehat J(\pi) - \lambda E_{s\sim D} \left[D_{\mathrm{KL}}(\pi(\cdot\mid s)\|\widehat\pi_b(\cdot\mid s))\right]. \]

30.11.1 Interactive: policy constraint versus value improvement

A strong constraint keeps the learned policy close to the behavior policy. A weak constraint may allow improvement but risks unsupported actions.

30.12 27.11 Pessimism and lower confidence bounds

A mathematically clean approach is pessimism under uncertainty. Instead of choosing the policy with the largest estimated value, choose the policy with the largest lower confidence bound.

For a policy \(\pi\), suppose

\[ |\widehat J(\pi)-J(\pi)|\leq b(\pi) \]

with high probability. A pessimistic objective is

\[ \widehat \pi \in \arg\max_\pi \left[\widehat J(\pi)-b(\pi)\right]. \]

The penalty \(b(\pi)\) should be large when the target policy depends on poorly covered state-action pairs. This mirrors optimism in online exploration, but with the sign reversed.

Online versus offline uncertainty. Online RL is often optimistic: try uncertain actions to gather information. Offline RL is often pessimistic: avoid uncertain actions because no new information can be gathered.

30.13 27.12 Conservative Q-learning idea

Conservative Q-learning, or CQL, modifies value learning so that unsupported actions receive lower values. A common regularizer compares the value of actions sampled from the learned policy with actions sampled from the dataset.

A simplified tabular conservative objective at state \(s\) is

\[ \frac{1}{2}\left(Q(s,a_D)-y\right)^2 + \alpha \left( \log\sum_{a\in A}\exp(Q(s,a)) - Q(s,a_D) \right), \]

where \(a_D\) is the action observed in the dataset and \(y\) is a Bellman target. The term

\[ \log\sum_{a\in A}\exp(Q(s,a)) \]

pushes down the values of all actions unless supported by data. The subtraction \(-Q(s,a_D)\) prevents the penalty from simply shrinking all values equally.

A more conceptual version is

\[ \text{Bellman fitting loss} + \alpha \left( E_{s\sim D,a\sim \pi}[Q(s,a)] - E_{s\sim D,a\sim D}[Q(s,a)] \right). \]

This penalizes assigning high value to actions chosen by the learned policy but not well represented in the data.

30.13.1 Interactive: conservative penalty

Conservatism lowers values for actions not supported by the dataset. The penalty strength controls how much the learned Q-function distrusts unsupported actions.

import numpy as np

# A single-state two-action example.
# Action 0 is well observed and modestly good.
# Action 1 is rarely observed and has an uncertain value estimate.
Q_data_fit = np.array([1.0, 1.3])
visit_counts = np.array([200, 5])

# A simple uncertainty penalty proportional to 1 / sqrt(N).
alpha = 1.2
penalty = alpha / np.sqrt(visit_counts)
Q_pessimistic = Q_data_fit - penalty

print("data-fit Q:", np.round(Q_data_fit, 3))
print("visit counts:", visit_counts)
print("uncertainty penalty:", np.round(penalty, 3))
print("pessimistic Q:", np.round(Q_pessimistic, 3))
print("greedy action before pessimism:", int(np.argmax(Q_data_fit)))
print("greedy action after pessimism:", int(np.argmax(Q_pessimistic)))

data-fit Q: [1.  1.3]
visit counts: [200   5]
uncertainty penalty: [0.085 0.537]
pessimistic Q: [0.915 0.763]
greedy action before pessimism: 1
greedy action after pessimism: 0

This toy example is not CQL itself, but it illustrates the main idea: do not trust high estimated values for rarely observed actions.

30.14 27.13 Fitted Q iteration in offline RL

Fitted Q iteration, or FQI, uses the control Bellman target

\[ y_i^{(k)} = R_i+ \gamma\max_{a'}Q_k(S_i',a'). \]

Then it fits \(Q_{k+1}\) to the supervised dataset

\[ \bigl((S_i,A_i),y_i^{(k)}\bigr)_{i=1}^n. \]

In a tabular setting, this is simple averaging. In a large state space, one uses regression models such as linear features, random forests, neural networks, or gradient-boosted trees.

The offline danger is that the max over \(a'\) may select actions not well covered at \(S_i'\). Conservative or constrained variants modify the target or the optimization procedure to reduce this error.

import numpy as np

rng = np.random.default_rng(7339)
S = 5
A = 2
gamma = 0.90

P = np.zeros((S, A, S))
for s in range(S):
    P[s, 0, max(s - 1, 0)] = 0.70
    P[s, 0, s] += 0.30
    P[s, 1, min(s + 1, S - 1)] = 0.80
    P[s, 1, s] += 0.20
r = np.zeros((S, A))
r[:, 0] = 0.1
r[:, 1] = np.linspace(0.0, 1.0, S)

# Behavior mostly takes action 0, so action 1 is under-covered.
pi_b = np.tile(np.array([0.85, 0.15]), (S, 1))

n = 3000
s = 0
data = []
counts = np.zeros((S, A))
for _ in range(n):
    a = rng.choice(A, p=pi_b[s])
    sp = rng.choice(S, p=P[s, a])
    reward = r[s, a] + 0.02 * rng.normal()
    data.append((s, a, reward, sp))
    counts[s, a] += 1
    s = sp

Q_plain = np.zeros((S, A))
Q_pess = np.zeros((S, A))
coverage_penalty = 0.8 / np.sqrt(np.maximum(counts, 1))

for _ in range(80):
    sums_plain = np.zeros((S, A))
    sums_pess = np.zeros((S, A))
    c = np.zeros((S, A))
    for s, a, reward, sp in data:
        y_plain = reward + gamma * np.max(Q_plain[sp])
        y_pess = reward + gamma * np.max(Q_pess[sp] - coverage_penalty[sp])
        sums_plain[s, a] += y_plain
        sums_pess[s, a] += y_pess
        c[s, a] += 1
    mask = c > 0
    Q_plain[mask] = sums_plain[mask] / c[mask]
    Q_pess[mask] = sums_pess[mask] / c[mask]

print("state-action counts")
print(counts.astype(int))
print("plain FQI greedy actions:", np.argmax(Q_plain, axis=1))
print("pessimistic FQI greedy actions:", np.argmax(Q_pess - coverage_penalty, axis=1))
print("plain Q")
print(np.round(Q_plain, 2))
print("pessimistic adjusted Q")
print(np.round(Q_pess - coverage_penalty, 2))

state-action counts
[[2002  359]
 [ 416   86]
 [  88   17]
 [  26    3]
 [   3    0]]
plain FQI greedy actions: [1 1 1 1 0]
pessimistic FQI greedy actions: [1 1 1 0 0]
plain Q
[[3.41 3.68]
 [3.55 4.19]
 [3.93 4.45]
 [4.09 4.39]
 [4.03 0.  ]]
pessimistic adjusted Q
[[ 0.91  0.92]
 [ 0.94  1.11]
 [ 0.99  1.03]
 [ 0.79  0.59]
 [ 0.33 -0.8 ]]

This code is deliberately simple. It shows how a coverage penalty can change the learned greedy policy.

30.15 27.14 Doubly robust estimation

Importance sampling uses action probabilities. Model-based evaluation uses an estimated value function. Doubly robust estimation combines both.

For a contextual bandit, a doubly robust estimator has the form

\[ \widehat J_{\mathrm{DR}}(\pi) = \frac{1}{n}\sum_{i=1}^n \left[ \sum_a \pi(a\mid X_i)\widehat q(X_i,a) + \frac{\pi(A_i\mid X_i)}{\pi_b(A_i\mid X_i)} \left(R_i-\widehat q(X_i,A_i)\right) \right]. \]

The first term is a model-based prediction. The second term corrects the prediction using importance-weighted residuals.

The estimator is called doubly robust because it can be consistent if either the behavior probabilities are correct or the outcome model is correct, under appropriate conditions.

30.15.1 Interactive: model bias and importance-weight variance

Doubly robust estimators balance model-based prediction and importance-weighted correction.

30.16 27.15 Offline RL as supervised learning plus constraints

Many offline RL methods can be summarized as supervised learning with Bellman targets plus a constraint or penalty.

Method family	Mathematical idea	Typical risk
Behavior cloning	maximize likelihood of logged actions	cannot improve much beyond behavior
Importance sampling	reweight behavior trajectories	high variance
FQE	fit Bellman expectation equation for fixed \(\pi\)	biased if function class is poor
FQI	fit Bellman optimality equation	extrapolation through max operator
Policy-constrained RL	keep \(\pi\) close to \(\pi_b\)	overly conservative if constraint is too strong
Conservative value learning	lower values for unsupported actions	may underestimate useful actions
Model-based offline RL	estimate \(P\) and \(r\)	model bias under distribution shift

The core mathematical tension is between improvement and support. A policy identical to \(\pi_b\) is safe but may be suboptimal. A policy far from \(\pi_b\) may improve, but its estimated value may be unreliable.

30.17 27.16 Practical offline-RL workflow

A careful offline-RL workflow should include the following steps.

1. Define the decision problem: states, actions, rewards, horizon, and discounting.
2. Audit the dataset: state-action coverage, missing actions, repeated trajectories, reward noise, censoring, and logging policy.
3. Estimate or approximate the behavior policy when needed.
4. Establish candidate policies that do not move too far outside the data distribution.
5. Use multiple off-policy evaluation methods, not only one.
6. Report uncertainty, not only point estimates.
7. Stress-test policies under perturbations of rewards, transitions, and behavior probabilities.
8. Deploy gradually only when the application allows safe monitoring.

Statistical warning. Offline RL should not be used as a black-box optimizer on historical data. Without support, value estimates can be confidently wrong.

30.18 27.17 AI-assisted learning components

30.18.1 AI derivation prompt

Ask an AI system:

Derive the ordinary importance sampling estimator for off-policy evaluation in an episodic MDP. State the support assumption clearly and explain why the variance grows with the horizon.

Then check whether the response includes:

• the trajectory likelihood ratio;
• the product over time;
• the behavior-policy denominator;
• the distinction between unbiasedness and variance;
• the support condition \(\pi_b(a\mid s)>0\) whenever \(\pi(a\mid s)>0\) on reachable histories.

30.18.2 AI code-review prompt

Give an AI system an implementation of fitted Q iteration and ask:

Find where this code may evaluate or maximize over actions that are not represented in the offline dataset. Suggest diagnostics using state-action counts.

A good response should identify the max backup as the dangerous step and recommend checking counts or density estimates for selected actions.

30.18.3 AI modeling prompt

For an applied project, ask:

Is offline RL appropriate for this dataset? Identify the state, action, reward, behavior policy, target policy class, and possible unobserved confounders.

Then manually verify that the AI did not assume causal validity merely because the dataset is large.

30.19 27.18 Summary

Offline RL studies sequential decision-making from fixed logged data. Its central challenge is distribution shift between the behavior policy that generated the data and the target policy we want to evaluate or optimize.

The main mathematical objects are:

\[ D=\{(S_i,A_i,R_i,S_i')\}_{i=1}^n, \]

\[ d^\pi_\rho(s,a), \qquad \mu_b(s,a), \qquad \frac{d^\pi_\rho(s,a)}{\mu_b(s,a)}. \]

Off-policy evaluation can be performed by importance sampling, model-based plug-in methods, fitted Q evaluation, or doubly robust estimators. Offline policy optimization requires additional protection against extrapolation error. Common protections include behavior constraints, pessimism, conservative Q-learning, and uncertainty penalties.

The guiding principle is simple:

\[ \text{improve only where the data provide support.} \]

30.20 Exercises

30.20.1 Conceptual exercises

1. Explain the difference between off-policy learning and offline RL.
2. Why is a large dataset not sufficient for offline RL if important actions were rarely taken?
3. Explain why ordinary importance sampling can be unbiased but practically unusable.
4. What is extrapolation error? Why does the max operator make it worse?
5. Compare behavior cloning and conservative Q-learning.
6. Explain why pessimism is natural in offline RL but optimism is natural in online exploration.

30.20.2 Mathematical exercises

1. Let \(\pi_b(a\mid s)>0\) whenever \(\pi(a\mid s)>0\). Derive the trajectory importance ratio for a finite-horizon MDP.
2. Show that trajectory importance sampling is unbiased under correct behavior probabilities and sufficient support.
3. Derive the FQE target for a fixed target policy \(\pi\).
4. For a one-state two-action MDP, compute how a KL constraint limits the target policy’s probability of the less observed action.
5. Consider the conservative objective \[ \frac{1}{2}(Q(s,a_D)-y)^2 +\alpha\left(\log\sum_a\exp(Q(s,a))-Q(s,a_D)\right). \] Compute the gradient with respect to \(Q(s,a)\) for each action.
6. Let \(b(s,a)=c/\sqrt{N(s,a)}\). Show how the pessimistic greedy policy changes when \(Q(s,a)\) is replaced by \(Q(s,a)-b(s,a)\).

30.20.3 Computational exercises

1. Simulate a two-action bandit with behavior policy \(\pi_b(1)=0.2\) and target policy \(\pi(1)=0.8\). Compare ordinary IS, weighted IS, and a model-based estimator.
2. Implement FQE for a finite MDP and compare the result with exact dynamic programming.
3. Implement fitted Q iteration using a fixed offline dataset. Track which actions the learned greedy policy selects and compare those actions with dataset counts.
4. Add a pessimistic count-based penalty \(c/\sqrt{N(s,a)}\) and study how the learned policy changes.
5. Compare behavior cloning, plain FQI, and pessimistic FQI on the same offline dataset.

30.20.4 AI-assisted exercises

1. Ask an AI tool to explain why offline RL is not just supervised learning. Critique the answer using the concepts of occupancy measures and Bellman backups.
2. Ask an AI tool to write FQE code. Check whether it evaluates the target policy or accidentally performs control with a max backup.
3. Ask an AI tool to propose offline-RL diagnostics for a healthcare dataset. Identify which suggestions are statistical, which are causal, and which are merely engineering checks.
4. Ask an AI tool to compare CQL and behavior cloning. Revise the answer so that it clearly distinguishes value pessimism from action imitation.

30.21 Instructor notes

This chapter should be taught as the statistical caution chapter of the book. Students have already seen Q-learning, DQN, policy gradients, and statistical learning theory. Offline RL forces them to combine those ideas with support, distribution shift, and finite-sample uncertainty.

A useful classroom demonstration is to create a dataset in which the behavior policy rarely takes a certain action. Plain fitted Q iteration may overvalue the rare action because of noisy targets. A count-based penalty or behavior constraint often fixes the issue, but can also prevent genuine improvement. This tradeoff is the main lesson.