20 Deep Q-Networks

Core idea. A deep Q-network, or DQN, replaces the tabular action-value table \(Q(s,a)\) by a neural approximation \(Q_ heta(s,a)\). The target is still the Bellman optimality equation,

\[ Q^*(s,a) = \mathbb E \left[ R_{t+1} + \gamma \max_{a'} Q^*(S_{t+1},a') \mid S_t=s,A_t=a \right], \]

but the fixed point is approximated by stochastic optimization over sampled transitions. The two stabilizing devices in the original DQN idea are experience replay and a target network. The mathematical viewpoint is

\[ \text{Q-learning} + \text{nonlinear function approximation} + \text{stabilized stochastic optimization}. \]

20.1 Learning goals

After reading this chapter, students should be able to:

explain how DQN generalizes tabular Q-learning;
write the DQN target and squared temporal-difference loss precisely;
distinguish the online network \(Q_ heta\) from the target network \(Q_{\theta^-}\);
explain why experience replay changes the statistical structure of the update data;
derive the gradient of the DQN loss with respect to the online-network parameters;
describe target-network updates, hard updates, and soft updates;
explain overestimation bias and the idea of Double DQN;
implement a small replay buffer and a NumPy action-value approximator;
identify the main instability mechanisms in deep value-based RL;
use AI tools to audit DQN equations, code, and modeling assumptions.

20.2 17.1 From tabular Q-learning to DQN

In tabular Q-learning, every state-action pair has its own parameter:

\[ Q(s,a) \quad \text{for each } (s,a). \]

The update from Chapter 12 is

\[ Q(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha \left[ R_{t+1} + \gamma \max_{a'} Q(S_{t+1},a') - Q(S_t,A_t) \right]. \]

This works when the state and action spaces are small enough to store a table. Many modern applications have high-dimensional states: images, text embeddings, sensor vectors, market features, or large simulation states. In that case, a table is not practical.

DQN replaces the table by a parameterized function

\[ Q_ heta:S\times A\to \mathbb R, \]

where \(\theta\) denotes all neural-network weights. For a finite action space, one common architecture maps a state \(s\) to a vector of action values:

\[ f_ heta(s) = \left(Q_ heta(s,a_1),\ldots,Q_ heta(s,a_m)\right). \]

The greedy action is then

\[ a_\theta(s) \in \arg\max_a Q_\theta(s,a). \]

The key mathematical change is that a single parameter vector \(\theta\) controls many state-action values simultaneously. One gradient update at one transition changes the predicted values of other states and actions as well.

DQN should not be viewed as a new Bellman equation. It is an approximate method for solving the same Bellman optimality equation, but in a nonlinear function class.

20.2.1 Interactive: From state to action values

This diagram shows the usual DQN architecture: a state vector is mapped through a feature representation to one scalar value for each discrete action.

20.3 17.2 The Bellman target

For a transition

\[ (s,a,r,s'), \]

the tabular Q-learning target is

\[ y = r+ \gamma \max_{a'} Q(s',a'). \]

In DQN, the current prediction is \(Q_\theta(s,a)\). A first attempt would use

\[ y_\theta = r+ \gamma\max_{a'}Q_\theta(s',a') \]

and minimize

\[ \left(y_\theta-Q_\theta(s,a)\right)^2. \]

However, this uses the same parameters both to define the target and to fit the prediction. As \(\theta\) changes, the target moves. This can create instability.

DQN introduces a separate target parameter vector \(\theta^-\). The target becomes

\[ y = r+ \gamma\max_{a'}Q_{\theta^-}(s',a'). \]

The online network \(Q_\theta\) is fitted to this temporarily fixed target. The target network is updated more slowly.

For terminal next states, the future term is omitted:

\[ y = r \quad \text{if } s' \text{ is terminal}. \]

A convenient way to write both cases is

\[ y = r+\gamma(1-d)\max_{a'}Q_{\theta^-}(s',a'), \]

where \(d=1\) for terminal transitions and \(d=0\) otherwise.

20.4 17.3 The DQN loss

Let \(D\) be a distribution over stored transitions. In practice, \(D\) is the empirical distribution induced by the replay buffer. The DQN objective is the expected squared temporal-difference error

\[ L(\theta) = \mathbb E_{(S,A,R,S',D_T)\sim D} \left[ \left( Y-Q_\theta(S,A) \right)^2 \right], \]

where

\[ Y = R+ \gamma(1-D_T)\max_{a'}Q_{\theta^-}(S',a'). \]

Here \(D_T\) is the terminal indicator. In a mini-batch \(B\), the empirical loss is

\[ \widehat L_B(\theta) = \frac{1}{|B|} \sum_{(s_i,a_i,r_i,s_i',d_i)\in B} \left( y_i-Q_\theta(s_i,a_i) \right)^2. \]

During one gradient step, \(y_i\) is treated as a constant because it is computed with the target network \(\theta^-\). Therefore

\[ \nabla_\theta \widehat L_B(\theta) = - \frac{2}{|B|} \sum_{i\in B} \left( y_i-Q_\theta(s_i,a_i) \right) \nabla_\theta Q_\theta(s_i,a_i). \]

Equivalently, using the TD error

\[ \delta_i = y_i-Q_\theta(s_i,a_i), \]

the update direction is proportional to

\[ \frac{1}{|B|}\sum_{i\in B}\delta_i\nabla_\theta Q_\theta(s_i,a_i). \]

This is the same algebraic structure as semi-gradient TD: the target is not differentiated through.

If one differentiates through the max target using the same network, the update is no longer the standard DQN semi-gradient update. The standard method treats the target as fixed for the current optimization step.

20.4.1 Interactive: Squared TD loss and Huber loss

Large TD errors can dominate a squared-error objective. Many DQN implementations use the Huber loss to reduce sensitivity to extreme errors.

20.5 17.4 Target networks

The target network is a slowly updated copy of the online network. In a hard update, one sets

\[ \theta^-\leftarrow \theta \]

every \(C\) training steps. Between such updates, \(\theta^-\) is fixed.

In a soft update, one uses

\[ \theta^- \leftarrow (1-\tau)\theta^-+\tau\theta, \]

where \(0<\tau\ll 1\). This is also called Polyak averaging.

The target network helps because it slows down the movement of the regression target. If the target changes too quickly, the optimization problem changes while the optimizer is trying to solve it. This creates a moving-target problem.

20.5.1 Interactive: Target-network lag

The online network moves every step. The target network either jumps occasionally or tracks smoothly by averaging.

20.6 17.5 Experience replay

DQN stores observed transitions in a replay buffer:

\[ B = \{(s_i,a_i,r_i,s_i',d_i)\}_{i=1}^N. \]

At each training step, a mini-batch is sampled from this buffer. Replay has several mathematical roles.

First, it improves data efficiency. A transition can be used more than once.

Second, it weakens temporal correlation. Sequential RL data are dependent:

\[ S_t,A_t,R_{t+1},S_{t+1},A_{t+1},\ldots \]

are not iid samples. Random mini-batches from a buffer are still not perfectly iid, but they are less correlated than consecutive transitions.

Third, replay changes the training distribution. The loss is not weighted by the current policy alone. It is weighted by the empirical distribution of the buffer, which is a mixture of past behavior policies.

Let \(\mu_B(s,a)\) denote the empirical state-action distribution in the buffer. Then DQN approximately minimizes a projected Bellman-error-like objective under the weighting \(\mu_B\):

\[ \mathbb E_{(S,A)\sim \mu_B} \left[ \left( R+\gamma\max_{a'}Q_{\theta^-}(S',a')-Q_\theta(S,A) \right)^2 \right]. \]

The buffer distribution matters. If important state-action pairs are rarely stored, the network may fit them poorly.

20.6.1 Interactive: Replay buffer age distribution

A replay buffer stores a moving window of transitions. Uniform sampling gives different probabilities to recent and old transitions depending on the buffer capacity.

20.7 17.6 Algorithm: DQN

Deep Q-network algorithm

Input: discount factor \(\gamma\), exploration schedule \(\epsilon_t\), replay capacity \(N\), mini-batch size \(m\), target update frequency \(C\).

Initialize online parameters \(\theta\).
Set target parameters \(\theta^-\leftarrow\theta\).
Initialize an empty replay buffer \(B\).
For each environment step:
1. Choose action \(A_t\) using an \(\epsilon_t\)-greedy policy from \(Q_\theta\).
2. Observe \(R_{t+1}\), \(S_{t+1}\), and terminal flag \(D_t\).
3. Store \((S_t,A_t,R_{t+1},S_{t+1},D_t)\) in \(B\).
4. Sample a mini-batch from \(B\).
5. Compute targets \[ y_i=r_i+\gamma(1-d_i)\max_{a'}Q_{\theta^-}(s_i',a'). \]
6. Take a gradient step on \[ \frac{1}{m}\sum_{i=1}^m\left(y_i-Q_\theta(s_i,a_i)\right)^2. \]
7. Every \(C\) steps, update \(\theta^-\leftarrow\theta\).

The policy used to collect data is usually \(\epsilon\)-greedy:

\[ A_t = \begin{cases} \text{a random action}, & \text{with probability } \epsilon_t,\\ \arg\max_a Q_\theta(S_t,a), & \text{with probability } 1-\epsilon_t. \end{cases} \]

Thus DQN is an off-policy algorithm: the target is greedy, while the behavior policy includes exploration.

20.8 17.7 Python example: replay buffer and DQN target

The following example implements a small replay buffer and computes DQN targets from a linear action-value model. The point is to make the data structure and target calculation transparent before using neural-network libraries.

import numpy as np

rng = np.random.default_rng(7243)

class ReplayBuffer:
    def __init__(self, capacity, state_dim):
        self.capacity = int(capacity)
        self.state_dim = int(state_dim)
        self.states = np.zeros((capacity, state_dim), dtype=float)
        self.actions = np.zeros(capacity, dtype=int)
        self.rewards = np.zeros(capacity, dtype=float)
        self.next_states = np.zeros((capacity, state_dim), dtype=float)
        self.done = np.zeros(capacity, dtype=float)
        self.position = 0
        self.size = 0

    def add(self, s, a, r, sp, done):
        j = self.position
        self.states[j] = s
        self.actions[j] = a
        self.rewards[j] = r
        self.next_states[j] = sp
        self.done[j] = float(done)
        self.position = (self.position + 1) % self.capacity
        self.size = min(self.size + 1, self.capacity)

    def sample(self, batch_size, rng):
        idx = rng.choice(self.size, size=batch_size, replace=False)
        return (
            self.states[idx],
            self.actions[idx],
            self.rewards[idx],
            self.next_states[idx],
            self.done[idx],
        )

state_dim = 3
n_actions = 2
buffer = ReplayBuffer(capacity=100, state_dim=state_dim)

# Add synthetic transitions.
for _ in range(50):
    s = rng.normal(size=state_dim)
    a = rng.integers(n_actions)
    sp = 0.8 * s + 0.2 * rng.normal(size=state_dim)
    r = 1.0 if (a == 1 and sp[0] > 0) else 0.0
    done = rng.random() < 0.05
    buffer.add(s, a, r, sp, done)

# A linear action-value approximator: Q(s, a) = s^T W[:, a] + b[a].
W_online = rng.normal(scale=0.1, size=(state_dim, n_actions))
b_online = np.zeros(n_actions)
W_target = W_online.copy()
b_target = b_online.copy()

def q_values(states, W, b):
    return states @ W + b

batch = buffer.sample(batch_size=8, rng=rng)
states, actions, rewards, next_states, done = batch

gamma = 0.95
q_next_target = q_values(next_states, W_target, b_target)
targets = rewards + gamma * (1.0 - done) * np.max(q_next_target, axis=1)
q_pred_all = q_values(states, W_online, b_online)
q_pred_selected = q_pred_all[np.arange(len(actions)), actions]
td_errors = targets - q_pred_selected

print("targets:", np.round(targets, 3))
print("predictions:", np.round(q_pred_selected, 3))
print("TD errors:", np.round(td_errors, 3))
print("mini-batch MSE:", float(np.mean(td_errors ** 2)))

targets: [0.09  0.026 1.118 0.053 0.065 0.047 0.076 0.054]
predictions: [-0.32  -0.033  0.152 -0.248  0.08   0.064  0.119 -0.312]
TD errors: [ 0.41   0.059  0.967  0.301 -0.015 -0.017 -0.042  0.365]
mini-batch MSE: 0.16655280124549315

The code separates four objects that are often mixed together in informal descriptions:

the sampled transitions;
the online prediction \(Q_\theta(s,a)\);
the target-network prediction \(Q_{\theta^-}(s',a')\);
the scalar target \(y\).

20.9 17.8 Python example: one gradient step for a linear DQN

For a linear action-value function

\[ Q_\theta(s,a)=s^T w_a+b_a, \]

the squared TD loss for one sample is

\[ \ell_i(\theta) = \left(y_i-s_i^T w_{a_i}-b_{a_i}\right)^2. \]

The semi-gradient update is

\[ w_{a_i} \leftarrow w_{a_i}+\alpha \delta_i s_i, \]

\[ b_{a_i} \leftarrow b_{a_i}+\alpha \delta_i, \]

up to a constant factor depending on whether the loss includes \(1/2\).

alpha = 0.05
W = W_online.copy()
b = b_online.copy()

q_before = q_values(states, W, b)[np.arange(len(actions)), actions]
loss_before = np.mean((targets - q_before) ** 2)

# Use the gradient of 1/2 times the squared TD error.
for s, a, y in zip(states, actions, targets):
    pred = s @ W[:, a] + b[a]
    delta = y - pred
    W[:, a] += alpha * delta * s
    b[a] += alpha * delta

q_after = q_values(states, W, b)[np.arange(len(actions)), actions]
loss_after = np.mean((targets - q_after) ** 2)

print("loss before:", round(float(loss_before), 5))
print("loss after: ", round(float(loss_after), 5))

loss before: 0.16655
loss after:  0.07794

This small example is not yet deep learning, but it contains the mathematical structure of a DQN update. A neural-network optimizer replaces the explicit linear update by backpropagation.

20.10 17.9 Neural networks as action-value approximators

A neural DQN can be written abstractly as

\[ Q_\theta(s,a) = \left[f_\theta(s)\right]_a, \]

where \(f_\theta(s)\in\mathbb R^{|A|}\) is a vector-valued neural network.

For a two-layer network,

\[ h = \sigma(W_1s+b_1), \]

\[ f_\theta(s) = W_2h+b_2. \]

The DQN loss for a mini-batch is

\[ \widehat L_B(\theta) = \frac{1}{m} \sum_{i=1}^m \left(y_i-[f_\theta(s_i)]_{a_i}\right)^2. \]

Backpropagation computes

\[ \nabla_\theta [f_\theta(s_i)]_{a_i}, \]

and the target \(y_i\) is treated as fixed during the online-network update.

For image inputs, the feature map may be a convolutional network. For vector states, it may be a multilayer perceptron. For structured or text states, it may involve embeddings or a transformer encoder. The Bellman target remains the same; the representation changes.

In DQN, the neural network is not the source of the objective. The objective comes from the Bellman optimality equation. The neural network defines the approximation class used to represent action values.

20.11 17.10 The deadly triad

Deep Q-learning combines three ingredients:

function approximation: \(Q_\theta\) is not tabular;
bootstrapping: targets depend on current or target value estimates;
off-policy learning: the behavior policy differs from the greedy target policy.

Together, these form the deadly triad. The term refers to the fact that the combination can produce instability or divergence.

Mathematically, the tabular Bellman optimality operator is a contraction in the sup norm. Once we restrict to a nonlinear function class and train by stochastic gradients under a replay-buffer distribution, we are no longer simply applying a contraction operator to a value table. Instead, the algorithm alternates between approximate target construction and approximate regression:

\[ Q_{\theta_k} \quad \longrightarrow \quad \text{targets from } Q_{\theta_k^-} \quad \longrightarrow \quad \theta_{k+1} \text{ by regression}. \]

The regression step is influenced by optimization error, approximation error, sampling error, and distribution shift.

A small training loss does not guarantee an optimal policy. The loss is measured on the replay-buffer distribution and against bootstrapped targets, not directly against \(Q^*\).

20.12 17.11 Overestimation bias and Double DQN

Q-learning uses a maximum over estimated action values:

\[ \max_a \widehat Q(s,a). \]

If the estimates contain noise, then the maximum tends to be biased upward. For example, if

\[ \widehat Q(s,a)=Q(s,a)+\varepsilon_a, \]

with zero-mean errors \(\varepsilon_a\), then often

\[ \mathbb E\left[\max_a \widehat Q(s,a)\right] \ge \max_a Q(s,a). \]

The inequality is a consequence of the convexity of the maximum function.

DQN uses

\[ y^{\text{DQN}} = r+ \gamma \max_{a'}Q_{\theta^-}(s',a'). \]

Double DQN separates action selection and action evaluation. The online network selects the action:

\[ a^* \in \arg\max_{a'}Q_\theta(s',a'), \]

and the target network evaluates it:

\[ y^{\text{Double}} = r+ \gamma Q_{\theta^-}(s',a^*). \]

Thus Double DQN uses

\[ y^{\text{Double}} = r+ \gamma Q_{\theta^-} \left(s',\arg\max_{a'}Q_\theta(s',a')\right). \]

20.12.1 Interactive: Overestimation and Double DQN

Noisy maximization produces an upward bias. Separating selection and evaluation can reduce this effect.

20.13 17.12 Exploration in DQN

The most common simple exploration mechanism is \(\epsilon\)-greedy exploration. With \(m=|A|\) actions,

\[ \pi_\epsilon(a\mid s) = \begin{cases} 1-\epsilon+\epsilon/m, & a\in\arg\max_b Q_\theta(s,b),\\ \epsilon/m, & \text{otherwise}. \end{cases} \]

In many implementations, \(\epsilon\) decreases from a large initial value to a smaller final value:

\[ \epsilon_t = \epsilon_{\min} + (\epsilon_0-\epsilon_{\min})\exp(-t/\kappa). \]

This is useful computationally, but it also raises statistical questions. If exploration decays too quickly, some actions may never be sampled enough. If exploration remains too large, the learned behavior may be inefficient.

DQN can also use noisy networks, entropy-based exploration, bootstrapped ensembles, count-based bonuses, or uncertainty-based exploration. Those methods are beyond the scope of this introductory chapter, but they all address the same mathematical problem: learning good action values requires enough coverage of state-action space.

20.14 17.13 Training diagnostics

DQN training curves can be difficult to interpret. Common diagnostics include:

episode return;
average loss;
average absolute TD error;
max action value;
replay-buffer coverage;
policy entropy or fraction of random actions;
target-network update times;
gradient norm.

A decreasing TD loss is not sufficient evidence of success. The learned policy may still be poor if the replay buffer is narrow, the targets are biased, or the network generalizes incorrectly.

20.14.1 Interactive: DQN diagnostics

A stable-looking loss does not always imply improving return. Multiple diagnostics should be read together.

20.15 17.14 Python example: a tiny DQN-style training loop

The following toy example trains a linear action-value model from replayed transitions generated by a simple environment. It is not intended to be a high-performance RL implementation. It is designed to show the mathematical ingredients of DQN in a small setting.

The state is a two-dimensional vector. The agent chooses one of two actions. Action \(1\) tends to increase the first coordinate and earns reward when the first coordinate is positive. Action \(0\) tends to decrease it.

import numpy as np

rng = np.random.default_rng(2026)

state_dim = 2
n_actions = 2
capacity = 500
batch_size = 32
gamma = 0.95
alpha = 0.03
epsilon = 0.20
target_update = 25
n_steps = 300

buffer = ReplayBuffer(capacity=capacity, state_dim=state_dim)

W = rng.normal(scale=0.1, size=(state_dim, n_actions))
b = np.zeros(n_actions)
W_targ = W.copy()
b_targ = b.copy()

s = rng.normal(scale=0.3, size=state_dim)
returns = []
losses = []
episode_return = 0.0

def env_step(s, a, rng):
    drift = np.array([-0.12, 0.00]) if a == 0 else np.array([0.12, 0.00])
    sp = 0.90 * s + drift + rng.normal(scale=0.08, size=state_dim)
    r = 1.0 if sp[0] > 0.45 else 0.0
    done = bool(abs(sp[0]) > 1.2 or rng.random() < 0.02)
    return sp, r, done

for t in range(n_steps):
    if rng.random() < epsilon:
        a = int(rng.integers(n_actions))
    else:
        a = int(np.argmax(q_values(s.reshape(1, -1), W, b)[0]))

    sp, r, done = env_step(s, a, rng)
    buffer.add(s, a, r, sp, done)
    episode_return += r

    if buffer.size >= batch_size:
        states, actions, rewards, next_states, done_flags = buffer.sample(batch_size, rng)
        q_next = q_values(next_states, W_targ, b_targ)
        y = rewards + gamma * (1.0 - done_flags) * np.max(q_next, axis=1)

        q_pred = q_values(states, W, b)[np.arange(batch_size), actions]
        delta = y - q_pred
        losses.append(float(np.mean(delta ** 2)))

        for si, ai, di in zip(states, actions, delta):
            W[:, ai] += alpha * di * si
            b[ai] += alpha * di

    if (t + 1) % target_update == 0:
        W_targ = W.copy()
        b_targ = b.copy()

    if done:
        returns.append(episode_return)
        episode_return = 0.0
        s = rng.normal(scale=0.3, size=state_dim)
    else:
        s = sp

print("number of completed episodes:", len(returns))
print("mean return over last 10 episodes:", round(float(np.mean(returns[-10:])), 3) if returns else None)
print("mean recent loss:", round(float(np.mean(losses[-20:])), 5) if losses else None)
print("learned weights:")
print(np.round(W, 3))
print("learned biases:", np.round(b, 3))

number of completed episodes: 10
mean return over last 10 episodes: 18.9
mean recent loss: 1.16994
learned weights:
[[ 2.785  2.798]
 [-0.025 -0.077]]
learned biases: [3.704 4.233]

This example contains the key DQN pattern:

collect a transition using an exploratory behavior policy;
store it in replay;
sample a mini-batch from replay;
compute targets using a target network;
update only the online network;
periodically copy the online network to the target network.

20.16 17.15 DQN as approximate value iteration

Value iteration applies the optimal Bellman operator:

\[ Q_{k+1} = TQ_k. \]

DQN can be informally viewed as approximate value iteration:

\[ Q_{\theta_{k+1}} \approx \Pi_{\mu_B}TQ_{\theta_k^-}, \]

where \(\Pi_{\mu_B}\) denotes projection or regression under the replay-buffer distribution. This notation should be read cautiously because the function class is nonlinear and the optimization may not find a global projection. Still, it captures the structure:

\[ \text{Bellman target construction} \quad + \quad \text{supervised regression to targets}. \]

The approximation errors can be grouped as:

sampling error from finite mini-batches;
optimization error from incomplete gradient descent;
approximation error from the neural function class;
distribution error from using replay-buffer weighting;
target error from bootstrapping with an imperfect target network.

This decomposition is useful when debugging a DQN system.

20.17 17.16 Practical implementation checklist

Checklist for a DQN implementation

Confirm that the action space is discrete.
Normalize or scale state features when appropriate.
Use a replay buffer large enough to diversify mini-batches.
Do not train before the buffer contains enough transitions.
Use a separate target network.
Handle terminal transitions correctly by removing the future term.
Track return, TD error, loss, gradient norm, and action frequencies.
Compare with a random policy and a simple heuristic baseline.
Test the target calculation on a tiny hand-checkable batch.
Set random seeds when running diagnostic experiments.

20.18 17.17 AI-assisted learning components

AI prompt: equation audit

Ask an AI assistant to check whether the following target is correct:

\[ y_i=r_i+\gamma\max_{a'}Q_\theta(s_i',a'). \]

Then ask it to explain why the standard DQN target usually uses \(Q_{\theta^-}\) instead of \(Q_\theta\), and why terminal transitions require a factor \(1-d_i\).

AI prompt: code review

Provide your replay-buffer sampling code and ask the AI assistant to check for the following mistakes: sampling uninitialized entries, using replacement unintentionally, mixing terminal and nonterminal targets incorrectly, and updating the target network before computing the target.

AI prompt: experiment critique

Give the assistant a plot of return, TD loss, average max-\(Q\), and epsilon. Ask it to list at least three possible explanations for a decreasing loss but flat return. Then ask which extra diagnostic it would collect next.

20.19 17.18 Summary

DQN is the first major deep RL method in this book. Its mathematical core is not mysterious: it is Q-learning with a neural action-value approximation. The central target is

\[ y = r+\gamma(1-d)\max_{a'}Q_{\theta^-}(s',a'), \]

and the online network minimizes a regression loss toward this target. Experience replay and target networks are not cosmetic tricks; they are attempts to control statistical dependence and target instability. Double DQN addresses maximization bias by separating action selection and action evaluation.

For applied mathematics students, DQN is approximate dynamic programming with nonlinear approximation and moving targets. For statistics students, DQN is sequential supervised learning with dependent data, biased targets, distribution shift, and exploration-driven sampling.

20.20 Conceptual exercises

Explain why a DQN needs one output per action when the action space is finite and discrete.
Why does DQN use a target network? Describe the moving-target problem in your own words.
Explain how experience replay changes the dependence structure of the training data.
Why is DQN considered off-policy?
Give an example where a small TD loss might not imply a good policy.
Explain the deadly triad and identify which three parts appear in DQN.
Compare DQN with tabular Q-learning. What is gained and what is lost?
Why does a terminal transition require removing the bootstrap term?

20.21 Mathematical exercises

Derive the gradient of \[ \frac{1}{2}\left(y-Q_\theta(s,a)\right)^2 \] with respect to \(\theta\), treating \(y\) as constant.
Suppose \(Q_\theta(s,a)=s^Tw_a+b_a\). Derive the update for \(w_a\) and \(b_a\) under the half-squared TD loss.
Prove that if \(\widehat Q_a=Q_a+\varepsilon_a\) with zero-mean noise, then \[ \mathbb E\max_a \widehat Q_a \ge \max_a Q_a. \] Hint: use Jensen’s inequality and the convexity of the maximum function.
Let the soft target update be \[ \theta^-_{t+1}=(1-\tau)\theta^-_t+\tau\theta_t. \] If \(\theta_t=\theta\) is constant for all \(t\), solve for \(\theta^-_t\).
In a mini-batch with terminal indicators \(d_i\), show how the DQN targets can be written as a vectorized NumPy expression.
Explain why the expression \[ Q_{\theta^-}\left(s',\arg\max_{a'}Q_\theta(s',a')\right) \] separates action selection from action evaluation.

20.22 Computational exercises

Modify the toy DQN example by changing the replay capacity. How do the recent losses and returns change?
Replace the hard target update with a soft target update. Compare training behavior for several values of \(\tau\).
Implement the Huber loss and compare it with the squared loss when occasional large rewards are added.
Implement Double DQN targets in the toy linear example.
Track action frequencies over training. Does the learned policy collapse to one action too early?
Build a small finite MDP and compare tabular Q-learning with a linear DQN using one-hot state features.
Create a diagnostic plot with three curves: return, TD loss, and average max-\(Q\).

20.23 AI-assisted exercises

Ask an AI assistant to generate a minimal DQN implementation. Then manually check whether it uses a target network and handles terminal states correctly.
Ask an AI assistant to explain the difference between DQN and Double DQN. Identify any vague or incorrect statements.
Provide your own training diagnostics to an AI assistant and ask for possible failure modes. Separate evidence-based comments from speculation.
Ask an AI assistant to convert a tabular Q-learning implementation to DQN. Check whether the resulting code still assumes a table anywhere.
Ask an AI assistant to design a unit test for the target calculation. Implement and run the test.

20.24 Notes for instructors

This chapter is a bridge from tabular RL to deep RL. It should not be taught as a collection of software tricks. The clean mathematical sequence is:

\[ \text{Bellman optimality equation} \to \text{Q-learning target} \to \text{function approximation} \to \text{semi-gradient regression} \to \text{stabilization by replay and target networks}. \]

For MA Applied Math students, emphasize approximate fixed points, nonlinear approximation, and the loss of contraction guarantees. For MS Statistics students, emphasize dependent data, replay-buffer distributions, target bias, and diagnostic uncertainty.