16  Function Approximation

Core idea. Tabular reinforcement learning stores one number for each state or each state-action pair. This is impossible when the state space is large, continuous, or combinatorial. Function approximation replaces a table by a parameterized function such as

\[ V_\theta(s) \approx V^\pi(s), \qquad Q_\theta(s,a) \approx Q^\pi(s,a), \]

where \(\theta \in \mathbb R^d\) is learned from samples. Mathematically, this chapter studies value functions as elements of a function space, approximation as projection, and reinforcement learning as noisy fixed-point computation in a lower-dimensional model class.

16.1 Learning goals

After reading this chapter, students should be able to:

  1. explain why tabular value functions fail for large state spaces;
  2. define linear value approximation \(V_\theta(s)=\phi(s)^T\theta\);
  3. interpret feature maps as coordinates for an approximation subspace;
  4. formulate weighted least-squares projection in a state distribution;
  5. derive the projected Bellman equation;
  6. distinguish the mean-squared value error, Bellman error, and projected Bellman error;
  7. implement linear least-squares value approximation in Python;
  8. derive semi-gradient TD updates for linear value functions;
  9. describe instability issues in off-policy and nonlinear approximation;
  10. use AI tools to audit feature choices, loss functions, and approximation errors without confusing supervised regression with reinforcement learning.

16.2 13.1 Why approximation is necessary

For a finite MDP with \(n\) states, a tabular value function is a vector in \(\mathbb R^n\):

\[ V = (V(s_1),\ldots,V(s_n))^T. \]

Tabular algorithms update one coordinate at a time. This is appropriate when \(n\) is small. In many applications, however, states are high-dimensional objects: images, inventories, portfolios, patient histories, recommendation contexts, or robot configurations. It is not possible to visit every state many times, and it is not possible to store an independent parameter for every state.

Function approximation introduces generalization. A value learned at one state influences value estimates at similar states. The key mathematical tradeoff is:

\[ \text{generalization reduces variance but introduces approximation bias.} \]

Suppose \(\mathcal S\) is large and \(V^\pi\) is the true value function. Instead of searching over all functions on \(\mathcal S\), we choose a parametric class

\[ \mathcal F = \{V_\theta: \theta \in \Theta\}. \]

The best possible approximation inside this class is usually not equal to \(V^\pi\). Thus the error has at least two components:

\[ V_\theta - V^\pi = \underbrace{V_\theta - V_{\theta^*}}_{\text{estimation or optimization error}} + \underbrace{V_{\theta^*}-V^\pi}_{\text{approximation error}}. \]

In tabular RL, the second term can be zero. In approximate RL, it is usually nonzero.

Function approximation changes the mathematical problem from solving for a vector \(V \in \mathbb R^n\) to finding the best element of a restricted function class \(\mathcal F\).

16.3 13.2 Feature maps and linear value functions

The simplest and most important approximation class is the linear class. Choose a feature map

\[ \phi:\mathcal S \to \mathbb R^d, \]

where

\[ \phi(s)= \begin{bmatrix} \phi_1(s) \\ \vdots \\ \phi_d(s) \end{bmatrix}. \]

A linear value approximation has the form

\[ V_\theta(s)=\phi(s)^T\theta = \sum_{j=1}^d \theta_j\phi_j(s). \]

Here the word linear means linear in the parameters \(\theta\), not necessarily linear in the state. For example, if \(s\in\mathbb R\), the feature vector

\[ \phi(s)= \begin{bmatrix} 1 \\ s \\ s^2 \\ \sin(\pi s) \end{bmatrix} \]

defines a linear model in a nonlinear basis.

For a finite state space \(\mathcal S=\{s_1,\ldots,s_n\}\), define the feature matrix

\[ \Phi= \begin{bmatrix} \phi(s_1)^T \\ \phi(s_2)^T \\ \vdots \\ \phi(s_n)^T \end{bmatrix} \in \mathbb R^{n\times d}. \]

Then the vector of approximate values is

\[ V_\theta = \Phi\theta. \]

Thus linear value approximation restricts value functions to the column space

\[ \mathcal L = \operatorname{col}(\Phi) \subseteq \mathbb R^n. \]

16.3.1 Interactive: Feature approximation of a value function

The exact value function may be nonlinear over the state index. A feature map chooses a lower-dimensional family of functions. Increasing the number of basis functions improves approximation but can increase statistical variance.

16.4 13.3 Weighted norms and state distributions

Approximation quality should be measured with respect to the states that matter. Under a fixed policy \(\pi\), the Markov chain visits some states often and others rarely. Let \(d(s)\) be a nonnegative state weighting satisfying

\[ \sum_s d(s)=1. \]

For finite state spaces, define the diagonal matrix

\[ D=\operatorname{diag}(d(s_1),\ldots,d(s_n)). \]

The \(D\)-weighted inner product is

\[ \langle u,v\rangle_D = u^T D v =\sum_s d(s)u(s)v(s), \]

and the associated norm is

\[ \lVert u\rVert_D^2 = u^T D u =\sum_s d(s)u(s)^2. \]

The best linear approximation to a target value vector \(V\) is the solution of

\[ \min_\theta \lVert V-\Phi\theta\rVert_D^2. \]

Assuming \(\Phi^T D\Phi\) is invertible, the normal equations are

\[ \Phi^T D\Phi\theta = \Phi^T D V, \]

so

\[ \theta^*=(\Phi^T D\Phi)^{-1}\Phi^T D V. \]

The fitted value vector is

\[ \Pi_D V = \Phi(\Phi^T D\Phi)^{-1}\Phi^T D V. \]

The matrix

\[ \Pi_D=\Phi(\Phi^T D\Phi)^{-1}\Phi^T D \]

is the \(D\)-orthogonal projection onto \(\operatorname{col}(\Phi)\).

Projection is the bridge between linear algebra and approximate dynamic programming. Because Bellman backups usually leave the feature space, we project them back into the feature space.

16.4.1 Interactive: Projection onto an approximation subspace

This picture shows the geometric idea behind value approximation. The true value function is a vector. The approximation class is a subspace. The fitted value is the weighted orthogonal projection of the true value onto that subspace.

16.5 13.4 Python example: least-squares value approximation

The following example creates a smooth value function over a one-dimensional state space and approximates it with polynomial features. This is ordinary weighted least squares, but the interpretation is RL-specific: the target is a value function and the weights represent a state-visitation distribution.

import numpy as np
import matplotlib.pyplot as plt

states = np.linspace(0, 1, 80)
true_V = np.sin(2 * np.pi * states) + 0.5 * states

# A state weighting: later states are visited more often.
d = 0.5 + states
d = d / d.sum()
D = np.diag(d)

def polynomial_features(x, degree):
    return np.column_stack([x**k for k in range(degree + 1)])

for degree in [1, 3, 7]:
    Phi = polynomial_features(states, degree)
    theta = np.linalg.solve(Phi.T @ D @ Phi, Phi.T @ D @ true_V)
    fitted = Phi @ theta
    weighted_mse = np.sum(d * (true_V - fitted)**2)
    print(f"degree={degree:2d}, weighted MSE={weighted_mse:.5f}")

Phi = polynomial_features(states, 5)
theta = np.linalg.solve(Phi.T @ D @ Phi, Phi.T @ D @ true_V)
fitted = Phi @ theta

plt.figure(figsize=(7, 4))
plt.plot(states, true_V, label="true value")
plt.plot(states, fitted, label="degree 5 approximation")
plt.scatter(states, fitted, s=8, alpha=0.4)
plt.xlabel("state")
plt.ylabel("value")
plt.title("Linear value approximation with polynomial features")
plt.legend()
plt.show()
degree= 1, weighted MSE=0.20496
degree= 3, weighted MSE=0.00485
degree= 7, weighted MSE=0.00000

Polynomial feature approximation of a value function.

This example is supervised regression only because the true \(V\) is supplied. In reinforcement learning, \(V^\pi\) is usually unknown. We observe sampled rewards and transitions, and the target itself depends on the current approximation.

16.6 13.5 Bellman backups leave the feature space

For a fixed policy \(\pi\), the Bellman expectation operator is

\[ T^\pi V = r_\pi + \gamma P_\pi V. \]

If \(V\) belongs to the feature space \(\mathcal L=\operatorname{col}(\Phi)\), then \(T^\pi V\) need not belong to \(\mathcal L\). That is,

\[ V \in \mathcal L \quad \not\Rightarrow \quad T^\pi V \in \mathcal L. \]

This is the fundamental reason projection enters approximate dynamic programming. We cannot usually solve

\[ V = T^\pi V \]

inside the feature space. Instead, we solve the projected Bellman equation

\[ \Phi\theta = \Pi_D T^\pi(\Phi\theta). \]

Equivalently,

\[ \Phi\theta = \Pi_D(r_\pi + \gamma P_\pi\Phi\theta). \]

Multiplying by \(\Phi^T D\) gives

\[ \Phi^T D\Phi\theta = \Phi^T D(r_\pi + \gamma P_\pi\Phi\theta). \]

Rearranging,

\[ \Phi^T D(I-\gamma P_\pi)\Phi\theta =\Phi^T D r_\pi. \]

Define

\[ A=\Phi^T D(I-\gamma P_\pi)\Phi, \qquad b=\Phi^T D r_\pi. \]

Then the projected Bellman equation becomes the linear system

\[ A\theta=b. \]

When \(A\) is nonsingular, the solution is

\[ \theta_{\mathrm{PBE}}=A^{-1}b. \]

16.6.1 Interactive: Bellman backup followed by projection

The Bellman operator moves the current approximate value function toward dynamic consistency. Projection moves it back into the chosen feature space. The projected Bellman fixed point is where these two operations balance.

16.7 13.6 Python example: projected Bellman equation

The following code compares the exact tabular value function with the projected Bellman solution using only two features.

import numpy as np

P = np.array([
    [0.70, 0.25, 0.05, 0.00],
    [0.10, 0.60, 0.25, 0.05],
    [0.05, 0.15, 0.60, 0.20],
    [0.00, 0.10, 0.20, 0.70]
])
r = np.array([0.0, 1.0, 2.0, 4.0])
gamma = 0.90

# Exact tabular solution.
V_exact = np.linalg.solve(np.eye(4) - gamma * P, r)

# Two-dimensional feature space: constant and normalized state index.
x = np.linspace(0, 1, 4)
Phi = np.column_stack([np.ones(4), x])

# Use a state weighting. Here we use a simple uniform distribution.
d = np.ones(4) / 4
D = np.diag(d)

A = Phi.T @ D @ (np.eye(4) - gamma * P) @ Phi
b = Phi.T @ D @ r
theta_pbe = np.linalg.solve(A, b)
V_pbe = Phi @ theta_pbe

# Best least-squares projection of the exact value, for comparison.
theta_proj = np.linalg.solve(Phi.T @ D @ Phi, Phi.T @ D @ V_exact)
V_proj = Phi @ theta_proj

print("Exact V:", np.round(V_exact, 3))
print("Projected Bellman V:", np.round(V_pbe, 3))
print("Projection of exact V:", np.round(V_proj, 3))
print("PBE residual norm:", np.linalg.norm(V_pbe - (r + gamma * P @ V_pbe)))
print("Projected residual norm:", np.linalg.norm(Phi.T @ D @ (V_pbe - (r + gamma * P @ V_pbe))))
Exact V: [12.856 17.072 20.344 24.86 ]
Projected Bellman V: [12.914 16.859 20.804 24.75 ]
Projection of exact V: [12.89  16.819 20.747 24.676]
PBE residual norm: 0.3472741502808864
Projected residual norm: 7.382996156779424e-16

Notice that the projected Bellman solution and the projection of the exact value are not generally the same. The projected Bellman solution solves a dynamic consistency equation after projection; the least-squares projection solves a pure approximation problem.

16.8 13.7 Three different errors

Approximate value prediction involves several losses. They are related but not identical.

The mean-squared value error is

\[ \operatorname{MSVE}(\theta) = \lVert V^\pi-\Phi\theta\rVert_D^2. \]

This measures distance to the true value function, but \(V^\pi\) is usually unknown.

The mean-squared Bellman error is

\[ \operatorname{MSBE}(\theta) = \lVert T^\pi(\Phi\theta)-\Phi\theta\rVert_D^2. \]

This measures Bellman inconsistency. It is natural but difficult to estimate from single samples because the squared conditional expectation is not the expectation of a squared single-sample target.

The mean-squared projected Bellman error is

\[ \operatorname{MSPBE}(\theta) = \lVert \Pi_D T^\pi(\Phi\theta)-\Phi\theta\rVert_D^2. \]

The projected Bellman equation is equivalent to minimizing the MSPBE to zero when an exact projected fixed point exists.

Do not confuse these objectives. Least-squares regression targets MSVE. Bellman residual methods target MSBE. TD methods with linear approximation are connected to the projected Bellman equation and MSPBE.

16.9 13.8 Semi-gradient TD with linear approximation

For a differentiable value approximation \(V_\theta(s)\), the TD error is

\[ \delta_t =R_{t+1}+\gamma V_\theta(S_{t+1})-V_\theta(S_t). \]

The semi-gradient TD update is

\[ \theta_{t+1} = \theta_t+\\alpha_t\delta_t\nabla_\theta V_\theta(S_t). \]

It is called a semi-gradient method because the target

\[ R_{t+1}+\gamma V_\theta(S_{t+1}) \]

also depends on \(\theta\), but the update treats that target as fixed when differentiating. For linear approximation,

\[ V_\theta(s)=\phi(s)^T\theta, \]

so

\[ \nabla_\theta V_\theta(s)=\phi(s). \]

Thus linear TD(0) becomes

\[ \theta_{t+1} = \theta_t+ \alpha_t \left[R_{t+1}+\gamma\phi(S_{t+1})^T\theta_t- \phi(S_t)^T\theta_t\right]\phi(S_t). \]

Taking conditional expectations under a fixed policy gives the mean update

\[ \mathbb E[\Delta\theta \mid \theta] =\alpha(b-A\theta), \]

where

\[ A=\Phi^T D(I-\gamma P_\pi)\Phi, \qquad b=\Phi^T D r_\pi. \]

The expected TD fixed point therefore satisfies

\[ A\theta=b, \]

which is the projected Bellman equation.

16.9.1 Interactive: Semi-gradient TD as noisy linear-system solving

Linear TD can be viewed as noisy iteration toward the solution of \(A\theta=b\). The learning rate controls the balance between speed and noise.

16.10 13.9 Python example: linear TD prediction

The following example simulates TD(0) with linear features for a small Markov reward process.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(7243)

P = np.array([
    [0.70, 0.25, 0.05, 0.00],
    [0.10, 0.60, 0.25, 0.05],
    [0.05, 0.15, 0.60, 0.20],
    [0.00, 0.10, 0.20, 0.70]
])
r = np.array([0.0, 1.0, 2.0, 4.0])
gamma = 0.90
x = np.linspace(0, 1, 4)
Phi = np.column_stack([np.ones(4), x])

# Reference projected Bellman solution under uniform weighting.
D = np.eye(4) / 4
A = Phi.T @ D @ (np.eye(4) - gamma * P) @ Phi
b = Phi.T @ D @ r
theta_star = np.linalg.solve(A, b)

alpha = 0.05
theta = np.zeros(2)
s = 0
errors = []

for t in range(6000):
    s_next = rng.choice(4, p=P[s])
    reward = r[s]
    td_error = reward + gamma * Phi[s_next] @ theta - Phi[s] @ theta
    theta = theta + alpha * td_error * Phi[s]
    errors.append(np.linalg.norm(theta - theta_star))
    s = s_next

print("TD theta:", np.round(theta, 3))
print("PBE theta:", np.round(theta_star, 3))

plt.figure(figsize=(7, 4))
plt.plot(errors)
plt.xlabel("TD update")
plt.ylabel("distance to projected Bellman solution")
plt.title("Linear TD(0) approaches the projected Bellman solution")
plt.show()
TD theta: [14.054 11.703]
PBE theta: [12.914 11.836]

The trajectory is noisy because each update uses one sampled transition. The expected update points toward the projected Bellman fixed point.

16.11 13.10 Approximate action values

For control, one often approximates action values:

\[ Q_\theta(s,a)=\phi(s,a)^T\theta. \]

The feature map now depends on both state and action:

\[ \phi:\mathcal S\times\mathcal A\to\mathbb R^d. \]

A semi-gradient SARSA update has the form

\[ \theta_{t+1} = \theta_t+ \alpha_t \left[R_{t+1}+\gamma Q_{\theta_t}(S_{t+1},A_{t+1})-Q_{\theta_t}(S_t,A_t)\right] \nabla_\theta Q_{\theta_t}(S_t,A_t). \]

For a linear action-value function,

\[ \nabla_\theta Q_\theta(s,a)=\phi(s,a), \]

so

\[ \theta_{t+1} = \theta_t+ \alpha_t \left[R_{t+1}+\gamma \phi(S_{t+1},A_{t+1})^T\theta_t-\phi(S_t,A_t)^T\theta_t\right] \phi(S_t,A_t). \]

Approximate Q-learning replaces \(A_{t+1}\) by a greedy maximization:

\[ \theta_{t+1} = \theta_t+ \alpha_t \left[R_{t+1}+\gamma \max_{a'}\phi(S_{t+1},a')^T\theta_t- \phi(S_t,A_t)^T\theta_t\right] \phi(S_t,A_t). \]

This update is widely used, but it is also a warning sign: the combination of bootstrapping, off-policy sampling, and function approximation can be unstable.

16.12 13.11 The deadly triad

Approximate RL is not merely tabular RL with fewer parameters. Stability can change dramatically. A classical warning is the deadly triad:

  1. function approximation, which generalizes across states;
  2. bootstrapping, which uses current value estimates inside the target;
  3. off-policy learning, where the data distribution differs from the target policy.

When all three are present, divergence can occur even in small linear examples. This is one reason modern deep RL uses stabilization devices such as replay buffers, target networks, trust-region updates, and conservative objectives.

For fixed-policy on-policy prediction with linear features, TD has strong convergence theory under suitable conditions. For nonlinear, off-policy, or control settings, the analysis is more delicate.

16.13 13.12 Bias, variance, and feature complexity

Feature choice controls an approximation tradeoff. With too few features, the model cannot represent the true value function well. With too many features, the estimate can become noisy or ill-conditioned.

Let \(\mathcal F_d\) be a sequence of approximation spaces of increasing dimension. The expected prediction error often follows the qualitative decomposition

\[ \mathbb E\lVert V^\pi-\widehat V_d\rVert_D^2 \approx \underbrace{\text{bias}(d)^2}_{\text{decreases with complexity}} + \underbrace{\text{variance}(d)}_{\text{increases with complexity}} + \underbrace{\text{irreducible noise}}_{\text{sampling uncertainty}}. \]

This is familiar from statistical learning, but RL adds complications: the data are dependent, the target is bootstrapped, and the state distribution depends on the policy.

16.13.1 Interactive: Feature complexity and prediction error

As the number of features increases, approximation error decreases at first. But with limited noisy data, high-complexity models can become unstable or overfit the sampled returns.

16.14 13.13 Neural value functions

A neural value function replaces linear features by learned nonlinear features. A neural network value function can be written abstractly as

\[ V_\theta(s)=f_\theta(s), \]

where \(\theta\) includes all weights and biases. A neural action-value function has the form

\[ Q_\theta(s,a)=f_\theta(s,a) \]

or, for discrete actions, a network may output a vector

\[ (Q_\theta(s,a_1),\ldots,Q_\theta(s,a_m)). \]

The TD loss used in many deep RL algorithms has the form

\[ L(\theta)= \frac{1}{2} \left(y_t-Q_\theta(S_t,A_t)\right)^2, \]

where the target is often

\[ y_t=R_{t+1}+\gamma\max_{a'}Q_{\theta^-}(S_{t+1},a'). \]

The parameter vector \(\theta^-\) is a delayed target-network parameter. This separates the rapidly changing prediction network from the target used in the update.

The mathematical picture remains the same:

\[ \text{choose a function class, define a Bellman-style target, and optimize a sample-based objective.} \]

The analysis becomes harder because the objective is nonconvex and the target distribution changes during learning.

16.15 13.14 AI-assisted component: choosing features responsibly

Large language models can help generate feature ideas, but they can also produce features that leak future information, violate the Markov property, or encode irrelevant proxies. For mathematical RL work, an AI assistant should be used as a critic rather than as an authority.

AI prompt: feature audit

Consider an RL problem with state variables, actions, rewards, and a proposed feature map \(\phi(s)\) or \(\phi(s,a)\). Ask an AI assistant:

  1. Which features are functions only of information available at decision time?
  2. Which features might leak future rewards or next states?
  3. Which features help preserve the Markov property?
  4. Which features create near-collinearity or poor conditioning?
  5. Which omitted variables could cause systematic value approximation bias?

Then translate the answer into a mathematical checklist involving \(\Phi^TD\Phi\), state coverage, policy dependence, and the target value function.

AI prompt: debugging approximate TD code

Paste a TD update and ask the assistant to identify:

  1. the sampled state \(S_t\) and next state \(S_{t+1}\);
  2. the TD target;
  3. the TD error;
  4. whether the update is a full gradient or semi-gradient;
  5. whether the method is on-policy or off-policy;
  6. whether terminal states are handled correctly.

Do not accept the answer until every item is matched to a line of code.

16.16 13.15 Summary

Function approximation is the mathematical step that allows reinforcement learning to scale beyond tables. The main ideas are:

  • a value function is approximated by a parameterized family \(V_\theta\);
  • linear approximation uses \(V_\theta=\Phi\theta\);
  • weighted least squares projects values using the state distribution \(D\);
  • Bellman backups generally leave the feature space;
  • projected Bellman equations combine dynamic consistency with approximation;
  • semi-gradient TD learns the projected Bellman solution from samples in the linear on-policy prediction setting;
  • approximate control is powerful but can be unstable;
  • feature complexity creates a bias-variance tradeoff;
  • neural value functions extend the same ideas to learned nonlinear representations.

The next chapter studies stochastic approximation more systematically. That theory explains why update rules such as TD can converge despite noisy samples.

16.17 Exercises

16.17.1 Conceptual exercises

  1. Explain why tabular value functions do not generalize across states.
  2. In \(V_\theta(s)=\phi(s)^T\theta\), what is linear and what may be nonlinear?
  3. Why should approximation error be measured using a state distribution \(d(s)\)?
  4. Explain the difference between projecting \(V^\pi\) and solving the projected Bellman equation.
  5. Why is semi-gradient TD not the same as ordinary supervised gradient descent?
  6. State the deadly triad and explain why each component matters.

16.17.2 Mathematical exercises

  1. Let \(D\) be a positive diagonal matrix and \(\Phi\) have full column rank. Prove that

\[ \Pi_D=\Phi(\Phi^TD\Phi)^{-1}\Phi^TD \]

satisfies \(\Pi_D^2=\Pi_D\).

  1. Prove that the residual \(V-\Pi_DV\) is \(D\)-orthogonal to \(\operatorname{col}(\Phi)\).

  2. Starting from

\[ \Phi\theta = \Pi_D(r+\gamma P\Phi\theta), \]

derive the linear system

\[ \Phi^TD(I-\gamma P)\Phi\theta=\Phi^TD r. \]

  1. For a two-state Markov reward process, choose one constant feature \(\phi(s)=1\). Derive the projected Bellman equation explicitly.

  2. Show that if \(\operatorname{col}(\Phi)=\mathbb R^n\), then the projected Bellman equation reduces to the exact Bellman equation.

16.17.3 Computational exercises

  1. Modify the polynomial approximation example using Fourier features instead of polynomial features.
  2. Estimate the state distribution \(d\) from a simulated trajectory and recompute the least-squares projection.
  3. Compare the projected Bellman solution with the least-squares projection of the exact value for several feature matrices.
  4. Implement linear TD(0) with three different learning rates and compare convergence behavior.
  5. Implement approximate SARSA using a feature vector \(\phi(s,a)\) for a small gridworld.
  6. Test how the condition number of \(\Phi^TD\Phi\) changes as more polynomial features are added.

16.17.4 AI-assisted exercises

  1. Ask an AI assistant to propose features for an inventory-control problem. Identify which features are mathematically valid state variables.
  2. Ask an AI assistant to explain the difference between MSVE, MSBE, and MSPBE. Correct any imprecise statements.
  3. Give an AI assistant a TD update and ask whether it is on-policy or off-policy. Verify its answer algebraically.
  4. Ask an AI assistant to rewrite the projected Bellman equation in matrix form, then check every dimension.
  5. Ask an AI assistant to suggest diagnostics for instability in approximate Q-learning. Translate each diagnostic into a mathematical quantity or plot.

16.18 Instructor notes

This chapter is a bridge between tabular dynamic programming and modern deep reinforcement learning. For MA Applied Math and MS Statistics students, emphasize three connections:

  1. Linear algebra: feature spaces, projections, normal equations, conditioning.
  2. Probability: state distributions, sampled transitions, dependent data.
  3. Optimization: semi-gradient updates, stochastic approximation, and objective mismatch.

A useful lecture sequence is: start with least-squares projection, then show that Bellman backups leave the feature space, then derive the projected Bellman equation, then derive linear TD as a stochastic approximation to that equation.

16.19 References

Key references for function approximation in RL include Sutton and Barto (sutton2018reinforcement?), Bertsekas and Tsitsiklis (bertsekas1996neuro?), Tsitsiklis and Van Roy (tsitsiklis1997analysis?), and Szepesvari (szepesvari2010algorithms?).