Lab 12. Least Squares and Data Fitting: Independent Study
This lab accompanies Chapter 12: Least Squares and Data Fitting.
The goal is to connect least squares with geometry, computation, and data modeling:
- least squares as projection onto a column space;
- residuals and orthogonality;
- normal equations;
- QR-based least squares computation;
- polynomial data fitting;
- weighted least squares;
- best approximation of functions.
This is an independent-study lab. Each main question includes a worked solution and a similar practice question.
Python practice notebook
You may use the Jupyter notebook version for longer Python practice:
Interactive lab
Study guide and worked questions
Question 1. A least squares mean
Let
\[ A=\begin{bmatrix}1\\1\\1\end{bmatrix}, \qquad \vec{b}=\begin{bmatrix}2\\4\\5\end{bmatrix}. \]
Find the least squares solution of \(Ax=\vec{b}\).
Solution
The normal equation is
\[ A^TAx=A^T\vec{b}. \]
Here
\[ A^TA=3, \qquad A^T\vec{b}=11. \]
Thus
\[ x=\frac{11}{3}. \]
This is the average of the data values \(2,4,5\).
Similar practice
For \(\vec{b}=(3,7,8,10)^T\) and \(A=(1,1,1,1)^T\), find the least squares solution.
Answer
The solution is the average:
\[ x=\frac{3+7+8+10}{4}=7. \]
Question 2. Fit a line by normal equations
Fit \(h(t)=c_0+c_1t\) to the points
\[ (0,1),\quad (1,2),\quad (2,4). \]
Solution
The design matrix and data vector are
\[ A=\begin{bmatrix}1&0\\1&1\\1&2\end{bmatrix}, \qquad \vec{b}=\begin{bmatrix}1\\2\\4\end{bmatrix}. \]
Then
\[ A^TA=\begin{bmatrix}3&3\\3&5\end{bmatrix}, \qquad A^T\vec{b}=\begin{bmatrix}7\\10\end{bmatrix}. \]
Solving gives
\[ \vec{c}=\begin{bmatrix}5/6\\3/2\end{bmatrix}. \]
So
\[ h(t)=\frac56+\frac32t. \]
Similar practice
Fit a line to
\[ (0,2),\quad (1,3),\quad (2,5). \]
Answer
The design matrix is the same, and \(\vec{b}=(2,3,5)^T\). Solving gives
\[ h(t)=\frac{11}{6}+\frac32t. \]
Question 3. Residual orthogonality
For the least squares line in Question 2, verify that the residual is orthogonal to the columns of \(A\).
Solution
The fitted vector is
\[ A\vec{c}= \begin{bmatrix} 5/6\\7/3\\23/6 \end{bmatrix}. \]
The residual is
\[ \vec{r}=\vec{b}-A\vec{c} = \begin{bmatrix} 1/6\\-1/3\\1/6 \end{bmatrix}. \]
The columns of \(A\) are \((1,1,1)^T\) and \((0,1,2)^T\). We check
\[ (1,1,1)\cdot \vec{r}=0, \qquad (0,1,2)\cdot \vec{r}=0. \]
Thus \(\vec{r}\perp \operatorname{Col}(A)\).
Similar practice
Use Python to verify residual orthogonality for the similar practice problem in Question 2.
Answer
The result should satisfy
\[ A^T\vec{r}=\vec{0} \]
up to numerical rounding error.
Question 4. Rank-deficient least squares
Let
\[ A=\begin{bmatrix}1&1\\1&1\\1&1\end{bmatrix}, \qquad \vec{b}=\begin{bmatrix}1\\2\\3\end{bmatrix}. \]
Find all least squares solutions.
Solution
The model output is
\[ A\vec{x}=(x_1+x_2)\begin{bmatrix}1\\1\\1\end{bmatrix}. \]
The best constant approximation to \((1,2,3)\) is its average \(2\). Therefore
\[ x_1+x_2=2. \]
All least squares solutions are
\[ \vec{x}=\begin{bmatrix}t\\2-t\end{bmatrix}, \qquad t\in\mathbb{R}. \]
Similar practice
For the same \(A\) and \(\vec{b}=(2,4,6)^T\), find all least squares solutions.
Answer
The average is \(4\), so
\[ x_1+x_2=4. \]
Thus
\[ \vec{x}=\begin{bmatrix}t\\4-t\end{bmatrix}. \]
Question 5. Polynomial fitting
Fit a quadratic polynomial
\[ g(t)=c_0+c_1t+c_2t^2 \]
to the four points
\[ (0,5),\quad (1,3),\quad (-1,13),\quad (2,1). \]
Solution
The design matrix is
\[ A=\begin{bmatrix} 1&0&0\\ 1&1&1\\ 1&-1&1\\ 1&2&4 \end{bmatrix}, \qquad \vec{b}=\begin{bmatrix}5\\3\\13\\1\end{bmatrix}. \]
Solving \(A^TA\vec{c}=A^T\vec{b}\) gives
\[ \vec{c}=\begin{bmatrix}29/5\\-16/5\\3/5\end{bmatrix}. \]
Therefore
\[ g(t)=\frac{29}{5}-\frac{16}{5}t+\frac35t^2. \]
Similar practice
Use Python to fit a line, a quadratic, and a cubic to the same four points. Compare the residual norms.
Answer
The cubic has residual norm \(0\) because a cubic can interpolate four generic data points exactly. The line and quadratic generally have nonzero residual norms.
Question 6. Best function approximation
Set up the normal equations for the best linear approximation to \(e^x\) on \([0,1]\).
Solution
Let
\[ p(x)=c_0+c_1x. \]
The orthogonality equations are
\[ \int_0^1(e^x-c_0-c_1x)\,dx=0, \]
and
\[ \int_0^1x(e^x-c_0-c_1x)\,dx=0. \]
Thus
\[ \begin{bmatrix} 1&1/2\\ 1/2&1/3 \end{bmatrix} \begin{bmatrix}c_0\\c_1\end{bmatrix} = \begin{bmatrix}e-1\\1\end{bmatrix}. \]
Similar practice
Set up the normal equations for the best quadratic approximation to \(\sin x\) on \([0,1]\).
Answer
Let \(p(x)=c_0+c_1x+c_2x^2\). Then
\[ \sum_{j=0}^2 c_j\int_0^1 x^{j+k}\,dx = \int_0^1 x^k\sin x\,dx, \qquad k=0,1,2. \]