1. Choose data points
Fit a polynomial $p(t)=c_0+c_1t+\cdots+c_dt^d$ to the points using least squares.
2. Least squares fit
Blue points are data. Orange curve is the least squares fit. Gray vertical segments show residuals.
3. Normal equations
The normal equations are $A^T A\vec{c}=A^T\vec{b}$, or in weighted form $A^TWA\vec{c}=A^TW\vec{b}$.
4. Residual orthogonality
For ordinary least squares, the residual satisfies $A^T\vec{r}=0$. For weighted least squares, it satisfies $A^TW\vec{r}=0$.
5. Projection example
Use the lecture matrix
$$A=\begin{bmatrix}-1&4\\1&8\\-1&4\end{bmatrix},\quad \vec b=\begin{bmatrix}14\\-4\\0\end{bmatrix}.$$
6. Best linear approximation to $e^x$
The best $p(x)=c_0+c_1x$ on $[0,1]$ satisfies
$$\begin{bmatrix}1&1/2\\1/2&1/3\end{bmatrix}\begin{bmatrix}c_0\\c_1\end{bmatrix}=\begin{bmatrix}e-1\\1\end{bmatrix}.$$
7. Independent-study practice tasks with answers
Task A. Fit a line to the Chapter data and record the residual norm.
Answer: Select degree 1. The coefficients should be approximately $c_0=37/6$ and $c_1=-61/18$.
Answer: Select degree 1. The coefficients should be approximately $c_0=37/6$ and $c_1=-61/18$.
Task B. Fit a quadratic to the Chapter data.
Answer: Select degree 2. The coefficients should be $c_0=29/5$, $c_1=-16/5$, $c_2=3/5$.
Answer: Select degree 2. The coefficients should be $c_0=29/5$, $c_1=-16/5$, $c_2=3/5$.
Task C. Fit a cubic to the Chapter data. What happens?
Answer: The cubic interpolates the four points exactly and the residual norm is $0$ up to rounding error.
Answer: The cubic interpolates the four points exactly and the residual norm is $0$ up to rounding error.
Task D. Turn on weights and make one data point have a large weight. What happens?
Answer: The fitted curve moves closer to the heavily weighted observation.
Answer: The fitted curve moves closer to the heavily weighted observation.
Task E. In the projection example, verify that $A^T\vec r=0$.
Answer: The residual is $(7,0,-7)^T$, and multiplying by $A^T$ gives the zero vector.
Answer: The residual is $(7,0,-7)^T$, and multiplying by $A^T$ gives the zero vector.