Chapter 15 · Linear Algebra Computation

Least Squares and Data Fitting

Approximate inconsistent systems and fit models to data

Normal equations Residuals Regression line QR least squares

Opening idea

Many linear systems in applications do not have exact solutions. A data set may contain noise. Measurements may be inconsistent. A model may be too simple to fit every observation exactly. In these situations, solving $$ Ax=b $$ exactly may be impossible. But we can still ask for the best approximate solution. The least-squares problem asks:

Big question. Approximate inconsistent systems and fit models to data

Learning goals

  1. Recognize when a linear system is inconsistent.
  2. Formulate the least-squares problem.
  3. Interpret least squares as an orthogonal projection problem.
  4. Derive the normal equations.
  5. Solve least-squares problems using normal equations.
  6. Solve least-squares problems using QR factorization.
  7. Fit lines and polynomials to data.
  8. Interpret residuals and fitted values.
  9. Understand why QR or np.linalg.lstsq is preferred computationally.
  10. Use Python to fit models, plot data, and analyze errors.

Core formulas

Least-squares problem
$$\hat x=\arg\min_x\|Ax-b\|^2$$
Normal equations
$$A^TA\hat x=A^Tb$$
Residual orthogonality
$$A^T(b-A\hat x)=0$$
Hat matrix
$$\hat b=A\hat x=P_{\operatorname{Col}(A)}b$$

Chapter map from the uploaded QMD

The interactive page follows the chapter sequence and turns key computational ideas into live tools.

  1. 15.1 Exact systems versus approximate systems
  2. 15.2 The least-squares problem
  3. 15.3 Geometry of least squares
  4. 15.4 Why they are called normal equations
  5. 15.5 A first least-squares example
  6. 15.6 Residual and orthogonality check
  7. 15.7 Python: solving by normal equations
  8. 15.8 Line fitting
  9. 15.9 Python plot: least-squares line
  10. 15.10 Residuals in data fitting
  11. 15.11 Visualizing residuals
  12. 15.12 QR solution of least squares
  13. 15.13 Why QR is better than normal equations
  14. 15.14 Python: least squares by QR
  15. 15.15 Python: least squares using np.linalg.lstsq
  16. 15.16 Polynomial fitting

Interactive Lab 1: Fit a line to data

Edit the data points. The tool builds the design matrix for $y=\beta_0+\beta_1x$, solves least squares, and draws residuals.

Model report

SSE0
RMSE0
$A^Tr$0

Interactive Lab 2: Normal equations calculator

Enter a matrix $A$ and vector $b$. This computes $A^TA$, $A^Tb$, and $\hat x$ for full-rank 2-column designs.

Interactive Lab 3: Polynomial degree

Use the current data and compare degree 1 versus degree 2 fitting.

Algorithmic summary

  1. 1Build a design matrix $A$ from the model.
  2. 2Choose a stable solver: QR or SVD preferred; normal equations for hand computation.
  3. 3Solve for coefficients $\hat x$.
  4. 4Compute fitted values $\hat b=A\hat x$ and residual $r=b-\hat b$.
  5. 5Assess error with SSE or RMSE.

Common mistakes to avoid

  • Expecting an exact solution to an overdetermined inconsistent system.
  • Forgetting to check residual orthogonality.
  • Confusing fitted values $\hat b$ with residuals $b-\hat b$.
  • Using normal equations blindly when $A$ is ill-conditioned.
  • Ignoring rank deficiency.
AI-age connection. Least squares is the first model-fitting algorithm: it introduces loss functions, gradients, projections, residuals, and the idea of learning from data.

Quick checks

What is the main computational object in this chapter?

Use the formulas and live labs above: identify the matrix/vector operation that turns the concept into computation.

What geometric picture should you keep in mind?

Follow the canvas visualization: a vector, subspace, transformed square, fitted line, or special direction depending on the chapter.

What should be checked after computation?

Check the defining property: orthogonality, $Q^TQ=I$, residual condition, determinant sign/size, or $Av=\lambda v$.

Chapter 15 interactive HTML · generated from uploaded QMD · no build system required