28  Review Problems

This page contains cumulative review problems for Computational Linear Algebra in \(\\mathbb{R}^n\).

The problems are organized by theme. Some are computational, some are conceptual, some ask for proofs, and some ask for interpretation. They are designed for exam review, independent study, recitation, or final projects.

Unless otherwise stated, all vectors and matrices have real entries.

28.1 How to use this review

A good review strategy is to work in layers.

  1. First, solve the computational problems by hand.
  2. Then explain the meaning of each answer geometrically.
  3. Then verify selected answers with Python.
  4. Finally, write short proofs for the conceptual claims.

When possible, connect every answer to one of the central ideas:

  • column space,
  • null space,
  • rank,
  • pivots,
  • basis,
  • coordinates,
  • transformation,
  • projection,
  • least squares,
  • eigenvectors,
  • diagonalization,
  • orthogonality,
  • symmetric matrices,
  • quadratic forms,
  • singular values,
  • low-rank structure.

28.2 Part I. Linear systems and row reduction

28.2.1 Problem 1. Solving and interpreting a system

Solve the system

\[ \begin{cases} x+2y-z=3,\\ 2x+5y+z=9,\\ -x-y+2z=0. \end{cases} \]

  1. Write the augmented matrix.
  2. Row reduce the augmented matrix.
  3. Determine whether the system is consistent.
  4. If it is consistent, write the solution.
  5. Interpret the solution geometrically as an intersection of planes.

28.2.2 Problem 2. Free variables

Consider the system with augmented matrix

\[ \left[ \begin{array}{cccc|c} 1&2&-1&3&4\\ 2&4&-2&6&8\\ 0&1&1&-1&2 \end{array} \right]. \]

  1. Row reduce the matrix.
  2. Identify pivot variables and free variables.
  3. Write the general solution in parametric vector form.
  4. Find a basis for the solution set of the associated homogeneous system.
  5. Explain why the nonhomogeneous solution set is not a subspace.

28.2.3 Problem 3. Consistency by column space

Let

\[ A= \begin{bmatrix} 1&2&0\\ 0&1&1\\ 1&3&1 \end{bmatrix}, \qquad b= \begin{bmatrix} 3\\ 2\\ 5 \end{bmatrix}. \]

  1. Determine whether \(Ax=b\) is consistent.
  2. Decide whether \(b\in\operatorname{Col}(A)\).
  3. Find all solutions if they exist.
  4. Determine \(\operatorname{rank}(A)\).
  5. Determine \(\dim\operatorname{Null}(A)\).

28.2.4 Problem 4. Parameters in a system

For which values of \(h\) is the system consistent?

\[ \left[ \begin{array}{ccc|c} 1&2&-1&3\\ 2&4&h&6\\ -1&-2&1&-3 \end{array} \right]. \]

For each case, describe the number of solutions.

28.2.5 Problem 5. Rank and pivots

Let

\[ A= \begin{bmatrix} 1&2&3&4\\ 2&4&6&8\\ 1&1&0&-1\\ 0&1&3&5 \end{bmatrix}. \]

  1. Find the reduced row echelon form.
  2. Find the pivot columns.
  3. Find a basis for \(\operatorname{Col}(A)\).
  4. Find a basis for \(\operatorname{Null}(A)\).
  5. Verify the rank-nullity theorem.

28.3 Part II. Matrix algebra and inverses

28.3.1 Problem 6. Matrix products

Let

\[ A= \begin{bmatrix} 1&2\\ 0&1\\ 3&-1 \end{bmatrix}, \qquad B= \begin{bmatrix} 2&0&1\\ 1&3&-2 \end{bmatrix}. \]

  1. Compute \(AB\).
  2. Compute \(BA\).
  3. Explain why both products are defined.
  4. Compare the sizes of \(AB\) and \(BA\).
  5. Explain why matrix multiplication is not commutative.

28.3.2 Problem 7. Inverses

Let

\[ A= \begin{bmatrix} 2&1\\ 5&3 \end{bmatrix}. \]

  1. Compute \(\det(A)\).
  2. Find \(A^{-1}\).
  3. Solve \(Ax=b\) for

\[ b= \begin{bmatrix} 1\\ 4 \end{bmatrix}. \]

  1. Verify your solution by multiplication.
  2. Interpret invertibility geometrically.

28.3.3 Problem 8. Elementary matrices

For each row operation, write the corresponding elementary matrix.

  1. Swap rows 1 and 2 in a \(3\times 3\) matrix.
  2. Multiply row 2 by \(5\).
  3. Add \(-3\) times row 1 to row 3.
  4. Explain why elementary matrices are invertible.
  5. Use elementary matrices to describe row reduction.

28.3.4 Problem 9. Invertible Matrix Theorem

Let \(A\) be a \(4\times 4\) matrix.

For each statement, decide whether it implies that \(A\) is invertible.

  1. The columns of \(A\) are linearly independent.
  2. The equation \(Ax=0\) has infinitely many solutions.
  3. \(\operatorname{rank}(A)=4\).
  4. \(\operatorname{Col}(A)=\mathbb{R}^4\).
  5. \(\det(A)=0\).
  6. \(Ax=b\) is consistent for every \(b\in\mathbb{R}^4\).
  7. \(A\) has four pivot columns.
  8. \(A^TA\) is invertible.

Explain each answer.

28.3.5 Problem 10. Block matrix thinking

Suppose

\[ A= \begin{bmatrix} B&0\\ 0&C \end{bmatrix}, \]

where \(B\) and \(C\) are square matrices.

  1. What is \(A^2\)?
  2. If \(B\) and \(C\) are invertible, find \(A^{-1}\).
  3. What is \(\det(A)\) in terms of \(\det(B)\) and \(\det(C)\)?
  4. What are the eigenvalues of \(A\) in terms of eigenvalues of \(B\) and \(C\)?

28.4 Part III. Span, basis, dimension, and subspaces

28.4.1 Problem 11. Span membership

Let

\[ v_1= \begin{bmatrix} 1\\ 2\\ 1 \end{bmatrix}, \quad v_2= \begin{bmatrix} 2\\ 1\\ 0 \end{bmatrix}, \quad v_3= \begin{bmatrix} 1\\ 0\\ 1 \end{bmatrix}, \quad b= \begin{bmatrix} 4\\ 5\\ 2 \end{bmatrix}. \]

  1. Determine whether \(b\in\operatorname{span}\{v_1,v_2,v_3\}\).
  2. If yes, find coefficients \(c_1,c_2,c_3\).
  3. Determine whether \(v_1,v_2,v_3\) are linearly independent.
  4. Find a basis for their span.
  5. State the dimension of the span.

28.4.2 Problem 12. Subspace test

For each set, determine whether it is a subspace of the indicated space.

\[ W_1=\left\{ \begin{bmatrix} x\\ y\\ z \end{bmatrix} :x+y+z=0 \right\}\subseteq\mathbb{R}^3. \]

\[ W_2=\left\{ \begin{bmatrix} x\\ y\\ z \end{bmatrix} :x+y+z=1 \right\}\subseteq\mathbb{R}^3. \]

\[ W_3=\left\{ \begin{bmatrix} x\\ y \end{bmatrix} :y=x^2 \right\}\subseteq\mathbb{R}^2. \]

\[ W_4=\operatorname{Null}(A) \]

for a matrix \(A\).

\[ W_5=\operatorname{Col}(A) \]

for an \(m\times n\) matrix \(A\).

Give reasons.

28.4.3 Problem 13. Bases for four fundamental subspaces

Let

\[ A= \begin{bmatrix} 1&2&1&0\\ 2&4&0&2\\ 1&2&-1&2 \end{bmatrix}. \]

  1. Find a basis for \(\operatorname{Col}(A)\).
  2. Find a basis for \(\operatorname{Null}(A)\).
  3. Find a basis for \(\operatorname{Row}(A)\).
  4. Find a basis for \(\operatorname{Null}(A^T)\).
  5. Verify all relevant dimension identities.

28.4.4 Problem 14. Coordinates in a basis

Let

\[ B= \left\{ \begin{bmatrix} 1\\ 1 \end{bmatrix}, \begin{bmatrix} 1\\ -1 \end{bmatrix} \right\}. \]

  1. Show that \(B\) is a basis of \(\mathbb{R}^2\).
  2. Find \([x]_B\) for

\[ x= \begin{bmatrix} 4\\ 2 \end{bmatrix}. \]

  1. Find the basis matrix \(P_B\).
  2. Verify that \(x=P_B[x]_B\).
  3. Describe the geometry of this coordinate system.

28.4.5 Problem 15. Change of basis

Let

\[ B= \left\{ \begin{bmatrix} 1\\ 1 \end{bmatrix}, \begin{bmatrix} 1\\ -1 \end{bmatrix} \right\}, \qquad C= \left\{ \begin{bmatrix} 2\\ 0 \end{bmatrix}, \begin{bmatrix} 0\\ 3 \end{bmatrix} \right\}. \]

  1. Find \(P_B\) and \(P_C\).
  2. Find the change-of-basis matrix from \(B\)-coordinates to \(C\)-coordinates.
  3. Convert \([x]_B=(2,-1)^T\) into \(C\)-coordinates.
  4. Convert back to standard coordinates and check consistency.

28.5 Part IV. Linear transformations

28.5.1 Problem 16. Standard matrix

Let \(T:\mathbb{R}^2\to\mathbb{R}^3\) be defined by

\[ T \left( \begin{bmatrix} x\\ y \end{bmatrix} \right) = \begin{bmatrix} x+2y\\ 3x-y\\ y \end{bmatrix}. \]

  1. Show that \(T\) is linear.
  2. Find the standard matrix of \(T\).
  3. Find \(\ker(T)\).
  4. Find \(\operatorname{im}(T)\).
  5. Determine whether \(T\) is one-to-one.
  6. Determine whether \(T\) is onto \(\mathbb{R}^3\).

28.5.2 Problem 17. Geometry of transformations

For each matrix, describe the transformation geometrically.

\[ A_1= \begin{bmatrix} 2&0\\ 0&1 \end{bmatrix} \]

\[ A_2= \begin{bmatrix} 0&-1\\ 1&0 \end{bmatrix} \]

\[ A_3= \begin{bmatrix} 1&1\\ 0&1 \end{bmatrix} \]

\[ A_4= \begin{bmatrix} 1&0\\ 0&0 \end{bmatrix} \]

For each matrix, also compute determinant, rank, and whether it is invertible.

28.5.3 Problem 18. Composition

Let

\[ R= \begin{bmatrix} 0&-1\\ 1&0 \end{bmatrix}, \qquad S= \begin{bmatrix} 1&1\\ 0&1 \end{bmatrix}. \]

  1. Compute \(RS\).
  2. Compute \(SR\).
  3. Describe the difference between applying \(S\) then \(R\) and applying \(R\) then \(S\).
  4. Explain why order matters in matrix multiplication.

28.5.4 Problem 19. Kernel and image

Let

\[ A= \begin{bmatrix} 1&2&3\\ 0&1&1 \end{bmatrix}. \]

For \(T(x)=Ax\):

  1. Find \(\ker(T)\).
  2. Find \(\operatorname{im}(T)\).
  3. Determine the rank and nullity of \(T\).
  4. Is \(T\) one-to-one?
  5. Is \(T\) onto \(\mathbb{R}^2\)?

28.6 Part V. Orthogonality, projection, QR, and least squares

28.6.1 Problem 20. Dot products and angles

Let

\[ u= \begin{bmatrix} 1\\ 2\\ 2 \end{bmatrix}, \qquad v= \begin{bmatrix} 2\\ 0\\ 1 \end{bmatrix}. \]

  1. Compute \(u\cdot v\).
  2. Compute \(\|u\|\) and \(\|v\|\).
  3. Compute the angle between \(u\) and \(v\).
  4. Find a unit vector in the direction of \(u\).
  5. Determine whether \(u\) and \(v\) are orthogonal.

28.6.2 Problem 21. Projection onto a line

Let

\[ a= \begin{bmatrix} 2\\ 1 \end{bmatrix}, \qquad b= \begin{bmatrix} 1\\ 3 \end{bmatrix}. \]

  1. Find \(\operatorname{proj}_{\operatorname{span}\{a\}}(b)\).
  2. Find the residual \(e=b-\operatorname{proj}(b)\).
  3. Verify that \(e\perp a\).
  4. Find the projection matrix onto \(\operatorname{span}\{a\}\).
  5. Verify that \(P^2=P\) and \(P^T=P\).

28.6.3 Problem 22. Projection onto a subspace

Let

\[ W=\operatorname{span} \left\{ \begin{bmatrix} 1\\ 0\\ 1 \end{bmatrix}, \begin{bmatrix} 0\\ 1\\ 1 \end{bmatrix} \right\}. \]

  1. Find an orthonormal basis for \(W\).
  2. Project

\[ b= \begin{bmatrix} 1\\ 2\\ 3 \end{bmatrix} \]

onto \(W\). 3. Find the residual. 4. Verify orthogonality of the residual to \(W\). 5. Find the projection matrix onto \(W\).

28.6.4 Problem 23. Gram–Schmidt

Apply Gram–Schmidt to

\[ a_1= \begin{bmatrix} 1\\ 1\\ 0 \end{bmatrix}, \qquad a_2= \begin{bmatrix} 1\\ 0\\ 1 \end{bmatrix}, \qquad a_3= \begin{bmatrix} 0\\ 1\\ 1 \end{bmatrix}. \]

  1. Find an orthogonal basis.
  2. Find an orthonormal basis.
  3. Form the matrix \(Q\).
  4. Verify that \(Q^TQ=I\).
  5. Explain what subspace these vectors span.

28.6.5 Problem 24. QR factorization

Let

\[ A= \begin{bmatrix} 1&1\\ 1&0\\ 0&1 \end{bmatrix}. \]

  1. Find a QR factorization of \(A\) by Gram–Schmidt.
  2. Verify \(A=QR\).
  3. Verify that \(Q^TQ=I\).
  4. Use \(Q\) to find the projection matrix onto \(\operatorname{Col}(A)\).

28.6.6 Problem 25. Least squares

Fit a line

\[ y=\beta_0+\beta_1t \]

to the data points

\[ (0,1),\quad (1,2),\quad (2,2),\quad (3,4). \]

  1. Build the design matrix.
  2. Solve the normal equations.
  3. Find the fitted values.
  4. Find the residual vector.
  5. Verify that the residual is orthogonal to the column space of the design matrix.
  6. Interpret the slope and intercept.

28.6.7 Problem 26. Least squares by QR

Use QR factorization to solve the least-squares problem from Problem 25.

  1. Compute \(A=QR\).
  2. Solve \(R\hat\beta=Q^Ty\).
  3. Compare with the normal-equation solution.
  4. Explain why QR is often preferred computationally.

28.7 Part VI. Determinants and volume

28.7.1 Problem 27. Determinants by row reduction

Compute the determinant of

\[ A= \begin{bmatrix} 1&2&3\\ 0&1&4\\ 5&6&0 \end{bmatrix} \]

using row reduction.

Track how each row operation changes the determinant.

28.7.2 Problem 28. Determinants and geometry

Let

\[ A= \begin{bmatrix} 2&1\\ 1&3 \end{bmatrix}. \]

  1. Compute \(\det(A)\).
  2. Find the area of the parallelogram spanned by the columns of \(A\).
  3. Decide whether \(A\) preserves or reverses orientation.
  4. Determine whether \(A\) is invertible.
  5. Describe what \(A\) does to area.

28.7.3 Problem 29. Determinant identities

Prove or disprove each statement.

  1. \(\det(AB)=\det(A)\det(B)\).
  2. \(\det(A+B)=\det(A)+\det(B)\).
  3. \(\det(A^T)=\det(A)\).
  4. If \(A\) is invertible, then \(\det(A^{-1})=1/\det(A)\).
  5. If \(\det(A)=0\), then \(A=0\).

For false statements, give counterexamples.

28.7.4 Problem 30. Determinant and eigenvalues

Let

\[ A= \begin{bmatrix} 4&1\\ 2&3 \end{bmatrix}. \]

  1. Compute \(\det(A)\).
  2. Find the eigenvalues.
  3. Verify that the product of eigenvalues equals \(\det(A)\).
  4. Verify that the sum of eigenvalues equals \(\operatorname{tr}(A)\).

28.8 Part VII. Eigenvalues, diagonalization, and dynamics

28.8.1 Problem 31. Eigenvalues and eigenspaces

For

\[ A= \begin{bmatrix} 4&1\\ 2&3 \end{bmatrix}, \]

  1. Find the characteristic polynomial.
  2. Find the eigenvalues.
  3. Find a basis for each eigenspace.
  4. Determine whether \(A\) is diagonalizable.
  5. If diagonalizable, find \(P\) and \(D\) such that \(A=PDP^{-1}\).

28.8.2 Problem 32. Repeated eigenvalue

Let

\[ A= \begin{bmatrix} 2&1\\ 0&2 \end{bmatrix}. \]

  1. Find the eigenvalues.
  2. Find the eigenspace.
  3. Determine algebraic and geometric multiplicity.
  4. Decide whether \(A\) is diagonalizable.
  5. Explain geometrically why this matrix does not have enough eigenvectors.

28.8.3 Problem 33. Matrix powers

Let

\[ A= \begin{bmatrix} 2&1\\ 1&2 \end{bmatrix}. \]

  1. Diagonalize \(A\).
  2. Find a formula for \(A^k\).
  3. Compute \(A^5\) using diagonalization.
  4. Compare with direct multiplication.
  5. Explain why diagonalization makes powers easier.

28.8.4 Problem 34. Discrete dynamical system

Consider

\[ x_{k+1}= \begin{bmatrix} 0.8&0.3\\ 0.2&0.7 \end{bmatrix} x_k. \]

  1. Find the eigenvalues.
  2. Find an eigenvector for eigenvalue \(1\).
  3. Normalize it so its entries sum to \(1\).
  4. Interpret it as a steady state.
  5. Starting from \(x_0=(1,0)^T\), describe the long-term behavior.

28.8.5 Problem 35. Stability

For each matrix, determine the long-term behavior of \(x_{k+1}=Ax_k\).

\[ A= \begin{bmatrix} 0.5&0\\ 0&0.2 \end{bmatrix} \]

\[ A= \begin{bmatrix} 1.2&0\\ 0&0.8 \end{bmatrix} \]

\[ A= \begin{bmatrix} 0&-1\\ 1&0 \end{bmatrix} \]

\[ A= \begin{bmatrix} 0.9&-0.2\\ 0.2&0.9 \end{bmatrix} \]

Use eigenvalues to explain your answers.

28.9 Part VIII. Orthogonal and symmetric matrices

28.9.1 Problem 36. Orthogonal matrices

For each matrix, determine whether it is orthogonal.

\[ Q_1= \begin{bmatrix} 0&-1\\ 1&0 \end{bmatrix} \]

\[ Q_2= \frac{1}{\sqrt{2}} \begin{bmatrix} 1&1\\ 1&-1 \end{bmatrix} \]

\[ Q_3= \begin{bmatrix} 2&0\\ 0&1/2 \end{bmatrix} \]

For each orthogonal matrix, compute its determinant and describe the transformation.

28.9.2 Problem 37. Symmetric matrices

Let

\[ A= \begin{bmatrix} 2&1\\ 1&2 \end{bmatrix}. \]

  1. Verify that \(A\) is symmetric.
  2. Find eigenvalues and eigenvectors.
  3. Normalize the eigenvectors.
  4. Form \(Q\) and \(D\).
  5. Verify that \(A=QDQ^T\).
  6. Write the spectral decomposition as a sum of projections.

28.9.3 Problem 38. Orthogonality of eigenvectors

Let \(A\) be symmetric and suppose

\[ Av=\lambda v, \qquad Aw=\mu w, \]

with \(\lambda\ne\mu\).

Prove that

\[ v\perp w. \]

28.9.4 Problem 39. Gram matrices

Let

\[ B= \begin{bmatrix} 1&2\\ 3&4\\ 5&6 \end{bmatrix}. \]

  1. Compute \(B^TB\).
  2. Verify that \(B^TB\) is symmetric.
  3. Show that \(x^TB^TBx\ge 0\) for all \(x\).
  4. Determine whether \(B^TB\) is positive definite.
  5. Relate your answer to the independence of the columns of \(B\).

28.10 Part IX. Quadratic forms and optimization

28.10.1 Problem 40. Matrix form of a quadratic form

Write each quadratic form as \(x^TAx\) with \(A\) symmetric.

\[ q(x_1,x_2)=3x_1^2+2x_1x_2+5x_2^2. \]

\[ q(x_1,x_2)=x_1^2-4x_1x_2+x_2^2. \]

\[ q(x_1,x_2,x_3)=2x_1^2+x_2^2+3x_3^2+6x_1x_3-2x_2x_3. \]

28.10.2 Problem 41. Classification by eigenvalues

Classify each symmetric matrix as positive definite, positive semidefinite, negative definite, negative semidefinite, or indefinite.

\[ A_1= \begin{bmatrix} 3&1\\ 1&3 \end{bmatrix} \]

\[ A_2= \begin{bmatrix} 1&2\\ 2&1 \end{bmatrix} \]

\[ A_3= \begin{bmatrix} 1&0\\ 0&0 \end{bmatrix} \]

\[ A_4= \begin{bmatrix} -2&0\\ 0&-3 \end{bmatrix} \]

28.10.3 Problem 42. Principal axes

Let

\[ A= \begin{bmatrix} 2&1\\ 1&2 \end{bmatrix}. \]

  1. Orthogonally diagonalize \(A\).
  2. Write \(x^TAx\) in principal-axis coordinates.
  3. Describe the level curves \(x^TAx=1\).
  4. Find the axis directions.
  5. Find the axis lengths.

28.10.4 Problem 43. Quadratic optimization

Let

\[ f(x)=\frac12x^T \begin{bmatrix} 3&1\\ 1&2 \end{bmatrix} x- \begin{bmatrix} 1\\ 4 \end{bmatrix}^Tx. \]

  1. Compute \(\nabla f(x)\).
  2. Find the critical point.
  3. Determine whether it is a minimum, maximum, or saddle.
  4. Complete the square in matrix form.
  5. Describe the level curves.

28.10.5 Problem 44. Gradient descent

For

\[ f(x)=\frac12x^T \begin{bmatrix} 8&0\\ 0&1 \end{bmatrix} x, \]

  1. Find the eigenvalues of the Hessian.
  2. Determine the range of step sizes \(\alpha\) for gradient descent convergence.
  3. Explain why the contours are elongated.
  4. Explain why gradient descent may zigzag.

28.11 Part X. Singular value decomposition and data

28.11.1 Problem 45. Singular values from \(A^TA\)

Let

\[ A= \begin{bmatrix} 3&0\\ 0&1\\ 0&0 \end{bmatrix}. \]

  1. Compute \(A^TA\).
  2. Find eigenvalues of \(A^TA\).
  3. Find singular values of \(A\).
  4. Find right singular vectors.
  5. Find left singular vectors corresponding to nonzero singular values.

28.11.2 Problem 46. SVD relations

For a matrix \(A\) with SVD

\[ A=U\Sigma V^T, \]

explain why:

  1. \(Av_i=\sigma_i u_i\).
  2. \(A^TAv_i=\sigma_i^2v_i\).
  3. \(AA^Tu_i=\sigma_i^2u_i\).
  4. \(\operatorname{rank}(A)\) equals the number of nonzero singular values.
  5. The SVD works even when \(A\) is rectangular.

28.11.3 Problem 47. Low-rank approximation

Suppose a matrix has singular values

\[ 10,\quad 4,\quad 1,\quad 0.2. \]

  1. What is the rank?
  2. What is the Frobenius norm?
  3. What fraction of energy is captured by the first singular value?
  4. What fraction is captured by the first two singular values?
  5. What is the Frobenius error of the best rank-\(2\) approximation?

28.11.4 Problem 48. Image compression

A grayscale image is represented by a \(500\times 400\) matrix.

A rank-\(k\) SVD approximation stores approximately

\[ k(m+n+1) \]

numbers.

  1. How many numbers are needed to store the original image?
  2. How many numbers are needed for a rank-\(20\) approximation?
  3. What is the compression ratio?
  4. What information is lost when using a low-rank approximation?
  5. Why might the largest singular values capture the main image structure?

28.11.5 Problem 49. PCA

Let \(X_c\) be a centered data matrix with SVD

\[ X_c=U\Sigma V^T. \]

  1. Explain why columns of \(V\) are principal component directions.
  2. Express the covariance matrix in terms of \(V\) and \(\Sigma\).
  3. Write the explained variance for each component.
  4. Write the formula for projecting data onto the first \(k\) principal components.
  5. Write the formula for reconstructing from the first \(k\) components.

28.11.6 Problem 50. Pseudoinverse

Let \(A=U\Sigma V^T\).

  1. Define \(A^+\).
  2. Explain why \(A^+b\) solves least squares in the minimum-norm sense.
  3. What happens to small singular values in the pseudoinverse?
  4. Why can this cause numerical instability?
  5. How might truncated SVD help?

28.12 Part XI. Computational review

28.12.1 Problem 51. Python: row reduction and rank

Use Python to study

\[ A= \begin{bmatrix} 1&2&3&4\\ 2&4&6&8\\ 1&1&0&-1\\ 0&1&3&5 \end{bmatrix}. \]

  1. Use SymPy to compute the RREF.
  2. Identify pivot columns.
  3. Compute rank using NumPy.
  4. Find a basis for the null space using SymPy.
  5. Verify rank-nullity.

28.12.2 Problem 52. Python: least squares

Generate data from

\[ y=2+3t+\epsilon, \]

where \(\epsilon\) is random noise.

  1. Build a design matrix.
  2. Solve least squares using np.linalg.lstsq.
  3. Plot the data and fitted line.
  4. Compute residuals.
  5. Verify \(A^Tr\approx 0\).

28.12.3 Problem 53. Python: eigenvalue dynamics

Choose a \(2\times 2\) diagonalizable matrix with one eigenvalue larger than \(1\) and one eigenvalue less than \(1\) in absolute value.

  1. Compute eigenvalues and eigenvectors.
  2. Simulate \(x_{k+1}=Ax_k\).
  3. Plot the trajectory.
  4. Explain the dominant direction.
  5. Compare with the eigenvector associated with the largest \(|\lambda|\).

28.12.4 Problem 54. Python: PCA and SVD

Generate a two-dimensional data cloud with correlated features.

  1. Center the data.
  2. Compute the covariance matrix.
  3. Find eigenvalues and eigenvectors.
  4. Compute the SVD of the centered data matrix.
  5. Plot principal directions.
  6. Project data onto the first principal component.
  7. Reconstruct and compute error.

28.12.5 Problem 55. Python: image-like matrix compression

Create a synthetic matrix using smooth functions, for example

\[ A_{ij}=\sin(x_i)\cos(y_j)+0.5\cos(2x_i)\sin(y_j). \]

  1. Compute the SVD.
  2. Plot singular values.
  3. Construct rank-\(1\), rank-\(2\), rank-\(5\), and rank-\(10\) approximations.
  4. Compare approximation errors.
  5. Visualize the approximations using imshow.

28.13 Part XII. Conceptual review

28.13.1 Problem 56. Explain in words

Write a paragraph explaining each concept.

  1. Column space.
  2. Null space.
  3. Rank.
  4. Basis.
  5. Orthogonal projection.
  6. Least squares.
  7. Eigenvector.
  8. Diagonalization.
  9. Spectral theorem.
  10. Singular value.

28.13.2 Problem 57. True or false

Decide whether each statement is true or false. Give a proof or counterexample.

  1. If \(AB=0\), then \(A=0\) or \(B=0\).
  2. If \(A\) is invertible, then \(Ax=b\) has a unique solution for every \(b\).
  3. If the columns of \(A\) span \(\mathbb{R}^m\), then \(Ax=b\) is consistent for every \(b\in\mathbb{R}^m\).
  4. If the columns of \(A\) are linearly independent, then \(\operatorname{Null}(A)=\{0\}\).
  5. If \(A\) is symmetric, then \(A\) is diagonalizable.
  6. If \(A\) is diagonalizable, then \(A\) is symmetric.
  7. If \(Q\) is orthogonal, then \(Q^{-1}=Q^T\).
  8. If \(A\) is positive definite, then all eigenvalues of \(A\) are positive.
  9. Singular values can be negative.
  10. The SVD works only for square matrices.

28.13.3 Problem 58. Connect the concepts

For each pair, explain the connection.

  1. Pivot columns and column space.
  2. Free variables and null space.
  3. Orthogonality and least squares.
  4. Projection and best approximation.
  5. Determinant and invertibility.
  6. Eigenvalues and matrix powers.
  7. Symmetric matrices and orthogonal diagonalization.
  8. Quadratic forms and optimization.
  9. Singular values and rank.
  10. PCA and SVD.

28.13.4 Problem 59. One matrix, many viewpoints

Let

\[ A= \begin{bmatrix} 2&1\\ 1&2 \end{bmatrix}. \]

Analyze \(A\) from as many viewpoints as possible.

  1. Matrix transformation.
  2. Determinant and area scaling.
  3. Eigenvalues and eigenvectors.
  4. Diagonalization.
  5. Orthogonal diagonalization.
  6. Quadratic form \(x^TAx\).
  7. Singular values.
  8. Condition number.
  9. Effect on the unit circle.
  10. Role in a dynamical system \(x_{k+1}=Ax_k\).

28.13.5 Problem 60. Final synthesis

Write a short essay answering:

How does linear algebra connect systems of equations, geometry, computation, and data?

Your essay should mention at least eight of the following terms:

  • matrix,
  • vector,
  • subspace,
  • basis,
  • coordinates,
  • projection,
  • least squares,
  • eigenvalue,
  • diagonalization,
  • symmetric matrix,
  • quadratic form,
  • SVD,
  • PCA,
  • low-rank approximation,
  • data matrix,
  • neural-network layer.

28.14 Part XIII. Challenge problems

28.14.1 Problem 61. Projection theorem

Let \(W\) be a subspace of \(\mathbb{R}^n\).

Prove that for every \(b\in\mathbb{R}^n\), there is a unique decomposition

\[ b=p+e, \]

where

\[ p\in W \]

and

\[ e\in W^\perp. \]

Use this to prove that \(p\) is the closest vector in \(W\) to \(b\).

28.14.2 Problem 62. Orthogonal complement dimensions

Let \(W\) be a subspace of \(\mathbb{R}^n\).

Prove that

\[ \dim(W)+\dim(W^\perp)=n. \]

Then apply this result to explain why

\[ \operatorname{rank}(A)+\dim\operatorname{Null}(A)=n. \]

28.14.3 Problem 63. Symmetric positive definite matrices

Let \(A\) be symmetric.

Prove that \(A\) is positive definite if and only if all eigenvalues of \(A\) are positive.

Use the spectral theorem.

28.14.4 Problem 64. Diagonalization and powers

Suppose \(A\) is diagonalizable and all eigenvalues satisfy

\[ |\lambda_i|<1. \]

Prove that

\[ A^k\to 0 \]

as

\[ k\to\infty. \]

What happens if one eigenvalue has absolute value greater than \(1\)?

28.14.5 Problem 65. SVD and rank

Use the SVD to prove that

\[ \operatorname{rank}(A)=\operatorname{rank}(A^T)=\operatorname{rank}(A^TA). \]

Explain why the nonzero singular values control all three ranks.

28.15 Part XIV. Checklist for final review

Before a final exam or cumulative assessment, make sure you can do the following.

28.15.1 Systems and matrices

  • Row reduce an augmented matrix.
  • Identify pivots and free variables.
  • Write a solution set in parametric vector form.
  • Interpret consistency using column space.
  • Compute matrix products and inverses.
  • Use the Invertible Matrix Theorem.

28.15.2 Vector spaces

  • Test whether a set is a subspace.
  • Determine span and linear independence.
  • Find bases for column spaces, null spaces, and row spaces.
  • Use rank-nullity.
  • Change coordinates between bases.

28.15.3 Geometry

  • Interpret matrices as transformations.
  • Compute dot products, norms, distances, and angles.
  • Find projections onto lines and subspaces.
  • Use Gram–Schmidt.
  • Compute QR factorizations.
  • Solve least-squares problems.

28.15.4 Spectral theory

  • Compute determinants.
  • Find eigenvalues and eigenspaces.
  • Diagonalize matrices when possible.
  • Use diagonalization to compute powers.
  • Analyze discrete dynamical systems.
  • Work with orthogonal and symmetric matrices.
  • Apply the spectral theorem.

28.15.5 Optimization and data

  • Classify quadratic forms.
  • Find minima of positive definite quadratic functions.
  • Compute singular values and SVDs.
  • Use SVD for rank and low-rank approximation.
  • Understand PCA through covariance and SVD.
  • Interpret data matrices, embeddings, and linear models.

28.16 Closing advice

Cumulative review should not be only about memorizing formulas.

For each problem, ask:

  1. What are the objects?
  2. What spaces do they live in?
  3. What is the matrix doing?
  4. What subspaces are involved?
  5. What geometry is present?
  6. What computation reveals the structure?
  7. What does the answer mean?

Linear algebra becomes powerful when computation, geometry, and interpretation work together.