Lab 14: Singular Value Decomposition

Purpose

This lab helps you practice the singular value decomposition computationally and visually. You will compute singular values, reconstruct matrices from SVD, build low-rank approximations, use the pseudoinverse for least squares, and interpret SVD geometrically.

Python practice notebook

Download and complete the Python practice notebook:

The notebook includes worked solutions and similar practice questions. Use it for independent study.

Interactive lab

Open the interactive HTML lab:

Open Lab 14 interactive page

Use the interactive page to explore:

  1. singular values of a \(2\times2\) matrix;
  2. the image of the unit circle;
  3. rank-\(k\) approximation;
  4. pseudoinverse least-squares solutions;
  5. PCA directions for a two-dimensional data cloud.

Background formulas

SVD

For a real matrix \(A\in\mathbb R^{m\times n}\),

\[ A=U\Sigma V^T, \]

where \(U\) and \(V\) are orthogonal and \(\Sigma\) is rectangular diagonal with nonnegative entries.

Singular values

The singular values are

\[ \sigma_i=\sqrt{\lambda_i(A^TA)}. \]

Rank from SVD

\[ \operatorname{rank}(A)=\#\{i:\sigma_i>0\}. \]

Spectral norm

\[ \|A\|_2=\sigma_1. \]

Rank-\(k\) approximation

\[ A_k=\sum_{i=1}^k \sigma_i u_i v_i^T. \]

Pseudoinverse

If \(A=U\Sigma V^T\), then

\[ A^+=V\Sigma^+U^T. \]

Independent-study tasks with solutions

TipTask 1: Compute singular values

Let

\[ A=\begin{bmatrix}3&0\\0&2\end{bmatrix}. \]

Find the singular values.

Solution

Since

\[ A^TA=\begin{bmatrix}9&0\\0&4\end{bmatrix}, \]

the singular values are \(3\) and \(2\).

TipTask 2: Rank from singular values

Suppose the singular values of \(A\) are

\[ 9,4,0,0. \]

Find \(\operatorname{rank}(A)\) and \(\|A\|_2\).

Solution

The rank is the number of nonzero singular values, so \(\operatorname{rank}(A)=2\). The spectral norm is the largest singular value, so \(\|A\|_2=9\).

TipTask 3: Best rank-one approximation

Suppose \(A\) has singular values \(10,3,1\). What is the spectral norm error of the best rank-one approximation?

Solution

By the Eckart–Young theorem,

\[ \|A-A_1\|_2=\sigma_2=3. \]

TipSimilar practice

Suppose \(A\) has singular values \(12,5,2,0.5\). What is the spectral norm error of the best rank-two approximation?

Answer

The error is the next singular value:

\[ \|A-A_2\|_2=\sigma_3=2. \]

Python warm-up

Code 
import numpy as np

A = np.array([[1, 4],
              [2, 2],
              [2,-4]], dtype=float)

U, s, Vt = np.linalg.svd(A, full_matrices=True)
print("singular values:", s)
print("rank:", np.sum(s > 1e-10))
print("spectral norm:", s[0])
singular values: [6. 3.]
rank: 2
spectral norm: 6.0

Reflection questions

  1. Why do singular values never become negative?
  2. Why is SVD more general than diagonalization?
  3. What geometric information is stored in the left singular vectors?
  4. What information is lost when small singular values are discarded?
  5. Why does PCA use the right singular vectors of a centered data matrix?

Submission suggestion

For an independent-study submission, include:

  1. screenshots or exported results from the interactive lab;
  2. completed Python notebook outputs;
  3. answers to the reflection questions;
  4. one short paragraph explaining how SVD connects geometry and data compression.