Big idea
A symmetric matrix creates an energy landscape: q(x)=xᵀAx. Eigenvectors give the main directions of the landscape. Eigenvalues tell whether those directions curve upward, downward, or remain flat.
positive definite = bowlindefinite = saddlezero eigenvalue = flat directioncondition number = landscape difficulty
1. Build a quadratic landscape
The matrix is A=[[a,b],[b,c]], so the quadratic form is q(x,y)=ax²+2bxy+cy².
2. Gradient descent on a quadratic bowl
3. Least squares as an energy landscape
Each point in parameter space represents a line y=w₀+w₁x. The loss L(w)=||Xw-y||² is a quadratic landscape.
Reflection: Why is the loss landscape a bowl when the columns of X are independent?
4. High-dimensional intuition
In high dimensions, we cannot draw the surface, but eigenvalues still tell us the shape. A large condition number means a narrow valley and slow gradient descent.
Student reflection prompts
- How do eigenvalues classify the shape of an energy landscape?
- Why does a cross term rotate the bowl?
- What does a flat direction mean in an optimization problem?
- Why does feature scaling help gradient descent?
- How is least squares a problem of finding the bottom of a bowl?