1. Covariance matrix
Use a symmetric matrix $\Sigma=\begin{bmatrix}a&c\\c&b\end{bmatrix}$. It is a valid covariance matrix exactly when it is positive semidefinite.
2. Covariance ellipse and principal directions
The ellipse axes are eigenvectors of $\Sigma$. Axis lengths are proportional to $\sqrt{\lambda_i}$.
3. Directional variance
For a unit direction $u=(\cos\theta,\sin\theta)$, the variance of the projection $u^TX$ is $u^T\Sigma u$.
4. Affine transformation
Let $Y=AX+b$. The covariance transforms by $\operatorname{Cov}(Y)=A\Sigma A^T$.
5. Whitening
If $\Sigma=Q\Lambda Q^T$, then $W=\Lambda^{-1/2}Q^T$ gives $W\Sigma W^T=I$ when $\Sigma$ is positive definite.
6. High-dimensional random directions
For random unit vectors $u,v\in\mathbb R^d$, a typical inner product has size about $1/\sqrt d$.
7. Independent-study tasks with answers
Task A. For $\Sigma=\begin{bmatrix}4&3\\3&4\end{bmatrix}$, find the largest variance direction.
Answer: The largest eigenvalue is $7$, with eigenvector proportional to $(1,1)$. The maximum variance direction is $\frac1{\sqrt2}(1,1)$.
Answer: The largest eigenvalue is $7$, with eigenvector proportional to $(1,1)$. The maximum variance direction is $\frac1{\sqrt2}(1,1)$.
Task B. Explain why $\Sigma$ cannot have a negative eigenvalue.
Answer: If $\Sigma q=\lambda q$ and $\|q\|=1$, then $\lambda=q^T\Sigma q=\operatorname{Var}(q^TX)\ge0$.
Answer: If $\Sigma q=\lambda q$ and $\|q\|=1$, then $\lambda=q^T\Sigma q=\operatorname{Var}(q^TX)\ge0$.
Similar practice. Try $\Sigma=\begin{bmatrix}5&2\\2&5\end{bmatrix}$. What are the eigenvalues?
Answer: $7$ and $3$.
Answer: $7$ and $3$.