1. Quadratic objective
Study $f(x)=\frac12 x^TQx-b^Tx$ with $Q=Q^T$.
Matrix Q
Vector b
Starting point x
2. Gradient descent path
Contours show the quadratic objective. Orange points show gradient descent iterates.
3. Gradient check
For $f(x)=\frac12x^TQx-b^Tx$, compare the analytic gradient $Qx-b$ with centered finite differences.
4. Newton step
For a quadratic objective, Newton's method uses $x_{new}=x-Q^{-1}(Qx-b)$.
5. Least squares gradient
Let $f(x)=\frac12\|Ax-y\|_2^2$. The gradient is $\nabla f(x)=A^T(Ax-y)$.
Matrix A
Vector y
Vector x
6. Matrix least squares
For $f(X)=\frac12\|MX-C\|_F^2$, $\nabla_Xf=M^T(MX-C)$.
M
X
C
7. Study tasks with answers
Task A. Find $\nabla(x^TAx)$.
Answer: $(A+A^T)x$. If $A$ is symmetric, this is $2Ax$.
Answer: $(A+A^T)x$. If $A$ is symmetric, this is $2Ax$.
Task B. Find the Hessian of $\frac12\|Ax-b\|^2$.
Answer: $A^TA$.
Answer: $A^TA$.
Task C. Find $\nabla_X\frac12\|X-C\|_F^2$.
Answer: $X-C$.
Answer: $X-C$.
Similar practice. Find $\nabla_X\frac12\|AXB-C\|_F^2$.
Answer: $A^T(AXB-C)B^T$.
Answer: $A^T(AXB-C)B^T$.