25 Formula Sheet
This formula sheet summarizes the main definitions, identities, algorithms, and computational patterns used throughout Computational Linear Algebra in \(\\mathbb{R}^n\).
It is intended as a quick reference. For meaning, geometry, derivations, and examples, see the corresponding chapters.
25.1 1. Core notation
| Symbol | Meaning |
|---|---|
| \(\mathbb{R}^n\) | real \(n\)-dimensional coordinate space |
| \(A\in\mathbb{R}^{m\times n}\) | matrix with \(m\) rows and \(n\) columns |
| \(x\in\mathbb{R}^n\) | column vector with \(n\) entries |
| \(A^T\) | transpose of \(A\) |
| \(I_n\) | \(n\times n\) identity matrix |
| \(0\) | zero vector or zero matrix, depending on context |
| \(\operatorname{Col}(A)\) | column space of \(A\) |
| \(\operatorname{Row}(A)\) | row space of \(A\) |
| \(\operatorname{Null}(A)\) | null space of \(A\) |
| \(\operatorname{rank}(A)\) | dimension of \(\operatorname{Col}(A)\) |
| \(\det(A)\) | determinant of a square matrix |
| \(\operatorname{tr}(A)\) | trace of a square matrix |
| \(\|x\|\) | Euclidean norm of \(x\) |
| \(x\cdot y\) | dot product of \(x\) and \(y\) |
| \(A^{-1}\) | inverse of an invertible square matrix |
| \(A^+\) | Moore–Penrose pseudoinverse |
25.2 2. Linear systems
A linear system can be written as
\[ Ax=b. \]
If
\[ A= \begin{bmatrix} |&|&&|\\ a_1&a_2&\cdots&a_n\\ |&|&&| \end{bmatrix}, \]
then
\[ Ax=x_1a_1+x_2a_2+\cdots+x_na_n. \]
Thus \(Ax=b\) asks whether \(b\) is a linear combination of the columns of \(A\).
25.2.1 Consistency
The system
\[ Ax=b \]
is consistent if and only if
\[ b\in\operatorname{Col}(A). \]
25.2.2 Homogeneous systems
The homogeneous system
\[ Ax=0 \]
is always consistent because \(x=0\) is a solution.
The solution set is
\[ \operatorname{Null}(A). \]
25.2.3 General solution structure
If \(x_p\) is one particular solution of
\[ Ax=b, \]
then all solutions have the form
\[ x=x_p+x_h, \]
where
\[ x_h\in\operatorname{Null}(A). \]
25.3 3. Row reduction
25.3.1 Elementary row operations
- Swap two rows.
- Multiply a row by a nonzero scalar.
- Add a multiple of one row to another row.
Row operations preserve the solution set of a linear system.
25.3.2 Echelon form
A matrix is in echelon form when:
- all zero rows are below nonzero rows,
- each leading entry is to the right of the leading entry above it,
- entries below each leading entry are zero.
25.3.3 Reduced row echelon form
A matrix is in reduced row echelon form when:
- it is in echelon form,
- each pivot is \(1\),
- each pivot is the only nonzero entry in its column.
25.3.4 Pivot and free variables
In a linear system:
- pivot columns correspond to pivot variables,
- nonpivot columns correspond to free variables.
25.3.5 Rank from pivots
\[ \operatorname{rank}(A)=\text{number of pivot columns of }A. \]
25.4 4. Matrix algebra
25.4.1 Matrix-vector product
If \(A\) is \(m\times n\) and \(x\in\mathbb{R}^n\), then
\[ Ax=x_1a_1+x_2a_2+\cdots+x_na_n. \]
25.4.2 Matrix multiplication
If \(A\) is \(m\times n\) and \(B\) is \(n\times p\), then \(AB\) is \(m\times p\).
Entry formula:
\[ (AB)_{ij}=\sum_{k=1}^n a_{ik}b_{kj}. \]
Column formula:
\[ AB= \begin{bmatrix} |&|&&|\\ Ab_1&Ab_2&\cdots&Ab_p\\ |&|&&| \end{bmatrix}. \]
25.4.3 Transpose identities
\[ (A+B)^T=A^T+B^T. \]
\[ (cA)^T=cA^T. \]
\[ (AB)^T=B^TA^T. \]
\[ (A^T)^T=A. \]
25.4.4 Powers
For a square matrix \(A\),
\[ A^0=I, \]
and for \(k\ge 1\),
\[ A^k=\underbrace{AA\cdots A}_{k\text{ factors}}. \]
25.5 5. Inverses
A square matrix \(A\) is invertible if there exists a matrix \(A^{-1}\) such that
\[ AA^{-1}=A^{-1}A=I. \]
25.5.1 Inverse of a \(2\times 2\) matrix
If
\[ A= \begin{bmatrix} a&b\\ c&d \end{bmatrix}, \]
then
\[ A^{-1} = \frac{1}{ad-bc} \begin{bmatrix} d&-b\\ -c&a \end{bmatrix}, \]
provided
\[ ad-bc\ne 0. \]
25.5.2 Solving with an inverse
If \(A\) is invertible, then
\[ Ax=b \]
has the unique solution
\[ x=A^{-1}b. \]
25.5.3 Inverse of a product
If \(A\) and \(B\) are invertible, then
\[ (AB)^{-1}=B^{-1}A^{-1}. \]
25.5.4 Transpose of an inverse
If \(A\) is invertible, then
\[ (A^T)^{-1}=(A^{-1})^T. \]
25.6 6. The Invertible Matrix Theorem
For an \(n\times n\) matrix \(A\), the following are equivalent:
- \(A\) is invertible.
- \(Ax=b\) has a unique solution for every \(b\in\mathbb{R}^n\).
- \(Ax=0\) has only the trivial solution.
- The columns of \(A\) are linearly independent.
- The columns of \(A\) span \(\mathbb{R}^n\).
- \(\operatorname{rank}(A)=n\).
- \(A\) has a pivot in every column.
- \(A\) has a pivot in every row.
- \(\operatorname{Col}(A)=\mathbb{R}^n\).
- \(\operatorname{Null}(A)=\{0\}\).
- \(\det(A)\ne 0\).
25.7 7. Span, linear independence, basis, dimension
25.7.1 Span
\[ \operatorname{span}\{v_1,\ldots,v_k\} = \{c_1v_1+\cdots+c_kv_k:c_i\in\mathbb{R}\}. \]
25.7.2 Linear independence
Vectors \(v_1,\ldots,v_k\) are linearly independent if
\[ c_1v_1+\cdots+c_kv_k=0 \]
implies
\[ c_1=\cdots=c_k=0. \]
They are linearly dependent if there is a nontrivial solution.
25.7.3 Basis
A set \(\{v_1,\ldots,v_k\}\) is a basis for a subspace \(W\) if:
- it spans \(W\),
- it is linearly independent.
25.7.4 Dimension
The dimension of \(W\) is the number of vectors in any basis for \(W\).
\[ \dim(W)=\text{number of basis vectors}. \]
25.8 8. Fundamental subspaces
For \(A\in\mathbb{R}^{m\times n}\):
25.8.1 Column space
\[ \operatorname{Col}(A)=\{Ax:x\in\mathbb{R}^n\}\subseteq\mathbb{R}^m. \]
25.8.2 Null space
\[ \operatorname{Null}(A)=\{x\in\mathbb{R}^n:Ax=0\}\subseteq\mathbb{R}^n. \]
25.8.3 Row space
\[ \operatorname{Row}(A)=\operatorname{Col}(A^T)\subseteq\mathbb{R}^n. \]
25.8.4 Left null space
\[ \operatorname{Null}(A^T)=\{y\in\mathbb{R}^m:A^Ty=0\}\subseteq\mathbb{R}^m. \]
25.8.5 Rank-nullity theorem
If \(A\) is \(m\times n\), then
\[ \operatorname{rank}(A)+\dim\operatorname{Null}(A)=n. \]
25.8.6 Orthogonal relationships
\[ \operatorname{Row}(A)^\perp=\operatorname{Null}(A). \]
\[ \operatorname{Col}(A)^\perp=\operatorname{Null}(A^T). \]
25.9 9. Coordinates and change of basis
Let
\[ B=\{b_1,\ldots,b_n\} \]
be a basis of \(\mathbb{R}^n\).
The basis matrix is
\[ P_B= \begin{bmatrix} |&|&&|\\ b_1&b_2&\cdots&b_n\\ |&|&&| \end{bmatrix}. \]
25.9.1 Coordinate vector
If
\[ x=c_1b_1+\cdots+c_nb_n, \]
then
\[ [x]_B= \begin{bmatrix} c_1\\ \vdots\\ c_n \end{bmatrix}. \]
25.9.2 From basis coordinates to standard coordinates
\[ x=P_B[x]_B. \]
25.9.3 From standard coordinates to basis coordinates
\[ [x]_B=P_B^{-1}x. \]
25.9.4 Change of basis from \(B\) to \(C\)
\[ [x]_C=P_C^{-1}P_B[x]_B. \]
The change-of-basis matrix from \(B\) to \(C\) is
\[ P_{C\leftarrow B}=P_C^{-1}P_B. \]
25.10 10. Linear transformations
A map \(T:\mathbb{R}^n\to\mathbb{R}^m\) is linear if
\[ T(x+y)=T(x)+T(y) \]
and
\[ T(cx)=cT(x). \]
Every linear transformation \(T:\mathbb{R}^n\to\mathbb{R}^m\) has a standard matrix \(A\) such that
\[ T(x)=Ax. \]
The standard matrix is
\[ A= \begin{bmatrix} |&|&&|\\ T(e_1)&T(e_2)&\cdots&T(e_n)\\ |&|&&| \end{bmatrix}. \]
25.10.1 Kernel and image
\[ \ker(T)=\{x:T(x)=0\}. \]
\[ \operatorname{im}(T)=\{T(x):x\in\mathbb{R}^n\}. \]
For \(T(x)=Ax\),
\[ \ker(T)=\operatorname{Null}(A), \]
and
\[ \operatorname{im}(T)=\operatorname{Col}(A). \]
25.11 11. Composition and inverse transformations
If
\[ T(x)=Ax \]
and
\[ S(x)=Bx, \]
then
\[ (T\circ S)(x)=A(Bx)=(AB)x. \]
Composition corresponds to matrix multiplication.
If \(T(x)=Ax\) and \(A\) is invertible, then
\[ T^{-1}(x)=A^{-1}x. \]
25.12 12. Dot product, norm, distance, and angle
25.12.1 Dot product
\[ x\cdot y=x^Ty=\sum_{i=1}^n x_iy_i. \]
25.12.2 Norm
\[ \|x\|=\sqrt{x\cdot x}. \]
25.12.3 Distance
\[ d(x,y)=\|x-y\|. \]
25.12.4 Angle
For nonzero \(x\) and \(y\),
\[ \cos\theta=\frac{x\cdot y}{\|x\|\|y\|}. \]
25.12.5 Orthogonality
\[ x\perp y \]
if and only if
\[ x\cdot y=0. \]
25.12.6 Cauchy–Schwarz inequality
\[ |x\cdot y|\le \|x\|\|y\|. \]
25.12.7 Triangle inequality
\[ \|x+y\|\le \|x\|+\|y\|. \]
25.13 13. Orthogonal sets
A set \(\{q_1,\ldots,q_k\}\) is orthogonal if
\[ q_i\cdot q_j=0 \]
for \(i\ne j\).
It is orthonormal if, in addition,
\[ \|q_i\|=1 \]
for all \(i\).
If \(Q\) has orthonormal columns, then
\[ Q^TQ=I. \]
If \(Q\) is square with orthonormal columns, then \(Q\) is orthogonal and
\[ Q^{-1}=Q^T. \]
25.14 14. Orthogonal projection
25.14.1 Projection onto a line
If \(a\ne 0\) and \(L=\operatorname{span}\{a\}\), then
\[ \operatorname{proj}_L(b)=\frac{a\cdot b}{a\cdot a}a. \]
The error is
\[ e=b-\operatorname{proj}_L(b), \]
and
\[ e\perp L. \]
25.14.2 Projection onto a unit vector
If \(\|q\|=1\), then
\[ \operatorname{proj}_{\operatorname{span}\{q\}}(b)=(q\cdot b)q. \]
25.14.3 Projection matrix onto a line
\[ P=\frac{aa^T}{a^Ta}. \]
Then
\[ \operatorname{proj}_L(b)=Pb. \]
25.14.4 Projection onto a subspace with orthonormal basis
If \(W=\operatorname{span}\{q_1,\ldots,q_k\}\) and \(q_i\) are orthonormal, then
\[ \operatorname{proj}_W(b)=\sum_{i=1}^k (b\cdot q_i)q_i. \]
If
\[ Q= \begin{bmatrix} |&|&&|\\ q_1&q_2&\cdots&q_k\\ |&|&&| \end{bmatrix}, \]
then
\[ \operatorname{proj}_W(b)=QQ^Tb. \]
25.14.5 Projection using a full-column-rank matrix
If \(A\) has independent columns, then the projection onto \(\operatorname{Col}(A)\) is
\[ P=A(A^TA)^{-1}A^T. \]
Then
\[ \operatorname{proj}_{\operatorname{Col}(A)}(b)=Pb. \]
25.14.6 Projection matrix properties
For an orthogonal projection matrix \(P\),
\[ P^2=P, \]
and
\[ P^T=P. \]
25.15 15. Gram–Schmidt and QR factorization
25.15.1 Gram–Schmidt
Given independent vectors \(a_1,\ldots,a_k\):
\[ u_1=a_1, \qquad q_1=\frac{u_1}{\|u_1\|}. \]
For \(j=2,\ldots,k\),
\[ u_j=a_j-\sum_{i=1}^{j-1}(a_j\cdot q_i)q_i. \]
Then
\[ q_j=\frac{u_j}{\|u_j\|}. \]
The vectors \(q_1,\ldots,q_k\) are orthonormal.
25.15.2 QR factorization
If \(A\) has independent columns, then
\[ A=QR, \]
where \(Q\) has orthonormal columns and \(R\) is upper triangular.
25.15.3 Projection using QR
If
\[ A=QR, \]
then
\[ \operatorname{proj}_{\operatorname{Col}(A)}(b)=QQ^Tb. \]
25.16 16. Least squares
The least-squares problem is
\[ \min_x \|b-Ax\|. \]
The fitted vector is
\[ \hat b=A\hat x. \]
The residual is
\[ r=b-A\hat x. \]
At the least-squares solution,
\[ r\perp \operatorname{Col}(A). \]
Thus
\[ A^Tr=0. \]
25.16.1 Normal equations
\[ A^T(b-A\hat x)=0. \]
Equivalently,
\[ A^TA\hat x=A^Tb. \]
If \(A\) has independent columns, then
\[ \hat x=(A^TA)^{-1}A^Tb. \]
25.16.2 Least squares using QR
If
\[ A=QR, \]
then solve
\[ R\hat x=Q^Tb. \]
25.16.3 Hat matrix
If \(A\) has independent columns, then
\[ P=A(A^TA)^{-1}A^T \]
and
\[ \hat b=Pb. \]
The matrix \(P\) is the hat matrix.
25.17 17. Determinants
25.17.1 \(2\times 2\) determinant
\[ \det \begin{bmatrix} a&b\\ c&d \end{bmatrix} = ad-bc. \]
25.17.2 \(3\times 3\) determinant
\[ \det \begin{bmatrix} a&b&c\\ d&e&f\\ g&h&i \end{bmatrix} = a(ei-fh)-b(di-fg)+c(dh-eg). \]
25.17.3 Area and volume
For a square matrix \(A\),
\[ |\det(A)| \]
is the volume scaling factor.
In \(\mathbb{R}^2\), it gives area scaling.
In \(\mathbb{R}^3\), it gives volume scaling.
25.17.4 Determinant and invertibility
\[ A \text{ is invertible} \quad\Longleftrightarrow\quad \det(A)\ne 0. \]
25.17.5 Determinant properties
\[ \det(AB)=\det(A)\det(B). \]
\[ \det(A^T)=\det(A). \]
If \(A\) is invertible,
\[ \det(A^{-1})=\frac{1}{\det(A)}. \]
For triangular \(A\),
\[ \det(A)=a_{11}a_{22}\cdots a_{nn}. \]
25.17.6 Row operation effects
- Swapping two rows changes the sign of the determinant.
- Multiplying one row by \(c\) multiplies the determinant by \(c\).
- Adding a multiple of one row to another row does not change the determinant.
25.18 18. Eigenvalues and eigenvectors
A nonzero vector \(v\) is an eigenvector of \(A\) with eigenvalue \(\lambda\) if
\[ Av=\lambda v. \]
Equivalently,
\[ (A-\lambda I)v=0. \]
25.18.1 Characteristic equation
Eigenvalues solve
\[ \det(A-\lambda I)=0. \]
25.18.2 Eigenspace
The eigenspace for eigenvalue \(\lambda\) is
\[ E_\lambda=\operatorname{Null}(A-\lambda I). \]
25.18.3 Triangular matrices
If \(A\) is triangular, its eigenvalues are its diagonal entries.
25.18.4 Trace and determinant
For a \(2\times 2\) matrix with eigenvalues \(\lambda_1,\lambda_2\),
\[ \lambda_1+\lambda_2=\operatorname{tr}(A), \]
and
\[ \lambda_1\lambda_2=\det(A). \]
More generally, the determinant is the product of eigenvalues counted with algebraic multiplicity.
25.18.5 Powers on eigenvectors
If
\[ Av=\lambda v, \]
then
\[ A^kv=\lambda^k v. \]
25.19 19. Diagonalization and dynamics
A matrix \(A\) is diagonalizable if
\[ A=PDP^{-1}, \]
where \(D\) is diagonal and \(P\) is invertible.
If
\[ P= \begin{bmatrix} v_1&\cdots&v_n \end{bmatrix}, \]
and
\[ D=\operatorname{diag}(\lambda_1,\ldots,\lambda_n), \]
then the columns \(v_i\) are eigenvectors and
\[ Av_i=\lambda_i v_i. \]
25.19.1 Coordinate form
\[ P^{-1}AP=D. \]
25.19.2 Powers
If
\[ A=PDP^{-1}, \]
then
\[ A^k=PD^kP^{-1}. \]
25.19.3 Discrete dynamics
For
\[ x_{k+1}=Ax_k, \]
we have
\[ x_k=A^kx_0. \]
If \(A\) is diagonalizable and
\[ x_0=c_1v_1+\cdots+c_nv_n, \]
then
\[ x_k=c_1\lambda_1^kv_1+\cdots+c_n\lambda_n^kv_n. \]
25.19.4 Stability rule
For an eigenvalue \(\lambda\):
- \(|\lambda|<1\): decay,
- \(|\lambda|>1\): growth,
- \(\lambda=1\): steady direction,
- \(\lambda=-1\): sign alternation,
- complex eigenvalues: oscillation or rotation.
25.20 20. Orthogonal matrices
A square matrix \(Q\) is orthogonal if
\[ Q^TQ=I. \]
Equivalently,
\[ Q^{-1}=Q^T. \]
25.20.1 Geometry preservation
If \(Q\) is orthogonal, then
\[ \|Qx\|=\|x\|. \]
Also,
\[ (Qx)\cdot(Qy)=x\cdot y. \]
Thus orthogonal matrices preserve lengths, distances, angles, and dot products.
25.20.2 Determinant
If \(Q\) is orthogonal, then
\[ \det(Q)=\pm 1. \]
25.21 21. Symmetric matrices and the spectral theorem
A square matrix \(A\) is symmetric if
\[ A^T=A. \]
25.21.1 Orthogonality of eigenvectors
For a symmetric matrix, eigenvectors corresponding to distinct eigenvalues are orthogonal.
25.21.2 Spectral theorem
If \(A\) is real and symmetric, then there exists an orthogonal matrix \(Q\) and a real diagonal matrix \(D\) such that
\[ A=QDQ^T. \]
Equivalently,
\[ Q^TAQ=D. \]
The columns of \(Q\) are orthonormal eigenvectors of \(A\).
25.21.3 Spectral decomposition
If
\[ A=QDQ^T, \]
with eigenvalues \(\lambda_i\) and orthonormal eigenvectors \(q_i\), then
\[ A=\sum_{i=1}^n \lambda_i q_iq_i^T. \]
25.22 22. Quadratic forms
A quadratic form is
\[ q(x)=x^TAx, \]
where \(A\) is symmetric.
For
\[ A= \begin{bmatrix} a&b\\ b&c \end{bmatrix}, \]
we have
\[ x^TAx=ax_1^2+2bx_1x_2+cx_2^2. \]
25.22.1 Principal axes
If
\[ A=QDQ^T \]
and
\[ y=Q^Tx, \]
then
\[ x^TAx=y^TDy=\lambda_1y_1^2+\cdots+\lambda_ny_n^2. \]
25.22.2 Definiteness by eigenvalues
For symmetric \(A\):
- all \(\lambda_i>0\): positive definite,
- all \(\lambda_i\ge 0\): positive semidefinite,
- all \(\lambda_i<0\): negative definite,
- all \(\lambda_i\le 0\): negative semidefinite,
- mixed positive and negative eigenvalues: indefinite.
25.22.3 Quadratic optimization
For
\[ f(x)=\frac12x^TAx-b^Tx+c, \]
with \(A\) symmetric,
\[ \nabla f(x)=Ax-b. \]
If \(A\) is positive definite, the unique minimizer satisfies
\[ Ax=b. \]
25.22.4 Least squares objective
\[ f(x)=\frac12\|Ax-b\|^2. \]
Gradient:
\[ \nabla f(x)=A^T(Ax-b). \]
Hessian:
\[ \nabla^2f(x)=A^TA. \]
Setting the gradient to zero gives
\[ A^TAx=A^Tb. \]
25.23 23. Singular value decomposition
For any real \(m\times n\) matrix \(A\),
\[ A=U\Sigma V^T. \]
Here:
- \(U\) is orthogonal or has orthonormal columns,
- \(V\) is orthogonal or has orthonormal columns,
- \(\Sigma\) is diagonal-shaped,
- singular values satisfy \(\sigma_i\ge 0\).
25.23.1 Singular vector relation
\[ Av_i=\sigma_i u_i. \]
25.23.2 Singular values from \(A^TA\)
If
\[ A^TAv_i=\lambda_i v_i, \]
then
\[ \sigma_i=\sqrt{\lambda_i}. \]
The \(v_i\) are right singular vectors.
For \(\sigma_i>0\),
\[ u_i=\frac{Av_i}{\sigma_i}. \]
25.23.3 Rank from singular values
\[ \operatorname{rank}(A)=\#\{i:\sigma_i>0\}. \]
25.23.4 Condition number
For invertible square \(A\),
\[ \kappa(A)=\frac{\sigma_{\max}}{\sigma_{\min}}. \]
25.23.5 Outer product form
If \(\operatorname{rank}(A)=r\), then
\[ A=\sum_{i=1}^r \sigma_i u_iv_i^T. \]
25.23.6 Rank-\(k\) truncated SVD
\[ A_k=\sum_{i=1}^k \sigma_i u_iv_i^T. \]
This is the best rank-\(k\) approximation to \(A\) in the Frobenius norm and spectral norm.
25.23.7 Frobenius norm and singular values
\[ \|A\|_F^2=\sum_i\sigma_i^2. \]
25.23.8 Pseudoinverse
If
\[ A=U\Sigma V^T, \]
then
\[ A^+=V\Sigma^+U^T, \]
where nonzero singular values are replaced by reciprocals.
The minimum-norm least-squares solution is
\[ \hat x=A^+b. \]
25.24 24. Data, PCA, and AI formulas
25.24.1 Data matrix
For \(m\) observations and \(n\) features,
\[ X\in\mathbb{R}^{m\times n}. \]
Rows are observations.
Columns are features.
25.24.2 Centered data
If \(\mu\) is the vector of column means,
\[ X_c=X-\mathbf{1}\mu^T. \]
25.24.3 Covariance matrix
For centered data,
\[ C=\frac{1}{m-1}X_c^TX_c. \]
25.24.4 PCA by covariance
If
\[ Cv_i=\lambda_i v_i, \]
then \(v_i\) is a principal component direction and \(\lambda_i\) is the variance in that direction.
25.24.5 PCA by SVD
If
\[ X_c=U\Sigma V^T, \]
then columns of \(V\) are principal component directions.
Explained variances are
\[ \frac{\sigma_i^2}{m-1}. \]
25.24.6 Projection to \(k\) principal components
If \(V_k\) contains the first \(k\) principal directions, then
\[ Z=X_cV_k. \]
Reconstruction:
\[ \hat X=ZV_k^T. \]
25.24.7 Cosine similarity
For nonzero vectors \(x\) and \(y\),
\[ \operatorname{cosine}(x,y) = \frac{x^Ty}{\|x\|\|y\|}. \]
25.24.8 Linear prediction
\[ \hat y=w^Tx+b. \]
With an intercept column included,
\[ \hat y=\beta^T\tilde x. \]
25.24.9 Neural-network affine layer
\[ z=Wx+b. \]
With activation,
\[ h=\phi(z). \]
25.24.10 Attention score matrix
For query, key, and value matrices,
\[ Q=XW_Q, \]
\[ K=XW_K, \]
\[ V=XW_V. \]
Similarity scores:
\[ S=QK^T. \]
25.24.11 Graph Laplacian
For an undirected graph with adjacency matrix \(A\) and degree matrix \(D\),
\[ L=D-A. \]
The graph Laplacian is symmetric positive semidefinite.
25.25 25. Python quick reference
25.25.1 NumPy imports
import numpy as np
import matplotlib.pyplot as plt25.25.2 Matrix and vector creation
A = np.array([[1, 2],
[3, 4]], dtype=float)
x = np.array([1, 2], dtype=float)25.25.3 Matrix multiplication
A @ x
A @ B25.25.4 Transpose
A.T25.25.5 Solve a square system
x = np.linalg.solve(A, b)25.25.6 Row reduction with SymPy
import sympy as sp
M = sp.Matrix([[1, 2, 3],
[4, 5, 6]])
M.rref()25.25.7 Rank
np.linalg.matrix_rank(A)25.25.8 Inverse
np.linalg.inv(A)25.25.9 Determinant
np.linalg.det(A)25.25.10 Dot product and norm
np.dot(x, y)
np.linalg.norm(x)25.25.11 Least squares
x_hat = np.linalg.lstsq(A, b, rcond=None)[0]25.25.12 QR factorization
Q, R = np.linalg.qr(A, mode="reduced")25.25.13 Eigenvalues and eigenvectors
eigvals, eigvecs = np.linalg.eig(A)For symmetric matrices:
eigvals, Q = np.linalg.eigh(A)25.25.14 Singular value decomposition
U, s, Vt = np.linalg.svd(A, full_matrices=False)Reconstruction:
A_reconstructed = (U * s) @ Vt25.25.15 Pseudoinverse
A_pinv = np.linalg.pinv(A)
x_hat = A_pinv @ b25.25.16 Condition number
np.linalg.cond(A)25.26 26. Concept map
| Concept | Formula | Meaning |
|---|---|---|
| Linear system | \(Ax=b\) | constraints and column combinations |
| Homogeneous system | \(Ax=0\) | null space |
| Matrix transformation | \(T(x)=Ax\) | linear map |
| Projection | \(\operatorname{proj}_W(b)\) | closest vector in \(W\) |
| Normal equations | \(A^TA\hat x=A^Tb\) | least-squares condition |
| Determinant | \(\det(A)\) | signed volume scaling |
| Eigenvalue equation | \(Av=\lambda v\) | invariant direction |
| Diagonalization | \(A=PDP^{-1}\) | eigenvector coordinates |
| Orthogonal matrix | \(Q^TQ=I\) | length and angle preservation |
| Spectral theorem | \(A=QDQ^T\) | symmetric matrix structure |
| Quadratic form | \(x^TAx\) | curvature and energy |
| SVD | \(A=U\Sigma V^T\) | singular directions and low rank |
| PCA | \(X_c=U\Sigma V^T\) | principal directions of data |
25.27 27. Most important ideas to remember
- \(Ax=b\) asks whether \(b\) lies in the column space of \(A\).
- Row reduction reveals pivots, rank, consistency, and free variables.
- A basis gives coordinates.
- A matrix is a linear transformation.
- Orthogonality gives length, distance, angle, and projection.
- Projection is best approximation.
- Least squares is projection onto a column space.
- Determinant measures signed volume scaling.
- Eigenvectors are directions preserved by a matrix.
- Diagonalization makes powers and dynamics simple.
- Orthogonal matrices preserve geometry.
- Symmetric matrices have orthonormal eigenvector bases.
- Quadratic forms describe curvature and optimization.
- The SVD works for every matrix and reveals low-rank structure.
- Data science and AI use linear algebra as a computational language.