Classes 14 · 16 · Test 3 · Final Q6–Q8

Singular Value
Decomposition

The SVD works for any real matrix — square or rectangular, full rank or singular, diagonalizable or not. It is the generalization of eigendecomposition and the foundation of PCA, data compression, and numerical linear algebra.

Derivation: From Eigenvalues to Singular Values

Transpose Products

For A ∈ ℝ^{m×n}: AᵀA ∈ ℝ^{n×n} is symmetric positive semidefinite AAᵀ ∈ ℝ^{m×m} is symmetric positive semidefinite AᵀA = VΛᵥVᵀ → columns of V are eigenvectors AAᵀ = UΛᵤUᵀ → columns of U are eigenvectors Key result: the r nonzero eigenvalues of AᵀA and AAᵀ are IDENTICAL.

Proof sketch: if $A\vec{v} = \sigma\vec{u}$, then $AA^T(A\vec{v}) = A(A^TA\vec{v}) = A\lambda\vec{v} = \lambda A\vec{v}$, so $\lambda$ is also an eigenvalue of $AA^T$.

The SVD

SVD Factorization

$$A = U\Sigma V^T \qquad A \in \mathbb{R}^{m \times n}$$
U ∈ ℝ^{m×m}: orthogonal (UᵀU = UUᵀ = I), columns = left singular vectors Σ ∈ ℝ^{m×n}: "diagonal" (σᵢⱼ = 0 for i≠j), diagonal entries σⱼ ≥ 0 V ∈ ℝ^{n×n}: orthogonal (VᵀV = VVᵀ = I), columns = right singular vectors σ₁ ≥ σ₂ ≥ ··· ≥ σᵣ > 0 = σᵣ₊₁ = ··· where r = rank(A)

The singular values $\sigma_j = \sqrt{\lambda_j}$ are the positive square roots of the nonzero eigenvalues of $A^TA$ (equivalently $AA^T$). MATLAB: [U,S,V] = svd(A).

Matrix Spaces from the SVD

Four Fundamental Spaces

Column space of A: first r columns of U (r = rank) Left null space of A: last (m−r) columns of U Row space of A: first r columns of V Null space of A: last (n−r) columns of V ← zero singular values Dimensions: col(A) = r, null(A) = n−r, row(A) = r, left-null(A) = m−r

The null space comes from the last columns of $V$ — those corresponding to zero singular values. If $\sigma_j = 0$, then $V$'s $j$-th column is a basis vector for $\mathcal{N}(A)$.

Eckart-Young: Low-Rank Approximation

SVD as a Series

$$A = \sum_{j=1}^r \sigma_j \vec{u}_j \vec{v}_j^T = \sigma_1\vec{u}_1\vec{v}_1^T + \sigma_2\vec{u}_2\vec{v}_2^T + \cdots + \sigma_r\vec{u}_r\vec{v}_r^T$$

Each term $\sigma_j\vec{u}_j\vec{v}_j^T$ is a rank-1 matrix. The Eckart-Young theorem says the optimal rank-$p$ approximation to $A$ is the first $p$ terms of this series — truncating to the $p$ largest singular values.

Key SVD Properties

Properties Checklist

Practice Problems

W26 Test 3 · Q1 $A_1 = \begin{bmatrix}0&2\\2&0\\4&4\end{bmatrix}$, $A_1^TA_1 = \begin{bmatrix}20&16\\16&20\end{bmatrix}$. Find the singular values. +
Worked Answer
Eigenvalues of $A_1^TA_1$: $\lambda^2 - 40\lambda + (400-256) = 0 \Rightarrow \lambda^2 - 40\lambda + 144 = 0$
$\lambda = \frac{40 \pm \sqrt{1600-576}}{2} = \frac{40 \pm 32}{2}$, so $\lambda_1 = 36$, $\lambda_2 = 4$.

$\sigma_1 = \sqrt{36} = 6$, $\sigma_2 = \sqrt{4} = 2$
W26 Test 3 · Q3 $\sigma_1 \approx 33.61$, $\sigma_2 \approx 11.04$, $\sigma_3 \approx 2.84$. How many components explain ≥96% of variance? +
Worked Answer
$\lambda_j = \sigma_j^2$: $\lambda_1 \approx 1129.6$, $\lambda_2 \approx 121.9$, $\lambda_3 \approx 8.1$. Total $\approx 1259.6$.
$p=1$: $1129.6/1259.6 \approx 89.7\%$ — not enough.
$p=2$: $(1129.6+121.9)/1259.6 \approx 99.4\%$ — exceeds 96%.

$p = 2$ components
W26 Test 3 · Q5 $A_5 = \text{diag}(1,2,3,0)_{4\times 3}$. Which of P5.1 (eigenvalues of $A^TA$ and $AA^T$ identical), P5.2 (nonzero singular values are 3, 2, 1), P5.3 (removing last row leaves nonzero singular values unchanged) are true? +
Worked Answer
P5.1 FALSE: $A_5^TA_5 \in \mathbb{R}^{3\times 3}$ has eigenvalues $\{1,4,9\}$. $A_5A_5^T \in \mathbb{R}^{4\times 4}$ has eigenvalues $\{1,4,9,0\}$. Not identical — $AA^T$ has an extra zero.

P5.2 TRUE: Nonzero singular values = $\sqrt{9}=3$, $\sqrt{4}=2$, $\sqrt{1}=1$. ✓

P5.3 TRUE: The last row of $A_5$ is all zeros. Removing it doesn't change the singular values. ✓

Answer: P5.2 and P5.3, not P5.1
W26 Test 3 · Q6 $A_6$ has singular values $12.4, 11.687, 5.713, 0$. Characterize the definiteness of $A_6$. +
Worked Answer
Insufficient information.

Singular values are eigenvalues of $A_6^TA_6$ (always non-negative). They don't tell us the eigenvalues of $A_6$ itself, which could be positive or negative. $\sigma_4 = 0$ means $A_6$ is singular, but that alone doesn't determine PSD, PD, or indefinite status.
2025 Final · Q6 $A_6$ has a 2D null space. $V_6$ is $4\times 4$. Which matrix $N$ derived from $V_6$ is a basis for $\text{null}(A_6)$? +
Worked Answer
Null space basis = right singular vectors corresponding to $\sigma = 0$. Since the null space is 2D (2 zero singular values), we want the last 2 columns of $V_6$ (columns 3 and 4 of the $4\times 4$ matrix $V_6$). Answer: (c).