Classes 17 · 19 · Test 3 · Final Q9–Q10

Principal Components
Analysis

PCA finds the orthogonal directions of greatest variance in a dataset. By projecting onto the first few principal components, we reduce dimensionality while preserving most information. It is exactly the SVD of the zero-mean data matrix.

Step 1: Zero-Mean Data

Computing the Zero-Mean Matrix $M$

$$\bar{A} \stackrel{\text{def}}{=} \frac{1}{m}\vec{1}^T A \qquad M \stackrel{\text{def}}{=} A - \vec{1}\bar{A}$$

Subtract the column mean from each column. $\bar{A}$ is a row vector of column means. In MATLAB: M = A - mean(A). This centers each variable (column) at zero.

CRITICAL: PCA is performed on M (zero-mean data), not on A. Using A directly gives wrong results.

Step 2: Covariance and Scatter Matrices

Sample Covariance Matrix

$$B = \frac{M^TM}{m-1} \in \mathbb{R}^{n \times n}$$

$B$ is symmetric positive semidefinite. It is PD if $M$ is full rank. Its eigenvalues are the variances of the data along the principal directions.

Scatter Matrix

$$S = M^TM = (m-1)B$$

The scatter matrix is just $(m-1)$ times the covariance matrix. The eigenvectors (loading vectors) are the same. The scatter matrix has the same PCA structure but is simpler to work with computationally.

Step 3: PCA via SVD

Connecting SVD to PCA

SVD of M: M = UΣVᵀ Eigendecomposition of scatter: S = MᵀM = VΣᵀΣVᵀ = VΣ²Vᵀ → Columns of V = loading vectors (eigenvectors of S and B) → Eigenvalues of B: λⱼ = σⱼ²/(m−1) → jth score vector: ⃗zⱼ = M⃗vⱼ = σⱼ⃗uⱼ

The columns of $V$ from the SVD of $M$ are the PCA loading vectors (principal components). The columns of $U$, scaled by $\sigma_j$, give the score vectors.

PCA Score Vectors

$$\vec{z}_j = M\vec{v}_j = \sigma_j\vec{u}_j$$

The $j$-th score vector projects the zero-mean data onto the $j$-th loading vector. $\vec{z}_1$ captures the most variance, $\vec{z}_2$ the second most, etc. To find the first score: multiply the first left singular vector of $M$ by $\sigma_1$.

Step 4: Explained Variance

Explained Variance Formula

$$q_p = \frac{\sum_{j=1}^p \lambda_j}{\sum_{j=1}^n \lambda_j} = \frac{\sum_{j=1}^p \sigma_j^2}{\sum_{j=1}^n \sigma_j^2}$$

The proportion of total variance explained by the first $p$ components. Find the smallest $p$ such that $q_p \geq \theta$ (e.g. 85% or 95%). The $\sigma_j^2$ values (not $\sigma_j$) are used — this is a common mistake.

Scree Plot

Scree plot: plot σⱼ or λⱼ vs. index j Look for an "elbow" — where the curve flattens. The number of components before the elbow is a good choice for p.

The scree plot is a visual tool for choosing the number of principal components. The cumulative explained variance plot ($q_p$ vs. $p$) directly shows you when you've exceeded a threshold $\theta$.

Dimensionality Reduction

Three Equivalent Methods

1. Compute first p PCA score vectors Zₚ = [⃗z₁ ··· ⃗zₚ] 2. Compute first p eigenvectors Vₚ of scatter S, then Zₚ = MVₚ 3. Compute SVD of M = UΣVᵀ, use first p left singular vectors Uₚ and Σₚ

All three methods give the same result. The data $A$ (in $n$ dimensions) is reduced to $Z_p$ (in $p$ dimensions) where $p \ll n$.

Practice Problems

W26 Test 2 · Q1 Find standardized values $Z_1$ for $A_1 = \begin{bmatrix}4&14\\2&4\\4&6\\8&6\\12&10\end{bmatrix}$. +
Worked Answer
Column 1: mean $= (4+2+4+8+12)/5 = 6$. Zero-mean: $[-2,-4,-2,2,6]$. Variance $= (4+16+4+4+36)/4 = 64/4 = 16$. Std $= 4$.
Standardized col 1: $[-0.5, -1.0, -0.5, 0.5, 1.5]$

Column 2: mean $= (14+4+6+6+10)/5 = 8$. Zero-mean: $[6,-4,-2,-2,2]$. Variance $= (36+16+4+4+4)/4 = 64/4 = 16$. Std $= 4$.
Standardized col 2: $[1.5, -1.0, -0.5, -0.5, 0.5]$

$Z_1 = \begin{bmatrix}-0.5&1.5\\-1.0&-1.0\\-0.5&-0.5\\0.5&-0.5\\1.5&0.5\end{bmatrix}$
W26 Test 3 · Q4 $\sigma_1 \approx 5.196$, $U_4$ first col $= [0.136,0.136,-0.816,0.544]^T$. Find first PCA score $\vec{z}_1$. +
Worked Answer
$\vec{z}_1 = \sigma_1\vec{u}_1 = 5.196 \times [0.136, 0.136, -0.816, 0.544]^T$

$\approx [0.707, 0.707, -4.243, 2.828]^T$
2025 Final · Q9 $M_9$ has SVD with $\sigma_1=4$, $\sigma_2=2$. $U_9$ first col $\approx [0.00, 0.71, 0.00, -0.71]^T$. Find first PCA score. +
Worked Answer
$\vec{z}_1 = \sigma_1\vec{u}_1 = 4 \times [0.00, 0.71, 0.00, -0.71]^T \approx [0.00, 2.83, 0.00, -2.83]^T$

Answer: (d)
2025 Final · Q10 $\sigma_1 \approx 31.85$, $\sigma_2 \approx 11.02$, $\sigma_3 \approx 2.80$. For $\theta = 85\%$ explained variance, what is the effective dimension? +
Worked Answer
$\lambda_1 = 31.85^2 \approx 1014.4$, $\lambda_2 = 11.02^2 \approx 121.4$, $\lambda_3 = 2.80^2 \approx 7.8$. Total $\approx 1143.6$.

$p=1$: $1014.4/1143.6 \approx 88.7\% \geq 85\%$ ✓

Effective dimension = 1 (Answer: a)