Classes 2 · 7 · 8 · Tests 1 & Final Q1–Q2

Eigenvalues &
Eigenvectors

The eigenvalue equation describes how a matrix transforms certain special vectors — scaling them without changing their direction. These appear throughout data analysis: in PCA, graph clustering, spectral decomposition, and more.

The Eigenvalue Equation

Definition

$$A\vec{v} = \lambda\vec{v}, \quad \vec{v} \neq \vec{0}$$

$A \in \mathbb{R}^{n \times n}$ is square. $\lambda \in \mathbb{R}$ is the eigenvalue (scalar). $\vec{v} \in \mathbb{R}^n$ is the eigenvector. The eigenvector is not changed in direction by the linear transformation $A$ — only its magnitude is scaled by $\lambda$.

Characteristic Polynomial & Equation

$$F(\lambda) \stackrel{\text{def}}{=} \det(A - \lambda I) = 0$$

Rearranging the eigenvalue equation: $(A - \lambda I)\vec{v} = \vec{0}$. For non-trivial $\vec{v} \neq \vec{0}$ to exist, the matrix $A - \lambda I$ must be singular, i.e. its determinant is zero. This gives the characteristic equation. For a $2 \times 2$ matrix $A = \begin{bmatrix}a&b\\c&d\end{bmatrix}$:

$$\lambda^2 - (a+d)\lambda + (ad - bc) = 0 \quad \Longrightarrow \quad \lambda^2 - \text{tr}(A)\lambda + \det(A) = 0$$

Eigenfacts — Matrix Properties and Eigenvalues

Trace and Determinant

$$\text{tr}(A) = \sum_{i=1}^n a_{ii} = \sum_{j=1}^n \lambda_j \qquad \det(A) = \prod_{j=1}^n \lambda_j$$

These two facts are the most powerful shortcuts for finding unknown eigenvalues. If you know some eigenvalues and the trace/det, you can solve for the rest without finding the characteristic polynomial.

Properties of Related Matrices

Transpose A^T: same eigenvalues as A (different eigenvectors) Inverse A^{-1}: eigenvalues = {1/λ₁, 1/λ₂, ..., 1/λₙ} Shift A + cI: eigenvalues = {λ₁+c, λ₂+c, ..., λₙ+c} Power A^k: eigenvalues = {λ₁^k, λ₂^k, ..., λₙ^k} Similar C = P⁻¹AP: same eigenvalues as A (for invertible P)

The eigenvectors are preserved for $A^{-1}$, $A + cI$, and $A^k$ — only the eigenvalues change. For $A^T$, the eigenvalues are the same but the eigenvectors are generally different.

MATLAB: [V,D] = eig(A) returns eigenvectors as columns of V, eigenvalues on diagonal of D.

Triangular Matrices

If A is upper or lower triangular, then: λⱼ = aⱼⱼ (diagonal entries ARE the eigenvalues)

This is the most tested fact on every CISC 271 exam. For a triangular matrix, you can read the eigenvalues directly off the diagonal without any calculation. This applies even if the matrix has complex off-diagonal entries.

Diagonalization

Diagonalization of A

$$A = V\Lambda V^{-1}$$

where $V = [\vec{v}_1 \; \vec{v}_2 \; \cdots \; \vec{v}_n]$ is the matrix of eigenvectors (columns) and $\Lambda = \text{diag}(\lambda_1, \ldots, \lambda_n)$. This holds if and only if $V$ is invertible (eigenvectors are linearly independent). A sufficient condition: all eigenvalues are distinct.

Similar Matrices

$$A \sim C \iff \exists \text{ invertible } P : A = P^{-1}CP$$

Similar matrices share the same eigenvalues (but generally different eigenvectors). Their traces and determinants are also equal.

Symmetric Matrices & Spectral Theorem

Properties of Symmetric Matrices ($B = B^T$)

1. Every eigenvalue is real 2. Eigenvectors for distinct eigenvalues are orthogonal 3. The eigenvectors form an orthonormal basis

Proof of orthogonality: for distinct $\lambda_1 \neq \lambda_2$, consider $(B\vec{v}_1) \cdot \vec{v}_2$. Compute this two ways: (1) $= \lambda_1(\vec{v}_1 \cdot \vec{v}_2)$ and (2) $= \lambda_2(\vec{v}_1 \cdot \vec{v}_2)$ (using $B^T = B$). Since $\lambda_1 \neq \lambda_2$, we must have $\vec{v}_1 \cdot \vec{v}_2 = 0$.

Spectral Decomposition (Spectral Theorem)

$$B = Q\Lambda Q^T$$

For any real symmetric matrix $B$: $Q$ is orthogonal (columns are unit eigenvectors of $B$), $\Lambda = \text{diag}(\lambda_1, \ldots, \lambda_n)$ with real eigenvalues. Since $Q^{-1} = Q^T$, this is a special case of diagonalization. Can also be written as a sum of rank-1 matrices:

$$B = \sum_{j=1}^n \lambda_j \vec{q}_j \vec{q}_j^T$$

This outer-product expansion is the foundation of PCA and the SVD.

Positive Definiteness

Classification of Real Symmetric Matrices

Symbol | Name | Eigenvalues | Quadratic Form -------|------------------|--------------------|---------------- B ≻ 0 | Positive Def. | all λⱼ > 0 | ū^T B ū > 0 B ⪰ 0 | Positive SemiDef | all λⱼ ≥ 0 | ū^T B ū ≥ 0 B ≺ 0 | Negative Def. | all λⱼ < 0 | ū^T B ū < 0 B ⪯ 0 | Negative SemiDef | all λⱼ ≤ 0 | ū^T B ū ≤ 0 | Indefinite | some + and some - | (neither above)

The quadratic form $\vec{u}^T B \vec{u}$ is the energy test. This can be proven using the spectral decomposition: $\vec{u}^T B \vec{u} = \sum_j \lambda_j \alpha_j^2$ where $\alpha_j = \vec{q}_j^T \vec{u}$.

IMPORTANT: For PSD, the sufficient (and necessary) condition is the energy condition $\vec{x}^T A \vec{x} \geq 0$. det(A) ≥ 0 alone is NOT sufficient (could have two negative eigenvalues).

Covariance Matrix is PSD

$$C = \frac{1}{m-1}M^TM \succeq 0$$

Any matrix of the form $M^TM$ is symmetric positive semidefinite. It is positive definite (PD) if and only if the columns of $M$ are linearly independent (i.e. $M$ is full rank). This is because $\vec{u}^T(M^TM)\vec{u} = \|M\vec{u}\|^2 \geq 0$, and equals zero iff $M\vec{u} = \vec{0}$, iff $\vec{u}$ is in the null space of $M$.

Rank-One Update Matrix

$$A_3 = I + \frac{1}{c}\vec{w}\vec{w}^T$$

The vector $\vec{w}$ is always an eigenvector with eigenvalue $\lambda = 1 + \|\vec{w}\|^2/c$. Any vector orthogonal to $\vec{w}$ is an eigenvector with eigenvalue $\lambda = 1$. This follows from: $A_3\vec{w} = \vec{w} + \frac{1}{c}\vec{w}(\vec{w}^T\vec{w}) = (1 + \|\vec{w}\|^2/c)\vec{w}$.

Matrix Powers & Stability

Matrix Powers via Diagonalization

$$A^k = V\Lambda^k V^{-1}$$

For diagonalizable $A = V\Lambda V^{-1}$: $A^2 = V\Lambda^2 V^{-1}$, etc. The eigenvalues are raised to the $k$th power; eigenvectors are unchanged. This implies a "square root" matrix $A^{1/2} = V\Lambda^{1/2}V^{-1}$ exists when all eigenvalues are non-negative.

Practice Problems

W26 Test 1 · Q4 For $A_4 = \begin{bmatrix}1+\epsilon & 2 \\ 0 & 1+\epsilon\end{bmatrix}$ with $0 < \epsilon < 1$, how do the eigenvalues change compared to $\epsilon = 0$? +
Worked Answer
$A_4$ is upper triangular. By the triangular matrix rule, eigenvalues = diagonal entries = $\lambda_1 = \lambda_2 = 1+\epsilon$.

At $\epsilon = 0$: both equal 1. For $\epsilon > 0$: both change to $1+\epsilon$.

Answer: Both $\lambda_1$ and $\lambda_2$ will change.
W26 Test 1 · Q5 $A_5$ is $4\times 4$ with $\text{tr}(A_5) = 13$, $\det(A_5) = 42$, $\lambda_a = 2$, $\lambda_b = 3$. Find the other two eigenvalues. +
Worked Answer
Use trace: $\lambda_c + \lambda_d = \text{tr} - \lambda_a - \lambda_b = 13 - 2 - 3 = 8$
Use determinant: $\lambda_c \cdot \lambda_d = \det / (\lambda_a \cdot \lambda_b) = 42 / (2 \times 3) = 7$

Solve the system: $\lambda_c + \lambda_d = 8$, $\lambda_c \lambda_d = 7$. This is a quadratic: $t^2 - 8t + 7 = 0$, so $(t-1)(t-7) = 0$.

$\lambda_c = 1$, $\lambda_d = 7$
W26 Test 1 · Q3 $A_3 = I + \frac{1}{22}\vec{w}\vec{w}^T$ where $\vec{w} = [1,3,3,5]^T$. Find the eigenvector $\vec{v}$ for eigenvalue $\lambda = 3$. +
Worked Answer
For any matrix of the form $I + c\vec{w}\vec{w}^T$, the vector $\vec{w}$ itself is an eigenvector. Verify:
$A_3\vec{w} = \vec{w} + \frac{1}{22}\vec{w}(\vec{w}^T\vec{w})$
$\|\vec{w}\|^2 = 1 + 9 + 9 + 25 = 44$
$A_3\vec{w} = \vec{w} + \frac{44}{22}\vec{w} = \vec{w} + 2\vec{w} = 3\vec{w}$ ✓

$\vec{v} = \vec{w} = [1,3,3,5]^T$
W26 Test 1 · Q6 $A_6 = P_6\begin{bmatrix}2&0&0\\0&1&0\\0&0&-1\end{bmatrix}P_6^T$, eigenvalues $36, 9, -2$. Describe the columns of $P_6$. +
Worked Answer
Since $A_6 = P_6 D_6 P_6^T$, the columns of $P_6$ are eigenvectors of $A_6$. The given $P_6 = \begin{bmatrix}-4&1&0\\1&2&-1\\1&2&1\end{bmatrix}$.

Check: col1 · col2 $= (-4)(1)+(1)(2)+(1)(2) = -4+2+2 = 0$ ✓
col1 · col3 $= (-4)(0)+(1)(-1)+(1)(1) = 0$ ✓
col2 · col3 $= (1)(0)+(2)(-1)+(2)(1) = 0$ ✓

$A_6$ is symmetric (given by the decomposition structure), so its eigenvectors must be orthogonal.

$P_6$ has orthogonal columns (not orthonormal — the columns are not unit length).
2019 Test 2 · Q1 $A$ is 2×2 symmetric, $\lambda_1 = 5$, $\lambda_2 = 10$, eigenvectors $\vec{x}_1 = [1/\sqrt{5},\, 2/\sqrt{5}]^T$, $\vec{x}_2 = [-2/\sqrt{5},\, 1/\sqrt{5}]^T$. Find $A$. +
Worked Answer
Use spectral decomposition: $A = \lambda_1\vec{x}_1\vec{x}_1^T + \lambda_2\vec{x}_2\vec{x}_2^T$

$\vec{x}_1\vec{x}_1^T = \frac{1}{5}\begin{bmatrix}1&2\\2&4\end{bmatrix}$, so $5 \cdot \vec{x}_1\vec{x}_1^T = \begin{bmatrix}1&2\\2&4\end{bmatrix}$

$\vec{x}_2\vec{x}_2^T = \frac{1}{5}\begin{bmatrix}4&-2\\-2&1\end{bmatrix}$, so $10 \cdot \vec{x}_2\vec{x}_2^T = \begin{bmatrix}8&-4\\-4&2\end{bmatrix}$

$A = \begin{bmatrix}1&2\\2&4\end{bmatrix} + \begin{bmatrix}8&-4\\-4&2\end{bmatrix} = \boxed{\begin{bmatrix}9&-2\\-2&6\end{bmatrix}}$
2019 Test 2 · Q2 $M = \begin{bmatrix}1+\epsilon & 1+\epsilon \\ 0 & 2\end{bmatrix}$. If $\epsilon$ changes from 0 to small positive, how many eigenvalues change? +
Worked Answer
$M$ is upper triangular → eigenvalues are diagonal entries: $\lambda_1 = 1+\epsilon$ and $\lambda_2 = 2$.

At $\epsilon = 0$: $\lambda_1 = 1$, $\lambda_2 = 2$. For $\epsilon > 0$: $\lambda_1 = 1+\epsilon$ changes, $\lambda_2 = 2$ stays fixed.

Exactly one eigenvalue changes, and both remain positive.
2025 Final · Q1 $A_1 = \begin{bmatrix}1+\epsilon & 0 \\ 2 & 2+\epsilon\end{bmatrix}$. At $\epsilon = 0$: $\lambda_1=1$, $\lambda_2=2$. For small positive $\epsilon$, how do eigenvalues change? +
Worked Answer
$A_1$ is lower triangular → eigenvalues are diagonal entries: $\lambda_1 = 1+\epsilon$, $\lambda_2 = 2+\epsilon$.
Both are functions of $\epsilon$, so both change for $\epsilon \neq 0$.

Both $\lambda_1$ and $\lambda_2$ will change. (Answer: d)
2025 Final · Q2 $A_2$ is symmetric, $A_2 = P_2\begin{bmatrix}3&0&0\\0&1&0\\0&0&-1\end{bmatrix}P_2^T$. What describes the columns of $P_2$? +
Worked Answer
$A_2$ is symmetric → eigenvectors for distinct eigenvalues are orthogonal (proven in class). The columns of $P_2$ are eigenvectors of $A_2$, which must be mutually orthogonal.

They are NOT orthonormal (unit length) since the decomposition is $A_2 = P_2 D P_2^T$, not $A_2 = Q\Lambda Q^T$ with $Q^TQ = I$.

Orthogonal columns. (Answer: c)
Conceptual State a sufficient condition for a symmetric 2×2 matrix $A$ to be positive semi-definite. +
Worked Answer
The energy condition: if $\vec{x} \neq \vec{0}$ then $\vec{x}^T A \vec{x} \geq 0$.

Common wrong answers: det(A) ≥ 0 (not sufficient: could have eigenvalues $-2$ and $-3$, det $= 6 > 0$). sum of diagonals ≥ 0 (not sufficient: could be 5 + (-3) = 2 but one eigenvalue is negative). The energy condition is both necessary and sufficient.