Classes 10 · 11 · 12 · 13 · Tests 2–3 · Final Q3–Q5

Projection &
Linear Regression

Linear regression is orthogonal projection. Finding the best linear fit to data is the same as finding the closest point in a subspace to a given vector. The normal equation solves both.

Data Preparation: Standardization

Mean, Zero-Mean Vector, Sample Variance

$$\bar{a} = \frac{1}{m}\sum_{i=1}^m a_i \qquad \vec{m} = \vec{a} - \bar{a}\vec{1} \qquad \sigma_{\vec{a}}^2 = \frac{\|\vec{m}\|^2}{m-1}$$

Sample variance uses $m-1$ in the denominator (Bessel's correction). Sample standard deviation $\sigma_{\vec{a}} = \|\vec{m}\|/\sqrt{m-1}$.

Standardized Data (Z-Score)

$$z(\vec{a}) = \frac{\vec{a} - \bar{a}\vec{1}}{\sigma_{\vec{a}}}$$

Standardized data has zero mean and unit variance. Applied column-by-column to a matrix. Used in the design matrix $X$ for standardized regression. Division by $\sigma = 0$ fails when a column is constant.

Orthogonal Projection

Projection to a 1D Subspace (Single Vector)

$$w = \frac{\vec{a}^T\vec{c}}{\vec{a}^T\vec{a}} \qquad \vec{p} = w\vec{a} = \frac{\vec{a}^T\vec{c}}{\vec{a}^T\vec{a}}\vec{a}$$

The scalar weight $w$ minimizes $\|\vec{c} - w\vec{a}\|^2$. The error $\vec{e} = \vec{c} - \vec{p}$ is perpendicular to $\vec{a}$: verify $\vec{a}^T\vec{e} = 0$.

Projection to a Subspace (General Case)

$$\vec{p} = A\vec{w} \qquad \text{where } \vec{w} \text{ solves the normal equation:}$$ $$\underbrace{A^TA}_{\text{symmetric}} \vec{w} = A^T\vec{c}$$ $$\Longrightarrow \vec{w} = (A^TA)^{-1}A^T\vec{c}$$

The projection matrix is $P = A(A^TA)^{-1}A^T$. Properties: $P^2 = P$ (idempotent), $P^T = P$ (symmetric). MATLAB shortcut: p = A*(A\c). The backslash solves the normal equation efficiently.

Error Vector

$$\vec{e} = \vec{c} - \vec{p} = \vec{c} - A\vec{w} \quad \perp \text{ column space of } A$$ $$A^T\vec{e} = \vec{0} \quad \text{(error is orthogonal to every column of A)}$$ $$\text{RMS}(\vec{e}) = \frac{\|\vec{e}\|}{\sqrt{m}}$$

The error is in the orthogonal complement of the column space of $A$ (left null space). This is the residual in linear regression.

Linear Regression

Regression With Intercept

$$A = \begin{bmatrix}\vec{a} & \vec{1}\end{bmatrix} \quad \vec{w} = \begin{bmatrix}w_1 \\ w_2\end{bmatrix} \quad A\vec{w} \approx \vec{c}$$

$w_1$ is the slope, $w_2$ is the intercept. The best-fit line always passes through the center point $(\bar{a}, \bar{c})$.

Regression Without Intercept (Hooke's Law)

$$c_i \approx k\, a_i \qquad k = \frac{\vec{a}^T\vec{c}}{\vec{a}^T\vec{a}} = \frac{\sum a_i c_i}{\sum a_i^2}$$

Single-parameter model. The data matrix $A = \vec{a}$ (just one column — no ones column). Normal equation is scalar: $(\vec{a}^T\vec{a})k = \vec{a}^T\vec{c}$.

Cross-Validation

K-Fold Cross-Validation

1. Divide data into K equal folds 2. For each fold k = 1, ..., K: - Train on all folds EXCEPT k (training set) - Test on fold k (hold-out set) 3. Compute validation residual ⃗v (on training subsets) 4. Compute test residual ⃗t (on hold-out subset) Key result: RMS(⃗t) > RMS(⃗v) always RMS(⃗t) ≫ RMS(⃗v) → overfitting / poor model

Validation residuals are optimistic because the model was trained on similar data. Test residuals represent true generalization. Leave-one-out (LOO) is a special case with $K = m$.

Practice Problems

W26 Test 2 · Q2 For $\mathbb{V}$ spanned by columns of $A_2 = \begin{bmatrix}2&1\\1&1\\-1&1\end{bmatrix}$, write a MATLAB expression for $\vec{p}_2$ (projection of $\vec{c}_2 = [1,2,3]^T$ into $\mathbb{V}$). +
Worked Answer
p = A*(A\c)

This computes $\vec{p} = A(A^TA)^{-1}A^T\vec{c}$. The backslash A\c solves the normal equation for weights $\hat{w}$, then A* projects back: $\vec{p} = A\hat{w}$.
W26 Test 2 · Q3 $A_3 = \begin{bmatrix}1&-1\\2&-1\\1&1\\1&2\end{bmatrix}$, $\vec{c}_3 = \begin{bmatrix}-11\\-7\\-2\\-1\end{bmatrix}$. Find the projection $\vec{p}_3$. +
Worked Answer
Normal equation $A_3^TA_3\vec{w} = A_3^T\vec{c}_3$:

$A_3^TA_3 = \begin{bmatrix}1\cdot1+2\cdot2+1\cdot1+1\cdot1 & \ldots\\\ldots & \ldots\end{bmatrix} = \begin{bmatrix}7 & 2\\2 & 7\end{bmatrix}$

$A_3^T\vec{c}_3 = \begin{bmatrix}(1)(-11)+(2)(-7)+(1)(-2)+(1)(-1)\\(-1)(-11)+(-1)(-7)+(1)(-2)+(2)(-1)\end{bmatrix} = \begin{bmatrix}-28\\14\end{bmatrix}$

Solve: $7w_1+2w_2=-28$ and $2w_1+7w_2=14$. From these: $w_1=-4$, $w_2=2$.

$\vec{p}_3 = A_3\vec{w} = -4[1,2,1,1]^T + 2[-1,-1,1,2]^T = \mathbf{[-6,-10,-2,0]^T}$
W26 Test 2 · Q4 Model $c_i \approx ka_i$ with data $a = [0,2,3,4]^T$, $c = [1.6, 3.5, 5.0, 8.5]^T$. Estimate $k$. +
Worked Answer
$k = \frac{\vec{a}^T\vec{c}}{\vec{a}^T\vec{a}} = \frac{0(1.6)+2(3.5)+3(5.0)+4(8.5)}{0^2+2^2+3^2+4^2} = \frac{0+7+15+34}{0+4+9+16} = \frac{56}{29} \approx \mathbf{1.931}$
2025 Final · Q3 Which sentence best describes specifying the projection of $\vec{c}$ into $\mathbb{V}$? (a) find spanning vectors minimizing distance to $\vec{c}$, (b) find weights minimizing Euclidean distance, (c) find the difference vector, (d) find the projection matrix. +
Worked Answer
Answer (b): Find the weights for spanning vectors that minimize the Euclidean distance to $\vec{c}$.

The projection specifies the scalar weights $w_j$ such that $\sum w_j\vec{a}_j$ is as close as possible to $\vec{c}$. We're not finding new spanning vectors (a), not finding the error vector (c), and not finding the projection matrix itself (d).
2025 Final · Q4 $A_4 = \begin{bmatrix}1&3\\2&-2\\1&1\\1&0\end{bmatrix}$, $\vec{c}_4 = \begin{bmatrix}-8\\-6\\-2\\-6\end{bmatrix}$. Find the error $\vec{e}_4 = \vec{c}_4 - \vec{p}_4$. +
Worked Answer
$A_4^TA_4 = \begin{bmatrix}1+4+1+1 & 3-4+1+0\\3-4+1+0 & 9+4+1+0\end{bmatrix} = \begin{bmatrix}7 & 0\\0 & 14\end{bmatrix}$ (diagonal — columns are orthogonal!)

$A_4^T\vec{c}_4 = \begin{bmatrix}(-8)+(2)(-6)+(-2)+(-6)\\ 3(-8)+(-2)(-6)+(-2)+0\end{bmatrix} = \begin{bmatrix}-28\\-14\end{bmatrix}$

$w_1 = -28/7 = -4$, $w_2 = -14/14 = -1$

$\vec{p}_4 = -4[1,2,1,1]^T - 1[3,-2,1,0]^T = [-4-3, -8+2, -4-1, -4-0]^T = [-7,-6,-5,-4]^T$

$\vec{e}_4 = \vec{c}_4 - \vec{p}_4 = [-8+7, -6+6, -2+5, -6+4]^T = \mathbf{[-1, 0, 3, -2]^T}$ (Answer: a)
2025 Final · Q5 Hooke's Law $c_i \approx ka_i$. Data: $a = [-1, 2, 3, 4]^T$, $c = [-3.1, 4.5, 5.0, 8.5]^T$. Find $k$. +
Worked Answer
$k = \frac{\vec{a}^T\vec{c}}{\vec{a}^T\vec{a}} = \frac{(-1)(-3.1) + 2(4.5) + 3(5.0) + 4(8.5)}{(-1)^2+2^2+3^2+4^2} = \frac{3.1+9+15+34}{1+4+9+16} = \frac{61.1}{30} \approx \mathbf{2.037}$ (Answer: b)
W26 Test 2 · Q5 In 3-fold CV, validation residuals $\vec{v}_5$ and test residuals $\vec{t}_5$ are given. Describe their RMS relationship. +
Worked Answer
$\text{RMS}(\vec{t}_5) \gg \text{RMS}(\vec{v}_5)$

Validation residuals are computed on data similar to (part of) the training set — they're optimistic. Test residuals measure true generalization to unseen data — always larger. When much larger, it indicates the model doesn't generalize well (possibly overfitting or poor model fit).