Linear algebra for machine learning: vectors, eigenvalues and SVD explained

In this article

1. Before you start
2. Why interviewers ask this
3. Linear algebra from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

Linear algebra is the grammar of every model: embeddings are vectors, layers are matrices, attention is a pair of matrix products. Interviewers ask it in three forms: relate linear algebra to transformers, walk through an SVD and its geometric meaning, or solve a puzzle such as the range of correlations three variables can share. A weak answer recites definitions: an eigenvector satisfies Av = λv. A strong answer says what the matrix does to space, uses the same picture to explain principal component analysis (PCA) and low-rank adaptation, and can carry out a short proof. This page builds those pictures in five steps and ends on the correlation puzzle, with every number computed.

Linear algebra from first principles

Vector

A vector is an ordered list of numbers, such as (3, 4), that can be read as a point or as an arrow from the origin with a length and a direction.

Matrix

A matrix is a rectangular table of numbers that, multiplied by a vector, turns it into another vector; it represents a transformation of space such as a stretch, a rotation or a shear.

Eigenvector

An eigenvector of a square matrix A is a nonzero vector v that A only scales, so that Av = λv; the scale factor λ is its eigenvalue.

Singular value decomposition

The singular value decomposition (SVD) writes any matrix A as UΣVᵀ, a rotation, a stretch along the axes by non-negative singular values, and a second rotation.

In plain words

Think of a matrix as a rubber sheet with a grid drawn on it. Multiplying by the matrix grabs the sheet and deforms it: stretch it, spin it, shear it. A vector is a point on the grid, and the product is where that point lands. The analogy breaks in one place: a real matrix may act on thousands of dimensions, where you cannot picture the sheet, so you trust the algebra and use two or three dimensions to build intuition.

In one sentence: machine learning stores data as vectors, applies learned matrices to them, and linear algebra tells you what those matrices do and how to simplify them.

Step 1: Vectors, dot products and norms

The dot product of two vectors multiplies matching entries and adds them. For a = (3, 4) and b = (4, 3) it is 3 × 4 + 4 × 3 = 24. The norm, or length, of a is the square root of 3² + 4², which is 5. Dividing the dot product by both lengths gives the cosine of the angle between them: 24 / (5 × 5) = 0.96, so the arrows point almost the same way.¹

This is how a model compares things. Two embeddings with a cosine near one mean similar items; a cosine near zero means unrelated. In attention, the score between two tokens is a dot product of a query vector and a key vector, divided by the square root of the key dimension, d_k, as in Vaswani et al. (2017). Input: two lists; output: one number that measures alignment.

1 · Dot product2 · Matrix = transformation3 · Eigenvector4 · SVD5 · Low rankReplay

Step-by-step description

1. Multiply matching entries and add: alignment of two vectors.
2. A matrix turns each vector into a new vector.
3. A direction the matrix only stretches, by the eigenvalue.
4. Any matrix: rotate, stretch by singular values, rotate.
5. Keep the top singular values for the best smaller matrix.

• The dot product of (3, 4) and (4, 3) is 24 and the cosine is 0.96.
• For the matrix [[2, 1], [1, 2]] the eigenvalues are 3 and 1.
• A rank-2 approximation of the 6 by 4 matrix keeps 98.5% of the energy.

Step 2: Matrices as transformations

A matrix times a vector computes a dot product of each row with the vector, producing a new vector. A matrix with m rows and n columns takes n-entry vectors to m-entry vectors, and two matrices multiply only when the inner sizes agree: (m by n) times (n by p) gives (m by p).

A neural-network layer is this operation plus a nonlinearity. Attention is two matrix products, Q Kᵀ then the softmax weights times V, after Q, K and V are produced by multiplying the input by three learned matrices, W_Q, W_K and W_V; a fourth, the output projection W_O, mixes the heads afterwards. For an illustrative sequence of 2,048 tokens and width 4,096 with 32 heads, the four projection matrices account for 80% of the attention block's multiply-adds and the two attention products for 20%; the feed-forward sublayer is not counted.¹ Every one of them is a matrix multiplication, which is why graphics processing units (GPUs) suit the work.

Step 3: Eigenvectors are directions that only get stretched

Most vectors change direction when a matrix acts on them. An eigenvector is the exception: the matrix stretches it, or flips it if λ is negative, but keeps it on the same line. Take A = [[2, 1], [1, 2]]. The vector (1, 1) maps to (3, 3), so it is an eigenvector with eigenvalue 3. The vector (−1, 1) maps to (−1, 1), with eigenvalue 1.¹

The two directions are perpendicular, which is no accident: a symmetric matrix always has real eigenvalues and perpendicular eigenvectors. Along these axes the matrix is just two independent stretches, by 3 and by 1. That is why eigenvectors simplify problems: they are the coordinates in which a matrix is easy.

A symmetric matrix is positive semi-definite when all its eigenvalues are at least zero; covariance matrices are, which will matter in the worked example. Eigenvectors exist only for square matrices and may fail to span the space, which leads to the next step.

Step 4: SVD rotates, stretches and rotates again

The SVD works for every matrix, even a rectangular one. It says any transformation is a rotation (Vᵀ), then a stretch along the coordinate axes by the singular values (Σ), then another rotation (U). The singular values are never negative and are listed largest first.

Here is where eigenvalues and singular values part ways. The shear matrix [[1, 1], [0, 1]] slides each row sideways. Its eigenvalues are both 1, and it has only one eigenvector direction, so eigenvalues say "no stretching at all". Its singular values are 1.618 and 0.618, the golden ratio and its inverse, because the shear really does stretch space along one axis and squeeze it along another.¹

Text description of the diagram

• Eigenvalues need a square matrix; every matrix has singular values.
• For the shear matrix the eigenvalues are 1 and 1 and the singular values are 1.618 and 0.618.

For a symmetric positive semi-definite matrix the two sets of numbers coincide: our matrix [[2, 1], [1, 2]] has singular values 3 and 1. For anything else, trust the singular values to tell you how much a matrix amplifies its input. Strang's MIT course covers eigenvalues, positive definite matrices and the SVD in that order.

Step 5: Keep the big singular values: low-rank approximation and PCA

If a few singular values are large and the rest small, the small ones can be dropped. Keeping the top k gives the best rank-k approximation, and the error equals the square root of the sum of the squared discarded singular values, which our script checks numerically. The rank of a matrix is the number of nonzero singular values, the number of independent directions it uses.

Principal component analysis (PCA) is this idea applied to data: centre each column, take the SVD, and the right singular vectors are the principal directions, the axes along which the data vary most. Low-rank adaptation (LoRA) applies it to fine-tuning: Hu et al. (2021) freeze the pre-trained weights and train two thin matrices whose product is a low-rank update. For a 4,096 by 4,096 weight and rank 8, that is 65,536 trainable numbers instead of 16,777,216, which is 0.39%.¹

Worked example

Two computations, both checked by code.¹

Part one: low-rank approximation. Take a 6 by 4 matrix of ratings, rows are users and columns are items, with two clear groups of items: [[5, 5, 0, 1], [5, 4, 1, 0], [4, 5, 0, 0], [0, 1, 5, 4], [1, 0, 4, 5], [0, 0, 5, 5]]. Its singular values are 12.27, 10.65, 1.58 and 1.22.¹ Keeping rank k stores k × (6 + 4 + 1) numbers: k columns of U (6 each), k rows of Vᵀ (4 each) and k singular values.

Text description of the diagram

• The singular values of the 6 by 4 matrix are 12.27, 10.65, 1.58 and 1.22.
• The first two singular values hold 98.5% of the energy.

The error at each rank matches the square root of the discarded squared singular values exactly (for rank 2, the square root of 1.58² + 1.22² = 2.00).¹ One honest caveat: at this tiny size, storing the factors takes 22 numbers against 24 for the original, a saving of only 2 numbers; the saving grows with size, as in the LoRA count above.

Part two: the correlation puzzle. Suppose n random variables have all pairwise correlations equal to c. What values can c take? The correlation matrix is C = (1 − c)I + cJ, where J is all ones. Its eigenvalues are 1 + (n − 1)c once, along the all-ones vector, and 1 − c repeated n − 1 times. A correlation matrix must be positive semi-definite, so both must be at least zero, which gives −1/(n − 1) ≤ c ≤ 1.

The last column shows the matrix becoming invalid just past the bound. For three variables with equal correlation r the answer is −1/2 ≤ r ≤ 1: three variables cannot all be negatively correlated beyond one half.

Trade-offs and edge cases

Eigendecomposition is not universal. It needs a square matrix, and a matrix such as the shear lacks enough eigenvectors. The SVD always exists.

Centre before PCA. PCA on uncentred data finds the mean, not the spread. Shift our seeded two-column sample by a mean of (10, 5): the uncentred first component points almost along the mean, (0.90, 0.44) against the mean direction (0.89, 0.45), and claims 99.4% of the variance, while the centred first component is (0.99, 0.14) and explains 95.2%, with the second at 4.8%, matching the covariance eigenvalues 5.26 and 0.27.¹ Scale columns when units differ.

Low rank is a modelling assumption. Real matrices are rarely exactly low-rank; the claim is that a few directions carry most of the energy, as in the ratings example. LoRA's bet is the same: Hu et al. report that the useful change in a weight during adaptation is low-dimensional.

Cost. Full SVD of a large matrix is expensive, so practitioners use truncated or randomized methods that compute only the top k directions.

Numerical care. Avoid forming AᵀA to get singular values; it squares sensitivity to rounding. Use a library SVD.

How to answer it in an interview

The spoken structure is: the factorization, the geometry in three moves, why it beats eigenvalues, then one use.

Common mistakes

"Singular values are the eigenvalues." (wrong) They coincide only for symmetric positive semi-definite matrices; the shear has eigenvalues 1, 1 and singular values 1.618, 0.618. (right) "Singular values are the square roots of the eigenvalues of AᵀA."

"Every matrix has an eigendecomposition." (wrong) Non-square matrices have none and some square ones lack enough eigenvectors. (right) "Every matrix has an SVD."

"PCA works on raw data." (wrong) Without centring it finds the mean direction. (right) "Centre, then take the SVD or the covariance eigenvectors."

"Dropping small singular values loses nothing." (wrong) It loses exactly the energy of the dropped values. (right) "It gives the best rank-k approximation, with a known error."

Follow-up questions

How can we relate linear algebra to transformers?

Embeddings are vectors, each attention head multiplies by learned projection matrices, scores are dot products and the output is a weighted sum of value vectors. The transformer hub shows the full path; the dominant cost is matrix multiplication.

Suppose n random variables have all pairwise correlations equal to c. What is the range of c?

−1/(n − 1) ≤ c ≤ 1. The proof is the eigenvalue argument above: 1 + (n − 1)c and 1 − c must both be non-negative for the correlation matrix to be positive semi-definite.

What is PCA, and how does it relate to the SVD?

PCA finds the directions of greatest variance. The right singular vectors of the centred data matrix are those directions, and the squared singular values divided by n − 1 are the variances along them.

Why does low-rank adaptation work?

The weight change needed to adapt a model appears to have low intrinsic rank, so two thin matrices can represent it. Hu et al. report 10,000 times fewer trainable parameters than full fine-tuning of a 175-billion-parameter model.

What changes if the matrix is not symmetric?

Eigenvalues may be complex and eigenvectors need not be perpendicular, so use the SVD for stretch factors. See the probability and statistics hub and deep learning hub for where each tool appears.

Sources

1. Linear Algebra (18.06SC, Fall 2011), Gilbert Strang — MIT OpenCourseWare, 1 Dec 2011
2. Deep Learning, chapter 2: Linear Algebra — Goodfellow, Bengio, Courville (MIT Press), 1 Nov 2016
3. Attention Is All You Need — arXiv (Vaswani et al.), v7, 2 Aug 2023
4. LoRA: Low-Rank Adaptation of Large Language Models — arXiv (Hu et al.), v2, 16 Oct 2021