What is a weight matrix in a neural network?

In this article

1. Before you start
2. Why interviewers ask this
3. Weight matrix from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page builds on how neural networks work, and it uses a little probability when it reaches random starting values (see probability and statistics for machine learning (ML)).

Why interviewers ask this

A question about the weight matrix is a fast way to see whether a candidate can reason about a neural network in shapes instead of slogans. If you cannot say what the matrix looks like, you cannot count parameters, size memory or debug the shape error that stops a model from running.

A weak answer says "the weights are the numbers the network learns." A strong answer says which axis is which, how many numbers there are, why two matrices with nothing between them are really one matrix, and why the starting values must differ from each other. A Staff-level answer adds the cost: memory per entry, and what the training loop stores for every entry.

The same few facts come back in questions on backpropagation, activation functions and exploding gradients, and the deep learning interview questions page collects the neighbours.

Weight matrix from first principles

Weight matrix

A weight matrix is a grid of learnable numbers, with one row for each neuron of a layer and one column for each input feature, that a layer multiplies with its input vector to produce one output number per neuron.

Linear layer

A linear layer is a layer that computes an affine map of its input, meaning a matrix product plus a bias vector, and applies no nonlinearity of its own.

In plain words

Picture a panel of judges scoring a contestant on the same four criteria. Each judge has four private importance numbers, one per criterion. A judge scores by multiplying each criterion by its importance and adding up. Stack the judges' numbers into a table, one judge per row. That table is a weight matrix. One multiplication by the table gives every judge's score at once.

One judge's numbers alone form a weight vector, which is a single row of the matrix; searchers who ask about a weight vector versus a weight matrix are asking about one row versus the whole stack.

The analogy breaks in one place: real judges keep their numbers, while a network's numbers start random and are adjusted by training.

In one precise sentence, a linear layer computes y = x·Wᵀ + b. Here x is the input, W is the weight matrix, b is the bias and y is the output. PyTorch documents exactly this affine map for torch.nn.Linear.

Step 1: Each row holds one neuron's weights

The weight matrix of a layer has one row per neuron and one column per input feature. The Stanford course notes put it this way: every neuron keeps its weights in a row, and one matrix holds all the connection strengths of the layer. Their example first layer, with three inputs and four neurons, has a weight matrix of four rows and three columns.

PyTorch uses the same layout. Its documentation gives the weight shape as (out_features, in_features), where in_features is the size of each input sample and out_features the size of each output sample. The bias has shape (out_features): one number per neuron.

Input: the layer's sizes. Output: a table of shape out by in. You now know what the object is: a single table for the whole layer, not one table per neuron.

Step 2: The layer multiplies the input by the transposed matrix

The layer computes every neuron's output with one operation: y = x·Wᵀ + b. The transpose Wᵀ is the matrix with rows and columns swapped, so an out-by-in matrix becomes in-by-out. That swap is why the shapes line up.

Take the toy layer with four inputs and three neurons: W has three rows and four columns, and x is a row of four numbers. A row of four times a grid of four by three gives a row of three, one value per neuron.¹ Each output value is the dot product of the input with one row of W, plus that neuron's bias.

The common error is to mix the two conventions: you store W as in by out but pass it to a layer that applies the transpose, or you take PyTorch's out-by-in weight and multiply x·W without transposing it. Either way the inner sizes (in and out) disagree, and unless they happen to be equal the framework refuses with a shape-mismatch error. Fix it by reading the shape as (out, in) and letting the layer apply the transpose. The input's last dimension must equal in_features; leading dimensions such as the batch pass through unchanged, per the same page.

Step 3: A nonlinearity between matrices stops them collapsing

Two weight matrices applied one after the other are the same as one matrix, because W2·(W1·x) equals (W2·W1)·x. The Stanford notes make the same point: without the nonlinearity, two matrices could be collapsed into a single matrix, and the scores would again be a linear function of the input.

So between layers the network applies an activation function, a fixed nonlinear function applied to each number. A common one is the rectified linear unit (ReLU), which replaces every negative number with zero. The activation functions page covers the others. Layers of this kind stacked in order form a feed forward network.

Input: the linear output y. Output: the activated values that feed the next layer's matrix. In the worked example below, a two-matrix stack with no activation produces exactly the same outputs as one combined matrix with fewer parameters here.¹

Step 4: Training changes every entry, from a random start

Training adjusts every entry of every weight matrix. Backpropagation is the method that computes the gradient of the loss with respect to each entry, and the gradient of a matrix has the same shape as the matrix, because there is one slope per entry. An optimizer then moves each entry a little against its slope.

The plain update is W ← W − η·∂L/∂W, where L is the loss and η (eta) is the learning rate, a small number that sets the step size. For one example, the gradient of a linear layer is ∂L/∂W = (∂L/∂y)ᵀ·x: a column of out numbers times a row of in numbers. That outer product has out rows and in columns, the shape of W.

Training also stores extra numbers per entry. Adam, a common optimizer, keeps a first-moment and a second-moment estimate for every parameter (Kingma and Ba, 2014). In 32-bit floats the weights, the gradient and the two moments take 4 × 4 = 16 bytes per entry, so the 4,096-by-4,096 layer with bias needs 268,500,992 bytes during training.¹

The starting values matter. PyTorch's documented default draws each weight from a uniform distribution between minus and plus the square root of k, where k is one divided by in_features. For four inputs that range is minus 0.5 to 0.5.¹

In a network with hidden layers, if every weight starts at the same value, the hidden neurons of a layer compute identical outputs and receive identical gradients (their outgoing weights are identical too), so they stay identical and the layer behaves like one neuron. Random draws break the tie.

1 · One row per neuron2 · Multiply3 · Activation4 · UpdateReplay

Step-by-step description

1. The matrix is out rows by in columns.
2. Input times the transposed matrix, plus the bias.
3. Apply the nonlinearity.
4. The gradient matches the matrix shape.

• The weight matrix has one row per neuron.
• The layer multiplies the input by the transposed matrix.
• A bias and an activation follow the product.
• The gradient has the same shape as the weight matrix.

Worked example

A layer with four inputs and three neurons shows every number. The weights below are illustrative, chosen to be easy to follow. Row 1 of W is (1, 0, 2, −1), row 2 is (0.5, 1, 0, 2) and row 3 is (−1, 1, 1, 0). The bias is (1, −1, 0) and the input is (2, 1, −1, 0.5).¹

Neuron 1 computes 2 + 0 − 2 − 0.5 = −0.5, then adds its bias of 1, giving 0.5. Neuron 2 computes 1 + 1 + 0 + 1 = 3, then adds −1, giving 2. Neuron 3 computes −2 + 1 − 1 + 0 = −2, plus 0, giving −2. The output is (0.5, 2, −2).¹ Feeding a batch works the same way: a second example (0, 1, 1, 1) stacked under the first makes a 2-by-4 input, and the output is a 2-by-3 grid whose second row is (2, 2, 2).¹

For the network with 3, 4, 4 and 1 neurons, the first matrix is 4 by 3 (the shape the Stanford notes use for their first layer), the second 4 by 4 and the third 1 by 4. Its parameters are 12 + 4, then 16 + 4, then 4 + 1, which total 41.¹

The collapse check uses a 2-by-3 matrix and the 3-by-4 matrix above, both without biases, so the stacked output comes from the pre-bias values (−0.5, 3, −2). Their product is a single 2-by-4 matrix with rows (2, 2, 2, 3) and (1.5, 0, −1, 2). On the input above, both routes give (5.5, 5), but the stack has 18 parameters and the collapsed matrix only 8.¹

For memory, multiply parameters by bytes per number: 4 for a 32-bit float, 2 for a 16-bit one. The 16-bit floating point (FP16) versus brain floating point 16 (BF16) page explains the two 16-bit formats.

Text description of the diagram

• The three weight matrices hold 12, 16 and 4 weights.
• The biases add 4, 4 and 1, so the network has 41 parameters.

Trade-offs and edge cases

The matrix is where a network's size lives. A layer's parameter count grows with the product of its input and output sizes, so doubling both widths quadruples the weights, and the memory with them. Training also keeps extra state for every entry, covered on the optimizer states page, so training memory is a multiple of the weights alone: in 32-bit training with Adam it is 16 bytes per entry against 4 for the weights (Step 4).

Conventions differ, and that is an edge case rather than a mistake. A column-vector convention writes W times x with W out by in; a row-vector convention may write x times W with W in by out. PyTorch stores out by in and computes x·Wᵀ. Always check which one a source uses.

Text description of the diagram

• PyTorch stores the weight out by in and computes x times W transposed.
• A column-vector convention writes W times x with W out by in.
• A row-vector convention may write x times W with W in by out.

Other edge cases:

• Bias is separate. In PyTorch the bias is its own vector, not a column of the matrix. Some texts fold it in by appending a constant 1 to the input; the arithmetic is the same.
• Not all parameters are weights. Biases and the scale and shift of normalization layers are parameters too; see batch versus layer normalization.
• Other layers reuse the idea. A convolutional layer shares one small set of weights across positions, and a long short-term memory (LSTM) cell uses several matrices. Very large entries can also blow up signals, the exploding gradients problem.

How to answer it in an interview

The spoken structure: say what a row and a column mean, give the shape and the parameter count, then the collapse argument and the reason for random initialization.

Common mistakes

• "PyTorch's weight is in by out, so I can multiply x times W." (wrong) PyTorch stores the weight out by in and computes x times W transposed. (right) "PyTorch stores out by in; other libraries and textbooks may use in by out, so I state my convention first."
• "Each neuron has its own matrix." (wrong) One matrix holds one row per neuron. (right) "One matrix per layer, one row per neuron."
• "The bias is part of the matrix." (wrong) In PyTorch it is a separate vector of length out. (right) "Weights plus a separate bias, so out times in plus out parameters."
• "Initialize everything to zero, it is neutral." (wrong) In a network with hidden layers, equal starting weights keep hidden neurons identical, and with all zeros the earlier layers get zero gradient. (right) "Random weights break the symmetry; zero biases are harmless."
• "More linear layers make a more expressive model." (wrong) Stacked matrices without an activation collapse to one. (right) "Depth needs a nonlinearity between the matrices."

Follow-up questions

What is the shape of a weight matrix?

For a layer with n inputs and m neurons, PyTorch stores it as m by n, that is (out_features, in_features), and computes x·Wᵀ + b. Some textbooks write it as n by m and skip the transpose, so state your convention first.

How many parameters does a layer have?

Out times in for the matrix, plus out for the bias. The toy layer with four inputs and three neurons has 12 + 3 = 15, and the network with layers 3, 4, 4 and 1 has 41.¹

Why can't you initialize all weights to zero?

In a network with hidden layers, if every weight starts at the same value, the hidden neurons of a layer compute identical outputs and receive identical gradients, because their outgoing weights are identical too, so they stay identical. With all weights exactly zero, the gradients reaching the earlier layers are zero as well. A single linear or softmax layer can start at zero and still learn distinct rows, but hidden layers need random starts. PyTorch's default draws from a small uniform range scaled by the number of inputs instead.

Why do you need an activation function between weight matrices?

Without one, W2·(W1·x) is just (W2·W1)·x, a single matrix, so extra layers add parameters but no power. See activation functions.

How much memory does a weight matrix take?

Entries times bytes per entry: an illustrative 4,096-by-4,096 layer with bias is 67,125,248 bytes in 32-bit floats and half that in 16-bit floats.¹ In 32-bit training with Adam, weights, gradient and two moments make 16 bytes per entry, 268,500,992 bytes for this layer.¹ Training needs more, because the optimizer stores extra values per entry.

Sources

1. torch.nn.Linear (PyTorch documentation, v2.9) — PyTorch / Linux Foundation (page undated; read 2026-10-26), 26 Oct 2026
2. Neural Networks Part 1: Setting up the Architecture — Stanford University course notes (page undated; read 2026-10-26), 26 Oct 2026
3. Adam: A Method for Stochastic Optimization (arXiv 1412.6980) — arXiv (Kingma and Ba), 22 Dec 2014