In this article
Foundations
What are the basics of deep learning?
Deep learning is machine learning with neural networks that have many layers, each transforming the output of the one before. A network is a function with adjustable numbers called weights; training adjusts them so the outputs match the data. Depth lets later layers build on simpler features from earlier ones, which is why the approach scales to images, speech and text.
Why it is asked: it checks you can define the field in plain words. Read the full explanation: how neural networks work.
What is a neuron, and how does a layer compute its output?
A neuron takes a weighted sum of its inputs, adds an offset called a bias, and passes the result through an activation function. A layer does this for many neurons at once, which is one matrix multiplication followed by an element-wise function. Stacking layers composes these steps.
Why it is asked: it is the building block everything else rests on. Read the full explanation: linear algebra for machine learning.
Why do we need activation functions?
Without a non-linear function between layers, a stack of linear layers multiplies into a single linear layer, so depth adds no power. The activation function supplies the bend. Rectified linear units, ReLU, which return the larger of zero and the input, are the common default for hidden layers; softmax turns final scores into probabilities. Sigmoid saturates, with a slope that never exceeds one quarter, which shrinks gradients; ReLU units can die when their input stays below zero.
Why it is asked: it separates memorised names from reasons. Read the full explanation: activation functions.
How many parameters does a small network have?
Count weights plus biases layer by layer. A multilayer perceptron (MLP) with 784 inputs, a hidden layer of 128 and 10 outputs has 784 × 128 + 128 = 100,480 parameters in the first layer and 128 × 10 + 10 = 1,290 in the second, 101,770 in all.1 The first layer dominates because the input is wide.
Why it is asked: it tests whether you can do the arithmetic that sizes a model. Read the full explanation: how neural networks work.
Why is a deep network better than a shallow one?
Depth lets a network reuse features: early layers learn simple pieces and later layers combine them, which can represent some functions with far fewer units than one wide layer would need. Practice, not only theory, supports it: very deep networks won image benchmarks once they became trainable, as in the residual-network results below.
Why it is asked: it tests whether you can argue from reuse, not slogans. Read the full explanation: how neural networks work.
Training
What is backpropagation?
Backpropagation is the algorithm that computes the gradient of the loss with respect to every weight by applying the chain rule backwards from the output, reusing values from the forward pass. It supplies the gradients; an optimiser such as gradient descent then uses them to update the weights. It is efficient because one backward pass gives all gradients.
Why it is asked: it is the core of training and a frequent derivation question. Read the full explanation: gradient descent.
What is the difference between gradient descent, stochastic gradient descent and mini-batch training?
Gradient descent computes the gradient over the whole dataset before each update. Stochastic gradient descent (SGD) updates from a single example, and mini-batch training updates from a small batch, the usual compromise. Smaller batches give noisier but cheaper steps and can help escape poor regions; larger batches give smoother steps and use hardware better.
Why it is asked: it tests the cost and noise trade-off. Read the full explanation: gradient descent.
What happens if the learning rate is too high or too low?
The learning rate is the factor that scales each weight update. Too high, and the loss oscillates or diverges, sometimes to not-a-number; too low, and training crawls or stalls in a flat region. A common practice is a short warm-up followed by a decay schedule, and a sweep over several values.
Why it is asked: it is the first knob anyone turns when training fails. Read the full explanation: gradient descent.
How does the Adam optimiser work?
Adam keeps running averages of the gradient and of its square and divides the running average of the gradient by the square root of the running average of its square, so each weight gets its own effective step size. The paper describes it as a first-order method that uses adaptive estimates of lower-order moments (Kingma and Ba, 2017). With default settings its first step is about the learning rate whether the gradient is 0.001, 1 or 100.1
Why it is asked: Adam is the default, so interviewers probe whether you know why. Read the full explanation: optimiser states and memory.
What is the difference between Adam and AdamW?
In Adam, adding an L2 penalty to the loss is not the same as weight decay, because the adaptive scaling changes the penalty's effect. AdamW decouples weight decay from the gradient step, which the authors say lets Adam generalise better (Loshchilov and Hutter, 2019). For plain SGD the two are equivalent when the decay is scaled by the learning rate.
Why it is asked: a precise detail that separates careful candidates. Read the full explanation: optimiser states and memory.
Which loss function do you use for classification, and why?
For single-label multi-class problems use softmax with cross-entropy loss (for binary or multi-label problems, a sigmoid per output with binary cross-entropy). Softmax turns logits, the final raw scores, into probabilities, and cross-entropy is the negative log of the probability given to the true class. For logits 2.0, 1.0 and 0.1 with class 0 correct, the probabilities are 0.659, 0.2424 and 0.0986, the loss is 0.417, and the gradient with respect to the logits is the probabilities minus the one-hot target (a vector with 1 for the true class and 0 elsewhere).1
Why it is asked: it links probability to the training signal. Read the full explanation: cross-entropy loss.
Why does weight initialisation matter?
Starting weights set the scale of signals and gradients in the first steps, and identical starting weights give identical gradients, so units would never differ: too small and signals fade layer by layer, too large and they blow up. He and colleagues introduced an initialisation designed for rectifier networks, which made very deep rectified networks trainable from scratch (He et al., 2015). It is commonly written as a standard deviation of the square root of 2 over the fan-in (the number of inputs to a unit), 0.0625 for 512 inputs.1
Why it is asked: it tests understanding of signal scale, not recipes. Read the full explanation: activation functions.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Regularisation and normalisation
What is overfitting and how do you reduce it?
Overfitting is when a model fits its training data, noise included, and does worse on new data. Reduce it with more data, data augmentation, a smaller model, weight decay, dropout, early stopping when validation loss stops improving, and a cleaner train and validation split. Measure the gap between training and validation loss to see whether it is happening.
Why it is asked: it is the central generalisation question. Read the full explanation: bias and variance in machine learning.
What is dropout and why does it work?
Dropout is a regulariser that randomly removes units during training and is turned off at inference. The original paper removes half of the feature detectors for each training case and reports that this stops units depending on specific others, called co-adaptation (Hinton et al., 2012). With the common inverted form, survivors are scaled by 1 over (1 − p), 2 at p = 0.5, so the mean activation is unchanged.1
Why it is asked: the scaling detail is a favourite follow-up. Read the full explanation: how neural networks work.
What does batch normalization do?
Batch normalization rescales a layer's inputs to zero mean and unit variance over each mini-batch, then applies learned scale and shift. Its authors report the same accuracy on an image model with 14 times fewer training steps (Ioffe and Szegedy, 2015). For the batch 1, 2, 3, 4 the normalised values are −1.3416, −0.4472, 0.4472 and 1.3416.1 During training it uses the current batch's statistics; at inference it uses running averages of mean and variance collected during training, so outputs do not depend on the batch.
Why it is asked: it tests both the mechanism and the training-versus-inference difference. Read the full explanation: batch normalization versus layer normalization.
When would you use layer normalization instead of batch normalization?
Layer normalization computes its statistics across the features of one example instead of across a batch, so it does not depend on batch size and behaves the same in training and inference. That makes it the standard in transformers and a better fit for small or variable batches and for sequence models. A common variant, RMSNorm, drops the mean-centring and only rescales (Zhang and Sennrich, 2019); LLaMA, for example, uses it (Touvron et al., 2023).
Why it is asked: it checks that you know why transformers avoid batch statistics. Read the full explanation: batch normalization versus layer normalization.
What are vanishing and exploding gradients, and how are they handled?
A vanishing gradient shrinks toward zero as it passes back through many layers, so early layers stop learning; an exploding gradient grows without bound. Remedies include ReLU-family activations, careful initialisation, normalisation layers, residual connections, and gradient clipping for the exploding case.
Why it is asked: it is the classic depth problem and ties many fixes together. Read the full explanation: activation functions.
What is a residual connection and why does it help?
A residual connection is a shortcut that adds a layer's input to its output, so the layer learns a correction instead of a whole transformation. The ResNet paper starts from the observation that deeper networks are harder to train and reaches 152 layers, about 8 times deeper than Visual Geometry Group (VGG) nets (He et al., 2015, residual networks). The shortcut also gives gradients a direct path.
Why it is asked: nearly every modern architecture uses one. Read the full explanation: residual connections.
Architectures
What is a convolutional neural network?
A convolutional neural network (CNN) applies small learned filters that slide across an input such as an image, sharing the same weights at every position. Small filters exploit locality, and sharing the same weights at every position makes the layer translation-equivariant; together they cut parameters sharply compared with a fully connected layer. Pooling or strides then shrink the map while deeper layers see larger regions.
Why it is asked: it tests whether you can explain the inductive bias. Read the full explanation: how neural networks work.
Why did LSTMs replace plain recurrent networks?
A recurrent neural network (RNN) passes a hidden state from step to step, and gradients through many steps shrink or explode. A long short-term memory (LSTM) network adds gates that control what to keep, write and output, which lets it carry information across longer spans: its cell state is updated by addition, so its gradient is scaled by the forget gate instead of being multiplied by the same matrix at every step. Transformers later replaced both for most language work.
Why it is asked: it tests the history behind today's defaults. Read the full explanation: LSTM networks.
What is the Transformer, and why did it replace recurrent models?
The Transformer is an architecture built solely on attention, with no recurrence or convolutions. Its authors report it is more parallelisable and needs much less training time, with 28.4 bilingual evaluation understudy (BLEU) points on English-to-German translation (Vaswani et al., 2017, v7 2023). Because every position attends to all others in one step, training parallelises across the sequence.
Why it is asked: it is the starting point for modern language models. Read the full explanation: the transformers hub.
What is attention?
Attention lets each position in a sequence compute a weighted average of the values at all positions, with weights from the similarity between its query and their keys. It gives direct access to any earlier token, at a cost that grows with the square of the sequence length.
Why it is asked: it is the follow-up to every transformer question. Read the full explanation: self-attention.
Debugging and practice
Training loss is not going down. What do you check?
Check the data and labels first, then the learning rate, then whether the loss is wired to the right outputs. Overfit a single small batch: if the model cannot drive that loss near zero, there is a bug, not a capacity problem. Then look at gradient norms (the size of the gradient per layer), initialisation and the optimiser settings.
Why it is asked: it reveals your debugging method. Read the full explanation: how neural networks work.
The loss became not-a-number. What are the likely causes?
Usual causes are a learning rate that is too high, a division by zero or logarithm of zero in a custom loss, overflow in the exponential of softmax, and low-precision numbers that overflow. Fix the loss with a stable formulation, lower the learning rate, add gradient clipping, and see how 16-bit formats differ in range.
Why it is asked: it separates experienced practitioners from tutorial followers. Read the full explanation: 16-bit floating point versus brain floating point.
How do you tell whether a model has high bias or high variance?
Compare training and validation error. High error on both means high bias: the model is too simple or under-trained. Low training error with high validation error means high variance: it has memorised the training data. The remedies are opposite, so diagnose first.
Why it is asked: it turns a vague symptom into a decision. Read the full explanation: bias and variance in machine learning.
Step-by-step description
- Start from what the loss curve does.
- Each symptom has its own first check.
- A flat loss points first to the data, labels and learning rate.
- Low training loss with high validation loss points to overfitting and regularisation.
- A not-a-number loss points to overflow, a log of zero or a learning rate that is too high.
- Define each term in one sentence, then give the number or the mechanism.
- Training is gradient descent with gradients from backpropagation; the learning rate is the first knob.
- Adam's first step is about the learning rate; AdamW decouples weight decay.
- Dropout, normalisation and residual connections each fix a named failure.
- When training fails, diagnose by symptom before changing the model.
Sources
- Adam: A Method for Stochastic Optimization (arXiv 1412.6980 v9) — arXiv (Kingma and Ba; ICLR 2015), 30 Jan 2017
- Root Mean Square Layer Normalization (arXiv 1910.07467 v1) — arXiv (Zhang and Sennrich), 16 Oct 2019
- LLaMA: Open and Efficient Foundation Language Models (arXiv 2302.13971 v1) — arXiv (Touvron et al.), 27 Feb 2023
- Decoupled Weight Decay Regularization (arXiv 1711.05101 v3) — arXiv (Loshchilov and Hutter; ICLR 2019), 4 Jan 2019
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (arXiv 1502.03167 v3) — arXiv (Ioffe and Szegedy), 2 Mar 2015
- Improving neural networks by preventing co-adaptation of feature detectors (arXiv 1207.0580) — arXiv (Hinton et al.), 3 Jul 2012
- Delving Deep into Rectifiers (arXiv 1502.01852) — arXiv (He, Zhang, Ren, Sun), 6 Feb 2015
- Deep Residual Learning for Image Recognition (arXiv 1512.03385) — arXiv (He, Zhang, Ren, Sun), 10 Dec 2015
- Attention Is All You Need (arXiv 1706.03762 v7) — arXiv (Vaswani et al.), 2 Aug 2023
