Machine learning algorithms: types, how they work and how to choose

In this article

1. Start here: the learning path
2. What are the main types of machine learning algorithms?
3. How do the main machine learning algorithms work?
4. Generative versus discriminative models
5. When does classical machine learning beat a large model?
6. How do you tell underfitting from overfitting?
7. Choosing a loss, an optimiser and a metric
8. Questions interviewers ask on machine learning fundamentals
9. All 40 guides in Classical machine learning
10. Follow-up questions
11. Sources

Start here: the learning path

Machine learning algorithm

A machine learning algorithm is a procedure that fits a model's parameters to data by minimising an error measure, so the model can then make outputs for inputs it has not seen.

Supervised learning

Supervised learning is learning from examples that each carry a correct answer, called a label, so the model learns to map inputs to labels. Classification and regression are its two main tasks.

Unsupervised learning

Unsupervised learning is an approach that learns structure from data that has no labels, such as grouping similar rows or compressing many features into a few.

Generative model

A generative model is a model of how the inputs and labels are jointly produced, so it can assign probabilities to inputs and generate new ones. Naive Bayes is a classic example.

Discriminative model

A discriminative model is a model of the label given the input only, with no model of how inputs arise. Logistic regression is a classic example.

Read the families section first, because every model you meet fits one of them. Then take the topics in this order: generative versus discriminative models, the loss a model minimises, the optimiser that minimises it, how to evaluate the result, and how to tell underfitting from overfitting. The cross-entropy loss and gradient descent articles cover the middle two, and the confusion matrix article covers evaluation.

Text description of the diagram

• Loss, gradient descent and evaluation connect every model family.
• Generative versus discriminative models, the loss, the optimiser and evaluation come in that order.

1. k-means vs KNN: what is the difference?
2. What are the assumptions of linear regression?
3. What is gradient descent in machine learning?
4. What is cross-entropy loss?
5. Exploration vs exploitation: what is the difference?
6. Logistic regression explained: how it works and how it is trained
7. Feature engineering techniques for encoding, scaling and missing data
8. BLEU vs ROUGE: what each metric measures and when to use it
9. What is a confusion matrix, and how do you read precision and recall?
10. Bias vs variance in machine learning: what is the difference?
11. L1 vs L2 regularization
12. What is data leakage in machine learning, and how do you prevent it?
13. Bagging vs boosting: what is the difference?
14. How does anomaly detection work on a time series?
15. What is distribution shift, and how do you detect it?
16. How do you handle an imbalanced dataset?
17. What is collaborative filtering, and how does it recommend?
18. SVMs, kernels, naive Bayes and PCA: margins, probabilities and variance
19. What are ranking and retrieval metrics?
20. What metrics do you use to eval models?
21. MLE vs MAP
22. Why did you choose a random forest? Variance, out-of-bag scores, baselines
23. What loss function did you employ?
24. If you had a binary classification problem, which loss function would you use?
25. What is multi-task ranking, and how does it work?
26. How do you get annotated training data?
27. Which one escapes local minima better? SGD vs full-batch GD
28. How do we decide which direction to take the step in?
29. What is overfitting or underfitting?
30. How do you determine if a model is overfitting?
31. How does K-means clustering work?
32. What are convolutional networks, and how do they work?
33. Machine learning breadth interview questions and answers

What are the main types of machine learning algorithms?

The four families differ in the supervision the data carries. Supervised learning has a label for each example. Unsupervised learning has none and looks for structure. Self-supervised learning builds its own labels from the data, such as hiding a word and learning to fill it in, which is how many language models are pretrained. Reinforcement learning has no labels but a reward signal, and the model learns a policy that earns more reward.

Text description of the diagram

• Supervised learning has a label for each example.
• Unsupervised learning has no labels and looks for structure.
• Self-supervised learning builds its own labels from the data.
• Reinforcement learning has no labels but a reward signal.

The scikit-learn estimator map, a flowchart the project calls a rough guide, shows how the supervised and unsupervised choices are made in practice. Its first step is blunt: with 50 samples or fewer, get more data. Treat the chart as a starting order to try, not a rule.

How do the main machine learning algorithms work?

Each common algorithm turns data into a model by a different rule, and knowing the rule tells you when it fits.

Linear and logistic regression

Linear regression predicts a number as a weighted sum of the input features, with the weights chosen to make the squared error on the training data as small as possible. Logistic regression passes the same weighted sum through the sigmoid function to get the probability of a class, and is trained with cross-entropy loss. Both are fast, and each weight shows how strongly a feature pushes the prediction.

Decision trees, bagging and boosting

A decision tree splits the rows again and again on the feature question that best separates the targets, and predicts from the leaf a row lands in. One deep tree overfits. Bagging trains many trees on random resamples of the rows and averages them, which cuts variance; a random forest is bagging plus a random subset of features at each split. Boosting trains small trees one after another, each fitted to the errors the earlier trees still make, which cuts bias; gradient boosting and XGBoost work this way.

Support vector machines

A support vector machine (SVM) finds the boundary between two classes with the widest margin to the nearest training points, the support vectors. A kernel lets it draw curved boundaries by measuring similarity in a richer feature space without building that space.

k-nearest neighbours and k-means

k-nearest neighbours is supervised: it stores the training rows and labels a new row by a vote of the k closest ones. k-means is unsupervised: it places k centres, assigns each row to its nearest centre, moves each centre to the mean of its rows, and repeats until nothing changes. Both depend on a distance, so features need a common scale.

Naive Bayes and principal component analysis

Naive Bayes is a generative classifier that applies Bayes' rule and assumes the features are independent given the class, so it needs only per-class counts; it is fast and strong on short text with little data. Principal component analysis (PCA) is unsupervised: it finds the directions in which the data varies most and keeps the first few, compressing many correlated features into fewer.

Regularisation

Regularisation adds a penalty on the size of the weights to the loss. A lasso (L1) penalty pushes some weights to exactly zero, which drops features; a ridge (L2) penalty shrinks all weights smoothly. Both trade a little bias for less variance, as Goodfellow et al. (2016) explain in chapter 5.

Generative versus discriminative models

A generative model learns how inputs and labels arise together, and a discriminative model learns only the boundary between labels. Ng and Jordan (2001) compared logistic regression with naive Bayes and challenged the belief that discriminative classifiers are almost always better. They report that there can often be two regimes as the training set grows: the discriminative model has the lower asymptotic error, while the generative model can approach its higher asymptotic error much faster and so does better when data is scarce.

The practical reading is that the right choice depends on how much labelled data you have. The same distinction explains why a generative model can also produce samples, which discriminative models cannot. It connects to probability, so the Gaussian distribution article is a useful prerequisite, and to information theory through the entropy, cross-entropy and Kullback-Leibler (KL) divergence article.

When does classical machine learning beat a large model?

Classical models often win on tabular data of moderate size. Grinsztajn, Oyallon and Varoquaux (2022) benchmarked 45 datasets and report that tree-based models such as XGBoost and random forests remain state of the art on medium-sized data, about 10,000 samples, even without counting their speed advantage.

The factors that decide the choice are the amount of labelled data, the cost and delay per prediction, the need to explain a result, and whether the input is structured columns or raw text, images or audio. For raw perceptual data, neural networks win, and the deep learning guide and the transformers guide cover them. A scenario worth practising is a small labelled text set, a few hundred reviews, and the question of whether to train a simple classifier or call a large language model. The answer is to name those factors and compare costs, not to pick a favourite.

How do you tell underfitting from overfitting?

Compare error on the training data with error on held-out data. A model that is too simple has high error on both, a model that is too flexible has the lowest training error but a held-out error that has started to rise again, and the best model sits between. Some gap between the two errors is normal for every model; overfitting is the rise. We fit polynomials to 15 noisy training points drawn from a sine curve, with noise variance 0.09, and scored them on 200 fresh points.¹

The data is synthetic and illustrative, drawn with seed 0, and the errors were computed in code¹. The cubic has the lowest test error, 0.133, against 0.354 for degree 9, while degree 9 has the lowest training error, 0.018, and overfits. In expectation, test error cannot fall below the noise variance of 0.09, which is the floor no model can beat.

Text description of the diagram

• The cubic has the lowest test error, 0.133, against 0.354 for degree 9.
• Degree 9 has the lowest training error, 0.018, but overfits.
• The noise variance of 0.09 is the floor for test error.

This is the classical bias-variance picture. Belkin et al. (2018) propose that for very large models the picture extends to a double-descent curve in which test error falls again past the point of exact fit, so the U shape is the classical story rather than the last word. The fix for overfitting in the classical regime is more data, a simpler model, or regularisation. Always keep a test set untouched: for 10,000 rows a common illustrative split of 70, 15 and 15 percent gives 7,000 for training, 1,500 for validation and 1,500 for the final test¹.

Choosing a loss, an optimiser and a metric

A model needs three decisions beyond its form: what error to minimise, how to minimise it, and how to judge the result. The loss is chosen from the output type, as the cross-entropy loss article shows. The optimiser, usually a variant of gradient descent, moves the parameters. The metric is a separate choice made from the business need.

Accuracy can mislead. On a problem with one positive in a hundred, a model that always answers negative scores 99% accuracy and finds none of the positives, so its recall is zero¹. The confusion matrix article builds precision, recall and the choice of threshold from the four cells, and linear algebra for machine learning supplies the vectors and matrices underneath most models.

Questions interviewers ask on machine learning fundamentals

Fundamentals questions usually ask for a definition and then for the reason behind it. Prompts to prepare: supervised versus unsupervised learning, generative versus discriminative models, bias and variance, bagging versus boosting, how to handle class imbalance, data leakage, and when classical models beat large generative AI models. The bagging and boosting answer is in the algorithms section above, and the follow-ups below answer the rest.

Follow-up questions

What is an ML model?

It is the fitted function that results from running a machine learning algorithm on data: a set of learned parameters plus the rule for turning an input into an output. The algorithm is the recipe, and the model is the thing it produces.

What is the difference between generative and discriminative models?

A generative model learns how inputs and labels arise together and can generate inputs; a discriminative model learns only the label given the input. Per Ng and Jordan, the discriminative one has lower asymptotic error and the generative one can win with little data.

What is the difference between supervised and unsupervised learning?

Supervised learning trains on examples with correct labels; unsupervised learning finds structure, such as clusters, in data with no labels. Self-supervised learning sits between them by deriving labels from the data itself.

How do you handle class imbalance?

Resample the classes or weight them in the loss so the rare class counts for more, tune the decision threshold instead of keeping the default, and judge the result with precision and recall rather than accuracy, which a majority-class guesser can inflate.

What is data leakage and how do you avoid it?

Data leakage is information from outside the training rows reaching the model, which makes held-out scores look better than real use will be. Split the data first, then fit every preprocessing step, such as scaling or imputing, on the training rows only and apply it unchanged to the rest.

When would you use classical machine learning instead of generative AI?

When the task is a narrow prediction on structured data, labelled examples exist or are cheap to collect, latency and cost matter, or results must be explainable. Tree-based models stay strong on medium-sized tabular data according to Grinsztajn et al.

How do you tell whether a model is overfitting?

Its training error keeps falling as flexibility grows while its held-out error rises: the degree-9 fit above has lower training error than the cubic, 0.018 against 0.025, but far higher test error, 0.354 against 0.133.¹ A gap between the two errors on its own is normal. Confirm with a validation set or cross-validation, and see the deep learning guide for how very large networks complicate the picture.

Which metric should I use for imbalanced data?

Not accuracy alone. Use precision and recall, or a precision-recall summary, read from the confusion matrix, and tune the decision threshold to the cost of each error.

Sources

1. Choosing the right estimator, scikit-learn 1.9.1 documentation (machine learning map) — scikit-learn developers (page shows version only; read 2026-10-24), 24 Oct 2026
2. On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes — NIPS 2001 (Ng and Jordan; year only), 1 Jan 2001
3. Why do tree-based models still outperform deep learning on tabular data? (arXiv 2207.08815) — arXiv (Grinsztajn, Oyallon, Varoquaux), 18 Jul 2022
4. Reconciling modern machine learning practice and the bias-variance trade-off (arXiv 1812.11118) — arXiv (Belkin, Hsu, Ma, Mandal), 28 Dec 2018
5. Deep Learning (contents: Part I, Applied Math and Machine Learning Basics) — MIT Press (Goodfellow, Bengio, Courville; year only), 1 Jan 2016