Maximum likelihood estimation explained: what it is and how to do it

In this article

1. Before you start
2. Why interviewers ask this
3. Maximum likelihood estimation from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the statistics and probability learning path for machine learning and data roles.

Why interviewers ask this

If you cannot explain where a loss function comes from, you are tuning it blind. Candidates report two kinds of question: derive the estimate for the bias of a coin from a sequence of flips, and explain how maximum likelihood and maximum a posteriori (MAP) estimation relate to cross-entropy loss and Kullback–Leibler (KL) divergence.

A weak answer recites "maximize the likelihood" and stops. A strong answer writes the likelihood for the stated model, takes the log, solves, and checks the result is a maximum. A Staff-level answer adds what the estimate does with tiny samples, why the Gaussian variance estimate is biased, and why training a classifier with cross-entropy is the same act as maximum likelihood.

Maximum likelihood estimation from first principles

Likelihood

A likelihood is a function of a model's parameters that gives the probability of the observed data when those parameters are assumed true, so it is read as a score for each parameter value, not as the probability of the parameter itself. For discrete data it is the product of probability mass function values at the observations; for continuous data it is the product of the probability densities at the observations.

Log-likelihood

The log-likelihood is the natural logarithm of the likelihood, which turns a product of many probabilities into a sum of their logarithms. Because the logarithm only rises as its input rises, the parameter value that maximizes the log-likelihood is the same one that maximizes the likelihood.

Biased estimator

A biased estimator is a formula for estimating a parameter whose average value over many samples of the same size differs from the true parameter. The maximum likelihood estimate of a Gaussian variance is one: on average it comes out too small by a factor that depends on the sample size.

In plain words

Picture three coins on a table with different biases. You flip one of them five times and see three heads. Which coin most likely produced that? You work out, for each coin, how probable your exact result was, and you pick the coin that scores highest.

Maximum likelihood does exactly that, except the candidates are every possible bias, not three coins. The analogy breaks in one place: the winning value is the best fit to the data, not a statement that it is probably the true value.

In one precise sentence, the estimate is the parameter value that maximizes the probability of the data you saw. Stanford's probability course notes put the idea as selecting the parameters that make the observed data the most likely.

Step 1: Write the likelihood of the data

Start with a model and its parameter. For a coin, the model is a Bernoulli distribution, a single flip with probability p of heads. The Stanford reader writes its probability mass function as $p^x(1-p{)}^{1-x}$, where x is one for heads and zero for tails.

Assume the flips are independent, so the probability of the whole sequence is the product of the single-flip probabilities. With k heads in n flips:

Input: the data and a model. Output: a function of p. For the five flips one, zero, zero, one, one, the likelihood of the data is p cubed times one minus p squared.¹ It is a score for each p, not a probability of p, and it does not add up to one across values of p.

Step 2: Take the logarithm so the product becomes a sum

The log-likelihood replaces the product with a sum:

Here f is the probability of one observation under p. Two reasons justify the move. First, the logarithm rises whenever its input rises, so the peak sits at the same p before and after taking it; only the height changes. Second, a product of many small probabilities becomes a tiny number that a computer can round to zero: the product of 2,000 probabilities of 0.5 (illustrative) underflows to exactly zero in double-precision arithmetic, while the sum of their logarithms is −1386.29.¹ Sums are also far easier to differentiate than products.

For the five flips the log-likelihood is three times the log of p plus two times the log of one minus p.¹ You now have a smooth curve with a single peak, and the next step finds where it is.

Step 3: Set the slope to zero and solve

A smooth curve has zero slope at its peak. Differentiate the log-likelihood, set the result to zero and solve for p. For the coin the derivative is k divided by p minus n minus k divided by one minus p, and its root is p equal to k divided by n.

The University of British Columbia (UBC) lecture notes show the same result: the derivative of the log-likelihood is zero at the number of heads divided by the number of flips. The estimate for a coin is therefore the fraction of heads. For the five flips with three heads it is 0.6.¹

Input: the log-likelihood. Output: a candidate value of p. The word candidate matters: a zero slope can also mark a valley, so one check remains.

Step 4: Check that the answer is a maximum

The second derivative tells a peak from a valley. If it is negative at the candidate, the curve bends downward there and the point is a maximum. For the coin the second derivative is $-k/p^2-(n-k)/(1-p{)}^2$, which is negative for every p between 0 and 1.

The slope of the log-likelihood is zero at p equal to 0.6, and its second derivative there is negative: about minus 20.83.¹ Also check the edges. If every flip is tails the estimate is zero, and if every flip is heads it is one, and a boundary answer deserves a second look because the slope need not be zero there. You now have the full recipe.

1 · Likelihood2 · Log3 · Zero slope4 · CheckReplay

Step-by-step description

1. Multiply the probability of each flip.
2. The product becomes a sum.
3. Set the derivative to zero and solve.
4. A negative second derivative means a maximum.

• The likelihood of the data is p cubed times one minus p squared.
• The slope of the log-likelihood is zero at p equal to 0.6, and its second derivative there is negative.

Worked example

Take five flips, one for heads and zero for tails: one, zero, zero, one, one (illustrative). The likelihood is p cubed times one minus p squared. Evaluating it at a few candidate values shows the peak directly.¹

The likelihood is largest at p equal to 0.6, where it is 0.03456. The likelihood at p equal to 0.5 is 0.03125, so a fair coin explains the data slightly worse.¹

Text description of the diagram

• The likelihood is largest at p equal to 0.6, where it is 0.03456.
• The likelihood at p equal to 0.5 is 0.03125.

Dividing the negative log-likelihood by the number of flips gives the average loss per flip. It is 0.6730 at p equal to 0.6 and 0.6931 at p equal to 0.5, so the maximum likelihood estimate is also the minimum-loss one.¹

A Gaussian example

For n independent observations from a Gaussian distribution, the log-likelihood uses the probability density of each one:

Setting the slope with respect to the mean to zero gives $\hat{\mu }=\bar{x}$, the sample mean. Setting the slope with respect to the variance to zero gives ${\hat{\sigma }}^2=\frac{1}{n}{\sum }_i(x_i-\bar{x}{)}^2$, the average squared distance from the sample mean; the Stanford reader works through parameter estimation in this way. Take the illustrative sample 3, 5, 6, 8, 13. The mean is 7 and the squared distances sum to 58, so the maximum likelihood variance, which divides by n, is 11.6.¹ The version that divides by n minus one gives 14.5.¹

The Book of Statistical Proofs shows that the expected value of the n-denominator estimate is (n−1)/n times the true variance. With five observations the factor is 0.8, so a true variance of ten (illustrative) averages 8 across samples.¹ Dividing by n−1 instead of n removes the bias.

Trade-offs and edge cases

Small samples are the weak spot of maximum likelihood. The National Institute of Standards and Technology (NIST) handbook says the estimates "become unbiased minimum variance estimators as the sample size increases", and that they are approximately normally distributed for large samples.

• Tiny samples overfit. Three heads in three flips gives an estimate of one, which says a tail is impossible. A prior fixes this: the UBC notes describe preferring parameter values less likely to overfit, which is MAP estimation. MAP estimation is a method that maximizes the likelihood multiplied by a prior over the parameter. The survey of loss functions on arXiv notes that maximum likelihood is the special case with a uniform prior. See MLE versus MAP and Bayes theorem questions.
• The variance estimate is biased. The bias shrinks as the sample grows: the factor (n−1)/n is 0.8 at n of five and 0.99 at n of one hundred.¹ Both estimates are acceptable; unbiased is not the only goal.
• No closed form. Many models have a log-likelihood whose zero-slope equation cannot be solved by algebra. Then the likelihood is maximized numerically, for example by gradient descent.
• Mean and median. The sample mean is the maximum likelihood estimate of a Gaussian mean, while the sample median is the estimate of the location of a Laplace distribution, so the right summary depends on the noise model; for the difference between summaries see mean versus median.

Cross-entropy loss is the average negative log-likelihood of the labels under the model. Goodfellow, Bengio and Courville (2016), section 5.5 show that maximum likelihood is the same as minimizing the cross-entropy between the data distribution and the model, which is the same as minimizing the KL divergence, and that Gaussian noise with a constant variance turns the likelihood into mean squared error. Bernoulli likelihood gives binary cross-entropy loss. A categorical likelihood over many classes gives cross-entropy loss.

Text description of the diagram

• Bernoulli likelihood gives binary cross-entropy loss.
• Gaussian noise with a constant variance gives mean squared error.

How to answer it in an interview

The spoken structure: say what is maximized, give the four steps, then name the small-sample and bias caveats.

Common mistakes

• "The likelihood is the probability that my parameter is correct." (wrong) It is the probability of the data given the parameter. (right) "It scores each parameter value by how probable the data are under it."
• "I set the derivative to zero, so I am done." (wrong) A zero slope can mark a valley or an edge case. (right) "I confirm the second derivative is negative and check the boundaries."
• "Maximum likelihood is always unbiased." (wrong) The Gaussian variance estimate is biased by the factor (n−1)/n. (right) "It is asymptotically unbiased, but not for every sample size."
• "Cross-entropy and maximum likelihood are different methods." (wrong) They are one computation seen from two sides. (right) "Minimizing cross-entropy is maximizing likelihood."
• "Three heads in three flips means the coin always lands heads." (wrong) The estimate is one, but the sample is tiny. (right) "I would add a prior or collect more data."

Follow-up questions

How do you derive the maximum likelihood estimate of a coin's bias from flip sequences?

Write the likelihood p to the power k times one minus p to the power n minus k, take the log, differentiate and set to zero. The solution is k divided by n, the fraction of heads, and a negative second derivative confirms it is a maximum.

How do maximum likelihood and maximum a posteriori estimation relate to cross-entropy loss and KL divergence?

The identity is $H(p,q)=H(p)+\mathrm{KL}(p,q)$: the cross-entropy of the model q against the data distribution p equals the entropy of p plus the KL divergence from p to q (Goodfellow et al., 2016, section 5.5). The entropy H(p) does not depend on the model, so minimizing cross-entropy is minimizing KL divergence, and on the empirical distribution of the data that is maximum likelihood. Maximum a posteriori estimation is cross-entropy plus the negative log-prior; a uniform prior adds a constant and gives maximum likelihood back, and a Gaussian prior adds an L2 penalty, which is ridge regression (survey on arXiv).

Why is the maximum likelihood estimate of a Gaussian variance biased?

It measures spread around the sample mean, which is itself fitted to the same data and so sits closer to the points than the true mean does. The expected estimate is (n−1)/n times the true variance, and dividing by n−1 removes the bias.

What if the likelihood has no closed-form solution?

Then you maximize the log-likelihood numerically, usually by gradient-based optimization. The objective is the same; only the way you reach the peak changes.

Is the median ever the maximum likelihood estimate?

Yes. For a Laplace distribution the maximum likelihood estimate of the location is the sample median, because its likelihood penalizes absolute rather than squared distance. More practice is in the probability interview questions.

Sources

1. NIST/SEMATECH e-Handbook of Statistical Methods, 8.4.1.2 Maximum likelihood estimation — NIST, 30 Oct 2013
2. Maximum Likelihood Estimation (lecture notes 20) — Stanford University, 1 Jan 2020
3. Deep Learning, section 5.5 Maximum likelihood estimation (Goodfellow, Bengio, Courville) — MIT Press, 18 Nov 2016
4. Parameter Estimation (course reader, Piech and Sahami) — Stanford University, 1 May 2017
5. MAP Estimation (CPSC 440, lecture 3) — University of British Columbia (M. Schmidt), 1 Jan 2022
6. Proof: Maximum likelihood estimator of variance is biased — The Book of Statistical Proofs, 1 Mar 2020
7. A survey and taxonomy of loss functions in machine learning (arXiv 2301.05579v1) — arXiv, 13 Jan 2023