Is the importance sampling ratio also called the policy or likelihood ratio?

In this article

1. Before you start
2. Why interviewers ask this
3. Importance sampling ratio from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the statistics interview guide for machine learning and data roles.

Why interviewers ask this

The question is a terminology trap, and a wrong answer costs the follow-up. Three different ideas share the words "ratio" and "likelihood", and a candidate who merges them cannot explain why a reinforcement learning update is stable or unstable. Interviewers use it to test whether you know the idea behind the names rather than the names.

A weak answer says "yes, they are all the same thing." A strong answer says the importance sampling ratio, the policy ratio and the statistician's likelihood ratio are one ratio of target over sampler in different settings, and separates it from the hypothesis-testing likelihood ratio. A Staff-level answer adds what the ratio costs: variance that grows with the length of a trajectory, and why PPO uses a single per-step ratio between nearby policies, clipped so one update cannot move the policy too far.

Importance sampling ratio from first principles

Importance sampling ratio

The importance sampling ratio is the probability or density of an outcome under the target distribution divided by its probability or density under the distribution that produced the sample, used as a weight so that samples from one distribution estimate an average under the other.

Behaviour policy

A behaviour policy is the policy that actually chose the actions in the collected data, as opposed to the target policy whose value you want to estimate.

Probability ratio

The probability ratio is the name PPO gives to the new policy's probability of an action divided by the old policy's probability of the same action.

1 · Target over sampler2 · PPO ratio3 · Product over steps4 · LookalikesReplay

Step-by-step description

1. Weight each sample by target density over sampling density.
2. PPO uses new over old policy and clips it.
3. Over a trajectory the ratios multiply.
4. The test statistic and score function differ.

• The importance sampling ratio is target density divided by sampling density.
• In proximal policy optimization the ratio is clipped to the interval from one minus epsilon to one plus epsilon.

In plain words

Imagine you poll people at a gym to learn how much the whole town exercises. Gym-goers are over-represented, so you give each answer a weight: how common that kind of person is in town, divided by how common they are in your poll. People who are rare in the poll but common in town count more. The importance sampling ratio is that weight. The analogy breaks in one place: a poll weight comes from group sizes, while the ratio here comes from two probability formulas you know exactly.

In one precise sentence, the ratio is the target density divided by the sampling density, evaluated at each sample.

Step 1: A ratio of target over sampler corrects the average

Suppose you want an average under a target density p but your samples come from a sampling density q, the setting of importance sampling. Weight each sample by the ratio:

In general statistics usage the ratio p(x)/q(x) is called the likelihood ratio or the importance weight; "policy ratio" is informal reinforcement learning usage, where Sutton and Barto write "importance-sampling ratio" and the PPO paper writes "probability ratio". The estimate is the average of f times the ratio over samples from q. A concrete case with two actions: the target policy picks action A with probability 0.8 and action B with 0.2, and the behaviour policy picks them with 0.4 and 0.6. The ratios are 2 for A and 0.3333 for B.¹

Let the returns be 10 for A and 0 for B (illustrative). The true average under the target is 8.0. The plain average of samples from the behaviour policy is 4.0, which is wrong. The ratio-weighted average is 8.0, which is right.¹ The method needs the sampler to cover the target: if the behaviour policy never takes an action the target would take, the ratio is undefined and no weighting can recover it. The related method rejection sampling also draws from a simple density to reach a target one, by discarding samples instead of reweighting them.

Step 2: PPO calls it the probability ratio and clips it

In reinforcement learning (RL) the target is the policy you want to evaluate or improve, and the sampler is the behaviour policy that gathered the data. PPO uses the same quantity between its new and old policy. Schulman et al. (2017) write the probability ratio as the new policy's probability of the action over the old policy's, so the ratio equals 1 when the new and old parameters are equal. The paper names it the probability ratio, not the importance sampling ratio:

Without a constraint, the authors note, maximising the ratio-weighted objective would lead to an excessively large policy update. So PPO clips the ratio to the interval from one minus epsilon to one plus epsilon, with epsilon equal to 0.2 as the paper's example (Schulman et al., 2017, §3), and takes the minimum of the clipped and unclipped terms, so the final objective is a lower bound (a pessimistic bound) on the unclipped objective.

A worked clip with a ratio of 1.5

Take a new policy probability of 0.9 and an old one of 0.6: the ratio is 1.5, above 1, so not a probability.¹ With an advantage of 2 (illustrative), the unclipped term is 3.0, the clipped term is 2.4, and the minimum is 2.4. With an advantage of minus 2, the minimum is minus 3.0, so a harmful move is not hidden.¹ The clip removes the incentive to push the ratio outside the interval; it does not forbid it.

Step 3: Over a whole trajectory the ratios multiply

An RL episode has many steps, and the chance of a whole trajectory is the product of the per-step action probabilities. The textbook treatment of this, Section 5.5 of Sutton and Barto on off-policy prediction via importance sampling, therefore weights a return by the product of the per-step ratios of target to behaviour policy:

Reinforcement learning books call this product the importance-sampling ratio. A product of ratios is unstable. With the two-action example repeated at every step, the average of the product stays at 1, but the variance of the trajectory ratio is five thirds to the power of the number of steps, minus one. It is 0.6667 at one step, 11.8601 at five steps and 164.3817 at ten steps.¹

Text description of the diagram

• The variance of the trajectory ratio is five thirds to the power of the number of steps, minus one.
• At ten steps the variance is 164.3817.

This is why long trajectories make off-policy estimates noisy. PPO sidesteps this by using one per-step ratio between two nearby policies, with advantage estimates instead of full-trajectory returns.

Step 4: Two lookalikes play different roles

Two other things share the name. The hypothesis-testing likelihood ratio has the same form, one density over another at the data, but there both densities are hypotheses about where fixed data came from, and the ratio is a test statistic to threshold, not a weight that corrects an average (the generalised test also maximises each likelihood over parameters). The score-function gradient, sometimes called the likelihood-ratio policy gradient, is the derivative of the importance ratio at the old parameters, because the gradient of p divided by p equals the gradient of log p; it is related, but it estimates a gradient and reweights nothing.

The same ratio of probabilities is behind the Kullback-Leibler divergence in entropy and cross-entropy, which averages the log of this ratio under the target.

The Staff-level answer

What makes it Staff-level: it maps each name to its setting, states the variance cost, and names the two lookalikes before the interviewer asks.

The rubric

Three variants of this question

What is the importance sampling ratio in off-policy learning?

It is the target policy's probability of an action divided by the behaviour policy's, multiplied over the steps of a trajectory. Weighting a return by it lets data from the behaviour policy estimate the value of the target policy.

Why does PPO clip the probability ratio?

Because maximising the unclipped ratio-weighted objective would push the new policy far from the old one, which breaks the assumption that the old policy's data still describes the new one. The clip removes the reward for moving the ratio outside the interval, and the minimum keeps the estimate pessimistic.

What happens if the behaviour policy never takes an action the target would?

The ratio is undefined there and the estimate is biased, because no samples exist to carry the weight. Off-policy methods therefore require that the behaviour policy gives every action the target might take a non-zero probability.

Common mistakes

• "The PPO ratio is a probability, so it stays below 1." (wrong) It is a ratio of two probabilities and can exceed 1, as the 1.5 example shows. (right) "It is near 1 and clipped to the interval around 1."
• "The clip stops the policy from changing." (wrong) It only removes the incentive to push the ratio past the interval for improving terms. (right) "The clip bounds the gain from large moves; it does not forbid them."
• "The importance sampling ratio is the hypothesis-testing likelihood ratio." (wrong) One reweights samples; the other compares two parameter values on fixed data. (right) "They share the form and the name likelihood ratio but play different roles."
• "A product of ratios is as stable as one ratio." (wrong) Its variance grows geometrically in the toy example. (right) "Long trajectories make the estimate noisy, so I keep policies close or use a weighted estimator."

Follow-up questions

What is the difference between ordinary and weighted importance sampling?

The ordinary estimator averages the ratio-weighted returns. The weighted estimator divides the sum of the ratio-weighted returns by the sum of the ratios; it is biased for a finite sample but usually has much lower variance, and it is consistent.

Is the importance sampling ratio a probability?

No. It is a ratio of two probabilities or densities, so it can be below or above 1. Its average under the sampling distribution is always 1 when the sampler covers the target, because the expectation of p over q under q is the integral of p, which is 1; the toy example confirms it.¹

What does the hypothesis-testing likelihood ratio compare?

It compares how well two parameter values explain the same fixed data. Related ideas such as maximum likelihood versus maximum a posteriori (MAP) estimation and Bayes' theorem also work with likelihoods, but none of them reweights samples.

How is the importance sampling ratio related to KL divergence?

The Kullback-Leibler divergence is the average, under the target, of the log of target over sampler. So the same ratio, taken as a log and averaged, measures how far two distributions are apart.

How does a plain Monte Carlo estimate relate to this?

A plain estimate such as estimating pi by random points samples straight from the target, so the ratio is 1 everywhere and no correction is needed.

Sources

1. Proximal Policy Optimization Algorithms (arXiv 1707.06347) — arXiv (Schulman, Wolski, Dhariwal, Radford, Klimov), 20 Jul 2017
2. Reinforcement Learning: An Introduction, 2nd edition, Section 5.5 Off-policy Prediction via Importance Sampling — MIT Press (Sutton and Barto), 1 Jan 2018