Exploration vs exploitation: what is the difference?

In this article

1. Before you start
2. Why interviewers ask this
3. What each one is
4. How they differ
5. When each wins
6. Worked example
7. Trade-offs and edge cases
8. How to answer it in an interview
9. Common mistakes
10. Follow-up questions
11. Sources

Before you start

Why interviewers ask this

The dilemma sits under every system that learns from its own choices: which ad to show, which video to recommend, which treatment arm of a trial to enlarge, which move a game agent tries. A team that only exploits locks in whatever looked good after its first few tries and never finds out that something better existed. A team that only explores spends its whole budget gathering information it never uses.

Interviewers ask the question because the answer shows whether a candidate sees that a learner's data depends on the learner's own choices, which is a different problem from fitting a model to a fixed table. A weak answer recites "exploration is trying new things". A strong answer says what each costs in numbers, names one method that balances them, and says how the balance should change over time. The question turns up in reinforcement learning rounds and in recommender-system design rounds, where it returns as the cold-start problem.

What each one is

Exploitation

Exploitation is a choice rule that picks the action whose estimated value is currently the highest, so that the learner collects the best reward its present knowledge can promise.

Exploration

Exploration is a choice rule that picks an action other than the current best estimate, so that the learner gathers information that can improve its estimates and its later choices.

Epsilon-greedy

Epsilon-greedy is a policy that exploits most of the time and, with a small probability epsilon, picks an action completely at random.

The two words in plain terms

Think of a person who eats lunch near one office for a year. Going to the best restaurant they already know is exploitation. Trying a new place on a Friday is exploration. Trying the new place is the only way to learn that a better one exists. The analogy breaks in one place: a learning system is judged on the total over many choices, so a bad exploratory step can pay back many times.

Sutton and Barto, in their textbook on reinforcement learning (second edition draft, read Oct 2026), state the idea directly: if you select a greedy action, "we say that you are exploiting your current knowledge", and if you select a nongreedy one, "you are exploring". They add that exploitation is right for the one step, but "exploration may produce the greater total reward in the long run" (Sutton and Barto, chapter 2). The easiest setting is the bandit. A multi-armed bandit is a row of slot machines (arms) with unknown average payouts and a fixed number of pulls, each pull both a choice and a measurement.

Why does a policy that never explores get stuck?

A learner that never explores only ever updates the estimate of the option it already prefers, and that is a trap with two doors. Estimates start at 0, so the first option that pays anything above 0 beats every untried option, and the learner keeps pulling it until its own average falls below 0. In the run described below, the never-explore policy tried a median of 2 of the 10 options, and in 59.2% of the problems it never pulled the truly best option once.¹

The rarer door is the textbook one: the best option is tried, its first results are unlucky and the learner never returns ("its initial samples of the optimal action were disappointing, and it never returned to it", Sutton and Barto, chapter 2). Here it explained only 95 of the 1,279 problems where greedy ended with the wrong favourite, 7.4%.¹ The result: 36.0% best-arm choices in the last 100 steps and 1.038 per step, against 80.1% for a policy that explored 10% of the time.¹ Exploring corrects a wrong early belief.

Text description of the diagram

• Exploitation picks the action with the highest current estimate and maximises the expected reward of this step.
• Exploration picks another action to learn its value, which may cost reward now and is the only source of new information.
• Using exploitation alone risks locking in a wrong early belief; using exploration alone wastes the budget.

How they differ

The difference shows in what each choice does to the learner's future data.

Regret is the formal way to score the balance. Auer, Cesa-Bianchi and Fischer (Machine Learning, 2002) describe it as the loss due to the fact that the best policy is not followed all the time, and say that Lai and Robbins showed it has to grow at least logarithmically in the number of plays (Auer et al., 2002). In plain words: some wasted pulls are unavoidable, and a good policy wastes fewer and fewer of them as it learns.

When each wins

The decision depends on two facts: how many choices remain, and how uncertain the options are. Two cases show the opposite ends.

Case one: a single choice. A traveller has one evening in a city and one dinner. Nothing learned there can be used again, so an exploratory choice has no later step to repay it. Exploit: pick the best-reviewed place.

Case two: a thousand choices on uncertain options. A product team shows one of ten headlines to each of many visitors and does not know which one converts best. Each visitor is a step, and information gathered early is used on every later visitor. Here a team that exploits too early risks locking in a headline that merely got lucky on its first views. This is the setting of the experiment below.

Text description of the diagram

• An exploratory choice has no later step to repay it, so exploit.
• Many choices with uncertain options justify exploration, and lower exploration as estimates firm up.
• If the options are already well measured, exploit with light exploration in case values change.

Worked example

The setup follows the test that Sutton and Barto call the 10-armed testbed: each of 10 arms has an average payout drawn from a normal distribution with mean 0 and variance 1, and each pull returns that average plus noise with mean 0 and variance 1 (Sutton and Barto, chapter 2). Here the test used 2,000 randomly generated problems of 1,000 pulls each.

On average the best arm in a problem has a true value of 1.535, so 1.535 per step is the ceiling no policy can beat.¹ Six policies faced identical rewards: pure greedy (epsilon 0), epsilon-greedy with epsilon 0.01, 0.1 and 0.3, the upper confidence bound policy (UCB), which adds to each option's average a bonus that is larger the less often that option has been tried, and Thompson sampling, explained in the trade-offs section below.¹

Read the table in three passes. First, greedy is stuck: its reward barely moved from 0.970 in the first 100 steps to 1.038 in the last 100.¹ Second, exploration pays slowly and then keeps paying: epsilon 0.1 was ahead of greedy from the start at 1.023 against 0.970 and widened the lead to 1.365 against 1.038 in the last 100 steps.¹

Epsilon 0.01 started almost level with greedy (0.983) and caught up only gradually, because exploring 1% of the time finds a better option rarely. Third, more is not better: epsilon 0.3 earned 0.873 in the first 100 steps and 1.071 in the last 100, below epsilon 0.1 on both, because it keeps spending 30% of its pulls at random long after the options are known.¹

Text description of the diagram

• Greedy earned 1.030 per step and epsilon 0.1 earned 1.305, against a ceiling of 1.535.
• Epsilon 0.3 fell back to 1.044, below epsilon 0.1.
• UCB earned 1.386 per step and Thompson sampling 1.466, the highest of the six.

Of the policies that explore by uncertainty, UCB earned 1.386 over all 1,000 steps and 1.482 over the last 100, and it chose the best arm 85.7% of the time at the end.¹ It did not start faster: its first-100-step average of 0.957 is below epsilon 0.1. Thompson sampling earned more still, 1.466 over all steps and 1.149 in the first 100, but it was given the testbed's own prior and noise level, which a real team rarely knows, so that gap flatters it.¹

UCB explores in a directed way, trying options whose estimates are most uncertain, instead of at random. Auer et al. show that a policy of this family achieves the logarithmic regret growth mentioned above uniformly over time, not only in the long run (Auer et al., 2002); Sutton and Barto report that UCB generally performs better than epsilon-greedy on this testbed, which this run reproduces. The test ran one set of settings, so a different noise level or number of arms could reorder the policies.

Trade-offs and edge cases

The question is never "explore or exploit" but "how much, and when". Four edge cases come up in follow-ups.

Decay the exploration over time. A fixed epsilon keeps wasting pulls forever, as the epsilon 0.3 row shows.¹ Many systems start with a high rate and lower it as the estimates firm up, so the cost of exploring is front-loaded when information is most valuable.

Random and directed exploration differ. Epsilon-greedy explores blindly: it is as likely to retry an option it has already tried a hundred times as one it has never tried. UCB and Thompson sampling direct the exploration toward uncertainty. Thompson sampling keeps, for each option, a range of plausible values that narrows as the option is tried. On each step it draws one random value from every range, as if each draw were a guess at the truth, and pulls the option whose guess is highest.

An option tried rarely has a wide range, so it often wins the draw and gets tried; an option known to be poor has a narrow range and is mostly skipped. The Russo et al. tutorial (arXiv, 7 July 2017) presents it as an algorithm for sequential decisions that balances exploiting current knowledge for good immediate results with gathering new information to improve future ones (Russo et al., 2017). The version run above used a normal range per option, and the statistics behind ranges that narrow as data arrives are in the statistics hub and the probability questions guide.

Policy-gradient methods explore through the policy itself. Methods such as proximal policy optimization (PPO) and soft actor-critic (SAC) do not have a table of option values to be greedy about. They learn a stochastic policy, a rule that gives each action a probability, and exploration comes from how spread out those probabilities are. PPO's authors say its objective can be augmented with an entropy bonus, a reward for keeping the action probabilities spread out, "to ensure sufficient exploration" (Schulman et al., 2017). SAC builds the spread into the goal: the actor maximises expected reward while also maximising entropy, "to succeed at the task while acting as randomly as possible" (Haarnoja et al., 2018).

The options may change. The test above assumed every arm's value was fixed. When values drift, as with news headlines or seasonal products, an estimate based on old data is wrong, and some exploration has to continue forever. Noisy estimates are also why a lucky early average misleads, the same lean-versus-scatter trade as in bias versus variance.

How to answer it in an interview

The spoken structure is: define both in one sentence each, name the cost of each extreme, give one number, then one method that balances them and how it changes over time.

Common mistakes

"Exploration is just randomness." (wrong) Random choice is one way to explore, and a blind one. (right) "Epsilon-greedy explores at random; UCB and Thompson sampling direct exploration toward the options they know least about, and PPO and SAC explore through a stochastic policy."

"More exploration is always safer." (wrong) Epsilon 0.3 earned 1.044 per step, below epsilon 0.1 at 1.305, in the run above.¹ (right) "I pick the rate on a test, and I lower it as estimates improve."

"The greedy choice is the optimal choice." (wrong) Greedy is optimal for the next step only if the estimates are right, and early estimates are noisy. (right) "Greedy maximises the estimated reward now, and the estimates may be wrong."

"This is only a reinforcement-learning topic." (wrong) It appears wherever a system learns from its own choices. (right) "Ad selection, recommendation and online experiments face the same dilemma."

Follow-up questions

On policy vs off policy?

An on-policy method learns the value of the policy it is actually following, exploration included. An off-policy method learns about one policy, usually the greedy one, while following another that explores. The link to this page is that the exploring policy is the one that generates the data.

What are the differences between the value, return and advantage function?

The return is the total reward collected from a step onward. The value of a state or action is the expected return from it. The advantage is how much better an action is than the average action in that state. Estimating these well is exactly what exploration pays for.

What is the difference between deep Q-learning and Q-learning?

Q-learning stores a value for each state and action pair in a table, while deep Q-learning replaces the table with a neural network so that it works on states too numerous to list. Both still need an exploration rule, and epsilon-greedy is the usual one.

How do you pick the exploration rate?

Treat it as a setting to tune on a test, not a constant. The run above shows the shape: too low gets stuck, too high wastes pulls. Start higher, decay it, and compare policies on identical data, as in an A/B test of the policies themselves.

Thompson sampling vs epsilon-greedy?

Epsilon-greedy explores blindly: with probability epsilon it picks any option, including ones it has already measured many times. Thompson sampling explores by uncertainty: it draws a plausible value for each option from what it has learned and pulls the highest draw, so options with wide uncertainty get tried more and settled poor options fade without a tuned rate. In the run above it earned 1.466 per step against 1.305 for epsilon 0.1, with the caveat that it was handed the testbed's prior and noise level.¹ Epsilon-greedy is simpler, needs no model of uncertainty and is easy to explain; Thompson sampling needs a model of each option's uncertainty (Russo et al., 2017).

How is exploration managed in algorithms like PPO or SAC?

Both learn a stochastic policy, so exploration is built into the action probabilities instead of an epsilon switch. PPO can add an entropy bonus that rewards a spread-out policy (Schulman et al., 2017); SAC puts entropy into its objective, so the actor maximises reward and randomness together (Haarnoja et al., 2018). Their papers describe the bonus or the temperature weight as a setting to tune, which is the same exploration-rate question as above. One caveat: the PPO paper's own continuous-control comparison ran without an entropy bonus, so the bonus is an option, not a requirement.

Sources

1. Reinforcement Learning: An Introduction, second edition (in progress), chapter 2: multi-armed bandits — Sutton and Barto (draft read as a PDF), 24 Oct 2026
2. Finite-time Analysis of the Multiarmed Bandit Problem — Machine Learning 47, 2002 (Auer, Cesa-Bianchi, Fischer), 24 Oct 2026
3. A Tutorial on Thompson Sampling — arXiv (Russo, Van Roy, Kazerouni, Osband, Wen), 7 Jul 2017
4. Proximal Policy Optimization Algorithms — arXiv (Schulman, Wolski, Dhariwal, Radford, Klimov; submitted 2017-07-20; PDF read), 24 Oct 2026
5. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor — arXiv (Haarnoja, Zhou, Abbeel, Levine; submitted 2018-01-04; PDF read), 24 Oct 2026