AI engineer coding interview questions: LRU cache, async batching and retries

In this article

1. Caching
2. Async batch calls and concurrency
3. Retries, backoff and timeouts
4. Provider pools and failover
5. Streaming and numerics
6. Follow-up questions
7. Sources

Caching

Caching questions test data-structure choice and shared state under concurrency. These tasks are reported by candidates in interviews at AI labs and AI startups. They sit beside the tensor-and-training tasks of the machine learning (ML) coding round guide; this page covers the application side. A strong answer in every entry below states the design, the failure case and the cost.

How do you implement a thread-safe LRU cache in Python?

An LRU cache, short for least recently used cache, is a fixed-size store that, when full, evicts the entry untouched for the longest time. In Python, an OrderedDict keeps order: call move_to_end on every hit and popitem(last=False) to evict. Guard reads and writes with one threading.Lock, because a hit changes the order. For production, do not hold the lock while computing a missing value, and expose hit and miss counts. With capacity two, putting a, b, then reading a and putting c evicts b.¹

Why it is asked: it tests data-structure choice, shared state and the habit of naming what breaks under concurrency. Further reading: Python documentation, OrderedDict objects.

How do you add batch operations to an LRU cache for model inference?

Add a get_batch(keys) method that takes the lock once, looks up every key, refreshes the order of each hit, and returns the hits together with the list of misses. The caller sends all misses to the model in one call, then writes the results back with put. Time is proportional to the number of keys k, and space is proportional to the capacity c. Never run the model call while holding the lock, or every other request waits.

Why it is asked: batching is how inference is made affordable, and the question checks that you keep the lock short. Further reading: Python documentation, OrderedDict objects.

Async batch calls and concurrency

How do you write an async batch caller for an LLM API with a concurrency cap, timeouts and retries?

Write one worker coroutine that handles a single request: wait for a slot on an undefined, which is where the concurrency cap is enforced, call the application programming interface (API) under asyncio.timeout, and retry transient failures with backoff. Start one task per request with asyncio.TaskGroup, whose async with block waits for them all on exit (Python documentation, asyncio tasks). Catch errors inside the worker and return them as results, so one failure does not hide the others, and return results in input order.

Why it is asked: it combines concurrency, failure handling and API cost control in one small program. Read the full explanation: Python documentation, coroutines and tasks.

How long does a batch of 100 requests take with a cap of 10 at a time?

A semaphore is a counter that lets only a fixed number of coroutines hold a slot at once; the rest wait. With 100 requests, a cap of 10 and an illustrative latency of 2 seconds each, the requests run in 10 waves, so the minimum wall time is 20 seconds.¹ The formula is the ceiling of requests divided by the cap, times latency. Raising the cap shortens the run until the provider throttles you.

Why it is asked: it checks that you can reason about throughput instead of just adding await. Read the full explanation: Python documentation, asyncio Semaphore.

Retries, backoff and timeouts

How do you retry with exponential backoff and jitter?

Wait longer after each failure, and randomize the wait. Jitter is randomness added to the backoff to spread retries around in time (Amazon Builders' Library, read October 2026). In the full-jitter form defined by Brooker, Amazon Web Services (AWS) Architecture Blog, March 2015, the sleep is a random value between zero and the smaller of the cap and base times two to the power (attempt minus 1), counting the first retry as attempt 1 here. Backoff alone is "good, but not sufficient", because clients stay in step (Brooker, 2015).

Why it is asked: it separates candidates who know why jitter exists from those who copy a loop. Read the full explanation: Brooker, Jitter: Making Things Better With Randomness.

With a base of half a second and a cap of 8 seconds, the caps are below.

The cap doubles with each attempt until it reaches 8 seconds, and each sleep is a random value between zero and the cap. In a seeded simulation of 100 clients retrying at attempt 3, all 100 land in one tenth-of-a-second window without jitter, and the busiest window holds 9 with full jitter.¹

Text description of the diagram

• The cap doubles with each attempt until it reaches 8 seconds.
• Each sleep is a random value between zero and the cap.

Why should only one layer of your system retry?

Retries multiply across layers. Amazon writes: "If each layer retries independently, the load on the database will increase 243x" (Amazon Builders' Library, read October 2026). Amazon's 243 is 3 to the power 5, which counts three attempts at each of five layers. If each layer makes one call plus three retries, the factor is 4 to the power 5, or 1,024.¹ Pick one layer, usually the one closest to the user-visible call, to retry, and let the others fail fast.

Why it is asked: it shows whether you think about the whole call stack and not just your function. Read the full explanation: Amazon Builders' Library, timeouts, retries and backoff with jitter.

Which errors should you retry?

Retry only transient failures, such as a timeout or a "try again later" response, and never errors caused by a bad request, which fail the same way each time. An idempotent operation is one that can be repeated safely with the same effect as doing it once. The Amazon article advises designing APIs to be idempotent so they can be retried safely (Amazon Builders' Library, read October 2026). For calls that are not, send an idempotency token.

Why it is asked: blind retries can charge a customer twice or send a duplicate message. Read the full explanation: Amazon Builders' Library.

Why must a retry loop re-raise CancelledError?

The TaskGroup and asyncio.timeout tools work through cancellation, and the Python documentation warns they might misbehave if a coroutine swallows CancelledError (Python documentation, read October 2026). That exception subclasses BaseException, so except Exception does not catch it, but a bare except does. Catch the specific transient exceptions you retry, and re-raise cancellation untouched.

Why it is asked: it is the classic bug in a retry helper that "works" until a timeout fires. Read the full explanation: Python documentation, coroutines and tasks.

How do you debug and extend a codebase to add an async retry with timeouts?

Read the call path first and find where the failing call is made. Write a failing test, then add one helper that wraps the call with a timeout, a bounded attempt count and full-jitter backoff, and change only that call site. Keep the existing error types, re-raise cancellation, and log each attempt. Say your plan aloud before you edit, because the reviewer is judging how you work as much as the result. The code review interview guide covers the reading habits.

Why it is asked: it mirrors real work on an unfamiliar codebase, not a blank page. Further reading: code review interview questions and answers.

Why must fire-and-forget tasks be stored?

A task started with asyncio.create_task and never referenced can be garbage-collected before it finishes. The Python documentation shows keeping a strong reference by adding each task to a background_tasks set (Python documentation, read October 2026). Remove the task from the set in a done callback so the set does not grow without limit.

Why it is asked: it exposes whether you know how background work can disappear silently. Read the full explanation: Python documentation, coroutines and tasks.

How do you test retry code without waiting?

Pass the sleep function and the random source in as arguments. In a test, give a sleep that only records the requested delays and a seeded random generator, then assert the delays stay between zero and the cap and the attempt count is bounded. Use a fake client that fails a fixed number of times before it succeeds. The test then runs in milliseconds and checks the logic, not the clock.

Why it is asked: injectable time and randomness is the sign of someone who has tested real retry code. Further reading: Brooker, Jitter: Making Things Better With Randomness, on why the sleeps are random.

Provider pools and failover

How do you implement a provider pool for external LLMs with failover?

A provider pool is one object that exposes a single interface in front of several model providers and moves to the next when one fails. Define a small base class with a complete(request) method, and one adapter per provider that maps its request and error types onto yours. The pool tries providers in order, skipping any marked unhealthy, and returns the first success. Keep the retry logic in one place, not in each adapter.

Text description of the diagram

• The pool tries providers in order.
• When every provider fails, the pool raises one error that lists each failure.

Why it is asked: it tests interface design and whether you can keep provider quirks out of callers. Related reading: the ML coding round guide, which covers the tensor side of the same round.

What should the pool do when every provider fails?

When every provider fails, the pool raises one error that lists each failure: it carries the list of each provider's failure, so the caller sees the whole story and can choose a fallback such as a cached answer or a clear message. Do not loop forever and do not return an empty result that looks like success. Bound the total time across all providers, not only each call, so failover cannot turn one timeout into a long hang.

Why it is asked: total failure is the case most solutions forget to write. Further reading: Amazon Builders' Library, on retries under overload.

Should the pool use a circuit breaker?

A circuit breaker is an option with a cost. The Amazon article notes that circuit breakers introduce modal behavior into systems that can be difficult to test, and it describes limiting retries locally with a token bucket instead (Amazon Builders' Library, read October 2026). A strong answer names both, picks the simpler one for the interview, and says what it would monitor.

Why it is asked: it checks whether you weigh a popular pattern against its testing cost. Read the full explanation: Amazon Builders' Library.

Streaming and numerics

How do you parse partial JSON that an LLM is still streaming?

Scan the text one character at a time and keep a stack of what is open: object, array or string. At any point, close the open string, drop an incomplete key or literal, and close each container from the top of the stack down to get a valid JavaScript Object Notation (JSON) value for what has arrived so far. Track escape characters inside strings. Call the function again as each chunk arrives and replace the displayed value.

Why it is asked: streaming output is the normal mode of LLM applications, and this is parsing with an unfinished input. Related reading: the ML coding round guide, which covers the tensor side of the same round.

How do you compute entropy over a stream of counts without overflow?

With counts $c_i$ and total $Z={\sum }_i{c}_i$, the entropy is $H=\log Z-\frac{1}{Z}{\sum }_i{c}_i\log c_i$. Keep only two running sums, the total $Z$ and $S={\sum }_i{c}_i\log c_i$, and never form probabilities or multiply raw likelihoods. When a count $c$ becomes $c+1$, add $(c+1)\log (c+1)-c\log c$ to $S$ and add 1 to $Z$. Use compensated summation, such as math.fsum, to limit accumulated rounding error in the sums; it does not remove the cancellation in the final subtraction when the two terms are close. The same stability idea underlies a numerically stable softmax.

Why it is asked: it tests numerical habits that matter whenever probabilities get small. Read the full explanation: numerically stable softmax.

Follow-up questions

What are AI engineer coding interview questions?

They are coding tasks about building applications around models: caches, async batch calls, retries, failover and streaming parsers, as listed above. They differ from questions about writing a model layer, which the guide to building a generative pre-trained transformer (GPT) from scratch covers.

Are AI engineer Python coding interview questions different from machine learning engineer ones?

Often in emphasis. The questions here stress reliability and concurrency in Python services, while machine learning engineer rounds more often write tensor code, as in beam search or attention. The same loop can contain both, so check the role description.

Can you use an AI coding assistant in an AI coding interview?

Rules differ by company and round, and this page has no public source for any company's format. Ask the recruiter in writing whether tools are allowed, and follow the answer exactly. Where an assistant is allowed, a strong answer shows that you review generated code line by line, explain why you keep or change it, and test it yourself instead of trusting it.

How do you practice these questions?

Write each one from a blank file, run it with a fake client that fails on purpose, then explain it aloud in under two minutes. A rehearsal out loud shows where your explanation of the trade-off is thin.

Sources

1. Timeouts, retries, and backoff with jitter (Amazon Builders' Library) — Amazon Web Services (Marc Brooker); undated page, read date given, 26 Oct 2026
2. Coroutines and tasks (Python documentation) — Python Software Foundation; undated build, read date given, 26 Oct 2026
3. Jitter: Making Things Better With Randomness — Marc Brooker's blog, 21 Mar 2015
4. Exponential Backoff And Jitter (AWS Architecture Blog) — Amazon Web Services (Marc Brooker), 4 Mar 2015
5. Synchronization primitives (Python documentation, asyncio Semaphore) — Python Software Foundation; undated build, read date given, 26 Oct 2026
6. collections, OrderedDict objects (Python documentation) — Python Software Foundation; undated build, read date given, 26 Oct 2026