In this article
Before you start
This page is one step of the system design interview path.
Why interviewers ask this
Designing a job scheduler is the one prompt that forces you to talk about time, failure and duplicates together. Candidates report prompts such as "design a job scheduler", "design a system to schedule jobs in a distributed environment", "design a distributed priority queue" and "design a continuous-integration (CI) system" in system design rounds at AI labs and large tech companies.
A weak answer draws a database, a queue and some workers, then says the system runs each job exactly once. A strong answer says who decides that a job is due, what stops two machines from both firing it, and what a worker crash does to a half-finished task. A Staff-level answer adds the policy question: for this job, is a skipped run or a doubled run worse? The same reasoning shows up in payment systems and in the wider backend interview questions.
A distributed job scheduler from first principles
- Launch record
- A launch record is a durable entry, keyed by job identifier and scheduled start time, that notes a launch is about to happen and later that it finished, so a new leader can tell which launches are still open after a failover.
- Lease
- A lease is a time-limited claim of ownership that the holder must keep renewing. If the renewals stop, the claim expires and another node may take it, which is how a cluster replaces a leader that has died.
- Visibility timeout
- A visibility timeout is the period after a worker receives a queue message during which other workers cannot see it. If the worker does not delete the message in time, it reappears and another worker can take it.
In plain words
Think of a restaurant kitchen with a ticket rail. A manager reads the booking list and puts a ticket on the rail when a table is due. Cooks take tickets and cook. The manager is the scheduler. The cooks are the workers.
The analogy breaks in one place. A kitchen has one manager who cannot vanish mid-sentence. In a distributed system the manager can crash after taking the booking list and before hanging the ticket. Nobody knows if the ticket was hung. Everything below exists to answer that doubt.
In one precise sentence, a distributed job scheduler keeps jobs in a durable store, lets exactly one node fire each due job, and runs the task on workers with at-least-once delivery.
Step 1: A durable store holds every job and its next run time
A durable store holds every job and its next run time, so a crash loses no schedule. Each row holds the job identifier, a schedule such as a cron expression or an interval, the next time it is due, a payload and a retry policy. A second table holds launch records: one row per job and start time.
Input: a job definition. Output: a row that survives machine loss. The store is replicated; Google's cron service keeps its state on several replicas and uses the Paxos consensus algorithm (Google's site reliability engineering (SRE) book, Chapter 24). A single relational database with failover is the simpler answer for most interviews. The key is to identify each launch by job plus start time, because the SRE chapter warns that otherwise launch tracking becomes ambiguous. You now have a place to remember what is due and what already ran.
How does the trigger find due jobs and move the schedule forward?
The trigger finds due jobs by polling the job store through an index on next_run_time, and it moves each job's schedule forward in the same transaction that records the launch. One common approach is a row-locking read that skips rows other transactions already hold (in many databases a select ... for update skip locked query), so two trigger workers never take the same row; the alternative is a time-bucketed scan that reads one slice of time at a time.
In that one transaction the trigger inserts the launch record and computes the next next_run_time from the cron expression, and a relay then publishes launched records to the queue. If the transaction fails, nothing moved and the job is still due. A misfire policy decides what happens to runs missed during an outage: fire once now, fire every missed run, or skip to the next slot. That is the skip-or-double choice, made per job. The minimal application programming interface (API) is create, update, pause and cancel a job, plus a query for a run's status and history.
Step 2: One leader or one row lock fires each due job once
One leader or one row lock fires each due job once. Running plain cron on several machines does not give high availability: without coordination every node fires every job.
There are two standard designs. In the first, replicas elect a leader, and the leader is the only replica that actively launches jobs (SRE book). Leadership is held by a lease. In the second, every node shares one database, and the first node to place a lock on a trigger fires it; only one node fires per firing (Quartz clustering docs).
Around each firing, write the launch record twice: before launching and after finishing. In the Paxos leader design the write goes through a quorum, which is a majority of replicas; in the shared-database design the record is a row written in the same transaction as the lock. After a failover the new leader reads the open records and finishes or skips them. Google also requires external operations to be idempotent or to have state it can look up.
Step 3: Workers run each task at least once from a queue
Workers run each task at least once from a queue. The scheduler does not run the code itself; it puts a message on a queue, and a pool of workers pulls messages and executes them. That keeps firing cheap and lets the worker pool scale on its own.
In Amazon Simple Queue Service (SQS), a received message cannot be seen by other consumers for the visibility timeout, 30 seconds by default. If the worker does not delete the message before the timeout, it becomes available again and another consumer can retrieve it. The documentation also warns that a message can still be delivered more than once.
So delivery is at-least-once, and a task must be idempotent. The word idempotent is used for a task where running it twice leaves the same result as running it once. A common technique is a key made of job identifier and start time, checked before the side effect. The visibility timeout is not a lock; it only hides the message.
Step 4: Heartbeats, retries and a dead-letter queue handle failure
Heartbeats, retries and a dead-letter queue handle failure. A long task extends its visibility timeout with a heartbeat, a periodic call that says "still working", which the SQS documentation recommends. The extension has a ceiling of 12 hours from first receipt, so a longer task must save checkpoints and split itself.
When a task fails, retry with exponential backoff, which doubles the wait after each attempt, and add jitter, a small random spread, so that retries of many failed tasks do not arrive together. After a set number of attempts, move the message to a dead-letter queue (DLQ), a side queue that holds messages that keep failing so a human can inspect them instead of the system retrying forever.
Crash recovery differs by design. In Quartz, jobs flagged as requesting recovery are re-executed by the remaining nodes; other jobs wait for their next trigger. You now have the whole system: store, single firing point, queue and workers, and a failure path.
Step-by-step description
- Jobs and next run times survive a crash.
- One leader or lock fires each job.
- Tasks run at least once.
- Repeated failures go to the dead-letter queue.
- A durable store holds every job and its next run time.
- One leader or one row lock fires each due job once.
- Workers run each task at least once from a queue.
- Retries with backoff end in a dead-letter queue.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Sizing shows why the shape above is enough. All inputs are illustrative assumptions, not sourced facts: 10,000,000 jobs per day, a peak five times the average, 200 firings per second that one lock domain can handle, one kilobyte per job record and thirty days of history. The average load is about 115.74 firings per second and the peak about 578.7 per second, which needs three lock domains, so the job set is split into three partitions.1 This matches the Quartz advice to partition jobs across several schedulers once the cluster-wide lock becomes the limit. Thirty days of records come to about 300 GB.1
For failover, assume a 15-second lease renewed every 5 seconds and a 2-second election. A dead leader is replaced after at most 17 seconds, far inside the one-minute failover that Google calls tolerable; the leader can miss two renewals in a row without losing the lease.1
| Quantity | Value (computed from illustrative inputs) |
|---|---|
| Average firings per second | 115.74 |
| Peak firings per second (five times average) | 578.7 |
| Lock domains needed at 200 firings per second each | 3 |
| Job records kept for thirty days | 300 GB |
| Worst-case failover (lease 15 s, election 2 s) | 17 seconds |
| Copies of a 90-second task running by second 89, no heartbeat | 3 |
| Copies with a heartbeat every 10 seconds | 1 |
| Retry waits, doubling from 2 seconds, five attempts | 2, 4, 8 and 16 seconds, 30 seconds in total |
In an illustrative case, a 90-second task with a 30-second visibility timeout is running on three workers by second 89; with a heartbeat it stays on one worker.1 After five failed attempts, with waits of 2, 4, 8 and 16 seconds, the message goes to the dead-letter queue.1
Text description of the diagram
- A 90-second task with a 30-second visibility timeout is running on three workers by second 89.
- With a heartbeat it stays on one worker.
Trade-offs and edge cases
The central trade-off is missing a launch against doubling it. Google states a general preference for skipping launches over risking double launches, as far as the infrastructure allows. That is a per-job policy: a garbage-collection job can skip a run, a newsletter must not go out twice, and payroll must not be skipped.
Text description of the diagram
- A leader lease and a shared-database lock both fire each job once.
- Plain cron on every node fires every job on every node.
Edge cases follow from the table. A deposed leader must stop acting at once, because holding leadership is what gives mutual exclusion toward the executor (SRE book); check the lease before every launch. That check alone is not enough: a leader can pass it, pause (in garbage collection or a slow network call) and launch after its lease expired and a new leader exists. The guard is a fencing token. Each lease carries a monotonically increasing epoch number, the leader writes it into the launch record and into every write to the queue or executor, and the store or executor rejects any write carrying an epoch lower than the highest it has seen.
Leases also assume that clock rates drift only a bounded amount, so leave margin between renewal and expiry. Clock skew breaks lock designs too: Quartz says clustered machines need clocks within a second of each other. More nodes do not mean more throughput, since the same documentation says the cluster-wide lock degrades performance beyond about three nodes, so partition the jobs instead. Duplicates are normal, so see concurrency interview questions for guarding shared state, and database interview questions for row locking. Hashing job identifiers to partitions follows the same idea as consistent hashing in a distributed cache. A task that calls out through a webhook needs the same idempotency key on every retry.
How to answer it in an interview
How would you design a distributed job scheduler?
I would split it into a store, a trigger layer, a queue and workers. Jobs and next run times live in a replicated database. One leader, held by a lease, or a row lock per trigger, fires each due job once, and I write a launch record before and after. Fired jobs go on a queue; workers hide a task for a timeout and extend it with a heartbeat. Delivery is at least once, so every handler is idempotent, keyed by job and start time. Failures retry with backoff and end in a dead-letter queue. For a missed launch, I ask whether skipping or doubling is worse for that job.
The spoken structure: name the four parts, say who fires and why only one node can, admit at-least-once delivery, then give the failure path and the skip-or-double policy.
Common mistakes
- "The scheduler runs each job exactly once." (wrong) The queue can redeliver a message. (right) "Delivery is at least once, so handlers are idempotent."
- "I will run cron on three machines for availability." (wrong) Every node fires every job without coordination. (right) "One leader or a trigger lock fires each job."
- "The visibility timeout locks the task." (wrong) It only hides the message and can expire while the worker still runs. (right) "I extend it with a heartbeat and keep handlers idempotent."
- "Always run a missed job late." (wrong) Some jobs must not run twice, and some can skip. (right) "I set skip or catch-up per job."
Follow-up questions
How do you design a job scheduler, with follow-ups on distributed execution, fault tolerance and orchestration internals?
Separate firing from execution: a leader or lock fires each due job once into a queue, and a worker pool runs it. Make the store replicated, the launch record two-phase, the handlers idempotent and failures end in a dead-letter queue. For orchestration, jobs with dependencies form a directed acyclic graph (DAG); a step is enqueued only when all its parents have succeeded, and run state is tracked per step so a failed step can be retried alone.
How do you design a CI system similar to GitHub Actions?
Treat each workflow run as a job whose steps are tasks forming a graph, and the machines that execute them as runners that register with a label and pull matching work from a queue. Store logs and artifacts per step, and give each step its own timeout and a cancellation path. The same concerns apply: at-least-once delivery, heartbeats for long runs, and a cap on retries. Say that you are designing from first principles, not describing any vendor's internals.
How do you design a distributed priority queue for job processing with multiple consumers?
One simple approach is a separate queue per priority class, with consumers polling the high class more often than the low class so that low-priority work is not starved. Each consumer still needs a visibility timeout and idempotent handlers.
Should a missed job run late, be skipped, or run twice?
It depends on the job. Prefer skipping to doubling when a double run is harmful, and write the choice per job.
Why not run cron on every machine?
Without coordination every machine fires every job, so each run happens once per machine. A leader lease or a trigger lock restores a single firing. Sharing state across nodes through one store is the same kind of problem as in a distributed rate limiter, and when you roll out a new scheduler version one node at a time, as in deployment strategies, the lease, not the rollout, decides which node is leader.
What stops an old leader that wakes up after a pause?
A fencing token. Each lease carries an increasing epoch, the leader attaches it to every launch record and queue write, and the store or executor rejects any write with an epoch lower than the highest it has seen.
- A scheduler stores jobs durably and lets exactly one node fire each due job.
- Use a leader lease or a trigger row lock, and write a launch record before and after.
- Queue delivery is at least once, so make every handler idempotent.
- Extend long tasks with a heartbeat, retry with backoff, and park failures in a dead-letter queue.
- Partition jobs across schedulers instead of adding lock-sharing nodes, and choose skip or catch-up per job.
Sources
- Chapter 24: Distributed Periodic Scheduling with Cron (Site Reliability Engineering, April 2016) — Google (sre.google), accessed 2026-10-26, 1 Apr 2016
- Amazon SQS visibility timeout — Amazon Web Services documentation, accessed 2026-10-26, 26 Oct 2026
- Configure Clustering with JDBC-JobStore (Quartz 2.3.0) — Quartz Scheduler documentation, accessed 2026-10-26, 26 Oct 2026
