In this article
Before you start
This page is one topic in the distributed systems interview round. It assumes you can follow a rough load calculation; the back-of-the-envelope estimation page teaches that.
Why interviewers ask this
A notification system tests whether a candidate can turn one big, bursty job into many small, safe ones. Candidates report this prompt in several forms in interviews at large software companies: deliver the emails of a marketing campaign, build something like the push gateways of the phone platforms, or send notifications for a streaming app. The scale in these reports is large: up to 100,000 emails per campaign and millions of campaigns (figures as reported by candidates, illustrative).
A weak answer draws a box called "notification service" and says it calls the email and push providers. A strong answer explains the queue between the campaign and the senders, what happens when a worker dies halfway through a batch, how a retry avoids a double send, and why a password reset must not wait behind a promotion. A Staff-level answer adds the feedback loop (bounces and complaints) and the compensation story when a campaign is cancelled midway.
A notification system from first principles
- Notification system
- A notification system is a service that accepts requests to tell users something, decides which channel and priority each message gets, and delivers it through push gateways, an email provider or other senders while handling retries, rate limits and failures.
- Suppression list
- A suppression list is a store of addresses and device tokens that must not be contacted again, filled from bounces, complaints and unsubscribes, and checked before every send.
In plain words
Picture a post office that gets a bag of ten thousand letters. It does not hand the whole bag to one courier. Clerks split the bag into trays and put each tray on a shelf. Several couriers pick up trays at their own speed. A tray is stamped when it is delivered. If a courier collapses, another takes the tray and skips the letters already stamped. The analogy breaks in one place: a post office sees a letter arrive, while a notification system often learns of a failed delivery later, from a bounce, or never.
In one precise sentence, a notification system turns one request into many small sends. Each send can be retried alone, carries an idempotency key and is throttled to what the provider allows.
Step 1: A campaign becomes batches on a queue
The campaign service does not send anything. It stores the campaign (content, audience, schedule), answers "accepted" and hands the work to a fan-out job, which reads the audience and cuts it into batches placed on a queue. A batch is a fixed number of recipients, for example 500 (illustrative), so one slow or failing batch affects few people.
Input: a campaign request. Output: many small batch jobs on a durable queue. The queue gives you three things: the campaign service answers fast, senders can be added or removed freely, and a crash loses nothing because jobs stay on the queue until acknowledged. A log such as Apache Kafka is a common choice; its documentation describes partitions and consumer groups, the mechanism that lets many workers share one stream of jobs. Key the batches by campaign so progress is easy to count. You now have work divided safely; the next step makes each send safe to repeat.
Step 2: An idempotency key makes retries safe
Most queues deliver at least once by default, and a call to an outside provider cannot be made exactly once, so a worker can send, crash before acknowledging, and see the same job again. Without protection that is a duplicate email or push. The fix is an idempotency key for each intended send, built from the campaign id and the recipient id, kept in a store with a state. The worker inserts the key as pending with an atomic "insert if absent", calls the provider, then marks the key sent once the provider confirms. If the queue redelivers a job whose key is sent, the worker skips it.
Input: a batch job. Output: no second send after a confirmed send, even across worker crashes. The ambiguous case is a key still pending after a timeout, because the provider may have accepted the message without confirming. There you either retry, which risks a duplicate and is the usual marketing choice, or skip, which risks a miss. A provider that accepts a deduplication id removes the gap. Money-moving systems design more carefully, as the payment system page shows. The same key idea protects the campaign application programming interface (API): a client that retries "create campaign" gets the same campaign back.
Step 3: Separate queues keep urgent messages urgent
Transactional messages (a login code, a receipt) and marketing messages (a weekly promotion) must not share a queue. A promotion to a very large audience would sit in front of the login code and delay it. Give each class its own queue and its own workers, and give the urgent class reserved capacity.
Input: a message with a class. Output: a queue chosen by priority. The phone gateways support this idea themselves: the documentation of Apple Push Notification service (APNs) and of Firebase Cloud Messaging (FCM) on message priority and collapsible messages describe priority levels, an expiry, the time to live (TTL), and collapsing of repeated messages. Use them: a login code gets high priority and a short expiry, because it is useless after a few minutes, while a promotion tolerates a normal priority and a longer expiry. Collapsing lets a newer "score update" replace an older one the user never saw.
Text description of the diagram
- Transactional messages go on a high-priority queue with short expiry.
- Marketing messages go on a separate throttled queue.
Transactional messages go on a high-priority queue with short expiry, and marketing messages go on a separate throttled queue.
Step 4: Throttled workers send through the gateways
Workers pull batches, skip suppressed recipients, check the idempotency key and call the channel provider. The provider sets the pace. Amazon documents that its email service limits sends per second and per day through sending quotas, so the sender must throttle itself rather than hammer the provider and collect errors. A shared rate limiter, like the one in the distributed rate limiter page, lets many workers respect one limit.
Push and email fail in different ways. Push goes to the platform gateways, which are best-effort: a message can expire or be collapsed, and the device may be off, so delivery is never promised. Email goes over the Simple Mail Transfer Protocol (SMTP), whose specification separates temporary failures, which should be retried later, from permanent ones, which should not. Retry temporary failures with growing delays; after a limit, move the job to a dead-letter queue. A dead-letter queue is a holding queue for jobs that keep failing, so a person or a repair job can inspect them. You now have a sender that is slow on purpose and safe to repeat.
Step 5: Bounces and complaints feed the suppression list
Delivery does not end at the send call. Providers report later events: a mailbox that does not exist, a recipient who marked the message as spam, a device token that is no longer valid. These arrive as callbacks, in the style of a webhook, or as feedback from the gateway. Each one writes to the suppression list, which every worker checks before sending.
Input: delivery events. Output: a list that shrinks future sends. Skipping dead addresses protects your sender reputation, which decides whether your email reaches inboxes at all, and avoids sending to people who asked to stop. The event stream also feeds campaign statistics, so use a separate store for those counts rather than the send path.
Step-by-step description
- A campaign becomes batches on a queue.
- Each send gets one key.
- Urgent and marketing traffic separate.
- Workers send through the gateways.
- Bounces feed the suppression list.
- A campaign becomes batches on a queue.
- Each send carries an idempotency key.
- Bounces flow back into a suppression list.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take one campaign of 100,000 emails and assume each worker holds one connection that may send 14 emails per second (illustrative, a per-worker rate, not an account limit). One worker takes about 119 minutes for 100,000 emails at 14 sends per second, because 100,000 divided by 14 is 7,142.86 seconds.1 The provider's account quota is the true ceiling: it must be at least the number of workers times 14 per second, otherwise the fleet is capped by the quota. With a 14 per second account quota, extra workers add nothing. The model below assumes the quota is high enough.
Ten workers take about 11.9 minutes.1 Cut into batches of 500, the campaign is 200 batches.1
Text description of the diagram
- One worker takes about 119 minutes for 100,000 emails at 14 sends per second.
- Ten workers take about 11.9 minutes.
Now the duplicates. Suppose a worker crash or queue redelivery hands 3% of jobs back after their send was already confirmed (illustrative). Without an idempotency key that sends 3,000 duplicate emails in this one campaign; with the key, zero.1 A timeout after the provider accepted a message but before it confirmed remains a possible duplicate under the retry policy.
Now the whole platform, with illustrative assumptions: 50,000 campaigns a day, 20,000 recipients on average, so 1,000,000,000 messages a day.1 That averages 11,574 sends per second. If the peak is five times the average, it is 57,870 per second, which needs about 4,134 workers at 14 sends per second each, and a provider quota of at least 57,870 per second.1 A log row of 200 bytes per message (illustrative) adds 200 gigabytes a day, or 6,000 gigabytes over 30 days of retention.1
| Quantity (illustrative inputs) | Value |
|---|---|
| One campaign, one worker at 14 per second | 119 minutes |
| One campaign, ten workers | 11.9 minutes |
| Batches of 500 per 100,000 recipients | 200 |
| Duplicates from 3% redelivered jobs, no key | 3,000 |
| Messages per day at 50,000 campaigns of 20,000 | 1,000,000,000 |
| Average sends per second | 11,574 |
| Peak sends per second at five times average | 57,870 |
| Workers needed at peak | 4,134 |
| Log storage per day at 200 bytes a row | 200 gigabytes |
Trade-offs and edge cases
Push or pull. Push delivery uses the platform gateways, which wake the device; pull has the app ask your server for new items. Push gives freshness but is best-effort; pull is reliable but costs battery and requests. A common answer is both: push as a nudge, pull to fetch the full content, so the payload stays small and a missed push is repaired the next time the app opens.
Exactly once. End-to-end exactly-once delivery is not achievable across a provider you do not control. Aim for at-least-once delivery with idempotent handling, as the Kafka design documentation frames the delivery semantics.
Cancelling a campaign. A campaign touches several services, so a single database transaction cannot cover it. A saga is a sequence of local transactions, each with a compensating action that undoes it if a later step fails; see the saga pattern in the Azure Architecture Center. Here the compensation is "stop the remaining batches and mark the campaign cancelled", because an email already sent cannot be taken back.
Tokens, preferences and caps. Apps register a device token with your server at each launch. When a gateway reports a token as invalid, the registry prunes it and adds it to the suppression list. A preference service stores opt-outs by channel and category, plus per-user frequency caps such as a daily maximum of marketing pushes (illustrative). Workers check preferences and the suppression list before every send. Templates fill personalisation fields at send time from recipient data, and schedules keep each recipient's time zone, so a morning-local-time campaign fans out across many hours.
Data store. The send log is write-heavy and append-only, so a wide-column or log-structured store fits. The idempotency store needs a key-value store with a conditional put, which is the atomic insert-if-absent. Campaigns and preferences are relational data. These are different needs, so use different stores; the database interview questions cover how to choose.
Slow tails. A few slow batches decide when a campaign finishes, so watch the slowest ones, not the average; the tail latency page explains why.
How to answer it in an interview
Design a system that delivers campaign emails, up to a hundred thousand per campaign and millions of campaigns.
I would not send from the campaign service. It stores the campaign, answers accepted, and a fan-out job cuts the audience into small batches on a durable queue. Workers pull batches, check an idempotency key per campaign and recipient, skip suppressed addresses and send through the email provider at a throttled rate. Transactional and marketing traffic get separate queues. Queues redeliver jobs, so the key is recorded as pending and marked sent after confirmation, which stops duplicates after a crash. Bounces and complaints feed a suppression list, and cancelling a campaign is a saga that stops the remaining batches.
The spoken structure: say why the queue is there, then the key, then priorities and throttling, then the feedback loop and the failure story.
Common mistakes
- "The campaign service calls the email provider directly." (wrong) A burst overloads the provider and a crash loses progress. (right) "The service enqueues batches and throttled workers send."
- "Retrying is always safe." (wrong) A retry after a timeout can duplicate a send. (right) "Every send carries an idempotency key, checked and recorded around the call."
- "Marketing and login codes can share a queue." (wrong) A large promotion delays the urgent message. (right) "Separate queues, with reserved capacity for transactional traffic."
- "Push always arrives." (wrong) The gateways are best-effort, with expiry and collapsing. (right) "Push is a nudge; the app can pull to repair a miss."
- "We can get exactly-once delivery." (wrong) The provider can accept and fail to confirm. (right) "At-least-once delivery plus idempotent handling."
Follow-up questions
How would you design a system like APNs or FCM that delivers billions of push notifications?
Treat it as the same pipeline with a long-lived connection layer: devices keep a connection to an edge fleet, a token-to-connection map finds the right edge server, and messages wait in per-device storage with an expiry until the device is reachable. The gateway stores and forwards the latest message per collapse key until the device reconnects or the expiry passes, and the connection fleet is sized by dividing the devices online by what one server can hold. Scale by partitioning devices across servers, for example with the consistent hashing of a distributed cache design.
How do you send up to 100,000 campaign emails when there are millions of campaigns?
Per campaign it is a small job; in total it is a throughput problem. Estimate peak sends per second, divide by the provider's rate to size the worker fleet as in the worked example, and keep campaigns isolated so one huge campaign cannot starve the others, for instance with per-tenant limits.
How do you handle distributed transactions and the saga pattern in a campaign system?
Avoid one big transaction. Split creation, audience snapshot, scheduling and sending into local steps, each with a compensation such as "cancel the remaining batches". Make every step idempotent so a replay after a crash is harmless.
What is resiliency, and how do horizontal and vertical scaling differ?
Resiliency is the ability to keep working and recover when parts fail, using queues, retries, idempotency and isolation. Vertical scaling makes one machine bigger; horizontal scaling adds more machines, which this design needs because workers are stateless and share work through the queue.
Push or pull: how do clients get notifications?
Both. Push wakes the device through the platform gateway; pull lets the app fetch the content and catch up on anything missed. Sending an id by push and fetching the body by pull also keeps the push payload small. The API design interview questions and backend interview questions cover the neighbouring topics.
- A campaign is accepted as a job and cut into small batches on a durable queue.
- Queues redeliver jobs, so an idempotency key per campaign and recipient, recorded around the send, prevents duplicate sends after a crash.
- Transactional and marketing traffic get separate queues, and gateway priority, expiry and collapsing support that split.
- Workers throttle themselves to the provider's rate limits.
- Bounces and complaints feed a suppression list, and cancelling a campaign is a saga.
Sources
- Sending notification requests to APNs — Apple Developer Documentation, 26 Oct 2026
- Setting the priority of a message (Firebase Cloud Messaging) — Firebase (Google), 26 Oct 2026
- Collapsible message types (Firebase Cloud Messaging) — Firebase (Google), 26 Oct 2026
- Saga pattern — Microsoft Azure Architecture Center, 26 Oct 2026
- Amazon SES: managing your sending quotas — Amazon Web Services Documentation, 26 Oct 2026
- Apache Kafka documentation: design and message delivery semantics — Apache Software Foundation, 26 Oct 2026
- Simple Mail Transfer Protocol (RFC 5321) — IETF, 1 Oct 2008
