Design a notification system

Every app eventually needs to tap its users on the shoulder: “Your OTP is 482913”, “Ravi is 5 minutes away with your biryani”, “₹1,240 paid to Sharma Medicals”, “Big Billion Days starts at midnight”. A notification system is the shared platform that does this for every product team, over push (through Apple’s and Google’s push services), SMS, email and an in-app inbox. On a whiteboard it looks like a queue in front of a few vendor APIs. It gets interesting because two very different kinds of traffic share the same pipes — one urgent OTP and a sale announcement to 100 million people — because the last mile belongs to providers with their own quotas and bad days, because users punish every unwanted buzz by turning notifications off, and because “deliver exactly once to a phone” is a promise no design can keep. In this problem you design that platform for a super-app with 300 million users: food delivery, payments and shopping in one.

Understand the problem

Start by agreeing who the customers are. They are not end users; they are a couple of hundred internal teams — orders, payments, rides, marketing, security — who today each call an SMS vendor or Apple’s push service themselves, each with their own retry bugs and their own idea of how many promotional pushes a day is too many. The platform gives them one API and takes the hard parts away. Then ask how the traffic differs, because nearly every decision follows from it.

Functional requirements

1. Product teams should be able to send a notification to a user, not to a device. They pass a user ID, a notification type and parameters; the platform finds the user’s phones, number and email address and picks the channels — push, SMS, email, in-app inbox.
2. Notifications should be written once as templates and arrive in the user’s language. Each type has a versioned template with a variant per channel and per locale (English, Hindi, Tamil, …).
3. Users should be able to control what they receive. They opt out per category and channel, set quiet hours, and are protected by frequency caps. Security messages cannot be muted.
4. Marketing should be able to schedule campaigns to tens of millions of users — for a segment, at a time, at a controlled pace, and with a working “stop” button.
5. Senders and users should be able to see what happened. Every notification has a status (accepted, sent, delivered, opened, failed), and the app has an inbox with read and unread items that updates live.

Non-functional requirements

• Scale. About 3B notifications a day — 1B transactional, 2B from campaigns — which become about 4B channel deliveries.
• Throughput. Transactional peaks around 50k/s (dinner time on a sale day), plus campaign fan-out to 100M users within 30–60 minutes, on top.
• Latency. OTP: accepted to handed to the SMS provider in under 500 ms at p99, and on the phone within a few seconds when carriers are healthy. Order updates: under 5 s at p99. Campaigns: minutes are fine, but the platform must hold the pace it promised.
• Durability. Once the API has returned 202, the notification is not lost: it is on several replicas before the API answers.
• No duplicate OTPs or receipts in normal operation; the rare duplicate after a crash should collapse on the device where the platform allows.
• Isolation. A campaign, or one team’s runaway loop, must not move transactional p99 latency.
• Availability. 99.99% for the accept API; each channel and provider degrades on its own.
• Respect and compliance. Opt-outs take effect within a minute; caps and quiet hours hold across all teams; email and SMS follow bulk-sender and Indian telecom rules.

Out of scope

• The transports themselves. APNs, FCM, SMS aggregators and the email delivery provider are external; we call their APIs. Running a mail service is its own problem.
• Segmentation and the campaign UI. Marketing tools produce a segment, a list of user IDs; we receive its ID.
• Deciding which notification is worth sending. Ranking candidates by predicted value is an ML system; we leave a hook for it (see the follow-up questions).
• Chat delivery. Messaging apps deliver over their own persistent connections and use push only to wake an offline phone — see design WhatsApp.

The set-up

Plan the approach

Say the shape out loud: accept durably and fast; decide once per user — where, in what words, and whether at all; send through per-channel workers; record what happened. Build it one requirement at a time, then spend most of the interview in the deep dives: priority isolation, campaign fan-out, providers, duplicates, preferences, and tokens and tracking.

Core entities

• Notification type. Registered once per use case, such as auth.otp or order.out_for_delivery: category, priority, allowed channels, time to live (TTL), collapse key and whether users may mute it. Callers pick a type, never a priority, so nobody can label a sale as urgent.
• Notification. One logical message to one user: a time-sortable ID from the unique ID generator, type, template parameters, sending team, idempotency key and an expires_at.
• Delivery. One target per channel and device: notification ID, channel, target (device token, phone number or email address), provider, attempt count, status and the provider’s message ID.
• Template. Versioned content per type, with variants per channel and locale; the SMS variant carries the template ID registered with the telecom operators.
• Contact book. Per user: devices (platform, token, app version, notification permission, last seen), verified phone and email, language and time zone.
• Preference. Per user, category and channel: on or off; plus quiet hours and a global mute for promotions.
• Campaign. Segment snapshot, template, schedule, pacing, state (scheduled, running, paused, cancelled, done) and progress.
• Inbox item. The in-app copy of a notification, with its read state.

The type registry is configuration that product teams submit for review. Three entries show how different the types are:

- type: auth.otp
  category: security          # cannot be muted
  priority: P0
  channels: [sms]
  ttl: 5m                     # useless after this: drop, never retry
  on_unknown_outcome: give_up # at-most-once; the user can tap "Resend"

- type: order.out_for_delivery
  category: order_updates
  priority: P1
  channels: [push, inapp]
  fallback: { sms: if_no_active_device }
  ttl: 30m
  collapse_key: "order:{order_id}"   # a newer update replaces an older one
  on_unknown_outcome: retry          # at-least-once

- type: promo.sale_announcement
  category: promotions
  priority: P2
  channels: [push, inapp, email]
  ttl: 6h
  respects: [quiet_hours, frequency_cap]

API

Product teams call the send endpoint; it is asynchronous by design.

POST /v1/notifications
Authorization: Bearer <service token for orders-svc>
Idempotency-Key: orders-svc:O-5521:out_for_delivery
Content-Type: application/json

{
  "type": "order.out_for_delivery",
  "user_id": "u_81723",
  "params": { "order_id": "O-5521", "rider": "Ravi", "eta_min": 12 },
  "channels": ["push", "inapp"]
}

HTTP/1.1 202 Accepted
Content-Type: application/json

{ "notification_id": "1852331870447001600", "status": "accepted" }

202 means “durably accepted”, not “delivered”: the notification is on the replicated log and will be processed even if every server that touched it dies a millisecond later. A retry with the same Idempotency-Key within 24 hours returns the same ID and sends nothing new. A missing template parameter or an unknown type is a 400 now, not a broken push to ten million phones later. A team over its quota gets 429.

Campaigns have their own endpoints:

POST /v1/campaigns
Content-Type: application/json

{
  "type": "promo.sale_announcement",
  "segment_id": "seg_bbd_2026_active",
  "template_version": 7,
  "send_at": "2026-10-28T19:07:00+05:30",
  "deliver_within": "45m",
  "respect_quiet_hours": true
}

POST /v1/campaigns/cmp_9921:pause
POST /v1/campaigns/cmp_9921:cancel

And the rest of the surface, for apps, users and providers:

GET  /v1/notifications/{id}               status per channel and device
PUT  /v1/users/{id}/devices/{device_id}   register or refresh a push token (every app start)
PUT  /v1/users/{id}/preferences           category × channel switches, quiet hours
GET  /v1/users/{id}/inbox?cursor=…        newest first, 20 per page
POST /v1/users/{id}/inbox:markRead        body: up_to = <notification_id>
POST /v1/providers/{name}/callbacks       delivery receipts, bounces, complaints
POST /u/{opaque_token}                    one-click email unsubscribe (RFC 8058)

Back-of-envelope estimates

Assume 300M registered users, 120M daily actives, 1.5 devices on file per user, 1.2 active devices per targeted user, and 85% of devices on Android. (For the method, see back-of-envelope estimation.)

Volume. 3B notifications a day ÷ 86,400 s ≈ 35k/s on average. Each becomes about 1.3 deliveries (several devices, plus an inbox copy): 3B × 1.3 ≈ 3.9B deliveries a day.

Transactional peak. 1B a day ÷ 86,400 ≈ 11.6k/s on average. Dinner time on a sale day runs about 4× that: ≈ 46k/s, so plan for 50k/s.

Campaign fan-out. 100M users in 30 minutes is 100M ÷ 1,800 s ≈ 56k users/s. At 1.2 devices each that is 67k pushes/s, and 85% of them — 57k/s, or 3.4M a minute — go to FCM. FCM’s scaling guide gives the HTTP v1 API a default of 600k messages per minute, so this campaign needs 5.7× the default; at the default rate it would take 102M ÷ 600k ≈ 170 minutes. And the same allowance must also carry every login alert and order update the app sends meanwhile, so assume the platform has negotiated about 4.8M a minute (80k/s) with FCM ahead of sale season.

SMS. 30M OTPs plus 20M other transactional SMS a day = 50M ≈ 580/s on average; the login rush when a sale opens peaks around 5k OTPs/s. Contract two aggregators at 5k/s each, 10k/s in total. At an assumed ₹0.15 per SMS, 50M a day costs ₹75 lakh a day — the reason every type prefers push and keeps SMS for OTPs and fallbacks.

Email. About 300M a day — receipts, statements, newsletters — ≈ 3.5k/s on average.

The log. A queued delivery is about 1 KB (IDs, parameters, routing — not rendered HTML). The worst moment is 50k/s transactional plus 67k/s campaign ≈ 120k deliveries/s → 120 MB/s into Kafka, 360 MB/s counting three replicas. Three days of retention: 3.9B × 1 KB ≈ 3.9 TB a day × 3 days × 3 replicas ≈ 35 TB.

Status. About 2.5 events per delivery (sent, delivered, sometimes opened or failed): 3.9B × 2.5 ≈ 9.75B a day ≈ 113k events/s. The latest status per delivery at about 200 bytes, kept 30 days: 3.9B × 200 B × 30 × 3 replicas ≈ 70 TB in a wide-column store.

Inbox. About a third of notifications land in the inbox: 1B a day × 500 B = 500 GB a day; 90 days × 3 replicas ≈ 135 TB.

Hot lookups. Contact book: 300M × 1.5 devices × 300 B ≈ 135 GB. Preferences and caps: 300M × 200 B ≈ 60 GB. Both fit comfortably in a Redis cluster in front of their databases.

Workers. Peak push is about 35k/s transactional (70% of the 50k/s goes by push) plus 67k/s campaign ≈ 100k/s. If one worker sustains 2,000 sends/s over pooled HTTP/2 connections, that is 50 workers; run about 80 across three zones.

High-level design

Build one requirement at a time. By the end of this section the platform works for one region at moderate traffic; the deep dives then make it survive a sale day.

Product teams should be able to send a notification to a user

The API does as little as possible while the caller waits: authenticate the calling service, check that the type exists and the parameters match its template, apply the team’s quota, record the idempotency key, assign an ID, append to a log and return 202. Everything slow — finding devices, rendering text, talking to Apple — happens behind the log.

1. The order service calls POST /v1/notifications with a user ID, a type and parameters.
2. The API checks the caller, the type and the parameters, records the idempotency key, appends the notification to the accepted log and returns 202 with its ID.
3. The orchestrator consumes the log and looks up the user’s devices, phone and email in the contact book.
4. It writes one delivery per channel and device onto that channel’s queue: two phones and the inbox make three deliveries.
5. Channel workers call the provider for their channel, or write the inbox.

Why a log in front, rather than calling Apple from the API? Providers take 100–500 ms and have bad hours; callers would inherit both. The log turns “accepted” into a durable fact: Kafka with acks=all and min.insync.replicas=2 acknowledges only once at least two replicas have the record (the reasoning is in design a distributed message queue). It also absorbs bursts and lets you replay a bad hour. Partition the log by user ID, so one user’s “order placed” is always processed before their “out for delivery” — per-key ordering, explained in message ordering.

Notifications should use templates in the user’s language

Teams register templates, not strings. A template belongs to a type and has a variant per channel — push title and body, SMS text, email subject and HTML, inbox card — and per locale. The orchestrator renders it with the request’s parameters in the user’s language, falling back from hi-IN to hi to en when a translation is missing.

Three decisions matter here:

• Render once, in the orchestrator. The rendered payload travels with the delivery, so a retry three minutes later sends exactly the same bytes even if someone published template v4 meanwhile.
• Cache templates in memory. There are a few thousand templates, a few megabytes in all; a change publishes a new version, and each orchestrator reloads within seconds.
• Validate at registration and at the API. At registration: the push variant must fit Apple’s 4 KB payload limit, and the SMS variant must match the text registered with the operators (more in the preferences deep dive). At the API: every required parameter is present.

Users should be able to control what they receive

Add a preference service: per user, category and channel, on or off; quiet hours; a time zone; and frequency caps. The orchestrator checks it for every delivery, so all teams share one set of rules. Reads come from a Redis cache in front of the preference database — 60 GB fits easily — and a change invalidates the user’s entry, so an opt-out takes effect in seconds.

Every outcome is a status — suppressed:opted_out, suppressed:cap, deferred:quiet_hours — so a sender who asks “why didn’t user X get it?” gets an answer instead of a shrug. Security and transactional types skip caps and quiet hours: nobody should be able to mute their OTP or the alert that someone just logged in from a new phone.

Marketing should be able to schedule campaigns to tens of millions of users

A campaign is not one big request. Add a campaign service that stores the definition and its state, and a scheduler that fires it at send_at — a classic delayed-job problem, covered in design a job scheduler. When it fires, the segment is frozen into a snapshot of user IDs in blob storage, split into chunks, and expansion workers turn each chunk into ordinary notifications on the lowest-priority lane.

A snapshot matters because segments are live queries: “users who ordered biryani in the last 30 days” changes by the minute. Freezing it when the campaign fires makes the audience reproducible, makes progress measurable (chunk 4,127 of 10,000) and lets a crashed worker’s chunk be re-run. At 8-byte user IDs the snapshot for 100M users is only 800 MB.

Senders and users should be able to see what happened

Two things close the loop: a status pipeline for senders, and an inbox for users.

Every stage emits a status event: the API (accepted), the orchestrator (suppressed, deferred, queued), workers (sent, failed, expired), providers through webhooks (delivered, bounced, SMS delivery receipts), and the app’s SDK (received, opened). Events go to one Kafka topic keyed by notification ID; a status writer keeps the latest state per delivery in a wide-column store and feeds per-campaign counters to an analytics store.

The inbox is the user-facing record. The orchestrator writes an inbox copy for types that have one; an online app hears about it within a second over a persistent connection, and an app that was offline fetches the page when it opens.

The realtime path is a standard real-time connection problem: each app holds a WebSocket or server-sent-events stream to a gateway node, a registry maps user to node, and the inbox writer tells that node “new item” after the write commits. That is a working notification system. Push it to a sale day and four things break: campaigns delay OTPs, naive expansion floods everything, providers start answering 429, and crashes produce duplicates.

Deep dives

How do we stop a campaign from delaying an OTP?

At 19:07 marketing starts the sale: 100M pushes and, for users without the app, 20M promotional SMS. At 19:08 Priya opens the app to grab a deal and asks for a login OTP. Where does her SMS queue?

Bad: one queue per channel. It is first in, first out, so the OTP joins behind every promotional SMS. With 20M ahead and 10k SMS/s of aggregator capacity, it waits 20M ÷ 10k = 2,000 s — over half an hour, for a code that expires in five minutes. The push channel has the same problem: an order update behind 100M campaign pushes drained at 55k/s waits about 30 minutes. You gain simplicity; you pay with head-of-line blocking, where the biggest sender sets everyone’s latency.

Good: a queue per priority, workers drain the highest first. The OTP now skips the promotions. You pay in three quieter ways: one shared worker pool can be fully busy with slow promotional calls when the OTP arrives; strict priority can starve promotions for as long as transactional traffic stays high; and nothing stops the campaign from spending the shared provider quota, so the OTP is first in line for a provider that answers 429.

Great: lanes by priority × channel, with their own workers and a reserved share of each provider.

• Topics per lane — sms.p0, sms.p1, sms.p2, push.p0, … — each with its own consumer pool, autoscaled on its own lag. P0 pools are small and always over-provisioned. Netflix’s RENO does the same: priority-specific queues and processing clusters, so each priority can be scaled and tuned independently. These are bulkheads.
• Reserved provider capacity. Keep one token bucket per provider account, as in the rate limiter design. P0 and P1 may drain it; P2 has its own cap and may take tokens only while the bucket holds more than a reserve floor, so a transactional burst always finds tokens. Of 10k SMS/s, promotions get at most 5k/s; of an 80k/s FCM allowance, campaigns get 55k/s and 25k/s is kept for transactional pushes.
• Admission control at the API. Each team has a quota per priority, so a team stuck in a loop gets 429 instead of hurting everyone — the noisy neighbour problem. Priority comes from the registered type, and P0 types need a review.
• A shedding order. Under overload, drop digests first, then delay P2, then slow bulk P1; never touch P0. That is load shedding decided in advance.
• The same split at the providers. Twilio’s scaling guide recommends a separate sender pool for OTPs and for promotions; Google’s sender guidelines recommend different sending IPs and From addresses for account notifications and promotions. Spam complaints about a sale email should not land receipts in the spam folder.

How do we send a campaign to 100M users in about 30 minutes?

Bad: one request loops over the segment. At 1 ms per user for preference and token lookups, a single loop takes 100M ms ≈ 28 hours. Parallelise it naively and you get the opposite failure: 100M messages land in the queue in minutes. A cancel must now purge a 100M-message backlog, preferences checked at enqueue time are stale by the time the last message goes out, and a crash halfway leaves you guessing who already received it.

Good: snapshot, chunks and stateless workers. Split the 100M-user snapshot into 100M ÷ 10k = 10,000 chunks. Workers claim a chunk, batch-read preferences and device tokens for its 10k users, drop opted-out users, build notifications and mark the chunk done. A crashed worker’s chunk is simply claimed again. That rerun may resend users from the first half of the chunk, so keep a per-campaign sent bitmap in Redis: SETBIT on the user’s dense sequence number (assigned at sign-up) returns the old bit, so a 1 means “already sent, skip”. One bit per user is 300M ÷ 8 = 37.5 MB per campaign. You gain parallelism and resumability; you still flood the queues if expansion outruns sending.

Great: expand just in time, at the pace the providers allow.

• Pace at the source. Each targeted user produces 1.2 devices × 85% Android ≈ 1.02 FCM messages. With a negotiated allowance of 80k/s and 25k/s reserved for transactional traffic, campaigns get 55k/s: 55k ÷ 1.02 ≈ 54k users/s, or 5.4 chunks a second, and 100M ÷ 54k ≈ 31 minutes. The dispatcher releases chunks at that rate; the token bucket in the push workers is an independent second guard. Each chunk takes about 2 s to expand, so 11 workers are busy at any moment; run 30.
• Ramp, and avoid the busy minutes. FCM’s scaling guide asks senders to ramp from zero to full rate over at least 60 seconds — longer for higher rates — and to avoid the two minutes around :00, :15, :30 and :45, when everyone else’s scheduled sends land (FCM’s traffic more than doubles at the top of the hour). Hence 19:07, not 19:00, and a two-minute ramp. The same guide asks you to contact FCM before changing your pattern by 100k requests/s globally, so a sale-day plan is agreed weeks ahead.
• Protect your own backend. A successful push makes people tap. If 2% of 100M open within 15 minutes, that is 2M ÷ 900 s ≈ 2,200 new sessions/s; at about 20 API calls per session in the first minute, ≈ 44k extra requests/s hit the catalogue and pricing services. Pace to what the landing page can serve, not only to what FCM accepts, or roll out by city or cohort.
• Time zones. “19:00 local” splits the snapshot into waves, one per time zone, each with its own release time.
• A fast kill switch. The dispatcher checks the campaign state before every release, and push workers drop P2 deliveries for a campaign in a cached cancelled set refreshed every few seconds. Because only a few chunks are in flight, a cancel stops the campaign within seconds, with about 50k users of blast radius.

How do we live within APNs, FCM, SMS and email limits?

Every provider is a different contract with different failure signals. Learn the handful of facts that change the design.

• APNs. Your servers keep long-lived HTTP/2 connections to Apple and multiplex many requests over each; Apple’s sending guide says to reuse connections for hours or days, spread traffic over many of them, and not assume a fixed number of concurrent streams. Authenticate with a signed provider token (a JWT) and refresh it, but no more than once every 20 minutes. Per request, apns-priority: 10 means send now and 5 lets the phone batch for battery; apns-expiration sets how long Apple keeps trying (0 means one attempt, no storage); apns-collapse-id, up to 64 bytes, makes a newer notification replace an older one; the payload is capped at 4 KB. If the phone is offline Apple stores only one pending notification per app — another reason the inbox matters.
• FCM. One message per HTTP v1 request, drawn from a per-minute allowance (600k by default). Android messages are normal or high priority: high may wake a dozing phone, normal may wait. FCM downgrades high-priority messages that don’t lead to a visible notification, judged on seven days of behaviour, so silent sync messages should go as normal.
• SMS aggregators. Throughput is sold per account and per sender: Twilio, for example, documents 100 messages/s for a US short code but 1/s for a US long code. Messages queue at the provider — 10 hours by default at Twilio — so give OTPs a short validity period (Twilio’s example uses two minutes) and they expire instead of arriving useless. Delivery receipts (DLRs) arrive later by webhook, matched by the provider’s message ID.
• Email. The provider meters a daily volume and a per-second rate. Gmail asks anyone sending more than 5,000 messages a day to authenticate with SPF, DKIM and DMARC, offer one-click unsubscribe, keep the spam rate below 0.3%, send at a steady rate, and, after a 4.7.28 rate-limit error, pause for 10 minutes and resume on a single connection.

Bad: workers call providers directly and retry anything that fails. Without a shared limit, 80 workers overrun FCM’s allowance together and get 429s, and immediate retries turn that into a retry storm. When one provider hangs, every worker thread waits on its timeout, so all channels in that pool stall. Retrying a dead token forever also costs throughput: Apple notes that 4xx errors slow your sending and too many get the connection closed.

Good: one adapter per provider. Each adapter owns a token bucket shared across workers (in Redis), a timeout — FCM suggests at least 10 seconds — and a classifier that turns each response into one action. Each provider gets its own worker pool, so a broken APNs cannot block SMS; RENO calls this bulkheaded delivery, best-effort per platform.

The classifier follows the providers’ own rules. Apple’s response guide lists codes never to retry — BadDeviceToken, DeviceTokenNotForTopic, Forbidden, ExpiredToken, Unregistered, PayloadTooLarge — and uses 429 for too many pushes to the same device token. FCM says: don’t retry 400, 401, 403 or 404; on 429 wait for retry-after (60 seconds if it is missing); back off on 5xx; never retry sooner than 10 seconds; and stop retrying once the message is no longer timely.

Great: breakers, failover and routing by measured health. Put a circuit breaker on each provider: if more than 20% of calls fail over 30 seconds, it opens, P0 and P1 traffic moves to the secondary at once, and P2 pauses rather than moving — the secondary’s capacity is for urgent traffic, and a promotion can wait ten minutes. For SMS, keep two or more aggregators and route per operator by measured receipt success and latency, because an aggregator can be healthy on one network and broken on another. Feed every signal back: dead tokens to the contact book, bounces and complaints to the email suppression list, failed OTP receipts to a channel fallback.

How do we avoid losing or duplicating notifications?

Walk the path and mark where things go wrong. The order service commits “out for delivery” and crashes before calling the API: lost. The API appends to the log, but the response times out and the caller retries: duplicate. A worker hands a push to FCM and crashes before committing its Kafka offset: the message is redelivered, duplicate. FCM times out: did it send or not? APNs is down for an hour: retry until when?

Bad: fire and forget. Commit the offset first, then send. Every crash loses whatever was in flight — at-most-once for everything, including receipts.

Good: at-least-once everywhere. Commit only after the provider answers, and retry everything. Nothing is lost, but every crash and timeout becomes a second push, a second OTP, a second “₹1,240 paid”.

Great: at-least-once transport, exactly-once effects inside the system, and a deliberate choice per type at the edge.

• Producer: a transactional outbox. The order service writes the status change and an outbox row in one database transaction; a relay sends the row to the API and uses its ID as the idempotency key. A crash cannot separate “order is out” from “tell the user”.
• API: an idempotency key. SET idem:{team}:{key} … NX EX 86400 in Redis maps the key to the first notification ID; a retry gets the same 202 and ID. The same key with a different body is rejected.
• Log: replicated before 202. acks=all with min.insync.replicas=2.
• Workers: a dedupe key per delivery. Before calling the provider, claim dlv:{notification_id}:{channel}:{device} with SET NX and a short expiry; after success, overwrite it as sent for a day. A redelivered copy finds the key and skips — the idea behind idempotent consumers.

That remaining window is why “exactly once to the phone” is impossible: the provider can accept a message and you can fail to learn that it did. Choose per type what an unknown outcome means:

def deliver(d):
    if now() > d.expires_at:
        return record(d, "expired")             # an OTP after 5 minutes is noise
    key = f"dlv:{d.notification_id}:{d.channel}:{d.target_id}"
    if not redis.set(key, "inflight", nx=True, ex=120):
        return                                  # already sent, or being sent
    try:
        result = adapters[d.provider].send(d)   # rate limit + breaker inside
    except (Timeout, ConnectionLost):
        if d.on_unknown_outcome == "give_up":   # OTP: never risk a second code
            return record(d, "unknown")         # the user can tap Resend
        raise RetryLater(d)                     # key expires in 120 s, then retry
    redis.set(key, "sent", ex=86400)
    record(d, result.status, result.provider_message_id)

OTPs are at-most-once per request. A second code is worse than a missing one, and the user has a Resend button that creates a new notification with a new code. Receipts and order updates are at-least-once. Send them with the notification ID as apns-collapse-id or FCM collapse key, so a rare duplicate replaces the first on the lock screen instead of stacking, and write the inbox as an upsert by notification ID.

Retries are bounded by the type’s TTL, not by a count. Back off exponentially with full jitter, never sooner than 10 seconds for FCM, and honour Retry-After. Run retries through delay topics (10 s, 1 min, 10 min) so a struggling message never blocks its partition. When expires_at passes, the delivery becomes expired and is dropped — RENO applies the same staleness filter to events that are too old to be useful. Only non-retryable failures (a bad payload, a broken template) go to a dead-letter queue with an alert.

How do we respect preferences, quiet hours and caps for 300M users?

Bad: each team checks preferences in its own code. Rules drift, and nobody can enforce “at most three promotions a day” across forty teams who each send one. Checking only at the API is no better: a campaign scheduled at noon for 19:07, or a promotion deferred overnight, is judged on preferences that are hours old.

Good: one policy check in the orchestrator at send time, reading the Redis cache. Rules are consistent and fresh. For a campaign, though, the check runs after expansion: users who turned promotions off still cost a queue entry and a pass through the pipeline.

Great: filter twice, enforce once.

• Bulk filter at expansion. The expansion worker reads preferences for its 10k users in one batch and drops opted-out users before they become deliveries. If 30% of the segment has promotional push off, that saves 100M × 30% × 1.2 ≈ 36M deliveries and the quota they would have used.
• Cheap re-check right before sending, for anything that waited — deferred by quiet hours, delayed by retries. An unsubscribe always wins: the preference write invalidates the cache entry and publishes an event, so it takes effect within seconds.
• Quiet hours in the user’s time zone, which the app reports on every start. A deferred promotion is scheduled for the end of the window only if its TTL allows. For campaigns, never defer millions of users to exactly 08:00 — that builds a new spike; spread them over the next hour with jitter, or skip users who are asleep.
• Frequency caps as atomic counters, per user, category and day in Redis: one Lua script checks the count and reserves a slot, exactly like the rate limiter. Over the cap, a promotion is dropped or folded into a digest. Transactional types neither count nor get blocked.
• The OS has the last word. A user can switch off notifications for the app entirely; FCM notes that high-priority messages to such apps are deprioritised. The app reports its permission state on start, and the orchestrator sends those users to the inbox (and, for transactional types, SMS) instead.

Two channels carry legal rules, and they belong in versioned configuration owned by a compliance team, not in code:

• Commercial SMS in India. Under TRAI’s 2018 customer preference regulations, commercial SMS runs on distributed ledgers (DLT) kept by the operators. Each sender registers its headers — sender IDs of up to 11 characters — and its content templates, with fixed text and marked variable parts; messages that don’t match are blocked. Subscribers can block commercial messages by category and by time band, and the 2018 schedule keeps the bands from 21:00 to 10:00 off by default unless the subscriber opts in; transactional messages are exempt from these blocks. The regulation has been amended several times since, so treat the windows and categories as data.
• Email unsubscribe. RFC 8058 one-click unsubscribe needs a List-Unsubscribe header with an HTTPS URI and List-Unsubscribe-Post: List-Unsubscribe=One-Click, both covered by the DKIM signature. The mailbox provider POSTs to that URI with no cookies or login, so the URI itself must carry an opaque, hard-to-forge token naming the user and list; the endpoint must not redirect, and it simply flips the preference. Gmail requires this for marketing mail from bulk senders.

How do we keep device tokens fresh and know what actually arrived?

Device tokens. The app registers on every start: device ID, push token, platform, app version, notification permission, locale and time zone. The server upserts by device ID and updates last_seen, which FCM’s token guide recommends.

• Bad: one token column per user. A second phone overwrites the first, and dead tokens are never removed, so a growing share of sends goes nowhere — and Apple throttles connections that produce too many errors.
• Good: a device table, cleaned up on provider signals. APNs answers 410 Unregistered with a timestamp; delete the token unless the app re-registered it after that time. FCM answers UNREGISTERED (404), or INVALID_ARGUMENT (400) for a payload you know is valid.
• Great: also expire by silence, and enforce ownership. FCM treats a registration as stale when the app hasn’t connected for over a month, and Android registrations expire after 270 days. Stop sending promotions to devices silent for 30 days and delete them at 60. A token belongs to one user at a time: when someone else logs in on the same phone, the device row moves to them, and logout deletes it — otherwise the previous owner’s OTPs and payment alerts land on a stranger’s lock screen. Dead-token rates per campaign double as an uninstall signal.

Delivery status. Be precise about what each signal means. A 200 from APNs or FCM means the provider accepted the message, not that anyone saw it. SMS receipts come from the operator and are fairly reliable; email gives delivered, bounced and complained events, while “opened” relies on image loading and is weak. For push, received and opened can only come from the app’s SDK. Events arrive out of order — a webhook saying “delivered” can beat the worker’s “sent” — so the status writer upserts with a rank (accepted < queued < sent < delivered < opened) and never moves backwards. The status store is a wide-column table (Cassandra or DynamoDB) partitioned by notification ID with a 30-day TTL; campaign dashboards read per-minute counts built by a stream processor, never by scanning rows.

The inbox. Partition by user ID, cluster by notification ID descending — time-sortable IDs give “newest first” for free — and page with a cursor, not an offset. Keep 90 days or 500 items per user. The unread badge is a Redis counter incremented by the inbox writer and reset by markRead up_to=<id>, recomputed from the table now and then. Online apps hear about new items through the realtime gateway; offline apps pull when they open. That is the hybrid push-and-pull model RENO uses — push to devices that are online, let every device call home at fixed points in its lifecycle — and it means a notification the OS dropped is still waiting in the inbox.

The final design

Put the pieces together. The hot path is the API, the log, the orchestrator, the lanes and the channel workers; campaigns enter through the same log, and everything that reports back flows into one status topic.

Follow one “out for delivery” notification, and then a campaign:

1. Produce. The order service commits the status change with an outbox row; a relay calls the API with the row ID as the idempotency key.
2. Accept. The API checks the type, parameters and team quota, records the key, appends to the P1 lane of the log with acks=all and returns 202.
3. Decide. The P1 orchestrator pool resolves the user’s active devices, checks preferences (cached), renders the Hindi template, and writes the inbox copy.
4. Route. It writes one delivery per device to push.p1 and one to the in-app lane, each carrying its rendered payload, expires_at and collapse ID.
5. Send. A push worker claims the dedupe key, takes a token from the FCM bucket, passes the breaker and calls FCM; the classifier records sent, deletes a dead token, or schedules a retry within the TTL.
6. Show. The inbox writer upserts the item; the gateway tells the app if it is online; otherwise the app pulls on open.
7. Campaigns. The campaign service releases 10k-user chunks at the paced rate into the P2 lane, where the same steps run with separate pools and a capped share of provider quota.
8. Report. Provider responses, receipts and app events flow into status events, then into the status store, the campaign dashboards and token clean-up.

Check it against the requirements: 202 only after the log has the notification on two replicas; OTPs on a P0 lane with reserved workers and SMS capacity, so a campaign cannot delay them; 100M users in about 31 minutes at 55k FCM sends/s, cancellable within seconds; duplicates prevented by idempotency and dedupe keys, with OTPs at-most-once and receipts collapsed on the device; preferences, caps and quiet hours enforced in one place and re-checked before sending; and every outcome visible as a status.

What interviewers look for by level

Follow-up questions

What if a campaign goes out by mistake?

Hit the kill switch: the campaign state flips to cancelled, the dispatcher stops releasing chunks, and workers drop queued P2 deliveries for it within seconds — at our pace, about 50k users had it in flight. Messages already accepted by Apple or Google cannot be recalled, but you can limit the damage: delete or replace the inbox items, and send a correction with the same collapse ID, which replaces the original wherever it is still pending or on display. Prevent the next one with a mandatory test send to staff, a 1% canary for the first five minutes, and a second approver for audiences above a million.

What if APNs or FCM is degraded for an hour?

The breaker for that provider opens. P2 pauses rather than retrying in a loop. P0 and P1 retry with backoff within their TTL, and types that allow it fall back to another channel — an order update moves to the inbox, and to SMS only if it is critical. When the provider recovers, drain the backlog with a ramp, not a flood, and drop everything whose TTL has passed. FCM’s own advice is that a request still failing after an hour of backed-off retries is either misclassified or caught in an outage that more retries will only worsen.

What if users complain about too many notifications?

Make attention a budget. Add a global cap across all teams — say three promotional pushes a day per user — and when more candidates compete than the budget allows, let a ranking service choose by predicted value (the ML part we left out). Fold low-value items into a daily digest. Track opt-out and dead-token rates per campaign and team as guardrails: a campaign that triggers a spike of unsubscribes is costing future reach.

What if we need digests such as “5 new messages”?

Buffer instead of sending. The first eligible event for a user and type starts a window — a sorted set in Redis and a timer from the job scheduler; later events join it. When the window closes, or the count crosses a threshold, render one notification from the buffered items. Use a collapse key so the digest replaces any earlier one on the device, and keep the individual items in the inbox.

What if we must add WhatsApp as a channel?

It becomes one more adapter, a lane per priority, a worker pool and a preference column; the API and orchestrator do not change, because channels are routing decisions made from type configuration. The real work is the provider’s contract — its own template approval, opt-in rules, pricing and rate limits — which slots into the same adapter, classifier and template registry as SMS.

How to say it in the interview

“I’ll design one notification platform for a 300-million-user super-app: push, SMS, email and an in-app inbox, about 3 billion notifications a day, transactional peaks of 50 thousand a second, and campaigns to 100 million users in under an hour.

Teams call one API with a user ID, a notification type and parameters. The API validates, records an idempotency key, appends to Kafka with acks from two replicas and returns 202. An orchestrator resolves the user’s devices and preferences, applies quiet hours and frequency caps, renders the localised template and writes deliveries to lanes split by priority and channel. Channel workers call APNs, FCM, SMS aggregators and the email provider.

The key decision is isolation. Priority comes from the registered type, and each lane has its own workers and a reserved share of provider quota, so an OTP never waits behind a campaign or finds FCM’s allowance spent.

Campaigns are frozen into a snapshot and released in 10-thousand-user chunks at the pace the providers allow: with 55 thousand FCM sends a second, 100 million users take about 31 minutes, and cancel takes effect within seconds.

For reliability: an outbox at producers, idempotency keys at the API, at-least-once workers with a dedupe key in Redis, and retries bounded by each type’s time to live. Exactly-once to a phone is impossible, so OTPs are at-most-once and receipts at-least-once with collapse IDs.

Providers get adapters with rate limits, circuit breakers and SMS failover; dead tokens are deleted on 410 and UNREGISTERED. Status events from workers, webhooks and the app feed a Cassandra status store, and the inbox is the source of truth, pushed live over WebSockets and pulled when the app opens.”

Concepts used in this design

• Message queues — the lanes between orchestrator and channel workers.
• Design a distributed message queue (Kafka) — why acks=all makes a 202 durable.
• Delivery guarantees, exactly-once and DLQs — at-least-once workers and the dead-letter queue.
• Idempotent consumers and the inbox pattern — dedupe keys per delivery.
• Idempotency — safe retries of the send API.
• Transactional outbox — never losing the “tell the user” step.
• Backpressure and load shedding — pacing campaigns and the shedding order.
• Bulkheads — separate pools per lane and provider.
• Circuit breaker — failing over when a provider is sick.
• Timeouts, retries and backoff with jitter — retry rules bounded by TTL.
• Design a rate limiter — provider token buckets and frequency caps.
• Noisy neighbours and fairness — per-team quotas on a shared platform.
• Redis — preference cache, dedupe keys, caps and the sent bitmap.
• Cassandra and DynamoDB — status store and inbox.
• Real-time connections — live inbox updates.
• Design a job scheduler — scheduled campaigns, deferrals and digests.

References and further reading

1. Sending notification requests to APNs (Apple Developer Documentation)developer.apple.comHTTP/2 connections, apns-priority 10, 5 and 1, apns-expiration, the 64-byte apns-collapse-id, the 4 KB payload limit, and APNs storing only one pending notification per app.
2. Handling notification responses from APNs (Apple Developer Documentation)developer.apple.comStatus codes and reasons: 410 Unregistered with its timestamp, 429 for one device token, the errors never to retry, and provider-token updates no more than every 20 minutes.
3. Best practices for FCM registration management (Firebase)firebase.google.comStore a timestamp per registration, treat a month without contact as stale, Android expiry after 270 days, and delete on UNREGISTERED or INVALID_ARGUMENT.
4. Scale FCM: best practices for sending at scale (Firebase)firebase.google.comThe 600k-per-minute default quota, ramping over at least 60 seconds, avoiding the quarter-hour marks, 10-second timeouts, and how to retry 429s and 5xx errors.
5. Setting the priority of a message (Firebase)firebase.google.comNormal versus high priority on Android, Doze, and FCM downgrading high-priority messages that do not lead to a visible notification.
6. Build to scale: queueing and latency on Twiliotwilio.comSeparate sender pools for promotions and OTPs, short validity periods, the 10-hour default queue, and per-sender throughput limits.
7. RFC 8058: Signaling One-Click Functionality for List Email Headersrfc-editor.orgList-Unsubscribe-Post, a DKIM signature covering both headers, an opaque token in the URI, and no cookies or redirects on the POST.
8. Email sender guidelines (Google)support.google.comWhat senders of more than 5,000 messages a day to Gmail must do: SPF, DKIM, DMARC, one-click unsubscribe, spam rate under 0.3%, and steady sending rates.
9. Rapid Event Notification System at Netflix (Netflix Technology Blog)netflixtechblog.comPriority-specific queues and clusters, a staleness filter, bulkheaded delivery per platform, and a hybrid push-and-pull model at about 150k events/s.
10. Telecom Commercial Communications Customer Preference Regulations, 2018 (TRAI)trai.gov.inIndia’s rules for commercial SMS: registered headers and content templates on DLT ledgers, and customer preferences by category and time band.