Design a price tracker

A price tracker watches other people’s shops for you. Paste an Amazon.in or Flipkart link — or open the product page with the browser extension installed — and you see how the price has moved over the past year, set “tell me when this TV drops below ₹45,000”, and your phone buzzes when it does. Keepa and camelcamelcamel made this popular for Amazon; in India, shoppers lean on such tools before Big Billion Days and the Great Indian Festival to tell a genuine discount from a strike-through price nobody ever paid. It looks like a CRUD app with a chart. It is interesting because you own none of the data: every price must be fetched — from an API with a quota, a feed, a page you may or may not be allowed to read, or a user’s browser — and with a fixed fetch budget for hundreds of millions of products, deciding what to look at next is the real system. Then a sale opens at midnight, a popular TV drops by ₹7,000, and two hundred thousand people want to hear about it in the same minute.

Understand the problem

A price tracker sounds like a cron job and a chart. Push on it and three hard questions appear: where the prices come from, how fresh they can be when you cannot look at everything all the time, and how to turn one price change into exactly the right notifications — sometimes three, sometimes two hundred thousand — without ever crying wolf. Pin down the product first.

Functional requirements

1. Users should be able to add a product and see its price history. Paste a URL, or open the product page with the extension, and see a chart for the last month, year or all time, with the lowest and highest prices marked.
2. Users should be able to set a price alert on a product: below a target (“under ₹45,000”), a percentage drop (“15% below today’s price”), or back in stock.
3. Users should be notified soon after an alert’s condition is met, by push or email, with a link straight to the product.
4. Users should be able to browse a deals feed of the biggest genuine price drops, by category.

Non-functional requirements

• Freshness where people are watching. For the 1M most-watched products, notice a change within 15 minutes, or about 5 minutes during a big sale. The other 29M watched products: within about 4 hours. Unwatched products: within a few days.
• Fast once detected. From a confirmed change to the push or email provider: under 1 minute at p99, under 3 minutes in the first minutes of a big sale.
• No false alarms. A parsing glitch must never tell 200,000 people about a ₹499 television. Budget: under 1 wrong notification in 10,000.
• Never miss, never repeat. Each alert fires once per crossing of its target, even when components crash and replay.
• Fast reads. Chart and extension lookups under 200 ms at p99; the extension must never slow the retailer’s page.
• Polite and within quota. Never exceed a retailer’s API quota or agreed fetch rate; follow robots.txt; back off on errors.
• Scale. About 500M tracked products across ten retailers, 20M monthly users, 100M active alerts, and a budget of about 5,000 product refreshes a second.
• Availability. 99.9% for charts and alert management. The alert pipeline may lag during an incident but must not lose an alert.

Out of scope

• Buying. We link to the retailer; cart, checkout and payment are theirs (see checkout and cart).
• Coupon and cashback aggregation. We record offers separately but don’t compute “effective prices”.
• Price prediction (“buy now or wait?”), touched on in the follow-ups.
• Getting around anti-bot protection. If a retailer says no, the design works without its pages.
• Accounts, login and paid plans.
• A store’s own price-drop alerts — see the follow-ups for how the design shrinks.

The set-up

Plan the approach

Draw two halves joined by one stream. The collection side decides what to refresh, fetches it through an allowed channel, validates it and emits confirmed price changes. The user side stores alerts, matches changes against them, notifies, and serves charts and deals. The stream of price changes, keyed by product, is the contract between them. Satisfy each requirement simply, then spend the time on the deep dives.

Core entities

• Product. One item at one retailer, keyed by the retailer’s own ID: amazon_in:B0C1XYZ123. Title, brand, model, a GTIN or EAN barcode when known, and a parent ID when it is one size or colour of a family.
• Price series. One kind of price: the retailer’s own offer, the lowest new offer from any seller, the featured offer. Keepa keeps dozens of series per product; we start with three.
• Price point. A change point (time, price) in one series, price in paise, with a special value for “not available”.
• Fetch state. Current prices, last checked, next due, estimated change rate, watcher count and the nearest armed target.
• Alert. User, product, series, kind (below a target, percentage drop, back in stock), target in paise, state (armed or fired), re-arm rule and channels.
• Notification. The alert, the change that fired it, channel and status — also the deduplication record.
• Deal. Product, old and new price, the drop against its typical price, and a score.

API

Adding a product resolves any URL shape to one product ID:

POST /v1/products:resolve
Content-Type: application/json

{ "url": "https://www.amazon.in/TV-55/dp/B0C1XYZ123/ref=sr_1_3?crid=2K8&keywords=tv" }

{
  "product_id": "amazon_in:B0C1XYZ123",
  "title": "55-inch 4K TV (2026)",
  "current": { "series": "featured", "price_paise": 5499900, "in_stock": true },
  "last_checked": "2026-10-27T21:40:00+05:30",
  "tracking_since": "2025-03-14",
  "watchers": 18234
}

The chart asks for one series and one range, and gets a pre-built summary — per-day minimum, maximum and closing price for a one-year view:

GET /v1/products/amazon_in:B0C1XYZ123/history?series=featured&range=1y

{
  "resolution": "1d",
  "points": [
    ["2026-10-26", 5499900, 5499900, 5499900],
    ["2026-10-27", 5299900, 5499900, 5499900]
  ],
  "lowest": { "price_paise": 4599900, "at": "2025-10-03T00:04:00+05:30" },
  "last_checked": "2026-10-27T21:40:00+05:30"
}

Creating an alert is idempotent, so a double tap does not create two:

POST /v1/alerts
Idempotency-Key: 7d0c2f6e-91aa-4c1b-b2b0-3f1d55a0e9c4
Content-Type: application/json

{ "product_id": "amazon_in:B0C1XYZ123", "series": "featured",
  "kind": "below", "target_paise": 4500000, "channels": ["push", "email"] }

{ "alert_id": "al_9QeT", "state": "armed", "target_paise": 4500000, "current_paise": 5499900 }

The rest, sketched:

GET    /v1/alerts?cursor=…                       my alerts, with current price and distance to target
DELETE /v1/alerts/{alert_id}
GET    /v1/deals?category=televisions&min_drop=20&cursor=…
GET    /v1/lookup?retailer=amazon_in&item=B0C1XYZ123   extension: chart summary by retailer ID
POST   /v1/observations                           extension: the price this user just saw (opt-in, signed)

An observation from the extension is small and untrusted until checked:

{ "retailer": "amazon_in", "item": "B0C1XYZ123", "series": "featured",
  "price_paise": 4799900, "in_stock": true, "seen_at": "2026-10-28T00:01:12+05:30",
  "install_id": "ins_4fa2", "signature": "…" }

Back-of-envelope estimates

Assume 500M tracked products, 20M monthly and 5M daily users, 5 alerts per monthly user, and a budget of 5,000 product refreshes a second across all retailers and channels. (For the method, see back-of-envelope estimation.)

Alerts. 20M × 5 = 100M active alerts. They concentrate: assume about 30M distinct products carry at least one alert, and the 1M most-watched carry half of them — 50M alerts, about 50 each. A popular TV before a sale can have 200,000.

The fetch budget. 5,000 × 86,400 = 432M refreshes a day. Spread evenly over 500M products, each is refreshed every 500M ÷ 432M ≈ 1.16 days — about every 28 hours, so a midnight drop is noticed the next night. The budget must be steered. A tiered plan that fits:

Price changes. If about 1 refresh in 10 finds a different price: 427M × 0.1 ≈ 43M change events a day, about 500 a second. In sale mode the hot tier gets about 4,000 refreshes a second; if 40% of those find a new price, that is 4,000 × 0.4 = 1,600 changes a second.

Notifications. On a normal day perhaps 1% of alerts fire: 1M ÷ 86,400 ≈ 12 a second. When a big sale opens, assume 6M alerts (6%) fire in the first five minutes: 6M ÷ 300 s = 20,000 a second on average, and 2.4M ÷ 60 s = 40,000 a second in the first minute — a thousand times the normal rate. (Check: 1,600 changes a second for 300 s is 480,000 changes, about 12 alerts each — plausible when hot products carry about 50.)

Reads. 5M extension users × 8 product pages a day = 40M lookups, plus 10M chart views on the site and app: 50M ÷ 86,400 ≈ 580 reads a second, about 5,800 at a sale-evening peak. Those 40M extension page views can also report the price they saw: 40M ÷ 86,400 ≈ 460 free observations a second — about a tenth of the fetch budget, aimed at exactly the products people look at.

Storage.

• Price history: 43M change points a day × 365 ≈ 15.7B points a year. At about 4 bytes a point once compressed: 15.7B × 4 B ≈ 63 GB a year. Storing every observation as a 40-byte row instead: 427M × 365 ≈ 156B rows × 40 B ≈ 6.2 TB a year — a hundred times more.
• Fetch state: 500M × 200 B = 100 GB. Alert index: 100M × about 100 B per sorted-set entry ≈ 10 GB of memory. Alerts of record: 100M × 300 B = 30 GB.

Storage is small. The scarce resource is the fetch budget, and the dangerous moment is the five minutes after a sale opens.

High-level design

Build one requirement at a time. By the end of this section the tracker works for one retailer and a few million products; the deep dives make it work for 500M products and a midnight sale.

Users should be able to add a product and see its price history

When a user pastes a URL, the Product API hands it to a URL resolver, which turns any URL shape for an item into one canonical product ID such as amazon_in:B0C1XYZ123. For a new product, the resolver creates a catalog DB row and asks for an immediate first fetch, so the user sees today’s price within seconds. A fetcher gets the price; a validator compares it with the last known price and, only if it changed, appends a change point to the history store, from which the chart is read.

Canonicalisation deserves a minute in the interview: get it wrong and one TV gets five histories, its watchers split five ways.

• Strip the noise. Tracking parameters (ref, tag, utm_*), title slugs and mobile hosts do not change the item.
• Resolve short links such as amzn.in/d/… by following the redirect once and caching the answer.
• Extract the retailer’s own ID with a rule per retailer: the ten-character ASIN after /dp/ or /gp/product/ on Amazon, the product ID in a Flipkart URL.
• Keep variants apart, but grouped. A 55-inch and a 65-inch TV have different prices, so each is its own product under a parent. Keepa’s product records carry exactly this — a parent ASIN, plus any UPC, EAN and GTIN barcodes.

The extension skips the resolver: it reads the item ID from the page it is on, calls GET /v1/lookup, draws the chart beside the retailer’s price and, if the user opted in, reports the price it saw.

Users should be able to set a price alert

The Alerts API writes the alert to an alerts DB, the source of truth, sharded by user so “show me my alerts” is one shard. Two views are derived from it through change data capture, both keyed by product instead:

• the alert index: one sorted set per product, scored by target price, which the matcher uses in step 3;
• the product’s fetch state: its watcher count and the target closest to the current price, which the scheduler uses to refresh the product sooner.

A percentage alert is stored as an absolute target computed when it is created: 15% below ₹54,999 is ₹54,999 × 0.85 ≈ ₹46,749. If the current price already meets the target, say so in the response rather than arming an alert that would fire a second later.

Users should be notified soon after an alert’s condition is met

Now connect collection to alerts. The scheduler marks a product due; a fetcher gets its price; the validator compares it with the fetch state and, if it changed, stores the change point and publishes a confirmed price change to the Kafka topic price-changes, keyed by product so one product’s changes stay in order (see message ordering per key). An alert matcher finds the armed alerts whose targets lie between the old and new price and writes one notification job each. A notifier sends a push or email whose link carries the affiliate tag — a common way for trackers to earn money.

Note the arithmetic in the figure: on a product refreshed every 15 minutes, a price change waits 7.5 minutes on average to be noticed. No pipeline tuning beats a better schedule.

Users should be able to browse a deals feed

A deals job — a small stream processor — consumes the same topic. It compares each drop with the product’s typical price over the last 90 days, read from the chart summaries, never with the strike-through list price, which may never have been charged. It skips out-of-stock and unconfirmed drops, scores the rest and keeps the top few thousand per category in a sorted set for the Deals API — a small cousin of top-K trending.

You now have a working tracker for one retailer and a few million products. The deep dives are about what breaks with 500 million products, ten retailers with their own rules, and a sale that opens at midnight.

Deep dives

How do we get prices at all?

Every other part of the design depends on this answer, and it is as much a business and legal question as a technical one.

Bad: crawl every product page as fast as the retailer will tolerate. It needs no partnership and covers everything public. You pay in every currency: load on someone else’s servers, layout changes that silently break your parser, terms of service that often forbid automated access, and an arms race with bot protection that you will lose — and should not be fighting.

Good: official APIs and feeds first.

• Affiliate product APIs. Amazon’s Creators API, the successor to PA-API 5, returns item and offer data for up to 10 ASINs per GetItems call, counted as one transaction. Access needs an Associates account with at least 10 qualifying sales in the last 30 days. New credentials get 1 transaction a second and 8,640 a day; after that the quota follows the sales you refer — one more transaction a second per $4,320 of shipped revenue in the previous 30 days, up to 10. At the ceiling, 10 calls × 10 ASINs = 100 products a second, which refreshes 100 × 900 = 90,000 products every 15 minutes — the very top of the hot tier, or the confirmation fetches in a later deep dive, nowhere near 200M Amazon items.
• Product and delta feeds. Flipkart’s affiliate API publishes product feeds per category and a delta feed of only the products added, updated or removed since a version you name, 500 per page. Polling it costs in proportion to what changed, not to catalogue size — exactly the shape a tracker wants.

You gain sanctioned, structured, stable data. You pay with quotas you must earn, coverage and freshness someone else decides, and different rules for every retailer.

Great: a source router that blends channels by cost, plus the extension as a free sensor. Per retailer, an adapter knows which channels exist and what each costs. The source router serves each due product through the cheapest allowed channel and enforces a budget per retailer and channel as a token bucket (rate limiting, pointed outwards).

• Delta feeds first, where they exist: you spend nothing on what did not change.
• Extension observations next. About 460 a second, free, and aimed at the products people look at. They are untrusted, so they go through the validator, and one report never fires an alert on its own.
• API calls for the hottest products and for confirmations, within the quota.
• Page fetches last, only where the retailer’s terms and robots.txt allow, at a modest fixed rate per host. RFC 9309 is the rulebook: obey the most specific matching rule, refresh your copy of /robots.txt at least every 24 hours, treat a 4xx answer as “no rules” but a 5xx or network failure as “disallow everything”. On 429, 500 or 503, slow down for the whole host and honour Retry-After, as Google’s crawlers do. Politeness queues are covered in design a web crawler.

You gain the cheapest fresh price per product, sanctioned sources first, and graceful degradation when a channel disappears. You pay with an adapter per retailer and channel to keep healthy, and freshness that differs by retailer — which the product should state honestly (“checked 3 hours ago”).

How do we decide what to refresh next?

The budget is 5,000 refreshes a second for 500M products. Every refresh spent on a product whose price did not change, or that nobody watches, is not spent on the TV that two hundred thousand people are waiting on. This is a job scheduler whose jobs are worth different amounts.

Bad: round-robin. Each product every 28 hours. About 94% of the budget goes to products nobody watches, and the hot TV is checked once a day, so its alerts arrive hours late.

Good: tiers by watchers. The table from the estimates: the 1M most-watched products every 15 minutes, the other 29M watched ones every 4 hours, everything else every 3 days. A new alert promotes its product. You gain a simple, explainable system and a hundredfold improvement for watched products on the same budget. You pay with bluntness inside each tier: a popular book whose price changes twice a year gets 96 refreshes a day, while a phone repriced hourly sits in the warm tier, seen every 4 hours.

Great: spend each refresh where it is most likely to fire an alert. Give every product a value of refreshing it now, built from four ingredients:

• How likely the price has changed. Model price changes as a Poisson process with rate λ per hour: the chance of at least one change in the Δt hours since the last look is 1 − e^(−λ·Δt). A refresh tells you only one bit — changed or not — which is the setting of crawl-freshness research such as Learning to Crawl. Counting changes underestimates λ, because two changes between visits look like one. Invert the formula instead: if refreshes I hours apart find a change in a fraction X of visits, λ = −ln(1 − X) ÷ I. Refreshed every 4 hours with changes on half the visits gives λ = 0.693 ÷ 4 ≈ 0.17 an hour, about 4.2 changes a day; the naive count says 3. New products start from their category’s average.
• How much it matters. Watchers plus recent extension views, on a log scale so one viral product cannot starve the rest.
• How close a target is. If the nearest armed target is 2% below the price and this product typically moves 5%, a change will probably fire something; if it is 40% away, probably not.
• The sale calendar. Big Billion Days, the Great Indian Festival and Prime Day are announced weeks ahead. Multiply λ for those windows, and schedule refreshes for 00:00:30 instead of stumbling on the drop.

from math import exp, log1p

def refresh_value(p, now_h):
    dt = now_h - p.last_checked_h                     # hours since the last look
    p_changed = 1 - exp(-p.rate_per_h * dt)           # Poisson: anything new?
    demand = log1p(p.watchers) + 0.3 * log1p(p.views_24h)
    near = p_cross(p.gap_to_nearest_target, p.typical_move)   # 0..1
    boost = sale_boost(p.retailer, p.category, now_h)         # 1, or about 5 in a sale window
    return demand * p_changed * (0.2 + 0.8 * near) * boost / p.channel_cost

Turning values into a schedule. Values grow with time, so don’t rescore 500M products every second. After each refresh, compute when the product’s value will cross a per-retailer water level v* — from K · (1 − e^(−λ·Δt)) = v*, where K is the rest of the formula, Δt = −ln(1 − v*/K) ÷ λ — and drop the product into that minute’s bucket (due:{retailer}:{minute}), a timing wheel. Each second a dispatcher drains the current bucket, highest value first, as fast as the retailer’s token bucket allows. A controller raises the water level when buckets back up and lowers it when budget sits idle. Clamp intervals between 5 minutes and 7 days; without the ceiling, a rarely refreshed product looks stable, is refreshed even more rarely, and its estimate never corrects.

Sale mode. For the hour around a sale opening, pause most of the cold tier (about 1,800 refreshes a second) and halve the warm tier (about 1,000 more). The hot tier then gets about 4,000 a second: its 1M products every 1M ÷ 4,000 = 250 s — about every 4 minutes. Extension observations surge at the same moment.

You gain far more alerts caught on time for the same budget, flowing on its own to volatile, watched, nearly-triggered products and to sale windows. You pay in complexity: estimates need history, the feedback loop needs floors and ceilings, and support must explain why a product was checked only twice yesterday. Keepa’s tracking API calls its update interval a target — “as close to that as it is efficient” — which is the right way to promise freshness.

How do we store price history and draw charts fast?

Charts are read about 580 times a second, ten times that on a sale evening, and each wants a year or more of one product’s history.

Bad: a row per refresh. (product_id, checked_at, price) in a relational table: 156B rows a year. A hot product refreshed every 15 minutes gains 96 × 365 = 35,040 rows a year, so a three-year chart reads about 105,000 rows, nearly all saying “same as before”.

Good: store change points only, partitioned by product. A price history is a step function, and a step function only needs its steps. Keepa’s API returns exactly this: per price type, one flat array of time, price, time, price, …, time in minutes since a fixed epoch, price in the smallest currency unit, −1 meaning “no offer”. A refresh that finds the same price updates “last checked” in the fetch state and writes nothing to history. Store points in a wide-column store such as Cassandra, partitioned by (product_id, series, year) and clustered by time: a year of history is one partition read. You gain ten times fewer rows. You pay by having to tell “unchanged” from “unknown”: if fetching fails for two days, the line must not claim the price held steady, so write explicit gap markers.

Great: compressed blocks plus pre-built chart summaries.

• Compressed series blocks. Once a month, fold each product’s recent points into a compressed block: delta-encode timestamps (minutes between changes fit in a byte or two) and prices, then pack the small integers with simple-8b and run-length encoding — the toolbox TimescaleDB uses for integer and timestamp columns. Prices in paise are integers, so no floating-point tricks. About 4 bytes a point gives the 63 GB a year from the estimates; long histories for analysis go to columnar storage.
• Chart summaries. Beyond the last month, the chart never reads raw points. For a one-year view keep per-day minimum, maximum and closing price; for all-time, per week; plus “lowest ever” and “lowest in 90 days”. Each is one blob of about 2 KB per product per range, updated incrementally — a change only touches the last bucket — and cached (cache-aside), so a chart is one key read. Pre-warm the hot tier’s blobs before a sale.
• Keep the minimum. Downsampling must keep each bucket’s minimum and maximum, never just the average. A two-hour flash price is the most important point on the chart, and an average erases it.

You gain tiny storage, constant-time chart reads and raw history when you need it. You pay with two copies to keep consistent — rebuild a summary from the points when in doubt — and a monthly compaction job.

How do we match a price change against 100 million alerts?

A confirmed change arrives: product P went from ₹54,999 to ₹47,999. Which alerts fire?

Bad: poll the alerts. Every few minutes, join all 100M alerts with current prices: 100M comparisons a pass, almost all repeats, and alerts still minutes late.

Good: look up alerts by product. Index alerts by product_id; on each change, load P’s armed alerts and compare in memory. Cheap for the many products with a handful of alerts. You pay on the products that matter most: the hot TV’s 200,000 alerts are loaded on every change — every few minutes during a sale — though a change usually fires few of them.

Great: a sorted set per product, and one range scan between the old and new price. Keep P’s armed alerts in a Redis sorted set, alerts:{P}:armed, scored by target price. An armed “below” alert always has a target under the current price — otherwise it would already have fired. So when the price drops from old to new, the alerts that fire are exactly those with targets in [new, old):

ZRANGEBYSCORE alerts:{P}:armed 47999 (54999

The minimum is inclusive; ( makes the maximum exclusive. The cost is O(log N + M), where M is the number of alerts that fire — proportional to work you must do anyway. A price rise fires no below-alerts.

The details that make it production-grade:

• Claim atomically. One Lua script reads the crossed alerts, moves them from armed to a fired set and returns them, so two matchers — or one replaying after a crash — never fire the same alert twice. The {P} hash tag keeps both sets in one Redis Cluster slot.
• Batch the giants. For the 200,000-watcher TV, take 5,000 at a time with LIMIT 0 5000, removing each batch as you go. Never page with a growing offset: Redis walks past every skipped member, so each later batch costs more.
• Back in stock is a drop from infinity. Treat “unavailable” as a price of +∞. When the TV returns at ₹47,999, the scan becomes [47999, +inf]. A pure back-in-stock alert is a below-alert with an infinite target.
• Hysteresis, so flapping prices don’t spam. A repricer bouncing between ₹47,999 and ₹48,999 hourly would fire a ₹48,500 alert hourly. A fired alert re-arms only when the price climbs back above its target by a margin, say 3%, or after a re-arm timer — Keepa’s tracking API has such a per-alert interval, where 0 means “notify only once”. Score the fired set by re-arm price, and a rise re-arms with the mirror-image scan.
• Deduplicate downstream. Each job carries (alert_id, price_change_id); the notifier records it before sending, so a replayed Kafka message is a no-op (see effectively-once consumers).
• Per-user caps. Three colours of the same TV produce one message; beyond a handful an hour, the rest roll into a digest.
• Rebuildable. The alerts DB stays the source of truth and the index is derived by CDC, so a lost shard is rebuilt from it. About 10 GB in total — but watch hot keys: the TV with 200,000 alerts lives on one shard.

How do we avoid alerting on bogus prices?

The worst failure is not a missed alert. It is 200,000 people opening a link to a ₹499 television that was never ₹499. Bogus prices come from predictable places:

• Parsing. The strike-through M.R.P., an EMI amount (“₹2,292/month”), or ₹4,999 read as ₹49.99 because of locale formatting.
• Placeholders. An out-of-stock page showing ₹1 or an old price. Store “no offer” as its own value — Keepa uses −1 — never as a price.
• The wrong price. A bank-card or coupon price shown as the price; the cheapest seller with ₹2,000 of shipping; a different colour. Keepa keeps the retailer’s own offer, lowest new offer, featured offer with shipping, list price and used prices as separate series, and coupons as separate fields. Do the same, and let an alert name one series.
• Spoofing. A forged extension report planting a fake deal.
• Real but bizarre. A genuine pricing mistake by the retailer — which users want.

Bad: trust every parsed number. You gain the fastest possible alert. You pay when one parser breaks after a redesign and fires millions of false alerts in minutes, and users stop trusting you.

Good: validate before publishing. Check schema, currency and units; flag anything far outside the product’s history — say more than 50% below its 90-day low — or its category’s band; require the stock flag and seller to make sense. You catch most junk. You pay by also blocking real flash deals and pricing mistakes — the alerts people want most.

Great: confirm before alerting, and cap the blast radius.

• Confirm suspicious changes through a second channel. A suspicious observation triggers an immediate refresh through a different channel — the API if there is quota, else a page fetch, else three independent extension reports — ahead of everything else in the scheduler. Confirmed: publish it, flagged, so the message can say “possible pricing error — the seller may cancel”. Not confirmed: quarantine.
• A blast-radius rule. A change that would fire over 10,000 alerts and lies more than 30% below the 90-day low needs confirmation even from a trusted channel — cheap insurance against the most expensive mistake.
• Earned trust for extension reports. Signed install IDs, per-install rate limits, and reputation from agreeing with fetched prices. A single report never fires an alert; at most it makes the scheduler refresh the product now.
• Adapter health with a breaker. Per retailer and channel, track parse failures and the share of suspicious changes. When either jumps — usually a layout change — a circuit breaker pauses the adapter, pages its owner, and the router shifts budget elsewhere.

You gain rare, bounded false alarms. You pay a minute or two of delay on the biggest drops, plus budget: reserving 2% of fetches for confirmations is 100 a second — conveniently, the Creators API ceiling.

How do we deliver millions of alerts when a sale opens at midnight?

At 00:00 on the first night of Big Billion Days, thousands of hot products change price within minutes: 6M alerts fire in the first five minutes, 40,000 a second in the first minute, and one TV can fire 150,000 alerts from a single change.

Bad: send from the matcher. The matcher calls push and email providers inline. The partition holding the TV spends minutes sending 150,000 messages while every other product on it waits, and when a provider throttles, matching stalls.

Good: a queue between matching and sending. The matcher only writes notification jobs to a Kafka topic partitioned by user ID; a pool of notifiers sends them. Matching stays fast, senders scale on their own, and retries don’t block matching (see message queues). You pay at the peak: at 10,000 sends a second, the first five minutes’ 6M alerts take about 10 minutes to drain, and an alert fired at 00:04 waits about six.

Great: pre-scaled, prioritised and stale-checked.

• Pre-scale by calendar. Sales are announced weeks ahead, and autoscaling on queue depth is too slow for a spike that is over in five minutes. Run notifiers, and reserve provider throughput, at about 30,000 sends a second for the opening hour: the backlog clears in about 2 minutes.
• Priority lanes. Live price alerts get their own topic; digests and “you might like” emails go to a low lane that can wait an hour. Oldest first within a lane (more in backpressure and load shedding).
• Stale checks. A job older than about 5 minutes re-reads the current price. If it has bounced back above the target, drop the message or rewrite it honestly: “was ₹47,999 for 6 minutes at 00:04”.
• Cheapest channel first. Push first, email batched, each provider behind its own token bucket. Device tokens, provider quotas and retries are in design a notification system.
• Links that pay for the system. Each message deep-links with the affiliate tag. Referred sales are the revenue — and, with the Creators API, what raises your API quota.

You gain alerts within a minute or two through a thousandfold spike. You pay for capacity reserved for one hour a year, and for lanes and stale checks to build and test. The retailer’s side of the same midnight is design a flash sale.

The final design

Put the pieces together. The collection side runs top to bottom through the middle; the user side wraps around it; one Kafka topic of confirmed price changes joins them.

Follow one TV’s price drop on sale night through the system:

1. Schedule. The timing wheel releases the TV at 00:00:30: 18,000 watchers, a target 2% below the price and a sale boost give it a high value, and the retailer’s token bucket has room.
2. Fetch. The source router uses the cheapest channel that retailer allows — delta feed, affiliate API or a polite page fetch — honouring robots.txt and backing off the whole host on 429 or 503.
3. Observe. Extension users opening the TV’s page report the price they see; the API forwards these signed, rate-limited reports to the validator as untrusted observations.
4. Validate and store. The validator parses the price into paise and checks it against the fetch state and history; a suspicious drop is confirmed through a second channel first. The change point goes to the history store, which updates the chart summaries in the read models.
5. Publish. The confirmed change goes to price-changes, keyed by product, so consumers see this TV’s changes in order.
6. Match. The matcher range-scans the TV’s sorted set between the old and new price and claims those alerts atomically, 5,000 at a time.
7. Notify. Jobs enter the live-alert lane. The notifier deduplicates, re-checks the price if a job is a few minutes old, and sends push or email with an affiliate deep link.
8. Deals. The deals job scores the drop against the TV’s 90-day typical price and, if it is in stock, adds it to the televisions deals set.
9. Read and manage. The API serves charts and deals from the read models in one key read each; new alerts go to the alerts DB, and CDC updates the index and the scheduler’s watcher counts.

Check it against the requirements. Freshness: hot products every 15 minutes (about 4 in sale mode), watched ones every 4 hours, the long tail every few days, within 5,000 refreshes a second. Alerts: O(log N + M) matching per change; delivery under a minute normally and about two minutes at a pre-scaled sale peak. False alarms: plausibility checks, second-channel confirmation, the blast-radius rule. Never miss, never repeat: Kafka at-least-once, an atomic claim and a deduplication key per notification. Reads: one cached blob per chart. Politeness: per-channel budgets, robots.txt, whole-host back-off, no evasion.

What interviewers look for by level

Follow-up questions

What if users want one alert across Amazon and Flipkart?

Add a product group above retailer products. Match listings by barcode — GTIN or EAN — where retailers expose one (Keepa’s product records carry UPC, EAN and GTIN lists). Otherwise match on brand, model number and the attributes that change price, with a confidence score, and ask users to confirm weak matches (“Is this the same TV?”). A group alert fires when the cheapest member crosses the target, so keep a group-level sorted set fed by every member’s changes. The risk is a wrong match — last year’s model under this year’s name — so always show which listings are in the group.

What if a retailer blocks us, or redesigns its pages overnight?

A redesign shows up as a spike in parse failures; the adapter’s breaker pauses it, the router shifts that retailer to its other channels, and the gap is recorded so charts do not invent a flat line. A block is a business decision, not a technical puzzle: stop page fetches for that retailer, rely on its API or feeds and on user-initiated extension reports, lower the freshness promise in the product, and talk to the retailer — a tracker that sends buyers is often welcome as an affiliate partner.

What if we are the store itself?

Then most of this design disappears. A store owns its prices and already publishes every change — the online store streams price changes through CDC into Kafka, keyed by product. Subscribe to that stream instead of fetching: no scheduler, no budget, no bogus parses, and freshness in seconds. The alert index, the range scan, hysteresis and the notification lanes stay exactly as they are, and you can even check stock at send time.

What if the price depends on pincode, membership or bank card?

Make the context part of the series key: record pincode, membership and the offers applied with every observation, and fetch with one fixed default context. Keep base price and “effective price with the best card offer” as separate series and let each alert choose. Extension reports carry the user’s own context — useful to that user, but never merged into a shared series with a different context.

What if users ask whether to buy now or wait for the sale?

The history is the dataset: how this product behaved in previous sales, days since its last drop, the gap to its all-time and 90-day lows. Start with an honest summary rather than a black box — “in the last two Big Billion Days this TV dropped 12–15%; it is 4% above its 90-day low today” — computed nightly over the history and stored in the chart summary. Show the evidence, not a promise.

How to say it in the interview

“I’ll design a price tracker like Keepa for Amazon.in and Flipkart: 500 million tracked products, 20 million users, 100 million alerts, and the requirement that a drop on a popular product reaches people within about 15 minutes — 5 during a big sale — without false alarms.

The core constraint is that we don’t own the data. A source router gets each price through the cheapest channel the retailer allows: delta feeds where they exist, the affiliate API — whose quota is small and grows with the sales we refer — polite page fetches only where terms and robots.txt allow, plus free observations from our browser extension. Every channel has its own token-bucket budget, and we back off a whole host on 429s and 503s.

With about 5,000 refreshes a second, round-robin means each product once every 28 hours, so the scheduler spends each refresh where it is most likely to matter: watchers, the chance the price has changed — I model changes as a Poisson process and estimate each product’s rate from whether past refreshes saw a change — how close the nearest alert target is, and a sale calendar. Hot products end up every 15 minutes, every 4 during a sale; the long tail every few days.

A validator compares each observation with the last known price, confirms suspicious drops through a second channel, stores a change point — not every observation — and publishes the change to Kafka keyed by product. History is stored as compressed change points, and charts come from pre-built daily and weekly min–max summaries, so a chart is one cached read.

For matching I invert the problem: a product’s armed alerts sit in a Redis sorted set scored by target, and a drop from old to new fires exactly the alerts in between, claimed atomically, with hysteresis so flapping prices don’t spam.

Notifications go through a queue with priority lanes and stale checks, pre-scaled for sale nights, when six million alerts can fire in five minutes.”

Concepts used in this design

• Back-of-envelope estimation — the fetch-budget arithmetic that shapes everything else.
• Rate limiting — token buckets per retailer and channel, pointed outwards.
• Push vs pull architecture — delta feeds and extension reports versus polling.
• Redis — sorted sets for the alert index, the deals feed and the due buckets.
• Kafka — the price-change log that joins collection to alerts.
• Message ordering: per-key queues — one product’s changes in order.
• Cassandra and DynamoDB — one wide partition per product, series and year.
• OLTP vs OLAP, warehouses and columnar storage — compressed column blocks for long histories.
• Change data capture — keeping the alert index and fetch state in step with the alerts DB.
• Effectively-once: idempotent consumers and the inbox pattern — one notification per crossing despite replays.
• Caching strategies — chart summaries as one cached read.
• Hot keys, cache stampede and invalidation — the viral TV’s chart and its 200,000 alerts.
• Message queues — decoupling matching from sending.
• Backpressure and load shedding — priority lanes during the midnight spike.
• Circuit breaker — pausing a retailer adapter that starts producing garbage.
• Timeouts, retries and backoff with jitter — polite back-off when a retailer says 429 or 503.

References and further reading

1. Creators API: Introduction (Amazon Associates)affiliate-program.amazon.comThe successor to PA-API 5: GetItems, GetVariations and SearchItems, and the prerequisite of 10 qualifying sales in the last 30 days.
2. Creators API: API Rates (Amazon Associates)affiliate-program.amazon.comInitial 1 TPS and 8,640 TPD, 10 ASINs per GetItems call as one transaction, quota growing with shipped revenue to at most 10 TPS, and 429 when exceeded.
3. Flipkart Affiliate API: Product APIs referenceaffiliate.flipkart.comPer-category product feeds and a delta feed of products added, updated or removed since a version, in pages of 500.
4. Keepa API backend: Product object (GitHub)github.comPrice history as flat [time, price, …] arrays per price type, prices in the smallest currency unit, −1 for no offer, separate coupon fields, parent ASINs and UPC/EAN/GTIN lists.
5. Keepa API backend: Tracking object (GitHub)github.comAlert thresholds per price type, a per-tracking rearm timer (0 = notify once), an update interval in hours that is a target rather than a timer, and notification channels.
6. RFC 9309: Robots Exclusion Protocolrfc-editor.orgMost-specific-match rules, 4xx means no restrictions, 5xx means disallow everything, cache for at most 24 hours, and robots.txt is not access control.
7. Reduce the Google crawl rate (Google crawling documentation)developers.google.comA large crawler slows down across the whole hostname on 500, 503 and 429 responses and honours Retry-After.
8. Learning to Crawl (Upadhyay et al., AAAI 2020)arxiv.orgKeeping copies fresh under a bandwidth limit, with Poisson change processes whose rates are unknown and observed only as “changed since the last visit or not”.
9. Compression methods in hypercore (TimescaleDB documentation)tigerdata.comDelta, delta-of-delta, simple-8b and run-length encoding for integer and timestamp columns.
10. ZRANGEBYSCORE (Redis documentation)redis.ioScore-range queries in O(log N + M), inclusive and “(” exclusive bounds, and why large LIMIT offsets are expensive.