Amazon Bestseller List

What is being asked

Design the pipeline that computes "top 100 products" for every category, and the service that shows them. Unlike a live counter, this is a batch problem over a large history.

Requirements

Functional

• Top N products per category and sub-category, per marketplace.
• Updated hourly; based on recent sales with more weight on recent days.

Non-functional

• Billions of order lines; tens of thousands of categories.
• Lists are read extremely often.
• A failed run must not publish a broken list.

Design

1. Orders are appended to a data lake, partitioned by hour.
2. A batch job reads recent partitions. Map: for each order line, emit a key of category and product with the units sold, once for each ancestor category so that a book counts in "Fiction" and in "Books".
3. Reduce: sum units per key, applying a decay weight by age.
4. A second step groups by category and keeps the top N with a bounded heap.
5. Results are checked, written as a new version, and then the "current" pointer is switched atomically. The API serves the current version from cache.

Key decisions and trade-offs

• Incremental computation. Re-reading thirty days every hour is wasteful. Keep hourly aggregates and combine them, adding the newest and dropping the oldest.
• Skew. One category or product may hold a large share of all sales, overloading a single reducer; pre-aggregating on the map side and salting hot keys spreads the work.
• Versioned publishing lets a bad run be rolled back by moving a pointer, and readers never see a half-written list.
• Batch is the right tool because an hourly list tolerates delay and benefits from complete, corrected data; a streaming design would add cost for freshness nobody needs.

Follow-up questions

• How do returns and cancellations affect the count?
• How would you prevent sellers from buying their way onto the list?
• How do you move this to updates every five minutes?
• How do you test that today's list is plausible before publishing?