CamelCamelCamel Tracker

What is being asked

Design a price history and alert service: collect the current price of a very large catalogue repeatedly, store every change, show a chart per product and notify people watching it.

Requirements

Functional

• Collect prices for tracked products on a schedule.
• Store price history; show charts over months and years.
• Let users set a target price and be notified.

Non-functional

• Tens of millions of products, each checked several times a day.
• The price source limits how fast it may be queried.
• History grows for ever; charts must still load fast.

Design

1. A scheduler keeps a priority queue of products by next check time. Products with many watchers or volatile prices are checked more often.
2. Collectors fetch the current price through the retailer's product data interface, within its permitted rate.
3. The price is compared with the last stored value. Only changes are written, so a product whose price is stable for a month costs one row.
4. Changes flow on a stream to the history store and to an alert matcher, which finds watchers whose target is now met.

Key decisions and trade-offs

• Store changes, not samples. Prices are step functions; recording only the steps cuts storage by an order of magnitude and loses nothing.
• Adaptive scheduling spends the limited request budget where prices actually move.
• Down-sampling for charts. A five-year chart is drawn from daily minimum and maximum values, not from every change.
• Respecting the source. Use official product data feeds where they exist, honour rate limits and terms of use, and back off on errors.

Follow-up questions

• How do you find the watchers affected by a price change efficiently?
• How would you avoid alerting on a pricing error that lasts five minutes?
• How do you partition the history store as it grows?
• What do you do when a product disappears from the catalogue?