Performance Metric Collection

What is being asked

Design the system that collects numeric performance metrics from every machine and service in a company, stores them efficiently as time series, and serves dashboards and alert rules.

Requirements

Functional

• Collect gauges, counters and histograms with labels (service, host, region).
• Query a range for charts; evaluate alert rules every minute.
• Down-sample old data and keep it for a year.

Non-functional

• Millions of active time series, sampled every 10 to 60 seconds.
• The monitoring system must survive the failures it is meant to report.

Design

1. Each host exposes its metrics on an endpoint. Collectors learn the list of targets from service discovery and scrape them on a schedule.
2. Samples are appended to a time-series database that keeps the newest block in memory and writes compressed blocks to disk. Consecutive timestamps and values compress to a byte or two each.
3. A compactor merges blocks and writes down-sampled versions (5-minute and 1-hour averages) to cheap storage.
4. A rule evaluator runs alert expressions against recent data and hands firing alerts to an alert manager, which groups, silences and routes them.

Key decisions and trade-offs

• Pull keeps control with the collector: it sets the rate, and a failed scrape is itself a signal. Push is needed for batch jobs and for hosts behind firewalls.
• Shard by region and by team, with a query layer that fans out. One global store is a single point of failure.
• Label cardinality is the usual cause of outages in metrics systems. Enforce limits per metric.
• Run collectors in pairs scraping the same targets; duplicates are removed at query time. This is simpler than replicating the database.

Follow-up questions

• How do you alert on a machine that has stopped reporting altogether?
• How would you compute a 99th percentile across 500 hosts from histograms?
• What do you do when a deployment doubles the number of series?
• How is the monitoring system itself monitored?