#61 TSDB/Pull Semantics
Metrics Monitoring (Prometheus)
A monitoring system that scrapes metrics from services, stores them as compressed time series on local disk, and evaluates alert rules.
What is being asked
Design a system like Prometheus. The focus is the pull model, the storage engine for time series, and how queries and alerts run on it.
Requirements
Functional
- Discover targets and scrape their metrics endpoints on an interval.
- Store samples as time series identified by a name and labels.
- A query language for rates and aggregations; alerting rules.
Non-functional
- Millions of series on one server.
- Keeps working in isolation, since it is needed most when other things are broken.
- Efficient storage: samples are tiny and arrive in vast numbers.
Design
- The scrape manager fetches each target's metrics page every interval. Each line becomes a sample: a timestamp and a value for a series identified by its labels.
- Recent samples live in memory in the head block and are also appended to a write-ahead log, which rebuilds the head after a crash.
- Every two hours the head is written to an immutable block on disk with an index from label values to series. A compactor merges small blocks and applies retention by deleting whole blocks.
- Samples are compressed: timestamps by the difference of differences, values by XOR with the previous value, giving a byte or two per sample.
- Queries find series through the label index, read the chunks covering the time range and compute. Rules are queries run on a schedule.
Key decisions and trade-offs
- Pull. The server decides the rate, can tell "target down" from "no data", and targets need no configuration of where to send. Short-lived jobs need a gateway to push to.
- Local storage, single node. Simple and dependable, but limited to one machine's disk. Long retention and a global view are added with remote storage or by federation.
- Series churn. Each new label combination is a new series. Labels with unbounded values fill memory.
- Run two identical servers for availability; alerts are de-duplicated downstream.
Follow-up questions
- Why can you not take a rate over a gauge?
- How does the system notice a counter reset after a restart?
- How would you query across twenty of these servers?
- What does a 15-second scrape interval imply for the shortest alert you can write?