Distributed Metrics Logging

What is being asked

Every service emits logs and metrics. Design the pipeline that gathers them from all machines, stores them cheaply, and lets engineers search recent logs and chart counts per minute during an incident.

Requirements

Functional

• Ingest log lines and numeric metrics tagged with service, host and severity.
• Search logs by time range and text; chart aggregates such as errors per minute.
• Keep raw data for days and rolled-up aggregates for months.

Non-functional

• Hundreds of thousands of events per second, with bursts during incidents, which is exactly when it must not fall over.
• Losing a little data is acceptable; delaying the producing service is not.

Design

1. A lightweight agent on each host tails log files, batches and compresses events, and ships them. It buffers on local disk when the network is slow, so the application never blocks.
2. The gateway authenticates, applies per-service quotas and writes to Kafka, which absorbs bursts and lets several consumers read the same stream independently.
3. A stream processor computes per-minute aggregates into a time-series store; an indexer makes recent logs searchable; an archiver writes compressed raw files to object storage for cheap long-term retention.

Key decisions and trade-offs

• Push or pull. Agents pushing suits logs and short-lived jobs; pulling (scraping) suits metrics because the collector controls the rate. Many systems do both.
• Sampling under pressure. When a service floods the pipeline, sample its debug logs and keep errors, protecting everyone else.
• Hot and cold tiers. Index only the last few days; older data stays in object storage and is queried slowly on demand. This is the main cost lever.
• Cardinality. A tag with unbounded values, such as a user id, multiplies time series and can break the metrics store. Limit tags to bounded sets.

Follow-up questions

• How do you keep one noisy service from delaying everyone's logs?
• What happens to data in transit when a Kafka broker fails?
• How would you support alerts that fire within ten seconds?
• How do you remove a customer's personal data from archived logs?