Cluster Health Monitor

What is being asked

Design the service that tracks the health of every node in a large cluster and tells schedulers and load balancers which nodes to avoid.

Requirements

Functional

• Detect a dead or unhealthy node quickly.
• Expose the current membership and health to other systems.
• Record history for diagnosis.

Non-functional

• Tens of thousands of nodes.
• Few false alarms: a slow network must not mark half the fleet dead.
• No single point of failure.

Design

1. An agent on each node sends a heartbeat every few seconds with a few vital signs: load, disk, the health of local services.
2. Collectors in each zone receive heartbeats and write each node's last-seen time to a replicated store.
3. A failure detector marks a node suspect after a few missed heartbeats and dead only after a longer period, or after several independent collectors agree.
4. Consumers read or subscribe to the membership list.

Key decisions and trade-offs

• Push heartbeats or pull probes. Heartbeats scale well and need no inbound access; active probes prove the node is reachable from outside. Using both catches more failures.
• Speed against false positives. A short timeout detects failure fast but mistakes a pause for a death. Adaptive detectors use the observed variation in heartbeat arrival instead of a fixed timeout.
• Central collectors or gossip. Central collection is simple and consistent. Gossip protocols such as SWIM have no central point and scale further, but membership is eventually consistent.
• Mass failure guard. If most nodes go silent at once, the likelier cause is the network or the monitor. Stop evicting nodes and alert a human.

Follow-up questions

• How do you tell a dead node from a network partition?
• What is the bandwidth cost of heartbeats from 50,000 nodes?
• How would you detect a node that is alive but answering very slowly?
• How do you stop a node that was wrongly declared dead from causing harm when it returns?