On-Call Escalation

What is being asked

Design an on-call paging system: schedules say who is on duty, alerts create incidents, and an escalation policy notifies people in turn until someone takes responsibility.

Requirements

Functional

• Rotations and overrides; escalation policies with steps and timeouts.
• Notify by push, SMS and phone call; acknowledge and resolve.
• De-duplicate repeated alerts into one incident.

Non-functional

• Must be more reliable than the systems it watches.
• A page must never be silently lost.
• Timers must fire on time even when servers restart.

Design

1. An alert arrives with a de-duplication key. A matching open incident absorbs it; otherwise a new incident is created in state triggered.
2. The incident is a state machine: triggered, acknowledged, resolved. Every transition is persisted with its time and actor.
3. On trigger, the service asks the schedule who is on call for step one, sends notifications, and sets a durable timer for the step's timeout.
4. If the timer fires and the incident is still triggered, it moves to the next step: the secondary, then the manager.
5. Acknowledging cancels pending timers.

Key decisions and trade-offs

• Durable timers. Timers are rows with a due time, polled by several workers that claim them atomically. An in-memory timer disappears with its server.
• Redundant delivery. Several providers per channel, and channels tried in order, because phone networks fail too.
• Independence. Host the paging system away from the infrastructure it monitors, or the outage that needs paging also stops the pages.
• Idempotent transitions. "Acknowledge" arriving twice, or after resolution, must be harmless.

Follow-up questions

• How do you avoid waking twenty people for one database failure?
• How would you compute "who is on call right now" with overrides and time zones?
• How do you test that paging works without paging someone?
• What if the acknowledgement arrives one second after the escalation fired?