Job Scheduler

What is being asked

Design a distributed job scheduler: users submit jobs to run at a time or on a schedule such as cron. The system must fire them on time, run them on workers, retry failures and report status.

Requirements

Functional

• One-off jobs at a time; recurring jobs by schedule.
• Retries with back-off, timeouts, cancellation, job history.
• Priorities and per-tenant limits.

Non-functional

• A job must not be lost, and should not run twice.
• Start within a second or two of the scheduled time.
• Scheduler nodes and workers can die at any moment.

Design

1. A job is stored with its next run time and an index on that time.
2. Schedulers each own a slice of the jobs. Every second a scheduler reads jobs due in the next short interval, marks them queued in the same transaction, and places them on a queue.
3. Workers take jobs, send heartbeats while running, and report the result.
4. For a recurring job, the scheduler computes the following run time after each firing.
5. If a worker's heartbeat stops, the job's lease expires and it is queued again.

Key decisions and trade-offs

• Exactly-once is not available; aim for at-least-once with idempotent jobs. The claim step uses a conditional update so two schedulers cannot both enqueue one job, but a worker can still die after finishing.
• Partition the schedule. One scheduler scanning every job does not scale; partition by job id and assign partitions through a coordinator.
• Polling the database or a timing wheel. Database polling is simple and durable. An in-memory timing wheel is faster but must be rebuilt after a restart.
• Missed runs. After an outage, decide per job whether to run all missed occurrences, only the latest, or none.

Follow-up questions

• How do you handle a million jobs scheduled for exactly midnight?
• How would you support jobs that depend on other jobs?
• What stops one tenant's long jobs from starving everyone else?
• How do you change a job's schedule while an instance of it is running?