#24 Orchestration/Metadata
Distributed Database Control Plane
The management layer of a database-as-a-service: create, resize, patch and repair thousands of database clusters automatically.
What is being asked
A cloud provider offers managed databases. Design the control plane: the system customers call to create or change a database, and which keeps every cluster healthy, separate from the data plane that actually serves queries.
Requirements
Functional
- Create, delete, resize, back up, restore, upgrade.
- Detect failed nodes and replace them.
- Show status and metrics; meter usage for billing.
Non-functional
- A control plane outage must not take running databases down.
- Operations are long, take many steps, and must survive crashes halfway.
- Thousands of clusters across regions.
Design
- The API does not act directly. It records the desired state ("cluster X: version 16, 3 nodes, 100 GB") and returns an operation id.
- Reconcilers continuously compare desired state with what really exists and take the next step to close the gap: provision a machine, attach storage, start the database, join it to the cluster.
- Multi-step operations run as durable workflows whose progress is persisted, so a crashed worker resumes where it stopped.
- An agent on each node reports health. On failure, the control plane promotes a replica and provisions a replacement.
Key decisions and trade-offs
- Declarative, not imperative. Storing the goal and reconciling towards it makes every operation retryable and self-healing; a script of commands breaks at the first failure.
- Static stability. The data plane must keep working with the control plane unreachable, so failover logic lives close to the databases and depends on little else.
- Idempotent steps. Every step may be executed twice, so each must check before acting.
- Blast radius. Roll changes out cluster by cluster and region by region, stopping automatically on errors.
Follow-up questions
- How do you upgrade ten thousand clusters without a mass outage?
- How does failover avoid two primaries?
- What stops a buggy reconciler from deleting healthy nodes?
- How would you design quotas so one customer cannot exhaust capacity?