#60 Idempotent Callbacks
Payment Gateway Integration
Take payments through an external provider and stay correct when its confirmations arrive late, twice, out of order or not at all.
What is being asked
Your shop integrates with a payment provider. The provider tells you the outcome by calling your webhook. Design the integration so an order is fulfilled exactly once, whatever the network does.
Requirements
Functional
- Start a payment for an order; receive the result; fulfil the order.
- Handle refunds and disputes notified later.
Non-functional
- Never fulfil an unpaid order; never fail to fulfil a paid one.
- Webhooks may be duplicated, delayed, reordered or forged.
- Our own service may be down when the provider calls.
Design
- Create a payment record in state pending before calling the provider, and send our own id as the provider's idempotency key so a retry does not create a second charge.
- The webhook endpoint first verifies the signature, then looks up the event id in a processed-events table. If present, it answers 200 and stops.
- Otherwise, in one database transaction, it records the event id and moves the payment through its state machine, ignoring transitions that are not allowed, such as "succeeded" after "refunded".
- The same transaction writes a fulfilment message to an outbox; a worker delivers it.
- The endpoint returns 200 only after the commit. Any error returns non-200 so the provider retries.
Key decisions and trade-offs
- Idempotent handlers are the whole point: the unique event id makes a duplicate harmless.
- Order of events is not guaranteed. Decide by the state machine and, where in doubt, fetch the payment's current status from the provider rather than trusting the event's content.
- Do not rely on the customer's browser returning to your success page; the webhook is the source of truth.
- Reconciliation catches the rare lost webhook by listing the provider's payments periodically and comparing.
Follow-up questions
- What if the webhook arrives before our own create-payment call has returned?
- How long do you keep processed event ids?
- How do you safely replay a day of webhooks after an outage?
- How do you test all this without moving real money?