A/B Testing Framework

What is being asked

Design an experimentation platform: define an experiment with variants and a traffic split, assign each user consistently to a variant, log what they saw and did, and report whether the difference is real.

Requirements

Functional

• Create experiments with variants, targeting rules and percentages.
• Assign a user to the same variant every time.
• Record exposures and outcomes; compute metrics per variant with statistical confidence.

Non-functional

• Assignment in under a millisecond, inside every request.
• A configuration outage must not take the product down.
• Many experiments running at once without interfering.

Design

1. Experiment definitions are stored centrally and pushed as a snapshot to an SDK inside each service.
2. The SDK assigns locally: hash the user id together with the experiment id, map the hash to a number from 0 to 99, and pick the variant whose range contains it. No network call is needed and the answer never changes.
3. The first time a user actually sees a variant, the SDK logs an exposure.
4. A pipeline joins exposures with later outcome events (purchases, clicks) and computes each metric per variant with a confidence interval.

Key decisions and trade-offs

• Hashing instead of storing assignments. Deterministic hashing needs no lookup and scales without limit. Salting with the experiment id keeps experiments independent of each other.
• Log exposure, not assignment. Counting users who were assigned but never reached the feature dilutes the result.
• Layers. Experiments that touch the same screen go in the same layer so a user is in only one of them; different layers overlap freely.
• Peeking. Checking results daily and stopping at the first good-looking number produces false wins. Fix the sample size in advance or use sequential tests.

Follow-up questions

• How do you ramp a risky change from 1% to 100% and roll back in seconds?
• How would you assign users who are not logged in?
• What is a sample ratio mismatch and how would you detect it?
• How do you keep a user in the same variant across devices?