#63 Request Queuing/GPU Orchestration
ChatGPT/LLM Backend
Serve a large language model to millions of users: queue requests fairly, pack them onto scarce GPUs, and stream answers word by word.
What is being asked
Design the serving side of an AI chat product. Each request needs a GPU for several seconds, capacity is limited and expensive, and users expect the reply to start appearing at once.
Requirements
Functional
- Send a conversation; receive a streamed reply.
- Conversation history; several models and tiers.
- Rate limits and usage accounting per user.
Non-functional
- Time to first token of about a second.
- GPUs fully used, since they dominate cost.
- Graceful behaviour when demand exceeds capacity.
Design
- The gateway authenticates and applies limits counted in tokens, not requests, since cost follows length.
- The conversation service assembles the prompt from history and enqueues the request by model and priority.
- A scheduler assigns requests to GPU workers. Workers use continuous batching: several requests share one model on one GPU, each generating a token per step, with new requests joining as others finish.
- Tokens are streamed back to the client as they are produced.
- The finished exchange is stored and usage recorded.
Key decisions and trade-offs
- Queue instead of failing. A short wait with a visible position is better than an error, and priority queues protect paying tiers.
- Batching raises throughput but adds latency per token; the batch size is tuned to meet the latency target.
- Autoscaling is slow. Loading a model onto a GPU takes minutes and GPUs may not be available, so capacity is planned ahead and overflow is routed to smaller models.
- Caching the prompt prefix. Reusing the computed state for an unchanged conversation history or system prompt saves much of the work on each turn.
- Routing by affinity. Sending a conversation back to the worker that holds its cached state is faster but conflicts with even load balancing.
Follow-up questions
- How do you cancel generation when the user closes the tab?
- How would you keep one user with very long prompts from starving others?
- What happens to in-flight requests when a GPU node fails?
- How do you roll out a new model version safely?