#31 Context Propagation/Logging
Distributed Tracing
Follow one request across dozens of microservices and see where the time went.
What is being asked
A user request fans out through many services. Design the system that records what each service did for that request and assembles a single timeline, a trace, that engineers can search and inspect.
Requirements
Functional
- Give each request a trace id and propagate it through every call.
- Record spans: service, operation, start, duration, parent, tags.
- Search traces by service, duration, error and tag; show the tree.
Non-functional
- Negligible overhead on the traced services.
- Enormous volume: every request creates many spans.
- Tracing failures must never break the application.
Design
- The first service creates a trace id. Each unit of work is a span with its own id and its parent's id.
- Context propagation: the trace id and current span id travel in request headers (the W3C
traceparentheader) and in message metadata, so the next service continues the same trace. - Libraries record spans in memory and hand them asynchronously to a local agent, which batches them to collectors.
- Collectors buffer in a log. A processor decides which traces to keep, groups spans by trace id and writes them to storage indexed by service, time and tags.
Key decisions and trade-offs
- Sampling is unavoidable. Keeping everything costs more than the systems being traced. Head sampling decides at the start (cheap, but may miss the rare slow request). Tail sampling decides after the trace completes and can keep every error and slow trace, but must buffer all spans briefly.
- Asynchronous and lossy by design. Spans are dropped under pressure rather than slowing the application.
- Clock skew. Machines disagree on time, so trees are built from parent ids and child spans are adjusted to sit inside their parents.
- Standard instrumentation (OpenTelemetry) avoids each team inventing its own.
Follow-up questions
- How does the trace id cross a message queue or a batch job?
- How would you find the traces behind a spike in the 99th percentile?
- How do you keep personal data out of span tags?
- How would you derive service dependency maps from traces?