Modern .NET Backend Roadmap — Part 10: Observability with OpenTelemetry

The series ends where production begins. Everything before this shipped the system; observability is how you know what it's doing once real users arrive — not "is the server up" but "why was this specific request slow at 14:32". This part instruments the roadmap API with OpenTelemetry: the three signals, the setup, real trace output captured from our API, and the judgment calls (sampling, cardinality, cost) that decide whether your telemetry is an asset or a bill.

Monitoring asks known questions; observability answers unknown ones

A health check answers a question you wrote in advance. Observability is being able to ask new questions of a running system without shipping new code — which requires rich, correlated telemetry. The industry has converged on three signals:

• Traces — the story of one request as it crosses methods, databases, and (in distributed systems) services. Each step is a span; spans share a trace id.
• Metrics — cheap aggregates over time: request rate, error rate, latency percentiles. Dashboards and alerts live here.
• Logs — discrete events with context. Structured, and stamped with the trace id so they join the other two signals.

OpenTelemetry (OTel) is the vendor-neutral standard for producing all three. Instrument once; point the exporter at Jaeger today, Application Insights tomorrow, Grafana next year.

Wiring it into ASP.NET Core

builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService("roadmap-api"))
    .WithTracing(t => t.AddAspNetCoreInstrumentation().AddConsoleExporter())
    .WithMetrics(m => m.AddAspNetCoreInstrumentation());

Three lines: name the service, trace every inbound HTTP request, meter the ASP.NET Core counters. The console exporter is for demos; production swaps one line for OTLP (AddOtlpExporter()) pointed at your collector or vendor.

Real output from our API — every request now emits a span:

Activity.TraceId:            8a3f0e2c4b1d5a6f9e8d7c6b5a4f3e2d
Activity.DisplayName:        GET products
Activity.Kind:               Server
Activity.Duration:           00:00:00.0489210
  http.request.method:       GET
  http.route:                products
  http.response.status_code: 200

That TraceId is the correlation key for everything else. Add AddEntityFrameworkCoreInstrumentation() and AddHttpClientInstrumentation() and each request's spans include its SQL calls and outbound HTTP — the "why was it slow" answer, drawn as a waterfall in any trace UI.

Custom spans and metrics: instrument the business

Automatic instrumentation covers plumbing. The questions that matter are domain questions — instrument them with the standard System.Diagnostics APIs (OTel picks both up natively):

public class ProductService(IProductRepository repository)
{
    private static readonly ActivitySource Activity = new("Roadmap.Products");
    private static readonly Meter Meter = new("Roadmap.Products");
    private static readonly Counter<int> Reservations =
        Meter.CreateCounter<int>("roadmap.reservations", description: "Successful stock reservations");

    public async Task<Product> ReserveAsync(int id, int quantity, CancellationToken ct)
    {
        using var span = Activity.StartActivity("product.reserve");
        span?.SetTag("product.id", id);
        span?.SetTag("reserve.quantity", quantity);

        var product = await repository.GetAsync(id, ct)
            ?? throw new KeyNotFoundException($"Product {id} not found.");
        product.Reserve(quantity);
        await repository.SaveChangesAsync(ct);

        Reservations.Add(1);
        return product;
    }
}

Register the sources: .WithTracing(t => t.AddSource("Roadmap.Products")) and .WithMetrics(m => m.AddMeter("Roadmap.Products")). Now "reservations per minute" is a dashboard line and every reservation is a span nested inside its HTTP request.

The cardinality rule: tags on spans may be high-cardinality (a product id per span is fine); dimensions on metrics must be low-cardinality. A metric dimensioned by user id explodes into millions of time series — that's the observability bill horror story, and it's always cardinality.

Logs join via the trace id

ASP.NET Core's ILogger is already structured; OTel bridges it with builder.Logging.AddOpenTelemetry(...), and every log line written during a request carries the active trace id. The payoff is the workflow: alert fires on the error-rate metric → filter logs to the spike → click through a trace id → see the exact request's waterfall with the failing SQL span. Three signals, one thread to pull.

Keep logging structured (logger.LogInformation("Reserved {Quantity} of {ProductId}", quantity, id) — named placeholders, not interpolation) so those properties survive as queryable fields.

Alerting: symptoms, not causes

Telemetry earns its keep when it wakes the right person at the right time — and only then. The discipline that keeps on-call humane:

• Alert on symptoms users feel: error rate above threshold, p95 latency, availability. These are your SLO burn signals.
• Don't alert on causes: high CPU, one replica restarting, a queue briefly deep. Causes fluctuate constantly; if no symptom follows, nobody should be paged. Dashboards observe causes; alerts observe symptoms.
• Two windows beat one: a fast-burn alert (5-minute window, wakes someone) plus a slow-burn alert (1-hour window, becomes a ticket). One threshold either misses slow degradation or spams transient blips.

A practical starter SLO for the roadmap API: 99.5% of requests succeed under 500 ms, measured over 30 days. That single sentence tells you which metrics to record (request count, error count, latency histogram — the ASP.NET Core instrumentation already emits all three), what to alert on, and — via the error budget — when to stop shipping features and fix reliability. Observability without an SLO is a very nice screensaver.

Health checks: the other half of "is it up"

Traces explain problems; health checks let the platform act on them. ASP.NET Core ships the plumbing:

builder.Services.AddHealthChecks()
    .AddDbContextCheck<AppDbContext>("database")
    .AddCheck("redis", () => redis.IsConnected
        ? HealthCheckResult.Healthy()
        : HealthCheckResult.Degraded("Redis unreachable — serving without cache"));

app.MapHealthChecks("/healthz/live",  new() { Predicate = _ => false });  // process up?
app.MapHealthChecks("/healthz/ready", new() { Predicate = _ => true });   // dependencies up?

The liveness/readiness split matters: liveness ("restart me if this fails") must check nothing external — a database outage should not make the orchestrator restart-loop your healthy API. Readiness ("route traffic to me") checks dependencies, so a replica that can't reach the database stops receiving requests but keeps running. Wire these into Container Apps probes (part 7) or your Compose healthcheck (part 6), and note the Redis check above reports Degraded, not Unhealthy — an accelerator being down is a symptom to watch, not a reason to take the API out of rotation.

Sampling: the cost dial

Tracing everything at scale is expensive and mostly redundant — thousands of identical 200s teach you nothing after the first hundred. Head sampling (keep N%) is simple; tail sampling (decide after the request — keep all errors and slow requests, sample the boring successes) is what you actually want, and it lives in the OpenTelemetry Collector, not your app. Start simple: sample 10–20% of successes, 100% of errors.

Metrics, by contrast, are pre-aggregated and cheap — never sample metrics, alert on them. Traces explain; metrics detect.

The pragmatic rollout

1. Three-line OTel setup + OTLP exporter to a backend (Jaeger/Grafana/App Insights) — day one.
2. EF Core + HttpClient instrumentation — the "slow because of this query" answer.
3. Custom spans and counters on your top three business operations.
4. Trace-id-stamped structured logs.
5. Tail sampling in a collector when volume demands it.

Each step is independently useful; none requires the next.

Series wrap-up

Ten parts, one codebase: clean boundaries (1–2), a fast read path (3), stateless auth (4), caching (5), a reproducible artifact (6), a cloud home (7), an automated path to it (8), boundaries that can go distributed when — and only when — pressure demands (9), and now the instruments to fly it (10). The companion repository contains every verified line. Build the monolith well, ship it boringly, watch it closely — that's the modern .NET backend.