Skip to Content
Cloud Engineering Academy
11. Monitoring, Logging & Troubleshooting

Distributed tracing with OpenTelemetry and AWS X-Ray for .NET (ADOT)

Instrument an ASP.NET Core 10 API with the OpenTelemetry SDK, add your own spans, export to a local Jaeger and to AWS (X-Ray / CloudWatch traces) through the ADOT collector, sample sensibly, and jump from a log line to its trace.

advanced⏱ 2 h 🟡 Low costSOA-C03DOP-C02DVA-C03

Assumes: .NET 10 LTS · OpenTelemetry .NET 1.21.0 · ADOT .NET auto-instrumentation 1.15.0 · AWS X-Ray OTLP endpoint · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain a trace, a span, context propagation and sampling, and how they map to .NET's Activity and ActivitySource.
  • Instrument an ASP.NET Core 10 API with the OpenTelemetry SDK, add custom spans around a database call and a partner call, and record errors on spans.
  • Run the lab's API, an OpenTelemetry collector and Jaeger locally and read a trace for a normal, a slow-database and a partner-outage request.
  • Describe the three ways to get traces into AWS (collector-less OTLP, the ADOT collector, Lambda and ECS integrations), their prerequisites, and what changed for .NET.
  • Choose a sampling strategy that keeps errors and controls cost, and connect a log line to its trace using the trace ID.

Prerequisites

  • Observability fundamentals: why traces exist and how they link to logs.
  • CloudWatch Logs for .NET: the formatter that writes traceId and spanId into every log line.
  • API monitoring: the edge-to-API view of tracing for API Gateway.
  • Tools: Docker (for Jaeger and the collector), .NET 10 SDK or Docker, AWS CLI v2 (for the AWS part).

Why this matters in banking

A customer's transfer takes eleven seconds. The API's p50 is 40 ms and its p99 is 900 ms, so it is not "slow" by the numbers. Logs show the request finished with status 201 after 11,034 ms, which tells you that it was slow but not why. The transfer touched the API, a sanctions-screening partner, an idempotency check in DynamoDB, a Postgres insert and a message to a queue. Which one cost ten seconds?

A distributed trace answers this in one picture: a timeline of every step of that one request, nested by who called whom, with each step's duration and status. The ten-second bar is the partner call. Without a trace you would be comparing timestamps across five log streams by hand. In a bank, where one payment crosses many services and some belong to third parties, tracing is also how you hold a dependency to its service-level agreement with evidence.

The idea in plain language

A trace is the story of one request. It is made of spans: each span is one unit of work with a name, a start time, a duration, a status and attributes (key/value tags). Spans form a tree: the incoming HTTP request is the root span, the database insert and the partner call are child spans.

To link spans across processes, the caller passes the trace context to the callee in an HTTP header. The standard is W3C Trace Context:

traceparent: 00-4efaaf4d1e8720b39541901950019ee5-53995c3f42cd8ad8-01
             ver  trace-id (32 hex)                parent span-id    flags (01 = sampled)

The callee starts its own spans with the same trace ID and the caller's span as parent. That is context propagation. AWS services also use their own header, X-Amzn-Trace-Id (for example Root=1-58406520-a006649127e371903a2de979;Parent=53995c3f42cd8ad8;Sampled=1), which load balancers and API Gateway add.

Analogy: A trace is a parcel's tracking history. The tracking number (trace ID) never changes; every depot it passes adds a scan (span) with its own times. The scans nest: "sorting centre" contains "conveyor 3" and "weighing". When a parcel is late you do not ask each depot to describe its day; you read the one tracking page and see which scan sat for nine hours.

.NET and OpenTelemetry: the part you already know

.NET has had distributed tracing built in since .NET 5, through System.Diagnostics:

.NET typeOpenTelemetry concept
ActivitySourceA tracer: the named source of spans
ActivityA span
Activity.CurrentThe current span
activity.SetTag(k, v)A span attribute
activity.AddEvent(...)A span event (a timestamped note)
activity.SetStatus(ActivityStatusCode.Error, ...)Span status
Activity.TraceId / SpanIdThe IDs you put in logs

OpenTelemetry (OTel) is the vendor-neutral standard and SDK that listens to those ActivitySources, applies a sampler, and exports spans with OTLP (the OpenTelemetry protocol) to a backend. ASP.NET Core, HttpClient and many libraries already create Activity objects; the OTel packages add the exporter and the instrumentation glue.

What problem the AWS services solve

AWS provides the backend for traces and the distribution of OpenTelemetry (ADOT):

  • AWS X-Ray and CloudWatch traces: store, search and visualise traces and build service maps.
  • ADOT, the AWS Distro for OpenTelemetry: AWS-supported builds of the collector and of language instrumentation, including a .NET auto-instrumentation package.
  • CloudWatch Application Signals (see Application Signals): service maps and SLOs built on the same telemetry.

What changed, verified on 2026-11-18 against the AWS documentation (links under Further reading):

  • X-Ray accepts spans directly on an OTLP endpoint, https://xray.<region>.amazonaws.com/v1/traces, over HTTP with protobuf only (not gRPC). To send traces there, Transaction Search must be enabled. Requests are signed with SigV4 and the sender needs the X-Ray write permissions.
  • The X-Ray SDKs and daemon are in maintenance mode since 2026-02-25, as the API monitoring lesson also states ⚠ VERIFY on the X-Ray documentation. New .NET code should use OpenTelemetry, with X-Ray as the backend.
  • With the ADOT .NET auto-instrumentation, collector-less export needs ADOT .NET 1.8.0 or later, plus two settings that are unique to .NET: OTEL_TRACES_EXPORTER=none and OTEL_AWS_SIG_V4_ENABLED=true. Compression and logs export are not supported for .NET in that mode.
  • When you send traces without a collector, the SDK default sampler is parentbased_always_on: 100% of traces. The documentation warns this can raise ingestion volume and cost by up to 20 times compared with a recommended 5% rate. Set OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.05 (or the equivalent in code) unless you really want full capture.
  • The AWS.Distro.OpenTelemetry.AutoInstrumentation NuGet package is at 1.15.0 (checked 2026-11-18).
  • X-Ray trace ID format is 1-<8 hex digits of Unix time>-<24 hex digits>; a W3C trace ID must be formatted this way when sent to X-Ray (for example 4efaaf4d1e8720b39541901950019ee5 becomes 1-4efaaf4d-1e8720b39541901950019ee5). The lab's XRayTraceId class converts both ways.
  • X-Ray free tier (pricing page): 100,000 traces recorded and 1,000,000 retrieved or scanned per month ⚠ VERIFY current limits and the Transaction Search price.

Three ways to export from a .NET service to AWS:

RouteHowWhen
Collector-lessADOT .NET auto-instrumentation sends OTLP straight to the X-Ray endpoint with SigV4Simple; small services; accept the .NET-specific settings above
ADOT collector (sidecar on ECS, or a daemon on EC2/EKS)App sends OTLP to a local collector; the collector batches, samples, scrubs attributes and exports to X-RayProduction: central control of sampling and data scrubbing
LambdaActive tracing for X-Ray, or the ADOT Lambda layer ⚠ VERIFY current layer names and .NET 10 supportServerless APIs; see Lambda with C#

Honest limitations:

  • Tracing is sampled: the trace you want may not exist unless you sample errors and slow requests deliberately.
  • Spans carry attributes; attributes can leak personal data (URLs with account numbers, SQL text with values). Scrub them.
  • Instrumentation libraries differ in maturity. The OpenTelemetry.Instrumentation.EntityFrameworkCore package was still published as a beta at the time of checking (1.21.0-beta.1), so decide deliberately whether to take a pre-release dependency in a regulated service.
  • OpenTelemetry.Contrib.Extensions.AWSXRay (1.2.0) provides an X-Ray propagator and ID generator, but when tested on 2026-11-18 it pulled in Newtonsoft.Json 11.0.1, which NuGet flags with a known high-severity advisory and which fails a build that treats audit warnings as errors. Test before you adopt it, and prefer W3C context propagation where you can.
  • Trace backends have retention and query limits; keep your own evidence elsewhere if regulators require it.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Application Insights SDK (TelemetryClient, operation_Id)OpenTelemetry SDK + ActivitySource; trace IDOne SDK for all signals; trace ID replaces operation_Id
Azure Monitor OpenTelemetry Distro (UseAzureMonitor())ADOT / OTLP exporter to X-RaySame OTel API; only the exporter and the backend change
Application MapX-Ray service map / CloudWatch trace map / Application SignalsBuilt from spans automatically
Dependency tracking (automatic)AddHttpClientInstrumentation(), AddAWSInstrumentation(), EF Core instrumentationEach library has its own instrumentation package
Adaptive samplingHead sampling in the SDK, tail sampling in the collector, X-Ray sampling rulesYou choose and configure the strategy
Correlation via Request-Id / traceparenttraceparent plus X-Amzn-Trace-IdAdd the AWS propagator only if you need ALB or API Gateway IDs
Live MetricsLive Tail on logs; no direct equivalent for tracesTraces appear after export and indexing delays

Architecture

Security-boundary view. The collector is a trust boundary: it is the one place that holds the AWS permission to write traces (the task role with X-Ray write actions), and the best place to delete sensitive attributes before they leave. Keep the application's own role free of tracing permissions when a collector sits in front. The collector's OTLP port (4318) must be reachable only from the same task (localhost) or a private network, never from the internet. In the lab's Docker network it is container-to-container only.

Step-by-step implementation

Add the OpenTelemetry packages

In the API project: OpenTelemetry.Extensions.Hosting, OpenTelemetry.Exporter.OpenTelemetryProtocol, OpenTelemetry.Instrumentation.AspNetCore, OpenTelemetry.Instrumentation.Http (all 1.21.0 on 2026-11-18). Add OpenTelemetry.Instrumentation.AWS when the service calls AWS SDK clients.

Register tracing, with your own source name

Call AddOpenTelemetry().WithTracing(...), AddSource("Harbourline.Transfers") for your manual spans, the AspNetCore and HttpClient instrumentations, and the OTLP exporter. Set the service name and environment on the resource: that is what appears on the service map.

Filter noise

Exclude /health from tracing. Health checks every few seconds would dominate your trace volume and cost.

Add spans where the time goes

Around the database insert and the partner call. Use ActivityKind.Client, set standard attributes (db.system, peer.service), and mark failures with SetStatus(ActivityStatusCode.Error, ...).

Put the trace ID in the logs

The lab's formatter already writes traceId and spanId from Activity.Current. That is the bridge from a log line to a trace.

Run it locally first

docker compose up --build starts the API, a collector and Jaeger. Send traffic with scripts/load.sh. Learn to read traces before you pay for them.

Choose a sampling strategy

Start with a parent-based ratio sampler (5%) in the SDK. Add tail sampling in the collector to keep 100% of errors and slow traces ⚠ VERIFY that your collector build includes the tail-sampling processor.

Point the exporter at AWS

On ECS: an ADOT collector sidecar with an awsxray exporter and a task role with the X-Ray write actions, and Transaction Search enabled in the account. Verify spans in the CloudWatch Traces view and the X-Ray service map.

AWS Console instructions

Enable Transaction Search (required for OTLP to X-Ray): Console → CloudWatch → Settings → Transaction Search → Enable (also reachable from X-Ray traces) ⚠ VERIFY the current wording; the documentation links a "Getting started with Transaction Search" page. It makes spans flow into a CloudWatch Logs group (aws/spans) and sets an indexing percentage that controls cost.

View traces: Console → CloudWatch → Application Signals → Traces (or X-Ray traces → Traces). Filter by service name, status or an annotation, open a trace to see the timeline and the trace map. To find the trace for a log line, convert the W3C trace ID in the log to X-Ray format (lowercase hex, 1- + first 8 + - + remaining 24) and paste it into the trace search.

AWS CLI

# Enable Transaction Search: send spans to CloudWatch Logs (needs a log resource policy; follow the console wizard first time)
aws xray update-trace-segment-destination --destination CloudWatchLogs --profile learn-sandbox
aws xray get-trace-segment-destination --profile learn-sandbox

# Recent slow or failed traces for the service (X-Ray filter expressions)
aws xray get-trace-summaries --profile learn-sandbox \
  --start-time "$(date -u -d '-15 minutes' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
  --filter-expression 'service("harbourline-transfers-api") AND (responsetime > 1 OR error OR fault)' \
  --query 'TraceSummaries[].[Id,Duration,HasError,HasFault]' --output table

# Fetch one full trace: the W3C id from a log line (4efaaf4d...) becomes 1-4efaaf4d-...
aws xray batch-get-traces --trace-ids 1-4efaaf4d-1e8720b39541901950019ee5 --profile learn-sandbox

# The service graph (who calls whom, error and latency per edge)
aws xray get-service-graph --profile learn-sandbox \
  --start-time "$(date -u -d '-15 minutes' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" --query 'Services[].Name'

The commands exist in AWS CLI v2; the exact behaviour with OTLP-sourced spans and Transaction Search (for example, which of your spans are indexed and therefore searchable by these calls) depends on your indexing settings ⚠ VERIFY. With Transaction Search you can also query the aws/spans log group with Logs Insights (Logs Insights queries).

Infrastructure as Code

The collector sidecar's configuration (ADOT or the upstream contrib collector; the exporter name awsxray is part of the AWS-supported distribution ⚠ VERIFY for your image version):

adot-collector-config.yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 128
  attributes/scrub:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: user.email
        action: delete
  batch: {}

exporters:
  awsxray:
    region: us-east-1

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, attributes/scrub, batch]
      exporters: [awsxray]

The task role needs only write access to X-Ray. A least-privilege policy for the task role:

TaskRoleXRay.cs
using Amazon.CDK.AWS.IAM;

// taskRole is the ECS task role the collector sidecar runs under.
taskRole.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
    Effect = Effect.ALLOW,
    Actions =
    [
        "xray:PutTraceSegments",
        "xray:PutTelemetryRecords",
        "xray:GetSamplingRules",
        "xray:GetSamplingTargets",
    ],
    Resources = ["*"], // X-Ray write actions do not support resource-level restrictions
}));

AWS publishes a managed policy for this, referred to in the OTLP documentation as AWSXrayWriteOnlyPolicy ⚠ VERIFY the exact managed-policy name in IAM before attaching; the explicit policy above shows what it must allow. For the sidecar container definition see Deploy to Fargate and add a second container with the collector image and --config pointing at the file (shipped via an environment variable or a config store ⚠ VERIFY image name and tag from the ADOT documentation).

Complete code

The registration (lab API)

labs/m11-observe-banking-api/src/Harbourline.Transfers.Api/Program.cs:

Program.cs (excerpt)
var otlpConfigured = !string.IsNullOrWhiteSpace(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]);
builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService(serviceName, serviceVersion: "1.0.0")
        .AddAttributes([new("deployment.environment.name", environmentName)]))
    .WithTracing(t =>
    {
        t.AddSource(Telemetry.SourceName)                      // "Harbourline.Transfers": our manual spans
         .AddAspNetCoreInstrumentation(o => o.Filter = ctx => !ctx.Request.Path.StartsWithSegments("/health"))
         .AddHttpClientInstrumentation();
        if (otlpConfigured) t.AddOtlpExporter();               // only when an endpoint is configured
    });

The manual spans

Transfers.cs: the database and the partner call, each as a span with a failure status:

Transfers.cs (excerpt)
public static class Telemetry
{
    public const string SourceName = "Harbourline.Transfers";
    public static readonly ActivitySource Source = new(SourceName);
}

public sealed class TransferStore
{
    public async Task<string> SaveAsync(TransferRequest request, int extraDelayMs, CancellationToken ct)
    {
        using var span = Telemetry.Source.StartActivity("db.insert transfers", ActivityKind.Client);
        span?.SetTag("db.system", "postgresql");
        span?.SetTag("db.operation", "INSERT");

        await Task.Delay(5 + extraDelayMs, ct);          // the slow-db fault makes this bar long in the trace

        var id = "PAY-" + Guid.CreateVersion7();
        span?.SetTag("payment.id", id);
        return id;
    }
}

public sealed class SanctionsPartner(Func<double> random)
{
    public async Task ScreenAsync(string creditorName, double failureRate, CancellationToken ct)
    {
        using var span = Telemetry.Source.StartActivity("partner.sanctions-screen", ActivityKind.Client);
        span?.SetTag("peer.service", "sanctions-partner");

        await Task.Delay(8, ct);

        if (failureRate > 0 && random() < failureRate)
        {
            span?.SetStatus(ActivityStatusCode.Error, "partner unavailable");   // the bar turns red
            throw new PartnerUnavailableException("Sanctions screening partner did not respond.");
        }
    }
}

A production-style SDK configuration

Sampling and, when you call AWS SDK clients, instrumentation for them (this compiles against OpenTelemetry 1.21.0 packages, checked in Docker on 2026-11-18):

TracingSetup.cs
using OpenTelemetry;
using OpenTelemetry.Context.Propagation;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

// W3C traceparent plus baggage. (An X-Ray propagator exists in a contrib package; see the dependency note above.)
Sdk.SetDefaultTextMapPropagator(new CompositeTextMapPropagator(
[
    new TraceContextPropagator(),
    new BaggagePropagator(),
]));

builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService("harbourline-transfers-api", serviceVersion: "1.0.0"))
    .WithTracing(t => t
        .AddSource("Harbourline.Transfers")
        .AddAspNetCoreInstrumentation(o => o.Filter = ctx => !ctx.Request.Path.StartsWithSegments("/health"))
        .AddHttpClientInstrumentation()
        .AddAWSInstrumentation()                                                  // spans for AWS SDK calls
        .SetSampler(new ParentBasedSampler(new TraceIdRatioBasedSampler(0.05)))   // keep 5%, honour the caller's decision
        .AddOtlpExporter());                                                      // endpoint from OTEL_EXPORTER_OTLP_ENDPOINT

var app = builder.Build();
app.MapGet("/", () => "ok");
app.Run();

Converting between the two trace ID formats

Tracing/XRayTraceId.cs in the lab (unit-tested):

XRayTraceId usage
XRayTraceId.FromW3C("4efaaf4d1e8720b39541901950019ee5");        // "1-4efaaf4d-1e8720b39541901950019ee5"
XRayTraceId.ToW3C("1-4efaaf4d-1e8720b39541901950019ee5");       // "4efaaf4d1e8720b39541901950019ee5"
XRayTraceId.StartTime("1-58406520-a006649127e371903a2de979");   // the request start time encoded in the ID
XRayTraceId.BuildHeader("58406520a006649127e371903a2de979", "53995c3f42cd8ad8", sampled: true);
// "Root=1-58406520-a006649127e371903a2de979;Parent=53995c3f42cd8ad8;Sampled=1"

Code explained line by line

  1. AddSource("Harbourline.Transfers") tells the OTel SDK to listen to your ActivitySource. If the names do not match exactly, your manual spans are created and silently dropped; this is the most common "my spans are missing" bug.
  2. ConfigureResource(... AddService(...)) sets service.name, which is how the service appears on the service map. Without it every service is unknown_service.
  3. Filter = ctx => !ctx.Request.Path.StartsWithSegments("/health") removes health-check requests from tracing before they create spans.
  4. if (otlpConfigured) t.AddOtlpExporter(); keeps local runs and tests free of exporter errors. The exporter reads its endpoint and protocol from the standard OTEL_EXPORTER_OTLP_* environment variables.
  5. StartActivity(..., ActivityKind.Client) marks the span as an outgoing call, which backends use to draw dependency edges. It returns null when no listener samples it, so every use is span?..
  6. SetStatus(ActivityStatusCode.Error, ...) is what turns a span red in Jaeger and counts it as an error in X-Ray; throwing the exception alone does not always mark the span.
  7. ParentBasedSampler(new TraceIdRatioBasedSampler(0.05)) means: if the caller already decided (the sampled flag in traceparent), follow it; otherwise keep 5% of new traces, chosen deterministically from the trace ID so every service in the path makes the same decision.
  8. CompositeTextMapPropagator lists the header formats the app reads and writes. BaggagePropagator carries key/value baggage; leave it out unless you need it, because baggage values travel to every downstream service.

Security considerations

⚠ Learning shortcut — not for production

The lab sends traces over plain HTTP between containers on a private Docker network, with tls.insecure: true towards Jaeger, and traces every request. Production: the collector runs as a sidecar reachable only on localhost, exports to AWS over TLS with SigV4, samples (for example 5% plus all errors) and deletes sensitive attributes before export.

  • Spans can carry personal data. HTTP instrumentation records URLs; if the path contains an account number (/accounts/ACC-0000000001) it is now in your trace store. Use route templates (/accounts/{id}) as the span name, and do not record request bodies or SQL parameter values. EF Core instrumentation can record SQL text: check the option that controls it before enabling.
  • Baggage travels everywhere. Never put identifiers or secrets in baggage.
  • Least privilege for tracing. X-Ray write permissions on the collector's role only; xray:Get* read permissions for engineers' operations role.
  • Trace retention. Transaction Search stores spans in a log group with a retention setting you control; set it, as with any log group (lesson 2).
  • Third-party propagation. Sending traceparent to external partners tells them your trace ID. That is normally harmless, but for some partners you may need to strip the header; check the contract.
  • Supply chain. Instrumentation packages run in your process with full privileges. Pin versions, review advisories (the NuGet audit warning mentioned above is exactly this), and avoid pre-release packages in regulated services unless risk-accepted.

Banking use case

Harbourline's payments team traces POST /v1/transfers end to end: API Gateway, the Transfers API, the sanctions partner, the database and the queue consumer. They sample 5% of ordinary traffic and keep every error and every request over one second (tail sampling in the collector). When a partner's latency degrades, the trace shows the sanctions span growing from 80 ms to 4 s while every other span stays flat, and the team attaches the traces to the partner's incident ticket as evidence for the service-level review. Each log line carries traceId, so support can go from a customer's correlation ID to the log lines to the trace in three steps. Attributes are scrubbed in the collector: no account numbers, names or authorisation headers leave the task. All data is fictional.

Common mistakes

  • AddSource name mismatch. Manual spans vanish. Fix: use a constant shared by the ActivitySource and the registration.
  • Tracing everything at 100%. The collector-less default is full capture. Fix: set a sampler, and watch the trace count.
  • No error marking. A failed call that is caught and handled looks green. Fix: SetStatus(Error) and record the exception.
  • High-cardinality span names (GET /accounts/ACC-0000000001). Fix: route templates; put the ID in an attribute only if policy allows.
  • Tracing health checks. Fix: filter them out.
  • Forgetting Transaction Search. The OTLP endpoint rejects or ignores spans. Fix: enable it first and check the documentation for the current prerequisite.
  • Wrong .NET settings with ADOT auto-instrumentation. Missing OTEL_TRACES_EXPORTER=none and OTEL_AWS_SIG_V4_ENABLED=true. Fix: follow the .NET-specific instructions exactly.
  • Putting the exporter in the request path. Direct synchronous export slows requests. Fix: the SDK batches spans; export to a local collector.

Troubleshooting

SymptomLikely causeFix
No spans in JaegerOTEL_EXPORTER_OTLP_ENDPOINT not set, collector down, or wrong protocol/portdocker compose logs collector; the lab sets http://collector:4318 and http/protobuf
Server spans appear but custom spans do notAddSource name differs from ActivitySource nameMake both use Telemetry.SourceName
Spans appear in Jaeger but not in X-RayTransaction Search not enabled, task role lacks X-Ray write, or wrong endpoint/RegionEnable Transaction Search; check the role; confirm https://xray.us-east-1.amazonaws.com/v1/traces
Collector-less ADOT .NET exports nothingMissing OTEL_TRACES_EXPORTER=none or OTEL_AWS_SIG_V4_ENABLED=trueSet both, as required for .NET
Trace is broken in two halvesContext not propagated (a custom HTTP client, a queue message without headers)Propagate traceparent; for queues put it in message attributes (see Module 8)
Unexpectedly high trace cost100% samplingSet a ratio sampler or collector sampling
traceId in logs is all zeros or missingNo Activity is active (code outside a request or tracing off)Ensure the ASP.NET Core instrumentation or an ActivityListener is registered

Cost and cleanup

Local (free): the Docker Compose stack costs nothing. AWS (credits): the free tier lists 100,000 traces recorded and 1,000,000 retrieved or scanned per month; Transaction Search stores spans in CloudWatch Logs at log ingestion prices (USD 0.50 per GB for Standard class at the time of checking ⚠ VERIFY), so a lab with a few hundred requests costs fractions of a cent, and the real risk is leaving 100% sampling on under load. For the lab, use the free local path for all practice, and if you want the AWS view, run a few dozen requests with a 100% sampler, look, and tear down within the hour.

# Local
docker compose down --rmi local

# AWS: remove what you created
aws logs delete-log-group --log-group-name aws/spans --profile learn-sandbox   # only if you created it for the lab; it holds spans
aws xray update-trace-segment-destination --destination XRay --profile learn-sandbox   # switch Transaction Search off ⚠ VERIFY
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab --profile learn-sandbox

Check with your team before deleting aws/spans or switching Transaction Search off in a shared account.

Exercises

  • 🟢 Guided: Run docker compose up --build, send 20 requests with scripts/load.sh, open Jaeger at http://localhost:16686, and find a normal trace. Note the three spans (server, partner, database) and their durations. Then send 10 with slow-db and 10 with partner-down and describe what changed.
  • 🟡 Challenge: Add a span queue.publish payment-event (kind Producer) after the database insert, tagged with messaging.system = sqs. Send a request and confirm the new span appears under the server span in Jaeger. Then change the sampler to ParentBasedSampler(new TraceIdRatioBasedSampler(0.5)) and observe how many of 20 requests produce traces.
  • 🔴 Live problem: "Some transfers take 6 to 12 seconds on Monday mornings, only for customers whose payee is at one partner bank. p99 over the day looks fine and logs show nothing wrong." You have traces for 5% of requests. Describe how you would capture the problem traces reliably (sampling change, attributes, tail sampling), how you would find the pattern, and what you would give the partner as evidence.

Assignment

Instrument a service of your own with OpenTelemetry: the SDK registration, two custom spans with failure status, a health-check filter, a 5% parent-based sampler, a collector configuration that deletes two sensitive attributes, and a README with a screenshot description of a normal and a failing trace. If you use AWS, add the least-privilege X-Ray policy in IaC.

Solutions

Knowledge check

Interview questions

  • Explain trace, span and context propagation. A trace is one request's journey; spans are timed units of work forming a tree; propagation passes the trace ID and parent span ID between services in headers such as traceparent.
  • Why OpenTelemetry instead of the X-Ray SDK for new .NET code? It is vendor-neutral, built on .NET's own Activity types, and the X-Ray SDKs and daemon are in maintenance mode; X-Ray remains a backend via OTLP or the ADOT collector.
  • How do you control tracing cost? Sampling (head ratio in the SDK, tail sampling in the collector to keep errors and slow requests), filtering health checks, and limiting attributes.
  • What can go wrong with traces in a bank? Personal data in attributes (URLs, SQL), unbounded retention, over-privileged tracing roles, and untested sampling that drops the one trace you need.
  • Where does a collector add value? Central batching, sampling, scrubbing of attributes and credentials handling in one place, instead of in every service.

Certification practice

Completion checklist

  • I can explain trace, span, propagation and sampling using .NET Activity terms
  • I ran the Docker Compose stack and read a normal, a slow and a failed trace in Jaeger
  • I added or read a custom span with failure status
  • I can state the .NET-specific ADOT settings and the Transaction Search prerequisite
  • I converted a W3C trace ID to X-Ray format by hand and with XRayTraceId
  • I tore down any AWS resources and confirmed nothing tagged Project=cea-lab remains

Key takeaways

  • Traces answer "where did the time go?"; traceId in every log line links logs to traces.
  • New .NET code uses OpenTelemetry over ActivitySource; X-Ray is a backend reached through OTLP or the ADOT collector.
  • Collector-less export to X-Ray needs Transaction Search, SigV4, and for .NET two specific settings; its default 100% sampling can be costly.
  • Put a collector in front for sampling, scrubbing and a single IAM boundary in production.
  • Spans are data: scrub attributes, filter health checks and set retention.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page