Distributed tracing with OpenTelemetry and AWS X-Ray for .NET (ADOT)
Instrument an ASP.NET Core 10 API with the OpenTelemetry SDK, add your own spans, export to a local Jaeger and to AWS (X-Ray / CloudWatch traces) through the ADOT collector, sample sensibly, and jump from a log line to its trace.
Assumes: .NET 10 LTS · OpenTelemetry .NET 1.21.0 · ADOT .NET auto-instrumentation 1.15.0 · AWS X-Ray OTLP endpoint · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain a trace, a span, context propagation and sampling, and how they map to .NET's
ActivityandActivitySource. - Instrument an ASP.NET Core 10 API with the OpenTelemetry SDK, add custom spans around a database call and a partner call, and record errors on spans.
- Run the lab's API, an OpenTelemetry collector and Jaeger locally and read a trace for a normal, a slow-database and a partner-outage request.
- Describe the three ways to get traces into AWS (collector-less OTLP, the ADOT collector, Lambda and ECS integrations), their prerequisites, and what changed for .NET.
- Choose a sampling strategy that keeps errors and controls cost, and connect a log line to its trace using the trace ID.
Prerequisites
- Observability fundamentals: why traces exist and how they link to logs.
- CloudWatch Logs for .NET: the formatter that writes
traceIdandspanIdinto every log line. - API monitoring: the edge-to-API view of tracing for API Gateway.
- Tools: Docker (for Jaeger and the collector), .NET 10 SDK or Docker, AWS CLI v2 (for the AWS part).
Why this matters in banking
A customer's transfer takes eleven seconds. The API's p50 is 40 ms and its p99 is 900 ms, so it is not "slow" by the numbers. Logs show the request finished with status 201 after 11,034 ms, which tells you that it was slow but not why. The transfer touched the API, a sanctions-screening partner, an idempotency check in DynamoDB, a Postgres insert and a message to a queue. Which one cost ten seconds?
A distributed trace answers this in one picture: a timeline of every step of that one request, nested by who called whom, with each step's duration and status. The ten-second bar is the partner call. Without a trace you would be comparing timestamps across five log streams by hand. In a bank, where one payment crosses many services and some belong to third parties, tracing is also how you hold a dependency to its service-level agreement with evidence.
The idea in plain language
A trace is the story of one request. It is made of spans: each span is one unit of work with a name, a start time, a duration, a status and attributes (key/value tags). Spans form a tree: the incoming HTTP request is the root span, the database insert and the partner call are child spans.
To link spans across processes, the caller passes the trace context to the callee in an HTTP header. The standard is W3C Trace Context:
traceparent: 00-4efaaf4d1e8720b39541901950019ee5-53995c3f42cd8ad8-01
ver trace-id (32 hex) parent span-id flags (01 = sampled)The callee starts its own spans with the same trace ID and the caller's span as parent. That is context propagation. AWS services also use their own header, X-Amzn-Trace-Id (for example Root=1-58406520-a006649127e371903a2de979;Parent=53995c3f42cd8ad8;Sampled=1), which load balancers and API Gateway add.
Analogy: A trace is a parcel's tracking history. The tracking number (trace ID) never changes; every depot it passes adds a scan (span) with its own times. The scans nest: "sorting centre" contains "conveyor 3" and "weighing". When a parcel is late you do not ask each depot to describe its day; you read the one tracking page and see which scan sat for nine hours.
.NET and OpenTelemetry: the part you already know
.NET has had distributed tracing built in since .NET 5, through System.Diagnostics:
| .NET type | OpenTelemetry concept |
|---|---|
ActivitySource | A tracer: the named source of spans |
Activity | A span |
Activity.Current | The current span |
activity.SetTag(k, v) | A span attribute |
activity.AddEvent(...) | A span event (a timestamped note) |
activity.SetStatus(ActivityStatusCode.Error, ...) | Span status |
Activity.TraceId / SpanId | The IDs you put in logs |
OpenTelemetry (OTel) is the vendor-neutral standard and SDK that listens to those ActivitySources, applies a sampler, and exports spans with OTLP (the OpenTelemetry protocol) to a backend. ASP.NET Core, HttpClient and many libraries already create Activity objects; the OTel packages add the exporter and the instrumentation glue.
What problem the AWS services solve
AWS provides the backend for traces and the distribution of OpenTelemetry (ADOT):
- AWS X-Ray and CloudWatch traces: store, search and visualise traces and build service maps.
- ADOT, the AWS Distro for OpenTelemetry: AWS-supported builds of the collector and of language instrumentation, including a .NET auto-instrumentation package.
- CloudWatch Application Signals (see Application Signals): service maps and SLOs built on the same telemetry.
What changed, verified on 2026-11-18 against the AWS documentation (links under Further reading):
- X-Ray accepts spans directly on an OTLP endpoint,
https://xray.<region>.amazonaws.com/v1/traces, over HTTP with protobuf only (not gRPC). To send traces there, Transaction Search must be enabled. Requests are signed with SigV4 and the sender needs the X-Ray write permissions. - The X-Ray SDKs and daemon are in maintenance mode since 2026-02-25, as the API monitoring lesson also states ⚠ VERIFY on the X-Ray documentation. New .NET code should use OpenTelemetry, with X-Ray as the backend.
- With the ADOT .NET auto-instrumentation, collector-less export needs ADOT .NET 1.8.0 or later, plus two settings that are unique to .NET:
OTEL_TRACES_EXPORTER=noneandOTEL_AWS_SIG_V4_ENABLED=true. Compression and logs export are not supported for .NET in that mode. - When you send traces without a collector, the SDK default sampler is
parentbased_always_on: 100% of traces. The documentation warns this can raise ingestion volume and cost by up to 20 times compared with a recommended 5% rate. SetOTEL_TRACES_SAMPLER=parentbased_traceidratioandOTEL_TRACES_SAMPLER_ARG=0.05(or the equivalent in code) unless you really want full capture. - The
AWS.Distro.OpenTelemetry.AutoInstrumentationNuGet package is at 1.15.0 (checked 2026-11-18). - X-Ray trace ID format is
1-<8 hex digits of Unix time>-<24 hex digits>; a W3C trace ID must be formatted this way when sent to X-Ray (for example4efaaf4d1e8720b39541901950019ee5becomes1-4efaaf4d-1e8720b39541901950019ee5). The lab'sXRayTraceIdclass converts both ways. - X-Ray free tier (pricing page): 100,000 traces recorded and 1,000,000 retrieved or scanned per month ⚠ VERIFY current limits and the Transaction Search price.
Three ways to export from a .NET service to AWS:
| Route | How | When |
|---|---|---|
| Collector-less | ADOT .NET auto-instrumentation sends OTLP straight to the X-Ray endpoint with SigV4 | Simple; small services; accept the .NET-specific settings above |
| ADOT collector (sidecar on ECS, or a daemon on EC2/EKS) | App sends OTLP to a local collector; the collector batches, samples, scrubs attributes and exports to X-Ray | Production: central control of sampling and data scrubbing |
| Lambda | Active tracing for X-Ray, or the ADOT Lambda layer ⚠ VERIFY current layer names and .NET 10 support | Serverless APIs; see Lambda with C# |
Honest limitations:
- Tracing is sampled: the trace you want may not exist unless you sample errors and slow requests deliberately.
- Spans carry attributes; attributes can leak personal data (URLs with account numbers, SQL text with values). Scrub them.
- Instrumentation libraries differ in maturity. The
OpenTelemetry.Instrumentation.EntityFrameworkCorepackage was still published as a beta at the time of checking (1.21.0-beta.1), so decide deliberately whether to take a pre-release dependency in a regulated service. OpenTelemetry.Contrib.Extensions.AWSXRay(1.2.0) provides an X-Ray propagator and ID generator, but when tested on 2026-11-18 it pulled inNewtonsoft.Json11.0.1, which NuGet flags with a known high-severity advisory and which fails a build that treats audit warnings as errors. Test before you adopt it, and prefer W3C context propagation where you can.- Trace backends have retention and query limits; keep your own evidence elsewhere if regulators require it.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
Application Insights SDK (TelemetryClient, operation_Id) | OpenTelemetry SDK + ActivitySource; trace ID | One SDK for all signals; trace ID replaces operation_Id |
Azure Monitor OpenTelemetry Distro (UseAzureMonitor()) | ADOT / OTLP exporter to X-Ray | Same OTel API; only the exporter and the backend change |
| Application Map | X-Ray service map / CloudWatch trace map / Application Signals | Built from spans automatically |
| Dependency tracking (automatic) | AddHttpClientInstrumentation(), AddAWSInstrumentation(), EF Core instrumentation | Each library has its own instrumentation package |
| Adaptive sampling | Head sampling in the SDK, tail sampling in the collector, X-Ray sampling rules | You choose and configure the strategy |
Correlation via Request-Id / traceparent | traceparent plus X-Amzn-Trace-Id | Add the AWS propagator only if you need ALB or API Gateway IDs |
| Live Metrics | Live Tail on logs; no direct equivalent for traces | Traces appear after export and indexing delays |
Architecture
Security-boundary view. The collector is a trust boundary: it is the one place that holds the AWS permission to write traces (the task role with X-Ray write actions), and the best place to delete sensitive attributes before they leave. Keep the application's own role free of tracing permissions when a collector sits in front. The collector's OTLP port (4318) must be reachable only from the same task (localhost) or a private network, never from the internet. In the lab's Docker network it is container-to-container only.
Step-by-step implementation
Add the OpenTelemetry packages
In the API project: OpenTelemetry.Extensions.Hosting, OpenTelemetry.Exporter.OpenTelemetryProtocol, OpenTelemetry.Instrumentation.AspNetCore, OpenTelemetry.Instrumentation.Http (all 1.21.0 on 2026-11-18). Add OpenTelemetry.Instrumentation.AWS when the service calls AWS SDK clients.
Register tracing, with your own source name
Call AddOpenTelemetry().WithTracing(...), AddSource("Harbourline.Transfers") for your manual spans, the AspNetCore and HttpClient instrumentations, and the OTLP exporter. Set the service name and environment on the resource: that is what appears on the service map.
Filter noise
Exclude /health from tracing. Health checks every few seconds would dominate your trace volume and cost.
Add spans where the time goes
Around the database insert and the partner call. Use ActivityKind.Client, set standard attributes (db.system, peer.service), and mark failures with SetStatus(ActivityStatusCode.Error, ...).
Put the trace ID in the logs
The lab's formatter already writes traceId and spanId from Activity.Current. That is the bridge from a log line to a trace.
Run it locally first
docker compose up --build starts the API, a collector and Jaeger. Send traffic with scripts/load.sh. Learn to read traces before you pay for them.
Choose a sampling strategy
Start with a parent-based ratio sampler (5%) in the SDK. Add tail sampling in the collector to keep 100% of errors and slow traces ⚠ VERIFY that your collector build includes the tail-sampling processor.
Point the exporter at AWS
On ECS: an ADOT collector sidecar with an awsxray exporter and a task role with the X-Ray write actions, and Transaction Search enabled in the account. Verify spans in the CloudWatch Traces view and the X-Ray service map.
AWS Console instructions
Enable Transaction Search (required for OTLP to X-Ray): Console → CloudWatch → Settings → Transaction Search → Enable (also reachable from X-Ray traces) ⚠ VERIFY the current wording; the documentation links a "Getting started with Transaction Search" page. It makes spans flow into a CloudWatch Logs group (aws/spans) and sets an indexing percentage that controls cost.
View traces: Console → CloudWatch → Application Signals → Traces (or X-Ray traces → Traces). Filter by service name, status or an annotation, open a trace to see the timeline and the trace map. To find the trace for a log line, convert the W3C trace ID in the log to X-Ray format (lowercase hex, 1- + first 8 + - + remaining 24) and paste it into the trace search.
AWS CLI
# Enable Transaction Search: send spans to CloudWatch Logs (needs a log resource policy; follow the console wizard first time)
aws xray update-trace-segment-destination --destination CloudWatchLogs --profile learn-sandbox
aws xray get-trace-segment-destination --profile learn-sandbox
# Recent slow or failed traces for the service (X-Ray filter expressions)
aws xray get-trace-summaries --profile learn-sandbox \
--start-time "$(date -u -d '-15 minutes' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
--filter-expression 'service("harbourline-transfers-api") AND (responsetime > 1 OR error OR fault)' \
--query 'TraceSummaries[].[Id,Duration,HasError,HasFault]' --output table
# Fetch one full trace: the W3C id from a log line (4efaaf4d...) becomes 1-4efaaf4d-...
aws xray batch-get-traces --trace-ids 1-4efaaf4d-1e8720b39541901950019ee5 --profile learn-sandbox
# The service graph (who calls whom, error and latency per edge)
aws xray get-service-graph --profile learn-sandbox \
--start-time "$(date -u -d '-15 minutes' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" --query 'Services[].Name'The commands exist in AWS CLI v2; the exact behaviour with OTLP-sourced spans and Transaction Search (for example, which of your spans are indexed and therefore searchable by these calls) depends on your indexing settings ⚠ VERIFY. With Transaction Search you can also query the aws/spans log group with Logs Insights (Logs Insights queries).
Infrastructure as Code
The collector sidecar's configuration (ADOT or the upstream contrib collector; the exporter name awsxray is part of the AWS-supported distribution ⚠ VERIFY for your image version):
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_mib: 128
attributes/scrub:
actions:
- key: http.request.header.authorization
action: delete
- key: user.email
action: delete
batch: {}
exporters:
awsxray:
region: us-east-1
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, attributes/scrub, batch]
exporters: [awsxray]The task role needs only write access to X-Ray. A least-privilege policy for the task role:
using Amazon.CDK.AWS.IAM;
// taskRole is the ECS task role the collector sidecar runs under.
taskRole.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
Effect = Effect.ALLOW,
Actions =
[
"xray:PutTraceSegments",
"xray:PutTelemetryRecords",
"xray:GetSamplingRules",
"xray:GetSamplingTargets",
],
Resources = ["*"], // X-Ray write actions do not support resource-level restrictions
}));AWS publishes a managed policy for this, referred to in the OTLP documentation as AWSXrayWriteOnlyPolicy ⚠ VERIFY the exact managed-policy name in IAM before attaching; the explicit policy above shows what it must allow. For the sidecar container definition see Deploy to Fargate and add a second container with the collector image and --config pointing at the file (shipped via an environment variable or a config store ⚠ VERIFY image name and tag from the ADOT documentation).
Complete code
The registration (lab API)
labs/m11-observe-banking-api/src/Harbourline.Transfers.Api/Program.cs:
var otlpConfigured = !string.IsNullOrWhiteSpace(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]);
builder.Services.AddOpenTelemetry()
.ConfigureResource(r => r.AddService(serviceName, serviceVersion: "1.0.0")
.AddAttributes([new("deployment.environment.name", environmentName)]))
.WithTracing(t =>
{
t.AddSource(Telemetry.SourceName) // "Harbourline.Transfers": our manual spans
.AddAspNetCoreInstrumentation(o => o.Filter = ctx => !ctx.Request.Path.StartsWithSegments("/health"))
.AddHttpClientInstrumentation();
if (otlpConfigured) t.AddOtlpExporter(); // only when an endpoint is configured
});The manual spans
Transfers.cs: the database and the partner call, each as a span with a failure status:
public static class Telemetry
{
public const string SourceName = "Harbourline.Transfers";
public static readonly ActivitySource Source = new(SourceName);
}
public sealed class TransferStore
{
public async Task<string> SaveAsync(TransferRequest request, int extraDelayMs, CancellationToken ct)
{
using var span = Telemetry.Source.StartActivity("db.insert transfers", ActivityKind.Client);
span?.SetTag("db.system", "postgresql");
span?.SetTag("db.operation", "INSERT");
await Task.Delay(5 + extraDelayMs, ct); // the slow-db fault makes this bar long in the trace
var id = "PAY-" + Guid.CreateVersion7();
span?.SetTag("payment.id", id);
return id;
}
}
public sealed class SanctionsPartner(Func<double> random)
{
public async Task ScreenAsync(string creditorName, double failureRate, CancellationToken ct)
{
using var span = Telemetry.Source.StartActivity("partner.sanctions-screen", ActivityKind.Client);
span?.SetTag("peer.service", "sanctions-partner");
await Task.Delay(8, ct);
if (failureRate > 0 && random() < failureRate)
{
span?.SetStatus(ActivityStatusCode.Error, "partner unavailable"); // the bar turns red
throw new PartnerUnavailableException("Sanctions screening partner did not respond.");
}
}
}A production-style SDK configuration
Sampling and, when you call AWS SDK clients, instrumentation for them (this compiles against OpenTelemetry 1.21.0 packages, checked in Docker on 2026-11-18):
using OpenTelemetry;
using OpenTelemetry.Context.Propagation;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
var builder = WebApplication.CreateBuilder(args);
// W3C traceparent plus baggage. (An X-Ray propagator exists in a contrib package; see the dependency note above.)
Sdk.SetDefaultTextMapPropagator(new CompositeTextMapPropagator(
[
new TraceContextPropagator(),
new BaggagePropagator(),
]));
builder.Services.AddOpenTelemetry()
.ConfigureResource(r => r.AddService("harbourline-transfers-api", serviceVersion: "1.0.0"))
.WithTracing(t => t
.AddSource("Harbourline.Transfers")
.AddAspNetCoreInstrumentation(o => o.Filter = ctx => !ctx.Request.Path.StartsWithSegments("/health"))
.AddHttpClientInstrumentation()
.AddAWSInstrumentation() // spans for AWS SDK calls
.SetSampler(new ParentBasedSampler(new TraceIdRatioBasedSampler(0.05))) // keep 5%, honour the caller's decision
.AddOtlpExporter()); // endpoint from OTEL_EXPORTER_OTLP_ENDPOINT
var app = builder.Build();
app.MapGet("/", () => "ok");
app.Run();Converting between the two trace ID formats
Tracing/XRayTraceId.cs in the lab (unit-tested):
XRayTraceId.FromW3C("4efaaf4d1e8720b39541901950019ee5"); // "1-4efaaf4d-1e8720b39541901950019ee5"
XRayTraceId.ToW3C("1-4efaaf4d-1e8720b39541901950019ee5"); // "4efaaf4d1e8720b39541901950019ee5"
XRayTraceId.StartTime("1-58406520-a006649127e371903a2de979"); // the request start time encoded in the ID
XRayTraceId.BuildHeader("58406520a006649127e371903a2de979", "53995c3f42cd8ad8", sampled: true);
// "Root=1-58406520-a006649127e371903a2de979;Parent=53995c3f42cd8ad8;Sampled=1"Code explained line by line
AddSource("Harbourline.Transfers")tells the OTel SDK to listen to yourActivitySource. If the names do not match exactly, your manual spans are created and silently dropped; this is the most common "my spans are missing" bug.ConfigureResource(... AddService(...))setsservice.name, which is how the service appears on the service map. Without it every service isunknown_service.Filter = ctx => !ctx.Request.Path.StartsWithSegments("/health")removes health-check requests from tracing before they create spans.if (otlpConfigured) t.AddOtlpExporter();keeps local runs and tests free of exporter errors. The exporter reads its endpoint and protocol from the standardOTEL_EXPORTER_OTLP_*environment variables.StartActivity(..., ActivityKind.Client)marks the span as an outgoing call, which backends use to draw dependency edges. It returnsnullwhen no listener samples it, so every use isspan?..SetStatus(ActivityStatusCode.Error, ...)is what turns a span red in Jaeger and counts it as an error in X-Ray; throwing the exception alone does not always mark the span.ParentBasedSampler(new TraceIdRatioBasedSampler(0.05))means: if the caller already decided (thesampledflag intraceparent), follow it; otherwise keep 5% of new traces, chosen deterministically from the trace ID so every service in the path makes the same decision.CompositeTextMapPropagatorlists the header formats the app reads and writes.BaggagePropagatorcarries key/value baggage; leave it out unless you need it, because baggage values travel to every downstream service.
Security considerations
⚠ Learning shortcut — not for production
The lab sends traces over plain HTTP between containers on a private Docker network, with tls.insecure: true towards Jaeger, and traces every request. Production: the collector runs as a sidecar reachable only on localhost, exports to AWS over TLS with SigV4, samples (for example 5% plus all errors) and deletes sensitive attributes before export.
- Spans can carry personal data. HTTP instrumentation records URLs; if the path contains an account number (
/accounts/ACC-0000000001) it is now in your trace store. Use route templates (/accounts/{id}) as the span name, and do not record request bodies or SQL parameter values. EF Core instrumentation can record SQL text: check the option that controls it before enabling. - Baggage travels everywhere. Never put identifiers or secrets in baggage.
- Least privilege for tracing. X-Ray write permissions on the collector's role only;
xray:Get*read permissions for engineers' operations role. - Trace retention. Transaction Search stores spans in a log group with a retention setting you control; set it, as with any log group (lesson 2).
- Third-party propagation. Sending
traceparentto external partners tells them your trace ID. That is normally harmless, but for some partners you may need to strip the header; check the contract. - Supply chain. Instrumentation packages run in your process with full privileges. Pin versions, review advisories (the NuGet audit warning mentioned above is exactly this), and avoid pre-release packages in regulated services unless risk-accepted.
Banking use case
Harbourline's payments team traces POST /v1/transfers end to end: API Gateway, the Transfers API, the sanctions partner, the database and the queue consumer. They sample 5% of ordinary traffic and keep every error and every request over one second (tail sampling in the collector). When a partner's latency degrades, the trace shows the sanctions span growing from 80 ms to 4 s while every other span stays flat, and the team attaches the traces to the partner's incident ticket as evidence for the service-level review. Each log line carries traceId, so support can go from a customer's correlation ID to the log lines to the trace in three steps. Attributes are scrubbed in the collector: no account numbers, names or authorisation headers leave the task. All data is fictional.
Common mistakes
AddSourcename mismatch. Manual spans vanish. Fix: use a constant shared by theActivitySourceand the registration.- Tracing everything at 100%. The collector-less default is full capture. Fix: set a sampler, and watch the trace count.
- No error marking. A failed call that is caught and handled looks green. Fix:
SetStatus(Error)and record the exception. - High-cardinality span names (
GET /accounts/ACC-0000000001). Fix: route templates; put the ID in an attribute only if policy allows. - Tracing health checks. Fix: filter them out.
- Forgetting Transaction Search. The OTLP endpoint rejects or ignores spans. Fix: enable it first and check the documentation for the current prerequisite.
- Wrong .NET settings with ADOT auto-instrumentation. Missing
OTEL_TRACES_EXPORTER=noneandOTEL_AWS_SIG_V4_ENABLED=true. Fix: follow the .NET-specific instructions exactly. - Putting the exporter in the request path. Direct synchronous export slows requests. Fix: the SDK batches spans; export to a local collector.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| No spans in Jaeger | OTEL_EXPORTER_OTLP_ENDPOINT not set, collector down, or wrong protocol/port | docker compose logs collector; the lab sets http://collector:4318 and http/protobuf |
| Server spans appear but custom spans do not | AddSource name differs from ActivitySource name | Make both use Telemetry.SourceName |
| Spans appear in Jaeger but not in X-Ray | Transaction Search not enabled, task role lacks X-Ray write, or wrong endpoint/Region | Enable Transaction Search; check the role; confirm https://xray.us-east-1.amazonaws.com/v1/traces |
| Collector-less ADOT .NET exports nothing | Missing OTEL_TRACES_EXPORTER=none or OTEL_AWS_SIG_V4_ENABLED=true | Set both, as required for .NET |
| Trace is broken in two halves | Context not propagated (a custom HTTP client, a queue message without headers) | Propagate traceparent; for queues put it in message attributes (see Module 8) |
| Unexpectedly high trace cost | 100% sampling | Set a ratio sampler or collector sampling |
traceId in logs is all zeros or missing | No Activity is active (code outside a request or tracing off) | Ensure the ASP.NET Core instrumentation or an ActivityListener is registered |
Cost and cleanup
Local (free): the Docker Compose stack costs nothing. AWS (credits): the free tier lists 100,000 traces recorded and 1,000,000 retrieved or scanned per month; Transaction Search stores spans in CloudWatch Logs at log ingestion prices (USD 0.50 per GB for Standard class at the time of checking ⚠ VERIFY), so a lab with a few hundred requests costs fractions of a cent, and the real risk is leaving 100% sampling on under load. For the lab, use the free local path for all practice, and if you want the AWS view, run a few dozen requests with a 100% sampler, look, and tear down within the hour.
# Local
docker compose down --rmi local
# AWS: remove what you created
aws logs delete-log-group --log-group-name aws/spans --profile learn-sandbox # only if you created it for the lab; it holds spans
aws xray update-trace-segment-destination --destination XRay --profile learn-sandbox # switch Transaction Search off ⚠ VERIFY
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab --profile learn-sandboxCheck with your team before deleting aws/spans or switching Transaction Search off in a shared account.
Exercises
- 🟢 Guided: Run
docker compose up --build, send 20 requests withscripts/load.sh, open Jaeger athttp://localhost:16686, and find a normal trace. Note the three spans (server, partner, database) and their durations. Then send 10 withslow-dband 10 withpartner-downand describe what changed. - 🟡 Challenge: Add a span
queue.publish payment-event(kindProducer) after the database insert, tagged withmessaging.system = sqs. Send a request and confirm the new span appears under the server span in Jaeger. Then change the sampler toParentBasedSampler(new TraceIdRatioBasedSampler(0.5))and observe how many of 20 requests produce traces. - 🔴 Live problem: "Some transfers take 6 to 12 seconds on Monday mornings, only for customers whose payee is at one partner bank. p99 over the day looks fine and logs show nothing wrong." You have traces for 5% of requests. Describe how you would capture the problem traces reliably (sampling change, attributes, tail sampling), how you would find the pattern, and what you would give the partner as evidence.
Assignment
Instrument a service of your own with OpenTelemetry: the SDK registration, two custom spans with failure status, a health-check filter, a 5% parent-based sampler, a collector configuration that deletes two sensitive attributes, and a README with a screenshot description of a normal and a failing trace. If you use AWS, add the least-privilege X-Ray policy in IaC.
Solutions
A normal trace has a server span (POST /v1/transfers, roughly 20 to 40 ms), a partner.sanctions-screen child (about 8 ms) and a db.insert transfers child (about 5 ms). With slow-db the db.insert bar grows to about 800 ms while the partner bar is unchanged: the bottleneck is clearly the database. With partner-down the partner span is marked error (red), the database span never starts, and the server span ends with a 502. The point: the trace names the failing step directly, which neither the latency metric nor the error count can do.
using var publish = Telemetry.Source.StartActivity("queue.publish payment-event", ActivityKind.Producer);
publish?.SetTag("messaging.system", "sqs");
publish?.SetTag("messaging.destination.name", "hl-payments-events");Place it after store.SaveAsync inside the same request method so the current activity is the server span and the new span nests beneath it. At a 0.5 ratio about half of the 20 requests produce traces; because the sampler is trace-ID based, the same traces are sampled in every service of the path. The ParentBased wrapper means a downstream service follows an upstream sampled flag rather than re-deciding.
Add attributes that can explain the pattern: the payee's partner bank code (a small fixed vocabulary, not the IBAN) on the partner span, and the time of day. Switch to tail sampling in the collector so that any trace over 3 seconds or with an error is kept at 100% while normal traces stay at 5% ⚠ VERIFY tail-sampling availability in your collector build. Then query the kept traces grouped by partner bank code and look for the slow span. Cross-check with logs through traceId. Evidence for the partner: the trace timelines showing the partner span duration with timestamps, the count of affected requests, and the comparison to other banks. Retention: copy the evidence you need, because trace retention is limited.
Knowledge check
Answer: the trace ID, propagated in a header such as W3C traceparent (and X-Amzn-Trace-Id on AWS edge services). Each service's spans use the same trace ID and the caller's span as parent.
Answer: ActivitySource and Activity from System.Diagnostics. An Activity is a span; the OTel SDK listens to registered ActivitySource names.
Answer: the default is parentbased_always_on (100% of traces), which can raise ingestion volume and cost up to about 20 times compared with a 5% sample.
Answer: Transaction Search. The endpoint accepts OTLP over HTTP (protobuf) with SigV4.
Answer: read the traceId from the log line (it is a W3C trace ID), convert it to X-Ray format (1- + 8 hex, -, 24 hex) if needed, and search for it in the trace viewer.
Interview questions
- Explain trace, span and context propagation. A trace is one request's journey; spans are timed units of work forming a tree; propagation passes the trace ID and parent span ID between services in headers such as
traceparent. - Why OpenTelemetry instead of the X-Ray SDK for new .NET code? It is vendor-neutral, built on .NET's own
Activitytypes, and the X-Ray SDKs and daemon are in maintenance mode; X-Ray remains a backend via OTLP or the ADOT collector. - How do you control tracing cost? Sampling (head ratio in the SDK, tail sampling in the collector to keep errors and slow requests), filtering health checks, and limiting attributes.
- What can go wrong with traces in a bank? Personal data in attributes (URLs, SQL), unbounded retention, over-privileged tracing roles, and untested sampling that drops the one trace you need.
- Where does a collector add value? Central batching, sampling, scrubbing of attributes and credentials handling in one place, instead of in every service.
Certification practice
A developer needs to find which downstream call makes one API request slow across several services. Which should they use?
A. CloudWatch alarms on average latency. B. Distributed tracing with an X-Ray-compatible backend and OpenTelemetry instrumentation. C. VPC Flow Logs. D. CloudTrail data events.
Answer: B. Traces show the timed path of one request across services. A aggregates, C records network flows, D records AWS API activity.
Trace costs spike after a team sends OpenTelemetry traces straight to the X-Ray OTLP endpoint with default SDK settings. What is the most likely reason and fix?
A. The default sampler records 100% of traces; configure a ratio-based sampler (for example 5%). B. X-Ray bills per user; reduce users. C. Traces are billed per Availability Zone; use one. D. Compression is on by default; turn it off.
Answer: A. Collector-less export defaults to parentbased_always_on. The other options are invented.
A team wants spans scrubbed of authorisation headers and sampled centrally for many services, with only one component holding AWS tracing permissions. Which design fits?
A. Each service calls PutTraceSegments directly.
B. Each service exports OTLP to a local ADOT/OpenTelemetry collector sidecar that scrubs, samples and exports to X-Ray under its own role.
C. Store traces in S3 manually.
D. Disable tracing.
Answer: B. A collector centralises scrubbing, sampling and credentials. A spreads permissions and logic across every service, C is not a tracing pipeline, D removes the capability.
Completion checklist
- I can explain trace, span, propagation and sampling using .NET
Activityterms - I ran the Docker Compose stack and read a normal, a slow and a failed trace in Jaeger
- I added or read a custom span with failure status
- I can state the .NET-specific ADOT settings and the Transaction Search prerequisite
- I converted a W3C trace ID to X-Ray format by hand and with
XRayTraceId - I tore down any AWS resources and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- Traces answer "where did the time go?";
traceIdin every log line links logs to traces. - New .NET code uses OpenTelemetry over
ActivitySource; X-Ray is a backend reached through OTLP or the ADOT collector. - Collector-less export to X-Ray needs Transaction Search, SigV4, and for .NET two specific settings; its default 100% sampling can be costly.
- Put a collector in front for sampling, scrubbing and a single IAM boundary in production.
- Spans are data: scrub attributes, filter health checks and set retention.
Further reading
- Exporting collector-less telemetry using the ADOT SDK: endpoint, IAM, .NET settings and sampling note used in this lesson.
- Sending trace data to AWS X-Ray: the trace ID format and W3C conversion.
- Amazon CloudWatch pricing: X-Ray free tier and related prices.
- Application Signals and the Module 11 overview: the next lessons.
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Dashboards and Synthetics canaries: one screen for on-call and a customer's-eye availability check
Design an operational CloudWatch dashboard for a banking API as code, add a log-query widget, and write a Synthetics canary that probes the API from outside so you find out about a failure before your customers do.
Lab 11: observe a banking API (logs, metrics, alarms, dashboard and traces)
Run the Harbourline Transfers API with structured logs, EMF metrics and OpenTelemetry traces, inject a slow database and a partner outage, find each cause with Logs Insights and Jaeger, then deploy alarms, a composite page and a dashboard with CDK or Terraform and tear everything down.