CloudWatch Application Signals: service maps, SLOs and error budgets
Turn OpenTelemetry telemetry from an ASP.NET Core service into an automatic service map, standard Latency/Fault/Error metrics and service level objectives with burn-rate alarms, and know what it adds over raw X-Ray traces.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · ADOT .NET auto-instrumentation · CloudWatch agent · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain what Application Signals adds on top of raw traces: standard application metrics, an automatically discovered service map and service level objectives (SLOs).
- Describe the ECS sidecar pattern that gets a .NET service onto Application Signals, and say exactly which environment variables switch it on.
- Choose a good service level indicator (SLI), set an attainment goal, and compute an error budget by hand.
- Create a request-based and a period-based SLO, and add burn-rate alarms that page on how fast the budget is draining rather than on a single bad minute.
- Spot the .NET-specific sharp edge: the ADOT .NET SDK and the AWS SDK for .NET v4.
Prerequisites
- Distributed tracing with OpenTelemetry and X-Ray for .NET (ADOT), the instrumentation Application Signals builds on.
- Alarms and composite alarms and Dashboards and Synthetics canaries, which the SLO alarms and service views extend.
- Module 11 overview for the full lesson list.
- Deploy to Fargate, because the worked example is an ECS task.
- Tools: AWS CLI v2 with the
learn-sandboxSSO profile, .NET 10 SDK, Docker. The hands-on part is optional andcost: credits; the free path is described in Cost and cleanup.
Why this matters in banking
Harbourline's payments team has a promise: "99.9% of payment initiations succeed, and 95% complete in under 800 ms, measured over 30 days". Today that promise lives in a slide. On salary day the on-call engineer sees CPU at 40% and no alarm firing, yet customers are complaining. Nobody can answer, in one screen, "are we meeting our promise this month, and how much slack is left?".
An SLO makes the promise executable. It says what "good" means, over what window, and how much failure you can afford (the error budget). When the budget is draining fast, you stop shipping risky changes. When it is full, you can ship. That gives product, engineering and risk teams a shared, numeric language, which matters for the operational-resilience conversations that regulated firms have to hold.
Application Signals is the AWS feature that computes the numbers for you, from telemetry you are already emitting.
The idea in plain language
Three layers, from raw to useful:
- Telemetry. Your .NET process is instrumented with OpenTelemetry. It produces spans (one per HTTP request, database call, SDK call).
- Standard metrics. Application Signals turns those spans into three metrics for every service and operation it discovers:
Latency,Fault(server-side failures, HTTP 5xx or error spans) andError(client-side failures, HTTP 4xx). It also derives availability as one minus faults divided by total requests, so a 404 does not count against you but a 503 does. - Objectives. On top of those metrics you declare an SLO: "availability of
POST /paymentsat least 99.9% over a rolling 30 days". Application Signals tracks attainment, the remaining error budget and the burn rate.
It also draws a service map automatically: your API, the database it calls, the queue, the downstream partner, with health coloured on each node. You did not draw a diagram; the traces told it who calls whom.
Analogy: raw traces are the CCTV footage of every branch; Application Signals is the branch scorecard on the regional manager's desk. The scorecard says "Leeds branch missed its queue-time target on Tuesday". The footage tells you why. You need both, and the scorecard tells you which tape to pull.
What problem the AWS service solves
Before Application Signals, a team that wanted "latency p99 per operation, with an error budget" had to wire it together: choose metrics, create alarms, hand-build a dashboard, write budget maths. Application Signals packages that for supported platforms.
Honest limits to know before you commit:
- Platforms. AWS documents Application Signals as supported and tested on Amazon EKS, native Kubernetes, Amazon ECS and Amazon EC2, and says the EC2 instructions should work on any platform that supports the CloudWatch agent and ADOT. There is also a Lambda enablement path in the docs. Pick the one you deploy to.
- .NET versions. The documented runtime matrix lists .NET 8, 9 and 10 and .NET Framework 4.6.2 and higher, on Linux x64, Linux ARM64 and Windows Server 2022 x64.
- AWS SDK for .NET v4. The docs carry a note that the ADOT SDK for .NET does not support AWS SDK for .NET v4 and recommend v3 for full Application Signals support. This course uses v4 elsewhere, so you can lose automatic AWS-call spans (DynamoDB, SQS and so on) for those calls. Re-check this note before you build on it, because it is the kind of limitation that disappears in a later release. ⚠ VERIFY
- Cost. It is a metered feature. Application Signals is billed per "signal"; the pricing page lists $1.50 per million signals for the first 100 million, then $0.75 per million for the next 900 million, then $0.30 per million. ⚠ VERIFY current rates, and note that spans, logs and X-Ray traces are billed separately.
- Standard metric dimensions. Operation names can be up to 1,024 characters, but SLOs can only be set on operations whose name is 194 characters or fewer. High-cardinality route names (a URL with a customer id baked in) will hurt you here.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Application Insights Application Map | Application Signals service map (topology) | Built from OpenTelemetry traces; shows SLI health on nodes |
| Application Insights performance and failure blades | Services and Service detail pages | Operations and dependencies are first-class, with correlated traces one click away |
| Azure Monitor SLO tooling (Service Health, custom workbooks) | Application Signals SLOs | AWS has a native SLO object with error budget and burn rate |
| Azure Monitor autoinstrumentation agent | ADOT auto-instrumentation plus CloudWatch agent | You add an init container and a sidecar to the task; no code change |
| Smart detection alerts | SLO burn-rate alarms, anomaly detection alarms | Burn-rate alarms are explicit and tunable |
| Cloud role name | service.name resource attribute | Becomes the Service dimension on every standard metric |
Architecture
The security-boundary view: the sidecar talks to AWS using the task role (CloudWatchAgentServerPolicy plus your application's own permissions). The application container never holds telemetry credentials. Telemetry crosses to AWS over HTTPS. Anything you put in span attributes or log fields (an account number in a URL path, for instance) is stored by AWS, so scrub it first.
Step-by-step implementation
Enable Application Signals once per account and Region
Application Signals needs a one-time setup that creates a service-linked role (it reads the X-Ray service graph, runs Logs Insights queries and reads metrics for you). In the console this is the Enable Application Signals button; from the CLI use aws application-signals start-discovery. ⚠ VERIFY the exact command name for your CLI version.
Put the CloudWatch agent configuration in Parameter Store
The sidecar reads its configuration from the CW_CONFIG_CONTENT secret. The configuration only needs to switch on the two Application Signals receivers: traces and the logs-based metrics pipeline.
Add three containers to the task definition
An init container (copies the .NET auto-instrumentation files into a shared volume and exits), an ecs-cwagent sidecar, and your application container with the profiler environment variables set. The dependsOn on init with condition SUCCESS guarantees the files exist before .NET starts.
Name your service and environment
service.name becomes the Service dimension; deployment.environment becomes Environment. If you do not set the environment on ECS, the cluster name is used. Choose stable names: renaming a service creates a new service in the map and breaks SLO history.
Generate traffic, then create SLOs
An operation appears only after it has reported metrics, and AWS states you cannot create an SLO on a discovered operation until it has done so. Send a few minutes of traffic first (see the load script below), then create the SLOs.
Add burn-rate alarms and route them
Create burn-rate alarms on the SLO with an SNS action, so a fast-draining budget pages someone and a slow drain only opens a ticket.
AWS Console instructions
Console → CloudWatch → Application Signals (APM) → Services.
- First visit: choose Enable Application Signals and accept the service-linked role creation.
- Services page: confirm your service (
harbourline-payments-api) appears. If it does not after about five minutes of traffic, jump to the troubleshooting table. - Open the service: Operations lists each route with call volume, latency, availability, faults and errors. Select a point on a latency graph to see correlated traces.
- Application Signals → Service Level Objectives → Create SLO. Pick the service, the operation (
POST /payments), the SLI type (availability or latency), then the goal (for example 99.9%) and the interval (rolling 30 days or calendar month). - On the SLO details page choose Burn rate to add alarms with look-back windows and thresholds and attach an SNS topic.
- Application map shows the topology; click a dependency node to see its dependency metrics.
AWS CLI
# Enable discovery once per account and Region (creates the service-linked role).
# ⚠ VERIFY: command and flags for your CLI v2 version.
aws application-signals start-discovery --profile learn-sandbox --region us-east-1
# Store the CloudWatch agent config for the sidecar in Parameter Store.
cat > cwagent-appsignals.json <<'EOF'
{
"traces": { "traces_collected": { "application_signals": {} } },
"logs": { "metrics_collected": { "application_signals": {} } }
}
EOF
aws ssm put-parameter \
--name "/hl/m11/ecs-cwagent" \
--type String \
--value "file://cwagent-appsignals.json" \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m11-l8 Key=Environment,Value=learn Key=Owner,Value=you \
--profile learn-sandbox --region us-east-1
# List the services Application Signals has discovered.
# ⚠ VERIFY: the --start-time/--end-time values are epoch seconds in this command.
aws application-signals list-services \
--start-time $(date -u -d '-1 hour' +%s) --end-time $(date -u +%s) \
--profile learn-sandbox --region us-east-1
# List SLOs and look at one.
aws application-signals list-service-level-objectives --profile learn-sandbox --region us-east-1
aws application-signals get-service-level-objective --id harbourline-payments-availability \
--profile learn-sandbox --region us-east-1Infrastructure as Code
The CDK snippet below adds the three containers to a Fargate task definition and defines two SLOs with the L1 CfnServiceLevelObjective construct. Property names follow the CloudFormation AWS::ApplicationSignals::ServiceLevelObjective reference, which I checked; I have not compiled this exact snippet against Amazon.CDK.Lib 2.277.0. ⚠ VERIFY by running dotnet build.
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.ApplicationSignals;
using Amazon.CDK.AWS.ECS;
using Amazon.CDK.AWS.IAM;
using Constructs;
namespace Harbourline.Monitoring.Infra;
public sealed class AppSignalsStack : Stack
{
public AppSignalsStack(Construct scope, string id, IStackProps? props = null)
: base(scope, id, props)
{
// Amazon.CDK.Tags.Of(this) would resolve to the stack's Tags property (CS0176), so qualify it.
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m11-l8");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
var taskRole = new Role(this, "TaskRole", new RoleProps
{
AssumedBy = new ServicePrincipal("ecs-tasks.amazonaws.com"),
ManagedPolicies = new[]
{
// Lets the sidecar publish metrics and traces.
ManagedPolicy.FromAwsManagedPolicyName("CloudWatchAgentServerPolicy"),
},
});
var task = new FargateTaskDefinition(this, "Task", new FargateTaskDefinitionProps
{
Cpu = 512,
MemoryLimitMiB = 1024,
TaskRole = taskRole,
});
task.AddVolume(new Amazon.CDK.AWS.ECS.Volume { Name = "otel-auto-instrumentation" });
var agentConfig = Amazon.CDK.AWS.SSM.StringParameter.FromStringParameterName(
this, "AgentConfig", "/hl/m11/ecs-cwagent");
var init = task.AddContainer("init", new ContainerDefinitionOptions
{
// Pin an exact tag from the ADOT .NET auto-instrumentation repository on ECR Public.
// ⚠ VERIFY the current tag; this needs v1.1.0 or later per the AWS docs.
Image = ContainerImage.FromRegistry(
"public.ecr.aws/aws-observability/adot-autoinstrumentation-dotnet:v1.9.0"),
Essential = false,
Command = new[] { "cp", "-a", "autoinstrumentation/.", "/otel-auto-instrumentation" },
});
init.AddMountPoints(new MountPoint
{
SourceVolume = "otel-auto-instrumentation",
ContainerPath = "/otel-auto-instrumentation",
ReadOnly = false,
});
var agent = task.AddContainer("ecs-cwagent", new ContainerDefinitionOptions
{
// ⚠ VERIFY: pin a version tag instead of latest in real use.
Image = ContainerImage.FromRegistry("public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest"),
Essential = false,
Secrets = new Dictionary<string, Amazon.CDK.AWS.ECS.Secret>
{
["CW_CONFIG_CONTENT"] = Amazon.CDK.AWS.ECS.Secret.FromSsmParameter(agentConfig),
},
Logging = LogDriver.AwsLogs(new AwsLogDriverProps { StreamPrefix = "cwagent" }),
});
var app = task.AddContainer("app", new ContainerDefinitionOptions
{
Image = ContainerImage.FromRegistry("public.ecr.aws/docker/library/hello-world:latest"), // replace with your ECR image
Essential = true,
Environment = new Dictionary<string, string>
{
["OTEL_RESOURCE_ATTRIBUTES"] = "service.name=harbourline-payments-api,deployment.environment=learn",
["CORECLR_ENABLE_PROFILING"] = "1",
["CORECLR_PROFILER"] = "{918728DD-259F-4A6A-AC2B-B85E1B658318}",
["CORECLR_PROFILER_PATH"] = "/otel-auto-instrumentation/linux-x64/OpenTelemetry.AutoInstrumentation.Native.so",
["DOTNET_ADDITIONAL_DEPS"] = "/otel-auto-instrumentation/AdditionalDeps",
["DOTNET_SHARED_STORE"] = "/otel-auto-instrumentation/store",
["DOTNET_STARTUP_HOOKS"] = "/otel-auto-instrumentation/net/OpenTelemetry.AutoInstrumentation.StartupHook.dll",
["OTEL_DOTNET_AUTO_HOME"] = "/otel-auto-instrumentation",
["OTEL_DOTNET_AUTO_PLUGINS"] = "AWS.Distro.OpenTelemetry.AutoInstrumentation.Plugin, AWS.Distro.OpenTelemetry.AutoInstrumentation",
["OTEL_AWS_APPLICATION_SIGNALS_ENABLED"] = "true",
["OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT"] = "http://localhost:4316/v1/metrics",
["OTEL_EXPORTER_OTLP_PROTOCOL"] = "http/protobuf",
["OTEL_EXPORTER_OTLP_ENDPOINT"] = "http://localhost:4316",
["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"] = "http://localhost:4316/v1/traces",
["OTEL_METRICS_EXPORTER"] = "none",
["OTEL_LOGS_EXPORTER"] = "none",
["OTEL_TRACES_SAMPLER"] = "xray",
["OTEL_TRACES_SAMPLER_ARG"] = "endpoint=http://localhost:2000",
["OTEL_PROPAGATORS"] = "tracecontext,baggage,b3,xray",
},
Logging = LogDriver.AwsLogs(new AwsLogDriverProps { StreamPrefix = "app" }),
});
app.AddMountPoints(new MountPoint
{
SourceVolume = "otel-auto-instrumentation",
ContainerPath = "/otel-auto-instrumentation",
ReadOnly = false,
});
// The app must not start until the init container has copied the instrumentation.
app.AddContainerDependencies(new ContainerDependency
{
Container = init,
Condition = ContainerDependencyCondition.SUCCESS,
});
// SLO 1: request-based availability of POST /payments, 99.9% over a rolling 30 days.
new CfnServiceLevelObjective(this, "PaymentsAvailabilitySlo", new CfnServiceLevelObjectiveProps
{
Name = "hl-payments-availability",
Description = "99.9% of POST /payments requests succeed (no 5xx) over a rolling 30 days.",
RequestBasedSli = new CfnServiceLevelObjective.RequestBasedSliProperty
{
RequestBasedSliMetric = new CfnServiceLevelObjective.RequestBasedSliMetricProperty
{
KeyAttributes = new Dictionary<string, string>
{
["Type"] = "Service",
["Name"] = "harbourline-payments-api",
["Environment"] = "learn",
},
OperationName = "POST /payments",
MetricType = "AVAILABILITY",
},
},
Goal = new CfnServiceLevelObjective.GoalProperty
{
AttainmentGoal = 99.9,
WarningThreshold = 99.95,
Interval = new CfnServiceLevelObjective.IntervalProperty
{
RollingInterval = new CfnServiceLevelObjective.RollingIntervalProperty
{
DurationUnit = "DAY",
Duration = 30,
},
},
},
BurnRateConfigurations = new[]
{
new CfnServiceLevelObjective.BurnRateConfigurationProperty { LookBackWindowMinutes = 5 },
new CfnServiceLevelObjective.BurnRateConfigurationProperty { LookBackWindowMinutes = 60 },
},
});
// SLO 2: period-based latency, p95 under 800 ms in 99% of 1-minute periods.
new CfnServiceLevelObjective(this, "PaymentsLatencySlo", new CfnServiceLevelObjectiveProps
{
Name = "hl-payments-latency-p95",
Description = "p95 latency of POST /payments at or below 800 ms in 99% of 1-minute periods.",
Sli = new CfnServiceLevelObjective.SliProperty
{
SliMetric = new CfnServiceLevelObjective.SliMetricProperty
{
KeyAttributes = new Dictionary<string, string>
{
["Type"] = "Service",
["Name"] = "harbourline-payments-api",
["Environment"] = "learn",
},
OperationName = "POST /payments",
MetricType = "LATENCY",
Statistic = "p95",
PeriodSeconds = 60,
},
MetricThreshold = 800,
ComparisonOperator = "LessThanOrEqualTo",
},
Goal = new CfnServiceLevelObjective.GoalProperty
{
AttainmentGoal = 99.0,
WarningThreshold = 99.5,
Interval = new CfnServiceLevelObjective.IntervalProperty
{
CalendarInterval = new CfnServiceLevelObjective.CalendarIntervalProperty
{
StartTime = 1767225600, // 2026-01-01T00:00:00Z, ⚠ VERIFY epoch unit
DurationUnit = "MONTH",
Duration = 1,
},
},
},
});
}
}Complete code
Two pieces of application code are useful regardless of how you deploy. First, a small load generator so the operations appear in the service map, and second, a tiny middleware that stamps the business operation onto the current span without leaking personal data.
using System.Diagnostics;
namespace Harbourline.Payments.Api.Telemetry;
/// <summary>
/// Adds low-cardinality, PII-free attributes to the current request span so SLO
/// slices and service-map drill-downs have something meaningful to group by.
/// </summary>
public sealed class PaymentTelemetryMiddleware(RequestDelegate next)
{
public async Task InvokeAsync(HttpContext context)
{
var activity = Activity.Current;
// Route TEMPLATE, never the raw path: /payments/PAY-1234 would explode metric cardinality.
var template = (context.GetEndpoint() as RouteEndpoint)?.RoutePattern.RawText;
if (activity is not null && template is not null)
{
activity.SetTag("hl.route", template);
}
await next(context);
if (activity is not null)
{
activity.SetTag("hl.http.status_class", $"{context.Response.StatusCode / 100}xx");
// A bank-specific outcome tag. Values come from a closed set; no account numbers or names.
if (context.Items.TryGetValue("hl.payment.scheme", out var scheme) && scheme is string s)
{
activity.SetTag("hl.payment.scheme", s); // FPS, BACS, CHAPS, SEPA_CT, SEPA_INST
}
}
}
}// Sends a mix of good and deliberately bad requests so Fault and Error metrics both move.
// Run against YOUR sandbox endpoint only; never against anything you do not own.
using System.Net.Http.Json;
var baseUrl = args.Length > 0 ? args[0] : "http://localhost:8080";
using var http = new HttpClient { BaseAddress = new Uri(baseUrl), Timeout = TimeSpan.FromSeconds(10) };
var rng = new Random(20261117); // seeded, reproducible
for (var i = 0; i < 600; i++)
{
var bad = rng.Next(100) < 3; // about 3% bad requests
var body = new
{
debtorAccountId = "ACC-0000000001",
creditorName = "Example Supplier Ltd",
creditorIban = bad ? "GB00INVALID" : "GB82WEST12345698765432",
amount = 12.34m,
currency = "GBP",
scheme = "FPS"
};
using var req = new HttpRequestMessage(HttpMethod.Post, "/payments") { Content = JsonContent.Create(body) };
req.Headers.Add("Idempotency-Key", Guid.CreateVersion7().ToString());
try
{
var res = await http.SendAsync(req);
Console.WriteLine($"{i,4} {(int)res.StatusCode}");
}
catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException)
{
Console.WriteLine($"{i,4} FAIL {ex.GetType().Name}");
}
await Task.Delay(100);
}Code explained line by line
PaymentTelemetryMiddleware(RequestDelegate next)uses a primary constructor (C# 12) to capture the next middleware.Activity.Currentis the span OpenTelemetry started for this HTTP request. In .NET, an OpenTelemetry span is aSystem.Diagnostics.Activity, so you need no extra package to add tags.RouteEndpoint.RoutePattern.RawTextis the route template such as/payments/{id}. Using it avoids creating a new operation per payment id.hl.http.status_classrecords2xx,4xxor5xx, which makes it easy to slice traces by outcome class.- The scheme tag comes from a closed set. A rule worth adopting: span attributes are for low-cardinality, non-personal values; put detail in structured logs, which you can mask and expire.
- In the load generator,
new Random(20261117)keeps the traffic reproducible, in line with the course's seeded-data convention. Guid.CreateVersion7()creates a time-ordered idempotency key (UUID v7, available since .NET 9).- Only
HttpRequestExceptionandTaskCanceledExceptionare caught, so a genuine bug elsewhere still crashes the script.
Security considerations
⚠ Learning shortcut — not for production
The lab uses cloudwatch-agent:latest, a broad managed policy (CloudWatchAgentServerPolicy) and 100% of requests going to the agent. In production, pin image digests, scope the task role to what the agent needs, sample traffic, and keep the sidecar image in your own ECR repository so a public tag change cannot alter your task.
- Telemetry is data. Span names, attributes and URL query strings can contain account numbers, names or tokens. Strip them in the application or the collector before they leave the process. What may be recorded needs agreement with your data-protection officer under UK GDPR.
- Least privilege for dashboards. Application Signals shows request details to anyone who can read the X-Ray and CloudWatch data. Grant read access by role, not to everybody in the account.
- Alarm actions need permission. The SNS topic policy must allow
cloudwatch.amazonaws.comto publish, or the page never arrives and nothing tells you. - Supply chain. The init container runs code inside your task. Treat it like any other dependency: pin it, scan it and update it on a schedule.
Banking use case
Harbourline defines three customer-facing SLOs for the payments API:
| SLO | SLI | Goal | Window | Why |
|---|---|---|---|---|
| Payment initiation availability | Request-based availability of POST /payments | 99.9% | Rolling 30 days | Customers notice failures, regulators ask about them |
| Payment initiation latency | Period-based p95 latency at or below 800 ms, 1-minute periods | 99% of periods | Calendar month | Fast payments are the product |
| Balance read availability | Request-based availability of GET /accounts/{id}/balance | 99.95% | Rolling 30 days | Read traffic is 20 times the write traffic |
Error-budget arithmetic for the first SLO: 99.9% over 30 days leaves a 0.1% budget. For request-based SLOs that means one failed request in a thousand. For time, 0.1% of 43,200 minutes is about 43 minutes. If a bad deploy fails 2% of requests, the burn rate is 2% divided by 0.1%, which is 20, so a 30-day budget is gone in 1.5 days. That is why a burn-rate alarm with a short look-back window pages quickly, while a long window catches slow leaks.
A common two-alarm policy, which you should tune rather than copy:
- Fast burn: burn rate above 14 over a 5 to 60-minute window pages the on-call engineer.
- Slow burn: burn rate above 2 over a 6 to 24-hour window opens a ticket.
The numbers come from widely used SRE guidance and are a starting point. AWS's own page describes two methods for choosing thresholds, and you should read it before settling. ⚠ VERIFY
Common mistakes
- High-cardinality operation names. Record the route template, not the URL. Otherwise you get thousands of operations and cannot set SLOs on them.
- Counting 4xx as failures. Application Signals counts 4xx as
Error, notFault, so a validation error does not hurt availability. That is correct for a server-side SLO, but make sure your product owners agree that a blocked-by-sanctions 403 is not "downtime". - Setting the goal at 100%. It leaves a zero budget, so every blip is a breach and the SLO means nothing.
- Creating an SLO before the operation reports. The operation must have reported standard metrics first.
- Renaming
service.nameafter launch. New name, new service, empty history. - Forgetting the .NET v4 SDK caveat. If AWS calls are missing from the map, check whether you are on AWS SDK for .NET v4.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Service never appears | OTEL_AWS_APPLICATION_SIGNALS_ENABLED unset, or sidecar not running | Check task has three containers and the app log shows the auto-instrumentation banner |
init container exits non-zero | Wrong image or command path | Check the image page for the .NET init command (cp -a autoinstrumentation/. /otel-auto-instrumentation) |
| App starts but no spans | Profiler variables missing (CORECLR_PROFILER, CORECLR_PROFILER_PATH) | Compare to the full list in the CDK snippet; check the volume mount path matches |
| Dependencies missing from the map | AWS SDK v4 not supported by ADOT .NET | Use AWS SDK v3 for full support, or accept manual spans ⚠ VERIFY |
| Metrics appear but no traces | X-Ray / trace permissions or sampler | Check task role policy and OTEL_TRACES_SAMPLER=xray with the sampler endpoint |
| SLO creation says operation not found | No traffic yet | Send traffic for a few minutes, refresh |
| Alarm never notifies | SNS topic policy or confirmed subscription missing | Confirm the email subscription, check the topic policy, test with set-alarm-state |
Latency shows milliseconds you cannot match to the app | Server span vs. total, load balancer time excluded | Compare the span duration in the trace with the ALB TargetResponseTime |
Cost and cleanup
| Item | Label | Estimate |
|---|---|---|
| Fargate task (0.5 vCPU, 1 GB) for one hour | credits | A few cents ⚠ VERIFY current Fargate price |
| Application Signals signals for 600 requests | credits | Fractions of a cent; the first months include a free allowance on the pricing page |
| SLOs, burn-rate alarms | credits | Alarms are priced per alarm metric; delete after the lab |
| Log ingestion | credits | Tiny at lab volume; set 7-day retention |
Free alternative: run the API locally with OpenTelemetry and Jaeger as in the module's local lab, then read this lesson as a design exercise: write your SLO definitions as comments in a YAML file and compute the budgets by hand.
# Cleanup. SLOs created by the stack are deleted with the stack; SLOs created in the
# console are deleted in Console → Application Signals → Service Level Objectives.
aws cloudformation delete-stack --stack-name AppSignalsStack --profile learn-sandbox --region us-east-1
aws ssm delete-parameter --name /hl/m11/ecs-cwagent --profile learn-sandbox --region us-east-1
aws logs delete-log-group --log-group-name /ecs/ecs-cwagent --profile learn-sandbox --region us-east-1
# Check nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1 --query 'ResourceTagMappingList[].ResourceARN'Exercises
- 🟢 Guided: For the three SLOs in the banking table, calculate the total error budget in minutes (period-based) or failed requests per million (request-based). Then compute the burn rate for a 2% failure rate on the payments availability SLO.
- 🟡 Challenge: Write the CDK or Terraform for a balance-read availability SLO at 99.95% with a 5-minute and a 60-minute burn-rate configuration, and an SNS topic with an email subscription that the burn-rate alarms use.
- 🔴 Live problem: The payments SLO shows 99.4% attainment against a 99.9% goal, but nobody has been paged all month. List three reasons that can happen (for example, the alarm watches a different metric, burn is slow and below the alarm threshold, or the operation name changed) and describe how you would check each one.
Assignment
Write a one-page SLO specification for the Harbourline payments API: three SLOs, each with an SLI definition, goal, window, error budget in plain numbers, the alarms it drives and the runbook each alarm links to. Add it to your portfolio repository next to the IaC that creates the SLOs.
Solutions
Payments availability: goal 99.9% leaves 0.1%, so 1,000 failed requests per million. Payments latency: 99% of 1-minute periods over a 30-day month (43,200 periods) must be good, so 432 periods can breach, about 7.2 hours. Balance reads: 99.95% leaves 0.05%, so 500 failed requests per million.
Burn rate for a 2% failure rate on the payments SLO is 2% divided by the baseline error rate of 0.1%, which is 20. At burn rate 20 a 30-day budget lasts 30 divided by 20 = 1.5 days. The point of the exercise: burn rate is a multiple of "exactly consume the budget by the end of the window", so any value above 1 means you will miss the goal if it continues.
Reuse the CfnServiceLevelObjective pattern from the CDK tab with AttainmentGoal = 99.95, operation GET /accounts/{id}/balance, MetricType = "AVAILABILITY" and two BurnRateConfigurations with LookBackWindowMinutes 5 and 60. Create the topic with new Topic(...) and topic.AddSubscription(new EmailSubscription("oncall@example.com")). Then create burn-rate alarms in the console or with PutMetricAlarm pointing at the burn-rate metrics the SLO publishes, with the topic as the alarm action. Remember the topic policy must allow CloudWatch to publish. ⚠ VERIFY the burn-rate metric names in your Region before scripting the alarms.
Knowledge check
Answer: Error counts 4xx. It does not reduce availability, which is computed as one minus Fault divided by total requests; Fault counts 5xx and error spans. The distractor "4xx always lowers availability" is wrong because 4xx is considered a client problem.
Answer: 0.5%. Burn rate = observed error rate divided by (100% minus the goal). At exactly 0.5% errors the burn rate is 1 and you meet the goal exactly at the end of the interval.
Answer: The operation must first report standard metrics to Application Signals. Send traffic and wait for it to appear.
Answer: Request-based. A period-based SLO judges whole time periods, so a one-minute period containing a single slow request counts as a full bad minute, which exaggerates problems when traffic is sparse. A request-based SLO simply measures good requests divided by total requests, so each request weighs the same. Whichever you choose, you cannot convert one type into the other later, so decide up front.
Answer: It copies the .NET auto-instrumentation files into a shared volume and exits, and the app container depends on it with condition SUCCESS. It is not a runtime component.
Interview questions
- What is an error budget and how does it change team behaviour? The allowed amount of failure implied by the SLO goal. When it is healthy the team ships; when it is nearly spent the team slows risky changes and fixes reliability.
- Why use burn-rate alarms rather than a threshold alarm on error rate? Burn rate relates the current error rate to the goal and window, so the same rule makes sense for any SLO, and combining a short and a long window pages on fast burns without alerting on tiny blips.
- What would you check if the service map is missing a dependency? Whether the call was instrumented (HTTP client, SDK version), whether context propagates, and for .NET, the AWS SDK v4 note.
- Difference between Application Signals and X-Ray? X-Ray stores and shows individual traces; Application Signals adds standard metrics, topology and SLOs derived from the same telemetry.
Certification practice
An operations team runs a containerised API on ECS and must show a 99.9% monthly availability target, track how much failure remains, and be paged when the budget is burning quickly. Which approach meets this with the least custom work?
A. A CloudWatch dashboard of ALB 5xx counts with a manual spreadsheet. B. Application Signals with a request-based availability SLO and burn-rate alarms. C. A Lambda function that queries Logs Insights hourly and writes a custom metric. D. CloudTrail Insights on the ECS service.
Answer: B. Application Signals computes the SLI, attainment, error budget and burn rate. A needs manual work and has no budget concept. C works but is custom code to maintain. D detects unusual API activity, not application availability.
A .NET service on ECS Fargate must be enabled for Application Signals without code changes. Which combination is required?
A. A CloudWatch agent sidecar and an init container that provides the auto-instrumentation, with the profiler environment variables on the app container. B. An X-Ray daemon on the container instance only. C. An IAM user's access keys injected into the task. D. A Lambda layer attached to the task definition.
Answer: A. The sidecar receives OTLP data and the init container supplies instrumentation. B is the legacy X-Ray daemon and does not produce Application Signals metrics by itself. C is insecure and unnecessary, since the task role is used. D applies to Lambda, not ECS.
A developer sees traces for HTTP requests but AWS SDK calls to DynamoDB are missing from the .NET service's map. The service uses AWS SDK for .NET v4. What is the most likely cause?
A. DynamoDB does not support tracing. B. The ADOT .NET instrumentation may not support SDK v4, so automatic AWS SDK spans are absent. C. The application map only shows the first dependency. D. Traces expire after one hour.
Answer: B (⚠ VERIFY against the latest ADOT note). A is wrong, DynamoDB calls are traceable with a supported SDK. C and D are invented limits.
Completion checklist
- I can explain Latency, Fault and Error, and how availability is derived
- I can compute an error budget and a burn rate by hand
- I can list the three containers and key environment variables of the ECS pattern
- I know the SDK v4 caveat and where to re-check it
- I ran the cleanup steps and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- Application Signals converts OpenTelemetry telemetry into standard metrics, a service map and SLOs without custom dashboards.
- An SLO needs an SLI, a goal and a window; the error budget is the failure you can afford, and burn rate says how fast you are spending it.
- Keep span attributes low-cardinality and free of personal data.
- Page on fast burn and ticket on slow burn.
- Verify .NET and AWS SDK v4 support before you design around automatic spans.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Lab 11: observe a banking API (logs, metrics, alarms, dashboard and traces)
Run the Harbourline Transfers API with structured logs, EMF metrics and OpenTelemetry traces, inject a slow database and a partner outage, find each cause with Logs Insights and Jaeger, then deploy alarms, a composite page and a dashboard with CDK or Terraform and tear everything down.
AWS CloudTrail for operations: who changed what, and when
Use CloudTrail event history, a trail delivered to CloudWatch Logs and EventBridge rules to answer 'who changed this security group?' in minutes, alert on risky API calls, and keep audit trails separate from application logs.