API monitoring: access logs, metrics, traces and alarms that matter
Turn on structured API Gateway access logs, read the right CloudWatch metrics, trace a request from the edge into .NET with OpenTelemetry and X-Ray, and build a small set of alarms that page someone only when customers are affected.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.CloudWatch 4.x) · OpenTelemetry .NET 1.21 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain the four signals an API team needs: access logs, execution logs, metrics and traces, and what each one can and cannot answer.
- Turn on structured JSON access logs for API Gateway REST and HTTP APIs, choosing
$contextvariables that make incidents debuggable without logging personal data. - Read the API Gateway CloudWatch metrics (
Count,4XXError/4xx,5XXError/5xx,Latency,IntegrationLatency) and use the gap betweenLatencyandIntegrationLatencyto locate a slowdown. - Trace a request end to end with AWS X-Ray (REST APIs) and OpenTelemetry for .NET, and join traces to logs with a correlation ID.
- Create a small set of customer-impact alarms (error rate, p99 latency, throttling, authorizer failures) with sensible missing-data handling, from the CLI and from C#.
Prerequisites
- API Gateway REST vs HTTP vs WebSocket — stages and API types
- Lambda integrations and Container-backed APIs
- Calling partner APIs safely — timeouts and retries show up in your traces
- Module 11: Monitoring goes much deeper into CloudWatch, Logs Insights and Application Signals; this lesson is the API-specific slice
- Tools: .NET 10 SDK (or Docker), AWS CLI v2 with the
learn-sandboxSSO profile (only for the optional deploy)
Why this matters in banking
It is 08:55 on a Monday. Harbourline Bank's (fictional) mobile app shows "Something went wrong" on the accounts screen for about one customer in twenty. The API team's only dashboard shows average latency, which looks fine. Support escalates after 40 minutes, when the complaints queue is already long.
When the team finally looks, the answer takes five minutes: a new Lambda authorizer release fails for tokens issued by an older app version, so API Gateway returns 401 for those customers. Count was normal, average Latency was normal, and the 5XXError alarm stayed quiet because the failures were 4xx. Nobody was watching the one thing that mattered: the share of customers whose requests failed.
Regulators expect banks to know when an important business service is degraded and to evidence how quickly they reacted. Good API monitoring is how you produce that evidence, and how you avoid finding out from social media.
The idea in plain language
Monitoring an API means being able to answer four questions quickly:
- Is it broken right now, and for how many people? → Metrics and alarms: cheap numbers over time, like request count, error rate and latency percentiles.
- What exactly happened to this one request? → Access logs: one line per request, with status, latency, caller and IDs.
- Where inside the system did the time go? → Traces: a timeline of one request as it passes through API Gateway, the authorizer, your .NET code, DynamoDB and a partner call.
- What did API Gateway itself decide? → Execution logs (REST APIs only): API Gateway's internal steps, like "authorizer returned Deny" or "mapping template failed".
Analogy: Think of a bank branch. Metrics are the queue counter on the wall: how many people are waiting and how long the wait is. Access logs are the visitor book: who came in, when, which counter served them and how it ended. A trace is following one customer with a stopwatch from the door to the cashier to the back office and out again. Alarms are the branch manager's pager, set to go off only when the queue is long enough that customers will complain.
The big mindset shift from "monitoring servers" is that you alarm on symptoms customers feel (errors, slowness, rejection) rather than on causes (CPU, memory). Causes go on dashboards for diagnosis; symptoms page a human.
What problem the AWS services solve
| Need | AWS feature | Honest limitations |
|---|---|---|
| Per-request record of every call | API Gateway access logs to CloudWatch Logs (REST and HTTP APIs); REST APIs can also send access logs to Amazon Data Firehose | You choose the fields; anything not in the format is lost. Ingestion is billed per GB |
| API Gateway's internal decisions | Execution logs (REST only), level ERROR or INFO, optional full request/response data | "Log full requests/responses" writes bodies and headers, which can contain tokens and personal data; never enable it in production |
| Request counts, errors and latency | CloudWatch metrics in the AWS/ApiGateway namespace, sent automatically | One-minute granularity by default; per-method or per-route metrics need detailed metrics, which cost extra as custom metrics |
| End-to-end request timeline | AWS X-Ray active tracing on REST API stages; OpenTelemetry in your .NET code | HTTP APIs do not support X-Ray tracing. The X-Ray SDKs and daemon are in maintenance mode since 2026-02-25, so instrument new .NET code with OpenTelemetry (ADOT) and keep X-Ray as the trace backend |
| Being told when customers hurt | CloudWatch alarms and composite alarms, notifying Amazon SNS | Alarms evaluate per period; badly chosen thresholds and missing-data settings cause either noise or silence |
Two facts from the comparison table in the API Gateway guide shape every design decision here: execution logs, X-Ray tracing and Firehose access logs exist only for REST APIs. If deep edge observability matters (and in banking it usually does), that is one more reason the Account Information API uses a REST API, alongside WAF support covered in API security.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| APIM diagnostic logs to Log Analytics | API Gateway access logs to CloudWatch Logs | You write the log format yourself from $context variables |
| APIM gateway logs / trace policy | API Gateway execution logs (REST) | Enabled per stage; needs an account-level CloudWatch role |
| Application Insights requests and dependencies | OpenTelemetry in .NET → X-Ray (traces), CloudWatch (metrics, logs) | Same OpenTelemetry API you may already use with the Azure Monitor exporter; only the exporter changes |
| Kusto (KQL) queries | CloudWatch Logs Insights query language | Pipe-based, similar spirit; stats, filter, parse, pctile |
| Azure Monitor metric alerts + action groups | CloudWatch alarms + SNS topics | Alarms have three states (OK, ALARM, INSUFFICIENT_DATA) and an explicit missing-data policy |
| Application Map | X-Ray trace map / CloudWatch Application Signals | Application Signals adds SLOs and service maps; see Module 11 |
Activity and traceparent header | Same System.Diagnostics.Activity; X-Ray also uses X-Amzn-Trace-Id | Configure the AWS X-Ray propagator so both headers carry the same trace |
Architecture
Security-boundary view. Logs are data stores with their own access rules:
- What goes in: request IDs, trace IDs, status codes, latencies, the authenticated subject (
sub) or principal ID, client IP. Not access tokens, account numbers in query strings, request bodies or names. - Who can read them: an
ApiOperatorsrole withlogs:StartQueryandlogs:GetQueryResultson the API's log groups only. Developers do not get production log access by default. - How long they live: a retention period on every log group (for example 90 days hot in CloudWatch, then exported to S3 under Object Lock if your records policy needs longer). Infinite retention is never the default in this platform.
- Encryption: CloudWatch Logs encrypts at rest; associate a customer managed KMS key if your bank's policy requires key control.
Step-by-step implementation
Decide what "healthy" means before you collect anything
Write down a service-level objective (SLO) for the Accounts API, for example: "99.9% of GET /accounts requests in a rolling 30 days return a non-5xx response within 1,000 ms." Two things follow:
- You need error rate and latency percentile signals, not averages. An average hides the 1 in 20 customers who wait 8 seconds.
- 4xx errors are usually the caller's fault, but a sudden jump in 401/403 often means your auth change broke clients, as in the scenario above. Watch them as a rate.
Give API Gateway permission to write logs (REST APIs, once per Region)
REST API execution logs and access logs are written by API Gateway using an IAM role set at the account level (cloudwatchRoleArn). The role trusts apigateway.amazonaws.com and has the AWS managed policy AmazonAPIGatewayPushToCloudWatchLogs. HTTP APIs do not need this setting; they write access logs using a service-linked mechanism and a log group you specify.
Turn on structured access logs
Create a log group with retention, then set the stage's access log destination and a JSON format. JSON makes Logs Insights discover fields automatically. A good banking format:
{
"requestId": "$context.requestId",
"extendedRequestId": "$context.extendedRequestId",
"traceId": "$context.xrayTraceId",
"ip": "$context.identity.sourceIp",
"userAgent": "$context.identity.userAgent",
"requestTime": "$context.requestTime",
"httpMethod": "$context.httpMethod",
"resourcePath": "$context.resourcePath",
"status": "$context.status",
"protocol": "$context.protocol",
"responseLength": "$context.responseLength",
"responseLatency": "$context.responseLatency",
"integrationLatency": "$context.integrationLatency",
"integrationStatus": "$context.integration.status",
"integrationError": "$context.integration.error",
"authorizerError": "$context.authorizer.error",
"authorizerLatency": "$context.authorizer.latency",
"principalId": "$context.authorizer.principalId",
"wafResponse": "$context.wafResponseCode",
"errorType": "$context.error.responseType",
"errorMessage": "$context.error.message"
}Notes on the choices:
requestIdcan be overridden by clients;extendedRequestIdis always generated by API Gateway, so log both.resourcePathlogs the template (/accounts/{accountId}), not the real path with the account ID in it. That keeps identifiers out of logs and makes grouping easy.$context.pathwould include the real value.principalIdis whatever your authorizer returned. Make it the opaque customer ID (CUS-00012345) or tokensub, never an email address.errorTypetells you why API Gateway rejected something (THROTTLED,UNAUTHORIZED,ACCESS_DENIED,WAF_FILTERED,INTEGRATION_TIMEOUTand so on).
HTTP APIs support a smaller set of variables (no xrayTraceId, no WAF fields); keep the same JSON keys where they exist so one query works across both.
Turn on execution logs at ERROR level only (REST)
Set the stage's method settings to loggingLevel=ERROR and dataTraceEnabled=false. INFO is useful for a short debugging session in a non-production stage; full data tracing logs headers and bodies, including Authorization headers and customer data, and must stay off in production.
Enable tracing at the edge and in the code
- On the REST API stage, enable X-Ray tracing (
tracingEnabled=true). API Gateway records a segment and passes the trace context to the integration in theX-Amzn-Trace-Idheader. - In the .NET service, use OpenTelemetry with the AWS X-Ray ID generator and propagator, export spans over OTLP to an ADOT collector (a sidecar on ECS, or the ADOT Lambda layer), and let the collector send them to X-Ray.
- Write the trace ID into every log line so you can jump from a log entry to its trace and back.
Create a handful of alarms, then a composite
Start with four, all on the production stage:
| Alarm | Metric | Rule of thumb |
|---|---|---|
| Server error rate | 5XXError (REST) Average = fraction of requests | > 0.01 (1%) for 3 of 5 one-minute periods |
| Client error spike | 4XXError Average | > 0.10 for 3 of 5 periods, or anomaly detection |
| Slow for some customers | Latency p99 | > 2000 ms for 5 of 5 periods |
| Backend struggling | IntegrationLatency p99 | > 1500 ms for 5 of 5 periods |
Then a composite alarm "AccountsApiCustomerImpact" = ALARM(server-errors) OR ALARM(latency-p99), which is the only one that pages. The others notify a chat channel. Composite alarms reduce noise when one failure trips several metrics.
Practise the investigation path
An alarm fires. Your runbook is: open the dashboard → see which metric moved → run the saved Logs Insights query for that time window → pick a failing extendedRequestId → open its trace → find the slow or failing span → read the app log lines with the same trace ID. If any step in that chain is missing, fix the instrumentation before the next incident, not during it.
AWS Console instructions
- Account CloudWatch role (REST): Console → API Gateway → in the left navigation choose Settings → Logging → Edit → paste the ARN of the role that has
AmazonAPIGatewayPushToCloudWatchLogs. - Access logs (REST): API Gateway → APIs → your API → Stages →
prod→ Logs and tracing → Edit → turn on Custom access logging, enter the log group ARN and the JSON format. In the same panel set CloudWatch logs to Errors only, leave Data tracing off, and turn on X-Ray tracing. - Access logs (HTTP API): API Gateway → your HTTP API → Monitor → Logging → select the stage → Edit → enable access logging, log group ARN, format.
- Detailed metrics: for REST, the same Logs and tracing panel has Detailed metrics; for HTTP APIs it is under the stage's settings. Leave them off unless you need per-route numbers.
- Alarms: Console → CloudWatch → Alarms → Create alarm → Select metric → ApiGateway → By API name and stage (REST) or By Stage (HTTP) → pick the metric → choose the statistic (for example
p99) and period → set the threshold and Datapoints to alarm → under Additional configuration set Missing data treatment → choose the SNS topic. - Traces: Console → CloudWatch → X-Ray traces → Traces, filter by URL or status, open one to see the timeline.
Console layouts change often; if a label differs, the setting names (access logging, tracing, detailed metrics) stay the same.
AWS CLI
# Variables used below (fictional IDs: replace with yours)
PROFILE=learn-sandbox
REGION=us-east-1
API_ID=a1b2c3d4e5 # REST API ID
STAGE=prod
LG=/aws/apigateway/hl-m04-accounts-access
# 1. Create the access log group with 90-day retention and the mandatory tags
aws logs create-log-group --log-group-name "$LG" \
--tags Project=cea-lab,Lab=m04-l17,Environment=learn,Owner=you \
--profile $PROFILE --region $REGION
aws logs put-retention-policy --log-group-name "$LG" --retention-in-days 90 \
--profile $PROFILE --region $REGION
# 2. Point API Gateway (REST, account level) at a role that can write logs
aws apigateway update-account \
--patch-operations op=replace,path=/cloudwatchRoleArn,value=arn:aws:iam::111122223333:role/hl-apigw-cloudwatch \
--profile $PROFILE --region $REGION
# 3. Enable JSON access logs on the stage (format stored in a local file, minified)
LG_ARN=$(aws logs describe-log-groups --log-group-name-prefix "$LG" \
--query 'logGroups[0].arn' --output text --profile $PROFILE --region $REGION)
FORMAT=$(jq -c . access-log-format.json)
aws apigateway update-stage --rest-api-id $API_ID --stage-name $STAGE \
--patch-operations \
"op=replace,path=/accessLogSettings/destinationArn,value=${LG_ARN%:*}" \
"op=replace,path=/accessLogSettings/format,value=$FORMAT" \
--profile $PROFILE --region $REGION
# 4. Execution logs at ERROR, no data tracing, and X-Ray tracing on
aws apigateway update-stage --rest-api-id $API_ID --stage-name $STAGE \
--patch-operations \
'op=replace,path=/*/*/logging/loglevel,value=ERROR' \
'op=replace,path=/*/*/logging/dataTrace,value=false' \
'op=replace,path=/tracingEnabled,value=true' \
--profile $PROFILE --region $REGION
# 5. Server-error-rate alarm: Average of 5XXError is the fraction of requests that failed
aws cloudwatch put-metric-alarm --alarm-name hl-m04-accounts-5xx-rate \
--namespace AWS/ApiGateway --metric-name 5XXError \
--dimensions Name=ApiName,Value=hl-m04-accounts-api Name=Stage,Value=$STAGE \
--statistic Average --period 60 --evaluation-periods 5 --datapoints-to-alarm 3 \
--threshold 0.01 --comparison-operator GreaterThanThreshold \
--treat-missing-data notBreaching \
--alarm-actions arn:aws:sns:us-east-1:111122223333:hl-m04-oncall \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m04-l17 Key=Environment,Value=learn Key=Owner,Value=you \
--profile $PROFILE --region $REGION
# 6. p99 latency alarm (percentiles use --extended-statistic)
aws cloudwatch put-metric-alarm --alarm-name hl-m04-accounts-latency-p99 \
--namespace AWS/ApiGateway --metric-name Latency \
--dimensions Name=ApiName,Value=hl-m04-accounts-api Name=Stage,Value=$STAGE \
--extended-statistic p99 --period 60 --evaluation-periods 5 --datapoints-to-alarm 5 \
--threshold 2000 --comparison-operator GreaterThanThreshold \
--treat-missing-data notBreaching \
--alarm-actions arn:aws:sns:us-east-1:111122223333:hl-m04-oncall \
--profile $PROFILE --region $REGION
# 7. Composite alarm: page only when customers are affected
aws cloudwatch put-composite-alarm --alarm-name hl-m04-accounts-customer-impact \
--alarm-rule 'ALARM("hl-m04-accounts-5xx-rate") OR ALARM("hl-m04-accounts-latency-p99")' \
--alarm-actions arn:aws:sns:us-east-1:111122223333:hl-m04-pager \
--profile $PROFILE --region $REGION
# 8. Ask a question of the access logs with Logs Insights (last hour)
aws logs start-query --log-group-name "$LG" \
--start-time $(( $(date +%s) - 3600 )) --end-time $(date +%s) \
--query-string 'fields @timestamp, status, resourcePath, errorType | filter status >= 400 | stats count(*) as n by status, errorType, resourcePath | sort n desc | limit 20' \
--profile $PROFILE --region $REGION
# then: aws logs get-query-results --query-id <id from the previous output>Line notes: the access log destination ARN must not end in :* (that suffix appears in describe-log-groups output, hence ${LG_ARN%:*}). /*/*/ in a patch path means "all resources and methods" in the stage. For HTTP APIs the dimension names differ: ApiId and Stage, and metric names are lower case (4xx, 5xx).
Logs Insights queries worth saving
# Error rate per minute from access logs
fields @timestamp, status
| stats sum(status >= 500) / count(*) * 100 as pct5xx, sum(status >= 400 and status < 500) / count(*) * 100 as pct4xx by bin(1m)
# Where the time goes: edge overhead vs backend
fields responseLatency, integrationLatency
| filter ispresent(integrationLatency)
| stats pct(responseLatency, 99) as p99Total, pct(integrationLatency, 99) as p99Backend, pct(responseLatency - integrationLatency, 99) as p99Edge by bin(5m)
# Why are callers being rejected at the edge?
fields @timestamp, errorType, authorizerError, principalId
| filter errorType in ["UNAUTHORIZED", "ACCESS_DENIED", "AUTHORIZER_FAILURE", "THROTTLED", "WAF_FILTERED"]
| stats count(*) by errorType, authorizerErrorInfrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.APIGateway;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.CloudWatch.Actions;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.SNS;
using Constructs;
namespace Harbourline.Accounts.Infra;
public sealed class MonitoringStack : Stack
{
public MonitoringStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
var accessLogs = new LogGroup(this, "AccessLogs", new LogGroupProps
{
LogGroupName = "/aws/apigateway/hl-m04-accounts-access",
Retention = RetentionDays.THREE_MONTHS,
RemovalPolicy = RemovalPolicy.DESTROY // lab only: production keeps logs
});
var api = new RestApi(this, "AccountsApi", new RestApiProps
{
RestApiName = "hl-m04-accounts-api",
CloudWatchRole = true, // creates and sets the account-level logging role
DeployOptions = new StageOptions
{
StageName = "prod",
TracingEnabled = true,
LoggingLevel = MethodLoggingLevel.ERROR,
DataTraceEnabled = false,
MetricsEnabled = false, // detailed per-method metrics off by default
AccessLogDestination = new LogGroupLogDestination(accessLogs),
AccessLogFormat = AccessLogFormat.Custom(System.Text.Json.JsonSerializer.Serialize(new
{
requestId = "$context.requestId",
extendedRequestId = "$context.extendedRequestId",
traceId = "$context.xrayTraceId",
ip = "$context.identity.sourceIp",
httpMethod = "$context.httpMethod",
resourcePath = "$context.resourcePath",
status = "$context.status",
responseLatency = "$context.responseLatency",
integrationLatency = "$context.integrationLatency",
principalId = "$context.authorizer.principalId",
errorType = "$context.error.responseType"
}))
}
});
api.Root.AddResource("health").AddMethod("GET", new MockIntegration(new IntegrationOptions
{
RequestTemplates = new Dictionary<string, string> { ["application/json"] = "{\"statusCode\": 200}" },
IntegrationResponses = [new IntegrationResponse { StatusCode = "200" }]
}), new MethodOptions { MethodResponses = [new MethodResponse { StatusCode = "200" }] });
var oncall = new Topic(this, "OnCall", new TopicProps { TopicName = "hl-m04-oncall" });
var errorRate = new Alarm(this, "ServerErrorRate", new AlarmProps
{
AlarmName = "hl-m04-accounts-5xx-rate",
Metric = api.MetricServerError(new MetricOptions { Statistic = "Average", Period = Duration.Minutes(1) }),
Threshold = 0.01,
EvaluationPeriods = 5,
DatapointsToAlarm = 3,
ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING
});
var latencyP99 = new Alarm(this, "LatencyP99", new AlarmProps
{
AlarmName = "hl-m04-accounts-latency-p99",
Metric = api.MetricLatency(new MetricOptions { Statistic = "p99", Period = Duration.Minutes(1) }),
Threshold = 2000,
EvaluationPeriods = 5,
DatapointsToAlarm = 5,
ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING
});
var impact = new CompositeAlarm(this, "CustomerImpact", new CompositeAlarmProps
{
CompositeAlarmName = "hl-m04-accounts-customer-impact",
AlarmRule = AlarmRule.AnyOf(errorRate, latencyP99)
});
impact.AddAlarmAction(new SnsAction(oncall));
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m04-l17");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "you");
}
}Complete code
The service side: an ASP.NET Core minimal API that writes JSON logs with trace and correlation IDs, emits OpenTelemetry traces in X-Ray-compatible form, and records one business metric. It runs locally with no AWS account; traces go to any OTLP endpoint (an ADOT collector in AWS, or Jaeger in Docker on your laptop).
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.21.0" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.21.0" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.21.0" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.21.0" />
<PackageReference Include="OpenTelemetry.Extensions.AWS" Version="1.21.0" />
</ItemGroup>
</Project>using System.Diagnostics;
using System.Diagnostics.Metrics;
using OpenTelemetry;
using OpenTelemetry.Extensions.AWS.Trace;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
var builder = WebApplication.CreateBuilder(args);
// 1. Structured JSON logs that include the current trace and span IDs.
builder.Logging.ClearProviders();
builder.Logging.AddJsonConsole(o =>
{
o.IncludeScopes = true;
o.UseUtcTimestamp = true;
o.TimestampFormat = "yyyy-MM-ddTHH:mm:ss.fffZ";
});
builder.Logging.Configure(o => o.ActivityTrackingOptions =
ActivityTrackingOptions.TraceId | ActivityTrackingOptions.SpanId);
// 2. Propagate X-Ray's header (X-Amzn-Trace-Id) as well as W3C traceparent.
Sdk.SetDefaultTextMapPropagator(new AWSXRayPropagator());
// 3. Tracing and metrics, exported over OTLP (endpoint from OTEL_EXPORTER_OTLP_ENDPOINT).
builder.Services.AddOpenTelemetry()
.ConfigureResource(r => r.AddService("hl-accounts-api", serviceVersion: "1.4.0"))
.WithTracing(t => t
.AddXRayTraceId() // X-Ray-compatible trace IDs
.AddAspNetCoreInstrumentation(o => o.Filter = ctx => ctx.Request.Path != "/health")
.AddHttpClientInstrumentation()
.AddSource(AccountsTelemetry.SourceName)
.AddOtlpExporter())
.WithMetrics(m => m
.AddAspNetCoreInstrumentation()
.AddMeter(AccountsTelemetry.MeterName)
.AddOtlpExporter());
builder.Services.AddSingleton<AccountsTelemetry>();
builder.Services.AddSingleton<IAccountStore, InMemoryAccountStore>();
builder.Services.AddProblemDetails();
var app = builder.Build();
// 4. Correlation ID: reuse the caller's, otherwise API Gateway's request ID, otherwise generate one.
app.Use(async (ctx, next) =>
{
var correlationId = ctx.Request.Headers["X-Correlation-Id"].FirstOrDefault()
?? ctx.Request.Headers["X-Request-Id"].FirstOrDefault()
?? Guid.CreateVersion7().ToString();
ctx.Response.Headers["X-Correlation-Id"] = correlationId;
Activity.Current?.SetTag("correlation.id", correlationId);
var logger = ctx.RequestServices.GetRequiredService<ILoggerFactory>().CreateLogger("Request");
using (logger.BeginScope(new Dictionary<string, object> { ["correlationId"] = correlationId }))
{
await next();
}
});
app.MapGet("/health", () => Results.Ok(new { status = "ok" }));
app.MapGet("/accounts", async (HttpContext ctx, IAccountStore store, AccountsTelemetry telemetry, ILogger<Program> log) =>
{
// In the real API the customer ID comes from the validated token's "sub" claim.
var customerId = ctx.Request.Headers["X-Customer-Id"].FirstOrDefault();
if (string.IsNullOrWhiteSpace(customerId))
{
telemetry.Rejected("missing_customer");
return Results.Problem(statusCode: 401, title: "Unauthenticated");
}
using var span = telemetry.Source.StartActivity("LoadAccounts");
span?.SetTag("customer.id.hash", AccountsTelemetry.Hash(customerId)); // never the raw ID in traces
var sw = Stopwatch.StartNew();
var accounts = await store.GetForCustomerAsync(customerId, ctx.RequestAborted);
telemetry.RecordLookup(sw.Elapsed, accounts.Count);
log.LogInformation("Returned {AccountCount} accounts in {ElapsedMs} ms", accounts.Count, sw.ElapsedMilliseconds);
return Results.Ok(new { items = accounts });
});
app.Run();
public sealed class AccountsTelemetry : IDisposable
{
public const string SourceName = "Harbourline.Accounts";
public const string MeterName = "Harbourline.Accounts";
private readonly Meter _meter = new(MeterName, "1.0.0");
private readonly Histogram<double> _lookupMs;
private readonly Counter<long> _rejected;
public ActivitySource Source { get; } = new(SourceName);
public AccountsTelemetry()
{
_lookupMs = _meter.CreateHistogram<double>("accounts.lookup.duration", unit: "ms",
description: "Time to load a customer's accounts from the store");
_rejected = _meter.CreateCounter<long>("accounts.requests.rejected",
description: "Requests rejected by the API before reaching the store");
}
public void RecordLookup(TimeSpan elapsed, int count) =>
_lookupMs.Record(elapsed.TotalMilliseconds, new KeyValuePair<string, object?>("result.empty", count == 0));
public void Rejected(string reason) =>
_rejected.Add(1, new KeyValuePair<string, object?>("reason", reason));
public static string Hash(string value) =>
Convert.ToHexString(System.Security.Cryptography.SHA256.HashData(System.Text.Encoding.UTF8.GetBytes(value)))[..16];
public void Dispose()
{
_meter.Dispose();
Source.Dispose();
}
}
public sealed record AccountSummary(string AccountId, string Currency, string Product, decimal Balance);
public interface IAccountStore
{
Task<IReadOnlyList<AccountSummary>> GetForCustomerAsync(string customerId, CancellationToken ct);
}
public sealed class InMemoryAccountStore : IAccountStore
{
private static readonly Dictionary<string, AccountSummary[]> Data = new()
{
["CUS-00012345"] = [new("ACC-0000123456", "GBP", "current", 1520.40m), new("ACC-0000123457", "GBP", "savings", 8000.00m)],
["CUS-00067890"] = [new("ACC-0000678901", "EUR", "current", 310.15m)]
};
public async Task<IReadOnlyList<AccountSummary>> GetForCustomerAsync(string customerId, CancellationToken ct)
{
await Task.Delay(Random.Shared.Next(5, 40), ct); // simulate a store call
return Data.TryGetValue(customerId, out var a) ? a : [];
}
}Run it locally with Jaeger as a free trace viewer:
# Jaeger all-in-one accepts OTLP on 4317 (gRPC) and shows a UI on 16686
docker run --rm -d --name jaeger -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:1.62.0
cd src/Harbourline.Accounts.Api
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 dotnet run --urls http://localhost:5080
# In another terminal: one good call, one rejected call
curl -s -H "X-Customer-Id: CUS-00012345" -H "X-Correlation-Id: demo-001" http://localhost:5080/accounts
curl -s -i http://localhost:5080/accounts
# Open http://localhost:16686 and search for service hl-accounts-apiThe Jaeger image tag is pinned for reproducibility; any recent jaegertracing/all-in-one tag works ⚠ VERIFY the tag still exists on Docker Hub before you run it.
Code explained line by line
AddJsonConsolewrites one JSON object per log line to stdout. On ECS with theawslogsdriver and on Lambda, stdout goes straight into CloudWatch Logs, and JSON fields become queryable in Logs Insights withoutparse.ActivityTrackingOptions.TraceId | SpanIdadds the current OpenTelemetry trace and span IDs to every log scope. This is the join key between a log line and a trace.Sdk.SetDefaultTextMapPropagator(new AWSXRayPropagator())reads and writes theX-Amzn-Trace-Idheader. When API Gateway has tracing on, it passes this header to your integration, so your spans become children of the API Gateway segment instead of a separate trace..AddXRayTraceId()makes OpenTelemetry generate trace IDs in the X-Ray format (an epoch timestamp prefix plus random bits), which X-Ray needs to accept spans that started in your code.AddAspNetCoreInstrumentation(o => o.Filter = ...)creates a server span per request and skips/health, which load balancers call constantly and would otherwise drown real traffic in traces (and cost).AddHttpClientInstrumentation()creates client spans for outbound calls, so a slow partner call shows up as a long child span..AddOtlpExporter()sends spans and metrics over OTLP. The endpoint comes fromOTEL_EXPORTER_OTLP_ENDPOINT: an ADOT collector sidecar in AWS, Jaeger locally. The app never needs AWS credentials for tracing; the collector does.- The correlation middleware reuses a caller's
X-Correlation-Idif present, falls back toX-Request-Id, and otherwise creates a time-ordered UUID v7. Configure API Gateway to pass$context.requestIdasX-Request-Idto the integration so edge and app logs share an ID. logger.BeginScope(...)attachescorrelationIdto every log line written during the request, including logs from deeper services.customer.id.hashputs a hash of the customer ID in the trace. Traces are viewed by more people than the database is; treat span attributes like log fields and keep personal data out.Histogram<double>with unitmsgives you percentiles of the store lookup in your metrics backend;Counter<long>counts rejections by reason. Keep tag values low-cardinality (a reason code, a boolean), never a customer or account ID, or your metric bill explodes.Results.Problem(...)returns RFC 9457 ProblemDetails, consistent with REST API design.
Security considerations
⚠ Learning shortcut — not for production
The local demo reads the customer ID from an X-Customer-Id header so you can test without Cognito. In production the customer ID comes only from the validated token's sub claim (see Lambda authorizers and Cognito user pools); a header like this would let anyone read anyone's accounts.
⚠ Learning shortcut — not for production
The CDK stack sets RemovalPolicy.DESTROY on the log group so teardown is clean. Production log groups are retained on stack deletion and, where your records policy requires it, exported to an S3 bucket with Object Lock (see S3 protection).
- Logs are personal data stores. Under UK GDPR, IP addresses and customer identifiers are personal data. Minimise what you log, set retention, restrict who can query, and document the purpose. Your data-protection officer decides the retention period, not the developer.
- Never log tokens. Do not include
$context.identity.*headers that carry credentials, and never enable execution-log data tracing in production. If a token does reach logs, treat it as a credential leak: revoke and rotate. - CloudWatch Logs data protection policies can detect and mask some sensitive data types in log groups (for example, credit card numbers) as a safety net; they are not a substitute for not logging the data ⚠ VERIFY which managed data identifiers fit your fields.
- Least privilege for observability. The ADOT collector role needs
xray:PutTraceSegmentsandxray:PutTelemetryRecords(or the AWS managed policy for the X-Ray daemon write access), pluscloudwatch:PutMetricDataif it exports metrics; nothing else. - Integrity for audit. Access logs are operational evidence, not your audit trail. Who changed the API configuration is answered by CloudTrail, covered in Module 11.
Banking use case
Harbourline's Accounts API is classified as supporting an important business service ("customers can view balances"). The operations standard says:
- SLO: 99.9% of balance requests succeed within 1 second over 30 days.
- Paging alarm: the composite customer-impact alarm. It pages the on-call engineer through SNS → the incident tool.
- Non-paging alarms: 4xx spike, authorizer latency, integration p99, throttle count. They post to the team's chat channel during working hours.
- Dashboard: request count, 4xx and 5xx rate, p50/p90/p99 latency, edge overhead (
Latency - IntegrationLatency), WAF blocks, and the business histogramaccounts.lookup.duration. - Evidence: every page creates an incident record with the alarm history, the saved Logs Insights query results and the trace IDs inspected. That record is what the bank shows if asked how quickly it detected and responded to the disruption.
When the Monday incident from the opening happens again, the 4xx-spike alarm fires within 3 minutes, the "Why are callers being rejected?" query shows UNAUTHORIZED with authorizer error text pointing at the token's issuer claim, and the team rolls back the authorizer before support even hears about it.
Common mistakes
- Alarming on averages. Average latency stays flat while the slowest 1% suffer. Use p90/p99 for latency and averages only for 0/1 metrics like
5XXError, where the average is the error rate. - Using
Sumof5XXErrorwith a fixed threshold. Ten errors per minute is a disaster at 50 requests and noise at 50,000. Alarm on the rate. - Ignoring 4xx. Auth and validation regressions show up as 4xx. Watch for sudden changes, not absolute levels.
treat-missing-dataleft asmissing. A quiet API at night produces no datapoints; alarms flip toINSUFFICIENT_DATAand people learn to ignore that state. For error-rate alarms,notBreachingis usually right; for "the API must receive traffic" heartbeat alarms, usebreaching.- Logging the real path.
$context.pathand full URLs put account IDs in logs. LogresourcePath. - Full request/response execution logging in production. It writes tokens and customer data to logs.
- High-cardinality metric dimensions. One custom metric per customer ID creates millions of metrics and a large bill.
- Instrumenting new code with the X-Ray SDK. It is in maintenance mode; use OpenTelemetry and keep X-Ray as the backend.
- No retention. Default log group retention is "never expire", so cost grows forever.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
CloudWatch Logs role ARN must be set in account settings to enable logging | REST API account-level role not set | aws apigateway update-account with a role trusted by apigateway.amazonaws.com |
| Access log group stays empty | Destination ARN ends in :*, wrong Region, or no requests since change | Use the ARN without :*; send a request; check the stage, not a different stage |
| Execution log group missing | Logging level OFF, or you are using an HTTP API | Set ERROR; HTTP APIs have no execution logs, rely on access logs |
| App spans appear as a separate trace from API Gateway | Propagator not set, or trace ID format not X-Ray | AWSXRayPropagator and .AddXRayTraceId(); confirm the X-Amzn-Trace-Id header reaches the app |
| No traces in X-Ray at all | Collector role lacks X-Ray permissions, or OTLP endpoint wrong | Check collector logs; grant xray:PutTraceSegments; verify OTEL_EXPORTER_OTLP_ENDPOINT |
Alarm stuck in INSUFFICIENT_DATA | No traffic, wrong dimension names or values | Compare dimensions with aws cloudwatch list-metrics --namespace AWS/ApiGateway; set a missing-data policy |
Latency high but IntegrationLatency low | Time spent at the edge: authorizer, WAF, mapping templates | Log authorizerLatency and waf.latency; enable authorizer caching |
| HTTP API alarm never fires | Using REST metric names | HTTP APIs use 4xx, 5xx and dimension ApiId |
Cost and cleanup
This lesson is free when run locally. The .NET service, Jaeger and the Logs Insights syntax all work without an AWS account.
If you deploy the optional stack, costs are small but not zero (us-east-1, check the CloudWatch and X-Ray pricing pages ⚠ VERIFY current rates):
- CloudWatch Logs ingestion is charged per GB (around USD 0.50/GB for standard logs) and storage per GB-month; a lab session produces kilobytes.
- Standard-resolution metric alarms cost about USD 0.10 per alarm per month, prorated; composite alarms are priced separately. Four alarms for one day cost cents.
- X-Ray includes a monthly free allowance of recorded traces (100,000) and then charges per million.
- API Gateway's built-in metrics are free; detailed metrics are charged as custom metrics.
- Logs Insights queries are charged per GB scanned.
Teardown:
aws cloudwatch delete-alarms --alarm-names hl-m04-accounts-customer-impact --profile learn-sandbox
aws cloudwatch delete-alarms --alarm-names hl-m04-accounts-5xx-rate hl-m04-accounts-latency-p99 --profile learn-sandbox
aws logs delete-log-group --log-group-name /aws/apigateway/hl-m04-accounts-access --profile learn-sandbox
# If you used CDK: cdk destroy | Terraform: terraform destroy
docker rm -f jaeger
# Confirm nothing tagged for this lab remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--query 'ResourceTagMappingList[].ResourceARN' --profile learn-sandboxDelete the composite alarm before the alarms it references, or the delete call fails.
Exercises
- 🟢 Guided: Run the local API with Jaeger. Make 20 good and 5 bad calls. In Jaeger, find a trace for a good call and identify the
LoadAccountsspan. In the console output, find a log line and confirm itsTraceIdmatches the trace. - 🟡 Challenge: Write a Logs Insights query that returns, per 5 minutes, the p99 of
responseLatency - integrationLatency(edge overhead) only for requests where an authorizer ran. Then design an alarm on authorizer latency using theAuthorizerLatencyfield from access logs via a metric filter. - 🔴 Live problem: Ticket from the service desk: "Since 14:10, some business customers say the statements page spins forever, but the dashboard is green." The API is a REST API with a VPC link to Fargate.
5XXErroraverage is 0.002, p50 latency 180 ms, p99 latency 28,900 ms. Write down your investigation steps, the queries you would run and the two most likely causes.
Assignment
Add an observability.md to your portfolio repository for the Account Information API containing: the SLO; the access log format with a one-line justification per field; the list of alarms with thresholds, datapoints-to-alarm and missing-data policy; and your three saved Logs Insights queries. Include a short "what we deliberately do not log" section.
Solutions
You should see one trace per request, with a server span GET /accounts and a child LoadAccounts. The rejected calls have only the server span and status 401. Each console JSON log entry includes a Scopes array containing TraceId, SpanId and correlationId; the TraceId equals the trace ID in Jaeger. That match is why you configured ActivityTrackingOptions: in production, the same ID lets you jump from a CloudWatch log line to the X-Ray trace.
fields responseLatency, integrationLatency, authorizerLatency
| filter ispresent(authorizerLatency) and authorizerLatency > 0
| stats pct(responseLatency - integrationLatency, 99) as p99Edge, pct(authorizerLatency, 99) as p99Auth by bin(5m)For an alarm, create a metric filter on the access log group with pattern { $.authorizerLatency > 0 } and metric value $.authorizerLatency, namespace Harbourline/Api, then alarm on its p99. Why: API Gateway does not publish authorizer latency as a standard REST metric, but the access log has it, and metric filters turn log fields into metrics cheaply. Keep the metric dimensionless to avoid cardinality costs.
A p99 of 28,900 ms with a near-zero error rate is the signature of requests that almost hit the REST API's default 29-second integration timeout. Steps: (1) Logs Insights on access logs filtered to resourcePath = "/statements" with integrationLatency > 10000, grouped by bin(1m), to confirm it is one route; (2) check whether some still time out as INTEGRATION_TIMEOUT (504) via errorType; (3) open traces for slow request IDs and look for the long span; (4) check the Fargate service's own metrics and the database. Most likely causes: a slow database query for customers with many transactions (missing index or no pagination limit), or a downstream partner/core banking call without a client timeout. The dashboard was green because the alarms watched averages and 5xx rate; add a p99 alarm and a per-route slow-request metric filter.
Knowledge check
Answer: About 1,500 ms is spent inside API Gateway before or after the integration: typically a Lambda authorizer without caching, WAF inspection, or request/response mapping. IntegrationLatency measures only the backend call; Latency covers the whole request at the gateway.
Answer: Each request reports 1 if it was a 5xx and 0 otherwise, so the average over a period is the fraction of requests that failed. That is an error rate, which scales with traffic, unlike Sum.
Answer: HTTP APIs do not support X-Ray tracing. Either trace from the backend (OpenTelemetry in the service, starting the trace there) or use a REST API, which supports X-Ray active tracing, execution logs and WAF.
Answer: $context.resourcePath, which logs the route template like /accounts/{accountId} rather than the actual path. It reduces personal data in logs and makes grouping by route easy.
Answer: Missing periods are treated as healthy, so the alarm stays OK instead of drifting to INSUFFICIENT_DATA. For an alarm whose purpose is detecting absence of traffic, you would choose breaching instead.
Interview questions
- "How would you know if your API was failing for 5% of users?" — Alarm on error rate (Average of
5XXError, plus a 4xx spike alarm) and p99 latency rather than averages or raw counts; drill down with structured access logs and traces joined by a correlation/trace ID. - "What's the difference between
LatencyandIntegrationLatencyin API Gateway?" —Latencyis the full time API Gateway takes to respond;IntegrationLatencyis just the backend portion. The gap is edge overhead: authorizers, WAF, mapping. - "Would you use the X-Ray SDK for a new .NET service?" — No; it is in maintenance mode. Use OpenTelemetry with the AWS X-Ray propagator and ID generator, export via ADOT, and keep X-Ray as the trace store.
- "What should never be in API logs at a bank?" — Tokens, secrets, full request bodies, card data, and unnecessary personal data. Log opaque IDs, route templates and status codes; set retention and restrict access.
- "How do you avoid alert fatigue?" — Page only on customer-impact symptoms via a composite alarm; route cause-level alarms to chat; use M-of-N datapoints; review every page in post-incident reviews.
Certification practice
A REST API in API Gateway invokes a Lambda function. Users report slow responses. CloudWatch shows Latency p99 of 3,200 ms and IntegrationLatency p99 of 400 ms. The API uses a Lambda REQUEST authorizer. What should the developer do FIRST?
A. Increase the memory of the backend Lambda function. B. Enable authorizer result caching with an appropriate TTL and identity sources. C. Switch the backend to a VPC link. D. Enable API Gateway stage caching for all methods.
Answer: B. Almost all latency is outside the integration, and an uncached authorizer runs on every request. A targets the backend, which is fast. C changes nothing about the edge. D caches backend responses, which is not where time is spent and risks serving one customer's data to another if keys are wrong.
A team instruments a new .NET 10 service behind an API Gateway REST API with X-Ray tracing enabled. Traces in X-Ray show the API Gateway segment, but the service's spans appear as separate, unconnected traces. What is the MOST likely fix?
A. Enable detailed metrics on the stage.
B. Configure the OpenTelemetry SDK with the AWS X-Ray propagator and X-Ray trace ID generation.
C. Turn on execution logging at INFO level.
D. Grant the service xray:GetTraceSummaries.
Answer: B. The service must read X-Amzn-Trace-Id and generate X-Ray-compatible IDs to continue the gateway's trace. A and C produce metrics and logs, not trace context. D is a read permission used by consoles and tools, not for writing linked spans.
A bank needs per-request API logs delivered to a central log pipeline through Amazon Data Firehose, X-Ray traces starting at the gateway, and AWS WAF protection. Which API Gateway option meets all requirements?
A. HTTP API with a JWT authorizer B. REST API (Regional endpoint) C. WebSocket API D. HTTP API behind an Application Load Balancer
Answer: B. REST APIs support Firehose access logging, X-Ray tracing and WAF association. HTTP APIs support none of those three. WebSocket APIs are for bidirectional messaging. D adds an ALB but still lacks gateway-level tracing and Firehose logs.
A security review finds full Authorization headers and JSON bodies containing customer names in API Gateway logs. Which change BEST addresses the finding while keeping troubleshooting ability?
A. Delete the log groups weekly.
B. Disable all API Gateway logging.
C. Turn off execution-log data tracing, use a custom JSON access log format with only non-sensitive $context fields, and set log group retention and least-privilege read access.
D. Encrypt the log group with a customer managed KMS key.
Answer: C. It removes the sensitive data at the source while keeping useful per-request records. A and B destroy the evidence you need. D protects data at rest but still stores tokens and personal data that should never have been logged.
Completion checklist
- I can explain access logs vs execution logs vs metrics vs traces in two sentences
- I wrote a JSON access log format and can justify each field
- I know which observability features exist only on REST APIs
- I can read
LatencyvsIntegrationLatencyto locate a slowdown - My local .NET service writes JSON logs with trace IDs and exports OpenTelemetry spans
- I created (or designed) a composite customer-impact alarm with M-of-N datapoints
- I ran the cleanup steps and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- Alarm on what customers feel: error rates and latency percentiles, combined into one paging composite alarm.
- Structured JSON access logs with
extendedRequestId,resourcePath, latencies anderrorTypemake most incidents a five-minute query. - REST APIs give you execution logs, X-Ray tracing, Firehose logs and WAF; HTTP APIs do not.
- Instrument .NET with OpenTelemetry plus the X-Ray propagator; the X-Ray SDK is in maintenance mode.
- Logs and traces hold personal data: minimise, mask, restrict and set retention.
Further reading
- Choose between REST APIs and HTTP APIs
- Variables for access logging for API Gateway
- Amazon API Gateway dimensions and metrics
- Set up CloudWatch logging for REST APIs in API Gateway
- Trace user requests to REST APIs using X-Ray
- Using Amazon CloudWatch alarms
- AWS Distro for OpenTelemetry
- OpenTelemetry .NET
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Lab: Hand-sign an AWS request with SigV4 (Lab 4.5)
Implement AWS Signature Version 4 in plain C#, prove it against AWS's published test vectors, sign a real STS GetCallerIdentity call and compare it with the SDK, then reuse the signer as a DelegatingHandler for IAM-protected APIs.
API security: AWS WAF and the OWASP API Security Top 10 for a banking API
Map every OWASP API Security Top 10 (2023) risk to a concrete control on AWS or in .NET, attach a WAF web ACL with managed and rate-based rules to an API Gateway REST API, and enforce object- and property-level authorisation in ASP.NET Core so a customer can only ever see their own accounts.