Skip to Content
Cloud Engineering Academy
4. API Development & Integration

API monitoring: access logs, metrics, traces and alarms that matter

Turn on structured API Gateway access logs, read the right CloudWatch metrics, trace a request from the edge into .NET with OpenTelemetry and X-Ray, and build a small set of alarms that page someone only when customers are affected.

intermediate⏱ 2 h 🟢 FreeDVA-C03SAA-C03SCS-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.CloudWatch 4.x) · OpenTelemetry .NET 1.21 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain the four signals an API team needs: access logs, execution logs, metrics and traces, and what each one can and cannot answer.
  • Turn on structured JSON access logs for API Gateway REST and HTTP APIs, choosing $context variables that make incidents debuggable without logging personal data.
  • Read the API Gateway CloudWatch metrics (Count, 4XXError/4xx, 5XXError/5xx, Latency, IntegrationLatency) and use the gap between Latency and IntegrationLatency to locate a slowdown.
  • Trace a request end to end with AWS X-Ray (REST APIs) and OpenTelemetry for .NET, and join traces to logs with a correlation ID.
  • Create a small set of customer-impact alarms (error rate, p99 latency, throttling, authorizer failures) with sensible missing-data handling, from the CLI and from C#.

Prerequisites

Why this matters in banking

It is 08:55 on a Monday. Harbourline Bank's (fictional) mobile app shows "Something went wrong" on the accounts screen for about one customer in twenty. The API team's only dashboard shows average latency, which looks fine. Support escalates after 40 minutes, when the complaints queue is already long.

When the team finally looks, the answer takes five minutes: a new Lambda authorizer release fails for tokens issued by an older app version, so API Gateway returns 401 for those customers. Count was normal, average Latency was normal, and the 5XXError alarm stayed quiet because the failures were 4xx. Nobody was watching the one thing that mattered: the share of customers whose requests failed.

Regulators expect banks to know when an important business service is degraded and to evidence how quickly they reacted. Good API monitoring is how you produce that evidence, and how you avoid finding out from social media.

The idea in plain language

Monitoring an API means being able to answer four questions quickly:

  1. Is it broken right now, and for how many people? → Metrics and alarms: cheap numbers over time, like request count, error rate and latency percentiles.
  2. What exactly happened to this one request? → Access logs: one line per request, with status, latency, caller and IDs.
  3. Where inside the system did the time go? → Traces: a timeline of one request as it passes through API Gateway, the authorizer, your .NET code, DynamoDB and a partner call.
  4. What did API Gateway itself decide? → Execution logs (REST APIs only): API Gateway's internal steps, like "authorizer returned Deny" or "mapping template failed".

Analogy: Think of a bank branch. Metrics are the queue counter on the wall: how many people are waiting and how long the wait is. Access logs are the visitor book: who came in, when, which counter served them and how it ended. A trace is following one customer with a stopwatch from the door to the cashier to the back office and out again. Alarms are the branch manager's pager, set to go off only when the queue is long enough that customers will complain.

The big mindset shift from "monitoring servers" is that you alarm on symptoms customers feel (errors, slowness, rejection) rather than on causes (CPU, memory). Causes go on dashboards for diagnosis; symptoms page a human.

What problem the AWS services solve

NeedAWS featureHonest limitations
Per-request record of every callAPI Gateway access logs to CloudWatch Logs (REST and HTTP APIs); REST APIs can also send access logs to Amazon Data FirehoseYou choose the fields; anything not in the format is lost. Ingestion is billed per GB
API Gateway's internal decisionsExecution logs (REST only), level ERROR or INFO, optional full request/response data"Log full requests/responses" writes bodies and headers, which can contain tokens and personal data; never enable it in production
Request counts, errors and latencyCloudWatch metrics in the AWS/ApiGateway namespace, sent automaticallyOne-minute granularity by default; per-method or per-route metrics need detailed metrics, which cost extra as custom metrics
End-to-end request timelineAWS X-Ray active tracing on REST API stages; OpenTelemetry in your .NET codeHTTP APIs do not support X-Ray tracing. The X-Ray SDKs and daemon are in maintenance mode since 2026-02-25, so instrument new .NET code with OpenTelemetry (ADOT) and keep X-Ray as the trace backend
Being told when customers hurtCloudWatch alarms and composite alarms, notifying Amazon SNSAlarms evaluate per period; badly chosen thresholds and missing-data settings cause either noise or silence

Two facts from the comparison table in the API Gateway guide shape every design decision here: execution logs, X-Ray tracing and Firehose access logs exist only for REST APIs. If deep edge observability matters (and in banking it usually does), that is one more reason the Account Information API uses a REST API, alongside WAF support covered in API security.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
APIM diagnostic logs to Log AnalyticsAPI Gateway access logs to CloudWatch LogsYou write the log format yourself from $context variables
APIM gateway logs / trace policyAPI Gateway execution logs (REST)Enabled per stage; needs an account-level CloudWatch role
Application Insights requests and dependenciesOpenTelemetry in .NET → X-Ray (traces), CloudWatch (metrics, logs)Same OpenTelemetry API you may already use with the Azure Monitor exporter; only the exporter changes
Kusto (KQL) queriesCloudWatch Logs Insights query languagePipe-based, similar spirit; stats, filter, parse, pctile
Azure Monitor metric alerts + action groupsCloudWatch alarms + SNS topicsAlarms have three states (OK, ALARM, INSUFFICIENT_DATA) and an explicit missing-data policy
Application MapX-Ray trace map / CloudWatch Application SignalsApplication Signals adds SLOs and service maps; see Module 11
Activity and traceparent headerSame System.Diagnostics.Activity; X-Ray also uses X-Amzn-Trace-IdConfigure the AWS X-Ray propagator so both headers carry the same trace

Architecture

Security-boundary view. Logs are data stores with their own access rules:

  • What goes in: request IDs, trace IDs, status codes, latencies, the authenticated subject (sub) or principal ID, client IP. Not access tokens, account numbers in query strings, request bodies or names.
  • Who can read them: an ApiOperators role with logs:StartQuery and logs:GetQueryResults on the API's log groups only. Developers do not get production log access by default.
  • How long they live: a retention period on every log group (for example 90 days hot in CloudWatch, then exported to S3 under Object Lock if your records policy needs longer). Infinite retention is never the default in this platform.
  • Encryption: CloudWatch Logs encrypts at rest; associate a customer managed KMS key if your bank's policy requires key control.

Step-by-step implementation

Decide what "healthy" means before you collect anything

Write down a service-level objective (SLO) for the Accounts API, for example: "99.9% of GET /accounts requests in a rolling 30 days return a non-5xx response within 1,000 ms." Two things follow:

  • You need error rate and latency percentile signals, not averages. An average hides the 1 in 20 customers who wait 8 seconds.
  • 4xx errors are usually the caller's fault, but a sudden jump in 401/403 often means your auth change broke clients, as in the scenario above. Watch them as a rate.

Give API Gateway permission to write logs (REST APIs, once per Region)

REST API execution logs and access logs are written by API Gateway using an IAM role set at the account level (cloudwatchRoleArn). The role trusts apigateway.amazonaws.com and has the AWS managed policy AmazonAPIGatewayPushToCloudWatchLogs. HTTP APIs do not need this setting; they write access logs using a service-linked mechanism and a log group you specify.

Turn on structured access logs

Create a log group with retention, then set the stage's access log destination and a JSON format. JSON makes Logs Insights discover fields automatically. A good banking format:

{
  "requestId": "$context.requestId",
  "extendedRequestId": "$context.extendedRequestId",
  "traceId": "$context.xrayTraceId",
  "ip": "$context.identity.sourceIp",
  "userAgent": "$context.identity.userAgent",
  "requestTime": "$context.requestTime",
  "httpMethod": "$context.httpMethod",
  "resourcePath": "$context.resourcePath",
  "status": "$context.status",
  "protocol": "$context.protocol",
  "responseLength": "$context.responseLength",
  "responseLatency": "$context.responseLatency",
  "integrationLatency": "$context.integrationLatency",
  "integrationStatus": "$context.integration.status",
  "integrationError": "$context.integration.error",
  "authorizerError": "$context.authorizer.error",
  "authorizerLatency": "$context.authorizer.latency",
  "principalId": "$context.authorizer.principalId",
  "wafResponse": "$context.wafResponseCode",
  "errorType": "$context.error.responseType",
  "errorMessage": "$context.error.message"
}

Notes on the choices:

  • requestId can be overridden by clients; extendedRequestId is always generated by API Gateway, so log both.
  • resourcePath logs the template (/accounts/{accountId}), not the real path with the account ID in it. That keeps identifiers out of logs and makes grouping easy. $context.path would include the real value.
  • principalId is whatever your authorizer returned. Make it the opaque customer ID (CUS-00012345) or token sub, never an email address.
  • errorType tells you why API Gateway rejected something (THROTTLED, UNAUTHORIZED, ACCESS_DENIED, WAF_FILTERED, INTEGRATION_TIMEOUT and so on).

HTTP APIs support a smaller set of variables (no xrayTraceId, no WAF fields); keep the same JSON keys where they exist so one query works across both.

Turn on execution logs at ERROR level only (REST)

Set the stage's method settings to loggingLevel=ERROR and dataTraceEnabled=false. INFO is useful for a short debugging session in a non-production stage; full data tracing logs headers and bodies, including Authorization headers and customer data, and must stay off in production.

Enable tracing at the edge and in the code

  • On the REST API stage, enable X-Ray tracing (tracingEnabled=true). API Gateway records a segment and passes the trace context to the integration in the X-Amzn-Trace-Id header.
  • In the .NET service, use OpenTelemetry with the AWS X-Ray ID generator and propagator, export spans over OTLP to an ADOT collector (a sidecar on ECS, or the ADOT Lambda layer), and let the collector send them to X-Ray.
  • Write the trace ID into every log line so you can jump from a log entry to its trace and back.

Create a handful of alarms, then a composite

Start with four, all on the production stage:

AlarmMetricRule of thumb
Server error rate5XXError (REST) Average = fraction of requests> 0.01 (1%) for 3 of 5 one-minute periods
Client error spike4XXError Average> 0.10 for 3 of 5 periods, or anomaly detection
Slow for some customersLatency p99> 2000 ms for 5 of 5 periods
Backend strugglingIntegrationLatency p99> 1500 ms for 5 of 5 periods

Then a composite alarm "AccountsApiCustomerImpact" = ALARM(server-errors) OR ALARM(latency-p99), which is the only one that pages. The others notify a chat channel. Composite alarms reduce noise when one failure trips several metrics.

Practise the investigation path

An alarm fires. Your runbook is: open the dashboard → see which metric moved → run the saved Logs Insights query for that time window → pick a failing extendedRequestId → open its trace → find the slow or failing span → read the app log lines with the same trace ID. If any step in that chain is missing, fix the instrumentation before the next incident, not during it.

AWS Console instructions

  • Account CloudWatch role (REST): Console → API Gateway → in the left navigation choose Settings → Logging → Edit → paste the ARN of the role that has AmazonAPIGatewayPushToCloudWatchLogs.
  • Access logs (REST): API Gateway → APIs → your API → Stages → prod → Logs and tracing → Edit → turn on Custom access logging, enter the log group ARN and the JSON format. In the same panel set CloudWatch logs to Errors only, leave Data tracing off, and turn on X-Ray tracing.
  • Access logs (HTTP API): API Gateway → your HTTP API → Monitor → Logging → select the stage → Edit → enable access logging, log group ARN, format.
  • Detailed metrics: for REST, the same Logs and tracing panel has Detailed metrics; for HTTP APIs it is under the stage's settings. Leave them off unless you need per-route numbers.
  • Alarms: Console → CloudWatch → Alarms → Create alarm → Select metric → ApiGateway → By API name and stage (REST) or By Stage (HTTP) → pick the metric → choose the statistic (for example p99) and period → set the threshold and Datapoints to alarm → under Additional configuration set Missing data treatment → choose the SNS topic.
  • Traces: Console → CloudWatch → X-Ray traces → Traces, filter by URL or status, open one to see the timeline.

Console layouts change often; if a label differs, the setting names (access logging, tracing, detailed metrics) stay the same.

AWS CLI

# Variables used below (fictional IDs: replace with yours)
PROFILE=learn-sandbox
REGION=us-east-1
API_ID=a1b2c3d4e5          # REST API ID
STAGE=prod
LG=/aws/apigateway/hl-m04-accounts-access

# 1. Create the access log group with 90-day retention and the mandatory tags
aws logs create-log-group --log-group-name "$LG" \
  --tags Project=cea-lab,Lab=m04-l17,Environment=learn,Owner=you \
  --profile $PROFILE --region $REGION
aws logs put-retention-policy --log-group-name "$LG" --retention-in-days 90 \
  --profile $PROFILE --region $REGION

# 2. Point API Gateway (REST, account level) at a role that can write logs
aws apigateway update-account \
  --patch-operations op=replace,path=/cloudwatchRoleArn,value=arn:aws:iam::111122223333:role/hl-apigw-cloudwatch \
  --profile $PROFILE --region $REGION

# 3. Enable JSON access logs on the stage (format stored in a local file, minified)
LG_ARN=$(aws logs describe-log-groups --log-group-name-prefix "$LG" \
  --query 'logGroups[0].arn' --output text --profile $PROFILE --region $REGION)
FORMAT=$(jq -c . access-log-format.json)
aws apigateway update-stage --rest-api-id $API_ID --stage-name $STAGE \
  --patch-operations \
    "op=replace,path=/accessLogSettings/destinationArn,value=${LG_ARN%:*}" \
    "op=replace,path=/accessLogSettings/format,value=$FORMAT" \
  --profile $PROFILE --region $REGION

# 4. Execution logs at ERROR, no data tracing, and X-Ray tracing on
aws apigateway update-stage --rest-api-id $API_ID --stage-name $STAGE \
  --patch-operations \
    'op=replace,path=/*/*/logging/loglevel,value=ERROR' \
    'op=replace,path=/*/*/logging/dataTrace,value=false' \
    'op=replace,path=/tracingEnabled,value=true' \
  --profile $PROFILE --region $REGION

# 5. Server-error-rate alarm: Average of 5XXError is the fraction of requests that failed
aws cloudwatch put-metric-alarm --alarm-name hl-m04-accounts-5xx-rate \
  --namespace AWS/ApiGateway --metric-name 5XXError \
  --dimensions Name=ApiName,Value=hl-m04-accounts-api Name=Stage,Value=$STAGE \
  --statistic Average --period 60 --evaluation-periods 5 --datapoints-to-alarm 3 \
  --threshold 0.01 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:hl-m04-oncall \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m04-l17 Key=Environment,Value=learn Key=Owner,Value=you \
  --profile $PROFILE --region $REGION

# 6. p99 latency alarm (percentiles use --extended-statistic)
aws cloudwatch put-metric-alarm --alarm-name hl-m04-accounts-latency-p99 \
  --namespace AWS/ApiGateway --metric-name Latency \
  --dimensions Name=ApiName,Value=hl-m04-accounts-api Name=Stage,Value=$STAGE \
  --extended-statistic p99 --period 60 --evaluation-periods 5 --datapoints-to-alarm 5 \
  --threshold 2000 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:hl-m04-oncall \
  --profile $PROFILE --region $REGION

# 7. Composite alarm: page only when customers are affected
aws cloudwatch put-composite-alarm --alarm-name hl-m04-accounts-customer-impact \
  --alarm-rule 'ALARM("hl-m04-accounts-5xx-rate") OR ALARM("hl-m04-accounts-latency-p99")' \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:hl-m04-pager \
  --profile $PROFILE --region $REGION

# 8. Ask a question of the access logs with Logs Insights (last hour)
aws logs start-query --log-group-name "$LG" \
  --start-time $(( $(date +%s) - 3600 )) --end-time $(date +%s) \
  --query-string 'fields @timestamp, status, resourcePath, errorType | filter status >= 400 | stats count(*) as n by status, errorType, resourcePath | sort n desc | limit 20' \
  --profile $PROFILE --region $REGION
# then: aws logs get-query-results --query-id <id from the previous output>

Line notes: the access log destination ARN must not end in :* (that suffix appears in describe-log-groups output, hence ${LG_ARN%:*}). /*/*/ in a patch path means "all resources and methods" in the stage. For HTTP APIs the dimension names differ: ApiId and Stage, and metric names are lower case (4xx, 5xx).

Logs Insights queries worth saving

# Error rate per minute from access logs
fields @timestamp, status
| stats sum(status >= 500) / count(*) * 100 as pct5xx, sum(status >= 400 and status < 500) / count(*) * 100 as pct4xx by bin(1m)

# Where the time goes: edge overhead vs backend
fields responseLatency, integrationLatency
| filter ispresent(integrationLatency)
| stats pct(responseLatency, 99) as p99Total, pct(integrationLatency, 99) as p99Backend, pct(responseLatency - integrationLatency, 99) as p99Edge by bin(5m)

# Why are callers being rejected at the edge?
fields @timestamp, errorType, authorizerError, principalId
| filter errorType in ["UNAUTHORIZED", "ACCESS_DENIED", "AUTHORIZER_FAILURE", "THROTTLED", "WAF_FILTERED"]
| stats count(*) by errorType, authorizerError

Infrastructure as Code

infra/cdk/MonitoringStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.APIGateway;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.CloudWatch.Actions;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.SNS;
using Constructs;

namespace Harbourline.Accounts.Infra;

public sealed class MonitoringStack : Stack
{
    public MonitoringStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        var accessLogs = new LogGroup(this, "AccessLogs", new LogGroupProps
        {
            LogGroupName = "/aws/apigateway/hl-m04-accounts-access",
            Retention = RetentionDays.THREE_MONTHS,
            RemovalPolicy = RemovalPolicy.DESTROY // lab only: production keeps logs
        });

        var api = new RestApi(this, "AccountsApi", new RestApiProps
        {
            RestApiName = "hl-m04-accounts-api",
            CloudWatchRole = true, // creates and sets the account-level logging role
            DeployOptions = new StageOptions
            {
                StageName = "prod",
                TracingEnabled = true,
                LoggingLevel = MethodLoggingLevel.ERROR,
                DataTraceEnabled = false,
                MetricsEnabled = false, // detailed per-method metrics off by default
                AccessLogDestination = new LogGroupLogDestination(accessLogs),
                AccessLogFormat = AccessLogFormat.Custom(System.Text.Json.JsonSerializer.Serialize(new
                {
                    requestId = "$context.requestId",
                    extendedRequestId = "$context.extendedRequestId",
                    traceId = "$context.xrayTraceId",
                    ip = "$context.identity.sourceIp",
                    httpMethod = "$context.httpMethod",
                    resourcePath = "$context.resourcePath",
                    status = "$context.status",
                    responseLatency = "$context.responseLatency",
                    integrationLatency = "$context.integrationLatency",
                    principalId = "$context.authorizer.principalId",
                    errorType = "$context.error.responseType"
                }))
            }
        });
        api.Root.AddResource("health").AddMethod("GET", new MockIntegration(new IntegrationOptions
        {
            RequestTemplates = new Dictionary<string, string> { ["application/json"] = "{\"statusCode\": 200}" },
            IntegrationResponses = [new IntegrationResponse { StatusCode = "200" }]
        }), new MethodOptions { MethodResponses = [new MethodResponse { StatusCode = "200" }] });

        var oncall = new Topic(this, "OnCall", new TopicProps { TopicName = "hl-m04-oncall" });

        var errorRate = new Alarm(this, "ServerErrorRate", new AlarmProps
        {
            AlarmName = "hl-m04-accounts-5xx-rate",
            Metric = api.MetricServerError(new MetricOptions { Statistic = "Average", Period = Duration.Minutes(1) }),
            Threshold = 0.01,
            EvaluationPeriods = 5,
            DatapointsToAlarm = 3,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
            TreatMissingData = TreatMissingData.NOT_BREACHING
        });

        var latencyP99 = new Alarm(this, "LatencyP99", new AlarmProps
        {
            AlarmName = "hl-m04-accounts-latency-p99",
            Metric = api.MetricLatency(new MetricOptions { Statistic = "p99", Period = Duration.Minutes(1) }),
            Threshold = 2000,
            EvaluationPeriods = 5,
            DatapointsToAlarm = 5,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
            TreatMissingData = TreatMissingData.NOT_BREACHING
        });

        var impact = new CompositeAlarm(this, "CustomerImpact", new CompositeAlarmProps
        {
            CompositeAlarmName = "hl-m04-accounts-customer-impact",
            AlarmRule = AlarmRule.AnyOf(errorRate, latencyP99)
        });
        impact.AddAlarmAction(new SnsAction(oncall));

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m04-l17");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        Amazon.CDK.Tags.Of(this).Add("Owner", "you");
    }
}

Complete code

The service side: an ASP.NET Core minimal API that writes JSON logs with trace and correlation IDs, emits OpenTelemetry traces in X-Ray-compatible form, and records one business metric. It runs locally with no AWS account; traces go to any OTLP endpoint (an ADOT collector in AWS, or Jaeger in Docker on your laptop).

src/Harbourline.Accounts.Api/Harbourline.Accounts.Api.csproj
<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.21.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.21.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.21.0" />
    <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.21.0" />
    <PackageReference Include="OpenTelemetry.Extensions.AWS" Version="1.21.0" />
  </ItemGroup>
</Project>
src/Harbourline.Accounts.Api/Program.cs
using System.Diagnostics;
using System.Diagnostics.Metrics;
using OpenTelemetry;
using OpenTelemetry.Extensions.AWS.Trace;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

// 1. Structured JSON logs that include the current trace and span IDs.
builder.Logging.ClearProviders();
builder.Logging.AddJsonConsole(o =>
{
    o.IncludeScopes = true;
    o.UseUtcTimestamp = true;
    o.TimestampFormat = "yyyy-MM-ddTHH:mm:ss.fffZ";
});
builder.Logging.Configure(o => o.ActivityTrackingOptions =
    ActivityTrackingOptions.TraceId | ActivityTrackingOptions.SpanId);

// 2. Propagate X-Ray's header (X-Amzn-Trace-Id) as well as W3C traceparent.
Sdk.SetDefaultTextMapPropagator(new AWSXRayPropagator());

// 3. Tracing and metrics, exported over OTLP (endpoint from OTEL_EXPORTER_OTLP_ENDPOINT).
builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService("hl-accounts-api", serviceVersion: "1.4.0"))
    .WithTracing(t => t
        .AddXRayTraceId()                 // X-Ray-compatible trace IDs
        .AddAspNetCoreInstrumentation(o => o.Filter = ctx => ctx.Request.Path != "/health")
        .AddHttpClientInstrumentation()
        .AddSource(AccountsTelemetry.SourceName)
        .AddOtlpExporter())
    .WithMetrics(m => m
        .AddAspNetCoreInstrumentation()
        .AddMeter(AccountsTelemetry.MeterName)
        .AddOtlpExporter());

builder.Services.AddSingleton<AccountsTelemetry>();
builder.Services.AddSingleton<IAccountStore, InMemoryAccountStore>();
builder.Services.AddProblemDetails();

var app = builder.Build();

// 4. Correlation ID: reuse the caller's, otherwise API Gateway's request ID, otherwise generate one.
app.Use(async (ctx, next) =>
{
    var correlationId = ctx.Request.Headers["X-Correlation-Id"].FirstOrDefault()
        ?? ctx.Request.Headers["X-Request-Id"].FirstOrDefault()
        ?? Guid.CreateVersion7().ToString();
    ctx.Response.Headers["X-Correlation-Id"] = correlationId;
    Activity.Current?.SetTag("correlation.id", correlationId);

    var logger = ctx.RequestServices.GetRequiredService<ILoggerFactory>().CreateLogger("Request");
    using (logger.BeginScope(new Dictionary<string, object> { ["correlationId"] = correlationId }))
    {
        await next();
    }
});

app.MapGet("/health", () => Results.Ok(new { status = "ok" }));

app.MapGet("/accounts", async (HttpContext ctx, IAccountStore store, AccountsTelemetry telemetry, ILogger<Program> log) =>
{
    // In the real API the customer ID comes from the validated token's "sub" claim.
    var customerId = ctx.Request.Headers["X-Customer-Id"].FirstOrDefault();
    if (string.IsNullOrWhiteSpace(customerId))
    {
        telemetry.Rejected("missing_customer");
        return Results.Problem(statusCode: 401, title: "Unauthenticated");
    }

    using var span = telemetry.Source.StartActivity("LoadAccounts");
    span?.SetTag("customer.id.hash", AccountsTelemetry.Hash(customerId)); // never the raw ID in traces

    var sw = Stopwatch.StartNew();
    var accounts = await store.GetForCustomerAsync(customerId, ctx.RequestAborted);
    telemetry.RecordLookup(sw.Elapsed, accounts.Count);

    log.LogInformation("Returned {AccountCount} accounts in {ElapsedMs} ms", accounts.Count, sw.ElapsedMilliseconds);
    return Results.Ok(new { items = accounts });
});

app.Run();

public sealed class AccountsTelemetry : IDisposable
{
    public const string SourceName = "Harbourline.Accounts";
    public const string MeterName = "Harbourline.Accounts";

    private readonly Meter _meter = new(MeterName, "1.0.0");
    private readonly Histogram<double> _lookupMs;
    private readonly Counter<long> _rejected;

    public ActivitySource Source { get; } = new(SourceName);

    public AccountsTelemetry()
    {
        _lookupMs = _meter.CreateHistogram<double>("accounts.lookup.duration", unit: "ms",
            description: "Time to load a customer's accounts from the store");
        _rejected = _meter.CreateCounter<long>("accounts.requests.rejected",
            description: "Requests rejected by the API before reaching the store");
    }

    public void RecordLookup(TimeSpan elapsed, int count) =>
        _lookupMs.Record(elapsed.TotalMilliseconds, new KeyValuePair<string, object?>("result.empty", count == 0));

    public void Rejected(string reason) =>
        _rejected.Add(1, new KeyValuePair<string, object?>("reason", reason));

    public static string Hash(string value) =>
        Convert.ToHexString(System.Security.Cryptography.SHA256.HashData(System.Text.Encoding.UTF8.GetBytes(value)))[..16];

    public void Dispose()
    {
        _meter.Dispose();
        Source.Dispose();
    }
}

public sealed record AccountSummary(string AccountId, string Currency, string Product, decimal Balance);

public interface IAccountStore
{
    Task<IReadOnlyList<AccountSummary>> GetForCustomerAsync(string customerId, CancellationToken ct);
}

public sealed class InMemoryAccountStore : IAccountStore
{
    private static readonly Dictionary<string, AccountSummary[]> Data = new()
    {
        ["CUS-00012345"] = [new("ACC-0000123456", "GBP", "current", 1520.40m), new("ACC-0000123457", "GBP", "savings", 8000.00m)],
        ["CUS-00067890"] = [new("ACC-0000678901", "EUR", "current", 310.15m)]
    };

    public async Task<IReadOnlyList<AccountSummary>> GetForCustomerAsync(string customerId, CancellationToken ct)
    {
        await Task.Delay(Random.Shared.Next(5, 40), ct); // simulate a store call
        return Data.TryGetValue(customerId, out var a) ? a : [];
    }
}

Run it locally with Jaeger as a free trace viewer:

# Jaeger all-in-one accepts OTLP on 4317 (gRPC) and shows a UI on 16686
docker run --rm -d --name jaeger -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:1.62.0

cd src/Harbourline.Accounts.Api
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 dotnet run --urls http://localhost:5080

# In another terminal: one good call, one rejected call
curl -s -H "X-Customer-Id: CUS-00012345" -H "X-Correlation-Id: demo-001" http://localhost:5080/accounts
curl -s -i http://localhost:5080/accounts
# Open http://localhost:16686 and search for service hl-accounts-api

The Jaeger image tag is pinned for reproducibility; any recent jaegertracing/all-in-one tag works ⚠ VERIFY the tag still exists on Docker Hub before you run it.

Code explained line by line

  1. AddJsonConsole writes one JSON object per log line to stdout. On ECS with the awslogs driver and on Lambda, stdout goes straight into CloudWatch Logs, and JSON fields become queryable in Logs Insights without parse.
  2. ActivityTrackingOptions.TraceId | SpanId adds the current OpenTelemetry trace and span IDs to every log scope. This is the join key between a log line and a trace.
  3. Sdk.SetDefaultTextMapPropagator(new AWSXRayPropagator()) reads and writes the X-Amzn-Trace-Id header. When API Gateway has tracing on, it passes this header to your integration, so your spans become children of the API Gateway segment instead of a separate trace.
  4. .AddXRayTraceId() makes OpenTelemetry generate trace IDs in the X-Ray format (an epoch timestamp prefix plus random bits), which X-Ray needs to accept spans that started in your code.
  5. AddAspNetCoreInstrumentation(o => o.Filter = ...) creates a server span per request and skips /health, which load balancers call constantly and would otherwise drown real traffic in traces (and cost).
  6. AddHttpClientInstrumentation() creates client spans for outbound calls, so a slow partner call shows up as a long child span.
  7. .AddOtlpExporter() sends spans and metrics over OTLP. The endpoint comes from OTEL_EXPORTER_OTLP_ENDPOINT: an ADOT collector sidecar in AWS, Jaeger locally. The app never needs AWS credentials for tracing; the collector does.
  8. The correlation middleware reuses a caller's X-Correlation-Id if present, falls back to X-Request-Id, and otherwise creates a time-ordered UUID v7. Configure API Gateway to pass $context.requestId as X-Request-Id to the integration so edge and app logs share an ID.
  9. logger.BeginScope(...) attaches correlationId to every log line written during the request, including logs from deeper services.
  10. customer.id.hash puts a hash of the customer ID in the trace. Traces are viewed by more people than the database is; treat span attributes like log fields and keep personal data out.
  11. Histogram<double> with unit ms gives you percentiles of the store lookup in your metrics backend; Counter<long> counts rejections by reason. Keep tag values low-cardinality (a reason code, a boolean), never a customer or account ID, or your metric bill explodes.
  12. Results.Problem(...) returns RFC 9457 ProblemDetails, consistent with REST API design.

Security considerations

⚠ Learning shortcut — not for production

The local demo reads the customer ID from an X-Customer-Id header so you can test without Cognito. In production the customer ID comes only from the validated token's sub claim (see Lambda authorizers and Cognito user pools); a header like this would let anyone read anyone's accounts.

⚠ Learning shortcut — not for production

The CDK stack sets RemovalPolicy.DESTROY on the log group so teardown is clean. Production log groups are retained on stack deletion and, where your records policy requires it, exported to an S3 bucket with Object Lock (see S3 protection).

  • Logs are personal data stores. Under UK GDPR, IP addresses and customer identifiers are personal data. Minimise what you log, set retention, restrict who can query, and document the purpose. Your data-protection officer decides the retention period, not the developer.
  • Never log tokens. Do not include $context.identity.* headers that carry credentials, and never enable execution-log data tracing in production. If a token does reach logs, treat it as a credential leak: revoke and rotate.
  • CloudWatch Logs data protection policies can detect and mask some sensitive data types in log groups (for example, credit card numbers) as a safety net; they are not a substitute for not logging the data ⚠ VERIFY which managed data identifiers fit your fields.
  • Least privilege for observability. The ADOT collector role needs xray:PutTraceSegments and xray:PutTelemetryRecords (or the AWS managed policy for the X-Ray daemon write access), plus cloudwatch:PutMetricData if it exports metrics; nothing else.
  • Integrity for audit. Access logs are operational evidence, not your audit trail. Who changed the API configuration is answered by CloudTrail, covered in Module 11.

Banking use case

Harbourline's Accounts API is classified as supporting an important business service ("customers can view balances"). The operations standard says:

  • SLO: 99.9% of balance requests succeed within 1 second over 30 days.
  • Paging alarm: the composite customer-impact alarm. It pages the on-call engineer through SNS → the incident tool.
  • Non-paging alarms: 4xx spike, authorizer latency, integration p99, throttle count. They post to the team's chat channel during working hours.
  • Dashboard: request count, 4xx and 5xx rate, p50/p90/p99 latency, edge overhead (Latency - IntegrationLatency), WAF blocks, and the business histogram accounts.lookup.duration.
  • Evidence: every page creates an incident record with the alarm history, the saved Logs Insights query results and the trace IDs inspected. That record is what the bank shows if asked how quickly it detected and responded to the disruption.

When the Monday incident from the opening happens again, the 4xx-spike alarm fires within 3 minutes, the "Why are callers being rejected?" query shows UNAUTHORIZED with authorizer error text pointing at the token's issuer claim, and the team rolls back the authorizer before support even hears about it.

Common mistakes

  • Alarming on averages. Average latency stays flat while the slowest 1% suffer. Use p90/p99 for latency and averages only for 0/1 metrics like 5XXError, where the average is the error rate.
  • Using Sum of 5XXError with a fixed threshold. Ten errors per minute is a disaster at 50 requests and noise at 50,000. Alarm on the rate.
  • Ignoring 4xx. Auth and validation regressions show up as 4xx. Watch for sudden changes, not absolute levels.
  • treat-missing-data left as missing. A quiet API at night produces no datapoints; alarms flip to INSUFFICIENT_DATA and people learn to ignore that state. For error-rate alarms, notBreaching is usually right; for "the API must receive traffic" heartbeat alarms, use breaching.
  • Logging the real path. $context.path and full URLs put account IDs in logs. Log resourcePath.
  • Full request/response execution logging in production. It writes tokens and customer data to logs.
  • High-cardinality metric dimensions. One custom metric per customer ID creates millions of metrics and a large bill.
  • Instrumenting new code with the X-Ray SDK. It is in maintenance mode; use OpenTelemetry and keep X-Ray as the backend.
  • No retention. Default log group retention is "never expire", so cost grows forever.

Troubleshooting

SymptomLikely causeFix
CloudWatch Logs role ARN must be set in account settings to enable loggingREST API account-level role not setaws apigateway update-account with a role trusted by apigateway.amazonaws.com
Access log group stays emptyDestination ARN ends in :*, wrong Region, or no requests since changeUse the ARN without :*; send a request; check the stage, not a different stage
Execution log group missingLogging level OFF, or you are using an HTTP APISet ERROR; HTTP APIs have no execution logs, rely on access logs
App spans appear as a separate trace from API GatewayPropagator not set, or trace ID format not X-RayAWSXRayPropagator and .AddXRayTraceId(); confirm the X-Amzn-Trace-Id header reaches the app
No traces in X-Ray at allCollector role lacks X-Ray permissions, or OTLP endpoint wrongCheck collector logs; grant xray:PutTraceSegments; verify OTEL_EXPORTER_OTLP_ENDPOINT
Alarm stuck in INSUFFICIENT_DATANo traffic, wrong dimension names or valuesCompare dimensions with aws cloudwatch list-metrics --namespace AWS/ApiGateway; set a missing-data policy
Latency high but IntegrationLatency lowTime spent at the edge: authorizer, WAF, mapping templatesLog authorizerLatency and waf.latency; enable authorizer caching
HTTP API alarm never firesUsing REST metric namesHTTP APIs use 4xx, 5xx and dimension ApiId

Cost and cleanup

This lesson is free when run locally. The .NET service, Jaeger and the Logs Insights syntax all work without an AWS account.

If you deploy the optional stack, costs are small but not zero (us-east-1, check the CloudWatch and X-Ray pricing pages ⚠ VERIFY current rates):

  • CloudWatch Logs ingestion is charged per GB (around USD 0.50/GB for standard logs) and storage per GB-month; a lab session produces kilobytes.
  • Standard-resolution metric alarms cost about USD 0.10 per alarm per month, prorated; composite alarms are priced separately. Four alarms for one day cost cents.
  • X-Ray includes a monthly free allowance of recorded traces (100,000) and then charges per million.
  • API Gateway's built-in metrics are free; detailed metrics are charged as custom metrics.
  • Logs Insights queries are charged per GB scanned.

Teardown:

aws cloudwatch delete-alarms --alarm-names hl-m04-accounts-customer-impact --profile learn-sandbox
aws cloudwatch delete-alarms --alarm-names hl-m04-accounts-5xx-rate hl-m04-accounts-latency-p99 --profile learn-sandbox
aws logs delete-log-group --log-group-name /aws/apigateway/hl-m04-accounts-access --profile learn-sandbox
# If you used CDK:  cdk destroy   |   Terraform:  terraform destroy
docker rm -f jaeger

# Confirm nothing tagged for this lab remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --query 'ResourceTagMappingList[].ResourceARN' --profile learn-sandbox

Delete the composite alarm before the alarms it references, or the delete call fails.

Exercises

  • 🟢 Guided: Run the local API with Jaeger. Make 20 good and 5 bad calls. In Jaeger, find a trace for a good call and identify the LoadAccounts span. In the console output, find a log line and confirm its TraceId matches the trace.
  • 🟡 Challenge: Write a Logs Insights query that returns, per 5 minutes, the p99 of responseLatency - integrationLatency (edge overhead) only for requests where an authorizer ran. Then design an alarm on authorizer latency using the AuthorizerLatency field from access logs via a metric filter.
  • 🔴 Live problem: Ticket from the service desk: "Since 14:10, some business customers say the statements page spins forever, but the dashboard is green." The API is a REST API with a VPC link to Fargate. 5XXError average is 0.002, p50 latency 180 ms, p99 latency 28,900 ms. Write down your investigation steps, the queries you would run and the two most likely causes.

Assignment

Add an observability.md to your portfolio repository for the Account Information API containing: the SLO; the access log format with a one-line justification per field; the list of alarms with thresholds, datapoints-to-alarm and missing-data policy; and your three saved Logs Insights queries. Include a short "what we deliberately do not log" section.

Solutions

Knowledge check

Interview questions

  • "How would you know if your API was failing for 5% of users?" — Alarm on error rate (Average of 5XXError, plus a 4xx spike alarm) and p99 latency rather than averages or raw counts; drill down with structured access logs and traces joined by a correlation/trace ID.
  • "What's the difference between Latency and IntegrationLatency in API Gateway?" — Latency is the full time API Gateway takes to respond; IntegrationLatency is just the backend portion. The gap is edge overhead: authorizers, WAF, mapping.
  • "Would you use the X-Ray SDK for a new .NET service?" — No; it is in maintenance mode. Use OpenTelemetry with the AWS X-Ray propagator and ID generator, export via ADOT, and keep X-Ray as the trace store.
  • "What should never be in API logs at a bank?" — Tokens, secrets, full request bodies, card data, and unnecessary personal data. Log opaque IDs, route templates and status codes; set retention and restrict access.
  • "How do you avoid alert fatigue?" — Page only on customer-impact symptoms via a composite alarm; route cause-level alarms to chat; use M-of-N datapoints; review every page in post-incident reviews.

Certification practice

Completion checklist

  • I can explain access logs vs execution logs vs metrics vs traces in two sentences
  • I wrote a JSON access log format and can justify each field
  • I know which observability features exist only on REST APIs
  • I can read Latency vs IntegrationLatency to locate a slowdown
  • My local .NET service writes JSON logs with trace IDs and exports OpenTelemetry spans
  • I created (or designed) a composite customer-impact alarm with M-of-N datapoints
  • I ran the cleanup steps and confirmed nothing tagged Project=cea-lab remains

Key takeaways

  • Alarm on what customers feel: error rates and latency percentiles, combined into one paging composite alarm.
  • Structured JSON access logs with extendedRequestId, resourcePath, latencies and errorType make most incidents a five-minute query.
  • REST APIs give you execution logs, X-Ray tracing, Firehose logs and WAF; HTTP APIs do not.
  • Instrument .NET with OpenTelemetry plus the X-Ray propagator; the X-Ray SDK is in maintenance mode.
  • Logs and traces hold personal data: minimise, mask, restrict and set retention.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page