Skip to Content
Cloud Engineering Academy
3. .NET Application Development on AWS

Logging and errors for .NET on AWS

Write structured JSON logs with correlation IDs from ASP.NET Core, ship them to CloudWatch Logs from Beanstalk, ECS and Lambda, query them with Logs Insights, and return safe RFC 9457 ProblemDetails from a global exception handler.

intermediate⏱ 2 h 🟢 FreeDVA-C03SAA-C03

Assumes: .NET 10 LTS · Serilog.AspNetCore 10.0.0 · AWS SDK for .NET v4 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain why structured JSON logs to stdout are the right default on every AWS compute service, and how each host ships them to CloudWatch Logs.
  • Configure Serilog (or plain ILogger) in ASP.NET Core to write one JSON object per line with consistent property names.
  • Give every request a correlation ID, accept a safe one from the caller or from the AWS X-Amzn-Trace-Id header, return it in the response and attach it to every log line.
  • Implement a global exception handler (IExceptionHandler) that returns RFC 9457 ProblemDetails, maps AWS SDK throttling and timeouts to the right status codes, and never leaks internals.
  • Query logs with CloudWatch Logs Insights, set retention, and keep personal data out of logs with pseudonymisation and CloudWatch Logs data protection.

Prerequisites

  • Module 3 overview and the Accounts API in labs/m03-accounts-api (this lesson walks through its real code)
  • Scaling .NET apps — why many identical copies make correlation essential
  • Tools: .NET 10 SDK or Docker; AWS CLI v2 with learn-sandbox for the CloudWatch parts

Why this matters in banking

A customer calls Harbourline Bank: "At 07:42 the app said something went wrong when I opened my savings account." The support agent has the customer's account ID and the time. The API runs as six Fargate tasks behind a load balancer, calls DynamoDB and a pseudonymisation service, and writes thousands of log lines per minute.

Without a correlation ID, someone greps through six log streams by timestamp and guesses. With one, the agent asks for the reference shown in the app's error screen (correlationId: 01a1eb9e-...) and a single Logs Insights query returns every line for that request, across every task, in order, including the exception and the DynamoDB error code.

Auditors ask related questions: Can you reconstruct what happened for a given request? Do your logs contain personal data, and for how long do you keep them? Can a caller see your stack traces? This lesson's habits answer all three.

The idea in plain language

A log is a diary the application writes as it works. Classic text logs (2026-11-18 07:42:01 ERROR Failed to load account ACC-...) are made for humans reading one server's file. In the cloud there are many copies, no files you can open, and you want to search logs like a database.

Structured logging writes each entry as a JSON object with named fields: {"@t":"...","@l":"Error","AccountId":"...","CorrelationId":"..."}. Tools can then filter, count and group on those fields.

A correlation ID is a reference number stamped on everything that happens for one request, the way a bank stamps a case number on every document in a complaint file.

ProblemDetails (RFC 9457) is a standard JSON shape for HTTP errors: type, title, status, detail, instance, plus your own extensions. Clients get a predictable error format; your internals stay private.

Analogy: structured logs are a well-kept case file in a filing system: every page has the case number, date and author in fixed boxes. Unstructured logs are a pile of handwritten notes. The ProblemDetails response is the polite letter to the customer, which gives the case number and what to do next but never photocopies the internal notes.

What problem CloudWatch Logs solves

Amazon CloudWatch Logs stores, searches and retains log events from AWS services and your applications. The vocabulary:

  • Log group — a container with one retention setting and one KMS key, usually one per application per environment (/hl-m03/accounts-api/ecs).
  • Log stream — a sequence of events from one source: one Fargate task, one Lambda execution environment, one EC2 instance file.
  • Log event — one line: timestamp + message (your JSON).
  • Logs Insights — a query language over log groups, billed per GB scanned.
  • Metric filters and subscription filters — turn matching lines into CloudWatch metrics, or stream them to Lambda, Kinesis Data Firehose or OpenSearch.

How each host gets your stdout into CloudWatch:

HostHow stdout reaches CloudWatchWhere to look
ECS on Fargateawslogs log driver in the task definition (or FireLens for routing elsewhere)The log group you name, stream prefix/container/task-id
LambdaBuilt in: everything written to stdout/stderr/aws/lambda/<function> by default, or a custom log group
Elastic BeanstalkPlatform agent streams instance log files when StreamLogs is enabled; the app's stdout lands in web.stdout.log/aws/elasticbeanstalk/<env>/var/log/web.stdout.log
EC2CloudWatch agent tails files or the journalWhatever you configure

Limitations to know: the default retention is never expire, which is both a cost trap and a data-protection problem. A single event is limited to 256 KB, so do not log whole payloads. Logs Insights queries scan data and cost money per GB, so narrow the time range and log group. CloudWatch is not a SIEM; for long-term security analytics you forward logs elsewhere (Module 11).

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
ILogger + Application Insights SDKILogger/Serilog writing JSON to stdout → CloudWatch LogsNo SDK call per log line; the platform collects stdout
Log Analytics workspace + KQLCloudWatch Logs Insights (its own query language), or OpenSearchPay per GB scanned per query, not per GB retained only
Application Insights operation_IdYour correlation ID + the X-Amzn-Trace-Id / X-Ray trace IDYou choose and propagate the ID yourself; X-Ray / OpenTelemetry adds distributed traces
App Service log streamingaws logs tail --followWorks across all streams of a group
Diagnostic settings retentionLog group retention (1 day … 10 years, or never)Default is never: always set it
UseExceptionHandler + ProblemDetailsExactly the same ASP.NET Core featuresNothing AWS-specific; the mapping of AWS SDK exceptions is

Architecture

Security boundary view. Two audiences see two different things. The caller sees a status code, a generic title, a correlation ID and nothing else. Operators see the full exception in CloudWatch Logs, which is protected by IAM (who can read the log group), encrypted at rest (optionally with your KMS key), retained for a defined period, and free of direct customer identifiers.

How the Accounts API does it

The module project's API (labs/m03-accounts-api) implements everything in this lesson. The relevant parts of Program.cs:

Program.cs (excerpt)
builder.Services.AddSerilog((services, logger) => logger
    .ReadFrom.Configuration(builder.Configuration)
    .ReadFrom.Services(services)
    .Enrich.FromLogContext()
    .Enrich.WithProperty("Application", "Harbourline.Accounts.Api")
    .WriteTo.Console(new RenderedCompactJsonFormatter()),
    preserveStaticLogger: true);

builder.Services.AddProblemDetails(o => o.CustomizeProblemDetails = ctx =>
{
    ctx.ProblemDetails.Instance = $"{ctx.HttpContext.Request.Method} {ctx.HttpContext.Request.Path}";
    ctx.ProblemDetails.Extensions["correlationId"] = CorrelationIdMiddleware.Get(ctx.HttpContext);
});
builder.Services.AddExceptionHandler<GlobalExceptionHandler>();

var app = builder.Build();

app.UseForwardedHeaders();
app.UseMiddleware<CorrelationIdMiddleware>();
app.UseSerilogRequestLogging(o =>
{
    o.GetLevel = (ctx, _, ex) =>
        ex is not null || ctx.Response.StatusCode >= 500 ? LogEventLevel.Error
        : ctx.Request.Path.StartsWithSegments("/health") ? LogEventLevel.Verbose
        : LogEventLevel.Information;
});
app.UseExceptionHandler();
app.UseStatusCodePages();

A real line it writes (from the lab's test run), reformatted for reading:

{
  "@t": "2026-11-17T22:26:45.9928372Z",
  "@m": "HTTP \"GET\" \"/v1/accounts/ACC-0000000001\" responded 500 in 1.2268 ms",
  "@i": "62d0885c",
  "@l": "Error",
  "@tr": "c709528ba268f118c04f4c68634a1e90",
  "@sp": "b0f66ecf9a1c250e",
  "RequestMethod": "GET",
  "RequestPath": "/v1/accounts/ACC-0000000001",
  "StatusCode": 500,
  "Elapsed": 1.226774,
  "SourceContext": "Serilog.AspNetCore.RequestLoggingMiddleware",
  "CorrelationId": "01a1eb9e-a867-7a7c-bd6d-535ce24626c0",
  "Application": "Harbourline.Accounts.Api"
}

@t is the timestamp, @m the rendered message, @i an event-type hash (the same for every line from the same message template, useful for grouping), @l the level (omitted for Information), and @tr/@sp the W3C trace and span IDs from Activity.Current. Everything else is a named property you can query.

Step-by-step implementation

Log JSON to stdout, nothing else

Use one console sink with a JSON formatter. Do not write log files inside containers and do not call the CloudWatch API from the app: the platform ships stdout for you, and it does so without blocking requests or needing extra IAM permissions.

Set levels in configuration

appsettings.json sets Information by default and Warning for chatty framework namespaces (Microsoft.AspNetCore, AWSSDK). Because the logger reads configuration, you can raise a namespace to Debug for an investigation through an environment variable such as Serilog__MinimumLevel__Override__Harbourline.Accounts.Api=Debug and a redeploy, without code changes.

Add the correlation middleware early

Register it right after forwarded headers, before request logging and exception handling, so that every later log line, including the exception handler's, carries the ID.

Map exceptions centrally

Implement IExceptionHandler. Map known failures (AWS throttling, your own timeouts) to specific status codes; everything else is a 500 with a generic title. Log the full exception once, here.

Configure retention and encryption

Create log groups in IaC with a retention period (the lab uses 7 days) and, for production, a customer managed KMS key. Never rely on the auto-created group with infinite retention.

Prepare queries and alarms

Save Logs Insights queries for "everything for one correlation ID" and "5xx by path". Create a metric filter on @l = "Error" and an alarm on it.

Correlation IDs in detail

Hosting/CorrelationId.cs from the lab:

Hosting/CorrelationId.cs
using System.Diagnostics;
using System.Text.RegularExpressions;
using Serilog.Context;

namespace Harbourline.Accounts.Api.Hosting;

public sealed partial class CorrelationIdMiddleware(RequestDelegate next)
{
    public const string HeaderName = "X-Correlation-ID";
    public const string AmznTraceHeader = "X-Amzn-Trace-Id";
    public const string ItemKey = "CorrelationId";

    // Only short, safe characters: an attacker must not be able to inject text into our logs.
    [GeneratedRegex("^[A-Za-z0-9-]{8,64}$", RegexOptions.CultureInvariant)]
    private static partial Regex SafeId();

    [GeneratedRegex("Root=([A-Za-z0-9-]{8,64})", RegexOptions.CultureInvariant)]
    private static partial Regex AmznRoot();

    public async Task InvokeAsync(HttpContext context)
    {
        var correlationId = Resolve(context.Request.Headers[HeaderName], context.Request.Headers[AmznTraceHeader]);

        context.Items[ItemKey] = correlationId;
        Activity.Current?.SetTag("correlation.id", correlationId);
        context.Response.OnStarting(() =>
        {
            context.Response.Headers[HeaderName] = correlationId;
            return Task.CompletedTask;
        });

        using (LogContext.PushProperty("CorrelationId", correlationId))
        {
            await next(context);
        }
    }

    public static string Resolve(string? header, string? amznTraceId)
    {
        if (header is not null && SafeId().IsMatch(header)) return header;
        if (amznTraceId is not null && AmznRoot().Match(amznTraceId) is { Success: true } m) return m.Groups[1].Value;
        return Guid.CreateVersion7().ToString();
    }

    public static string? Get(HttpContext context) => context.Items[ItemKey] as string;
}
  1. HeaderName — the header a calling system (the mobile backend, another microservice) can send so its ID flows through.
  2. SafeId() — a source-generated regex allowing 8–64 letters, digits and hyphens. Without validation, a caller could send a header with newlines or JSON fragments and forge log lines (log injection). JSON formatters escape strings, but validation also stops absurdly long values.
  3. AmznRoot() — the ALB and API Gateway add X-Amzn-Trace-Id: Root=1-67a1b2c3-.... Reusing its root means the same ID appears in ALB access logs and X-Ray, so you can join them.
  4. Guid.CreateVersion7() — the fallback. UUID v7 is time-ordered, so IDs sort roughly by time (the same reason CONVENTIONS.md uses v7 for transaction IDs).
  5. context.Items[...] — stores the ID for the rest of the request so the ProblemDetails customisation can read it.
  6. Activity.Current?.SetTag(...) — adds it to the OpenTelemetry/Activity span, so traces and logs share it.
  7. Response.OnStarting — sets the response header just before headers are sent. Setting it immediately would fail if a later component cleared headers (the exception handler does).
  8. LogContext.PushProperty — Serilog's ambient context: every log event written during await next(context) gets a CorrelationId property, including events from deep inside the SDK wrappers or the exception handler. Enrich.FromLogContext() in Program.cs is what reads it.

The global exception handler

Hosting/GlobalExceptionHandler.cs:

Hosting/GlobalExceptionHandler.cs
public sealed class GlobalExceptionHandler(
    IProblemDetailsService problemDetails,
    ILogger<GlobalExceptionHandler> logger) : Microsoft.AspNetCore.Diagnostics.IExceptionHandler
{
    private static readonly HashSet<string> ThrottlingCodes = new(StringComparer.Ordinal)
    {
        "ThrottlingException", "ProvisionedThroughputExceededException", "RequestLimitExceeded", "TooManyRequestsException",
    };

    public async ValueTask<bool> TryHandleAsync(HttpContext httpContext, Exception exception, CancellationToken cancellationToken)
    {
        var (status, title) = exception switch
        {
            AmazonServiceException aws when ThrottlingCodes.Contains(aws.ErrorCode ?? string.Empty)
                => (StatusCodes.Status503ServiceUnavailable, "The service is busy. Please retry shortly."),
            OperationCanceledException when !httpContext.RequestAborted.IsCancellationRequested
                => (StatusCodes.Status504GatewayTimeout, "A downstream dependency did not respond in time."),
            _ => (StatusCodes.Status500InternalServerError, "An unexpected error occurred."),
        };

        logger.LogError(exception, "Unhandled exception; returning {StatusCode}", status);

        httpContext.Response.StatusCode = status;
        if (status == StatusCodes.Status503ServiceUnavailable)
        {
            httpContext.Response.Headers.RetryAfter = "2";
        }

        return await problemDetails.TryWriteAsync(new ProblemDetailsContext
        {
            HttpContext = httpContext,
            Exception = exception,
            ProblemDetails = new ProblemDetails { Status = status, Title = title },
        });
    }
}
  1. The fully qualified Microsoft.AspNetCore.Diagnostics.IExceptionHandler is deliberate: the AWS SDK v4 core assembly also defines an Amazon.Runtime.IExceptionHandler, and with both namespaces imported the short name is ambiguous (compiler error CS0104). A real trap when you add the SDK to a web project.
  2. ThrottlingCodes — AWS services signal throttling with these error codes. The SDK's standard retry mode has already retried with back-off; if it still fails, the honest answer to the client is 503 + Retry-After, not 500.
  3. OperationCanceledException when !RequestAborted... — the repository enforces a per-call time budget with a CancellationToken (in SDK v4, ClientConfig.Timeout does not bound async calls). If our budget expired while the client is still connected, it is a 504. If the client disconnected, the exception falls through to the default and nobody reads the response anyway.
  4. logger.LogError(exception, ...) — the only place the stack trace is written. It carries the correlation ID from LogContext.
  5. ProblemDetails { Status, Title } — no Detail with the exception message. AWS error messages include table names and ARNs; the lab's test Throttling_from_aws_becomes_503_with_retry_after_and_no_internal_details asserts that the table name never reaches the response.
  6. IProblemDetailsService.TryWriteAsync — runs the CustomizeProblemDetails callback from Program.cs, which adds instance and correlationId, and fills type with the RFC 9110 section link for the status.

What the client receives:

HTTP/1.1 503 Service Unavailable
Content-Type: application/problem+json
Retry-After: 2
X-Correlation-ID: 01a1eb9e-a809-7285-8969-feea1953715c

{
  "type": "https://tools.ietf.org/html/rfc9110#section-15.6.4",
  "title": "The service is busy. Please retry shortly.",
  "status": 503,
  "instance": "GET /v1/accounts/ACC-0000000001",
  "correlationId": "01a1eb9e-a809-7285-8969-feea1953715c"
}

Expected errors are not exceptions. Validation failures and "not found" are returned directly by the endpoints with TypedResults.Problem(...) (400 and 404). Exceptions are for the unexpected; using them for control flow makes logs noisy and slows the API.

Keeping personal data out of logs

UK GDPR's data-minimisation principle applies to logs. Logs are copied, retained and read by many people, so the safest personal data in a log is none.

  • Never log names, emails, phone numbers, full IBANs, card data (PCI DSS forbids logging full PANs and never allows sensitive authentication data), tokens or secrets.
  • Pseudonymise identifiers you need for correlation. The Accounts API logs CustomerRef, an HMAC-SHA256 of the customer ID with a secret key from Parameter Store or Secrets Manager (Security/Pseudonymisation.cs). The same customer always gets the same cref_… value, so support can follow one customer's journey, but the value cannot be reversed without the key. It is still personal data under UK GDPR (pseudonymised, not anonymised), so it still needs retention limits and access control.
  • Use message templates, not string interpolation. LogInformation("Listed {AccountCount} accounts for customer {CustomerRef}", count, cref) keeps fields separate and searchable. LogInformation($"...{customerId}") bakes values into the message and defeats both querying and redaction.
  • Add a safety net. CloudWatch Logs data protection policies can detect and mask managed data identifiers (for example email addresses, credit card numbers) in a log group, with audit findings. Treat it as a net, not as the primary control.

AWS Console instructions

  • CloudWatch → Logs → Log groups → /hl-m03/accounts-api/ecs — check Retention (should be 1 week in the lab) and KMS key ID. Actions → Edit retention setting changes it.
  • CloudWatch → Logs → Logs Insights — select the log group, set a narrow time range, paste a query from the CLI section, Run query. Save stores it for the team.
  • CloudWatch → Logs → Log groups → group → Metric filters → Create metric filter — pattern { $.@l = "Error" }, metric namespace Harbourline/AccountsApi, metric name ErrorCount, value 1.
  • CloudWatch → Logs → Log groups → group → Data protection → Create policy — choose managed data identifiers (for example EmailAddress, CreditCardNumber) and an audit destination.
  • Lambda → function → Configuration → Monitoring and operations tools → Logging configuration — log format JSON, application and system log levels, and the log group.

AWS CLI

# Follow the API's logs live (all tasks), JSON lines as written by the app
aws logs tail /hl-m03/accounts-api/ecs --follow --since 10m --profile learn-sandbox

# Only errors, using a filter pattern on the JSON field
aws logs tail /hl-m03/accounts-api/ecs --since 1h --filter-pattern '{ $.@l = "Error" }' --profile learn-sandbox

# Set retention on a log group (the default is never expire)
aws logs put-retention-policy --log-group-name /hl-m03/accounts-api/ecs --retention-in-days 7 --profile learn-sandbox

# Logs Insights: everything for one correlation id, oldest first
QUERY_ID=$(aws logs start-query \
  --log-group-name /hl-m03/accounts-api/ecs \
  --start-time $(date -d '-1 hour' +%s) --end-time $(date +%s) \
  --query-string 'fields @timestamp, @l, @m, StatusCode, SourceContext
                  | filter CorrelationId = "01a1eb9e-a867-7a7c-bd6d-535ce24626c0"
                  | sort @timestamp asc' \
  --query queryId --output text --profile learn-sandbox)
aws logs get-query-results --query-id "$QUERY_ID" --profile learn-sandbox

# Logs Insights: 5xx count and p95 latency per path in 5-minute bins
aws logs start-query --log-group-name /hl-m03/accounts-api/ecs \
  --start-time $(date -d '-3 hours' +%s) --end-time $(date +%s) \
  --query-string 'filter SourceContext = "Serilog.AspNetCore.RequestLoggingMiddleware"
                  | stats count(*) as requests, sum(StatusCode >= 500) as errors, pct(Elapsed, 95) as p95ms by RequestPath, bin(5m)' \
  --profile learn-sandbox

# Metric filter: one ErrorCount data point per error line
aws logs put-metric-filter --log-group-name /hl-m03/accounts-api/ecs \
  --filter-name api-errors --filter-pattern '{ $.@l = "Error" }' \
  --metric-transformations metricName=ErrorCount,metricNamespace=Harbourline/AccountsApi,metricValue=1,defaultValue=0 \
  --profile learn-sandbox

# Lambda: JSON log format and levels for a function
aws lambda update-function-configuration --function-name hl-m03-accounts-api \
  --logging-config LogFormat=JSON,ApplicationLogLevel=INFO,SystemLogLevel=WARN \
  --profile learn-sandbox

date -d is GNU syntax and works in Git Bash. In PowerShell use [DateTimeOffset]::UtcNow.AddHours(-1).ToUnixTimeSeconds().

Infrastructure as Code

LoggingConstruct.cs
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.Logs;
using Constructs;

namespace Harbourline.Accounts.Infra;

public sealed class LoggingConstruct : Construct
{
    public LogGroup LogGroup { get; }

    public LoggingConstruct(Construct scope, string id, IKey logKey) : base(scope, id)
    {
        // Explicit group: retention and KMS key are decided here, not by whoever writes first.
        LogGroup = new LogGroup(this, "ApiLogs", new LogGroupProps
        {
            LogGroupName = "/hl-m03/accounts-api/ecs",
            Retention = RetentionDays.ONE_MONTH,   // production: per your retention schedule
            EncryptionKey = logKey,                // key policy must allow logs.<region>.amazonaws.com
            RemovalPolicy = RemovalPolicy.RETAIN,  // logs are evidence: keep on stack deletion
        });

        var errors = new MetricFilter(this, "ErrorFilter", new MetricFilterProps
        {
            LogGroup = LogGroup,
            FilterPattern = FilterPattern.StringValue("$.@l", "=", "Error"),
            MetricNamespace = "Harbourline/AccountsApi",
            MetricName = "ErrorCount",
            MetricValue = "1",
            DefaultValue = 0,
        });

        _ = new Alarm(this, "ErrorAlarm", new AlarmProps
        {
            AlarmDescription = "More than 5 error log lines in 5 minutes",
            Metric = errors.Metric(new MetricOptions { Statistic = "Sum", Period = Duration.Minutes(5) }),
            Threshold = 5,
            EvaluationPeriods = 1,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
            TreatMissingData = TreatMissingData.NOT_BREACHING,
        });

        _ = new QueryDefinition(this, "ByCorrelationId", new QueryDefinitionProps
        {
            QueryDefinitionName = "accounts-api/by-correlation-id",
            QueryString = new QueryString(new QueryStringProps
            {
                Fields = ["@timestamp", "@l", "@m", "StatusCode", "SourceContext"],
                FilterStatements = new[] { "CorrelationId = \"REPLACE_ME\"" },
                Sort = "@timestamp asc",
            }),
            LogGroups = [LogGroup],
        });
    }
}

Complete code

A minimal, self-contained API that uses only the built-in ILogger JSON console formatter (no Serilog), for teams that prefer fewer dependencies. Run it with dotnet run and call /demo/fail and /demo/slow.

Harbourline.Logging.Demo.csproj
<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
  </PropertyGroup>
</Project>
Program.cs
using System.Diagnostics;
using Microsoft.AspNetCore.Diagnostics;
using Microsoft.AspNetCore.Mvc;

var builder = WebApplication.CreateBuilder(args);

// 1. Built-in JSON console logs: one JSON object per line, scopes included.
builder.Logging.ClearProviders();
builder.Logging.AddJsonConsole(o =>
{
    o.IncludeScopes = true;                 // correlation id arrives as a scope
    o.UseUtcTimestamp = true;
    o.TimestampFormat = "yyyy-MM-ddTHH:mm:ss.fffZ";
    o.JsonWriterOptions = new() { Indented = false };
});

// 2. ProblemDetails for every error response, stamped with the correlation id.
builder.Services.AddProblemDetails(o => o.CustomizeProblemDetails = ctx =>
    ctx.ProblemDetails.Extensions["correlationId"] = ctx.HttpContext.TraceIdentifier);
builder.Services.AddExceptionHandler<DemoExceptionHandler>();

var app = builder.Build();

// 3. Correlation: reuse a safe incoming id, else the W3C trace id; push it as a logging scope.
app.Use(async (ctx, next) =>
{
    var incoming = ctx.Request.Headers["X-Correlation-ID"].ToString();
    var id = incoming.Length is >= 8 and <= 64 && incoming.All(c => char.IsAsciiLetterOrDigit(c) || c == '-')
        ? incoming
        : Activity.Current?.TraceId.ToString() ?? Guid.CreateVersion7().ToString();

    ctx.TraceIdentifier = id;
    ctx.Response.Headers["X-Correlation-ID"] = id;
    var logger = ctx.RequestServices.GetRequiredService<ILoggerFactory>().CreateLogger("Correlation");
    using (logger.BeginScope(new Dictionary<string, object> { ["CorrelationId"] = id }))
    {
        await next(ctx);
    }
});

app.UseExceptionHandler();
app.UseStatusCodePages();

app.MapGet("/demo/ok", (ILogger<Program> log) =>
{
    log.LogInformation("Balance enquiry served for {AccountId}", "ACC-0000000001");
    return Results.Ok(new { balance = 1250.00m, currency = "GBP" });
});

app.MapGet("/demo/fail", () =>
{
    // Simulates a bug that reveals an internal detail in its message.
    throw new InvalidOperationException("Connection to core-banking-db-07.internal failed");
});

app.MapGet("/demo/slow", async (CancellationToken ct) =>
{
    using var budget = CancellationTokenSource.CreateLinkedTokenSource(ct);
    budget.CancelAfter(TimeSpan.FromMilliseconds(200));       // our time budget for the dependency
    await Task.Delay(TimeSpan.FromSeconds(2), budget.Token);   // the "dependency"
    return Results.Ok();
});

app.MapGet("/demo/missing/{id}", (string id) =>
    Results.Problem(statusCode: 404, title: "Account not found", detail: $"No account {id} is visible to you."));

app.Run();

internal sealed class DemoExceptionHandler(IProblemDetailsService problems, ILogger<DemoExceptionHandler> log) : IExceptionHandler
{
    public async ValueTask<bool> TryHandleAsync(HttpContext ctx, Exception ex, CancellationToken ct)
    {
        var (status, title) = ex switch
        {
            OperationCanceledException when !ctx.RequestAborted.IsCancellationRequested
                => (StatusCodes.Status504GatewayTimeout, "A downstream dependency did not respond in time."),
            _ => (StatusCodes.Status500InternalServerError, "An unexpected error occurred."),
        };

        log.LogError(ex, "Unhandled exception on {Path}; returning {Status}", ctx.Request.Path.Value, status);
        ctx.Response.StatusCode = status;
        return await problems.TryWriteAsync(new ProblemDetailsContext
        {
            HttpContext = ctx,
            Exception = ex,
            ProblemDetails = new ProblemDetails { Status = status, Title = title },
        });
    }
}

Code explained line by line

  1. builder.Logging.ClearProviders() then AddJsonConsole — removes the default text console provider and writes JSON instead. On Lambda you can alternatively let the runtime format logs as JSON with the function's logging configuration.
  2. IncludeScopes = true — the JSON includes a Scopes array; the correlation dictionary appears there for every line written inside the using block.
  3. UseUtcTimestamp and an ISO format — consistent UTC timestamps across hosts; CloudWatch also records its own ingestion time.
  4. AddProblemDetails with CustomizeProblemDetails — every ProblemDetails response (from the exception handler, UseStatusCodePages or Results.Problem) gets correlationId.
  5. The inline middleware — validates the incoming header (length 8–64, letters, digits, hyphen) before trusting it; otherwise uses the W3C trace ID of the current Activity, which ASP.NET Core creates per request.
  6. ctx.TraceIdentifier = id — ASP.NET Core's built-in per-request ID is reused as our correlation ID, so framework logs that include it match ours.
  7. logger.BeginScope(...) — ILogger's equivalent of Serilog's LogContext.PushProperty.
  8. UseExceptionHandler() without a path — uses the registered IExceptionHandler and the ProblemDetails service (ASP.NET Core 8+ behaviour).
  9. UseStatusCodePages() — turns empty 404/405 responses (no matching route) into ProblemDetails too.
  10. /demo/fail — the exception message contains a host name; the handler logs it but the response says only "An unexpected error occurred."
  11. /demo/slow — a linked token with CancelAfter is the pattern for a time budget; the resulting OperationCanceledException becomes a 504.
  12. /demo/missing/{id} — an expected outcome returned directly, no exception, no error-level log.

Security considerations

⚠ Learning shortcut — not for production

The lab log groups use 7-day retention, the AWS-owned encryption key and RemovalPolicy.DESTROY, so teardown leaves nothing behind. In production, set retention from the bank's records-retention schedule (agreed with compliance), encrypt with a customer managed KMS key whose policy limits who can decrypt, keep RemovalPolicy.RETAIN, and forward security-relevant logs to a central, write-once log archive account (Module 11).

  • Access to logs is access to data. Grant logs:GetLogEvents, logs:FilterLogEvents and logs:StartQuery per log group to support roles, not logs:* on *.
  • Validate everything you log from the request. Headers, query strings and bodies are attacker-controlled. Validate correlation IDs; never log raw bodies.
  • Never return exception text. Stack traces, SQL, table names and ARNs help an attacker map your system. Set ASPNETCORE_ENVIRONMENT=Production so the developer exception page is never used.
  • Health-check noise hides attacks. The lab logs /health requests at Verbose so they do not drown real traffic.

Banking use case

Harbourline's support process for API errors:

  1. The mobile app shows "Something went wrong. Reference: 01a1eb9e-…" using the correlationId from the ProblemDetails body.
  2. Level-2 support runs the saved Logs Insights query accounts-api/by-correlation-id with that reference and a 30-minute window around the reported time. They see the request line, the Unhandled exception; returning 503 line with ProvisionedThroughputExceededException, and the CustomerRef (not the customer ID).
  3. The ErrorCount metric filter alarm had already paged the on-call engineer, whose dashboard shows the throttling spike.
  4. Logs are retained for the period in the bank's retention schedule, encrypted with a KMS key that only the observability and security teams can use, and subscription-filtered to the central log archive.

Common mistakes

  • Logging with string interpolation → values are baked into the message, not queryable or maskable → use message templates.
  • Writing log files inside the container → lost when the task stops, invisible to CloudWatch → stdout only.
  • No retention set → logs kept forever, growing cost and a GDPR storage-limitation problem → set retention in IaC.
  • Logging the exception at every layer → the same stack trace five times → log once, at the boundary (the exception handler).
  • Returning ex.Message in detail → leaks internals → generic titles for 5xx; precise details only for 4xx validation errors.
  • Trusting X-Correlation-ID blindly → log injection, enormous values → validate format and length.
  • Using the static Log.Logger with several hosts in one process → in integration tests running in parallel, hosts overwrite each other's logger and log assertions fail randomly → AddSerilog(..., preserveStaticLogger: true) as in the lab (this exact flaky test was fixed in the module project).

Troubleshooting

SymptomLikely causeFix
No logs in CloudWatch from ECSTask execution role lacks logs:CreateLogStream/PutLogEvents, or wrong log group nameCheck the execution role; CDK's LogDrivers.AwsLogs grants it automatically
Lambda logs missingFunction role lacks the basic execution policy, or a custom log group in another nameAttach AWSLambdaBasicExecutionRole; check the function's logging config
Beanstalk: only nginx logs, no app logsLog streaming disabledSet aws:elasticbeanstalk:cloudwatch:logs StreamLogs=true; app stdout is in web.stdout.log
Logs Insights shows fields as plain textThe line is not valid JSON (e.g. a text log from another provider in the same stream)One JSON formatter only; ClearProviders() when using the built-in JSON console
CorrelationId missing on some linesLines written before the middleware or outside the request (startup, background work)Register the middleware early; push your own ID in background jobs
Error responses are empty bodiesAddProblemDetails() not registered, or UseStatusCodePages() missingRegister both, as in Program.cs
Clients see the developer exception pageASPNETCORE_ENVIRONMENT=Development in a deployed environmentSet Production in the task definition / environment properties
Log-based test assertions fail randomly in CIStatic Serilog logger shared across parallel test hostspreserveStaticLogger: true

Cost and cleanup

CloudWatch Logs bills for ingestion, storage and queries. In us-east-1 the Standard log class costs about USD 0.50 per GB ingested and USD 0.03 per GB-month stored, and Logs Insights about USD 0.005 per GB scanned ⚠ VERIFY current prices on the CloudWatch pricing page. The AWS Free Tier includes 5 GB of log data per month (ingestion, archive storage and Insights scanning) ⚠ VERIFY under the current free plan. The lab's traffic produces a few megabytes, so it is effectively free.

Cost traps: debug logging left on in production, logging whole request bodies, health checks at Information level, never-expiring retention, and Insights queries over 30 days of every log group.

# Remove resources created only for this lesson
aws logs delete-metric-filter --log-group-name /hl-m03/accounts-api/ecs --filter-name api-errors --profile learn-sandbox
aws logs delete-log-group --log-group-name /hl-m03/accounts-api/ecs --profile learn-sandbox   # if the stack is already destroyed

# Find log groups without retention (never expire) — fix or delete them
aws logs describe-log-groups --query "logGroups[?retentionInDays==null].logGroupName" --profile learn-sandbox

Exercises

  • 🟢 Guided: Run the Accounts API locally (dotnet run --project src/Harbourline.Accounts.Api --launch-profile http). Call /v1/accounts/ACC-0000000001 with -H "X-Correlation-ID: support-case-000123", then with a header containing a space, then with none. Compare the X-Correlation-ID response headers and the CorrelationId in the console logs.
  • 🟡 Challenge: Extend GlobalExceptionHandler so that a DynamoDB ResourceNotFoundException (table missing, a configuration error) returns 503 with title "The service is not available" and logs at Critical. Write an integration test in the style of Throttling_from_aws_becomes_503_with_retry_after_and_no_internal_details.
  • 🔴 Live problem: A penetration test report says: "Error responses from /v1/customers/{id}/accounts disclose the DynamoDB table name and AWS account ID." Logs show detail fields like Requested resource not found: Table: arn:aws:dynamodb:.... Find the likely cause in a codebase that otherwise looks like the lab, and fix it.

Assignment

Add a docs/observability.md to your portfolio repository describing the Accounts API's logging contract: the JSON fields and their meaning, the correlation ID rules, the ProblemDetails shapes per status code, the log levels per namespace, the retention period and its justification, what is never logged, and three saved Logs Insights queries.

Solutions

Knowledge check

Interview questions

  • "How do you trace one failed request across many containers?" — A correlation ID per request, accepted from callers when valid or taken from X-Amzn-Trace-Id, pushed into the logging context, returned in a response header and in ProblemDetails. Then one Logs Insights query on that field. For cross-service timing, add OpenTelemetry/X-Ray traces with the same ID as a tag.
  • "What goes in an error response from a banking API?" — RFC 9457 ProblemDetails: status, generic title for 5xx, precise detail for 4xx validation, instance, correlation ID. Never stack traces, exception messages, resource names or personal data of other customers.
  • "How do you keep personal data out of logs?" — Message templates, no payload logging, pseudonymised identifiers, code review rules, tests that assert raw identifiers are absent, and CloudWatch Logs data protection policies as a safety net; plus retention and access control on the log groups.
  • "Serilog or built-in ILogger?" — Both write structured JSON. Serilog brings enrichers, request logging and configuration-driven sinks; the built-in JSON console has no dependency. Either way the app logs through ILogger<T>.

Certification practice

Completion checklist

  • I can explain how ECS, Lambda and Beanstalk each ship stdout to CloudWatch Logs
  • I can read a Serilog compact JSON line and name each @ field
  • I can explain the correlation ID precedence and why the header is validated
  • I can map throttling, timeouts and bugs to 503, 504 and 500 ProblemDetails
  • I ran the guided exercise and saw the correlation header behaviour
  • I checked that my log groups have retention set

Key takeaways

  • Log structured JSON to stdout; let the platform ship it to CloudWatch Logs.
  • One correlation ID per request, validated, reused from X-Amzn-Trace-Id when present, on every log line and in every error response.
  • A global exception handler logs once and returns ProblemDetails without internals: 503 for throttling, 504 for timeouts, 500 for bugs.
  • Personal data stays out of logs: templates, pseudonymised references, retention, access control, data protection policies.
  • Set retention on every log group; never-expire is the default and the most common cost trap.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page