Logging and errors for .NET on AWS
Write structured JSON logs with correlation IDs from ASP.NET Core, ship them to CloudWatch Logs from Beanstalk, ECS and Lambda, query them with Logs Insights, and return safe RFC 9457 ProblemDetails from a global exception handler.
Assumes: .NET 10 LTS · Serilog.AspNetCore 10.0.0 · AWS SDK for .NET v4 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain why structured JSON logs to stdout are the right default on every AWS compute service, and how each host ships them to CloudWatch Logs.
- Configure Serilog (or plain
ILogger) in ASP.NET Core to write one JSON object per line with consistent property names. - Give every request a correlation ID, accept a safe one from the caller or from the AWS
X-Amzn-Trace-Idheader, return it in the response and attach it to every log line. - Implement a global exception handler (
IExceptionHandler) that returns RFC 9457 ProblemDetails, maps AWS SDK throttling and timeouts to the right status codes, and never leaks internals. - Query logs with CloudWatch Logs Insights, set retention, and keep personal data out of logs with pseudonymisation and CloudWatch Logs data protection.
Prerequisites
- Module 3 overview and the Accounts API in
labs/m03-accounts-api(this lesson walks through its real code) - Scaling .NET apps — why many identical copies make correlation essential
- Tools: .NET 10 SDK or Docker; AWS CLI v2 with
learn-sandboxfor the CloudWatch parts
Why this matters in banking
A customer calls Harbourline Bank: "At 07:42 the app said something went wrong when I opened my savings account." The support agent has the customer's account ID and the time. The API runs as six Fargate tasks behind a load balancer, calls DynamoDB and a pseudonymisation service, and writes thousands of log lines per minute.
Without a correlation ID, someone greps through six log streams by timestamp and guesses. With one, the agent asks for the reference shown in the app's error screen (correlationId: 01a1eb9e-...) and a single Logs Insights query returns every line for that request, across every task, in order, including the exception and the DynamoDB error code.
Auditors ask related questions: Can you reconstruct what happened for a given request? Do your logs contain personal data, and for how long do you keep them? Can a caller see your stack traces? This lesson's habits answer all three.
The idea in plain language
A log is a diary the application writes as it works. Classic text logs (2026-11-18 07:42:01 ERROR Failed to load account ACC-...) are made for humans reading one server's file. In the cloud there are many copies, no files you can open, and you want to search logs like a database.
Structured logging writes each entry as a JSON object with named fields: {"@t":"...","@l":"Error","AccountId":"...","CorrelationId":"..."}. Tools can then filter, count and group on those fields.
A correlation ID is a reference number stamped on everything that happens for one request, the way a bank stamps a case number on every document in a complaint file.
ProblemDetails (RFC 9457) is a standard JSON shape for HTTP errors: type, title, status, detail, instance, plus your own extensions. Clients get a predictable error format; your internals stay private.
Analogy: structured logs are a well-kept case file in a filing system: every page has the case number, date and author in fixed boxes. Unstructured logs are a pile of handwritten notes. The ProblemDetails response is the polite letter to the customer, which gives the case number and what to do next but never photocopies the internal notes.
What problem CloudWatch Logs solves
Amazon CloudWatch Logs stores, searches and retains log events from AWS services and your applications. The vocabulary:
- Log group — a container with one retention setting and one KMS key, usually one per application per environment (
/hl-m03/accounts-api/ecs). - Log stream — a sequence of events from one source: one Fargate task, one Lambda execution environment, one EC2 instance file.
- Log event — one line: timestamp + message (your JSON).
- Logs Insights — a query language over log groups, billed per GB scanned.
- Metric filters and subscription filters — turn matching lines into CloudWatch metrics, or stream them to Lambda, Kinesis Data Firehose or OpenSearch.
How each host gets your stdout into CloudWatch:
| Host | How stdout reaches CloudWatch | Where to look |
|---|---|---|
| ECS on Fargate | awslogs log driver in the task definition (or FireLens for routing elsewhere) | The log group you name, stream prefix/container/task-id |
| Lambda | Built in: everything written to stdout/stderr | /aws/lambda/<function> by default, or a custom log group |
| Elastic Beanstalk | Platform agent streams instance log files when StreamLogs is enabled; the app's stdout lands in web.stdout.log | /aws/elasticbeanstalk/<env>/var/log/web.stdout.log |
| EC2 | CloudWatch agent tails files or the journal | Whatever you configure |
Limitations to know: the default retention is never expire, which is both a cost trap and a data-protection problem. A single event is limited to 256 KB, so do not log whole payloads. Logs Insights queries scan data and cost money per GB, so narrow the time range and log group. CloudWatch is not a SIEM; for long-term security analytics you forward logs elsewhere (Module 11).
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
ILogger + Application Insights SDK | ILogger/Serilog writing JSON to stdout → CloudWatch Logs | No SDK call per log line; the platform collects stdout |
| Log Analytics workspace + KQL | CloudWatch Logs Insights (its own query language), or OpenSearch | Pay per GB scanned per query, not per GB retained only |
Application Insights operation_Id | Your correlation ID + the X-Amzn-Trace-Id / X-Ray trace ID | You choose and propagate the ID yourself; X-Ray / OpenTelemetry adds distributed traces |
| App Service log streaming | aws logs tail --follow | Works across all streams of a group |
| Diagnostic settings retention | Log group retention (1 day … 10 years, or never) | Default is never: always set it |
UseExceptionHandler + ProblemDetails | Exactly the same ASP.NET Core features | Nothing AWS-specific; the mapping of AWS SDK exceptions is |
Architecture
Security boundary view. Two audiences see two different things. The caller sees a status code, a generic title, a correlation ID and nothing else. Operators see the full exception in CloudWatch Logs, which is protected by IAM (who can read the log group), encrypted at rest (optionally with your KMS key), retained for a defined period, and free of direct customer identifiers.
How the Accounts API does it
The module project's API (labs/m03-accounts-api) implements everything in this lesson. The relevant parts of Program.cs:
builder.Services.AddSerilog((services, logger) => logger
.ReadFrom.Configuration(builder.Configuration)
.ReadFrom.Services(services)
.Enrich.FromLogContext()
.Enrich.WithProperty("Application", "Harbourline.Accounts.Api")
.WriteTo.Console(new RenderedCompactJsonFormatter()),
preserveStaticLogger: true);
builder.Services.AddProblemDetails(o => o.CustomizeProblemDetails = ctx =>
{
ctx.ProblemDetails.Instance = $"{ctx.HttpContext.Request.Method} {ctx.HttpContext.Request.Path}";
ctx.ProblemDetails.Extensions["correlationId"] = CorrelationIdMiddleware.Get(ctx.HttpContext);
});
builder.Services.AddExceptionHandler<GlobalExceptionHandler>();
var app = builder.Build();
app.UseForwardedHeaders();
app.UseMiddleware<CorrelationIdMiddleware>();
app.UseSerilogRequestLogging(o =>
{
o.GetLevel = (ctx, _, ex) =>
ex is not null || ctx.Response.StatusCode >= 500 ? LogEventLevel.Error
: ctx.Request.Path.StartsWithSegments("/health") ? LogEventLevel.Verbose
: LogEventLevel.Information;
});
app.UseExceptionHandler();
app.UseStatusCodePages();A real line it writes (from the lab's test run), reformatted for reading:
{
"@t": "2026-11-17T22:26:45.9928372Z",
"@m": "HTTP \"GET\" \"/v1/accounts/ACC-0000000001\" responded 500 in 1.2268 ms",
"@i": "62d0885c",
"@l": "Error",
"@tr": "c709528ba268f118c04f4c68634a1e90",
"@sp": "b0f66ecf9a1c250e",
"RequestMethod": "GET",
"RequestPath": "/v1/accounts/ACC-0000000001",
"StatusCode": 500,
"Elapsed": 1.226774,
"SourceContext": "Serilog.AspNetCore.RequestLoggingMiddleware",
"CorrelationId": "01a1eb9e-a867-7a7c-bd6d-535ce24626c0",
"Application": "Harbourline.Accounts.Api"
}@t is the timestamp, @m the rendered message, @i an event-type hash (the same for every line from the same message template, useful for grouping), @l the level (omitted for Information), and @tr/@sp the W3C trace and span IDs from Activity.Current. Everything else is a named property you can query.
Step-by-step implementation
Log JSON to stdout, nothing else
Use one console sink with a JSON formatter. Do not write log files inside containers and do not call the CloudWatch API from the app: the platform ships stdout for you, and it does so without blocking requests or needing extra IAM permissions.
Set levels in configuration
appsettings.json sets Information by default and Warning for chatty framework namespaces (Microsoft.AspNetCore, AWSSDK). Because the logger reads configuration, you can raise a namespace to Debug for an investigation through an environment variable such as Serilog__MinimumLevel__Override__Harbourline.Accounts.Api=Debug and a redeploy, without code changes.
Add the correlation middleware early
Register it right after forwarded headers, before request logging and exception handling, so that every later log line, including the exception handler's, carries the ID.
Map exceptions centrally
Implement IExceptionHandler. Map known failures (AWS throttling, your own timeouts) to specific status codes; everything else is a 500 with a generic title. Log the full exception once, here.
Configure retention and encryption
Create log groups in IaC with a retention period (the lab uses 7 days) and, for production, a customer managed KMS key. Never rely on the auto-created group with infinite retention.
Prepare queries and alarms
Save Logs Insights queries for "everything for one correlation ID" and "5xx by path". Create a metric filter on @l = "Error" and an alarm on it.
Correlation IDs in detail
Hosting/CorrelationId.cs from the lab:
using System.Diagnostics;
using System.Text.RegularExpressions;
using Serilog.Context;
namespace Harbourline.Accounts.Api.Hosting;
public sealed partial class CorrelationIdMiddleware(RequestDelegate next)
{
public const string HeaderName = "X-Correlation-ID";
public const string AmznTraceHeader = "X-Amzn-Trace-Id";
public const string ItemKey = "CorrelationId";
// Only short, safe characters: an attacker must not be able to inject text into our logs.
[GeneratedRegex("^[A-Za-z0-9-]{8,64}$", RegexOptions.CultureInvariant)]
private static partial Regex SafeId();
[GeneratedRegex("Root=([A-Za-z0-9-]{8,64})", RegexOptions.CultureInvariant)]
private static partial Regex AmznRoot();
public async Task InvokeAsync(HttpContext context)
{
var correlationId = Resolve(context.Request.Headers[HeaderName], context.Request.Headers[AmznTraceHeader]);
context.Items[ItemKey] = correlationId;
Activity.Current?.SetTag("correlation.id", correlationId);
context.Response.OnStarting(() =>
{
context.Response.Headers[HeaderName] = correlationId;
return Task.CompletedTask;
});
using (LogContext.PushProperty("CorrelationId", correlationId))
{
await next(context);
}
}
public static string Resolve(string? header, string? amznTraceId)
{
if (header is not null && SafeId().IsMatch(header)) return header;
if (amznTraceId is not null && AmznRoot().Match(amznTraceId) is { Success: true } m) return m.Groups[1].Value;
return Guid.CreateVersion7().ToString();
}
public static string? Get(HttpContext context) => context.Items[ItemKey] as string;
}HeaderName— the header a calling system (the mobile backend, another microservice) can send so its ID flows through.SafeId()— a source-generated regex allowing 8–64 letters, digits and hyphens. Without validation, a caller could send a header with newlines or JSON fragments and forge log lines (log injection). JSON formatters escape strings, but validation also stops absurdly long values.AmznRoot()— the ALB and API Gateway addX-Amzn-Trace-Id: Root=1-67a1b2c3-.... Reusing its root means the same ID appears in ALB access logs and X-Ray, so you can join them.Guid.CreateVersion7()— the fallback. UUID v7 is time-ordered, so IDs sort roughly by time (the same reason CONVENTIONS.md uses v7 for transaction IDs).context.Items[...]— stores the ID for the rest of the request so the ProblemDetails customisation can read it.Activity.Current?.SetTag(...)— adds it to the OpenTelemetry/Activityspan, so traces and logs share it.Response.OnStarting— sets the response header just before headers are sent. Setting it immediately would fail if a later component cleared headers (the exception handler does).LogContext.PushProperty— Serilog's ambient context: every log event written duringawait next(context)gets aCorrelationIdproperty, including events from deep inside the SDK wrappers or the exception handler.Enrich.FromLogContext()inProgram.csis what reads it.
The global exception handler
Hosting/GlobalExceptionHandler.cs:
public sealed class GlobalExceptionHandler(
IProblemDetailsService problemDetails,
ILogger<GlobalExceptionHandler> logger) : Microsoft.AspNetCore.Diagnostics.IExceptionHandler
{
private static readonly HashSet<string> ThrottlingCodes = new(StringComparer.Ordinal)
{
"ThrottlingException", "ProvisionedThroughputExceededException", "RequestLimitExceeded", "TooManyRequestsException",
};
public async ValueTask<bool> TryHandleAsync(HttpContext httpContext, Exception exception, CancellationToken cancellationToken)
{
var (status, title) = exception switch
{
AmazonServiceException aws when ThrottlingCodes.Contains(aws.ErrorCode ?? string.Empty)
=> (StatusCodes.Status503ServiceUnavailable, "The service is busy. Please retry shortly."),
OperationCanceledException when !httpContext.RequestAborted.IsCancellationRequested
=> (StatusCodes.Status504GatewayTimeout, "A downstream dependency did not respond in time."),
_ => (StatusCodes.Status500InternalServerError, "An unexpected error occurred."),
};
logger.LogError(exception, "Unhandled exception; returning {StatusCode}", status);
httpContext.Response.StatusCode = status;
if (status == StatusCodes.Status503ServiceUnavailable)
{
httpContext.Response.Headers.RetryAfter = "2";
}
return await problemDetails.TryWriteAsync(new ProblemDetailsContext
{
HttpContext = httpContext,
Exception = exception,
ProblemDetails = new ProblemDetails { Status = status, Title = title },
});
}
}- The fully qualified
Microsoft.AspNetCore.Diagnostics.IExceptionHandleris deliberate: the AWS SDK v4 core assembly also defines anAmazon.Runtime.IExceptionHandler, and with both namespaces imported the short name is ambiguous (compiler error CS0104). A real trap when you add the SDK to a web project. ThrottlingCodes— AWS services signal throttling with these error codes. The SDK's standard retry mode has already retried with back-off; if it still fails, the honest answer to the client is 503 +Retry-After, not 500.OperationCanceledException when !RequestAborted...— the repository enforces a per-call time budget with aCancellationToken(in SDK v4,ClientConfig.Timeoutdoes not bound async calls). If our budget expired while the client is still connected, it is a 504. If the client disconnected, the exception falls through to the default and nobody reads the response anyway.logger.LogError(exception, ...)— the only place the stack trace is written. It carries the correlation ID fromLogContext.ProblemDetails { Status, Title }— noDetailwith the exception message. AWS error messages include table names and ARNs; the lab's testThrottling_from_aws_becomes_503_with_retry_after_and_no_internal_detailsasserts that the table name never reaches the response.IProblemDetailsService.TryWriteAsync— runs theCustomizeProblemDetailscallback fromProgram.cs, which addsinstanceandcorrelationId, and fillstypewith the RFC 9110 section link for the status.
What the client receives:
HTTP/1.1 503 Service Unavailable
Content-Type: application/problem+json
Retry-After: 2
X-Correlation-ID: 01a1eb9e-a809-7285-8969-feea1953715c
{
"type": "https://tools.ietf.org/html/rfc9110#section-15.6.4",
"title": "The service is busy. Please retry shortly.",
"status": 503,
"instance": "GET /v1/accounts/ACC-0000000001",
"correlationId": "01a1eb9e-a809-7285-8969-feea1953715c"
}Expected errors are not exceptions. Validation failures and "not found" are returned directly by the endpoints with TypedResults.Problem(...) (400 and 404). Exceptions are for the unexpected; using them for control flow makes logs noisy and slows the API.
Keeping personal data out of logs
UK GDPR's data-minimisation principle applies to logs. Logs are copied, retained and read by many people, so the safest personal data in a log is none.
- Never log names, emails, phone numbers, full IBANs, card data (PCI DSS forbids logging full PANs and never allows sensitive authentication data), tokens or secrets.
- Pseudonymise identifiers you need for correlation. The Accounts API logs
CustomerRef, an HMAC-SHA256 of the customer ID with a secret key from Parameter Store or Secrets Manager (Security/Pseudonymisation.cs). The same customer always gets the samecref_…value, so support can follow one customer's journey, but the value cannot be reversed without the key. It is still personal data under UK GDPR (pseudonymised, not anonymised), so it still needs retention limits and access control. - Use message templates, not string interpolation.
LogInformation("Listed {AccountCount} accounts for customer {CustomerRef}", count, cref)keeps fields separate and searchable.LogInformation($"...{customerId}")bakes values into the message and defeats both querying and redaction. - Add a safety net. CloudWatch Logs data protection policies can detect and mask managed data identifiers (for example email addresses, credit card numbers) in a log group, with audit findings. Treat it as a net, not as the primary control.
AWS Console instructions
- CloudWatch → Logs → Log groups →
/hl-m03/accounts-api/ecs— check Retention (should be 1 week in the lab) and KMS key ID. Actions → Edit retention setting changes it. - CloudWatch → Logs → Logs Insights — select the log group, set a narrow time range, paste a query from the CLI section, Run query. Save stores it for the team.
- CloudWatch → Logs → Log groups → group → Metric filters → Create metric filter — pattern
{ $.@l = "Error" }, metric namespaceHarbourline/AccountsApi, metric nameErrorCount, value1. - CloudWatch → Logs → Log groups → group → Data protection → Create policy — choose managed data identifiers (for example
EmailAddress,CreditCardNumber) and an audit destination. - Lambda → function → Configuration → Monitoring and operations tools → Logging configuration — log format JSON, application and system log levels, and the log group.
AWS CLI
# Follow the API's logs live (all tasks), JSON lines as written by the app
aws logs tail /hl-m03/accounts-api/ecs --follow --since 10m --profile learn-sandbox
# Only errors, using a filter pattern on the JSON field
aws logs tail /hl-m03/accounts-api/ecs --since 1h --filter-pattern '{ $.@l = "Error" }' --profile learn-sandbox
# Set retention on a log group (the default is never expire)
aws logs put-retention-policy --log-group-name /hl-m03/accounts-api/ecs --retention-in-days 7 --profile learn-sandbox
# Logs Insights: everything for one correlation id, oldest first
QUERY_ID=$(aws logs start-query \
--log-group-name /hl-m03/accounts-api/ecs \
--start-time $(date -d '-1 hour' +%s) --end-time $(date +%s) \
--query-string 'fields @timestamp, @l, @m, StatusCode, SourceContext
| filter CorrelationId = "01a1eb9e-a867-7a7c-bd6d-535ce24626c0"
| sort @timestamp asc' \
--query queryId --output text --profile learn-sandbox)
aws logs get-query-results --query-id "$QUERY_ID" --profile learn-sandbox
# Logs Insights: 5xx count and p95 latency per path in 5-minute bins
aws logs start-query --log-group-name /hl-m03/accounts-api/ecs \
--start-time $(date -d '-3 hours' +%s) --end-time $(date +%s) \
--query-string 'filter SourceContext = "Serilog.AspNetCore.RequestLoggingMiddleware"
| stats count(*) as requests, sum(StatusCode >= 500) as errors, pct(Elapsed, 95) as p95ms by RequestPath, bin(5m)' \
--profile learn-sandbox
# Metric filter: one ErrorCount data point per error line
aws logs put-metric-filter --log-group-name /hl-m03/accounts-api/ecs \
--filter-name api-errors --filter-pattern '{ $.@l = "Error" }' \
--metric-transformations metricName=ErrorCount,metricNamespace=Harbourline/AccountsApi,metricValue=1,defaultValue=0 \
--profile learn-sandbox
# Lambda: JSON log format and levels for a function
aws lambda update-function-configuration --function-name hl-m03-accounts-api \
--logging-config LogFormat=JSON,ApplicationLogLevel=INFO,SystemLogLevel=WARN \
--profile learn-sandboxdate -d is GNU syntax and works in Git Bash. In PowerShell use [DateTimeOffset]::UtcNow.AddHours(-1).ToUnixTimeSeconds().
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.Logs;
using Constructs;
namespace Harbourline.Accounts.Infra;
public sealed class LoggingConstruct : Construct
{
public LogGroup LogGroup { get; }
public LoggingConstruct(Construct scope, string id, IKey logKey) : base(scope, id)
{
// Explicit group: retention and KMS key are decided here, not by whoever writes first.
LogGroup = new LogGroup(this, "ApiLogs", new LogGroupProps
{
LogGroupName = "/hl-m03/accounts-api/ecs",
Retention = RetentionDays.ONE_MONTH, // production: per your retention schedule
EncryptionKey = logKey, // key policy must allow logs.<region>.amazonaws.com
RemovalPolicy = RemovalPolicy.RETAIN, // logs are evidence: keep on stack deletion
});
var errors = new MetricFilter(this, "ErrorFilter", new MetricFilterProps
{
LogGroup = LogGroup,
FilterPattern = FilterPattern.StringValue("$.@l", "=", "Error"),
MetricNamespace = "Harbourline/AccountsApi",
MetricName = "ErrorCount",
MetricValue = "1",
DefaultValue = 0,
});
_ = new Alarm(this, "ErrorAlarm", new AlarmProps
{
AlarmDescription = "More than 5 error log lines in 5 minutes",
Metric = errors.Metric(new MetricOptions { Statistic = "Sum", Period = Duration.Minutes(5) }),
Threshold = 5,
EvaluationPeriods = 1,
ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING,
});
_ = new QueryDefinition(this, "ByCorrelationId", new QueryDefinitionProps
{
QueryDefinitionName = "accounts-api/by-correlation-id",
QueryString = new QueryString(new QueryStringProps
{
Fields = ["@timestamp", "@l", "@m", "StatusCode", "SourceContext"],
FilterStatements = new[] { "CorrelationId = \"REPLACE_ME\"" },
Sort = "@timestamp asc",
}),
LogGroups = [LogGroup],
});
}
}Complete code
A minimal, self-contained API that uses only the built-in ILogger JSON console formatter (no Serilog), for teams that prefer fewer dependencies. Run it with dotnet run and call /demo/fail and /demo/slow.
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
</Project>using System.Diagnostics;
using Microsoft.AspNetCore.Diagnostics;
using Microsoft.AspNetCore.Mvc;
var builder = WebApplication.CreateBuilder(args);
// 1. Built-in JSON console logs: one JSON object per line, scopes included.
builder.Logging.ClearProviders();
builder.Logging.AddJsonConsole(o =>
{
o.IncludeScopes = true; // correlation id arrives as a scope
o.UseUtcTimestamp = true;
o.TimestampFormat = "yyyy-MM-ddTHH:mm:ss.fffZ";
o.JsonWriterOptions = new() { Indented = false };
});
// 2. ProblemDetails for every error response, stamped with the correlation id.
builder.Services.AddProblemDetails(o => o.CustomizeProblemDetails = ctx =>
ctx.ProblemDetails.Extensions["correlationId"] = ctx.HttpContext.TraceIdentifier);
builder.Services.AddExceptionHandler<DemoExceptionHandler>();
var app = builder.Build();
// 3. Correlation: reuse a safe incoming id, else the W3C trace id; push it as a logging scope.
app.Use(async (ctx, next) =>
{
var incoming = ctx.Request.Headers["X-Correlation-ID"].ToString();
var id = incoming.Length is >= 8 and <= 64 && incoming.All(c => char.IsAsciiLetterOrDigit(c) || c == '-')
? incoming
: Activity.Current?.TraceId.ToString() ?? Guid.CreateVersion7().ToString();
ctx.TraceIdentifier = id;
ctx.Response.Headers["X-Correlation-ID"] = id;
var logger = ctx.RequestServices.GetRequiredService<ILoggerFactory>().CreateLogger("Correlation");
using (logger.BeginScope(new Dictionary<string, object> { ["CorrelationId"] = id }))
{
await next(ctx);
}
});
app.UseExceptionHandler();
app.UseStatusCodePages();
app.MapGet("/demo/ok", (ILogger<Program> log) =>
{
log.LogInformation("Balance enquiry served for {AccountId}", "ACC-0000000001");
return Results.Ok(new { balance = 1250.00m, currency = "GBP" });
});
app.MapGet("/demo/fail", () =>
{
// Simulates a bug that reveals an internal detail in its message.
throw new InvalidOperationException("Connection to core-banking-db-07.internal failed");
});
app.MapGet("/demo/slow", async (CancellationToken ct) =>
{
using var budget = CancellationTokenSource.CreateLinkedTokenSource(ct);
budget.CancelAfter(TimeSpan.FromMilliseconds(200)); // our time budget for the dependency
await Task.Delay(TimeSpan.FromSeconds(2), budget.Token); // the "dependency"
return Results.Ok();
});
app.MapGet("/demo/missing/{id}", (string id) =>
Results.Problem(statusCode: 404, title: "Account not found", detail: $"No account {id} is visible to you."));
app.Run();
internal sealed class DemoExceptionHandler(IProblemDetailsService problems, ILogger<DemoExceptionHandler> log) : IExceptionHandler
{
public async ValueTask<bool> TryHandleAsync(HttpContext ctx, Exception ex, CancellationToken ct)
{
var (status, title) = ex switch
{
OperationCanceledException when !ctx.RequestAborted.IsCancellationRequested
=> (StatusCodes.Status504GatewayTimeout, "A downstream dependency did not respond in time."),
_ => (StatusCodes.Status500InternalServerError, "An unexpected error occurred."),
};
log.LogError(ex, "Unhandled exception on {Path}; returning {Status}", ctx.Request.Path.Value, status);
ctx.Response.StatusCode = status;
return await problems.TryWriteAsync(new ProblemDetailsContext
{
HttpContext = ctx,
Exception = ex,
ProblemDetails = new ProblemDetails { Status = status, Title = title },
});
}
}Code explained line by line
builder.Logging.ClearProviders()thenAddJsonConsole— removes the default text console provider and writes JSON instead. On Lambda you can alternatively let the runtime format logs as JSON with the function's logging configuration.IncludeScopes = true— the JSON includes aScopesarray; the correlation dictionary appears there for every line written inside theusingblock.UseUtcTimestampand an ISO format — consistent UTC timestamps across hosts; CloudWatch also records its own ingestion time.AddProblemDetailswithCustomizeProblemDetails— every ProblemDetails response (from the exception handler,UseStatusCodePagesorResults.Problem) getscorrelationId.- The inline middleware — validates the incoming header (length 8–64, letters, digits, hyphen) before trusting it; otherwise uses the W3C trace ID of the current
Activity, which ASP.NET Core creates per request. ctx.TraceIdentifier = id— ASP.NET Core's built-in per-request ID is reused as our correlation ID, so framework logs that include it match ours.logger.BeginScope(...)—ILogger's equivalent of Serilog'sLogContext.PushProperty.UseExceptionHandler()without a path — uses the registeredIExceptionHandlerand the ProblemDetails service (ASP.NET Core 8+ behaviour).UseStatusCodePages()— turns empty 404/405 responses (no matching route) into ProblemDetails too./demo/fail— the exception message contains a host name; the handler logs it but the response says only "An unexpected error occurred."/demo/slow— a linked token withCancelAfteris the pattern for a time budget; the resultingOperationCanceledExceptionbecomes a 504./demo/missing/{id}— an expected outcome returned directly, no exception, no error-level log.
Security considerations
⚠ Learning shortcut — not for production
The lab log groups use 7-day retention, the AWS-owned encryption key and RemovalPolicy.DESTROY, so teardown leaves nothing behind. In production, set retention from the bank's records-retention schedule (agreed with compliance), encrypt with a customer managed KMS key whose policy limits who can decrypt, keep RemovalPolicy.RETAIN, and forward security-relevant logs to a central, write-once log archive account (Module 11).
- Access to logs is access to data. Grant
logs:GetLogEvents,logs:FilterLogEventsandlogs:StartQueryper log group to support roles, notlogs:*on*. - Validate everything you log from the request. Headers, query strings and bodies are attacker-controlled. Validate correlation IDs; never log raw bodies.
- Never return exception text. Stack traces, SQL, table names and ARNs help an attacker map your system. Set
ASPNETCORE_ENVIRONMENT=Productionso the developer exception page is never used. - Health-check noise hides attacks. The lab logs
/healthrequests atVerboseso they do not drown real traffic.
Banking use case
Harbourline's support process for API errors:
- The mobile app shows "Something went wrong. Reference: 01a1eb9e-…" using the
correlationIdfrom the ProblemDetails body. - Level-2 support runs the saved Logs Insights query
accounts-api/by-correlation-idwith that reference and a 30-minute window around the reported time. They see the request line, theUnhandled exception; returning 503line withProvisionedThroughputExceededException, and theCustomerRef(not the customer ID). - The
ErrorCountmetric filter alarm had already paged the on-call engineer, whose dashboard shows the throttling spike. - Logs are retained for the period in the bank's retention schedule, encrypted with a KMS key that only the observability and security teams can use, and subscription-filtered to the central log archive.
Common mistakes
- Logging with string interpolation → values are baked into the message, not queryable or maskable → use message templates.
- Writing log files inside the container → lost when the task stops, invisible to CloudWatch → stdout only.
- No retention set → logs kept forever, growing cost and a GDPR storage-limitation problem → set retention in IaC.
- Logging the exception at every layer → the same stack trace five times → log once, at the boundary (the exception handler).
- Returning
ex.Messageindetail→ leaks internals → generic titles for 5xx; precise details only for 4xx validation errors. - Trusting
X-Correlation-IDblindly → log injection, enormous values → validate format and length. - Using the static
Log.Loggerwith several hosts in one process → in integration tests running in parallel, hosts overwrite each other's logger and log assertions fail randomly →AddSerilog(..., preserveStaticLogger: true)as in the lab (this exact flaky test was fixed in the module project).
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| No logs in CloudWatch from ECS | Task execution role lacks logs:CreateLogStream/PutLogEvents, or wrong log group name | Check the execution role; CDK's LogDrivers.AwsLogs grants it automatically |
| Lambda logs missing | Function role lacks the basic execution policy, or a custom log group in another name | Attach AWSLambdaBasicExecutionRole; check the function's logging config |
| Beanstalk: only nginx logs, no app logs | Log streaming disabled | Set aws:elasticbeanstalk:cloudwatch:logs StreamLogs=true; app stdout is in web.stdout.log |
| Logs Insights shows fields as plain text | The line is not valid JSON (e.g. a text log from another provider in the same stream) | One JSON formatter only; ClearProviders() when using the built-in JSON console |
CorrelationId missing on some lines | Lines written before the middleware or outside the request (startup, background work) | Register the middleware early; push your own ID in background jobs |
| Error responses are empty bodies | AddProblemDetails() not registered, or UseStatusCodePages() missing | Register both, as in Program.cs |
| Clients see the developer exception page | ASPNETCORE_ENVIRONMENT=Development in a deployed environment | Set Production in the task definition / environment properties |
| Log-based test assertions fail randomly in CI | Static Serilog logger shared across parallel test hosts | preserveStaticLogger: true |
Cost and cleanup
CloudWatch Logs bills for ingestion, storage and queries. In us-east-1 the Standard log class costs about USD 0.50 per GB ingested and USD 0.03 per GB-month stored, and Logs Insights about USD 0.005 per GB scanned ⚠ VERIFY current prices on the CloudWatch pricing page. The AWS Free Tier includes 5 GB of log data per month (ingestion, archive storage and Insights scanning) ⚠ VERIFY under the current free plan. The lab's traffic produces a few megabytes, so it is effectively free.
Cost traps: debug logging left on in production, logging whole request bodies, health checks at Information level, never-expiring retention, and Insights queries over 30 days of every log group.
# Remove resources created only for this lesson
aws logs delete-metric-filter --log-group-name /hl-m03/accounts-api/ecs --filter-name api-errors --profile learn-sandbox
aws logs delete-log-group --log-group-name /hl-m03/accounts-api/ecs --profile learn-sandbox # if the stack is already destroyed
# Find log groups without retention (never expire) — fix or delete them
aws logs describe-log-groups --query "logGroups[?retentionInDays==null].logGroupName" --profile learn-sandboxExercises
- 🟢 Guided: Run the Accounts API locally (
dotnet run --project src/Harbourline.Accounts.Api --launch-profile http). Call/v1/accounts/ACC-0000000001with-H "X-Correlation-ID: support-case-000123", then with a header containing a space, then with none. Compare theX-Correlation-IDresponse headers and theCorrelationIdin the console logs. - 🟡 Challenge: Extend
GlobalExceptionHandlerso that a DynamoDBResourceNotFoundException(table missing, a configuration error) returns 503 with title "The service is not available" and logs atCritical. Write an integration test in the style ofThrottling_from_aws_becomes_503_with_retry_after_and_no_internal_details. - 🔴 Live problem: A penetration test report says: "Error responses from
/v1/customers/{id}/accountsdisclose the DynamoDB table name and AWS account ID." Logs showdetailfields likeRequested resource not found: Table: arn:aws:dynamodb:.... Find the likely cause in a codebase that otherwise looks like the lab, and fix it.
Assignment
Add a docs/observability.md to your portfolio repository describing the Accounts API's logging contract: the JSON fields and their meaning, the correlation ID rules, the ProblemDetails shapes per status code, the log levels per namespace, the retention period and its justification, what is never logged, and three saved Logs Insights queries.
Solutions
With support-case-000123 the API echoes the same value in the response header and every log line for that request has "CorrelationId":"support-case-000123". A value with a space fails the ^[A-Za-z0-9-]{8,64}$ check, so the middleware ignores it; locally there is no X-Amzn-Trace-Id, so it generates a UUID v7 (starting with the current time in hex) and returns that. With no header you also get a fresh UUID v7. Behind an ALB the third case would reuse the Root= value of X-Amzn-Trace-Id instead, so the ID would match the ALB access log.
Add an arm before the default: ResourceNotFoundException => (StatusCodes.Status503ServiceUnavailable, "The service is not available") (namespace Amazon.DynamoDBv2.Model), and log with logger.Log(status == 503 && exception is ResourceNotFoundException ? LogLevel.Critical : LogLevel.Error, exception, ...). The test replaces IAccountRepository with a fake that throws new ResourceNotFoundException("Requested resource not found: Table: hl-m03-accounts not found"), calls an endpoint, asserts status 503, asserts the body does not contain hl-m03-accounts, and asserts a log event at LogEventLevel.Fatal (Serilog's name for Critical) exists in CollectingSink. 503 rather than 500 because it is an availability problem the client can retry later, and Critical because it means a broken deployment, not a single bad request.
The global handler is probably not the culprit (it never sets Detail). Look for code that catches an AmazonServiceException and returns Results.Problem(detail: ex.Message), or a CustomizeProblemDetails callback that copies ctx.Exception?.Message into Detail. Another cause: ASPNETCORE_ENVIRONMENT=Development in the deployed task, where ProblemDetails can include exception details. Fix: remove ex.Message from responses, let the exception reach the global handler (or map it to a generic title), ensure Production, and add a regression test asserting that 5xx bodies contain no arn: and no table names. Then rotate nothing (no secret leaked) but record the finding and fix in the vulnerability tracker.
Knowledge check
Answer: The platform (awslogs driver, Lambda runtime, Beanstalk agent) already ships stdout to CloudWatch asynchronously. Calling the API from the app adds latency, failure modes, IAM permissions and cost per call, and ties the code to one destination.
Answer: From the Root= part of the X-Amzn-Trace-Id header that the ALB adds. That ties application logs to ALB access logs and X-Ray traces. Only if neither header is usable does it generate a UUID v7.
Answer: 503 Service Unavailable with a Retry-After header. The problem is temporary capacity, not a bug, and well-behaved clients back off and retry. A 500 suggests a permanent server error and gives no retry guidance.
Answer: No. Whoever holds the key can re-link it, so it is pseudonymised personal data. It reduces risk and is good practice, but the logs still need a lawful basis, retention limits and access control.
Answer: Never expire. Always set retention explicitly, ideally in IaC, because otherwise storage costs grow forever and you keep personal data longer than necessary.
Interview questions
- "How do you trace one failed request across many containers?" — A correlation ID per request, accepted from callers when valid or taken from
X-Amzn-Trace-Id, pushed into the logging context, returned in a response header and in ProblemDetails. Then one Logs Insights query on that field. For cross-service timing, add OpenTelemetry/X-Ray traces with the same ID as a tag. - "What goes in an error response from a banking API?" — RFC 9457 ProblemDetails: status, generic title for 5xx, precise detail for 4xx validation, instance, correlation ID. Never stack traces, exception messages, resource names or personal data of other customers.
- "How do you keep personal data out of logs?" — Message templates, no payload logging, pseudonymised identifiers, code review rules, tests that assert raw identifiers are absent, and CloudWatch Logs data protection policies as a safety net; plus retention and access control on the log groups.
- "Serilog or built-in
ILogger?" — Both write structured JSON. Serilog brings enrichers, request logging and configuration-driven sinks; the built-in JSON console has no dependency. Either way the app logs throughILogger<T>.
Certification practice
A developer must find all log entries for a single failed API request across 12 Amazon ECS tasks. The application writes JSON logs with a requestId field to Amazon CloudWatch Logs. What is the MOST efficient approach?
A. Download all log streams and search them locally.
B. Run a CloudWatch Logs Insights query filtering on requestId for a narrow time range.
C. Create a metric filter for the requestId and view the metric.
D. Enable AWS CloudTrail data events for CloudWatch Logs.
Answer: B. Logs Insights searches all streams in the group and can filter on JSON fields. A is slow. C counts matches but does not show the entries. D records API calls to CloudWatch, not application log content.
A security review finds that a .NET API returns exception messages containing internal host names to clients. The team wants consistent error responses that contain a reference for support staff but no internal details. What should the developer implement?
A. Disable logging for exceptions.
B. A global exception handler that logs the exception and returns a ProblemDetails response with a generic title and a correlation ID.
C. Return HTTP 200 with an error field for all failures.
D. Encrypt the exception message in the response.
Answer: B. Details go to the logs, the client gets a standard response and a reference. A removes the evidence needed to fix the bug. C breaks HTTP semantics and clients' retry logic. D still discloses data and is pointless complexity.
A company's CloudWatch Logs bill has grown every month for a year although traffic is flat. Applications run on AWS Lambda and Amazon ECS. What is the MOST likely cause and fix?
A. Lambda functions are over-provisioned; reduce memory. B. Log groups use the default retention of never expire; set retention periods on all log groups. C. CloudWatch Logs charges per log stream; reduce the number of tasks. D. Logs are encrypted with KMS; remove encryption.
Answer: B. With flat traffic, ingestion is flat, but stored data grows forever without retention. A affects Lambda compute, not logs. C is not how CloudWatch Logs is billed. D removes a security control for a negligible saving.
Completion checklist
- I can explain how ECS, Lambda and Beanstalk each ship stdout to CloudWatch Logs
- I can read a Serilog compact JSON line and name each
@field - I can explain the correlation ID precedence and why the header is validated
- I can map throttling, timeouts and bugs to 503, 504 and 500 ProblemDetails
- I ran the guided exercise and saw the correlation header behaviour
- I checked that my log groups have retention set
Key takeaways
- Log structured JSON to stdout; let the platform ship it to CloudWatch Logs.
- One correlation ID per request, validated, reused from
X-Amzn-Trace-Idwhen present, on every log line and in every error response. - A global exception handler logs once and returns ProblemDetails without internals: 503 for throttling, 504 for timeouts, 500 for bugs.
- Personal data stays out of logs: templates, pseudonymised references, retention, access control, data protection policies.
- Set retention on every log group; never-expire is the default and the most common cost trap.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Scaling .NET apps on AWS
Make an ASP.NET Core API stateless, size its connection pools, move session state to ElastiCache or DynamoDB, and scale it automatically with target tracking on ECS and concurrency controls on Lambda.
Authentication and authorisation for .NET APIs on AWS
Protect an ASP.NET Core API with Amazon Cognito access tokens, validate JWTs correctly (issuer, token_use, client_id, scopes), use policy-based authorisation, and keep user identity separate from the IAM role your code runs as.