Skip to Content
Cloud Engineering Academy
11. Monitoring, Logging & Troubleshooting

CloudWatch Logs for .NET: structured JSON, correlation IDs, retention and PII-safe logging

Build a .NET 10 console formatter that writes one flat, redacted JSON object per line, ship it to CloudWatch Logs from ECS or Lambda, set retention and encryption in code, and keep personal data out of the log group.

intermediate⏱ 1 h 30 min🟡 Low costSOA-C03DOP-C02DVA-C03

Assumes: .NET 10 LTS · Microsoft.Extensions.Logging 10.0 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Describe how a line printed to stdout by an ASP.NET Core app becomes an event in a CloudWatch log group and log stream, on ECS Fargate and on Lambda.
  • Write a custom ConsoleFormatter that emits flat JSON with a correlation ID, trace ID and PII redaction applied to every value.
  • Set retention, log class and encryption on a log group in CDK (C#) and Terraform, and explain why the default is a mistake.
  • Give the application only the logging permissions it needs, and give operators read-only access to named log groups.
  • Explain the limits that shape a logging design: event size, batch size, metric-filter and subscription-filter counts.

Prerequisites

  • Observability fundamentals: signals, cardinality, actionable alerts.
  • Logging and errors for .NET on AWS: the Serilog-based first pass, global exception handler and ProblemDetails. This lesson goes deeper on the formatter, redaction and governance.
  • Tools: .NET 10 SDK or Docker, AWS CLI v2 with the learn-sandbox SSO profile (only for the deployed part).

Why this matters in banking

Two incidents show why log design is a banking concern.

Incident one: the log that became a breach. A developer adds logger.LogInformation("Request: {Body}", body) to debug a payment bug. For three weeks, every transfer request body, including the creditor IBAN, the amount and a free-text reference field in which customers type their email address or phone number, is written to a log group with retention set to "never expire". A support engineer with log access can now see data they were never meant to see. The bank has an unplanned personal-data store with no retention limit and no deletion process. That is a data-protection problem under UK GDPR, discovered by a routine review rather than by the attacker you were worried about.

Incident two: the log you cannot search. An outage hits. The logs are plain English sentences with the interesting values buried in them, some across several lines because of stack traces. Finding "all lines for the failed transfer" means a regular expression written under pressure.

Good log design prevents both: structured so it is searchable, minimal so it holds nothing it should not, and governed with retention and access control set in code.

The idea in plain language

An ASP.NET Core app writes log lines to stdout, exactly as it would to the console on your laptop. On AWS, the platform collects stdout and sends it to Amazon CloudWatch Logs. You do not call a logging API from your code.

CloudWatch Logs organises data in three levels:

TermWhat it isTypical value in Harbourline
Log groupA container with one retention setting, one access policy, optionally one KMS key. Usually one per application per environment/hl-m11/transfers-api
Log streamOne source writing in order: one Fargate task, one Lambda execution environmentapi/api/4f2c...
Log eventOne record: a timestamp and a message string (your JSON line){"timestamp":"...","level":"Error",...}

Three rules make logs useful in an incident:

  1. One event, one line, one JSON object. A multi-line stack trace becomes many events that interleave with other requests. A JSON object with an exception field stays together.
  2. Flat fields. CloudWatch Logs Insights discovers top-level JSON keys automatically, so filter statusCode >= 500 just works. Nested arrays force awkward paths like Scopes.0.CorrelationId.
  3. Every line carries identity. correlationId joins all lines of a request; traceId joins the log to the trace.

Analogy: A structured log is a bank's paper form rather than a free-text diary entry. Every form has the same boxes (date, branch, account type, outcome), so a clerk can sort a thousand forms in minutes. A diary has more nuance but nobody can search it at 3 a.m.

What problem the AWS service solves

CloudWatch Logs stores, searches, retains and streams log data from your applications and from AWS services. The vocabulary for the features this lesson uses:

  • Retention is set per log group, from one day up to ten years, or "never expire". New log groups default to never expire, which is the cost and data-protection trap this lesson closes.
  • Log class is chosen when the group is created: Standard (the default, full features) or Infrequent Access (cheaper ingestion, fewer features such as no metric filters or subscription filters ⚠ VERIFY the exact feature list and price). You cannot change the class of an existing group ⚠ VERIFY.
  • KMS encryption with your own key is optional. CloudWatch Logs encrypts at rest by default.
  • Metric filters turn matching log lines into metrics; subscription filters stream matching lines to Lambda, Kinesis, Firehose or OpenSearch.
  • Data protection policies can detect and mask sensitive data patterns in log events at ingestion ⚠ VERIFY the supported identifiers, the price and the regions before relying on them.
  • Live Tail streams events to your screen in near real time.

Quotas that shape design (checked 2026-11-18 against the CloudWatch Logs quotas page; Region us-east-1):

LimitValueConsequence
Maximum log event size1,024 KBOlder material quotes 256 KB; do not rely on large events anyway. Never log whole payloads
Maximum PutLogEvents batch1 MBAgents and drivers batch for you
Metric filters per log group100Plenty, but each one is a design decision
Subscription filters per log group5Plan fan-out carefully
StartQuery rate10 per secondDashboards with many log widgets can throttle

Honest limitations: log ingestion is billed per GB (USD 0.50 per GB for Standard in US East at the time of checking ⚠ VERIFY), so chatty frameworks cost real money. CloudWatch Logs is not a long-term audit archive. If a records policy demands years of retention, export to S3 with Object Lock (see CloudTrail for banks for the pattern). And logs are not a metric store: do not scan logs to count things at scale; use EMF (lesson 4).

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
ILogger + Application Insights provider (HTTP call to an ingestion endpoint)ILogger + JSON console formatter, platform ships stdoutNo logging SDK in the request path; the log driver or Lambda runtime does the shipping
Log Analytics workspace (one place for many apps)Log groups (one per app/environment), queried together by nameRetention and encryption are per group, not per workspace
Table-level retention in Log Analyticsretention_in_days per log groupDefault is never expire
Data Collection Rules with transformationsData protection policies, field indexes, log transformers ⚠ VERIFYDifferent feature set; verify before assuming parity
Diagnostic settings to Event Hub for SIEMSubscription filters to Kinesis/Firehose/LambdaMaximum 5 per log group
Azure RBAC on a workspaceIAM policy on logs: actions and log-group ARNsGrant per log group ARN, not account-wide
Serilog RenderedCompactJsonFormatter (@t, @m, @l)A flat formatter (timestamp, message, level)@-prefixed names need backticks in Logs Insights; plain names are easier

Architecture

Security-boundary view. There are three separate permission sets, and mixing them up is a common mistake:

  • The task execution role (ECS) or function role (Lambda) lets the platform write to the log group. The application code itself needs no logging permission.
  • The operations role lets humans read and query specific log groups.
  • The KMS key policy, if you use your own key, must let the CloudWatch Logs service use the key and let operators decrypt through it.

Redaction happens inside the process, before the line leaves stdout. Nothing downstream, including the log driver, ever sees the raw value.

Step-by-step implementation

Write the formatter once, in a shared library

Create HlJsonFormatter : ConsoleFormatter so every service in the bank writes the same shape. It takes the log entry, adds timestamp, level, category, message, trace IDs from Activity.Current, then every property from the message template and the logging scopes, flattened and camel-cased. Every string passes through the redactor.

Register it and silence the noise

AddConsole(o => o.FormatterName = "hl-json") plus AddConsoleFormatter<HlJsonFormatter, ConsoleFormatterOptions>(). In appsettings.json set Microsoft.AspNetCore to Warning; the framework otherwise writes six lines per request that duplicate your own request-completed line.

Add correlation middleware early in the pipeline

Resolve an incoming X-Correlation-ID only if it is short and made of safe characters; otherwise generate a UUID v7. Echo it on the response, add it to a logging scope and tag the current span. Because scope properties are flattened by the formatter, every line in the request carries correlationId.

Write one request-completed line

In the middleware's finally block, log method, path, status code and elapsed milliseconds. This single line feeds the error-rate and latency queries in lesson 3.

Create the log group in code

CDK or Terraform, with explicit retention, tags and (production) a KMS key. Never let the first write auto-create the group, because that creates it with never expire.

Point the platform at the group

ECS: an awslogs log configuration in the task definition. Lambda: set the function's log group and JSON log format. Check that the execution role may write.

Prove redaction with a test

Unit-test the formatter with an IBAN, a card number, an email and a phone number in the message and in a property, and assert that none appear in the output. The lab's LoggingTests.cs does exactly this.

AWS Console instructions

Create a log group: Console → CloudWatch → Logs → Log management → Create log group. Enter the name, choose Retention setting (for example 1 week), optionally pick a KMS key and the Log class, then Create.

Change retention on an existing group: select the log group, Actions → Edit retention setting, choose a period, Save.

Read events: open the log group, choose a stream, or choose Start tail for Live Tail. For queries, use CloudWatch → Logs → Logs Insights (lesson 3). Console labels move between releases ⚠ VERIFY if a menu is not where described.

AWS CLI

# Create a log group with an explicit retention. Never leave the default (never expire).
aws logs create-log-group --log-group-name /hl-m11/transfers-api \
  --tags Project=cea-lab,Lab=m11-observe-banking-api,Environment=learn,Owner=you \
  --profile learn-sandbox
aws logs put-retention-policy --log-group-name /hl-m11/transfers-api --retention-in-days 7 --profile learn-sandbox

# Audit: which log groups in the account never expire? (this finds the cost and data-protection traps)
aws logs describe-log-groups --profile learn-sandbox \
  --query "logGroups[?retentionInDays==null].[logGroupName,storedBytes]" --output table

# Follow new events for a group, like tail -f, from the last 10 minutes
aws logs tail /hl-m11/transfers-api --follow --since 10m --profile learn-sandbox

# Search with a filter pattern (JSON pattern: all error events). Cheap for quick looks; use Insights for analysis.
aws logs filter-log-events --log-group-name /hl-m11/transfers-api \
  --filter-pattern '{ $.level = "Error" }' --max-items 5 --profile learn-sandbox

# Write a test line yourself (needs a stream); useful to try metric filters without running the app
aws logs create-log-stream --log-group-name /hl-m11/transfers-api --log-stream-name manual-test --profile learn-sandbox
aws logs put-log-events --log-group-name /hl-m11/transfers-api --log-stream-name manual-test --profile learn-sandbox \
  --log-events "timestamp=$(date +%s000),message='{\"level\":\"Error\",\"message\":\"test\",\"correlationId\":\"manual-0001\"}'"

Infrastructure as Code

LogGroupExample.cs
using Amazon.CDK;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.Logs;
using Constructs;

public sealed class LoggingStack : Stack
{
    public LoggingStack(Construct scope, string id) : base(scope, id)
    {
        // Production: a customer managed key with rotation, if your policy requires key control.
        var key = new Key(this, "LogsKey", new KeyProps { EnableKeyRotation = true });

        var logGroup = new LogGroup(this, "ApiLogs", new LogGroupProps
        {
            LogGroupName = "/hl-m11/transfers-api",
            Retention = RetentionDays.ONE_WEEK,      // never leave the default (never expire)
            EncryptionKey = key,                      // CDK adds the key-policy statement for the Logs service
            RemovalPolicy = RemovalPolicy.DESTROY,    // lab only
        });
    }
}

The lab's full stack is in labs/m11-observe-banking-api/infra/cdk and infra/terraform. The ECS task definition side (the awslogs driver) is shown in Deploy to Fargate. Two driver options deserve attention: non-blocking mode (so a logging problem cannot stall request threads, at the price of possibly dropping lines when the buffer is full) and the maximum buffer size ⚠ VERIFY the option names for your agent version.

Complete code

The formatter and redactor live in labs/m11-observe-banking-api/src/Harbourline.Observability/Logging. The essence of the formatter:

HlJsonFormatter.cs (excerpt)
public override void Write<TState>(in LogEntry<TState> logEntry, IExternalScopeProvider? scopeProvider, TextWriter textWriter)
{
    var message = logEntry.Formatter?.Invoke(logEntry.State, logEntry.Exception);
    if (message is null && logEntry.Exception is null) return;

    using var buffer = new MemoryStream();
    using (var json = new Utf8JsonWriter(buffer))
    {
        json.WriteStartObject();
        json.WriteString("timestamp", _clock.GetUtcNow().UtcDateTime.ToString("O"));
        json.WriteString("level", logEntry.LogLevel.ToString());
        json.WriteString("category", logEntry.Category);
        json.WriteString("message", Clean("message", message));

        var activity = Activity.Current;
        if (activity is not null)
        {
            json.WriteString("traceId", activity.TraceId.ToString());
            json.WriteString("spanId", activity.SpanId.ToString());
        }

        var written = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
        scopeProvider?.ForEachScope(static (scope, state) =>
        {
            if (scope is IEnumerable<KeyValuePair<string, object?>> pairs)
                foreach (var pair in pairs) WriteProperty(state.Json, state.Written, pair.Key, pair.Value);
        }, new ScopeState(json, written));

        if (logEntry.State is IEnumerable<KeyValuePair<string, object?>> props)
            foreach (var pair in props) WriteProperty(json, written, pair.Key, pair.Value);

        if (logEntry.Exception is not null)
            json.WriteString("exception", Clean("exception", $"{logEntry.Exception.GetType().Name}: {logEntry.Exception.Message}"));

        json.WriteEndObject();
    }
    textWriter.WriteLine(Encoding.UTF8.GetString(buffer.GetBuffer(), 0, (int)buffer.Length));
}

The redactor's public surface:

PiiRedactor.cs (excerpt)
public static string Redact(string? text);                       // masks IBANs, Luhn-valid card numbers, phones, emails
public static bool IsSensitiveKey(string key);                   // creditorIban, accessToken, password, pan, cvv ...
public static string RedactValue(string key, string? value);     // whole value hidden if the KEY is sensitive

And the correlation middleware, which runs first in the pipeline:

CorrelationMiddleware.cs (excerpt)
public async Task InvokeAsync(HttpContext context)
{
    var correlationId = CorrelationId.Resolve(context.Request.Headers["X-Correlation-ID"]);
    context.Items[ItemKey] = correlationId;
    context.Response.Headers["X-Correlation-ID"] = correlationId;
    Activity.Current?.SetTag("correlation.id", correlationId);

    var started = Stopwatch.GetTimestamp();
    using var scope = logger.BeginScope(new Dictionary<string, object?> { ["CorrelationId"] = correlationId });
    try { await next(context); }
    finally
    {
        var status = context.Response.StatusCode;
        logger.Log(status >= 500 ? LogLevel.Error : LogLevel.Information,
            "Request completed {Method} {Path} {StatusCode} {ElapsedMs}",
            context.Request.Method, context.Request.Path.Value, status,
            Math.Round(Stopwatch.GetElapsedTime(started).TotalMilliseconds, 1));
    }
}

A real line from the lab's container (formatted for reading), after a transfer reference of rent, call me on +44 7700 900123:

{
  "timestamp": "2026-11-18T08:00:13.1770679Z",
  "level": "Information",
  "category": "Program",
  "message": "Transfer requested, reference rent, call me on [phone]",
  "traceId": "5cf629daf8ba97bdcfffaf7929fca643",
  "spanId": "8e9c3eb4500a450a",
  "correlationId": "01a1edab-ab5a-7c55-bda9-d95e2a2212e4",
  "reference": "rent, call me on [phone]"
}

Code explained line by line

  1. ConsoleFormatter is the extension point of the built-in console logger. Writing your own formatter means you control the exact JSON shape and, importantly, run the redactor on every value before it is written.
  2. if (message is null && logEntry.Exception is null) return; mirrors the built-in formatters: a call that produces no message and no exception writes nothing.
  3. Utf8JsonWriter writes into a MemoryStream first so one textWriter.WriteLine emits the whole object; two threads therefore cannot interleave half-written lines.
  4. Activity.Current provides the W3C trace and span IDs when OpenTelemetry (or any ActivityListener) is active. Putting them in the log is what links the log to the trace.
  5. ForEachScope with a static lambda and a state struct avoids allocating a closure for every log call. Scope values (the correlation ID) are written before the message-template values.
  6. The written set stops duplicate keys: ASP.NET Core sometimes adds the same property from both a scope and the state, and JSON with duplicate keys confuses Logs Insights.
  7. Clean applies RedactValue(key, value): if the name is sensitive (creditorIban) the whole value becomes [redacted]; otherwise the content is scanned for IBANs, Luhn-valid card numbers, phones and emails. The Luhn check matters: it stops 16-digit order numbers being masked, and the code also ignores a run of identical digits, because the lab found that the all-zero parent span ID 0000000000000000 passes Luhn and was being masked as a card.
  8. In the middleware, the finally block guarantees one request-completed line even when the pipeline throws, so error requests are never missing from your latency and error-rate queries.

Security considerations

⚠ Learning shortcut — not for production

The lab log group uses the default AWS-owned encryption and RemovalPolicy.DESTROY, and the API deliberately logs a free-text field to demonstrate redaction. Production: a customer managed KMS key if policy requires it, RETAIN (or export to S3 with Object Lock) for logs with evidential value, and no free-text logging at all.

  • Don't log what you don't need. Redaction is a safety net. Regular expressions will miss formats (names, addresses, national insurance numbers). The primary control is an allow-list of what each log statement may contain, enforced in code review. See UK GDPR and data residency for the wider data-protection context, and agree masking rules and retention periods with your data-protection officer.
  • Refuse unsafe correlation IDs. A header value that contains newlines or quotes could forge extra log lines in plain-text logs. The lab accepts only 8 to 64 characters from [A-Za-z0-9._-] and replaces anything else.
  • Restrict who reads. Grant logs:StartQuery, logs:GetQueryResults, logs:GetLogEvents and logs:FilterLogEvents on named log-group ARNs, not *. In production, developers should reach production logs through an audited role, not a shared one.
  • Truncate. The formatter caps any value at 16,384 characters, so one careless LogInformation(body) cannot produce a giant event.
  • Logs are not tamper-proof. Anyone with logs:DeleteLogGroup can remove them. Protect critical log groups with an SCP or IAM deny, and send evidential logs to a write-once store.

Banking use case

The Harbourline Payments team runs the Transfers API on Fargate. Each task writes flat JSON to stdout; the awslogs driver ships it to /hl-prod/transfers-api with 30 days of retention in CloudWatch, encrypted with a team-owned KMS key. A nightly job exports one day of audit-relevant events (payment accepted, payment rejected, sanctions screening result) to an S3 bucket in a separate log-archive account with Object Lock for seven years, because the bank's records policy requires it. Everyday debugging logs never leave the 30-day window. During an incident the on-call engineer runs a saved query on the correlation ID from the failed customer response and sees the whole request. All data is fictional.

Common mistakes

  • Auto-created log groups. Lambda and some agents create groups with never expire. Create them in IaC first, and run the "which groups never expire" audit command monthly.
  • Multi-line text logs. Stack traces split into many events. Use one JSON line with an exception field.
  • Logging request and response bodies "for debugging". Both a cost and a data-protection problem. Log sizes and IDs instead.
  • Writing a CloudWatch client into the app. It adds latency, needs IAM permissions in the app, and loses lines when the process dies. Write to stdout and let the platform ship it.
  • A different log shape per service. Share the formatter as a library so queries work everywhere.
  • Not testing the redactor. Add unit tests with realistic data (IBANs with and without spaces, Luhn-valid test cards, +44 numbers).

Troubleshooting

SymptomLikely causeFix
No logs appear in the groupTask execution role cannot write, or wrong group name in the task definitionCheck the role has logs:CreateLogStream and logs:PutLogEvents, and that the group exists (the driver does not create it by default)
Logs appear but JSON fields are not searchableLine is not pure JSON (prefix text, or the console logger's default format)Confirm FormatterName = "hl-json" and that nothing else writes a prefix
Duplicate fields or Scopes.0... in queriesBuilt-in JSON console formatter in useUse the flat formatter
Costs higher than expectedFramework chatter at Information, or Debug left onSet Microsoft.AspNetCore to Warning; check ingestion by log group
AccessDenied reading logs with KMSOperator lacks kms:Decrypt on the keyAdd it to the operations role and the key policy
EMF metric lines look wrong in logsConsole formatter wrapped the EMF lineEmit EMF directly to Console.Out, not through ILogger (lesson 4)

Cost and cleanup

Rough cost at lab volumes: a few hundred log lines is a few hundred kilobytes, well inside the 5 GB free ingestion allowance for logs listed on the CloudWatch pricing page at the time of checking ⚠ VERIFY. The cost risk is not the lab but forgetting it: set retention to 7 days and delete the group afterwards.

# Delete the lab log group and confirm nothing from this lab remains
aws logs delete-log-group --log-group-name /hl-m11/transfers-api --profile learn-sandbox
aws logs describe-log-groups --log-group-name-prefix /hl-m11 --profile learn-sandbox
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab --profile learn-sandbox

If you deployed with CDK or Terraform, use cdk destroy or terraform destroy instead.

Exercises

  • 🟢 Guided: Run the lab tests (LoggingTests.cs), then add a new test: a log message containing a UK mobile number in international format and a 16-digit order number. Assert that the phone is masked and the order number is not.
  • 🟡 Challenge: Extend PiiRedactor to mask UK National Insurance numbers (two letters, six digits, a final letter A to D, in the fictional range you choose). Add tests for both positive and negative cases, including a near miss.
  • 🔴 Live problem: A security review finds creditorName values in the production log group. Retention is "never expire" and 14 months of data exist. Write the plan: stop the leak, decide what happens to the existing data (with your data-protection officer), prevent regressions in CI, and report the incident. List the AWS actions you would take and which need approval.

Assignment

Add a Harbourline.Observability style formatter to a .NET service of your own. Deliver: the formatter and redactor, five unit tests including one for each data type masked, appsettings.json log levels that remove framework noise, and an IaC snippet (CDK or Terraform) that creates the log group with retention. Add a short LOGGING.md describing what may and may not be logged.

Solutions

Knowledge check

Interview questions

  • How do logs get from a .NET container to CloudWatch? The app writes to stdout; the awslogs driver (or FireLens) batches events with PutLogEvents using the task execution role into a log group and stream.
  • How do you keep personal data out of logs? Don't log it (allow-list), redact at the formatter as a safety net, restrict access and set retention, and test with realistic data.
  • Why add a correlation ID, and where? To join every line of one request. Add it in the first middleware, echo it to the caller and put it in the logging scope.
  • What would you do about a log group that never expires? Set retention, estimate the cost, check for personal data with the data-protection officer, and export evidential logs to S3 with Object Lock.
  • What limits matter when designing log events? Event size (1,024 KB per event today), PutLogEvents batch size (1 MB), and the counts of metric filters (100) and subscription filters (5) per log group.

Certification practice

Completion checklist

  • I can explain how stdout becomes a CloudWatch log event on ECS and on Lambda
  • I ran the lab tests and read HlJsonFormatter and PiiRedactor
  • I created a log group with explicit retention (in code, not by hand)
  • I ran the "groups that never expire" audit command
  • I cleaned up and confirmed nothing tagged Project=cea-lab remains

Key takeaways

  • One event, one line, one flat JSON object; correlation ID and trace ID on every line.
  • Redact inside the process, and treat redaction as a safety net rather than a licence to log personal data.
  • Set retention, class and encryption in code; the default of never expire is a trap.
  • Separate write permission (execution role) from read permission (operations role), and grant both per log group.
  • Write EMF lines straight to stdout (lesson 4), not through the logger, so they remain pure JSON.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page