Skip to Content
Cloud Engineering Academy
11. Monitoring, Logging & Troubleshooting

Lab 11: observe a banking API (logs, metrics, alarms, dashboard and traces)

Run the Harbourline Transfers API with structured logs, EMF metrics and OpenTelemetry traces, inject a slow database and a partner outage, find each cause with Logs Insights and Jaeger, then deploy alarms, a composite page and a dashboard with CDK or Terraform and tear everything down.

intermediate⏱ 2 h 30 min🟡 Low costSOA-C03DOP-C02DVA-C03

Assumes: .NET 10 LTS · OpenTelemetry .NET 1.21.0 · Amazon.CDK.Lib 2.277.0 · Cdklabs.CdkNag 3.0.2 · Terraform AWS provider v6 · Docker · Region us-east-1 · labs/m11-observe-banking-api. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lab you can:

  • Run the Transfers API with flat JSON logs, a correlation ID, PII redaction, EMF business metrics and OpenTelemetry traces, and read each signal.
  • Inject two faults (a slow database and a partner outage) and find the cause of each from a trace and from a Logs Insights query.
  • Deploy a log group with 7-day retention, an SNS page topic, four metric alarms, a composite page alarm and a dashboard from CDK (C#) or Terraform.
  • Rehearse a page with set-alarm-state and read the alarm history.
  • Tear everything down and prove that nothing tagged Project=cea-lab remains.

Prerequisites

Lessons 1 to 7 of this module:

Tools: Docker Desktop. For the AWS part: AWS CLI v2 with aws sso login --profile learn-sandbox, jq, and either Node.js LTS with the CDK CLI (and a bootstrapped account) or Terraform 1.14. The .NET SDK is optional because every command also runs in mcr.microsoft.com/dotnet/sdk:10.0.

Why this matters in banking

The module project asks you to "observe and fix a banking API". This lab is the first half: make the Transfers API observable, break it on purpose, and practise finding the cause from the data rather than from guesses. The same exercise is how real teams run game days: a planned, safe failure that checks whether the alarms fire, whether the runbook works and whether on-call can find the cause quickly. Banks that must demonstrate operational resilience run these regularly and keep the results as evidence.

The idea in plain language

You will run the API on your machine. It writes three kinds of telemetry:

  1. Logs to stdout: one flat JSON object per line (with a correlation ID, and a trace ID when inside a request).
  2. Metrics to stdout, as Embedded Metric Format lines that CloudWatch Logs would turn into metrics.
  3. Traces over OTLP to a collector and Jaeger.

You then break it twice, find the causes, and deploy the AWS side (log group, alarms, dashboard) so the same signals would page a person if this were running in a real account.

Analogy: Think of fitting a test car with a dashboard, a black-box recorder and a GPS tracker, then driving it over a rough road on purpose to see whether the warning lights come on, and whether the recorder tells you which wheel hit the pothole.

What problem the AWS service solves

Locally you practise reading signals. In AWS the same lines feed CloudWatch Logs, CloudWatch metrics and alarms, X-Ray and dashboards. The lab is split so that most of it is free:

PartCost labelWhat runs
A. Unit tests🟢 freeDocker only
B. Local traces and logs, fault injection🟢 freeDocker Compose (API, collector, Jaeger)
C. Deploy alarms, topic and dashboard🟡 creditsCloudWatch, SNS (minutes; destroy the same day)
D. Logs Insights and the page rehearsal🟡 creditsA few kilobytes of logs

If you do not want to deploy anything, do parts A and B and read the infrastructure code for part C (a free design-only alternative).

Microsoft / Azure comparison

You know (Microsoft/Azure)Lab equivalentWhat's different
Application Insights end-to-end transaction viewJaeger trace view locally; X-Ray in AWSSame OpenTelemetry data, different backend
Azure Monitor metric alert + action groupCloudWatch alarm + composite + SNSThe composite is the one object that notifies
Log Analytics saved queriesIncidentQueries deployed as query definitionsInsights QL, not KQL
Chaos Studio experimentsX-Fault header and Faults__* settingsA tiny in-app switch; AWS Fault Injection Service is the managed option (Module 12) ⚠ VERIFY

Architecture

Security-boundary view. Everything local is on your machine. The AWS side is private to your sandbox account: the SNS topic allows only CloudWatch alarms from this account to publish and denies non-TLS access; the log group has 7-day retention; nothing is public.

Step-by-step implementation

Part A: run the tests (free)

Run the unit and API tests in Docker

From labs/m11-observe-banking-api, with your own NuGet cache volume:

MSYS_NO_PATHCONV=1 docker run --rm --memory 3g \
  -v "$PWD:/src" -v cea-nuget-m11:/root/.nuget/packages \
  -w /src mcr.microsoft.com/dotnet/sdk:10.0 \
  sh -c "dotnet test --solution ObserveBankingApi.slnx; find . -type d \( -name bin -o -name obj \) -prune -exec rm -rf {} +"

Expected on 2026-11-18: total: 61, failed: 0, succeeded: 60, skipped: 1. The skipped test needs RUN_AWS_TESTS=1 and a deployed stack. dotnet test does not compile the CDK project; dotnet build ObserveBankingApi.slnx does.

Read what the tests prove

Open tests/Harbourline.Observability.Tests. LoggingTests.cs shows redaction and correlation IDs; MetricsAndTracingTests.cs shows the EMF structure, cardinality guard, trace ID conversion and the saved-query checks; AlarmTests.cs shows M-out-of-N and composite rules; ApiTests.cs shows each outcome of the endpoint and the metric it emits.

Part B: traces, logs and faults on your machine (Lab 11.1, free)

Start the stack

docker compose up --build

This builds the API image, starts the collector and Jaeger, and exposes the API on http://localhost:8080 and the Jaeger UI on http://localhost:16686.

Send healthy traffic and read one line of each signal

bash scripts/load.sh http://localhost:8080 10
docker compose logs api --no-log-prefix | tail -n 6

Find (a) a request-completed line with method, path, statusCode, elapsedMs, correlationId; (b) an EMF line starting with {"_aws":; (c) the line "Transfer requested, reference ..." and check that the phone number in the reference appears as [phone]. The script puts +44 7700 900123 in every reference on purpose.

Read a normal trace

Open Jaeger, choose the service harbourline-transfers-api, and Find Traces. Open one: you should see a server span POST /v1/transfers with two children, partner.sanctions-screen (about 8 ms) and db.insert transfers (about 5 ms). Note the trace ID, then find the same ID in the API logs (traceId).

Inject fault 1: a slow database

bash scripts/load.sh http://localhost:8080 10 slow-db

In Jaeger, open one of the new traces. The db.insert transfers span is now about 800 ms and the server span is as long. Because only the database span grew, you know the partner and the API code are healthy. Check the log: elapsedMs for the request is about 800.

Inject fault 2: a partner outage

bash scripts/load.sh http://localhost:8080 10 partner-down

Every request returns HTTP 502. In Jaeger the partner.sanctions-screen span is marked as an error and there is no database span, because the code never reached it. In the logs, an Error line "Transfer failed: sanctions partner unavailable" appears with the same correlationId as the request-completed line.

Follow one failed request from the client

curl -i -X POST http://localhost:8080/v1/transfers \
  -H "Content-Type: application/json" -H "Idempotency-Key: demo-0001" -H "X-Fault: partner-down" \
  -d '{"debtorAccountId":"ACC-0000000001","creditorName":"Example Landlord Ltd","creditorIban":"GB29NWBK60161331926819","amount":50,"currency":"GBP","reference":"test"}'

The response carries X-Correlation-ID and a ProblemDetails body with the same correlationId. This is what a customer's support agent would read out to you. Search the logs with it: docker compose logs api --no-log-prefix | grep <correlation-id>.

Part C: deploy the AWS side (Labs 11.2 and 11.3, credits)

Sign in and choose a tool

aws sso login --profile learn-sandbox
aws sts get-caller-identity --profile learn-sandbox

Use either CDK or Terraform; both create the same resources. Use your own email address for the alarm subscription, or an address you control.

cd infra/cdk
cdk deploy -c alarmEmail=you@example.com -c owner=your-name --profile learn-sandbox

The first deployment in a new account needs cdk bootstrap. If cdk synth reports unacknowledged cdk-nag findings, read the report and acknowledge only with a written reason; the test run compiled the project but did not run a synth.

Confirm the SNS subscription

Open the email "AWS Notification - Subscription Confirmation" and click Confirm subscription. Until you do, the page cannot reach you.

Check what was created

aws logs describe-log-groups --log-group-name-prefix /hl-m11 --profile learn-sandbox \
  --query 'logGroups[].[logGroupName,retentionInDays]' --output table
aws cloudwatch describe-alarms --alarm-name-prefix hl-m11 --alarm-types MetricAlarm CompositeAlarm --profile learn-sandbox \
  --query '[MetricAlarms[].AlarmName, CompositeAlarms[].AlarmName]'

Expect one log group with retention 7 and five alarms in all (four metric alarms and the composite hl-m11-page). Newly created metric alarms may show INSUFFICIENT_DATA or OK until data arrives.

Part D: signals in CloudWatch, queries and a page rehearsal

Push the local logs into the log group

bash scripts/push-logs.sh /hl-m11/transfers-api learn-sandbox

The script wraps each JSON line from docker compose logs api as a log event and sends one batch. EMF extraction creates the metrics from the lines that carry an _aws block. This replaces running a container in ECS for the lab. Wait a minute or two, then open CloudWatch → Metrics → All metrics → Harbourline/Payments and confirm TransferCount and TransferLatency exist, under both dimension sets. Because all events share one send time, graphs show one spike, which is fine for the lab.

Run the five incident queries

In CloudWatch → Logs Insights, select /hl-m11/transfers-api, set the time range to the last hour and run each query from Logs Insights queries:

  1. Follow one request: paste a correlationId from a 502 response.
  2. Error rate: expect statusClass 2 and 5 rows.
  3. Latency by path: the p99 of /v1/transfers should be near 800 ms (the slow-db run).
  4. Top errors: "Transfer failed: sanctions partner unavailable" at the top.
  5. Slowest requests: the slow-db requests, with traceId values you can find in Jaeger.

Write down for each query what question it answered and what you did with the result.

Rehearse the page

aws cloudwatch set-alarm-state --alarm-name hl-m11-failed-transfers --state-value ALARM \
  --state-reason "Lab 11 game day rehearsal" --profile learn-sandbox
aws cloudwatch describe-alarm-history --alarm-name hl-m11-page --history-item-type StateUpdate --max-items 5 --profile learn-sandbox

Within a minute the composite hl-m11-page should move to ALARM and an email arrives. Notice that only the composite emailed you, even though a child alarm changed state: that is the "one incident, one page" design. The next evaluation resets the child's state, and you may get an OK email.

Open the dashboard

CloudWatch → Dashboards → hl-m11-transfers. Identify, in reading order: the alarm status, transfers by outcome, p99 latency and the latest-errors log table.

Write the mini post-incident review

In a file review.md, write six lines: what you injected, how the alarms and signals showed it, which query or trace found the cause, how long that took you, what you would add to the runbook (docs/runbooks/failed-transfers.md), and one thing that was not observable that you wish were. This is the seed of the module project's deliverable.

AWS Console instructions

  • Alarms: Console → CloudWatch → Alarms → All alarms, filter by hl-m11. Open hl-m11-page and read Alarm rule and Actions.
  • Metrics: CloudWatch → Metrics → All metrics → Harbourline/Payments.
  • Logs: CloudWatch → Logs → Log management → /hl-m11/transfers-api; Insights under Logs Insights.
  • Dashboard: CloudWatch → Dashboards → hl-m11-transfers.
  • SNS: Console → Amazon SNS → Topics → hl-m11-oncall-page → Subscriptions should show Confirmed.

Console labels change from time to time ⚠ VERIFY.

AWS CLI

# Prove what exists, by tag, before and after the lab
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --query 'ResourceTagMappingList[].ResourceARN' --profile learn-sandbox

# The series you are paying for
aws cloudwatch list-metrics --namespace Harbourline/Payments --profile learn-sandbox \
  --query 'Metrics[].[MetricName,Dimensions[].Value|join(`,`,@)]' --output table

# A quick Insights check from the CLI (last hour)
QID=$(aws logs start-query --log-group-name /hl-m11/transfers-api --profile learn-sandbox \
  --start-time $(date -d '-1 hour' +%s) --end-time $(date +%s) \
  --query-string 'filter level = "Error" | stats count(*) as occurrences by message | sort occurrences desc | limit 5' \
  --query queryId --output text)
sleep 5
aws logs get-query-results --query-id "$QID" --profile learn-sandbox --query 'results'

Infrastructure as Code

The stack is labs/m11-observe-banking-api/infra. The relevant resources, shown in lesson 5, are the log group (/hl-m11/transfers-api, 7 days), the SNS topic hl-m11-oncall-page (TLS-only), four alarms (hl-m11-failed-transfers, hl-m11-traffic-floor, hl-m11-p99-latency, hl-m11-failure-rate), the composite hl-m11-page (the only alarm with an action) and the dashboard hl-m11-transfers. Compare the two tools side by side: the CDK stack has a MathExpression for the failure rate and the Terraform configuration has metric_query blocks for the same expression. Both carry the mandatory tags (Project=cea-lab, Lab=m11-observe-banking-api, Environment=learn, Owner).

Complete code

Everything lives in labs/m11-observe-banking-api:

PathWhat it is
src/Harbourline.ObservabilityFormatter, PII redactor, correlation ID, EMF writer, alarm simulator, composite rule parser, X-Ray trace ID helper, incident queries
src/Harbourline.Transfers.ApiThe API: endpoint, fault injection, OpenTelemetry
tests/Harbourline.Observability.Tests61 tests (60 run offline, 1 gated)
infra/cdk, infra/terraformThe AWS side in two tools
dashboards/transfers.json.tftplDashboard JSON (also usable with put-dashboard)
otel/collector-config.yaml, docker-compose.yml, DockerfileThe local trace pipeline
scripts/load.sh, scripts/push-logs.shTraffic with faults, and log push
docs/runbooks/failed-transfers.mdThe runbook the alarm descriptions point to

The endpoint, trimmed to its observable parts:

Program.cs (excerpt)
app.MapPost("/v1/transfers", async (
    [FromBody] TransferRequest request, HttpContext http, TransferStore store, SanctionsPartner partner,
    BankingMetrics metrics, IOptions<FaultSettings> faultOptions, ILogger<Program> logger, CancellationToken ct) =>
{
    var started = Stopwatch.GetTimestamp();
    var correlationId = CorrelationMiddleware.Get(http);
    // ... fault switches (lab only), Idempotency-Key check, validation ...
    try
    {
        logger.LogInformation("Transfer requested, reference {Reference}", request.Reference);   // redacted by the formatter

        await partner.ScreenAsync(request.CreditorName!, failureRate, ct);                       // span: partner.sanctions-screen
        var paymentId = await store.SaveAsync(request, slowMs, ct);                              // span: db.insert transfers

        metrics.RecordTransfer("succeeded", ElapsedMs(), correlationId, "FPS");                  // EMF line
        return Results.Created($"/v1/transfers/{paymentId}", new TransferAccepted(paymentId, "received", request.Amount, request.Currency!));
    }
    catch (PartnerUnavailableException ex)
    {
        metrics.RecordTransfer("failed", ElapsedMs(), correlationId, "FPS");                     // EMF line, Outcome=failed
        logger.LogError(ex, "Transfer failed: sanctions partner unavailable");
        return Results.Problem(title: "Payment could not be screened right now", statusCode: StatusCodes.Status502BadGateway,
            extensions: new Dictionary<string, object?> { ["correlationId"] = correlationId });
    }
});

Code explained line by line

  1. CorrelationMiddleware.Get(http) reads the ID the middleware stored for this request; the same ID is in the logging scope, the response header, the ProblemDetails and the EMF line.
  2. LogInformation("... {Reference}", request.Reference) logs a customer-typed free-text value. The formatter redacts it. The lab does this deliberately; real code would not log free text.
  3. partner.ScreenAsync and store.SaveAsync each start their own span. When the partner fails, the exception stops execution before SaveAsync, which is why the failed trace has no database span.
  4. metrics.RecordTransfer("succeeded", ...) writes one EMF line. The outcome is checked against a fixed list, so dimension cardinality is bounded.
  5. The catch handles only PartnerUnavailableException, records the failed outcome, logs an error with the exception (type and redacted message), and returns a 502 ProblemDetails containing the correlation ID, with no internal detail.
  6. Anything else escapes to GlobalExceptionHandler, which logs the unhandled exception once and returns a safe 500.
  7. The elapsed time uses Stopwatch.GetTimestamp() taken at the start, so the latency covers everything from the endpoint's first line to the outcome.

Security considerations

⚠ Learning shortcut — not for production

Faults__AllowHeader=true lets any caller inject failures with a header; the stack uses an unconfirmed email subscription route, a 7-day log group that is destroyed with the stack, plain HTTP between containers and 100% trace sampling. In production: no caller-controlled fault switches (use a feature-flag service with access control), a paging tool subscribed over HTTPS, retention agreed with your data-protection officer, TLS between components and sampling with error retention.

  • The formatter redacts IBANs, Luhn-valid card numbers, phone numbers and emails, and hides any property whose name is sensitive. It is a safety net; a data-protection review decides what may be logged at all.
  • The log group and topic are private to the sandbox account. The topic policy allows only this account's hl-m11-* alarms to publish and denies non-TLS.
  • push-logs.sh pushes whatever the container wrote. Do not point it at a container that handled real data.
  • The AWS side uses your SSO session; no access keys are created or stored.

Banking use case

This lab is the platform team's observability baseline for any new payment endpoint: the same formatter, the same EMF vocabulary (Outcome), the same two paging alarms plus a composite, a dashboard in reading order and a runbook link in each alarm description. A service does not go live until its game day has shown that a partner outage pages someone within five minutes and that the on-call engineer can find the cause with the saved queries. All names and data are fictional.

Common mistakes

  • Pushing logs, then expecting graphs instantly. EMF extraction and alarms lag by a minute or more.
  • Dimension mismatch. An alarm that reads transfer-api while the code publishes transfers-api stays in INSUFFICIENT_DATA. Compare with list-metrics.
  • Forgetting to confirm the SNS subscription. The composite fires and nothing arrives.
  • Leaving the stack running. Alarms, a dashboard beyond the free three, and logs accumulate cost. Destroy it.
  • Running the compose stack with real data. It is a lab; use fictional data only.
  • Treating the local fault switch as a feature. It is a teaching device.

Troubleshooting

SymptomLikely causeFix
docker compose up fails on the API buildDocker out of memory or no network for NuGetClose other containers; retry; check docker compose build --no-cache api
Jaeger shows no harbourline-transfers-api serviceCollector not ready or wrong endpointdocker compose logs collector; restart with docker compose restart api
load.sh prints HTTP 000API not listening yetWait a few seconds; curl localhost:8080/health
Every response is 400Missing Idempotency-Key header (the script sets one)Use the script or add the header
push-logs.sh: no JSON linesThe api container has no logs (recreated)Send traffic again, then push
Metrics never appearEMF line altered (for example a prefixed line), or the push used the wrong groupLook at the event in the log group; it must be the raw JSON line
Alarm stays INSUFFICIENT_DATANo data yet or mismatched dimensionsWait; compare dimensions with list-metrics
No page emailSubscription unconfirmed or topic policy wrongConfirm the subscription; check describe-alarm-history for failed actions
cdk deploy fails on bootstrapAccount not bootstrappedcdk bootstrap aws://<account>/us-east-1 --profile learn-sandbox
terraform apply asks for ownerRequired variablePass -var owner=your-name

Cost and cleanup

Cost. Parts A and B are free. Part C and D at lab volumes: log ingestion of a few kilobytes (inside the 5 GB free allowance), 8 custom metrics (inside the free tier of 10 if nothing else uses it, otherwise USD 2.40 per month at list price), five alarms (at most USD 0.90 per month at list price if they ran for a month; about nothing for an hour), one dashboard (free tier covers three), an SNS topic with one email subscription (free). All prices ⚠ VERIFY on the CloudWatch pricing page, checked 2026-11-18. Destroy the same day.

Teardown (run every step).

# 1. Destroy the AWS stack (use the tool you deployed with)
cd infra/cdk && cdk destroy --profile learn-sandbox        # or:
cd infra/terraform && terraform destroy -var owner=your-name -var alarm_email=you@example.com

# 2. Stop and remove the local containers and images
cd ../.. && docker compose down --rmi local

# 3. Confirm nothing is left in AWS
aws logs describe-log-groups --log-group-name-prefix /hl-m11 --profile learn-sandbox
aws cloudwatch describe-alarms --alarm-name-prefix hl-m11 --profile learn-sandbox --query 'length(MetricAlarms) + length(CompositeAlarms)'
aws cloudwatch list-dashboards --dashboard-name-prefix hl-m11 --profile learn-sandbox
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab --profile learn-sandbox

Expected: no log group, 0 alarms, no dashboards and no resource from this lab in the tag listing. The custom metrics in Harbourline/Payments stay visible in the console for a while after you stop publishing and age out on their own; you cannot delete them ⚠ VERIFY. Also delete the saved Insights queries if you created any (see lesson 3), and the unconfirmed or confirmed email subscription disappears with the topic.

Exercises

  • 🟢 Guided: Complete Parts A to D, then fill in this table from your own run: fault, first signal that showed it, signal that found the cause, time from injection to cause.
  • 🟡 Challenge: Add a third fault slow-partner (partner call takes 3 seconds) to FaultSettings and the endpoint, with a test in ApiTests.cs. Predict what the trace and the p99 metric will show, then verify. Decide which of your alarms fires and which should.
  • 🔴 Live problem: The on-call engineer got paged at 09:07 by hl-m11-page, but when they opened the dashboard hl-m11-failed-transfers was already back in OK. By the time they ran the queries the errors had stopped. Write what you would change in the alarms, the dashboard and the saved queries so that the evidence of a short incident is still easy to find after it ends (think time ranges, trace sampling for errors, log retention and a state-change history).

Assignment

Deliver the first half of the module project in your portfolio repository: your review.md from the game day, an alarm table (alarm, metric, M of N, missing-data policy, page or ticket, reason), the dashboard JSON, a screenshot description of a normal, a slow and a failed Jaeger trace, and a RUNBOOK.md that extends failed-transfers.md with the commands you actually used. Add a note on one change you would make to the telemetry (a new field, metric or span) and why.

Solutions

Knowledge check

Interview questions

  • Describe a game day you would run for a payments API. Define the hypothesis (alarm pages within five minutes), inject a safe fault (partner outage in a test environment), observe alarms, logs and traces, time the diagnosis, and record gaps as actions.
  • How would you prove to an auditor that your alerting works? Keep alarm definitions in Git, run scheduled rehearsals with set-alarm-state and real faults, and retain the alarm history and the rehearsal records.
  • What telemetry would you add to a new payment endpoint on day one? A correlation ID, one request-completed log line, an outcome metric with bounded dimensions, spans around dependencies, health checks and an alarm per customer-visible symptom.
  • How do you separate a slow dependency from a failing one? Latency metrics and slow traces show slow; error outcomes and red spans show failing; the alarm design reflects the difference.
  • What stays out of logs and traces in a bank? Card numbers, full IBANs, names, tokens, authorisation headers, free-text customer input and request bodies.

Certification practice

Completion checklist

  • Tests passed in Docker (60 passed, 1 skipped)
  • I read a normal, a slow-database and a partner-outage trace in Jaeger
  • I confirmed the phone number appears as [phone] in the logs
  • I deployed the stack (or read the IaC), ran the five queries and rehearsed the page
  • I wrote review.md with timings
  • I ran every teardown step and confirmed nothing tagged Project=cea-lab remains

Key takeaways

  • The three signals are most useful together: the metric tells you when, the log tells you which request, the trace tells you where.
  • A fixed metric vocabulary, a composite alarm and a runbook link are what make an alert actionable.
  • Practise failure on purpose: a rehearsed page and a timed diagnosis are worth more than a clever dashboard.
  • Keep the lab cheap: local first, a few kilobytes in AWS, destroy the same day.
  • Telemetry is data: redact, retain briefly and restrict.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page