Lab 11: observe a banking API (logs, metrics, alarms, dashboard and traces)
Run the Harbourline Transfers API with structured logs, EMF metrics and OpenTelemetry traces, inject a slow database and a partner outage, find each cause with Logs Insights and Jaeger, then deploy alarms, a composite page and a dashboard with CDK or Terraform and tear everything down.
Assumes: .NET 10 LTS · OpenTelemetry .NET 1.21.0 · Amazon.CDK.Lib 2.277.0 · Cdklabs.CdkNag 3.0.2 · Terraform AWS provider v6 · Docker · Region us-east-1 · labs/m11-observe-banking-api. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lab you can:
- Run the Transfers API with flat JSON logs, a correlation ID, PII redaction, EMF business metrics and OpenTelemetry traces, and read each signal.
- Inject two faults (a slow database and a partner outage) and find the cause of each from a trace and from a Logs Insights query.
- Deploy a log group with 7-day retention, an SNS page topic, four metric alarms, a composite page alarm and a dashboard from CDK (C#) or Terraform.
- Rehearse a page with
set-alarm-stateand read the alarm history. - Tear everything down and prove that nothing tagged
Project=cea-labremains.
Prerequisites
Lessons 1 to 7 of this module:
- Observability fundamentals
- CloudWatch Logs for .NET
- Logs Insights queries
- CloudWatch metrics and custom metrics
- Alarms and composite alarms
- Dashboards and Synthetics canaries
- Distributed tracing with X-Ray and OpenTelemetry
Tools: Docker Desktop. For the AWS part: AWS CLI v2 with aws sso login --profile learn-sandbox, jq, and either Node.js LTS with the CDK CLI (and a bootstrapped account) or Terraform 1.14. The .NET SDK is optional because every command also runs in mcr.microsoft.com/dotnet/sdk:10.0.
Why this matters in banking
The module project asks you to "observe and fix a banking API". This lab is the first half: make the Transfers API observable, break it on purpose, and practise finding the cause from the data rather than from guesses. The same exercise is how real teams run game days: a planned, safe failure that checks whether the alarms fire, whether the runbook works and whether on-call can find the cause quickly. Banks that must demonstrate operational resilience run these regularly and keep the results as evidence.
The idea in plain language
You will run the API on your machine. It writes three kinds of telemetry:
- Logs to stdout: one flat JSON object per line (with a correlation ID, and a trace ID when inside a request).
- Metrics to stdout, as Embedded Metric Format lines that CloudWatch Logs would turn into metrics.
- Traces over OTLP to a collector and Jaeger.
You then break it twice, find the causes, and deploy the AWS side (log group, alarms, dashboard) so the same signals would page a person if this were running in a real account.
Analogy: Think of fitting a test car with a dashboard, a black-box recorder and a GPS tracker, then driving it over a rough road on purpose to see whether the warning lights come on, and whether the recorder tells you which wheel hit the pothole.
What problem the AWS service solves
Locally you practise reading signals. In AWS the same lines feed CloudWatch Logs, CloudWatch metrics and alarms, X-Ray and dashboards. The lab is split so that most of it is free:
| Part | Cost label | What runs |
|---|---|---|
| A. Unit tests | 🟢 free | Docker only |
| B. Local traces and logs, fault injection | 🟢 free | Docker Compose (API, collector, Jaeger) |
| C. Deploy alarms, topic and dashboard | 🟡 credits | CloudWatch, SNS (minutes; destroy the same day) |
| D. Logs Insights and the page rehearsal | 🟡 credits | A few kilobytes of logs |
If you do not want to deploy anything, do parts A and B and read the infrastructure code for part C (a free design-only alternative).
Microsoft / Azure comparison
| You know (Microsoft/Azure) | Lab equivalent | What's different |
|---|---|---|
| Application Insights end-to-end transaction view | Jaeger trace view locally; X-Ray in AWS | Same OpenTelemetry data, different backend |
| Azure Monitor metric alert + action group | CloudWatch alarm + composite + SNS | The composite is the one object that notifies |
| Log Analytics saved queries | IncidentQueries deployed as query definitions | Insights QL, not KQL |
| Chaos Studio experiments | X-Fault header and Faults__* settings | A tiny in-app switch; AWS Fault Injection Service is the managed option (Module 12) ⚠ VERIFY |
Architecture
Security-boundary view. Everything local is on your machine. The AWS side is private to your sandbox account: the SNS topic allows only CloudWatch alarms from this account to publish and denies non-TLS access; the log group has 7-day retention; nothing is public.
Step-by-step implementation
Part A: run the tests (free)
Run the unit and API tests in Docker
From labs/m11-observe-banking-api, with your own NuGet cache volume:
MSYS_NO_PATHCONV=1 docker run --rm --memory 3g \
-v "$PWD:/src" -v cea-nuget-m11:/root/.nuget/packages \
-w /src mcr.microsoft.com/dotnet/sdk:10.0 \
sh -c "dotnet test --solution ObserveBankingApi.slnx; find . -type d \( -name bin -o -name obj \) -prune -exec rm -rf {} +"Expected on 2026-11-18: total: 61, failed: 0, succeeded: 60, skipped: 1. The skipped test needs RUN_AWS_TESTS=1 and a deployed stack. dotnet test does not compile the CDK project; dotnet build ObserveBankingApi.slnx does.
Read what the tests prove
Open tests/Harbourline.Observability.Tests. LoggingTests.cs shows redaction and correlation IDs; MetricsAndTracingTests.cs shows the EMF structure, cardinality guard, trace ID conversion and the saved-query checks; AlarmTests.cs shows M-out-of-N and composite rules; ApiTests.cs shows each outcome of the endpoint and the metric it emits.
Part B: traces, logs and faults on your machine (Lab 11.1, free)
Start the stack
docker compose up --buildThis builds the API image, starts the collector and Jaeger, and exposes the API on http://localhost:8080 and the Jaeger UI on http://localhost:16686.
Send healthy traffic and read one line of each signal
bash scripts/load.sh http://localhost:8080 10
docker compose logs api --no-log-prefix | tail -n 6Find (a) a request-completed line with method, path, statusCode, elapsedMs, correlationId; (b) an EMF line starting with {"_aws":; (c) the line "Transfer requested, reference ..." and check that the phone number in the reference appears as [phone]. The script puts +44 7700 900123 in every reference on purpose.
Read a normal trace
Open Jaeger, choose the service harbourline-transfers-api, and Find Traces. Open one: you should see a server span POST /v1/transfers with two children, partner.sanctions-screen (about 8 ms) and db.insert transfers (about 5 ms). Note the trace ID, then find the same ID in the API logs (traceId).
Inject fault 1: a slow database
bash scripts/load.sh http://localhost:8080 10 slow-dbIn Jaeger, open one of the new traces. The db.insert transfers span is now about 800 ms and the server span is as long. Because only the database span grew, you know the partner and the API code are healthy. Check the log: elapsedMs for the request is about 800.
Inject fault 2: a partner outage
bash scripts/load.sh http://localhost:8080 10 partner-downEvery request returns HTTP 502. In Jaeger the partner.sanctions-screen span is marked as an error and there is no database span, because the code never reached it. In the logs, an Error line "Transfer failed: sanctions partner unavailable" appears with the same correlationId as the request-completed line.
Follow one failed request from the client
curl -i -X POST http://localhost:8080/v1/transfers \
-H "Content-Type: application/json" -H "Idempotency-Key: demo-0001" -H "X-Fault: partner-down" \
-d '{"debtorAccountId":"ACC-0000000001","creditorName":"Example Landlord Ltd","creditorIban":"GB29NWBK60161331926819","amount":50,"currency":"GBP","reference":"test"}'The response carries X-Correlation-ID and a ProblemDetails body with the same correlationId. This is what a customer's support agent would read out to you. Search the logs with it: docker compose logs api --no-log-prefix | grep <correlation-id>.
Part C: deploy the AWS side (Labs 11.2 and 11.3, credits)
Sign in and choose a tool
aws sso login --profile learn-sandbox
aws sts get-caller-identity --profile learn-sandboxUse either CDK or Terraform; both create the same resources. Use your own email address for the alarm subscription, or an address you control.
cd infra/cdk
cdk deploy -c alarmEmail=you@example.com -c owner=your-name --profile learn-sandboxThe first deployment in a new account needs cdk bootstrap. If cdk synth reports unacknowledged cdk-nag findings, read the report and acknowledge only with a written reason; the test run compiled the project but did not run a synth.
Confirm the SNS subscription
Open the email "AWS Notification - Subscription Confirmation" and click Confirm subscription. Until you do, the page cannot reach you.
Check what was created
aws logs describe-log-groups --log-group-name-prefix /hl-m11 --profile learn-sandbox \
--query 'logGroups[].[logGroupName,retentionInDays]' --output table
aws cloudwatch describe-alarms --alarm-name-prefix hl-m11 --alarm-types MetricAlarm CompositeAlarm --profile learn-sandbox \
--query '[MetricAlarms[].AlarmName, CompositeAlarms[].AlarmName]'Expect one log group with retention 7 and five alarms in all (four metric alarms and the composite hl-m11-page). Newly created metric alarms may show INSUFFICIENT_DATA or OK until data arrives.
Part D: signals in CloudWatch, queries and a page rehearsal
Push the local logs into the log group
bash scripts/push-logs.sh /hl-m11/transfers-api learn-sandboxThe script wraps each JSON line from docker compose logs api as a log event and sends one batch. EMF extraction creates the metrics from the lines that carry an _aws block. This replaces running a container in ECS for the lab. Wait a minute or two, then open CloudWatch → Metrics → All metrics → Harbourline/Payments and confirm TransferCount and TransferLatency exist, under both dimension sets. Because all events share one send time, graphs show one spike, which is fine for the lab.
Run the five incident queries
In CloudWatch → Logs Insights, select /hl-m11/transfers-api, set the time range to the last hour and run each query from Logs Insights queries:
- Follow one request: paste a
correlationIdfrom a 502 response. - Error rate: expect statusClass
2and5rows. - Latency by path: the p99 of
/v1/transfersshould be near 800 ms (theslow-dbrun). - Top errors: "Transfer failed: sanctions partner unavailable" at the top.
- Slowest requests: the
slow-dbrequests, withtraceIdvalues you can find in Jaeger.
Write down for each query what question it answered and what you did with the result.
Rehearse the page
aws cloudwatch set-alarm-state --alarm-name hl-m11-failed-transfers --state-value ALARM \
--state-reason "Lab 11 game day rehearsal" --profile learn-sandbox
aws cloudwatch describe-alarm-history --alarm-name hl-m11-page --history-item-type StateUpdate --max-items 5 --profile learn-sandboxWithin a minute the composite hl-m11-page should move to ALARM and an email arrives. Notice that only the composite emailed you, even though a child alarm changed state: that is the "one incident, one page" design. The next evaluation resets the child's state, and you may get an OK email.
Open the dashboard
CloudWatch → Dashboards → hl-m11-transfers. Identify, in reading order: the alarm status, transfers by outcome, p99 latency and the latest-errors log table.
Write the mini post-incident review
In a file review.md, write six lines: what you injected, how the alarms and signals showed it, which query or trace found the cause, how long that took you, what you would add to the runbook (docs/runbooks/failed-transfers.md), and one thing that was not observable that you wish were. This is the seed of the module project's deliverable.
AWS Console instructions
- Alarms: Console → CloudWatch → Alarms → All alarms, filter by
hl-m11. Openhl-m11-pageand read Alarm rule and Actions. - Metrics: CloudWatch → Metrics → All metrics → Harbourline/Payments.
- Logs: CloudWatch → Logs → Log management → /hl-m11/transfers-api; Insights under Logs Insights.
- Dashboard: CloudWatch → Dashboards → hl-m11-transfers.
- SNS: Console → Amazon SNS → Topics → hl-m11-oncall-page → Subscriptions should show Confirmed.
Console labels change from time to time ⚠ VERIFY.
AWS CLI
# Prove what exists, by tag, before and after the lab
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--query 'ResourceTagMappingList[].ResourceARN' --profile learn-sandbox
# The series you are paying for
aws cloudwatch list-metrics --namespace Harbourline/Payments --profile learn-sandbox \
--query 'Metrics[].[MetricName,Dimensions[].Value|join(`,`,@)]' --output table
# A quick Insights check from the CLI (last hour)
QID=$(aws logs start-query --log-group-name /hl-m11/transfers-api --profile learn-sandbox \
--start-time $(date -d '-1 hour' +%s) --end-time $(date +%s) \
--query-string 'filter level = "Error" | stats count(*) as occurrences by message | sort occurrences desc | limit 5' \
--query queryId --output text)
sleep 5
aws logs get-query-results --query-id "$QID" --profile learn-sandbox --query 'results'Infrastructure as Code
The stack is labs/m11-observe-banking-api/infra. The relevant resources, shown in lesson 5, are the log group (/hl-m11/transfers-api, 7 days), the SNS topic hl-m11-oncall-page (TLS-only), four alarms (hl-m11-failed-transfers, hl-m11-traffic-floor, hl-m11-p99-latency, hl-m11-failure-rate), the composite hl-m11-page (the only alarm with an action) and the dashboard hl-m11-transfers. Compare the two tools side by side: the CDK stack has a MathExpression for the failure rate and the Terraform configuration has metric_query blocks for the same expression. Both carry the mandatory tags (Project=cea-lab, Lab=m11-observe-banking-api, Environment=learn, Owner).
Complete code
Everything lives in labs/m11-observe-banking-api:
| Path | What it is |
|---|---|
src/Harbourline.Observability | Formatter, PII redactor, correlation ID, EMF writer, alarm simulator, composite rule parser, X-Ray trace ID helper, incident queries |
src/Harbourline.Transfers.Api | The API: endpoint, fault injection, OpenTelemetry |
tests/Harbourline.Observability.Tests | 61 tests (60 run offline, 1 gated) |
infra/cdk, infra/terraform | The AWS side in two tools |
dashboards/transfers.json.tftpl | Dashboard JSON (also usable with put-dashboard) |
otel/collector-config.yaml, docker-compose.yml, Dockerfile | The local trace pipeline |
scripts/load.sh, scripts/push-logs.sh | Traffic with faults, and log push |
docs/runbooks/failed-transfers.md | The runbook the alarm descriptions point to |
The endpoint, trimmed to its observable parts:
app.MapPost("/v1/transfers", async (
[FromBody] TransferRequest request, HttpContext http, TransferStore store, SanctionsPartner partner,
BankingMetrics metrics, IOptions<FaultSettings> faultOptions, ILogger<Program> logger, CancellationToken ct) =>
{
var started = Stopwatch.GetTimestamp();
var correlationId = CorrelationMiddleware.Get(http);
// ... fault switches (lab only), Idempotency-Key check, validation ...
try
{
logger.LogInformation("Transfer requested, reference {Reference}", request.Reference); // redacted by the formatter
await partner.ScreenAsync(request.CreditorName!, failureRate, ct); // span: partner.sanctions-screen
var paymentId = await store.SaveAsync(request, slowMs, ct); // span: db.insert transfers
metrics.RecordTransfer("succeeded", ElapsedMs(), correlationId, "FPS"); // EMF line
return Results.Created($"/v1/transfers/{paymentId}", new TransferAccepted(paymentId, "received", request.Amount, request.Currency!));
}
catch (PartnerUnavailableException ex)
{
metrics.RecordTransfer("failed", ElapsedMs(), correlationId, "FPS"); // EMF line, Outcome=failed
logger.LogError(ex, "Transfer failed: sanctions partner unavailable");
return Results.Problem(title: "Payment could not be screened right now", statusCode: StatusCodes.Status502BadGateway,
extensions: new Dictionary<string, object?> { ["correlationId"] = correlationId });
}
});Code explained line by line
CorrelationMiddleware.Get(http)reads the ID the middleware stored for this request; the same ID is in the logging scope, the response header, the ProblemDetails and the EMF line.LogInformation("... {Reference}", request.Reference)logs a customer-typed free-text value. The formatter redacts it. The lab does this deliberately; real code would not log free text.partner.ScreenAsyncandstore.SaveAsynceach start their own span. When the partner fails, the exception stops execution beforeSaveAsync, which is why the failed trace has no database span.metrics.RecordTransfer("succeeded", ...)writes one EMF line. The outcome is checked against a fixed list, so dimension cardinality is bounded.- The
catchhandles onlyPartnerUnavailableException, records thefailedoutcome, logs an error with the exception (type and redacted message), and returns a 502 ProblemDetails containing the correlation ID, with no internal detail. - Anything else escapes to
GlobalExceptionHandler, which logs the unhandled exception once and returns a safe 500. - The elapsed time uses
Stopwatch.GetTimestamp()taken at the start, so the latency covers everything from the endpoint's first line to the outcome.
Security considerations
⚠ Learning shortcut — not for production
Faults__AllowHeader=true lets any caller inject failures with a header; the stack uses an unconfirmed email subscription route, a 7-day log group that is destroyed with the stack, plain HTTP between containers and 100% trace sampling. In production: no caller-controlled fault switches (use a feature-flag service with access control), a paging tool subscribed over HTTPS, retention agreed with your data-protection officer, TLS between components and sampling with error retention.
- The formatter redacts IBANs, Luhn-valid card numbers, phone numbers and emails, and hides any property whose name is sensitive. It is a safety net; a data-protection review decides what may be logged at all.
- The log group and topic are private to the sandbox account. The topic policy allows only this account's
hl-m11-*alarms to publish and denies non-TLS. push-logs.shpushes whatever the container wrote. Do not point it at a container that handled real data.- The AWS side uses your SSO session; no access keys are created or stored.
Banking use case
This lab is the platform team's observability baseline for any new payment endpoint: the same formatter, the same EMF vocabulary (Outcome), the same two paging alarms plus a composite, a dashboard in reading order and a runbook link in each alarm description. A service does not go live until its game day has shown that a partner outage pages someone within five minutes and that the on-call engineer can find the cause with the saved queries. All names and data are fictional.
Common mistakes
- Pushing logs, then expecting graphs instantly. EMF extraction and alarms lag by a minute or more.
- Dimension mismatch. An alarm that reads
transfer-apiwhile the code publishestransfers-apistays inINSUFFICIENT_DATA. Compare withlist-metrics. - Forgetting to confirm the SNS subscription. The composite fires and nothing arrives.
- Leaving the stack running. Alarms, a dashboard beyond the free three, and logs accumulate cost. Destroy it.
- Running the compose stack with real data. It is a lab; use fictional data only.
- Treating the local fault switch as a feature. It is a teaching device.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
docker compose up fails on the API build | Docker out of memory or no network for NuGet | Close other containers; retry; check docker compose build --no-cache api |
Jaeger shows no harbourline-transfers-api service | Collector not ready or wrong endpoint | docker compose logs collector; restart with docker compose restart api |
load.sh prints HTTP 000 | API not listening yet | Wait a few seconds; curl localhost:8080/health |
| Every response is 400 | Missing Idempotency-Key header (the script sets one) | Use the script or add the header |
push-logs.sh: no JSON lines | The api container has no logs (recreated) | Send traffic again, then push |
| Metrics never appear | EMF line altered (for example a prefixed line), or the push used the wrong group | Look at the event in the log group; it must be the raw JSON line |
Alarm stays INSUFFICIENT_DATA | No data yet or mismatched dimensions | Wait; compare dimensions with list-metrics |
| No page email | Subscription unconfirmed or topic policy wrong | Confirm the subscription; check describe-alarm-history for failed actions |
cdk deploy fails on bootstrap | Account not bootstrapped | cdk bootstrap aws://<account>/us-east-1 --profile learn-sandbox |
terraform apply asks for owner | Required variable | Pass -var owner=your-name |
Cost and cleanup
Cost. Parts A and B are free. Part C and D at lab volumes: log ingestion of a few kilobytes (inside the 5 GB free allowance), 8 custom metrics (inside the free tier of 10 if nothing else uses it, otherwise USD 2.40 per month at list price), five alarms (at most USD 0.90 per month at list price if they ran for a month; about nothing for an hour), one dashboard (free tier covers three), an SNS topic with one email subscription (free). All prices ⚠ VERIFY on the CloudWatch pricing page, checked 2026-11-18. Destroy the same day.
Teardown (run every step).
# 1. Destroy the AWS stack (use the tool you deployed with)
cd infra/cdk && cdk destroy --profile learn-sandbox # or:
cd infra/terraform && terraform destroy -var owner=your-name -var alarm_email=you@example.com
# 2. Stop and remove the local containers and images
cd ../.. && docker compose down --rmi local
# 3. Confirm nothing is left in AWS
aws logs describe-log-groups --log-group-name-prefix /hl-m11 --profile learn-sandbox
aws cloudwatch describe-alarms --alarm-name-prefix hl-m11 --profile learn-sandbox --query 'length(MetricAlarms) + length(CompositeAlarms)'
aws cloudwatch list-dashboards --dashboard-name-prefix hl-m11 --profile learn-sandbox
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab --profile learn-sandboxExpected: no log group, 0 alarms, no dashboards and no resource from this lab in the tag listing. The custom metrics in Harbourline/Payments stay visible in the console for a while after you stop publishing and age out on their own; you cannot delete them ⚠ VERIFY. Also delete the saved Insights queries if you created any (see lesson 3), and the unconfirmed or confirmed email subscription disappears with the topic.
Exercises
- 🟢 Guided: Complete Parts A to D, then fill in this table from your own run: fault, first signal that showed it, signal that found the cause, time from injection to cause.
- 🟡 Challenge: Add a third fault
slow-partner(partner call takes 3 seconds) toFaultSettingsand the endpoint, with a test inApiTests.cs. Predict what the trace and the p99 metric will show, then verify. Decide which of your alarms fires and which should. - 🔴 Live problem: The on-call engineer got paged at 09:07 by
hl-m11-page, but when they opened the dashboardhl-m11-failed-transferswas already back inOK. By the time they ran the queries the errors had stopped. Write what you would change in the alarms, the dashboard and the saved queries so that the evidence of a short incident is still easy to find after it ends (think time ranges, trace sampling for errors, log retention and a state-change history).
Assignment
Deliver the first half of the module project in your portfolio repository: your review.md from the game day, an alarm table (alarm, metric, M of N, missing-data policy, page or ticket, reason), the dashboard JSON, a screenshot description of a normal, a slow and a failed Jaeger trace, and a RUNBOOK.md that extends failed-transfers.md with the commands you actually used. Add a note on one change you would make to the telemetry (a new field, metric or span) and why.
Solutions
Typical results. Slow database: the p99 latency alarm or the dashboard shows it first (latency rises; failures do not); the trace finds the cause (long db.insert span) in about two minutes; query 5 lists the slow requests with trace IDs. Partner outage: the failed-transfers alarm fires after three breaching minutes (in real traffic) and the composite pages; query 4 (top errors) names the partner failure in seconds; the trace shows the red partner span and no database span. Record your own timings; the exercise is about the habit of writing them down.
Add public int PartnerDelayMs { get; set; } to FaultSettings, a "slow-partner" header value that sets it to 3000, and pass it to ScreenAsync which does await Task.Delay(8 + partnerDelayMs, ct). The server span and the partner span both grow to about 3 seconds while db.insert stays at 5 ms, and p99 TransferLatency rises. The latency warning alarm should fire; the failed-transfers page should not, because nothing fails. If your page fires, you have found a design problem: slow is not failed. A test: send a request with the header and assert 201 and TransferLatency above 2,900 ms in the captured metric.
Short incidents leave little evidence unless you plan for it. (1) The dashboard should default to a time range that includes the incident (for example 3 hours), and the alarm status widget should show state history through the alarm history API or saved links ⚠ VERIFY. (2) Keep 100% of error traces and slow traces through tail sampling, so the evidence exists. (3) Put the incident window in the runbook as the first step: query the last 30 minutes, not "now". (4) Keep log retention long enough (7 days is short for a real service; 30 days is common) and export evidential logs. (5) Record alarm state transitions (EventBridge rule on alarm state change into a log or ticket) so the timeline survives. (6) Treat "alarm already green when I looked" as a signal that the alarm's period may be too short for on-call response, and consider whether a ticket-level follow-up is needed.
Knowledge check
Answer: the partner call failed first and threw, so the code never reached the database insert. The trace shows exactly how far the request got.
Answer: the composite alarm hl-m11-page. Child alarms change state but have no actions, so one incident gives one page.
Answer: to keep the lab cheap and fast. The production pipeline would use the awslogs driver; the observable data is identical.
Answer: alarms and graphs split by outcome (for example only failed), while the first set [Service, Environment] gives totals for rate calculations.
Answer: a caller-controlled fault switch is a vulnerability. It exists only as a lab device and is off by default.
Interview questions
- Describe a game day you would run for a payments API. Define the hypothesis (alarm pages within five minutes), inject a safe fault (partner outage in a test environment), observe alarms, logs and traces, time the diagnosis, and record gaps as actions.
- How would you prove to an auditor that your alerting works? Keep alarm definitions in Git, run scheduled rehearsals with
set-alarm-stateand real faults, and retain the alarm history and the rehearsal records. - What telemetry would you add to a new payment endpoint on day one? A correlation ID, one request-completed log line, an outcome metric with bounded dimensions, spans around dependencies, health checks and an alarm per customer-visible symptom.
- How do you separate a slow dependency from a failing one? Latency metrics and slow traces show slow; error outcomes and red spans show failing; the alarm design reflects the difference.
- What stays out of logs and traces in a bank? Card numbers, full IBANs, names, tokens, authorisation headers, free-text customer input and request bodies.
Certification practice
A team wants one notification per incident when either of two symptom alarms fires, and no notifications from the symptom alarms themselves. What should they configure?
A. SNS actions on both alarms.
B. A composite alarm with rule ALARM(a) OR ALARM(b) and the SNS action on the composite only.
C. A higher threshold on both alarms.
D. A CloudWatch dashboard.
Answer: B. The composite evaluates the children's states and notifies once; the children have no actions. A produces two notifications, C changes sensitivity not routing, D only displays.
A request returns HTTP 502. The logs show the correlation ID but no timing for the downstream call. What is the quickest way to see which dependency failed and for how long?
A. Increase the log level to Debug and redeploy. B. Open the distributed trace for the request using the trace ID in the log line. C. Add a CloudWatch alarm. D. Check CloudTrail.
Answer: B. A trace shows each dependency call's duration and status. A requires redeploying during an incident, C does not diagnose, D does not record application calls.
A team needs a failure rate alarm that does not fire when only a handful of requests flow at night. Which design works?
A. A static alarm on failed count above 1.
B. A metric-math alarm IF(total >= 20, 100 * failed / total, 0) with a percentage threshold and M-out-of-N evaluation.
C. Disable the alarm at night manually.
D. Alarm on CPU instead.
Answer: B. The guard returns 0 below a traffic floor so low volume cannot trigger the rate alarm. A fires on a single failure; C is manual and error-prone; D is a cause metric.
Completion checklist
- Tests passed in Docker (
60 passed, 1 skipped) - I read a normal, a slow-database and a partner-outage trace in Jaeger
- I confirmed the phone number appears as
[phone]in the logs - I deployed the stack (or read the IaC), ran the five queries and rehearsed the page
- I wrote
review.mdwith timings - I ran every teardown step and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- The three signals are most useful together: the metric tells you when, the log tells you which request, the trace tells you where.
- A fixed metric vocabulary, a composite alarm and a runbook link are what make an alert actionable.
- Practise failure on purpose: a rehearsed page and a timed diagnosis are worth more than a clever dashboard.
- Keep the lab cheap: local first, a few kilobytes in AWS, destroy the same day.
- Telemetry is data: redact, retain briefly and restrict.
Further reading
- Module 11 overview: the full lesson list and the module project.
- Amazon CloudWatch pricing: every price quoted in this lab.
- Using Amazon CloudWatch alarms: alarm behaviour used in Part D.
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Distributed tracing with OpenTelemetry and AWS X-Ray for .NET (ADOT)
Instrument an ASP.NET Core 10 API with the OpenTelemetry SDK, add your own spans, export to a local Jaeger and to AWS (X-Ray / CloudWatch traces) through the ADOT collector, sample sensibly, and jump from a log line to its trace.
CloudWatch Application Signals: service maps, SLOs and error budgets
Turn OpenTelemetry telemetry from an ASP.NET Core service into an automatic service map, standard Latency/Fault/Error metrics and service level objectives with burn-rate alarms, and know what it adds over raw X-Ray traces.