Skip to Content
Cloud Engineering Academy
11. Monitoring, Logging & Troubleshooting

Module 11: Monitoring, Logging & Troubleshooting

Make your .NET workloads observable with CloudWatch logs, metrics, alarms and dashboards, distributed tracing with X-Ray and OpenTelemetry, and CloudTrail for who-did-what. You then practise finding and fixing real production problems methodically, from thread-pool starvation to cold starts, and turn alerts into runbooks.

SOA-C03DOP-C02DVA-C03

Assumes: .NET 10 LTS · OpenTelemetry .NET / ADOT · AWS SDK for .NET v4 · Region us-east-1. Verified against official sources on 2026-11-18.

Make your .NET workloads observable with CloudWatch logs, metrics, alarms and dashboards, distributed tracing with X-Ray and OpenTelemetry, and CloudTrail for who-did-what. You then practise finding and fixing real production problems methodically, from thread-pool starvation to cold starts, and turn alerts into runbooks.

Estimated timeAbout 14 hours (lessons 8 h · labs 3 h · project 3 h)
Priority for youHigh — supports deployment (priority 1) and security detection (priority 3)
LevelIntermediate → advanced
CertificationsSOA-C03 (primary) · DOP-C02 · DVA-C03
Cost profile🟢 local OpenTelemetry + basic metrics · 🟡 log ingestion, custom metrics, canaries · 🔴 none required

Why this matters in banking

When the payments API slows down at 09:00 on salary day, the business wants an answer in minutes, not a war room that guesses. Operational-resilience expectations also mean you must detect incidents, know their customer impact and show regulators how quickly you recovered. Good observability lets you follow one failed transfer across the API, the queue and the database, while CloudTrail answers the separate question "who changed this?". Poor observability, by contrast, means noisy alerts that everyone ignores, or logs full of personal data that become a liability of their own.

From .NET/Azure to AWS

You know (Microsoft/Azure)AWS equivalentWhat's different
Application InsightsCloudWatch Application Signals + X-Ray (OpenTelemetry-based)Instrumentation is OpenTelemetry; ADOT is AWS's distribution
Log Analytics workspace + KQLCloudWatch Logs + Logs Insights query languageLog groups per app; retention is set per log group
Azure Monitor metrics and alertsCloudWatch metrics, alarms and composite alarmsAlarms notify through SNS; composite alarms cut noise
Custom metrics via TelemetryClientEmbedded Metric Format (EMF) or PutMetricDataEMF turns structured log lines into metrics cheaply
Availability testsCloudWatch Synthetics canariesScripted canaries run on a schedule
Azure Monitor workbooks / dashboardsCloudWatch dashboardsCross-account, cross-Region dashboards possible
Azure Activity LogAWS CloudTrailEvery API call, who made it and from where
Cost alertsAWS Budgets, Cost Anomaly Detection, Cost ExplorerCovered with FinOps in Module 15

Learning objectives

By the end of this module you can:

  • Ship structured JSON logs from ASP.NET Core to CloudWatch Logs with correlation IDs and sensible retention, without leaking personal data.
  • Query logs with Logs Insights to answer real incident questions quickly.
  • Publish custom business metrics with EMF and build alarms, composite alarms and dashboards that reflect customer impact.
  • Instrument a .NET 10 API with OpenTelemetry (ADOT) and follow a request end-to-end in X-Ray and Application Signals.
  • Use CloudTrail to answer "who did what, when" and distinguish audit trails from application logs.
  • Diagnose common performance bottlenecks: thread-pool starvation, connection-pool exhaustion, cold starts and N+1 queries.
  • Write on-call runbooks and run a structured troubleshooting process from symptom to root cause.

Prerequisites

How the pieces fit

Lessons

Work through them in this order. Lessons 1 to 7 build the instrumentation, then the lab is the hands-on checkpoint for those seven; lessons 9 to 15 add service-level views, audit, scale and the human side of operations.

  1. Observability fundamentals — logs, metrics and traces, the golden signals and what "actionable" really means.
  2. CloudWatch Logs for .NET — log groups, retention, structured JSON with ILogger/Serilog, correlation IDs and PII-safe logging.
  3. Logs Insights queries — filtering, parsing, statistics and five saved queries for common incident questions.
  4. CloudWatch metrics and custom metrics — namespaces, dimensions, resolution and the Embedded Metric Format from C#.
  5. Alarms and composite alarms — thresholds vs anomaly detection, missing-data handling and noise reduction.
  6. Dashboards and Synthetics canaries — an operational dashboard for a banking API and a scripted availability check.
  7. Distributed tracing with OpenTelemetry and X-Ray for .NET (ADOT) — instrumenting ASP.NET Core, HttpClient, the AWS SDK and EF Core.
  8. Lab 11: observe a banking API — the hands-on checkpoint for lessons 1 to 7 (see the labs section below).
  9. CloudWatch Application Signals — service maps, SLOs, error budgets and what it adds over raw traces.
  10. AWS CloudTrail for operations — management vs data events, finding a change, and how it differs from application logging.
  11. Log aggregation and error monitoring — log classes, centralising logs across accounts, error fingerprints and alert routing.
  12. Performance bottlenecks in .NET on AWS — thread-pool starvation, connection pools, cold starts and N+1 queries.
  13. Troubleshooting production issues methodically — symptom, hypothesis, evidence, fix, with a worked 5xx investigation.
  14. Incident response and on-call runbooks — severity, roles, communication, runbook structure and post-incident reviews.
  15. Cost monitoring for observability — CloudWatch's own costs (ingestion, retention, custom metrics), budgets and anomaly alerts.

Revise with the Module 11 cheat sheet.

Labs and module project

  • Lab 11: observe a banking API covers Lab 11.1 (🟢), Lab 11.2 (🟡) and Lab 11.3 (🟡): run the Harbourline Transfers API locally with flat JSON logs, PII redaction, EMF business metrics and OpenTelemetry traces into Jaeger, inject a slow database and a partner outage, find each cause, then deploy a 7-day log group, SNS page topic, alarms, a composite page alarm and a dashboard with CDK or Terraform, rehearse a page and tear everything down. 61 tests run without an AWS account (60 pass, 1 is skipped unless RUN_AWS_TESTS=1). The code is in labs/m11-observe-banking-api/ (src/, tests/, infra/, otel/, dashboards/, scripts/, docker-compose.yml).
  • Lab 11.4 (🟢) — Reproduce thread-pool starvation with sync-over-async code, diagnose it with dotnet-counters, then fix it, inside Performance bottlenecks in .NET on AWS.
  • Lab 11.5 (🟢) — CloudTrail event history: find who changed a security group in your sandbox, inside AWS CloudTrail for operations.

Module project — observe and fix a banking API. Monitor the .NET banking API, inject faults (slow database, partner timeout, exhausted connection pool, bad deploy), identify each performance problem, trace failed requests end-to-end, and configure actionable alerts with linked runbooks. Deliver a dashboard, an alarm list with rationale and one written post-incident review. The lab covers the slow-database and partner-outage faults plus the dashboard, alarms and review; Performance bottlenecks and Incident response and on-call runbooks supply the rest.

Cost profile

PartLabelFree alternative
OpenTelemetry, local collector, dotnet-counters🟢 freeRuns entirely on your machine
Basic service metrics and CloudTrail event history🟢 freeIncluded by default
CloudWatch log ingestion and storage🟡 creditsKeep lab volumes small and set short retention; delete log groups afterwards
Custom metrics, alarms and dashboards🟡 creditsAlways-free allowances cover a few ⚠ VERIFY current limits; delete after the module
Synthetics canaries🟡 creditsRun for one hour, then delete the canary and its S3 artefacts
X-Ray traces / Application Signals🟡 creditsSample at a low rate; local Jaeger for most practice

Certification alignment

ExamDomains (weight)What this module covers
SOA-C03Monitoring (22%) · Reliability (22%)CloudWatch logs, metrics, alarms, dashboards, troubleshooting
DOP-C02Monitoring (15%) · IR (14%)Observability design, event-driven alerting, incident response
DVA-C03Troubleshoot (22%)Instrumentation, X-Ray tracing, log-based debugging

The domain labels are short forms; see each exam page for the official wording.

Service availability notes

Services that changed — verified 2026-11-17

AWS CloudTrail Lake has been closed to new customers since 2026-05-31. AWS recommends Amazon CloudWatch (CloudWatch pipelines and its unified log data management) for similar audit-log analytics. CloudTrail trails themselves are fully supported.

AWS Systems Manager Incident Manager has been closed to new customers since 2025-11-07. This module teaches runbooks with CloudWatch alarms → SNS and does not build a lab on it; Incident response and on-call runbooks discusses the alternatives. Check AWS's migration guidance for the currently recommended replacement ⚠ VERIFY.

CloudWatch Logs "Infrequent Access" log class is now called Essentials ⚠ VERIFY the exact wording. It has no metric filters, subscription filters or EMF, and the class cannot change after the log group is created (Log aggregation and error monitoring).

CloudWatch Application Signals supports .NET applications through OpenTelemetry/ADOT, but the ADOT .NET SDK does not support AWS SDK for .NET v4; confirm .NET 10 support on your chosen platform before relying on automatic spans ⚠ VERIFY (Application Signals).

Tracing in the lab uses W3C Trace Context propagation rather than the X-Ray propagator, because OpenTelemetry.Contrib.Extensions.AWSXRay 1.2.0 pulled in a Newtonsoft.Json version with a known vulnerability when tested on 2026-11-18 (Distributed tracing).

Synthetics runtime name: the runtime in the dashboards lesson is a placeholder; check the current supported runtime before creating a canary ⚠ VERIFY. Synthetics has no .NET runtime (Dashboards and Synthetics canaries).

Logging and personal data

Logs often end up holding account numbers, names and IP addresses. What may be logged, masking rules and log retention periods need agreement with your data-protection officer under UK GDPR.

Key AWS services

Amazon CloudWatch (Logs, Logs Insights, Metrics, Embedded Metric Format, Alarms, composite alarms, Dashboards, Synthetics, Application Signals) · AWS X-Ray · AWS Distro for OpenTelemetry (ADOT) · AWS CloudTrail · Amazon SNS · AWS Systems Manager · AWS Budgets

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page