Module 11: Monitoring, Logging & Troubleshooting
Make your .NET workloads observable with CloudWatch logs, metrics, alarms and dashboards, distributed tracing with X-Ray and OpenTelemetry, and CloudTrail for who-did-what. You then practise finding and fixing real production problems methodically, from thread-pool starvation to cold starts, and turn alerts into runbooks.
Assumes: .NET 10 LTS · OpenTelemetry .NET / ADOT · AWS SDK for .NET v4 · Region us-east-1. Verified against official sources on 2026-11-18.
Make your .NET workloads observable with CloudWatch logs, metrics, alarms and dashboards, distributed tracing with X-Ray and OpenTelemetry, and CloudTrail for who-did-what. You then practise finding and fixing real production problems methodically, from thread-pool starvation to cold starts, and turn alerts into runbooks.
| Estimated time | About 14 hours (lessons 8 h · labs 3 h · project 3 h) |
| Priority for you | High — supports deployment (priority 1) and security detection (priority 3) |
| Level | Intermediate → advanced |
| Certifications | SOA-C03 (primary) · DOP-C02 · DVA-C03 |
| Cost profile | 🟢 local OpenTelemetry + basic metrics · 🟡 log ingestion, custom metrics, canaries · 🔴 none required |
Why this matters in banking
When the payments API slows down at 09:00 on salary day, the business wants an answer in minutes, not a war room that guesses. Operational-resilience expectations also mean you must detect incidents, know their customer impact and show regulators how quickly you recovered. Good observability lets you follow one failed transfer across the API, the queue and the database, while CloudTrail answers the separate question "who changed this?". Poor observability, by contrast, means noisy alerts that everyone ignores, or logs full of personal data that become a liability of their own.
From .NET/Azure to AWS
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Application Insights | CloudWatch Application Signals + X-Ray (OpenTelemetry-based) | Instrumentation is OpenTelemetry; ADOT is AWS's distribution |
| Log Analytics workspace + KQL | CloudWatch Logs + Logs Insights query language | Log groups per app; retention is set per log group |
| Azure Monitor metrics and alerts | CloudWatch metrics, alarms and composite alarms | Alarms notify through SNS; composite alarms cut noise |
| Custom metrics via TelemetryClient | Embedded Metric Format (EMF) or PutMetricData | EMF turns structured log lines into metrics cheaply |
| Availability tests | CloudWatch Synthetics canaries | Scripted canaries run on a schedule |
| Azure Monitor workbooks / dashboards | CloudWatch dashboards | Cross-account, cross-Region dashboards possible |
| Azure Activity Log | AWS CloudTrail | Every API call, who made it and from where |
| Cost alerts | AWS Budgets, Cost Anomaly Detection, Cost Explorer | Covered with FinOps in Module 15 |
Learning objectives
By the end of this module you can:
- Ship structured JSON logs from ASP.NET Core to CloudWatch Logs with correlation IDs and sensible retention, without leaking personal data.
- Query logs with Logs Insights to answer real incident questions quickly.
- Publish custom business metrics with EMF and build alarms, composite alarms and dashboards that reflect customer impact.
- Instrument a .NET 10 API with OpenTelemetry (ADOT) and follow a request end-to-end in X-Ray and Application Signals.
- Use CloudTrail to answer "who did what, when" and distinguish audit trails from application logs.
- Diagnose common performance bottlenecks: thread-pool starvation, connection-pool exhaustion, cold starts and N+1 queries.
- Write on-call runbooks and run a structured troubleshooting process from symptom to root cause.
Prerequisites
- Module 3: .NET on AWS — logging, health checks and the deployed API
- Module 1: AWS fundamentals — IAM and CloudTrail basics
- Helpful: Module 8: Messaging for tracing across queues, and Module 6: Databases for database performance
How the pieces fit
Lessons
Work through them in this order. Lessons 1 to 7 build the instrumentation, then the lab is the hands-on checkpoint for those seven; lessons 9 to 15 add service-level views, audit, scale and the human side of operations.
- Observability fundamentals — logs, metrics and traces, the golden signals and what "actionable" really means.
- CloudWatch Logs for .NET — log groups, retention, structured JSON with
ILogger/Serilog, correlation IDs and PII-safe logging. - Logs Insights queries — filtering, parsing, statistics and five saved queries for common incident questions.
- CloudWatch metrics and custom metrics — namespaces, dimensions, resolution and the Embedded Metric Format from C#.
- Alarms and composite alarms — thresholds vs anomaly detection, missing-data handling and noise reduction.
- Dashboards and Synthetics canaries — an operational dashboard for a banking API and a scripted availability check.
- Distributed tracing with OpenTelemetry and X-Ray for .NET (ADOT) — instrumenting ASP.NET Core, HttpClient, the AWS SDK and EF Core.
- Lab 11: observe a banking API — the hands-on checkpoint for lessons 1 to 7 (see the labs section below).
- CloudWatch Application Signals — service maps, SLOs, error budgets and what it adds over raw traces.
- AWS CloudTrail for operations — management vs data events, finding a change, and how it differs from application logging.
- Log aggregation and error monitoring — log classes, centralising logs across accounts, error fingerprints and alert routing.
- Performance bottlenecks in .NET on AWS — thread-pool starvation, connection pools, cold starts and N+1 queries.
- Troubleshooting production issues methodically — symptom, hypothesis, evidence, fix, with a worked 5xx investigation.
- Incident response and on-call runbooks — severity, roles, communication, runbook structure and post-incident reviews.
- Cost monitoring for observability — CloudWatch's own costs (ingestion, retention, custom metrics), budgets and anomaly alerts.
Revise with the Module 11 cheat sheet.
Labs and module project
- Lab 11: observe a banking API covers Lab 11.1 (🟢), Lab 11.2 (🟡) and Lab 11.3 (🟡): run the Harbourline Transfers API locally with flat JSON logs, PII redaction, EMF business metrics and OpenTelemetry traces into Jaeger, inject a slow database and a partner outage, find each cause, then deploy a 7-day log group, SNS page topic, alarms, a composite page alarm and a dashboard with CDK or Terraform, rehearse a page and tear everything down. 61 tests run without an AWS account (60 pass, 1 is skipped unless
RUN_AWS_TESTS=1). The code is inlabs/m11-observe-banking-api/(src/,tests/,infra/,otel/,dashboards/,scripts/,docker-compose.yml). - Lab 11.4 (🟢) — Reproduce thread-pool starvation with sync-over-async code, diagnose it with
dotnet-counters, then fix it, inside Performance bottlenecks in .NET on AWS. - Lab 11.5 (🟢) — CloudTrail event history: find who changed a security group in your sandbox, inside AWS CloudTrail for operations.
Module project — observe and fix a banking API. Monitor the .NET banking API, inject faults (slow database, partner timeout, exhausted connection pool, bad deploy), identify each performance problem, trace failed requests end-to-end, and configure actionable alerts with linked runbooks. Deliver a dashboard, an alarm list with rationale and one written post-incident review. The lab covers the slow-database and partner-outage faults plus the dashboard, alarms and review; Performance bottlenecks and Incident response and on-call runbooks supply the rest.
Cost profile
| Part | Label | Free alternative |
|---|---|---|
OpenTelemetry, local collector, dotnet-counters | 🟢 free | Runs entirely on your machine |
| Basic service metrics and CloudTrail event history | 🟢 free | Included by default |
| CloudWatch log ingestion and storage | 🟡 credits | Keep lab volumes small and set short retention; delete log groups afterwards |
| Custom metrics, alarms and dashboards | 🟡 credits | Always-free allowances cover a few ⚠ VERIFY current limits; delete after the module |
| Synthetics canaries | 🟡 credits | Run for one hour, then delete the canary and its S3 artefacts |
| X-Ray traces / Application Signals | 🟡 credits | Sample at a low rate; local Jaeger for most practice |
Certification alignment
| Exam | Domains (weight) | What this module covers |
|---|---|---|
| SOA-C03 | Monitoring (22%) · Reliability (22%) | CloudWatch logs, metrics, alarms, dashboards, troubleshooting |
| DOP-C02 | Monitoring (15%) · IR (14%) | Observability design, event-driven alerting, incident response |
| DVA-C03 | Troubleshoot (22%) | Instrumentation, X-Ray tracing, log-based debugging |
The domain labels are short forms; see each exam page for the official wording.
Service availability notes
Services that changed — verified 2026-11-17
AWS CloudTrail Lake has been closed to new customers since 2026-05-31. AWS recommends Amazon CloudWatch (CloudWatch pipelines and its unified log data management) for similar audit-log analytics. CloudTrail trails themselves are fully supported.
AWS Systems Manager Incident Manager has been closed to new customers since 2025-11-07. This module teaches runbooks with CloudWatch alarms → SNS and does not build a lab on it; Incident response and on-call runbooks discusses the alternatives. Check AWS's migration guidance for the currently recommended replacement ⚠ VERIFY.
CloudWatch Logs "Infrequent Access" log class is now called Essentials ⚠ VERIFY the exact wording. It has no metric filters, subscription filters or EMF, and the class cannot change after the log group is created (Log aggregation and error monitoring).
CloudWatch Application Signals supports .NET applications through OpenTelemetry/ADOT, but the ADOT .NET SDK does not support AWS SDK for .NET v4; confirm .NET 10 support on your chosen platform before relying on automatic spans ⚠ VERIFY (Application Signals).
Tracing in the lab uses W3C Trace Context propagation rather than the X-Ray propagator, because OpenTelemetry.Contrib.Extensions.AWSXRay 1.2.0 pulled in a Newtonsoft.Json version with a known vulnerability when tested on 2026-11-18 (Distributed tracing).
Synthetics runtime name: the runtime in the dashboards lesson is a placeholder; check the current supported runtime before creating a canary ⚠ VERIFY. Synthetics has no .NET runtime (Dashboards and Synthetics canaries).
Logging and personal data
Logs often end up holding account numbers, names and IP addresses. What may be logged, masking rules and log retention periods need agreement with your data-protection officer under UK GDPR.
Key AWS services
Amazon CloudWatch (Logs, Logs Insights, Metrics, Embedded Metric Format, Alarms, composite alarms, Dashboards, Synthetics, Application Signals) · AWS X-Ray · AWS Distro for OpenTelemetry (ADOT) · AWS CloudTrail · Amazon SNS · AWS Systems Manager · AWS Budgets
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Module 10 cheat sheet
One-page revision summary for Infrastructure as Code: tools, .NET and Azure mapping, CLI commands, limits, decision rules, exam traps and cost traps.
Observability fundamentals: logs, metrics, traces and alerts you can act on
Explain the difference between monitoring and observability, choose the right signal (log, metric or trace) for a question, apply the four golden signals to a banking API, and judge whether an alert is actionable.