Observability fundamentals: logs, metrics, traces and alerts you can act on
Explain the difference between monitoring and observability, choose the right signal (log, metric or trace) for a question, apply the four golden signals to a banking API, and judge whether an alert is actionable.
Assumes: .NET 10 LTS · OpenTelemetry .NET 1.21 · Amazon CloudWatch · AWS X-Ray · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain the difference between monitoring (known questions, known failures) and observability (asking new questions of a running system) in your own words.
- Choose between a log, a metric and a trace for a given question, and say what each costs.
- Apply the four golden signals (latency, traffic, errors, saturation) and the RED and USE shortcuts to a banking API.
- Write a service level indicator (SLI) and a service level objective (SLO) for a payments endpoint, and explain an error budget.
- Test any alert against five questions to decide whether it is actionable, and map each signal to the AWS service that stores it.
Prerequisites
- Logging and errors for .NET on AWS: you already write structured logs with a correlation ID.
- API monitoring: a first pass at access logs, metrics and alarms for one API.
- Tools: none for the reading. The lab at the end of the module (see Lab 11: observe a banking API) runs the sample code.
Why this matters in banking
It is 09:00 on salary day. Customers can log in and see balances, but "send money" spins for ten seconds and then fails for about one in five of them. The support line lights up. Three things now decide how the next hour goes:
- How fast you notice. If the first signal is a customer tweet, you are already late. An alert that fired at 09:02 gives you an hour's head start.
- How fast you find the cause. Is it the database, the sanctions-screening partner, a bad deploy, or the load balancer? With logs, metrics and traces tied together by IDs you can follow one failed transfer from the edge to the cause in minutes. Without them, five teams join a call and guess.
- How honest the evidence is. Operational-resilience expectations ask a bank to know its important business services, detect disruption and show how long recovery took. Your telemetry is that evidence. It also must not become a liability: a log full of IBANs and names is personal data you now have to protect, retain and delete under UK GDPR.
Observability is the engineering practice that makes all three possible. It is not a tool you buy; it is a property you design in.
The idea in plain language
Monitoring answers questions you thought of in advance: "Is CPU above 80%?", "Is the health check passing?" You build dashboards and alarms for known failure modes. It works well for failures you have seen before.
Observability is the ability to understand what a system is doing from the data it emits, including failures you did not predict. The test is simple: when something strange happens, can you answer "why?" without deploying new code to find out? If you have to add log lines and redeploy in the middle of an incident, the system was not observable enough.
Three kinds of data do most of the work. Each answers a different question:
| Signal | What it is | Best at answering | Cost shape |
|---|---|---|---|
| Metric | A number measured over time, aggregated (count, sum, average, percentile) | "Is something wrong, and how bad is it?" | Cheap per data point; expensive when you add many dimension combinations |
| Log | A timestamped record of one event, with detail | "What exactly happened in this request?" | Priced per GB ingested and stored; grows with traffic |
| Trace | The path of one request across services, as a tree of timed spans | "Where did the time go, and which dependency failed?" | Priced per trace or span; usually sampled |
Many teams add a fourth: events such as deployments, configuration changes and feature-flag flips. Those answer "what changed?", which is the most common cause of an incident. In AWS, CloudTrail (see CloudTrail for banks) records changes to AWS resources, and your pipeline records deployments.
Analogy: Think of a branch's queue system. The metric is the screen showing "average wait: 14 minutes". It tells you there is a problem but not why. The log is the ticket machine's record: who took ticket 57, at what time, for which service. The trace is a staff member following one customer with a stopwatch from the door to the counter to the back office and out, writing down every stop. You need the screen to know when to act, the ticket records for detail, and the stopwatch walk to find the slow step.
The signals are most powerful when they are linked. A metric spike leads you to a time window; the logs in that window show a correlationId; the traceId in the same log line opens the trace; the trace shows a slow span. Every lesson in this module adds one link to that chain.
Cardinality: the idea that explains most observability bills
Cardinality is the number of distinct values a field can take. A Outcome field with three values (succeeded, rejected, failed) has cardinality 3. A customerId field has cardinality in the millions.
- In a log or trace, high-cardinality fields are welcome: they are exactly what lets you find one customer's request. You pay by volume, not by distinct values.
- In a metric, every distinct combination of dimension values is a separate time series, and each one is billed. A
customerIddimension on a metric would create a million metrics.
The practical rule: put high-cardinality identifiers in logs and traces; keep metric dimensions to small, fixed vocabularies. The lab code enforces this: BankingMetrics refuses any Outcome that is not one of three known values.
What problem the AWS services solve
Here is the mapping from signal to AWS service. Later lessons cover each in depth.
| Need | AWS service or feature | Lesson |
|---|---|---|
| Store and search application logs | Amazon CloudWatch Logs (log groups, retention, Logs Insights) | CloudWatch Logs for .NET, Logs Insights queries |
| Numbers over time, business metrics | CloudWatch metrics, custom metrics, Embedded Metric Format (EMF) | Metrics and EMF |
| Tell a human when customers hurt | CloudWatch alarms, composite alarms, SNS | Alarms and composite alarms |
| One screen for the on-call engineer; scripted availability checks | CloudWatch dashboards, CloudWatch Synthetics canaries | Dashboards and Synthetics canaries |
| Follow one request across services | AWS X-Ray / CloudWatch traces with OpenTelemetry (ADOT) | Distributed tracing |
| Service maps and SLOs | CloudWatch Application Signals | Application Signals |
| Who changed what in AWS | AWS CloudTrail | CloudTrail for banks |
Honest limitations:
- Observability is not free. Logs, custom metrics, traces and dashboards all cost money, and the bill grows with traffic. The cost monitoring lesson treats that; for now, remember that "log everything forever" is not a strategy.
- More data is not more insight. A thousand unlabelled dashboards are worse than one good one.
- Telemetry has a blind spot at the edges: if the request never reaches your code (DNS failure, a WAF block, a TLS error), your application logs show nothing. That is why you also need external checks (canaries) and load-balancer metrics.
- Sampling (keeping only some traces) saves money but means the one trace you want may not exist. Lesson 7 covers how to sample so errors are always kept.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Application Insights (one SDK for requests, dependencies, exceptions, metrics) | Separate services stitched by IDs: CloudWatch Logs + CloudWatch metrics + X-Ray/Application Signals | No single SDK; OpenTelemetry is the common instrumentation layer, and you choose where each signal lands |
| Azure Monitor Logs, KQL | CloudWatch Logs Insights | Different query language; pay per GB scanned |
| Azure Monitor Metrics, metric alerts, action groups | CloudWatch metrics, alarms, SNS | Alarms notify through SNS topics rather than action groups |
| Application Map | X-Ray service map, Application Signals | Built from traces |
ILogger with the Application Insights provider | ILogger writing JSON to stdout | The platform ships stdout; you rarely call a logging SDK |
TelemetryClient.TrackMetric | EMF log line or PutMetricData | EMF is a log line CloudWatch turns into metrics: no extra API call |
| Activity Log | CloudTrail | Separate service, separate retention rules |
System.Diagnostics.Activity and ActivitySource | Exactly the same | OpenTelemetry .NET is built directly on these types, so your Azure-era instrumentation knowledge carries over |
Architecture
Security-boundary view. Telemetry is data with its own trust rules:
- What goes in: identifiers that are not personal (correlation ID, trace ID, payment ID, status, latency). Not names, full IBANs, card numbers, tokens or request bodies.
- Who can read it: a named operations role with read access to specific log groups, not every developer in production by default.
- How long it lives: a retention period on every log group, agreed with your data-protection officer.
- Where it travels: inside the AWS account over TLS. Anything exported to a third-party tool needs a third-party-risk review.
The four golden signals
Google's site-reliability book popularised four signals that, for a user-facing service, tell you most of what you need.
| Signal | Question | Banking example (Transfers API) | Typical AWS source |
|---|---|---|---|
| Latency | How long do requests take? Split successes from failures: a fast error is not a fast success | p50, p95, p99 of POST /v1/transfers | EMF TransferLatency, ALB TargetResponseTime, API Gateway Latency |
| Traffic | How much demand is there? | Transfers per minute | EMF TransferCount, ALB RequestCount |
| Errors | How many requests fail, including the quietly wrong ones? | 5xx rate; transfers with outcome failed; and "200 OK but the payment was not created" | EMF by Outcome, ALB HTTPCode_Target_5XX_Count |
| Saturation | How full is the most constrained resource? | Database connection pool in use; thread-pool queue length; Fargate CPU | Container Insights, RDS metrics, runtime counters |
Two shorthand sets help when you are in a hurry:
- RED for request-driven services: Rate, Errors, Duration. It is the golden signals without saturation.
- USE for resources (CPU, disk, connection pool): Utilisation, Saturation, Errors.
Use RED to decide whether customers are affected; use USE to find which resource is the bottleneck.
Percentiles, not averages. An average hides the slow tail. If 1 in 100 transfers takes 30 seconds, the average barely moves but one customer in a hundred is furious. Alarm on p99 (or p95) for latency and keep the average only as a trend line.
SLIs, SLOs and error budgets
An SLI (service level indicator) is a measurement of good service, expressed as a ratio: good events divided by valid events. An SLO (service level objective) is the target for that ratio over a window.
For the Transfers API:
- Availability SLI: requests that return a non-5xx response ÷ all valid requests. Requests rejected for bad input (HTTP 400) are valid but not errors on our side, so they count as good.
- Latency SLI: requests completed in under 1,000 ms ÷ all valid requests.
- SLO: 99.9% of requests meet both, over a rolling 30 days.
The SLO implies an error budget: the allowed amount of badness. 99.9% over 30 days of 10 million requests allows 10,000 bad requests. If a bad release burns 8,000 of them in one afternoon, the team has little budget left for risky changes this month, so it slows down and fixes reliability instead. The budget turns an argument ("is it reliable enough?") into a number.
Two cautions. First, an SLO is a business decision: a payments journey may warrant 99.95% while an internal report needs 99%. Second, SLOs do not replace regulatory or contractual targets; they feed into them. The Application Signals lesson covers CloudWatch Application Signals, which can compute SLOs for you.
Is this alert actionable?
Most teams do not have too little alerting, they have too much of the wrong kind. Test every alert against five questions:
- Does it mean a customer is, or soon will be, affected? CPU at 85% is a cause, not a symptom. Failed transfers is a symptom.
- Does a human need to act now? If the answer is "look at it tomorrow", it is a ticket or a dashboard tile, not a page.
- Can the person who gets it do something? If the fix belongs to another team, route it there.
- Does it say what to do first? Link a runbook in the alarm description.
- Is it unique? One incident should produce one page, not twelve. Composite alarms (lesson 5) exist for this.
An alert that fails these tests trains people to ignore alerts. Alert fatigue is a reliability risk in its own right.
Step-by-step: instrument one transfer for all three signals
This is the pattern you will build in the module. It uses the sample code in labs/m11-observe-banking-api (the API is Harbourline.Transfers.Api).
Decide the question each signal answers
Write it down before writing code. For POST /v1/transfers: the metric answers "are transfers failing or slow?"; the log answers "what happened to correlation ID X?"; the trace answers "which step was slow for that request?"
Give every request one identity
Resolve a correlationId at the edge (accept a safe incoming one, otherwise generate a UUID v7), echo it on the response, and add it to the logging scope and the current trace span. Lesson 2 shows the middleware.
Emit a log line per request, flat JSON
One line, one object, with method, path, statusCode, elapsedMs, correlationId and the traceId from Activity.Current. Flat fields are what make Logs Insights queries short.
Emit business metrics with a bounded vocabulary
Count transfers by Outcome (succeeded, rejected, failed) and record latency. Lesson 4 shows how EMF does this from a log line.
Start spans around the interesting work
One span for the database insert, one for the partner call. Lesson 7 shows ActivitySource and the OpenTelemetry setup.
Alarm on symptoms and link a runbook
Alarm on failed transfers and traffic stopping; show latency on the dashboard and page only when customers are affected.
Complete code
The smallest end-to-end demonstration of "one request, three signals" in .NET:
using System.Diagnostics;
using Microsoft.Extensions.Logging;
namespace Harbourline.Demo;
public sealed class TransferHandler(ILogger<TransferHandler> logger)
{
// A trace source: spans you start here are exported once the OpenTelemetry SDK listens to this name.
private static readonly ActivitySource Source = new("Harbourline.Transfers");
public async Task<string> HandleAsync(string correlationId, decimal amount)
{
var started = Stopwatch.GetTimestamp();
// TRACE: a span for the work. It becomes a bar in the trace view.
using var span = Source.StartActivity("handle transfer");
span?.SetTag("correlation.id", correlationId);
// LOG: one event with detail. traceId lets you jump from this line to the trace.
logger.LogInformation("Transfer requested {CorrelationId} {TraceId}", correlationId, span?.TraceId.ToString());
await Task.Delay(20); // pretend to work
var elapsedMs = Stopwatch.GetElapsedTime(started).TotalMilliseconds;
// METRIC: in the lab this is an EMF line (lesson 4). Here it is shown as a structured log field.
logger.LogInformation("Transfer completed {Outcome} {ElapsedMs}", "succeeded", elapsedMs);
return "PAY-" + Guid.CreateVersion7();
}
}Code explained line by line
ActivitySourceis the .NET type behind distributed tracing. OpenTelemetry does not invent a new API: it listens toActivitySourcenames you register. That is why this code has no AWS or OpenTelemetry reference.StartActivityreturnsnullwhen nothing is listening, sospan?.keeps the code free when tracing is off.- The first log line carries
correlationIdandtraceId. Those two fields are the glue between signals. Stopwatch.GetElapsedTimemeasures latency without allocating; it is the value you would record as a metric.Outcomeuses a fixed vocabulary, which is what keeps metric cardinality (and cost) predictable.
Security considerations
⚠ Learning shortcut — not for production
The lab logs a free-text reference field on purpose so you can watch a redactor mask a phone number. In production, do not log customer-typed free text at all; agree an allow-list of loggable fields with your data-protection officer, and treat the redactor as a safety net, not a policy.
- Least privilege for readers. Give operations staff
logs:StartQuery,logs:GetQueryResultsand read access to specific log groups; give the application only the permissions its log driver needs. - Retention is a data-protection control. Set it on every log group. "Never expire" keeps personal data longer than you can justify.
- Telemetry integrity. Application logs are not an audit trail. Audit evidence comes from CloudTrail and from append-only stores designed for it (see Audit logging and evidence collection).
- Never claim compliance from telemetry alone. Whether your logging meets a regulator's expectations is a decision for your compliance and legal teams.
Banking use case
Harbourline's payments team defines its top-level service as "customers can send money". It picks three SLIs (success ratio, p99 latency under 1,000 ms, and "payment status visible within 5 seconds"), writes a 99.9% SLO, and wires exactly two paging alarms: failed transfers and traffic stopped. Latency, queue depth and CPU live on one dashboard and feed tickets. Each page links a runbook. After the first quarter the team reviews every page: any alert that was not actionable is deleted or demoted. Page volume falls by more than half and mean time to acknowledge improves, because people trust the pager again. All of this uses fictional Harbourline data.
Common mistakes
- Alerting on causes. CPU, memory and queue length belong on dashboards. Page on symptoms. Why it happens: causes are easy to measure. Fix: ask "does a customer notice?"
- Averages instead of percentiles. Fix: alarm on p95 or p99.
- High-cardinality metric dimensions (customer ID, request ID). Fix: IDs go in logs and traces; metric dimensions stay in a small fixed set.
- Logs without a correlation ID. You can see an error but not what led to it. Fix: one ID, added in middleware, on every line.
- Collecting everything, reading nothing. Fix: start from the questions on-call needs answered and instrument for those.
- Treating the application log as an audit log. Fix: use CloudTrail and purpose-built audit stores for evidence.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Dashboard looks healthy but customers complain | You measure the server, not the customer path; a failure sits in front of your code (DNS, WAF, load balancer) | Add an external canary and load-balancer metrics; see Dashboards and Synthetics canaries |
| Error rate alarm never fires although errors exist | Errors counted per request but returned as HTTP 200 with an error body | Count outcomes in business terms (EMF Outcome), not only status codes |
| Metric bill higher than expected | A high-cardinality dimension | Remove the dimension; move the identifier to a log field |
| Cannot find the logs for one failing request | No correlation ID, or logs spread over several groups | Add the ID in middleware; query all relevant groups at once in Logs Insights |
| Nobody reacts to pages | Alert fatigue | Run the five-question test and delete or demote alerts |
Cost and cleanup
This lesson is conceptual and costs nothing. When you apply it, remember the cost drivers: log ingestion (per GB), custom metrics (per metric per month), alarms, dashboards beyond the free three, traces beyond the free tier. At the time of checking (2026-11-18), the CloudWatch pricing page lists, for US East (N. Virginia), custom metrics from USD 0.30 per metric per month, standard alarms at USD 0.10, composite alarms at USD 0.50, and log ingestion at USD 0.50 per GB for standard logs ⚠ VERIFY current prices before budgeting.
There is nothing to tear down for this lesson.
Exercises
- 🟢 Guided: For the three questions below, name the best signal (metric, log or trace) and why: (a) "Is the failure rate above normal right now?" (b) "What happened to the transfer a customer reported at 09:14?" (c) "Why did that transfer take eight seconds?"
- 🟡 Challenge: Write an availability SLI and a latency SLI for
GET /v1/accounts/{id}/balance. State what counts as a valid request and what counts as good. Then compute the error budget in requests for 99.95% over 30 days at 20 million requests. - 🔴 Live problem: The on-call phone has paged 14 times this week. Ten pages were "CPU above 80% on api-service", three were "queue depth high", one was "failed transfers above threshold". Your manager asks you to cut pages without missing real incidents. Write a one-page proposal: which alerts you keep as pages, which you move to dashboards or tickets, and what evidence you will use to review the change in a month.
Assignment
Create observability-plan.md for a service of your choice (the Transfers API is fine). Include: the three questions on-call must answer in the first five minutes, the signal that answers each, two SLIs and an SLO, the four golden signals with a named metric each, and an alert inventory marked "page", "ticket" or "dashboard only" with a one-line reason for each. Commit it to your portfolio repository.
Solutions
(a) Metric. A failure rate is an aggregate over many requests; metrics are cheap, fast to query and alarmable. (b) Log, found by correlation ID or timestamp. The log holds the detail of one request. (c) Trace. A trace shows the time taken by each step of that one request, so the slow span stands out. The three signals cooperate: the metric tells you when, the log tells you which request, the trace tells you where.
Valid request: an authenticated GET for an account the caller owns that reaches the service (exclude health checks and calls rejected by authentication, which are not service failures). Availability SLI: valid requests answered with a non-5xx status ÷ valid requests. A 404 for an unknown account is good (the service behaved correctly). Latency SLI: valid requests completed in under 300 ms ÷ valid requests (choose the threshold from a customer-experience target; 300 ms is an example). Error budget: 0.05% of 20,000,000 = 10,000 bad requests per 30 days. If an incident burns 3,000, 30% of the month's budget is gone, which is the number you use when deciding whether to freeze risky releases.
Keep failed transfers and traffic stopped as pages: they are customer symptoms. Move CPU to a dashboard tile and a ticket alarm only if sustained for hours, because CPU at 80% on a service that scales out is not a customer problem. Turn queue depth into an age of oldest message alarm, which measures how long customers wait, then page only if it breaches your processing objective. Group the surviving alarms under a composite alarm so one incident is one page. Review after a month: for every page, record whether action was needed; delete any alert with zero actions. The evidence is the page count, the action rate per page, and mean time to acknowledge.
Knowledge check
Answer: a metric. It is an aggregated number over time. Logs hold detail but cost per GB; traces show one request's path and are usually sampled. Metrics are best for alarms.
Answer: every distinct dimension combination is its own billed time series, so millions of customers would create millions of metrics. A log field is just text in a line that you pay for by volume, and it lets you find one customer's request.
Answer: most requests are fast but a small fraction (about 1 in 100) are very slow. Alarming on the average would miss it; alarm on p95 or p99.
Answer: failed transfers. It is a customer-visible symptom. CPU is a cause that may or may not hurt customers, so it belongs on a dashboard.
Answer: it quantifies how much unreliability is acceptable. While budget remains, the team can ship changes; when it is exhausted, the team prioritises reliability work over features.
Interview questions
- What is the difference between monitoring and observability? Monitoring checks known failure modes with predefined dashboards and alerts. Observability means emitting enough well-linked data (logs, metrics, traces) to ask new questions about unexpected behaviour without changing code.
- How do you stop a metrics bill from exploding? Keep dimensions to small, bounded sets; put identifiers in logs and traces; avoid per-request dimensions; review custom metrics regularly.
- What makes an alert good? It indicates customer impact, needs human action now, is routed to someone who can act, links a runbook, and appears once per incident.
- Why sample traces, and how do you avoid losing the important ones? Tracing every request is costly; sample a percentage but always keep errors and slow requests (tail-based or error-biased sampling).
- How would you follow one failed payment across services? Take its correlation ID, query logs for that ID, read the trace ID from the lines, and open the trace to find the failing span.
Certification practice
A payments team wants to be paged only when customers are affected, not whenever a server is busy. Which alarm best meets this?
A. CPU utilisation above 70% on any instance for one minute. B. The percentage of failed payment requests above a threshold for 3 of the last 5 minutes. C. Free memory below 20% on a single instance. D. Disk read operations above the daily average.
Answer: B. It measures a customer-visible symptom and uses M-out-of-N evaluation to ignore single spikes. A, C and D are resource causes that may not affect customers, and would page too often.
A team needs to find which dependency made one request slow in a microservices system. Which signal is most direct?
A. A CloudWatch metric for average latency. B. A distributed trace with spans per downstream call. C. A CloudWatch alarm on 5xx errors. D. CloudTrail management events.
Answer: B. A trace shows the timed path of a single request across services. Metrics (A, C) are aggregated and cannot isolate one request's slow step; CloudTrail (D) records AWS API calls, not application request timing.
A developer adds customerId as a dimension on a custom CloudWatch metric emitted per request. What is the most likely consequence?
A. Faster alarms. B. A large number of billed custom metrics, because each distinct value creates a separate metric. C. Automatic aggregation across customers. D. Metrics are rejected by CloudWatch.
Answer: B. Each unique dimension combination is its own metric. The identifier belongs in a log field or trace attribute. CloudWatch does not reject the data (D), and it does not roll the values up for you (C).
Completion checklist
- I can explain monitoring versus observability in two sentences
- I can pick the right signal (metric, log or trace) for a question
- I can write an SLI, an SLO and compute an error budget
- I can apply the five-question actionable-alert test to an alert
- I read the sample code's
LogFieldsandBankingMetricsand can say whyOutcomehas three values
Key takeaways
- Metrics tell you that something is wrong, logs tell you what happened, traces tell you where; link them with a correlation ID and a trace ID.
- Keep metric dimensions small and fixed; put identifiers in logs and traces.
- Alarm on symptoms customers feel (errors, latency percentiles, traffic stopping), and show causes on dashboards.
- An SLO turns reliability into a number, and the error budget turns it into a decision.
- Telemetry is data: set retention, restrict access and never log what you do not need.
Further reading
- Amazon CloudWatch documentation: the service overview.
- OpenTelemetry documentation: the vendor-neutral instrumentation standard.
- Module 11 overview: the full lesson list and labs.
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Module 11: Monitoring, Logging & Troubleshooting
Make your .NET workloads observable with CloudWatch logs, metrics, alarms and dashboards, distributed tracing with X-Ray and OpenTelemetry, and CloudTrail for who-did-what. You then practise finding and fixing real production problems methodically, from thread-pool starvation to cold starts, and turn alerts into runbooks.
CloudWatch Logs for .NET: structured JSON, correlation IDs, retention and PII-safe logging
Build a .NET 10 console formatter that writes one flat, redacted JSON object per line, ship it to CloudWatch Logs from ECS or Lambda, set retention and encryption in code, and keep personal data out of the log group.