Skip to Content
Cloud Engineering Academy
11. Monitoring, Logging & Troubleshooting

Observability fundamentals: logs, metrics, traces and alerts you can act on

Explain the difference between monitoring and observability, choose the right signal (log, metric or trace) for a question, apply the four golden signals to a banking API, and judge whether an alert is actionable.

beginner⏱ 1 h 🟢 FreeSOA-C03DOP-C02DVA-C03

Assumes: .NET 10 LTS · OpenTelemetry .NET 1.21 · Amazon CloudWatch · AWS X-Ray · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain the difference between monitoring (known questions, known failures) and observability (asking new questions of a running system) in your own words.
  • Choose between a log, a metric and a trace for a given question, and say what each costs.
  • Apply the four golden signals (latency, traffic, errors, saturation) and the RED and USE shortcuts to a banking API.
  • Write a service level indicator (SLI) and a service level objective (SLO) for a payments endpoint, and explain an error budget.
  • Test any alert against five questions to decide whether it is actionable, and map each signal to the AWS service that stores it.

Prerequisites

Why this matters in banking

It is 09:00 on salary day. Customers can log in and see balances, but "send money" spins for ten seconds and then fails for about one in five of them. The support line lights up. Three things now decide how the next hour goes:

  1. How fast you notice. If the first signal is a customer tweet, you are already late. An alert that fired at 09:02 gives you an hour's head start.
  2. How fast you find the cause. Is it the database, the sanctions-screening partner, a bad deploy, or the load balancer? With logs, metrics and traces tied together by IDs you can follow one failed transfer from the edge to the cause in minutes. Without them, five teams join a call and guess.
  3. How honest the evidence is. Operational-resilience expectations ask a bank to know its important business services, detect disruption and show how long recovery took. Your telemetry is that evidence. It also must not become a liability: a log full of IBANs and names is personal data you now have to protect, retain and delete under UK GDPR.

Observability is the engineering practice that makes all three possible. It is not a tool you buy; it is a property you design in.

The idea in plain language

Monitoring answers questions you thought of in advance: "Is CPU above 80%?", "Is the health check passing?" You build dashboards and alarms for known failure modes. It works well for failures you have seen before.

Observability is the ability to understand what a system is doing from the data it emits, including failures you did not predict. The test is simple: when something strange happens, can you answer "why?" without deploying new code to find out? If you have to add log lines and redeploy in the middle of an incident, the system was not observable enough.

Three kinds of data do most of the work. Each answers a different question:

SignalWhat it isBest at answeringCost shape
MetricA number measured over time, aggregated (count, sum, average, percentile)"Is something wrong, and how bad is it?"Cheap per data point; expensive when you add many dimension combinations
LogA timestamped record of one event, with detail"What exactly happened in this request?"Priced per GB ingested and stored; grows with traffic
TraceThe path of one request across services, as a tree of timed spans"Where did the time go, and which dependency failed?"Priced per trace or span; usually sampled

Many teams add a fourth: events such as deployments, configuration changes and feature-flag flips. Those answer "what changed?", which is the most common cause of an incident. In AWS, CloudTrail (see CloudTrail for banks) records changes to AWS resources, and your pipeline records deployments.

Analogy: Think of a branch's queue system. The metric is the screen showing "average wait: 14 minutes". It tells you there is a problem but not why. The log is the ticket machine's record: who took ticket 57, at what time, for which service. The trace is a staff member following one customer with a stopwatch from the door to the counter to the back office and out, writing down every stop. You need the screen to know when to act, the ticket records for detail, and the stopwatch walk to find the slow step.

The signals are most powerful when they are linked. A metric spike leads you to a time window; the logs in that window show a correlationId; the traceId in the same log line opens the trace; the trace shows a slow span. Every lesson in this module adds one link to that chain.

Cardinality: the idea that explains most observability bills

Cardinality is the number of distinct values a field can take. A Outcome field with three values (succeeded, rejected, failed) has cardinality 3. A customerId field has cardinality in the millions.

  • In a log or trace, high-cardinality fields are welcome: they are exactly what lets you find one customer's request. You pay by volume, not by distinct values.
  • In a metric, every distinct combination of dimension values is a separate time series, and each one is billed. A customerId dimension on a metric would create a million metrics.

The practical rule: put high-cardinality identifiers in logs and traces; keep metric dimensions to small, fixed vocabularies. The lab code enforces this: BankingMetrics refuses any Outcome that is not one of three known values.

What problem the AWS services solve

Here is the mapping from signal to AWS service. Later lessons cover each in depth.

NeedAWS service or featureLesson
Store and search application logsAmazon CloudWatch Logs (log groups, retention, Logs Insights)CloudWatch Logs for .NET, Logs Insights queries
Numbers over time, business metricsCloudWatch metrics, custom metrics, Embedded Metric Format (EMF)Metrics and EMF
Tell a human when customers hurtCloudWatch alarms, composite alarms, SNSAlarms and composite alarms
One screen for the on-call engineer; scripted availability checksCloudWatch dashboards, CloudWatch Synthetics canariesDashboards and Synthetics canaries
Follow one request across servicesAWS X-Ray / CloudWatch traces with OpenTelemetry (ADOT)Distributed tracing
Service maps and SLOsCloudWatch Application SignalsApplication Signals
Who changed what in AWSAWS CloudTrailCloudTrail for banks

Honest limitations:

  • Observability is not free. Logs, custom metrics, traces and dashboards all cost money, and the bill grows with traffic. The cost monitoring lesson treats that; for now, remember that "log everything forever" is not a strategy.
  • More data is not more insight. A thousand unlabelled dashboards are worse than one good one.
  • Telemetry has a blind spot at the edges: if the request never reaches your code (DNS failure, a WAF block, a TLS error), your application logs show nothing. That is why you also need external checks (canaries) and load-balancer metrics.
  • Sampling (keeping only some traces) saves money but means the one trace you want may not exist. Lesson 7 covers how to sample so errors are always kept.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Application Insights (one SDK for requests, dependencies, exceptions, metrics)Separate services stitched by IDs: CloudWatch Logs + CloudWatch metrics + X-Ray/Application SignalsNo single SDK; OpenTelemetry is the common instrumentation layer, and you choose where each signal lands
Azure Monitor Logs, KQLCloudWatch Logs InsightsDifferent query language; pay per GB scanned
Azure Monitor Metrics, metric alerts, action groupsCloudWatch metrics, alarms, SNSAlarms notify through SNS topics rather than action groups
Application MapX-Ray service map, Application SignalsBuilt from traces
ILogger with the Application Insights providerILogger writing JSON to stdoutThe platform ships stdout; you rarely call a logging SDK
TelemetryClient.TrackMetricEMF log line or PutMetricDataEMF is a log line CloudWatch turns into metrics: no extra API call
Activity LogCloudTrailSeparate service, separate retention rules
System.Diagnostics.Activity and ActivitySourceExactly the sameOpenTelemetry .NET is built directly on these types, so your Azure-era instrumentation knowledge carries over

Architecture

Security-boundary view. Telemetry is data with its own trust rules:

  • What goes in: identifiers that are not personal (correlation ID, trace ID, payment ID, status, latency). Not names, full IBANs, card numbers, tokens or request bodies.
  • Who can read it: a named operations role with read access to specific log groups, not every developer in production by default.
  • How long it lives: a retention period on every log group, agreed with your data-protection officer.
  • Where it travels: inside the AWS account over TLS. Anything exported to a third-party tool needs a third-party-risk review.

The four golden signals

Google's site-reliability book popularised four signals that, for a user-facing service, tell you most of what you need.

SignalQuestionBanking example (Transfers API)Typical AWS source
LatencyHow long do requests take? Split successes from failures: a fast error is not a fast successp50, p95, p99 of POST /v1/transfersEMF TransferLatency, ALB TargetResponseTime, API Gateway Latency
TrafficHow much demand is there?Transfers per minuteEMF TransferCount, ALB RequestCount
ErrorsHow many requests fail, including the quietly wrong ones?5xx rate; transfers with outcome failed; and "200 OK but the payment was not created"EMF by Outcome, ALB HTTPCode_Target_5XX_Count
SaturationHow full is the most constrained resource?Database connection pool in use; thread-pool queue length; Fargate CPUContainer Insights, RDS metrics, runtime counters

Two shorthand sets help when you are in a hurry:

  • RED for request-driven services: Rate, Errors, Duration. It is the golden signals without saturation.
  • USE for resources (CPU, disk, connection pool): Utilisation, Saturation, Errors.

Use RED to decide whether customers are affected; use USE to find which resource is the bottleneck.

Percentiles, not averages. An average hides the slow tail. If 1 in 100 transfers takes 30 seconds, the average barely moves but one customer in a hundred is furious. Alarm on p99 (or p95) for latency and keep the average only as a trend line.

SLIs, SLOs and error budgets

An SLI (service level indicator) is a measurement of good service, expressed as a ratio: good events divided by valid events. An SLO (service level objective) is the target for that ratio over a window.

For the Transfers API:

  • Availability SLI: requests that return a non-5xx response ÷ all valid requests. Requests rejected for bad input (HTTP 400) are valid but not errors on our side, so they count as good.
  • Latency SLI: requests completed in under 1,000 ms ÷ all valid requests.
  • SLO: 99.9% of requests meet both, over a rolling 30 days.

The SLO implies an error budget: the allowed amount of badness. 99.9% over 30 days of 10 million requests allows 10,000 bad requests. If a bad release burns 8,000 of them in one afternoon, the team has little budget left for risky changes this month, so it slows down and fixes reliability instead. The budget turns an argument ("is it reliable enough?") into a number.

Two cautions. First, an SLO is a business decision: a payments journey may warrant 99.95% while an internal report needs 99%. Second, SLOs do not replace regulatory or contractual targets; they feed into them. The Application Signals lesson covers CloudWatch Application Signals, which can compute SLOs for you.

Is this alert actionable?

Most teams do not have too little alerting, they have too much of the wrong kind. Test every alert against five questions:

  1. Does it mean a customer is, or soon will be, affected? CPU at 85% is a cause, not a symptom. Failed transfers is a symptom.
  2. Does a human need to act now? If the answer is "look at it tomorrow", it is a ticket or a dashboard tile, not a page.
  3. Can the person who gets it do something? If the fix belongs to another team, route it there.
  4. Does it say what to do first? Link a runbook in the alarm description.
  5. Is it unique? One incident should produce one page, not twelve. Composite alarms (lesson 5) exist for this.

An alert that fails these tests trains people to ignore alerts. Alert fatigue is a reliability risk in its own right.

Step-by-step: instrument one transfer for all three signals

This is the pattern you will build in the module. It uses the sample code in labs/m11-observe-banking-api (the API is Harbourline.Transfers.Api).

Decide the question each signal answers

Write it down before writing code. For POST /v1/transfers: the metric answers "are transfers failing or slow?"; the log answers "what happened to correlation ID X?"; the trace answers "which step was slow for that request?"

Give every request one identity

Resolve a correlationId at the edge (accept a safe incoming one, otherwise generate a UUID v7), echo it on the response, and add it to the logging scope and the current trace span. Lesson 2 shows the middleware.

Emit a log line per request, flat JSON

One line, one object, with method, path, statusCode, elapsedMs, correlationId and the traceId from Activity.Current. Flat fields are what make Logs Insights queries short.

Emit business metrics with a bounded vocabulary

Count transfers by Outcome (succeeded, rejected, failed) and record latency. Lesson 4 shows how EMF does this from a log line.

Start spans around the interesting work

One span for the database insert, one for the partner call. Lesson 7 shows ActivitySource and the OpenTelemetry setup.

Alarm on failed transfers and traffic stopping; show latency on the dashboard and page only when customers are affected.

Complete code

The smallest end-to-end demonstration of "one request, three signals" in .NET:

OneRequestThreeSignals.cs
using System.Diagnostics;
using Microsoft.Extensions.Logging;

namespace Harbourline.Demo;

public sealed class TransferHandler(ILogger<TransferHandler> logger)
{
    // A trace source: spans you start here are exported once the OpenTelemetry SDK listens to this name.
    private static readonly ActivitySource Source = new("Harbourline.Transfers");

    public async Task<string> HandleAsync(string correlationId, decimal amount)
    {
        var started = Stopwatch.GetTimestamp();

        // TRACE: a span for the work. It becomes a bar in the trace view.
        using var span = Source.StartActivity("handle transfer");
        span?.SetTag("correlation.id", correlationId);

        // LOG: one event with detail. traceId lets you jump from this line to the trace.
        logger.LogInformation("Transfer requested {CorrelationId} {TraceId}", correlationId, span?.TraceId.ToString());

        await Task.Delay(20); // pretend to work

        var elapsedMs = Stopwatch.GetElapsedTime(started).TotalMilliseconds;

        // METRIC: in the lab this is an EMF line (lesson 4). Here it is shown as a structured log field.
        logger.LogInformation("Transfer completed {Outcome} {ElapsedMs}", "succeeded", elapsedMs);
        return "PAY-" + Guid.CreateVersion7();
    }
}

Code explained line by line

  1. ActivitySource is the .NET type behind distributed tracing. OpenTelemetry does not invent a new API: it listens to ActivitySource names you register. That is why this code has no AWS or OpenTelemetry reference.
  2. StartActivity returns null when nothing is listening, so span?. keeps the code free when tracing is off.
  3. The first log line carries correlationId and traceId. Those two fields are the glue between signals.
  4. Stopwatch.GetElapsedTime measures latency without allocating; it is the value you would record as a metric.
  5. Outcome uses a fixed vocabulary, which is what keeps metric cardinality (and cost) predictable.

Security considerations

⚠ Learning shortcut — not for production

The lab logs a free-text reference field on purpose so you can watch a redactor mask a phone number. In production, do not log customer-typed free text at all; agree an allow-list of loggable fields with your data-protection officer, and treat the redactor as a safety net, not a policy.

  • Least privilege for readers. Give operations staff logs:StartQuery, logs:GetQueryResults and read access to specific log groups; give the application only the permissions its log driver needs.
  • Retention is a data-protection control. Set it on every log group. "Never expire" keeps personal data longer than you can justify.
  • Telemetry integrity. Application logs are not an audit trail. Audit evidence comes from CloudTrail and from append-only stores designed for it (see Audit logging and evidence collection).
  • Never claim compliance from telemetry alone. Whether your logging meets a regulator's expectations is a decision for your compliance and legal teams.

Banking use case

Harbourline's payments team defines its top-level service as "customers can send money". It picks three SLIs (success ratio, p99 latency under 1,000 ms, and "payment status visible within 5 seconds"), writes a 99.9% SLO, and wires exactly two paging alarms: failed transfers and traffic stopped. Latency, queue depth and CPU live on one dashboard and feed tickets. Each page links a runbook. After the first quarter the team reviews every page: any alert that was not actionable is deleted or demoted. Page volume falls by more than half and mean time to acknowledge improves, because people trust the pager again. All of this uses fictional Harbourline data.

Common mistakes

  • Alerting on causes. CPU, memory and queue length belong on dashboards. Page on symptoms. Why it happens: causes are easy to measure. Fix: ask "does a customer notice?"
  • Averages instead of percentiles. Fix: alarm on p95 or p99.
  • High-cardinality metric dimensions (customer ID, request ID). Fix: IDs go in logs and traces; metric dimensions stay in a small fixed set.
  • Logs without a correlation ID. You can see an error but not what led to it. Fix: one ID, added in middleware, on every line.
  • Collecting everything, reading nothing. Fix: start from the questions on-call needs answered and instrument for those.
  • Treating the application log as an audit log. Fix: use CloudTrail and purpose-built audit stores for evidence.

Troubleshooting

SymptomLikely causeFix
Dashboard looks healthy but customers complainYou measure the server, not the customer path; a failure sits in front of your code (DNS, WAF, load balancer)Add an external canary and load-balancer metrics; see Dashboards and Synthetics canaries
Error rate alarm never fires although errors existErrors counted per request but returned as HTTP 200 with an error bodyCount outcomes in business terms (EMF Outcome), not only status codes
Metric bill higher than expectedA high-cardinality dimensionRemove the dimension; move the identifier to a log field
Cannot find the logs for one failing requestNo correlation ID, or logs spread over several groupsAdd the ID in middleware; query all relevant groups at once in Logs Insights
Nobody reacts to pagesAlert fatigueRun the five-question test and delete or demote alerts

Cost and cleanup

This lesson is conceptual and costs nothing. When you apply it, remember the cost drivers: log ingestion (per GB), custom metrics (per metric per month), alarms, dashboards beyond the free three, traces beyond the free tier. At the time of checking (2026-11-18), the CloudWatch pricing page lists, for US East (N. Virginia), custom metrics from USD 0.30 per metric per month, standard alarms at USD 0.10, composite alarms at USD 0.50, and log ingestion at USD 0.50 per GB for standard logs ⚠ VERIFY current prices before budgeting.

There is nothing to tear down for this lesson.

Exercises

  • 🟢 Guided: For the three questions below, name the best signal (metric, log or trace) and why: (a) "Is the failure rate above normal right now?" (b) "What happened to the transfer a customer reported at 09:14?" (c) "Why did that transfer take eight seconds?"
  • 🟡 Challenge: Write an availability SLI and a latency SLI for GET /v1/accounts/{id}/balance. State what counts as a valid request and what counts as good. Then compute the error budget in requests for 99.95% over 30 days at 20 million requests.
  • 🔴 Live problem: The on-call phone has paged 14 times this week. Ten pages were "CPU above 80% on api-service", three were "queue depth high", one was "failed transfers above threshold". Your manager asks you to cut pages without missing real incidents. Write a one-page proposal: which alerts you keep as pages, which you move to dashboards or tickets, and what evidence you will use to review the change in a month.

Assignment

Create observability-plan.md for a service of your choice (the Transfers API is fine). Include: the three questions on-call must answer in the first five minutes, the signal that answers each, two SLIs and an SLO, the four golden signals with a named metric each, and an alert inventory marked "page", "ticket" or "dashboard only" with a one-line reason for each. Commit it to your portfolio repository.

Solutions

Knowledge check

Interview questions

  • What is the difference between monitoring and observability? Monitoring checks known failure modes with predefined dashboards and alerts. Observability means emitting enough well-linked data (logs, metrics, traces) to ask new questions about unexpected behaviour without changing code.
  • How do you stop a metrics bill from exploding? Keep dimensions to small, bounded sets; put identifiers in logs and traces; avoid per-request dimensions; review custom metrics regularly.
  • What makes an alert good? It indicates customer impact, needs human action now, is routed to someone who can act, links a runbook, and appears once per incident.
  • Why sample traces, and how do you avoid losing the important ones? Tracing every request is costly; sample a percentage but always keep errors and slow requests (tail-based or error-biased sampling).
  • How would you follow one failed payment across services? Take its correlation ID, query logs for that ID, read the trace ID from the lines, and open the trace to find the failing span.

Certification practice

Completion checklist

  • I can explain monitoring versus observability in two sentences
  • I can pick the right signal (metric, log or trace) for a question
  • I can write an SLI, an SLO and compute an error budget
  • I can apply the five-question actionable-alert test to an alert
  • I read the sample code's LogFields and BankingMetrics and can say why Outcome has three values

Key takeaways

  • Metrics tell you that something is wrong, logs tell you what happened, traces tell you where; link them with a correlation ID and a trace ID.
  • Keep metric dimensions small and fixed; put identifiers in logs and traces.
  • Alarm on symptoms customers feel (errors, latency percentiles, traffic stopping), and show causes on dashboards.
  • An SLO turns reliability into a number, and the error budget turns it into a decision.
  • Telemetry is data: set retention, restrict access and never log what you do not need.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page