Skip to Content
Cloud Engineering Academy
11. Monitoring, Logging & Troubleshooting

Dashboards and Synthetics canaries: one screen for on-call and a customer's-eye availability check

Design an operational CloudWatch dashboard for a banking API as code, add a log-query widget, and write a Synthetics canary that probes the API from outside so you find out about a failure before your customers do.

intermediate⏱ 1 h 30 min🟡 Low costSOA-C03DOP-C02DVA-C03

Assumes: .NET 10 LTS · AWSSDK.CloudWatch 4.1.1.2 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · CloudWatch Synthetics · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Lay out an operational dashboard in the order an on-call engineer reads it (customer impact, golden signals, dependencies, resources) and explain what to leave off.
  • Write dashboard JSON with metric, alarm status, log query and text widgets, deploy it with the CLI, CDK, Terraform and a small .NET tool, and keep it in source control.
  • Explain what a CloudWatch Synthetics canary is, which runtimes it supports, what it costs, and why an external probe sees failures your application metrics cannot.
  • Write a heartbeat canary for the Transfers API health endpoint, alarm on its success percentage, and tear it down.
  • Know when a dashboard is the wrong tool.

Prerequisites

  • Alarms and composite alarms: the alarms the dashboard displays and the canary alarm builds on.
  • CloudWatch metrics and custom metrics: the Harbourline/Payments metrics.
  • Health checks: the /health endpoint the canary calls.
  • Tools: AWS CLI v2 (learn-sandbox). For the canary: an HTTPS URL reachable from the internet, which a sandbox test deployment can provide (see Cost and cleanup for the free alternative).

Why this matters in banking

At 08:55, three minutes before the branches open, a certificate on the public load balancer expires. Every internal metric is healthy: containers run, the database is fine, CPU is low. No request ever reaches the application, so there are no application errors, no logs and no failed-transfer metric. The only way to see this problem is to be a customer: make a real request from outside and notice that TLS fails.

That is the job of a canary: a small script run on a schedule, from outside the application, that behaves like a customer and records success or failure. And when the canary fails, the next question is "what else is red?", which is what a dashboard answers at a glance. Together they close the blind spot that internal metrics leave at the edge.

The idea in plain language

A CloudWatch dashboard is a customisable page of widgets, each showing a metric graph, a number, an alarm status, a log-query result or text. You build it once as JSON and keep it in Git. A good dashboard is built around one question: "Are customers OK, and if not, where should I look first?"

A CloudWatch Synthetics canary is a script that CloudWatch runs on a schedule (from one minute to many hours) inside AWS-managed Lambda infrastructure in your account. It calls your endpoints, checks results, uploads artifacts (logs, screenshots, HAR files) to S3, and publishes metrics such as success percent and duration.

Analogy: The dashboard is the instrument panel in the cockpit: speed, altitude, fuel, warning lights. The canary is the mystery shopper the bank sends into each branch at 09:00 every day to buy a coffee and see if the doors open, the till works and the receipt prints. Instruments tell you the engine is running; only the mystery shopper tells you the customer can actually get in.

Dashboard design, in reading order

An on-call engineer should read a dashboard top-down in about ten seconds:

RowContentsWhy
1. Customer impactAlarm status widget (composite first), canary success, transfers by outcome"Is it bad?"
2. Golden signalsLatency percentiles (p50, p99), traffic, error rate"What kind of bad?"
3. DependenciesPartner call latency and failures, database latency, queue age"Which part?"
4. ResourcesCPU, memory, connection pool, task count"Which resource?"
5. EvidenceLatest errors (log widget), recent deployments (text or annotation)"What changed?"

Leave off anything nobody has ever used during an incident. A dashboard with 60 widgets is a wall of noise. Prefer several small dashboards with a clear purpose (transfers-overview, partner-gateway, database) over one huge one.

What problem the AWS services solve

Dashboards. Pricing page, checked 2026-11-18: the free tier includes 3 custom dashboards with up to 50 metrics each per month; beyond that, dashboards are billed per dashboard per month ⚠ VERIFY the current price. Dashboards can show metrics from other Regions and, with cross-account observability, other accounts ⚠ VERIFY the setup.

Widgets you will use: metric (time series, single value, gauge, bar, pie), alarm (a status panel), log (a Logs Insights query result as a table or chart), text (Markdown for runbook links), and explorer for tag-driven views ⚠ VERIFY. Dashboard bodies are JSON with a widgets array; each widget has type, a position (x, y) on a 24-column grid, width, height and properties.

Synthetics. From the CloudWatch Synthetics documentation (checked 2026-11-18): canary runtimes use Node.js, Python or Java, with the Puppeteer, Playwright and Selenium frameworks. There is no .NET runtime, so a canary script for a .NET service is written in Node.js or Python and calls your service over HTTP. It is testing from outside, so the language of the service does not matter. Pricing page: 100 canary runs per month are free ⚠ VERIFY the per-run price afterwards, plus the costs of the S3 artifacts, the CloudWatch metrics and logs the canary writes, and any VPC networking.

Typical blueprints ⚠ VERIFY current names: a heartbeat monitor (load a URL), an API canary (call endpoints and assert on responses), a broken-link checker, a visual or UI workflow canary. This lesson writes an API-style heartbeat by hand so you see exactly what it does.

Honest limitations:

  • Runtime versions are retired. Pick the latest runtime when you create the canary and plan upgrades; the documentation has a support policy page for runtime deprecation. The lab's default syn-nodejs-puppeteer-9.1 is a placeholder: check the current list before applying ⚠ VERIFY.
  • A canary tests one path at a fixed frequency; a failure between runs is invisible, and a five-minute canary cannot detect a 90-second blip.
  • It costs money continuously while it runs, unlike an alarm on existing metrics.
  • Credentials for authenticated flows must live in Secrets Manager and be fetched by the script; never inline them.
  • Canaries do not replace load balancer metrics and alarms; they add an outside-in check.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Monitor workbooks and dashboardsCloudWatch dashboardsJSON bodies you can store in Git; no workbook parameters ⚠ VERIFY dashboard variables
Application Insights availability tests (URL ping, multi-step web test)CloudWatch Synthetics canariesYou write the script (Node.js, Python or Java) and it runs as Lambda in your account
Availability test locations worldwideCanaries run in the Region you create them (and VPCs you choose)Multi-Region coverage means creating canaries in several Regions
Dashboard tile JSONDashboard widget JSONBoth are exportable; CloudWatch uses a 24-column grid
Alert on availability test failureAlarm on the canary's SuccessPercent metricThe canary publishes standard CloudWatch metrics you alarm on
Pinned Log Analytics query tileLog widget with a Logs Insights queryRuns as the viewer and costs a query each refresh
Grafana (Azure Managed Grafana)Amazon Managed Grafana (separate service)Worth considering for cross-source dashboards; not covered here

Architecture

Security-boundary view. The canary runs under its own IAM role that can write to one S3 bucket, write its own log group and publish to the CloudWatchSynthetics namespace, nothing else. It sends real requests, so point it at a dedicated health or probe endpoint with no side effects: never at an endpoint that moves money. If it must authenticate, use a purpose-made probe identity with no access to customer data, and fetch the credential from Secrets Manager at run time. The artifact bucket holds response details; keep it private with Block Public Access and a short lifecycle.

Step-by-step implementation

Decide the dashboard's one question

"Are customers able to send money, and if not where do I look?" Write it at the top of the dashboard as a text widget with the runbook link.

Sketch rows before writing JSON

Use the five-row layout above. Put the composite alarm and the canary first.

Write the dashboard body as code

Start from the lab's dashboards/transfers.json.tftpl. Keep the file in Git and deploy it with IaC or put-dashboard. The console's Actions → View/edit source shows the JSON for any dashboard you created by hand, so you can copy one into the repository.

Add a log widget for evidence

A table of the latest errors with correlationId saves the on-call engineer a trip to Logs Insights. Remember each refresh runs a query and costs scanned data, so keep the time range short.

Write the canary script

Node.js, no dependencies beyond the built-in https module and the Synthetics logger. Call the health endpoint; fail the run on any status other than 200 or on a timeout.

Create the canary with a least-privilege role

A role that can write to the artifact bucket, its own log group and the CloudWatchSynthetics namespace. Schedule rate(5 minutes) for the lab; choose the interval from your detection target.

Alarm on SuccessPercent and add it to the dashboard

Alarm when SuccessPercent is below 100 in 2 of the last 3 runs, with missing data treated as breaching (a canary that stops running is a failure). Add the alarm to the composite if it should page.

Delete the canary when you finish the lab

Canaries bill per run until deleted. Delete the canary, its Lambda layers and the artifact bucket.

AWS Console instructions

Dashboard: Console → CloudWatch → Dashboards → Create dashboard, name it (hl-m11-transfers), choose a first widget type (Line, Number, Logs table, Alarm status, Text), pick the metric or query, and Save. To edit as JSON: Actions → View/edit source. Console labels change from time to time ⚠ VERIFY.

Canary: Console → CloudWatch → Application Signals (or Synthetics Canaries) → Synthetics Canaries → Create canary. Choose Use a blueprint (for example Heartbeat monitoring) or Author with inline code, enter the name (names are limited in length and character set ⚠ VERIFY), the runtime version, the endpoint, the schedule (for example Run continuously every 5 minutes), the S3 location for artifacts, and the IAM role (let the console create one for the lab). Under CloudWatch alarms the console can create an alarm on the canary for you.

AWS CLI

# Deploy the lab dashboard from the template (replace the two placeholders the Terraform template uses)
sed -e 's/${region}/us-east-1/g' -e 's#${log_group}#/hl-m11/transfers-api#g' \
  labs/m11-observe-banking-api/dashboards/transfers.json.tftpl > /tmp/transfers.json
aws cloudwatch put-dashboard --dashboard-name hl-m11-transfers \
  --dashboard-body file:///tmp/transfers.json --profile learn-sandbox

# Read it back (the JSON you can commit to Git)
aws cloudwatch get-dashboard --dashboard-name hl-m11-transfers --profile learn-sandbox --query DashboardBody --output text

# List and delete dashboards
aws cloudwatch list-dashboards --dashboard-name-prefix hl-m11 --profile learn-sandbox
aws cloudwatch delete-dashboards --dashboard-names hl-m11-transfers --profile learn-sandbox

# Canaries: list, check the last run, stop and delete
aws synthetics describe-canaries --profile learn-sandbox --query 'Canaries[].[Name,Status.State,RuntimeVersion]' --output table
aws synthetics get-canary-runs --name hl-m11-health --max-results 3 --profile learn-sandbox --query 'CanaryRuns[].Status.State'
aws synthetics stop-canary --name hl-m11-health --profile learn-sandbox
aws synthetics delete-canary --name hl-m11-health --delete-lambda --profile learn-sandbox

On PowerShell, write the sed substitution as (Get-Content file -Raw).Replace('${region}','us-east-1').

Infrastructure as Code

The lab's stack includes the dashboard (CDK widgets, and a JSON template for Terraform) and, in Terraform, an optional canary. The CDK widgets in code:

ObservabilityStack.cs (excerpt)
var dashboard = new Dashboard(this, "Dashboard", new DashboardProps
{
    DashboardName = "hl-m11-transfers",
    DefaultInterval = Duration.Hours(3),
});

dashboard.AddWidgets(
    new AlarmStatusWidget(new AlarmStatusWidgetProps { Title = "Alarms", Alarms = [page, failed, floor, slow, rate], Width = 8, Height = 4 }),
    new SingleValueWidget(new SingleValueWidgetProps { Title = "Transfers by outcome (last 3 h)", Metrics = [Count("succeeded"), Count("rejected"), Count("failed")], Width = 8, Height = 4 }),
    new GraphWidget(new GraphWidgetProps { Title = "p99 transfer latency (ms)", Left = [latencyAll], LeftAnnotations = [new HorizontalAnnotation { Value = 1500, Label = "warning" }], Width = 8, Height = 4 }));

dashboard.AddWidgets(
    new LogQueryWidget(new LogQueryWidgetProps
    {
        Title = "Latest errors",
        LogGroupNames = [logGroup.LogGroupName],
        QueryString = "fields @timestamp, correlationId, path, message | filter level = \"Error\" | sort @timestamp desc | limit 20",
        Width = 24, Height = 6,
    }));

A CDK canary uses the Amazon.CDK.AWS.Synthetics construct library; check its runtime constants for your CDK version before writing it ⚠ VERIFY, because runtime names change as AWS retires versions.

Complete code

The canary script

The script the lab's Terraform packages (Node.js; no third-party dependencies):

heartbeat.js
const log = require('SyntheticsLogger');
const https = require('https');

// Calls the health URL and fails the run on any status other than 200.
const heartbeat = async function () {
  const url = process.env.TARGET_URL;
  const status = await new Promise((resolve, reject) => {
    https.get(url, (res) => { res.resume(); resolve(res.statusCode); }).on('error', reject);
  });
  log.info('health returned ' + status);
  if (status !== 200) { throw new Error('Health check returned ' + status); }
};

exports.handler = async () => { return await heartbeat(); };

The file is zipped as nodejs/node_modules/heartbeat.js, the path layout Synthetics expects for Node.js canaries, and the handler is heartbeat.handler ⚠ VERIFY the layout for your chosen runtime.

A .NET tool that builds and saves the dashboard

Dashboards are JSON, so a small .NET tool can generate them with type-safe helpers. It compiles against AWSSDK.CloudWatch 4.1.1.2:

DashboardTool.cs
using System.Text.Json.Nodes;
using Amazon.CloudWatch;
using Amazon.CloudWatch.Model;

const string region = "us-east-1";
const string logGroup = "/hl-m11/transfers-api";

// A metric widget: a time series on the 24-column grid.
static JsonObject MetricWidget(int x, int y, string title, string region, JsonArray metrics, string stat = "Sum") => new()
{
    ["type"] = "metric",
    ["x"] = x, ["y"] = y, ["width"] = 12, ["height"] = 6,
    ["properties"] = new JsonObject
    {
        ["title"] = title,
        ["region"] = region,
        ["view"] = "timeSeries",
        ["stacked"] = false,
        ["stat"] = stat,
        ["period"] = 60,
        ["metrics"] = metrics,
    },
};

var body = new JsonObject
{
    ["widgets"] = new JsonArray
    {
        MetricWidget(0, 0, "Transfers by outcome", region, new JsonArray
        {
            new JsonArray("Harbourline/Payments", "TransferCount", "Service", "transfers-api", "Environment", "learn", "Outcome", "succeeded"),
            new JsonArray("Harbourline/Payments", "TransferCount", "Service", "transfers-api", "Environment", "learn", "Outcome", "failed"),
        }),
        new JsonObject
        {
            ["type"] = "log", ["x"] = 12, ["y"] = 0, ["width"] = 12, ["height"] = 6,
            ["properties"] = new JsonObject
            {
                ["title"] = "Latest errors",
                ["region"] = region,
                ["view"] = "table",
                ["query"] = $"SOURCE '{logGroup}' | fields @timestamp, correlationId, path, message | filter level = \"Error\" | sort @timestamp desc | limit 20",
            },
        },
    },
};

using var cw = new AmazonCloudWatchClient();
var response = await cw.PutDashboardAsync(new PutDashboardRequest
{
    DashboardName = "hl-m11-transfers",
    DashboardBody = body.ToJsonString(),
});

foreach (var message in response.DashboardValidationMessages)
    Console.WriteLine($"{message.DataPath}: {message.Message}");
Console.WriteLine(response.DashboardValidationMessages.Count == 0 ? "Dashboard saved." : "Dashboard saved with validation messages.");

Code explained line by line

  1. MetricWidget returns a JsonObject rather than a string template, so a typo is a compile-time error and quotes need no escaping.
  2. ["type"] = "metric" selects the widget kind; x, y, width, height place it on a grid 24 columns wide. Two widgets of width 12 fill a row.
  3. The metrics array uses CloudWatch's compact notation: namespace, metric name, then dimension name/value pairs, and an optional options object at the end (for example {"stat": "p99"}). The lab spells every row out in full. CloudWatch also accepts a "..." shorthand that repeats parts of the previous row, but its exact rules are easy to get wrong, so check the dashboard-body documentation before using it ⚠ VERIFY.
  4. stat = "Sum" with period = 60 matches the alarm definitions; dashboards that use different statistics from the alarm confuse the person reading them.
  5. The log widget's query starts with SOURCE '<log group>' followed by Logs Insights syntax. It runs when the dashboard loads, as the viewer.
  6. PutDashboardAsync replaces the whole dashboard (it is an upsert by name). DashboardValidationMessages lists problems with a widget while still saving the rest, so always print them.
  7. In the canary script, res.resume() drains the response so the socket closes, throw marks the run as failed, and exports.handler is what the Synthetics runtime calls.

Security considerations

⚠ Learning shortcut — not for production

The lab canary calls a plain health URL with no authentication, from the public internet, with the artifact bucket allowed to be force-destroyed. In production: probe a dedicated internal-safe endpoint, run the canary inside the VPC for private APIs, keep artifacts for a defined time in a private, encrypted bucket, store any probe credential in Secrets Manager, and use a probe identity with no customer-data access.

  • Probe endpoints must be side-effect free. A canary that creates a transfer every five minutes creates real or test money movements, fraud signals and clutter. Use read-only health endpoints, or an isolated synthetic account that the bank's controls recognise.
  • Dashboards expose information. Operational dashboards reveal architecture and volumes. Restrict cloudwatch:GetDashboard and the data behind widgets to operations roles, and never put customer data in a widget.
  • Log widgets run as the viewer. A viewer without log permissions sees an error, not the data, which is correct behaviour.
  • Sharing. CloudWatch can share dashboards publicly or with specific accounts ⚠ VERIFY; do not enable public sharing for operational dashboards of a bank.
  • Cost as a control. A runaway canary frequency of one minute on many endpoints adds up; review it like any recurring spend.

Banking use case

Harbourline's payments team runs a five-minute canary from us-east-1 against the Transfers API's public health endpoint and, from a private subnet, against the internal partner gateway. The canary success alarm feeds the page composite (a failed canary means customers probably cannot reach the service even though no internal metric is red). The on-call dashboard has the composite and canary status at the top, transfers by outcome and p99 latency below, a partner row (call latency, failures, circuit-breaker state) and a log table of the latest errors. A text widget links the runbook and the status page. On a certificate-expiry incident the canary alarm fired two minutes after expiry while every internal metric was green, which is exactly the blind spot it exists for. All data is fictional.

Common mistakes

  • Dashboards with 60 widgets. Fix: the five-row rule; small purpose-built dashboards.
  • Dashboards built by hand in the console. They are lost or drift. Fix: JSON in Git and IaC.
  • Mixing statistics and periods between the dashboard and the alarm. Fix: the same Sum/p99 and 60-second period.
  • A canary against a state-changing endpoint. Fix: a dedicated health endpoint or a read-only call.
  • No alarm on the canary. It runs, fails, and nobody notices. Fix: alarm on SuccessPercent, with missing data breaching.
  • Never deleting a lab canary. It bills per run. Fix: the cleanup commands below.
  • Assuming a canary language must match your service. It does not; Node.js or Python calling HTTP is fine for a .NET service.
  • Heavy log widgets refreshing every minute for many viewers. Fix: narrow time range, few such widgets.

Troubleshooting

SymptomLikely causeFix
Widget shows "No data available"Dimension values or period do not match the published metricOpen the metric in All metrics and copy exact values
put-dashboard returns validation messagesInvalid widget property or unknown metric notationRead DashboardValidationMessages; fix the named DataPath
Log widget empty or access errorViewer lacks logs:StartQuery, or the log group name is wrongGrant read permissions; check the SOURCE string
Canary stays in ERROR / FAILEDScript error, endpoint unreachable, bad runtime version or roleCheck the canary's run artifacts and log group under /aws/lambda/cwsyn-<name>-...
Canary succeeds but customers failIt tests a different path than customers useAdd a probe for the failing path; compare with load balancer metrics
Canary alarm in INSUFFICIENT_DATACanary not started, or missing-data policy wrongStart it; use breaching for the missing-data policy
Dashboard edits disappear after deployIaC overwrites console editsMake the change in code

Cost and cleanup

Dashboard: within the free tier of three dashboards with up to 50 metrics each, so the lab dashboard costs nothing if it is among your first three; beyond that a per-dashboard monthly charge applies ⚠ VERIFY. Canary: the first 100 runs per month are free; at a five-minute schedule a canary runs about 8,640 times per month, so leaving it on is not free. Run it for an hour (12 runs) and delete it. Estimate: well under USD 0.10 for the hour at list prices plus pennies for S3 and logs ⚠ VERIFY the per-run price. If you do not want to create a canary at all, use the free design-only alternative: read the script, run node locally against your own docker compose API (set TARGET_URL to http://localhost:8080/health and replace https with http in a local copy), and write the alarm definition without deploying it.

# Delete the canary, its Lambda, the artifact bucket and the dashboard
aws synthetics stop-canary --name hl-m11-health --profile learn-sandbox
aws synthetics delete-canary --name hl-m11-health --delete-lambda --profile learn-sandbox
aws s3 rb s3://hl-m11-canary-<account-id> --force --profile learn-sandbox
aws cloudwatch delete-dashboards --dashboard-names hl-m11-transfers --profile learn-sandbox
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab --profile learn-sandbox

With Terraform, terraform destroy removes all of it (the bucket uses force_destroy for the lab).

Exercises

  • 🟢 Guided: Deploy the lab dashboard with put-dashboard. Open it, find each widget, then open View/edit source and change one widget's title in the JSON; save it and compare with the file in the repository.
  • 🟡 Challenge: Add a text widget at the top with the runbook link and an alarm status widget showing the composite, and add a "Partner latency" row with a made-up metric name, spelling out each metric row in full.
  • 🔴 Live problem: Last Tuesday customers could not reach the API for twelve minutes (an expired certificate on the load balancer). Nothing paged; the dashboards were green. Write the post-incident actions: what monitoring would have detected it, where it should run, how fast, what it costs, and how you prevent expiry in the first place (see Certificates, TLS and mTLS).

Assignment

Create dashboards/ and canary/ folders in a service repository containing: a dashboard JSON following the five-row layout with at least one alarm widget and one log widget, a canary script and its IaC (CDK or Terraform), an alarm on the canary's success percentage, and a README explaining what each widget is for and who looks at it.

Solutions

Knowledge check

Interview questions

  • What should be on an on-call dashboard? Customer impact first (composite alarm, canary, outcomes), then golden signals, dependencies, resources, and evidence such as latest errors; omit anything nobody uses in incidents.
  • What is a canary and why do you need one? A scheduled script from outside the application that checks real endpoints; it finds edge failures (DNS, TLS, WAF, load balancer) that internal metrics cannot see.
  • How do you avoid a canary causing harm? Use a read-only or synthetic-account endpoint, least-privilege role, secrets from Secrets Manager and a sensible frequency.
  • Dashboard-as-code: how? JSON body in Git deployed with IaC or put-dashboard, with the console used only to prototype.
  • What does a canary cost to run continuously? Per-run charges beyond the free runs, plus S3, logs and metrics; frequency drives cost.

Certification practice

Completion checklist

  • I can lay out a dashboard in reading order and justify each row
  • I deployed (or reviewed) the dashboard JSON as code
  • I read the canary script and know why it tests from outside
  • I priced the canary and either ran it for an hour or used the free alternative
  • I deleted the canary, its bucket and the dashboard, and confirmed nothing with Project=cea-lab remains

Key takeaways

  • Build dashboards in the order on-call reads them: impact, golden signals, dependencies, resources, evidence.
  • Keep dashboards in Git; the console is for prototyping.
  • A canary is a customer's-eye probe that sees edge failures your internal metrics cannot; point it at a side-effect-free endpoint.
  • Synthetics has no .NET runtime; a Node.js or Python script calls your service over HTTP.
  • Alarm on the canary with missing data treated as breaching, and delete lab canaries promptly: they bill per run.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page