Skip to Content
Cloud Engineering Academy
11. Monitoring, Logging & Troubleshooting

CloudWatch alarms and composite alarms: thresholds, missing data and noise reduction

Design alarms that page a human only when customers are affected: choose statistics and periods, use M-out-of-N evaluation, pick the right missing-data policy, add metric math and anomaly detection where they fit, and combine alarms into one composite page.

intermediate⏱ 1 h 40 min🟡 Low costSOA-C03DOP-C02DVA-C03

Assumes: .NET 10 LTS · AWSSDK.CloudWatch 4.1.1.2 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Describe a CloudWatch metric alarm: its metric, statistic, period, evaluation periods, datapoints to alarm, threshold, comparison operator and treat-missing-data policy.
  • Explain the three alarm states (OK, ALARM, INSUFFICIENT_DATA) and exactly when an alarm sends a notification.
  • Choose the right missing-data policy for a failure-count alarm and for a heartbeat or traffic-floor alarm.
  • Build a metric-math alarm for a failure rate with a minimum-traffic guard, and say when anomaly detection is the better choice.
  • Combine alarms with a composite alarm rule so one incident produces one page, and rehearse the page with set-alarm-state.

Prerequisites

Why this matters in banking

Two failure patterns, both costly.

The pager that cried wolf. Operations set an alarm "CPU above 70% for one minute" on forty containers. During salary-day load it fires twenty times in an hour. The on-call engineer silences the topic. At 11:20 transfers really do start failing, and nobody is listening.

The silence that looked like health. A deployment breaks the metric publisher: no data is sent at all. The alarm on "failed transfers" stays green, because no failures were reported. Customers cannot pay for forty minutes. Silence was the symptom, and the alarm treated it as good news.

Good alarm design addresses both: alarm only on symptoms customers feel, require sustained breach (so a single spike does not page), decide deliberately what missing data means, and funnel related alarms through a composite so one incident means one page.

The idea in plain language

A CloudWatch alarm watches one metric (or a calculation over metrics) and compares each period's value with a threshold. It is always in one of three states:

StateMeaning
OKThe metric is within the threshold
ALARMThe metric breached the threshold enough times (see M of N)
INSUFFICIENT_DATANot enough data to decide, for example a new alarm or a metric that stopped reporting

An alarm triggers its actions only when the state changes (the exception is Auto Scaling actions, which repeat each minute while the alarm stays in the state). That is the verified behaviour from the CloudWatch documentation: an alarm that stays in ALARM for an hour sends one notification, not sixty.

The knobs, in the order you should think about them:

SettingQuestion it answersBanking example
Metric and statisticWhat do I measure and how do I combine data points?TransferCount filtered to Outcome=failed, Sum
PeriodHow long is one evaluation window?60 seconds
Threshold and comparisonWhat is "bad"?Greater than 5
Evaluation periods (N)How many recent periods do I look at?5
Datapoints to alarm (M)How many of those N must be bad?3 (so "3 out of 5")
Treat missing dataWhat does "no data" mean?notBreaching for failures; breaching for a traffic floor
ActionsWhat happens on a state change?Publish to an SNS topic

M out of N is the most useful noise filter. "Three of the last five minutes above 5" ignores a one-minute spike, still reacts within about three to five minutes, and tolerates a single good minute in the middle of a bad patch. Setting M equal to N ("5 of 5") is stricter and slower.

Analogy: A smoke alarm that sounded every time someone made toast would be unplugged within a week. M-out-of-N is the alarm waiting to see smoke for several seconds, and the composite alarm is the building's fire panel that sounds the bell once when the kitchen and the corridor both report smoke, instead of ringing every detector separately.

What problem the AWS service solves

CloudWatch alarms turn metrics into decisions: notify a person, run an automated action, or both.

Alarm types (from the CloudWatch alarms documentation, checked 2026-11-18):

TypeWatchesUse when
Metric alarmOne metric, or the result of a metric-math expressionAlmost always
Composite alarmThe states of other alarms through a rule such as ALARM(a) OR ALARM(b)Reduce noise; express "page only if" logic
Log alarmThe result of a Logs Insights query run on a scheduleThe signal exists only in logs; billed for scanned data on every run ⚠ VERIFY
PromQL alarmA Prometheus-style query on metrics ingested through CloudWatch's OTLP endpointYou already run OpenTelemetry metrics into CloudWatch ⚠ VERIFY availability

This lesson uses metric and composite alarms, which every exam covers.

What alarms can do: send to SNS topics; perform EC2 actions (stop, terminate, reboot, recover) and Auto Scaling actions (metric alarms only); and create investigations, Systems Manager OpsItems or incidents. Composite alarms can send SNS notifications and create investigations, OpsItems or incidents, but cannot perform EC2 or Auto Scaling actions. The incident action depends on Systems Manager Incident Manager, which has been closed to new customers since 2025-11-07, so this module routes pages through alarm → SNS → on-call tool ⚠ VERIFY the currently recommended replacement.

Honest limitations:

  • Evaluation lag. Alarms evaluate on metric periods; with a one-minute period and "3 of 5", detection takes minutes. If you need seconds, use high-resolution metrics and a shorter period, accepting higher cost and noise.
  • Maximum evaluation range. Period × evaluation periods is limited: up to one day for periods under one hour, up to seven days for periods of an hour or longer.
  • Alarm history is kept for 30 days. Copy state-change events elsewhere if you need longer evidence.
  • Alarms are only as honest as their metric. A broken publisher makes data disappear; that is what the missing-data policy is for.
  • CloudWatch does not check that your actions exist. A typo in an SNS ARN gives you an alarm that silently notifies nobody. Rehearse every alarm with set-alarm-state.
  • Pricing (checked 2026-11-18, US East examples ⚠ VERIFY): USD 0.10 per standard alarm metric per month, USD 0.50 per composite alarm, and an anomaly-detection alarm bills its three underlying metrics (USD 0.30). Free tier: 10 alarm metrics.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Monitor metric alert ruleCloudWatch metric alarmBoth threshold-based; CloudWatch has explicit M-out-of-N and a missing-data policy
Action group (email, SMS, webhook, Logic App)SNS topic with subscriptions, or EventBridge on alarm state changeAlarm → SNS is the standard; add subscribers (email, HTTPS, Lambda, chat bridges) separately
Dynamic thresholds (ML baseline)Anomaly detection alarmNeeds about two weeks of history to learn; model band set with a width parameter
Alert processing rules (suppression)Composite alarms, alarm actions suppression, DisableAlarmActionsComposite rules express logic; suppression needs a deliberate design
Smart groups / alert correlationComposite alarm rulesYou write the logic yourself
Log search alert ruleLog alarm (new) or metric filter + metric alarmMetric filter plus alarm is the long-standing route
Alert severity 0 to 4No built-in severityUse alarm naming, tags and which topic the alarm publishes to
Test action groupset-alarm-stateSets any state until the next evaluation

Architecture

Security-boundary view. The SNS topic policy must allow cloudwatch.amazonaws.com to publish, scoped with aws:SourceArn to your alarms, and should deny non-TLS publishes. Control who can change alarms (cloudwatch:PutMetricAlarm, DeleteAlarms, DisableAlarmActions): disabling actions silences the pager, so treat that permission as sensitive and review it in CloudTrail (see CloudTrail for banks).

Choosing the right missing-data policy

When a period has no data point, the alarm needs a rule. The four options (from the CloudWatch documentation on missing data; the lab's AlarmSimulator models them):

PolicyTreats missing points asRight forWrong for
missing (default)Not counted; if the whole window is missing, the alarm goes to INSUFFICIENT_DATARarely: it hides outagesAnything you must page on
notBreachingGood (inside the threshold)Error counts that are only published when errors happenTraffic floors
breachingBad (outside the threshold)Heartbeats, traffic floors, "no transfers at all"Error counts (every quiet night would page)
ignoreKeep the current stateMetrics that are legitimately sparse and where flapping would hurtAnything where silence matters

The rule of thumb: for each alarm ask "if the metric vanished, is that good news or bad news?" Failures vanishing is good news (notBreaching). Traffic vanishing is bad news (breaching). Write the answer in the alarm description.

You can try all of these in code without AWS:

AlarmSimulatorTests.cs (excerpt)
// Alarm on failed transfers: more than 5 in a minute, 3 out of 5 minutes.
var def = new AlarmDefinition(5, Comparison.GreaterThanThreshold, EvaluationPeriods: 5, DatapointsToAlarm: 3);

AlarmSimulator.Evaluate(def, [1, 9, 2, 8, 7]);       // Alarm: three minutes above 5
AlarmSimulator.Evaluate(def, [0, 0, 40, 0, 0]);      // Ok: one spike does not page
AlarmSimulator.Evaluate(def, [null, null, null, null, null]);   // InsufficientData with the default policy

// A traffic floor: no data means no traffic, which is the problem.
var floor = new AlarmDefinition(1, Comparison.LessThanThreshold, 3, 3, TreatMissingData.Breaching);
AlarmSimulator.Evaluate(floor, [null, null, null]);  // Alarm

The simulator is a teaching model: CloudWatch has additional edge cases (for example, for sparse metrics it may look back past the window), so treat the documentation as the source of truth.

Metric math and anomaly detection

A failure rate with a traffic guard

"More than 5 failures per minute" is a good first alarm, but it behaves differently at 02:00 (five failures may be the only traffic) and at 09:00 (five failures is noise in ten thousand transfers). A rate with a minimum-traffic guard is more robust:

IF(total >= 20, 100 * failed / total, 0)

where failed is TransferCount with Outcome=failed and total is TransferCount without the outcome dimension (this is why the lab publishes two dimension sets: the all-outcomes series exists so this expression works). When fewer than 20 transfers flow in a minute the expression returns 0, so one failure at 03:00 cannot page. An alarm on this expression with threshold 5 (percent) and "3 of 5" reads: "more than 5% of transfers failed in at least three of the last five minutes, when real traffic exists".

Use the rate alarm together with the count alarm: counts catch outages at low traffic through the traffic-floor alarm, rates catch partial degradation at high traffic.

Anomaly detection

An anomaly detection alarm learns a model of the metric from its history (the documentation describes about two weeks of training ⚠ VERIFY) and alarms when the value leaves an expected band, set by ANOMALY_DETECTION_BAND(metric, k) where larger k widens the band. It suits metrics with daily or weekly shape, such as transfers per minute (a quiet Sunday morning versus a busy Friday evening) and is a good fit for the traffic signal. It suits badly: brand-new metrics, metrics that shift legitimately after a release, and anything with a hard business limit (use a static threshold). Anomaly alarms also bill three underlying metrics.

Composite alarms

A composite alarm has a rule expression over other alarms' states:

ALARM("hl-m11-failed-transfers") OR ALARM("hl-m11-traffic-floor")

The syntax supports ALARM(...), OK(...), INSUFFICIENT_DATA(...), TRUE, FALSE, AND, OR, NOT and parentheses. Typical designs:

GoalRule
One page for a customer-impacting incidentALARM(failed) OR ALARM(floor)
Require two independent symptoms before pagingALARM(failed) AND ALARM(latency)
Do not page while a planned deployment alarm is activeALARM(failed) AND NOT ALARM(deploy-in-progress)
Page only for the whole service, not one taskALARM(az-a-errors) AND ALARM(az-b-errors)

Put actions on the composite only, not on its children. The child alarms still change state, which you see on the dashboard and in history, but only the composite notifies. CloudWatch also lets you suppress a composite's actions while another alarm is in ALARM (an actions suppressor) ⚠ VERIFY the exact property names for your IaC tool.

The lab's CompositeAlarmRule parses and evaluates this syntax offline so you can test a rule before deploying it:

CompositeAlarmRuleTests.cs (excerpt)
var rule = CompositeAlarmRule.Parse(
    "ALARM(\"hl-failed-transfers\") AND ALARM(\"hl-p99-latency\") AND NOT ALARM(\"hl-deploy-in-progress\")");

rule.Evaluate(new Dictionary<string, AlarmState>
{
    ["hl-failed-transfers"] = AlarmState.Alarm,
    ["hl-p99-latency"] = AlarmState.Alarm,
    ["hl-deploy-in-progress"] = AlarmState.Ok,
});   // true: page

Limits to remember: cross-account composite alarms are not supported, and a composite is billed at its own price (USD 0.50 per month at the checked list price).

Step-by-step implementation

List symptoms, not causes

For the Transfers API: failed transfers, traffic stopped, high failure rate, slow tail. Everything else (CPU, memory, connection pool) goes on the dashboard.

Fix each alarm's statistic, period and M of N

Sum per 60 seconds for counts; p99 per 60 seconds for latency; M of N of 3 of 5 for noisy signals, 10 of 10 for a traffic floor (ten quiet minutes in a row).

Decide the missing-data policy and write it down

Put the reason in the alarm description, for example "silence is good news".

Add the guard for ratios

Use metric math with a minimum-traffic condition, so low-volume periods cannot page.

Create the page topic and the composite

One SNS topic for pages, one composite alarm with the only action. Include the runbook path in AlarmDescription; the notification carries it.

Subscribe a human and confirm

For an email subscription, click the confirmation link. Then rehearse the page.

Rehearse and review

aws cloudwatch set-alarm-state on a child alarm and check the page arrives. In a game day, inject a real fault (the lab's X-Fault: partner-down). Review every alarm after real incidents: did it fire, was it actionable, was it fast enough?

AWS Console instructions

Create an alarm: Console → CloudWatch → Alarms → All alarms → Create alarm → Select metric. Browse to Harbourline/Payments → the Service, Environment, Outcome dimensions → TransferCount for failed; choose the statistic Sum and period 1 minute. Conditions: Static, Greater than, 5. Additional configuration: Datapoints to alarm 3 out of 5, Missing data treatment = Treat missing data as good (not breaching). Next: choose an SNS topic notification for In alarm (and OK), name and describe the alarm, Create alarm.

Composite alarm: Alarms → Create alarm → Create composite alarm, choose the child alarms and the condition (the console builds the rule, or you can type the expression), then set the actions on the composite.

Rehearse: open an alarm, there is no console button for SetAlarmState; use the CLI below. Console labels change over time ⚠ VERIFY.

AWS CLI

# A metric alarm: failed transfers, 3 of the last 5 minutes above 5. Silence is good news.
aws cloudwatch put-metric-alarm --profile learn-sandbox \
  --alarm-name hl-m11-failed-transfers \
  --alarm-description "More than 5 failed transfers per minute in 3 of 5 minutes. Silence is good news. Runbook: docs/runbooks/failed-transfers.md" \
  --namespace Harbourline/Payments --metric-name TransferCount \
  --dimensions Name=Service,Value=transfers-api Name=Environment,Value=learn Name=Outcome,Value=failed \
  --statistic Sum --period 60 --evaluation-periods 5 --datapoints-to-alarm 3 \
  --threshold 5 --comparison-operator GreaterThanThreshold --treat-missing-data notBreaching

# A composite alarm. It is the ONLY alarm with an action.
aws cloudwatch put-composite-alarm --profile learn-sandbox \
  --alarm-name hl-m11-page \
  --alarm-rule 'ALARM("hl-m11-failed-transfers") OR ALARM("hl-m11-traffic-floor")' \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:hl-m11-oncall-page

# Rehearse the page: force a child alarm into ALARM (it resets at the next evaluation)
aws cloudwatch set-alarm-state --alarm-name hl-m11-failed-transfers --state-value ALARM \
  --state-reason "Game day rehearsal" --profile learn-sandbox

# See what happened and why
aws cloudwatch describe-alarm-history --alarm-name hl-m11-page --history-item-type StateUpdate --max-items 5 --profile learn-sandbox

# List every alarm in ALARM state
aws cloudwatch describe-alarms --state-value ALARM --query 'MetricAlarms[].AlarmName' --profile learn-sandbox

# Silence actions during planned maintenance (and remember to re-enable!)
aws cloudwatch disable-alarm-actions --alarm-names hl-m11-page --profile learn-sandbox
aws cloudwatch enable-alarm-actions  --alarm-names hl-m11-page --profile learn-sandbox

Infrastructure as Code

The lab's stack (labs/m11-observe-banking-api/infra) contains four metric alarms, the composite and the SNS topic, in both tools. Key excerpts:

ObservabilityStack.cs (excerpt)
var failed = new Alarm(this, "FailedTransfersAlarm", new AlarmProps
{
    AlarmName = "hl-m11-failed-transfers",
    Metric = Count("failed"),                       // TransferCount, Outcome=failed, Sum, 1 minute
    Threshold = 5,
    EvaluationPeriods = 5,
    DatapointsToAlarm = 3,                          // 3 of 5
    ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
    TreatMissingData = TreatMissingData.NOT_BREACHING,
});

var failureRate = new MathExpression(new MathExpressionProps
{
    Expression = "IF(total >= 20, 100 * failed / total, 0)",
    UsingMetrics = new Dictionary<string, IMetric> { ["failed"] = Count("failed"), ["total"] = totalCount },
    Label = "Failure rate %",
    Period = Duration.Minutes(1),
});

var page = new CompositeAlarm(this, "PageAlarm", new CompositeAlarmProps
{
    CompositeAlarmName = "hl-m11-page",
    AlarmRule = AlarmRule.AnyOf(failed, floor),    // the only alarm with an action
});
page.AddAlarmAction(new SnsAction(topic));
page.AddOkAction(new SnsAction(topic));

Both stacks were compiled (CDK) and validated (Terraform) on 2026-11-18; they have not been deployed by the test run.

Complete code

Creating and rehearsing alarms from .NET with the AWS SDK v4 (compiles against AWSSDK.CloudWatch 4.1.1.2), useful for tooling such as a "game day" console app:

AlarmTool.cs
using Amazon.CloudWatch;
using Amazon.CloudWatch.Model;

using var cw = new AmazonCloudWatchClient();

await cw.PutMetricAlarmAsync(new PutMetricAlarmRequest
{
    AlarmName = "hl-m11-failed-transfers",
    AlarmDescription = "More than 5 failed transfers per minute in 3 of the last 5 minutes",
    Namespace = "Harbourline/Payments",
    MetricName = "TransferCount",
    Dimensions =
    [
        new Dimension { Name = "Service", Value = "transfers-api" },
        new Dimension { Name = "Environment", Value = "learn" },
        new Dimension { Name = "Outcome", Value = "failed" },
    ],
    Statistic = Statistic.Sum,
    Period = 60,
    EvaluationPeriods = 5,
    DatapointsToAlarm = 3,
    Threshold = 5,
    ComparisonOperator = ComparisonOperator.GreaterThanThreshold,
    TreatMissingData = "notBreaching",
});

await cw.PutCompositeAlarmAsync(new PutCompositeAlarmRequest
{
    AlarmName = "hl-m11-page",
    AlarmRule = "ALARM(\"hl-m11-failed-transfers\")",
    AlarmActions = ["arn:aws:sns:us-east-1:111122223333:hl-m11-oncall-page"],
});

// Rehearse the page without waiting for a real failure.
await cw.SetAlarmStateAsync(new SetAlarmStateRequest
{
    AlarmName = "hl-m11-failed-transfers",
    StateValue = StateValue.ALARM,
    StateReason = "Game day rehearsal",
});

var described = await cw.DescribeAlarmsAsync(new DescribeAlarmsRequest
{
    AlarmNames = ["hl-m11-failed-transfers"],
    AlarmTypes = [AlarmType.MetricAlarm],
});
Console.WriteLine(described.MetricAlarms[0].StateValue);

The offline simulators are labs/m11-observe-banking-api/src/Harbourline.Observability/Alarms/AlarmSimulator.cs and CompositeAlarmRule.cs, with tests in AlarmTests.cs.

Code explained line by line

  1. DatapointsToAlarm = 3 with EvaluationPeriods = 5 is the "3 out of 5" rule. Leaving DatapointsToAlarm unset means every evaluation period must breach.
  2. TreatMissingData takes a string in the SDK ("breaching", "notBreaching", "ignore", "missing"), unlike the typed constants for statistics and states.
  3. Statistic.Sum, ComparisonOperator.GreaterThanThreshold and StateValue.ALARM are SDK v4 constant classes. Passing the wrong type is a compile error, which is the point of using them.
  4. PutCompositeAlarmAsync references child alarms by name inside the rule string; child alarms must exist when the composite is created, and the rule's quotes are escaped as \".
  5. AlarmActions is an array of ARNs. Putting the action on the composite and not on the child alarms is what gives you one page per incident.
  6. SetAlarmStateAsync forces a state until the next evaluation. It is a rehearsal tool: it also proves the SNS topic, subscription and policy are wired correctly.
  7. DescribeAlarmsAsync filters by AlarmTypes so you do not accidentally get composites when you want metric alarms.

Security considerations

⚠ Learning shortcut — not for production

The lab subscribes a plain email address and gives your sandbox identity broad CloudWatch rights. In production, page through an on-call tool subscribed to the topic over HTTPS, encrypt the topic with a customer managed KMS key whose policy allows CloudWatch, restrict cloudwatch:DisableAlarmActions, DeleteAlarms and PutMetricAlarm to the owning team, and monitor changes to alarms in CloudTrail.

  • Protect the pager. An attacker (or a tired engineer) who disables alarm actions silences the bank's detection. Alert on DisableAlarmActions and DeleteAlarms events.
  • Notification content. Alarm descriptions and state reasons travel through SNS to email and chat; they must contain no personal data or secrets.
  • Topic policy. Allow only cloudwatch.amazonaws.com with aws:SourceArn matching your alarms, and deny aws:SecureTransport = false. The lab's Terraform shows both statements.
  • Alarms as controls. A detection alarm may be part of your operational-resilience evidence. Version it in Git and review changes like code. Whether it meets a regulator's expectation is for your compliance team.

Banking use case

Harbourline's payments platform has four alarms for the Transfers API and one page. Failed transfers and traffic floor feed the composite hl-prod-page, which is the only alarm wired to the on-call tool. Failure rate creates a ticket and is shown on the dashboard, and p99 latency is a dashboard tile that engineers consult during an incident. A planned deployment raises hl-deploy-in-progress for the release window; the composite rule includes AND NOT ALARM(hl-deploy-in-progress) for the failure-rate alarm only, so a bad deploy is still paged after the window. Each quarter the team lists every page and records whether it needed human action; pages with zero actions are deleted or demoted. All of this uses fictional data.

Common mistakes

  • Wrong missing-data policy. The default hides outages; breaching on an error counter pages every quiet night. Fix: ask "is silence good or bad?" per alarm.
  • Alarming on single data points. One bad minute pages. Fix: M of N.
  • Alarming on averages of latency. Fix: p95 or p99.
  • Actions on every child alarm. One incident produces many pages. Fix: actions on the composite only.
  • Never testing the page. The topic ARN has a typo, or the subscription is unconfirmed. Fix: set-alarm-state rehearsal and a periodic game day.
  • Absolute thresholds that ignore traffic. Fix: rate alarms with a minimum-traffic guard.
  • Alarm sprawl. Hundreds of alarms nobody owns. Fix: tags with owner and runbook, and a review.
  • Forgetting to re-enable actions after maintenance. Fix: put re-enabling in the maintenance checklist, and alarm on DisableAlarmActions events lasting too long.

Troubleshooting

SymptomLikely causeFix
Alarm stays in INSUFFICIENT_DATANo data yet, dimensions differ from the published metric, or period shorter than the publish intervallist-metrics and copy exact dimension values; check the period
Alarm in ALARM but no notificationUnconfirmed SNS subscription, wrong topic ARN, or topic policy blocks CloudWatchConfirm the subscription; check describe-alarm-history for action failures; fix the topic policy
Notification arrives once and never againActions fire on state changes onlyExpected; if you need reminders, use an on-call tool that re-notifies
Alarm flaps between OK and ALARMThreshold close to normal values, M equals N, or ignore policyUse M of N, widen the period or add hysteresis through a composite
Alarm never fires during a real outageMissing-data policy notBreaching on a metric that stops when the service diesAdd a traffic-floor alarm with breaching
Composite creation failsA child alarm does not exist yet, or quoting error in the ruleCreate children first; escape quotes
set-alarm-state is overwritten immediatelyExpected: the next evaluation resets the stateRehearse by watching the notification, not the state

Cost and cleanup

The lab's four metric alarms and one composite cost, at the checked list prices, 4 × USD 0.10 + USD 0.50 = USD 0.90 per month if left running (the metric-math alarm bills per underlying metric ⚠ VERIFY), and a few of those fall inside the free tier of 10 alarm metrics. Deploy, test and destroy within the hour.

# Delete the alarms (composite first: a composite blocks deleting its children)
aws cloudwatch delete-alarms --alarm-names hl-m11-page --profile learn-sandbox
aws cloudwatch delete-alarms --alarm-names hl-m11-failed-transfers hl-m11-traffic-floor hl-m11-p99-latency hl-m11-failure-rate --profile learn-sandbox
aws cloudwatch describe-alarms --alarm-name-prefix hl-m11 --query 'length(MetricAlarms) + length(CompositeAlarms)' --profile learn-sandbox

If you deployed with the stack, cdk destroy or terraform destroy removes everything, including the SNS topic.

Exercises

  • 🟢 Guided: In AlarmTests.cs, add a test where TreatMissingData.NotBreaching and the window is [9, 9, null, null, null] with "3 of 5 above 5"; predict the result before running. Then deploy the lab stack, confirm the email subscription, and rehearse with set-alarm-state.
  • 🟡 Challenge: Write a composite rule for "page when failures are high or the service is silent, unless the planned-maintenance alarm is active", test it with CompositeAlarmRule for all eight combinations of three alarm states, and deploy it in CDK or Terraform.
  • 🔴 Live problem: The page fired 11 times last Monday between 09:00 and 09:40, all failed-transfers, all for a partner that rate-limited you every few minutes. The on-call engineer was woken once, but the notifications overwhelmed the channel. Propose changes to the alarms (period, M of N, composite, suppression) and to the process, with the trade-offs for detection time.

Assignment

Design the alarm set for a service of your choice: a table with each alarm's metric, statistic, period, M of N, threshold, missing-data policy and reason, which are pages, tickets or dashboard-only, and one composite rule. Implement it in CDK or Terraform, rehearse it with set-alarm-state, and write the rehearsal results (who received what, how long it took) in a short ALARMS.md.

Solutions

Knowledge check

Interview questions

  • How do you reduce alarm noise? Alarm on symptoms, use M-out-of-N evaluation, pick the correct missing-data policy, use rates with a traffic guard, and combine alarms into composite alarms with actions only on the composite.
  • Static threshold versus anomaly detection? Static when there is a hard business limit; anomaly detection when the metric has daily or weekly shape and a fixed number would be wrong at some times. Anomaly detection needs history and costs three metrics.
  • How do you know an alarm will really page you? Rehearse with set-alarm-state and a game day; confirm the SNS subscription, topic policy and on-call tool routing.
  • What happens if the metric stops being published? It depends on the missing-data policy; for heartbeats use breaching, and add an external canary as an independent check.
  • Why are alarm-modification permissions sensitive? Disabling or deleting alarms silences detection; restrict and monitor those actions.

Certification practice

Completion checklist

  • I can explain M out of N and the four missing-data policies
  • I chose a missing-data policy for each lab alarm and wrote down why
  • I tested a composite rule offline with CompositeAlarmRule
  • I rehearsed the page with set-alarm-state (if I deployed)
  • I deleted every alarm, the SNS topic and the stack, and confirmed nothing with Project=cea-lab remains

Key takeaways

  • Alarm on customer-visible symptoms, sustained over several periods, and put the cost of noise in the design.
  • Missing data is a decision: silence is good news for failures and bad news for traffic.
  • Rates with a minimum-traffic guard are more robust than absolute counts; anomaly detection fits metrics with a shape.
  • Put actions on the composite so one incident means one page, and rehearse the page.
  • Treat alarm configuration and alarm-disable permissions as security-relevant.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page