CloudWatch alarms and composite alarms: thresholds, missing data and noise reduction
Design alarms that page a human only when customers are affected: choose statistics and periods, use M-out-of-N evaluation, pick the right missing-data policy, add metric math and anomaly detection where they fit, and combine alarms into one composite page.
Assumes: .NET 10 LTS · AWSSDK.CloudWatch 4.1.1.2 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Describe a CloudWatch metric alarm: its metric, statistic, period, evaluation periods, datapoints to alarm, threshold, comparison operator and treat-missing-data policy.
- Explain the three alarm states (
OK,ALARM,INSUFFICIENT_DATA) and exactly when an alarm sends a notification. - Choose the right missing-data policy for a failure-count alarm and for a heartbeat or traffic-floor alarm.
- Build a metric-math alarm for a failure rate with a minimum-traffic guard, and say when anomaly detection is the better choice.
- Combine alarms with a composite alarm rule so one incident produces one page, and rehearse the page with
set-alarm-state.
Prerequisites
- CloudWatch metrics and custom metrics: the
Harbourline/Paymentsmetrics the alarms read. - Observability fundamentals: the five-question test for an actionable alert.
- Tools: AWS CLI v2 (
learn-sandbox), .NET 10 SDK or Docker. CDK CLI or Terraform if you deploy the lab stack.
Why this matters in banking
Two failure patterns, both costly.
The pager that cried wolf. Operations set an alarm "CPU above 70% for one minute" on forty containers. During salary-day load it fires twenty times in an hour. The on-call engineer silences the topic. At 11:20 transfers really do start failing, and nobody is listening.
The silence that looked like health. A deployment breaks the metric publisher: no data is sent at all. The alarm on "failed transfers" stays green, because no failures were reported. Customers cannot pay for forty minutes. Silence was the symptom, and the alarm treated it as good news.
Good alarm design addresses both: alarm only on symptoms customers feel, require sustained breach (so a single spike does not page), decide deliberately what missing data means, and funnel related alarms through a composite so one incident means one page.
The idea in plain language
A CloudWatch alarm watches one metric (or a calculation over metrics) and compares each period's value with a threshold. It is always in one of three states:
| State | Meaning |
|---|---|
OK | The metric is within the threshold |
ALARM | The metric breached the threshold enough times (see M of N) |
INSUFFICIENT_DATA | Not enough data to decide, for example a new alarm or a metric that stopped reporting |
An alarm triggers its actions only when the state changes (the exception is Auto Scaling actions, which repeat each minute while the alarm stays in the state). That is the verified behaviour from the CloudWatch documentation: an alarm that stays in ALARM for an hour sends one notification, not sixty.
The knobs, in the order you should think about them:
| Setting | Question it answers | Banking example |
|---|---|---|
| Metric and statistic | What do I measure and how do I combine data points? | TransferCount filtered to Outcome=failed, Sum |
| Period | How long is one evaluation window? | 60 seconds |
| Threshold and comparison | What is "bad"? | Greater than 5 |
| Evaluation periods (N) | How many recent periods do I look at? | 5 |
| Datapoints to alarm (M) | How many of those N must be bad? | 3 (so "3 out of 5") |
| Treat missing data | What does "no data" mean? | notBreaching for failures; breaching for a traffic floor |
| Actions | What happens on a state change? | Publish to an SNS topic |
M out of N is the most useful noise filter. "Three of the last five minutes above 5" ignores a one-minute spike, still reacts within about three to five minutes, and tolerates a single good minute in the middle of a bad patch. Setting M equal to N ("5 of 5") is stricter and slower.
Analogy: A smoke alarm that sounded every time someone made toast would be unplugged within a week. M-out-of-N is the alarm waiting to see smoke for several seconds, and the composite alarm is the building's fire panel that sounds the bell once when the kitchen and the corridor both report smoke, instead of ringing every detector separately.
What problem the AWS service solves
CloudWatch alarms turn metrics into decisions: notify a person, run an automated action, or both.
Alarm types (from the CloudWatch alarms documentation, checked 2026-11-18):
| Type | Watches | Use when |
|---|---|---|
| Metric alarm | One metric, or the result of a metric-math expression | Almost always |
| Composite alarm | The states of other alarms through a rule such as ALARM(a) OR ALARM(b) | Reduce noise; express "page only if" logic |
| Log alarm | The result of a Logs Insights query run on a schedule | The signal exists only in logs; billed for scanned data on every run ⚠ VERIFY |
| PromQL alarm | A Prometheus-style query on metrics ingested through CloudWatch's OTLP endpoint | You already run OpenTelemetry metrics into CloudWatch ⚠ VERIFY availability |
This lesson uses metric and composite alarms, which every exam covers.
What alarms can do: send to SNS topics; perform EC2 actions (stop, terminate, reboot, recover) and Auto Scaling actions (metric alarms only); and create investigations, Systems Manager OpsItems or incidents. Composite alarms can send SNS notifications and create investigations, OpsItems or incidents, but cannot perform EC2 or Auto Scaling actions. The incident action depends on Systems Manager Incident Manager, which has been closed to new customers since 2025-11-07, so this module routes pages through alarm → SNS → on-call tool ⚠ VERIFY the currently recommended replacement.
Honest limitations:
- Evaluation lag. Alarms evaluate on metric periods; with a one-minute period and "3 of 5", detection takes minutes. If you need seconds, use high-resolution metrics and a shorter period, accepting higher cost and noise.
- Maximum evaluation range. Period × evaluation periods is limited: up to one day for periods under one hour, up to seven days for periods of an hour or longer.
- Alarm history is kept for 30 days. Copy state-change events elsewhere if you need longer evidence.
- Alarms are only as honest as their metric. A broken publisher makes data disappear; that is what the missing-data policy is for.
- CloudWatch does not check that your actions exist. A typo in an SNS ARN gives you an alarm that silently notifies nobody. Rehearse every alarm with
set-alarm-state. - Pricing (checked 2026-11-18, US East examples ⚠ VERIFY): USD 0.10 per standard alarm metric per month, USD 0.50 per composite alarm, and an anomaly-detection alarm bills its three underlying metrics (USD 0.30). Free tier: 10 alarm metrics.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Monitor metric alert rule | CloudWatch metric alarm | Both threshold-based; CloudWatch has explicit M-out-of-N and a missing-data policy |
| Action group (email, SMS, webhook, Logic App) | SNS topic with subscriptions, or EventBridge on alarm state change | Alarm → SNS is the standard; add subscribers (email, HTTPS, Lambda, chat bridges) separately |
| Dynamic thresholds (ML baseline) | Anomaly detection alarm | Needs about two weeks of history to learn; model band set with a width parameter |
| Alert processing rules (suppression) | Composite alarms, alarm actions suppression, DisableAlarmActions | Composite rules express logic; suppression needs a deliberate design |
| Smart groups / alert correlation | Composite alarm rules | You write the logic yourself |
| Log search alert rule | Log alarm (new) or metric filter + metric alarm | Metric filter plus alarm is the long-standing route |
| Alert severity 0 to 4 | No built-in severity | Use alarm naming, tags and which topic the alarm publishes to |
| Test action group | set-alarm-state | Sets any state until the next evaluation |
Architecture
Security-boundary view. The SNS topic policy must allow cloudwatch.amazonaws.com to publish, scoped with aws:SourceArn to your alarms, and should deny non-TLS publishes. Control who can change alarms (cloudwatch:PutMetricAlarm, DeleteAlarms, DisableAlarmActions): disabling actions silences the pager, so treat that permission as sensitive and review it in CloudTrail (see CloudTrail for banks).
Choosing the right missing-data policy
When a period has no data point, the alarm needs a rule. The four options (from the CloudWatch documentation on missing data; the lab's AlarmSimulator models them):
| Policy | Treats missing points as | Right for | Wrong for |
|---|---|---|---|
missing (default) | Not counted; if the whole window is missing, the alarm goes to INSUFFICIENT_DATA | Rarely: it hides outages | Anything you must page on |
notBreaching | Good (inside the threshold) | Error counts that are only published when errors happen | Traffic floors |
breaching | Bad (outside the threshold) | Heartbeats, traffic floors, "no transfers at all" | Error counts (every quiet night would page) |
ignore | Keep the current state | Metrics that are legitimately sparse and where flapping would hurt | Anything where silence matters |
The rule of thumb: for each alarm ask "if the metric vanished, is that good news or bad news?" Failures vanishing is good news (notBreaching). Traffic vanishing is bad news (breaching). Write the answer in the alarm description.
You can try all of these in code without AWS:
// Alarm on failed transfers: more than 5 in a minute, 3 out of 5 minutes.
var def = new AlarmDefinition(5, Comparison.GreaterThanThreshold, EvaluationPeriods: 5, DatapointsToAlarm: 3);
AlarmSimulator.Evaluate(def, [1, 9, 2, 8, 7]); // Alarm: three minutes above 5
AlarmSimulator.Evaluate(def, [0, 0, 40, 0, 0]); // Ok: one spike does not page
AlarmSimulator.Evaluate(def, [null, null, null, null, null]); // InsufficientData with the default policy
// A traffic floor: no data means no traffic, which is the problem.
var floor = new AlarmDefinition(1, Comparison.LessThanThreshold, 3, 3, TreatMissingData.Breaching);
AlarmSimulator.Evaluate(floor, [null, null, null]); // AlarmThe simulator is a teaching model: CloudWatch has additional edge cases (for example, for sparse metrics it may look back past the window), so treat the documentation as the source of truth.
Metric math and anomaly detection
A failure rate with a traffic guard
"More than 5 failures per minute" is a good first alarm, but it behaves differently at 02:00 (five failures may be the only traffic) and at 09:00 (five failures is noise in ten thousand transfers). A rate with a minimum-traffic guard is more robust:
IF(total >= 20, 100 * failed / total, 0)where failed is TransferCount with Outcome=failed and total is TransferCount without the outcome dimension (this is why the lab publishes two dimension sets: the all-outcomes series exists so this expression works). When fewer than 20 transfers flow in a minute the expression returns 0, so one failure at 03:00 cannot page. An alarm on this expression with threshold 5 (percent) and "3 of 5" reads: "more than 5% of transfers failed in at least three of the last five minutes, when real traffic exists".
Use the rate alarm together with the count alarm: counts catch outages at low traffic through the traffic-floor alarm, rates catch partial degradation at high traffic.
Anomaly detection
An anomaly detection alarm learns a model of the metric from its history (the documentation describes about two weeks of training ⚠ VERIFY) and alarms when the value leaves an expected band, set by ANOMALY_DETECTION_BAND(metric, k) where larger k widens the band. It suits metrics with daily or weekly shape, such as transfers per minute (a quiet Sunday morning versus a busy Friday evening) and is a good fit for the traffic signal. It suits badly: brand-new metrics, metrics that shift legitimately after a release, and anything with a hard business limit (use a static threshold). Anomaly alarms also bill three underlying metrics.
Composite alarms
A composite alarm has a rule expression over other alarms' states:
ALARM("hl-m11-failed-transfers") OR ALARM("hl-m11-traffic-floor")The syntax supports ALARM(...), OK(...), INSUFFICIENT_DATA(...), TRUE, FALSE, AND, OR, NOT and parentheses. Typical designs:
| Goal | Rule |
|---|---|
| One page for a customer-impacting incident | ALARM(failed) OR ALARM(floor) |
| Require two independent symptoms before paging | ALARM(failed) AND ALARM(latency) |
| Do not page while a planned deployment alarm is active | ALARM(failed) AND NOT ALARM(deploy-in-progress) |
| Page only for the whole service, not one task | ALARM(az-a-errors) AND ALARM(az-b-errors) |
Put actions on the composite only, not on its children. The child alarms still change state, which you see on the dashboard and in history, but only the composite notifies. CloudWatch also lets you suppress a composite's actions while another alarm is in ALARM (an actions suppressor) ⚠ VERIFY the exact property names for your IaC tool.
The lab's CompositeAlarmRule parses and evaluates this syntax offline so you can test a rule before deploying it:
var rule = CompositeAlarmRule.Parse(
"ALARM(\"hl-failed-transfers\") AND ALARM(\"hl-p99-latency\") AND NOT ALARM(\"hl-deploy-in-progress\")");
rule.Evaluate(new Dictionary<string, AlarmState>
{
["hl-failed-transfers"] = AlarmState.Alarm,
["hl-p99-latency"] = AlarmState.Alarm,
["hl-deploy-in-progress"] = AlarmState.Ok,
}); // true: pageLimits to remember: cross-account composite alarms are not supported, and a composite is billed at its own price (USD 0.50 per month at the checked list price).
Step-by-step implementation
List symptoms, not causes
For the Transfers API: failed transfers, traffic stopped, high failure rate, slow tail. Everything else (CPU, memory, connection pool) goes on the dashboard.
Fix each alarm's statistic, period and M of N
Sum per 60 seconds for counts; p99 per 60 seconds for latency; M of N of 3 of 5 for noisy signals, 10 of 10 for a traffic floor (ten quiet minutes in a row).
Decide the missing-data policy and write it down
Put the reason in the alarm description, for example "silence is good news".
Add the guard for ratios
Use metric math with a minimum-traffic condition, so low-volume periods cannot page.
Create the page topic and the composite
One SNS topic for pages, one composite alarm with the only action. Include the runbook path in AlarmDescription; the notification carries it.
Subscribe a human and confirm
For an email subscription, click the confirmation link. Then rehearse the page.
Rehearse and review
aws cloudwatch set-alarm-state on a child alarm and check the page arrives. In a game day, inject a real fault (the lab's X-Fault: partner-down). Review every alarm after real incidents: did it fire, was it actionable, was it fast enough?
AWS Console instructions
Create an alarm: Console → CloudWatch → Alarms → All alarms → Create alarm → Select metric. Browse to Harbourline/Payments → the Service, Environment, Outcome dimensions → TransferCount for failed; choose the statistic Sum and period 1 minute. Conditions: Static, Greater than, 5. Additional configuration: Datapoints to alarm 3 out of 5, Missing data treatment = Treat missing data as good (not breaching). Next: choose an SNS topic notification for In alarm (and OK), name and describe the alarm, Create alarm.
Composite alarm: Alarms → Create alarm → Create composite alarm, choose the child alarms and the condition (the console builds the rule, or you can type the expression), then set the actions on the composite.
Rehearse: open an alarm, there is no console button for SetAlarmState; use the CLI below. Console labels change over time ⚠ VERIFY.
AWS CLI
# A metric alarm: failed transfers, 3 of the last 5 minutes above 5. Silence is good news.
aws cloudwatch put-metric-alarm --profile learn-sandbox \
--alarm-name hl-m11-failed-transfers \
--alarm-description "More than 5 failed transfers per minute in 3 of 5 minutes. Silence is good news. Runbook: docs/runbooks/failed-transfers.md" \
--namespace Harbourline/Payments --metric-name TransferCount \
--dimensions Name=Service,Value=transfers-api Name=Environment,Value=learn Name=Outcome,Value=failed \
--statistic Sum --period 60 --evaluation-periods 5 --datapoints-to-alarm 3 \
--threshold 5 --comparison-operator GreaterThanThreshold --treat-missing-data notBreaching
# A composite alarm. It is the ONLY alarm with an action.
aws cloudwatch put-composite-alarm --profile learn-sandbox \
--alarm-name hl-m11-page \
--alarm-rule 'ALARM("hl-m11-failed-transfers") OR ALARM("hl-m11-traffic-floor")' \
--alarm-actions arn:aws:sns:us-east-1:111122223333:hl-m11-oncall-page
# Rehearse the page: force a child alarm into ALARM (it resets at the next evaluation)
aws cloudwatch set-alarm-state --alarm-name hl-m11-failed-transfers --state-value ALARM \
--state-reason "Game day rehearsal" --profile learn-sandbox
# See what happened and why
aws cloudwatch describe-alarm-history --alarm-name hl-m11-page --history-item-type StateUpdate --max-items 5 --profile learn-sandbox
# List every alarm in ALARM state
aws cloudwatch describe-alarms --state-value ALARM --query 'MetricAlarms[].AlarmName' --profile learn-sandbox
# Silence actions during planned maintenance (and remember to re-enable!)
aws cloudwatch disable-alarm-actions --alarm-names hl-m11-page --profile learn-sandbox
aws cloudwatch enable-alarm-actions --alarm-names hl-m11-page --profile learn-sandboxInfrastructure as Code
The lab's stack (labs/m11-observe-banking-api/infra) contains four metric alarms, the composite and the SNS topic, in both tools. Key excerpts:
var failed = new Alarm(this, "FailedTransfersAlarm", new AlarmProps
{
AlarmName = "hl-m11-failed-transfers",
Metric = Count("failed"), // TransferCount, Outcome=failed, Sum, 1 minute
Threshold = 5,
EvaluationPeriods = 5,
DatapointsToAlarm = 3, // 3 of 5
ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING,
});
var failureRate = new MathExpression(new MathExpressionProps
{
Expression = "IF(total >= 20, 100 * failed / total, 0)",
UsingMetrics = new Dictionary<string, IMetric> { ["failed"] = Count("failed"), ["total"] = totalCount },
Label = "Failure rate %",
Period = Duration.Minutes(1),
});
var page = new CompositeAlarm(this, "PageAlarm", new CompositeAlarmProps
{
CompositeAlarmName = "hl-m11-page",
AlarmRule = AlarmRule.AnyOf(failed, floor), // the only alarm with an action
});
page.AddAlarmAction(new SnsAction(topic));
page.AddOkAction(new SnsAction(topic));Both stacks were compiled (CDK) and validated (Terraform) on 2026-11-18; they have not been deployed by the test run.
Complete code
Creating and rehearsing alarms from .NET with the AWS SDK v4 (compiles against AWSSDK.CloudWatch 4.1.1.2), useful for tooling such as a "game day" console app:
using Amazon.CloudWatch;
using Amazon.CloudWatch.Model;
using var cw = new AmazonCloudWatchClient();
await cw.PutMetricAlarmAsync(new PutMetricAlarmRequest
{
AlarmName = "hl-m11-failed-transfers",
AlarmDescription = "More than 5 failed transfers per minute in 3 of the last 5 minutes",
Namespace = "Harbourline/Payments",
MetricName = "TransferCount",
Dimensions =
[
new Dimension { Name = "Service", Value = "transfers-api" },
new Dimension { Name = "Environment", Value = "learn" },
new Dimension { Name = "Outcome", Value = "failed" },
],
Statistic = Statistic.Sum,
Period = 60,
EvaluationPeriods = 5,
DatapointsToAlarm = 3,
Threshold = 5,
ComparisonOperator = ComparisonOperator.GreaterThanThreshold,
TreatMissingData = "notBreaching",
});
await cw.PutCompositeAlarmAsync(new PutCompositeAlarmRequest
{
AlarmName = "hl-m11-page",
AlarmRule = "ALARM(\"hl-m11-failed-transfers\")",
AlarmActions = ["arn:aws:sns:us-east-1:111122223333:hl-m11-oncall-page"],
});
// Rehearse the page without waiting for a real failure.
await cw.SetAlarmStateAsync(new SetAlarmStateRequest
{
AlarmName = "hl-m11-failed-transfers",
StateValue = StateValue.ALARM,
StateReason = "Game day rehearsal",
});
var described = await cw.DescribeAlarmsAsync(new DescribeAlarmsRequest
{
AlarmNames = ["hl-m11-failed-transfers"],
AlarmTypes = [AlarmType.MetricAlarm],
});
Console.WriteLine(described.MetricAlarms[0].StateValue);The offline simulators are labs/m11-observe-banking-api/src/Harbourline.Observability/Alarms/AlarmSimulator.cs and CompositeAlarmRule.cs, with tests in AlarmTests.cs.
Code explained line by line
DatapointsToAlarm = 3withEvaluationPeriods = 5is the "3 out of 5" rule. LeavingDatapointsToAlarmunset means every evaluation period must breach.TreatMissingDatatakes a string in the SDK ("breaching","notBreaching","ignore","missing"), unlike the typed constants for statistics and states.Statistic.Sum,ComparisonOperator.GreaterThanThresholdandStateValue.ALARMare SDK v4 constant classes. Passing the wrong type is a compile error, which is the point of using them.PutCompositeAlarmAsyncreferences child alarms by name inside the rule string; child alarms must exist when the composite is created, and the rule's quotes are escaped as\".AlarmActionsis an array of ARNs. Putting the action on the composite and not on the child alarms is what gives you one page per incident.SetAlarmStateAsyncforces a state until the next evaluation. It is a rehearsal tool: it also proves the SNS topic, subscription and policy are wired correctly.DescribeAlarmsAsyncfilters byAlarmTypesso you do not accidentally get composites when you want metric alarms.
Security considerations
⚠ Learning shortcut — not for production
The lab subscribes a plain email address and gives your sandbox identity broad CloudWatch rights. In production, page through an on-call tool subscribed to the topic over HTTPS, encrypt the topic with a customer managed KMS key whose policy allows CloudWatch, restrict cloudwatch:DisableAlarmActions, DeleteAlarms and PutMetricAlarm to the owning team, and monitor changes to alarms in CloudTrail.
- Protect the pager. An attacker (or a tired engineer) who disables alarm actions silences the bank's detection. Alert on
DisableAlarmActionsandDeleteAlarmsevents. - Notification content. Alarm descriptions and state reasons travel through SNS to email and chat; they must contain no personal data or secrets.
- Topic policy. Allow only
cloudwatch.amazonaws.comwithaws:SourceArnmatching your alarms, and denyaws:SecureTransport = false. The lab's Terraform shows both statements. - Alarms as controls. A detection alarm may be part of your operational-resilience evidence. Version it in Git and review changes like code. Whether it meets a regulator's expectation is for your compliance team.
Banking use case
Harbourline's payments platform has four alarms for the Transfers API and one page. Failed transfers and traffic floor feed the composite hl-prod-page, which is the only alarm wired to the on-call tool. Failure rate creates a ticket and is shown on the dashboard, and p99 latency is a dashboard tile that engineers consult during an incident. A planned deployment raises hl-deploy-in-progress for the release window; the composite rule includes AND NOT ALARM(hl-deploy-in-progress) for the failure-rate alarm only, so a bad deploy is still paged after the window. Each quarter the team lists every page and records whether it needed human action; pages with zero actions are deleted or demoted. All of this uses fictional data.
Common mistakes
- Wrong missing-data policy. The default hides outages;
breachingon an error counter pages every quiet night. Fix: ask "is silence good or bad?" per alarm. - Alarming on single data points. One bad minute pages. Fix: M of N.
- Alarming on averages of latency. Fix: p95 or p99.
- Actions on every child alarm. One incident produces many pages. Fix: actions on the composite only.
- Never testing the page. The topic ARN has a typo, or the subscription is unconfirmed. Fix:
set-alarm-staterehearsal and a periodic game day. - Absolute thresholds that ignore traffic. Fix: rate alarms with a minimum-traffic guard.
- Alarm sprawl. Hundreds of alarms nobody owns. Fix: tags with owner and runbook, and a review.
- Forgetting to re-enable actions after maintenance. Fix: put re-enabling in the maintenance checklist, and alarm on
DisableAlarmActionsevents lasting too long.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Alarm stays in INSUFFICIENT_DATA | No data yet, dimensions differ from the published metric, or period shorter than the publish interval | list-metrics and copy exact dimension values; check the period |
Alarm in ALARM but no notification | Unconfirmed SNS subscription, wrong topic ARN, or topic policy blocks CloudWatch | Confirm the subscription; check describe-alarm-history for action failures; fix the topic policy |
| Notification arrives once and never again | Actions fire on state changes only | Expected; if you need reminders, use an on-call tool that re-notifies |
| Alarm flaps between OK and ALARM | Threshold close to normal values, M equals N, or ignore policy | Use M of N, widen the period or add hysteresis through a composite |
| Alarm never fires during a real outage | Missing-data policy notBreaching on a metric that stops when the service dies | Add a traffic-floor alarm with breaching |
| Composite creation fails | A child alarm does not exist yet, or quoting error in the rule | Create children first; escape quotes |
set-alarm-state is overwritten immediately | Expected: the next evaluation resets the state | Rehearse by watching the notification, not the state |
Cost and cleanup
The lab's four metric alarms and one composite cost, at the checked list prices, 4 × USD 0.10 + USD 0.50 = USD 0.90 per month if left running (the metric-math alarm bills per underlying metric ⚠ VERIFY), and a few of those fall inside the free tier of 10 alarm metrics. Deploy, test and destroy within the hour.
# Delete the alarms (composite first: a composite blocks deleting its children)
aws cloudwatch delete-alarms --alarm-names hl-m11-page --profile learn-sandbox
aws cloudwatch delete-alarms --alarm-names hl-m11-failed-transfers hl-m11-traffic-floor hl-m11-p99-latency hl-m11-failure-rate --profile learn-sandbox
aws cloudwatch describe-alarms --alarm-name-prefix hl-m11 --query 'length(MetricAlarms) + length(CompositeAlarms)' --profile learn-sandboxIf you deployed with the stack, cdk destroy or terraform destroy removes everything, including the SNS topic.
Exercises
- 🟢 Guided: In
AlarmTests.cs, add a test whereTreatMissingData.NotBreachingand the window is[9, 9, null, null, null]with "3 of 5 above 5"; predict the result before running. Then deploy the lab stack, confirm the email subscription, and rehearse withset-alarm-state. - 🟡 Challenge: Write a composite rule for "page when failures are high or the service is silent, unless the planned-maintenance alarm is active", test it with
CompositeAlarmRulefor all eight combinations of three alarm states, and deploy it in CDK or Terraform. - 🔴 Live problem: The page fired 11 times last Monday between 09:00 and 09:40, all
failed-transfers, all for a partner that rate-limited you every few minutes. The on-call engineer was woken once, but the notifications overwhelmed the channel. Propose changes to the alarms (period, M of N, composite, suppression) and to the process, with the trade-offs for detection time.
Assignment
Design the alarm set for a service of your choice: a table with each alarm's metric, statistic, period, M of N, threshold, missing-data policy and reason, which are pages, tickets or dashboard-only, and one composite rule. Implement it in CDK or Terraform, rehearse it with set-alarm-state, and write the rehearsal results (who received what, how long it took) in a short ALARMS.md.
Solutions
With [9, 9, null, null, null], two points breach (9 is above 5) and three are missing but treated as not breaching, so breaches (2) are below DatapointsToAlarm (3): the result is Ok. With the breaching policy the three missing points would count as breaches and the result would be Alarm. That is the whole difference between the two policies in one example. The rehearsal should deliver an email when the composite moves to ALARM; if it does not, check the subscription confirmation and the topic policy.
Rule: (ALARM(failed) OR ALARM(floor)) AND NOT ALARM(maintenance). Truth table over the eight combinations of (failed, floor, maintenance) each in Alarm or Ok: the composite is true only for (Alarm, any, Ok) and (Ok, Alarm, Ok), that is three of the eight combinations; whenever maintenance is Alarm the composite is false. In CDK use AlarmRule.AllOf(AlarmRule.AnyOf(failed, floor), AlarmRule.Not(AlarmRule.FromAlarm(maintenance, AlarmState.ALARM))) ⚠ VERIFY the helper names in your CDK version; in Terraform write the string. Note the risk: a maintenance alarm left active hides real outages, so alarm on its duration.
Rate-limiting causes repeated short bursts. Options: raise M of N to "4 of 5" and the period to 5 minutes (slower detection, fewer pages); add a rate alarm with the traffic guard so only large relative failures page; route partner-specific failures (a separate Outcome or a Dependency dimension with a small fixed vocabulary) to a ticket-level alarm and keep the composite for broad impact; ensure state-change notifications are grouped by the on-call tool. Trade-off: every change delays detection of a real partner outage; mitigate with a canary that tests the partner path end to end. Process: record the partner's published limits, negotiate a higher limit or add client-side back-off (see Retry strategies), and review after a month.
Knowledge check
Answer: one. Alarm actions fire when the state changes. (Auto Scaling actions are the exception and repeat each minute.)
Answer: the alarm goes to ALARM when at least 3 of the last 5 evaluation periods breach the threshold (DatapointsToAlarm = 3, EvaluationPeriods = 5).
Answer: breaching. No data means no traffic, which is the condition you want to detect. notBreaching would hide it.
Answer: no. Composite alarms can notify through SNS and create investigations, OpsItems or incidents, but cannot perform EC2 or Auto Scaling actions.
Answer: at very low traffic one failure is a huge percentage. The guard returns 0 (not breaching) below a volume threshold so a single failure at night cannot page.
Interview questions
- How do you reduce alarm noise? Alarm on symptoms, use M-out-of-N evaluation, pick the correct missing-data policy, use rates with a traffic guard, and combine alarms into composite alarms with actions only on the composite.
- Static threshold versus anomaly detection? Static when there is a hard business limit; anomaly detection when the metric has daily or weekly shape and a fixed number would be wrong at some times. Anomaly detection needs history and costs three metrics.
- How do you know an alarm will really page you? Rehearse with
set-alarm-stateand a game day; confirm the SNS subscription, topic policy and on-call tool routing. - What happens if the metric stops being published? It depends on the missing-data policy; for heartbeats use
breaching, and add an external canary as an independent check. - Why are alarm-modification permissions sensitive? Disabling or deleting alarms silences detection; restrict and monitor those actions.
Certification practice
An alarm on HealthyHostCount stays green when an application stops publishing metrics entirely. What setting should be changed so silence triggers the alarm?
A. Set treat missing data to breaching.
B. Increase the period to one hour.
C. Use the Maximum statistic.
D. Disable the alarm's actions.
Answer: A. With breaching, missing data points count as breaching, so a silent metric moves the alarm to ALARM. B, C and D do not change how missing data is treated.
Twelve related alarms fire together during one incident and flood the on-call channel. Which design gives one notification per incident?
A. Attach the SNS action to each alarm. B. Create a composite alarm over the related alarms and attach the action only to the composite. C. Lower every threshold. D. Increase the SNS topic's retry policy.
Answer: B. Composite alarms evaluate a rule over other alarms' states, and notify once. A multiplies notifications; C makes more alarms fire; D is delivery tuning.
A developer creates an alarm for a custom metric with dimensions Service and Outcome, but the alarm stays in INSUFFICIENT_DATA although data flows. What is the most likely cause?
A. CloudWatch does not support custom metrics in alarms. B. The alarm's dimensions or namespace do not exactly match the published metric. C. The SNS topic is in another Region. D. The period is too long.
Answer: B. Alarms must include the same namespace, metric name and all dimensions as the published metric. A is false; C would affect notifications, not state; D is possible but less likely than a dimension mismatch.
Completion checklist
- I can explain M out of N and the four missing-data policies
- I chose a missing-data policy for each lab alarm and wrote down why
- I tested a composite rule offline with
CompositeAlarmRule - I rehearsed the page with
set-alarm-state(if I deployed) - I deleted every alarm, the SNS topic and the stack, and confirmed nothing with
Project=cea-labremains
Key takeaways
- Alarm on customer-visible symptoms, sustained over several periods, and put the cost of noise in the design.
- Missing data is a decision: silence is good news for failures and bad news for traffic.
- Rates with a minimum-traffic guard are more robust than absolute counts; anomaly detection fits metrics with a shape.
- Put actions on the composite so one incident means one page, and rehearse the page.
- Treat alarm configuration and alarm-disable permissions as security-relevant.
Further reading
- Using Amazon CloudWatch alarms: alarm types, actions and limits as used here.
- Amazon CloudWatch pricing: alarm and composite alarm prices.
- Dashboards and Synthetics canaries: the next lesson.
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
CloudWatch metrics and custom business metrics with the Embedded Metric Format
Understand namespaces, dimensions, units, statistics and resolution, publish a business metric (failed transfers per minute) from C# as an Embedded Metric Format log line, and price the result before you deploy it.
Dashboards and Synthetics canaries: one screen for on-call and a customer's-eye availability check
Design an operational CloudWatch dashboard for a banking API as code, add a log-query widget, and write a Synthetics canary that probes the API from outside so you find out about a failure before your customers do.