Cost monitoring for observability: CloudWatch bills, budgets and anomaly alerts
Understand what makes CloudWatch and related observability services expensive, find your largest log groups and metric sources, cut waste without losing signal, and set up a budget and anomaly detection so a logging mistake is caught in a day rather than at month end.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- List the main CloudWatch cost drivers (log ingestion and storage, custom metrics, alarms, dashboards, Insights queries, Application Signals, Synthetics) and estimate a monthly bill from a volume.
- Rank log groups by ingested bytes and find groups with no retention, using a small C# tool.
- Choose the cheapest log class, retention and metric design that still meets the need.
- Create an AWS Budget scoped to CloudWatch and a Cost Anomaly Detection monitor with alerts.
- Explain why high-cardinality metric dimensions and debug logging in production are the classic cost traps.
Prerequisites
- Log aggregation and error monitoring for log classes (Standard and Essentials) and what each supports.
- The earlier Module 11 lessons: CloudWatch Logs for .NET, CloudWatch metrics and custom metrics and Alarms and composite alarms.
- Module 15 (FinOps) covers budgets and cost allocation across the whole platform. This lesson applies them to observability.
- Tools: AWS CLI v2 with the
learn-sandboxSSO profile, .NET 10 SDK. A Cost Explorer-enabled account helps but is not required.
Why this matters in banking
Observability is one of the few cloud costs that grows by itself. Every new microservice adds log groups, every outage adds temporary debug logging that nobody removes, every dashboard adds widgets. In regulated firms, logs also get kept "just in case" for years. Left alone, the monitoring bill can approach the size of the compute bill it is watching.
A typical story: during an incident an engineer sets the payments service to Debug level to see more. The incident ends, the setting stays, and ingestion rises from 20 GB to 300 GB a day. Nobody notices until the monthly invoice, by which time the extra cost is five figures. There is a second cost too: debug logs often contain request bodies, which means personal data is now sitting in a log group with unlimited retention. Good cost monitoring is also good data hygiene.
The idea in plain language
CloudWatch is metered like a utility: you pay for what flows in, what you keep and what you ask of it. Each dimension has its own meter:
- Flow in: every gigabyte of logs ingested, every custom metric data stream, every canary run, every signal.
- Keep: every gigabyte-month of log storage; every custom metric that exists.
- Ask: every gigabyte scanned by Logs Insights, every
GetMetricDatarequest from third-party tools, every dashboard widget refresh by an API client.
The surprising fact for most developers: ingestion is usually the biggest meter, and cardinality (how many distinct metric series you create) is the sneakiest.
Analogy: a water bill with a charge for every litre that enters the building (ingestion), a storage-tank rent (retention), and a fee per tap that exists (metrics, alarms). Fixing a leak helps most; leaving every tap running for no reason costs the rest.
What problem the AWS service solves
AWS gives you the meters, not the discipline. The tools:
| Tool | What it does | Notes |
|---|---|---|
| Cost Explorer | Breaks costs down by service, usage type, tag | Grouping by usage type is the fastest way to find which CloudWatch meter is growing |
| AWS Budgets | Alerts when cost or forecast exceeds a threshold, filterable by service and tag | Alerts are not hard limits: they notify, they do not stop spending |
| Cost Anomaly Detection | Machine-learning monitors that alert on unusual spend | Needs a few days of history; alerts by email or SNS ⚠ VERIFY current behaviour |
| Cost allocation tags | Attribute costs to a team or service | Tags on log groups let you split the logging bill; tags must be activated for cost reports ⚠ VERIFY |
AWS/Logs metrics | IncomingBytes per log group | Free, near-real-time, the best signal for "who is logging a lot" |
| Billing metrics | EstimatedCharges in AWS/Billing (us-east-1 only, needs billing alerts enabled) | A coarse early warning ⚠ VERIFY |
Price points used in this lesson
Prices change and differ by Region. These come from the CloudWatch pricing page worked examples for US East at the time of checking, and are for estimation only. ⚠ VERIFY every number on the pricing page before using it in a budget.
| Meter | Price used | Notes |
|---|---|---|
| Log ingestion, Standard class | $0.50 per GB (first tier) | Lower tiers exist at higher volume |
| Log ingestion, Essentials (formerly Infrequent Access) | $0.25 per GB | Fewer features: no metric filters, subscription filters, EMF, Live Tail |
| Log storage | $0.03 per GB-month | Charged per month stored; retention drives this |
| Logs Insights | per GB of data scanned ⚠ VERIFY the current rate | Not shown in the portion of the page I could read |
| Custom metrics | $0.30 per metric per month (first 10,000) | A metric is one unique name plus dimension combination |
| Standard alarm | $0.10 per alarm metric per month | High-resolution alarm $0.30 |
| Composite alarm | $0.50 per alarm per month | |
| Dashboards | per dashboard per month beyond the free allowance ⚠ VERIFY price | Free tier: 3 dashboards, up to 50 metrics each |
| Application Signals | $1.50 per million signals (first 100 million), less beyond | Billed per signal; spans and X-Ray traces billed separately |
| Synthetics | $0.0012 per canary run | Free tier: 100 runs per month |
| Contributor Insights | $0.50 per rule after the first | Plus matched events |
| Metric streams | $0.003 per 1,000 metric updates |
Free tier highlights (monthly): 5 GB of logs, 10 custom metrics, 10 alarm metrics, 3 dashboards and 1 million API requests. ⚠ VERIFY
The worked estimate
Harbourline's payments platform ingests 50 GB of logs per day at Standard class with 30-day retention, runs 2,000 custom metrics, 60 alarms, 8 dashboards and one canary every five minutes.
| Item | Calculation | Monthly |
|---|---|---|
| Log ingestion | 50 GB × 30 days × $0.50 | $750 |
| Log storage (steady state, 30 days held) | 1,500 GB × $0.03 | about $45 (the stored size is usually smaller than raw ingestion because it is compressed; treat this as an upper bound ⚠ VERIFY) |
| Custom metrics | 2,000 × $0.30 | $600 |
| Alarms | 60 × $0.10 | $6 |
| Dashboards | (8 − 3 free) × dashboard price ($3 assumed ⚠ VERIFY) | about $15 |
| Canary | 12 runs/hour × 24 × 30 = 8,640 runs, minus 100 free, × $0.0012 | about $10 |
| Total (before Insights queries and Application Signals) | about $1,430 |
Two lines dominate: ingestion and custom metrics. That tells you where to look first, and it is usual: a handful of chatty log groups and a handful of high-cardinality metrics cause most of the bill.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Log Analytics ingestion pricing (per GB), commitment tiers | CloudWatch Logs ingestion per GB, volume tiers | AWS also has a lower-priced Essentials log class; storage is charged separately |
| Basic Logs vs Analytics Logs tables | Essentials vs Standard log classes | Class chosen at creation, cannot be changed |
| Daily cap on a workspace | None built in; use a budget alert and IncomingBytes alarm | AWS does not stop ingestion when a budget is exceeded |
| Application Insights sampling and daily cap | OpenTelemetry sampling, X-Ray sampling rules, Application Signals | Sampling is configured in the SDK or collector |
| Azure Cost Management budgets and anomaly alerts | AWS Budgets and Cost Anomaly Detection | Similar: notifications, not enforcement |
| Metric pricing for custom metrics in Azure Monitor | Per custom metric per month | Every distinct dimension combination is a separate billed metric |
| Table-level retention in Log Analytics | Per log group retention in days | Default for new log groups is to never expire |
Architecture
Boundary view: cost data is visible to billing roles, not to every engineer. Tags on log groups (Owner, Service, Environment) let you attribute the bill to a team without exposing log contents.
Step-by-step implementation
Find where the money goes
In Cost Explorer, filter by service CloudWatch and group by usage type. The names tell you the meter: ingestion, storage, metrics, alarms, API requests. Note the biggest two.
Rank log groups by ingestion
Use IncomingBytes per log group over the last seven days, multiplied by the ingestion price, to rank groups. The C# tool below does this.
Fix retention
Find log groups with no retention and set one that matches the business need (for example 14 days for debug-heavy application logs, longer only where a documented requirement exists). Archive long-term evidence to S3 with a lifecycle policy rather than keeping it in CloudWatch.
Reduce volume at the source
Raise the minimum log level, stop logging bodies and health checks, sample high-volume info logs and move rarely read logs to the Essentials class (new log groups only).
Tame metric cardinality
Remove unbounded dimensions (customer id, request id, URL path). Keep dimensions to bounded sets such as service, environment and route template.
Add guardrails
An alarm on IncomingBytes for the noisiest groups, a CloudWatch-scoped budget, and Cost Anomaly Detection by service. Route all three to a topic someone reads.
AWS Console instructions
- Cost Explorer: Console → Billing and Cost Management → Cost Explorer → Filters: Service = CloudWatch → Group by: Usage type. Switch to daily granularity to see spikes.
- Budgets: Billing and Cost Management → Budgets → Create budget → Cost budget, amount
20(lab) or your real figure; under Filters choose Service = CloudWatch; add alerts at 80% (actual) and 100% (forecast) with an email recipient. - Cost Anomaly Detection: Billing and Cost Management → Cost Anomaly Detection → Create monitor → AWS services; add an alert subscription with your email or an SNS topic and a daily summary. ⚠ VERIFY console wording.
- Log groups: CloudWatch → Logs → Log groups. Add the columns Retention and Stored bytes; sort by stored bytes. The console shows log class and data protection status too.
- Retention: select a group → Actions → Edit retention setting.
- Cost allocation tags: Billing and Cost Management → Cost allocation tags → activate
Owner,Service,Environment. Activation applies from the date you activate. ⚠ VERIFY
AWS CLI
# 1) CloudWatch cost by usage type for the last 14 days (Cost Explorer API: small charge per request ⚠ VERIFY).
aws ce get-cost-and-usage \
--time-period Start=$(date -u -d '-14 days' +%F),End=$(date -u +%F) \
--granularity DAILY --metrics UnblendedCost \
--filter '{"Dimensions":{"Key":"SERVICE","Values":["AmazonCloudWatch"]}}' \
--group-by Type=DIMENSION,Key=USAGE_TYPE \
--profile learn-sandbox --region us-east-1
# 2) Log groups with NO retention (they never expire) and their stored size.
aws logs describe-log-groups \
--query 'logGroups[?retentionInDays==`null`].[logGroupName,storedBytes,logGroupClass]' \
--output table --profile learn-sandbox --region us-east-1
# 3) Set 14-day retention on one log group.
aws logs put-retention-policy --log-group-name /hl/payments/api --retention-in-days 14 \
--profile learn-sandbox --region us-east-1
# 4) Bytes ingested by one log group over the last 7 days, daily sums.
aws cloudwatch get-metric-statistics --namespace AWS/Logs --metric-name IncomingBytes \
--dimensions Name=LogGroupName,Value=/hl/payments/api \
--start-time $(date -u -d '-7 days' +%Y-%m-%dT%H:%M:%SZ) --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
--period 86400 --statistics Sum --profile learn-sandbox --region us-east-1
# 5) Create a budget scoped to CloudWatch (JSON files below).
aws budgets create-budget --account-id 123456789012 \
--budget file://budget.json --notifications-with-subscribers file://notifications.json \
--profile learn-sandbox --region us-east-1
# 6) Create an anomaly monitor by service and subscribe an email daily.
aws ce create-anomaly-monitor --anomaly-monitor 'MonitorName=hl-m11-services,MonitorType=DIMENSIONAL,MonitorDimension=SERVICE' \
--profile learn-sandbox --region us-east-1
# 7) Count custom metrics per namespace (cost driver: each series is billed).
aws cloudwatch list-metrics --namespace Harbourline/Payments --query 'length(Metrics)' \
--profile learn-sandbox --region us-east-1budget.json and notifications.json for step 5:
{
"BudgetName": "hl-m11-cloudwatch-monthly",
"BudgetType": "COST",
"TimeUnit": "MONTHLY",
"BudgetLimit": { "Amount": "20", "Unit": "USD" },
"CostFilters": { "Service": ["Amazon CloudWatch"] }
}[
{
"Notification": {
"NotificationType": "ACTUAL",
"ComparisonOperator": "GREATER_THAN",
"Threshold": 80,
"ThresholdType": "PERCENTAGE"
},
"Subscribers": [{ "SubscriptionType": "EMAIL", "Address": "finops@example.com" }]
},
{
"Notification": {
"NotificationType": "FORECASTED",
"ComparisonOperator": "GREATER_THAN",
"Threshold": 100,
"ThresholdType": "PERCENTAGE"
},
"Subscribers": [{ "SubscriptionType": "EMAIL", "Address": "finops@example.com" }]
}
]The usage-type names Cost Explorer returns for CloudWatch (for example ones containing DataProcessing-Bytes for log ingestion, TimedStorage-ByteHrs for log storage, MetricMonitorUsage for custom metrics and AlarmMonitorUsage for alarms, usually prefixed by a Region code) are from memory; read the actual strings from your own output before filtering on them. ⚠ VERIFY The service name in the budget filter (Amazon CloudWatch) differs from the Cost Explorer dimension value (AmazonCloudWatch), and the exact strings should be checked in each console. ⚠ VERIFY
Infrastructure as Code
The stack creates a CloudWatch-scoped monthly budget, an anomaly monitor by service with a subscription, a log group with sensible retention, and an alarm on its ingestion volume. L1 constructs are used for Budgets and Cost Explorer; property names follow the CloudFormation references, compile before relying on them. ⚠ VERIFY
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.Budgets;
using Amazon.CDK.AWS.CE;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.CloudWatch.Actions;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.SNS;
using Constructs;
namespace Harbourline.Monitoring.Infra;
public sealed class CostGuardrailsStack : Stack
{
public CostGuardrailsStack(Construct scope, string id, string finopsEmail, IStackProps? props = null)
: base(scope, id, props)
{
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m11-l14");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
var topic = new Topic(this, "FinOpsTopic", new TopicProps { TopicName = "hl-m11-finops-alerts" });
// --- Budget: CloudWatch only, 20 USD per month, alert at 80% actual and 100% forecast ---
new CfnBudget(this, "CloudWatchBudget", new CfnBudgetProps
{
Budget = new CfnBudget.BudgetDataProperty
{
BudgetName = "hl-m11-cloudwatch-monthly",
BudgetType = "COST",
TimeUnit = "MONTHLY",
BudgetLimit = new CfnBudget.SpendProperty { Amount = 20, Unit = "USD" },
CostFilters = new Dictionary<string, object>
{
["Service"] = new[] { "Amazon CloudWatch" },
},
},
NotificationsWithSubscribers = new[]
{
new CfnBudget.NotificationWithSubscribersProperty
{
Notification = new CfnBudget.NotificationProperty
{
NotificationType = "ACTUAL",
ComparisonOperator = "GREATER_THAN",
Threshold = 80,
ThresholdType = "PERCENTAGE",
},
Subscribers = new[]
{
new CfnBudget.SubscriberProperty { SubscriptionType = "EMAIL", Address = finopsEmail },
},
},
new CfnBudget.NotificationWithSubscribersProperty
{
Notification = new CfnBudget.NotificationProperty
{
NotificationType = "FORECASTED",
ComparisonOperator = "GREATER_THAN",
Threshold = 100,
ThresholdType = "PERCENTAGE",
},
Subscribers = new[]
{
new CfnBudget.SubscriberProperty { SubscriptionType = "EMAIL", Address = finopsEmail },
},
},
},
});
// --- Cost Anomaly Detection: one monitor across AWS services, daily email ---
var monitor = new CfnAnomalyMonitor(this, "ServiceMonitor", new CfnAnomalyMonitorProps
{
MonitorName = "hl-m11-services",
MonitorType = "DIMENSIONAL",
MonitorDimension = "SERVICE",
});
new CfnAnomalySubscription(this, "AnomalyAlerts", new CfnAnomalySubscriptionProps
{
SubscriptionName = "hl-m11-anomaly-daily",
Frequency = "DAILY",
MonitorArnList = new[] { monitor.AttrMonitorArn },
Subscribers = new[]
{
new CfnAnomalySubscription.SubscriberProperty { Type = "EMAIL", Address = finopsEmail },
},
// Alert when the total impact is at least 10 USD ⚠ VERIFY: Threshold is documented as deprecated in favour of ThresholdExpression.
Threshold = 10,
});
// --- A log group with deliberate retention, class and tags ---
var logGroup = new LogGroup(this, "PaymentsLogs", new LogGroupProps
{
LogGroupName = "/hl/payments/api",
Retention = RetentionDays.TWO_WEEKS,
LogGroupClass = LogGroupClass.STANDARD, // alarms/metric filters need Standard
RemovalPolicy = RemovalPolicy.DESTROY, // lab only
});
// --- Alarm on ingestion: more than 5 GB per hour from this one group ---
var incoming = new Metric(new MetricProps
{
Namespace = "AWS/Logs",
MetricName = "IncomingBytes",
DimensionsMap = new Dictionary<string, string> { ["LogGroupName"] = logGroup.LogGroupName },
Statistic = "Sum",
Period = Duration.Hours(1),
});
new Alarm(this, "IngestionSpike", new AlarmProps
{
AlarmName = "hl-m11-payments-logs-ingestion-spike",
AlarmDescription = "SEV3: payments log ingestion over 5 GB in an hour. Check log level and new deploys. Runbook: https://wiki.example.com/runbooks/log-volume",
Metric = incoming,
Threshold = 5L * 1024 * 1024 * 1024,
EvaluationPeriods = 1,
ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING,
}).AddAlarmAction(new SnsAction(topic));
}
}Complete code
A weekly cost-hygiene report in C#. It ranks log groups by bytes ingested over the last seven days, estimates their ingestion cost at a price you supply, and lists log groups with no retention. It is read-only by default; with --fix-retention <days> it sets retention on groups that have none, after printing what it will do.
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<!-- Look up exact 4.x versions at https://api.nuget.org/v3-flatcontainer/<id>/index.json -->
<PackageReference Include="AWSSDK.CloudWatch" Version="4.*" />
<PackageReference Include="AWSSDK.CloudWatchLogs" Version="4.*" />
</ItemGroup>
</Project>using Amazon;
using Amazon.CloudWatch;
using Amazon.CloudWatch.Model;
using Amazon.CloudWatchLogs;
using Amazon.CloudWatchLogs.Model;
// Usage: dotnet run -- [--price 0.50] [--fix-retention 14] [--top 15]
var price = 0.50m; // USD per GB, Standard class. ⚠ VERIFY on the pricing page
int? fixRetention = null;
var top = 15;
for (var i = 0; i < args.Length - 1; i++)
{
switch (args[i])
{
case "--price" when decimal.TryParse(args[i + 1], System.Globalization.CultureInfo.InvariantCulture, out var p):
price = p; break;
case "--fix-retention" when int.TryParse(args[i + 1], out var d):
fixRetention = d; break;
case "--top" when int.TryParse(args[i + 1], out var t):
top = t; break;
}
}
var region = RegionEndpoint.USEast1;
using var logs = new AmazonCloudWatchLogsClient(region);
using var cw = new AmazonCloudWatchClient(region);
// 1) All log groups (paged).
var groups = new List<LogGroup>();
string? token = null;
do
{
var res = await logs.DescribeLogGroupsAsync(new DescribeLogGroupsRequest { NextToken = token });
groups.AddRange(res.LogGroups ?? []);
token = res.NextToken;
} while (token is not null);
Console.WriteLine($"{groups.Count} log group(s) found.\n");
// 2) Bytes ingested in the last 7 days per group. GetMetricData accepts up to 500 queries per call.
var end = DateTime.UtcNow;
var start = end.AddDays(-7);
var weeklyBytes = new Dictionary<string, double>();
foreach (var batch in groups.Chunk(400))
{
var queries = batch.Select((g, idx) => new MetricDataQuery
{
Id = $"g{idx}",
ReturnData = true,
MetricStat = new MetricStat
{
Period = 7 * 24 * 3600,
Stat = "Sum",
Metric = new Metric
{
Namespace = "AWS/Logs",
MetricName = "IncomingBytes",
Dimensions = [new Dimension { Name = "LogGroupName", Value = g.LogGroupName }],
},
},
}).ToList();
var data = await cw.GetMetricDataAsync(new GetMetricDataRequest
{
StartTime = start,
EndTime = end,
MetricDataQueries = queries,
});
foreach (var r in data.MetricDataResults ?? [])
{
var idx = int.Parse(r.Id[1..]);
weeklyBytes[batch[idx].LogGroupName] = (r.Values ?? []).Sum();
}
}
// 3) Rank, convert to GB and estimated monthly ingestion cost (weekly volume x 30/7).
const double GiB = 1024d * 1024 * 1024;
Console.WriteLine($"Top {top} log groups by ingestion (last 7 days). Price assumed: ${price}/GB\n");
Console.WriteLine($"{"Log group",-52} {"GB/week",9} {"~USD/month",11} Retention");
foreach (var g in groups
.OrderByDescending(g => weeklyBytes.GetValueOrDefault(g.LogGroupName))
.Take(top))
{
var gb = weeklyBytes.GetValueOrDefault(g.LogGroupName) / GiB;
var monthly = (decimal)gb * 30m / 7m * price;
var retention = g.RetentionInDays is int r ? $"{r} d" : "NEVER EXPIRES";
Console.WriteLine($"{Truncate(g.LogGroupName, 52),-52} {gb,9:F2} {monthly,11:F2} {retention}");
}
// 4) Groups with no retention.
var noRetention = groups.Where(g => g.RetentionInDays is null).ToList();
Console.WriteLine($"\n{noRetention.Count} group(s) have no retention policy.");
if (fixRetention is int days && noRetention.Count > 0)
{
Console.WriteLine($"Setting {days}-day retention on {noRetention.Count} group(s). Ctrl+C within 5 seconds to cancel.");
await Task.Delay(TimeSpan.FromSeconds(5));
foreach (var g in noRetention)
{
await logs.PutRetentionPolicyAsync(new PutRetentionPolicyRequest
{
LogGroupName = g.LogGroupName,
RetentionInDays = days,
});
Console.WriteLine($" set {g.LogGroupName}");
}
}
else if (noRetention.Count > 0)
{
Console.WriteLine("Re-run with --fix-retention <days> after agreeing the period with the data owner.");
}
return 0;
static string Truncate(string s, int max) => s.Length <= max ? s : "..." + s[^(max - 3)..];Code explained line by line
- The argument loop uses
switch ... whenpatterns to parse three optional flags without a command-line library.CultureInfo.InvariantCulturemakes0.50parse the same on every machine. DescribeLogGroupsAsyncis paged, so the do/while followsNextTokenuntil it is null. Missing this is the classic cause of "report shows only 50 groups".groups.Chunk(400)keeps eachGetMetricDatacall safely under its 500-query limit. ⚠ VERIFY the current limit.- Each query asks for a single data point:
Period = 7 * 24 * 3600with theSumstatistic gives one total for the whole week, which is all the report needs. Smaller periods return more data points and more API cost for no benefit here. - The query
Idmust begin with a lowercase letter, sog0,g1and so on.r.Id[1..]parses the index back. - Fully qualified or aliased
MetricandDimensioncome fromAmazon.CloudWatch.Model; no otherMetrictype is imported, so no ambiguity here. - The monthly estimate multiplies the weekly gigabytes by 30/7 and the price. It estimates ingestion only, not storage or queries.
RetentionInDays is int ruses a pattern to unwrap the nullable.nullmeans never expires, which is the default for new log groups unless you set one.- The destructive part is opt-in and delayed:
--fix-retentionprints a message and waits five seconds, so a mistaken run can be cancelled. Shortening retention deletes older logs permanently. Agree the period with the data owner first.
Security considerations
⚠ Learning shortcut — not for production
The lab tool runs with your sandbox role and can change retention on every log group in the account. In production, give the cost report a read-only role, apply retention through infrastructure as code (so changes are reviewed), and treat any shortening of retention as a controlled change that needs sign-off from the data owner and, where applicable, the compliance team.
- Retention is a compliance decision, not only a cost decision. Cutting retention can destroy evidence you are obliged to keep; keeping logs too long can breach data-minimisation expectations under UK GDPR. Cost pressure alone must not decide it.
- Debug logging is a data risk. Request bodies in logs may include personal data. Raising the log level back to a normal one reduces both cost and exposure.
- Cost data reveals the architecture. Restrict Cost Explorer and Budgets to billing and FinOps roles.
- Budgets and anomaly alerts do not stop spending. They notify. If you need hard limits, use quotas, SCPs or automation with human approval.
- Do not auto-delete log groups. Automation that deletes data on a cost trigger can remove evidence during an incident, when logging is highest.
- Tag everything. Untagged log groups cannot be attributed, which turns every cost conversation into a hunt.
Banking use case
Harbourline's observability cost policy:
| Area | Policy | Why |
|---|---|---|
| Retention | 14 days for application logs in Standard; 30 days for audit-relevant services; archive evidence to S3 with Object Lock and a lifecycle rule | Cost and data minimisation; the evidence period comes from policy, confirmed with Compliance |
| Log class | Standard for logs that feed alarms or EMF; Essentials for rarely read, high-volume logs (flow logs, access logs) | Essentials is cheaper but has no metric filters |
| Levels | Information in production; Debug only through a time-limited feature flag that expires in 4 hours | Prevents the forgotten debug flag |
| Metrics | Dimensions limited to service, environment, route template, status class; never customer, account or request id | Cardinality creates cost and slows queries |
| Dashboards | One per service plus the triage dashboard; delete those unused for 90 days | Dashboards are billed |
| Canaries | Production only; 5-minute frequency for critical journeys, 15 minutes elsewhere | Cost scales with frequency |
| Guardrails | Budget on CloudWatch, anomaly monitor by service, IncomingBytes alarms on the five noisiest groups | Catch a mistake in a day |
| Review | Monthly top-10 log group review by service owners | Ownership |
A common result of the first review is a 30 to 60 percent reduction in log volume, achieved without losing any alert, by removing health-check logs, request bodies and repeated retry messages. ⚠ This range is anecdotal; measure your own baseline.
Common mistakes
- Leaving retention at "never expire". Storage grows forever and old personal data lingers.
- Debug logging left on after an incident. The largest single cause of surprise logging bills.
- High-cardinality dimensions. Each unique combination is a separate billed metric: 50,000 customers × 3 metrics is 150,000 metrics.
- Logging health checks and static assets. Pure noise at high volume.
- Choosing Essentials, then needing a metric filter. The class cannot be changed; you must create a new log group.
- Duplicating data. Centralization plus subscription plus S3 export each add cost; replicate only what you need.
- Running broad Logs Insights queries over months of data repeatedly. You pay per gigabyte scanned; narrow the time range and log groups, and save results.
- Ignoring third-party API polling. Tools calling
GetMetricDataorListMetricsconstantly create request charges. - Setting a budget with no recipient who reads it. Alerts need an owner.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| CloudWatch bill jumped overnight | Ingestion spike (debug level, retry storm) or a new high-cardinality metric | Cost Explorer daily by usage type; IncomingBytes per group; list-metrics counts |
| Cost Explorer shows no CloudWatch data | Cost Explorer not enabled or data delay (up to about a day) | Enable it; wait; check the account is the payer or has access |
| Budget never alerts | Service filter string does not match, or notification recipients unconfirmed | Check the filter in the console; confirm email subscription |
| Anomaly detection silent | Not enough history, threshold too high, or subscription frequency daily | Wait several days; lower the impact threshold; use SNS for faster alerts |
IncomingBytes alarm in INSUFFICIENT_DATA | Log group received no data in the period | Expected for quiet groups; use treatMissingData = notBreaching |
| Report shows 0 for a busy group | Metric lag, wrong dimension value or different Region | Check the Region and exact LogGroupName; wait a few minutes |
put-retention-policy rejected | Invalid number of days (only specific values are accepted) | Use an allowed value such as 7, 14, 30, 60, 90, 180, 365 ⚠ VERIFY list |
| Retention set but storage did not fall | Deletion of expired data is asynchronous | Allow time; check the stored bytes later |
| Custom metric count keeps growing | Unbounded dimension values | Find the dimension with list-metrics, remove it in the code, wait for old series to age out |
Cost and cleanup
| Item | Label | Estimate |
|---|---|---|
| Cost Explorer API calls from the CLI | credits | A small charge per paginated request, so run it a handful of times ⚠ VERIFY |
| Budget | free | The first budgets are free; check current terms ⚠ VERIFY |
| Anomaly monitor and subscription | free | Cost Anomaly Detection has no additional charge ⚠ VERIFY |
| One log group, one alarm, one SNS topic | credits | Pennies |
| The C# report | free | Reads metrics and log group metadata |
Free alternative: read the worked estimate, plug in your own numbers in a spreadsheet and rank which line you would attack first. Run the report tool only if you have an AWS account; it reads the sandbox's own few log groups.
# Remove the lab resources
cdk destroy CostGuardrailsStack --profile learn-sandbox # or: terraform destroy
aws budgets delete-budget --account-id 123456789012 --budget-name hl-m11-cloudwatch-monthly --profile learn-sandbox --region us-east-1
aws ce list-anomaly-monitors --profile learn-sandbox --region us-east-1 # find the monitor ARN
aws ce delete-anomaly-monitor --monitor-arn <monitor-arn> --profile learn-sandbox --region us-east-1
aws ce list-anomaly-subscriptions --profile learn-sandbox --region us-east-1 # find the subscription ARN
aws ce delete-anomaly-subscription --subscription-arn <subscription-arn> --profile learn-sandbox --region us-east-1
# Confirm nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1 --query 'ResourceTagMappingList[].ResourceARN'Exercises
- 🟢 Guided: Redo the worked estimate for a smaller service: 8 GB/day of Standard logs, 14-day retention, 150 custom metrics, 12 alarms, 1 dashboard (within the free allowance) and no canary. Write the total and the single biggest line.
- 🟡 Challenge: Extend
LogCostReportwith a--classsummary that lists how many log groups are in Standard and Essentials (useLogGroupClass) and flags any Standard group whose log name containsfloworaccessas a candidate for Essentials in a new group. Explain why you cannot convert it in place. - 🔴 Live problem: Ticket: "CloudWatch costs doubled in October and the owner of the platform account says nothing changed." You have read access to Cost Explorer, CloudTrail and CloudWatch. Describe the order of investigation (usage type, day of the jump,
IncomingBytesby group, metric counts, new alarms or canaries, deploys that day, CloudTrail for log-level or configuration changes) and the three most likely root causes with the evidence that would confirm each.
Assignment
Write a "CloudWatch cost policy" for Harbourline: retention by log category with the source of each requirement, log class rules, log level rules, metric dimension rules, dashboard and canary rules, the three guardrails, and a monthly review checklist. Include the worked estimate for your own environment and a one-page explanation for the finance team of why the bill moves with traffic.
Solutions
Ingestion: 8 GB × 30 × $0.50 = $120. Storage: about 8 × 14 = 112 GB held × $0.03 = about $3.4 (upper bound). Custom metrics: 150 × $0.30 = $45, minus the free 10 = about $42. Alarms: 12 × $0.10 = $1.20, minus the free 10 alarm metrics, about $0.20. Dashboards: one, within the free three. Total about $166. The biggest line is ingestion, so the first lever is log volume (level, bodies, health checks), then metric cardinality. Notice that retention cuts storage but not ingestion, which is why shortening retention alone rarely fixes a large bill.
Use g.LogGroupClass (an enum-like string, STANDARD or INFREQUENT_ACCESS/ESSENTIALS depending on the API version; compare case-insensitively and accept both names) and count by class with GroupBy. Flag Standard groups whose names match flow or access. You cannot convert in place because a log group's class is fixed at creation: create a new Essentials group, point the producer at it, and let the old group expire under its retention. Remember Essentials groups have no metric filters, no subscription filters and no EMF, so check that nothing depends on them first.
Knowledge check
Answer: Ingestion, by a wide margin in typical workloads (about $0.50 per GB ingested versus about $0.03 per GB-month stored in the pricing examples). Shortening retention helps storage only.
Answer: Each unique combination of metric name and dimension values is a separate custom metric, billed per month. 50,000 customers create 50,000 series per metric name.
Answer: Never expire. You must set a retention period explicitly.
Answer: No. Budgets notify (and can trigger actions you configure); they are not hard limits by default.
Answer: Essentials (formerly Infrequent Access), for the lower ingestion price. You give up metric filters, subscription filters, EMF, Live Tail and anomaly detection, and the class cannot be changed later.
Interview questions
- CloudWatch costs doubled. How do you find the cause? Cost Explorer by usage type and day to find the meter and the date, then
IncomingBytesper log group, metric counts per namespace, new alarms and canaries, and deployments and configuration changes that day. - How do you reduce logging cost without losing observability? Lower levels, drop noisy lines and bodies, sample high-volume info logs, set retention, use Essentials for rarely read logs, and keep alarms on metrics rather than raw log scans.
- What is metric cardinality and why does it matter? The number of distinct series created by dimension combinations; each is billed and makes queries slower, so dimensions must come from bounded sets.
- Who should decide log retention? The data owner with Compliance and the data-protection officer, based on documented requirements; engineering supplies the cost impact.
Certification practice
A company notices that CloudWatch Logs charges rose sharply after an application upgrade. Which metric helps identify the log group responsible?
A. AWS/Logs IncomingBytes by log group.
B. AWS/EC2 CPUUtilization.
C. AWS/S3 NumberOfObjects.
D. AWS/RDS ReadIOPS.
Answer: A. IncomingBytes per LogGroupName shows which groups ingest the most. The others measure unrelated resources.
A team wants an automatic email when the daily cost of any AWS service is unusually high compared with its history, with no thresholds to tune per service. Which feature fits?
A. AWS Cost Anomaly Detection with a service-dimension monitor. B. A CloudWatch dashboard. C. An S3 lifecycle rule. D. A Route 53 health check.
Answer: A. Cost Anomaly Detection learns spend patterns and alerts on unusual changes. The other options do not analyse cost.
An application writes verbose logs that are almost never read but must be searchable occasionally during investigations. No alarms depend on them. Which approach reduces cost the MOST while keeping searchability?
A. Use a new log group in the Essentials (Infrequent Access) class with an appropriate retention period. B. Disable logging. C. Switch the log group to Essentials in place. D. Export to S3 every minute.
Answer: A. Essentials has a lower ingestion price and still supports Logs Insights queries. C is not possible because the class cannot change after creation. B loses evidence. D adds complexity and cost.
Completion checklist
- I can name the main CloudWatch cost drivers and which is usually largest
- I can estimate a monthly bill from ingestion, metrics, alarms and dashboards
- I ran or read the cost report and found log groups with no retention
- I created (or designed) a budget, an anomaly monitor and an ingestion alarm
- I can explain why retention is a compliance decision as well as a cost decision
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- Ingestion and metric cardinality usually dominate the CloudWatch bill.
- New log groups never expire by default: set retention deliberately, with the data owner's agreement.
- Choose the log class at creation; it cannot be changed, and Essentials lacks metric filters, subscriptions and EMF.
- Use
IncomingBytes, a CloudWatch-scoped budget and anomaly detection to catch mistakes within a day. - Debug logging left on is both a cost and a data-protection problem.
Further reading
- Amazon CloudWatch pricing
- Log classes
- AWS CloudTrail pricing
- Log aggregation and error monitoring (this course)
- Lab 11: observe a banking API for the module project, then the Module 11 overview.
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Incident response and on-call runbooks: severity, roles, communication and reviews
Design the human side of operations: severity levels, incident roles, communication cadence, runbooks that an engineer can follow at 3 a.m., a semi-automated rollback with Systems Manager Automation, and a blameless post-incident review.
Module 11 cheat sheet
One-page revision summary for Monitoring, Logging & Troubleshooting: services, Azure mapping, CLI commands, limits, decision rules, exam traps and cost traps.