Skip to Content
Cloud Engineering Academy
11. Monitoring, Logging & Troubleshooting

Cost monitoring for observability: CloudWatch bills, budgets and anomaly alerts

Understand what makes CloudWatch and related observability services expensive, find your largest log groups and metric sources, cut waste without losing signal, and set up a budget and anomaly detection so a logging mistake is caught in a day rather than at month end.

intermediate⏱ 1 h 20 min🟢 FreeSOA-C03DOP-C02SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • List the main CloudWatch cost drivers (log ingestion and storage, custom metrics, alarms, dashboards, Insights queries, Application Signals, Synthetics) and estimate a monthly bill from a volume.
  • Rank log groups by ingested bytes and find groups with no retention, using a small C# tool.
  • Choose the cheapest log class, retention and metric design that still meets the need.
  • Create an AWS Budget scoped to CloudWatch and a Cost Anomaly Detection monitor with alerts.
  • Explain why high-cardinality metric dimensions and debug logging in production are the classic cost traps.

Prerequisites

Why this matters in banking

Observability is one of the few cloud costs that grows by itself. Every new microservice adds log groups, every outage adds temporary debug logging that nobody removes, every dashboard adds widgets. In regulated firms, logs also get kept "just in case" for years. Left alone, the monitoring bill can approach the size of the compute bill it is watching.

A typical story: during an incident an engineer sets the payments service to Debug level to see more. The incident ends, the setting stays, and ingestion rises from 20 GB to 300 GB a day. Nobody notices until the monthly invoice, by which time the extra cost is five figures. There is a second cost too: debug logs often contain request bodies, which means personal data is now sitting in a log group with unlimited retention. Good cost monitoring is also good data hygiene.

The idea in plain language

CloudWatch is metered like a utility: you pay for what flows in, what you keep and what you ask of it. Each dimension has its own meter:

  • Flow in: every gigabyte of logs ingested, every custom metric data stream, every canary run, every signal.
  • Keep: every gigabyte-month of log storage; every custom metric that exists.
  • Ask: every gigabyte scanned by Logs Insights, every GetMetricData request from third-party tools, every dashboard widget refresh by an API client.

The surprising fact for most developers: ingestion is usually the biggest meter, and cardinality (how many distinct metric series you create) is the sneakiest.

Analogy: a water bill with a charge for every litre that enters the building (ingestion), a storage-tank rent (retention), and a fee per tap that exists (metrics, alarms). Fixing a leak helps most; leaving every tap running for no reason costs the rest.

What problem the AWS service solves

AWS gives you the meters, not the discipline. The tools:

ToolWhat it doesNotes
Cost ExplorerBreaks costs down by service, usage type, tagGrouping by usage type is the fastest way to find which CloudWatch meter is growing
AWS BudgetsAlerts when cost or forecast exceeds a threshold, filterable by service and tagAlerts are not hard limits: they notify, they do not stop spending
Cost Anomaly DetectionMachine-learning monitors that alert on unusual spendNeeds a few days of history; alerts by email or SNS ⚠ VERIFY current behaviour
Cost allocation tagsAttribute costs to a team or serviceTags on log groups let you split the logging bill; tags must be activated for cost reports ⚠ VERIFY
AWS/Logs metricsIncomingBytes per log groupFree, near-real-time, the best signal for "who is logging a lot"
Billing metricsEstimatedCharges in AWS/Billing (us-east-1 only, needs billing alerts enabled)A coarse early warning ⚠ VERIFY

Price points used in this lesson

Prices change and differ by Region. These come from the CloudWatch pricing page worked examples for US East at the time of checking, and are for estimation only. ⚠ VERIFY every number on the pricing page before using it in a budget.

MeterPrice usedNotes
Log ingestion, Standard class$0.50 per GB (first tier)Lower tiers exist at higher volume
Log ingestion, Essentials (formerly Infrequent Access)$0.25 per GBFewer features: no metric filters, subscription filters, EMF, Live Tail
Log storage$0.03 per GB-monthCharged per month stored; retention drives this
Logs Insightsper GB of data scanned ⚠ VERIFY the current rateNot shown in the portion of the page I could read
Custom metrics$0.30 per metric per month (first 10,000)A metric is one unique name plus dimension combination
Standard alarm$0.10 per alarm metric per monthHigh-resolution alarm $0.30
Composite alarm$0.50 per alarm per month
Dashboardsper dashboard per month beyond the free allowance ⚠ VERIFY priceFree tier: 3 dashboards, up to 50 metrics each
Application Signals$1.50 per million signals (first 100 million), less beyondBilled per signal; spans and X-Ray traces billed separately
Synthetics$0.0012 per canary runFree tier: 100 runs per month
Contributor Insights$0.50 per rule after the firstPlus matched events
Metric streams$0.003 per 1,000 metric updates

Free tier highlights (monthly): 5 GB of logs, 10 custom metrics, 10 alarm metrics, 3 dashboards and 1 million API requests. ⚠ VERIFY

The worked estimate

Harbourline's payments platform ingests 50 GB of logs per day at Standard class with 30-day retention, runs 2,000 custom metrics, 60 alarms, 8 dashboards and one canary every five minutes.

ItemCalculationMonthly
Log ingestion50 GB × 30 days × $0.50$750
Log storage (steady state, 30 days held)1,500 GB × $0.03about $45 (the stored size is usually smaller than raw ingestion because it is compressed; treat this as an upper bound ⚠ VERIFY)
Custom metrics2,000 × $0.30$600
Alarms60 × $0.10$6
Dashboards(8 − 3 free) × dashboard price ($3 assumed ⚠ VERIFY)about $15
Canary12 runs/hour × 24 × 30 = 8,640 runs, minus 100 free, × $0.0012about $10
Total (before Insights queries and Application Signals)about $1,430

Two lines dominate: ingestion and custom metrics. That tells you where to look first, and it is usual: a handful of chatty log groups and a handful of high-cardinality metrics cause most of the bill.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Log Analytics ingestion pricing (per GB), commitment tiersCloudWatch Logs ingestion per GB, volume tiersAWS also has a lower-priced Essentials log class; storage is charged separately
Basic Logs vs Analytics Logs tablesEssentials vs Standard log classesClass chosen at creation, cannot be changed
Daily cap on a workspaceNone built in; use a budget alert and IncomingBytes alarmAWS does not stop ingestion when a budget is exceeded
Application Insights sampling and daily capOpenTelemetry sampling, X-Ray sampling rules, Application SignalsSampling is configured in the SDK or collector
Azure Cost Management budgets and anomaly alertsAWS Budgets and Cost Anomaly DetectionSimilar: notifications, not enforcement
Metric pricing for custom metrics in Azure MonitorPer custom metric per monthEvery distinct dimension combination is a separate billed metric
Table-level retention in Log AnalyticsPer log group retention in daysDefault for new log groups is to never expire

Architecture

Boundary view: cost data is visible to billing roles, not to every engineer. Tags on log groups (Owner, Service, Environment) let you attribute the bill to a team without exposing log contents.

Step-by-step implementation

Find where the money goes

In Cost Explorer, filter by service CloudWatch and group by usage type. The names tell you the meter: ingestion, storage, metrics, alarms, API requests. Note the biggest two.

Rank log groups by ingestion

Use IncomingBytes per log group over the last seven days, multiplied by the ingestion price, to rank groups. The C# tool below does this.

Fix retention

Find log groups with no retention and set one that matches the business need (for example 14 days for debug-heavy application logs, longer only where a documented requirement exists). Archive long-term evidence to S3 with a lifecycle policy rather than keeping it in CloudWatch.

Reduce volume at the source

Raise the minimum log level, stop logging bodies and health checks, sample high-volume info logs and move rarely read logs to the Essentials class (new log groups only).

Tame metric cardinality

Remove unbounded dimensions (customer id, request id, URL path). Keep dimensions to bounded sets such as service, environment and route template.

Add guardrails

An alarm on IncomingBytes for the noisiest groups, a CloudWatch-scoped budget, and Cost Anomaly Detection by service. Route all three to a topic someone reads.

AWS Console instructions

  • Cost Explorer: Console → Billing and Cost Management → Cost Explorer → Filters: Service = CloudWatch → Group by: Usage type. Switch to daily granularity to see spikes.
  • Budgets: Billing and Cost Management → Budgets → Create budget → Cost budget, amount 20 (lab) or your real figure; under Filters choose Service = CloudWatch; add alerts at 80% (actual) and 100% (forecast) with an email recipient.
  • Cost Anomaly Detection: Billing and Cost Management → Cost Anomaly Detection → Create monitor → AWS services; add an alert subscription with your email or an SNS topic and a daily summary. ⚠ VERIFY console wording.
  • Log groups: CloudWatch → Logs → Log groups. Add the columns Retention and Stored bytes; sort by stored bytes. The console shows log class and data protection status too.
  • Retention: select a group → Actions → Edit retention setting.
  • Cost allocation tags: Billing and Cost Management → Cost allocation tags → activate Owner, Service, Environment. Activation applies from the date you activate. ⚠ VERIFY

AWS CLI

# 1) CloudWatch cost by usage type for the last 14 days (Cost Explorer API: small charge per request ⚠ VERIFY).
aws ce get-cost-and-usage \
  --time-period Start=$(date -u -d '-14 days' +%F),End=$(date -u +%F) \
  --granularity DAILY --metrics UnblendedCost \
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["AmazonCloudWatch"]}}' \
  --group-by Type=DIMENSION,Key=USAGE_TYPE \
  --profile learn-sandbox --region us-east-1

# 2) Log groups with NO retention (they never expire) and their stored size.
aws logs describe-log-groups \
  --query 'logGroups[?retentionInDays==`null`].[logGroupName,storedBytes,logGroupClass]' \
  --output table --profile learn-sandbox --region us-east-1

# 3) Set 14-day retention on one log group.
aws logs put-retention-policy --log-group-name /hl/payments/api --retention-in-days 14 \
  --profile learn-sandbox --region us-east-1

# 4) Bytes ingested by one log group over the last 7 days, daily sums.
aws cloudwatch get-metric-statistics --namespace AWS/Logs --metric-name IncomingBytes \
  --dimensions Name=LogGroupName,Value=/hl/payments/api \
  --start-time $(date -u -d '-7 days' +%Y-%m-%dT%H:%M:%SZ) --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 86400 --statistics Sum --profile learn-sandbox --region us-east-1

# 5) Create a budget scoped to CloudWatch (JSON files below).
aws budgets create-budget --account-id 123456789012 \
  --budget file://budget.json --notifications-with-subscribers file://notifications.json \
  --profile learn-sandbox --region us-east-1

# 6) Create an anomaly monitor by service and subscribe an email daily.
aws ce create-anomaly-monitor --anomaly-monitor 'MonitorName=hl-m11-services,MonitorType=DIMENSIONAL,MonitorDimension=SERVICE' \
  --profile learn-sandbox --region us-east-1

# 7) Count custom metrics per namespace (cost driver: each series is billed).
aws cloudwatch list-metrics --namespace Harbourline/Payments --query 'length(Metrics)' \
  --profile learn-sandbox --region us-east-1

budget.json and notifications.json for step 5:

budget.json
{
  "BudgetName": "hl-m11-cloudwatch-monthly",
  "BudgetType": "COST",
  "TimeUnit": "MONTHLY",
  "BudgetLimit": { "Amount": "20", "Unit": "USD" },
  "CostFilters": { "Service": ["Amazon CloudWatch"] }
}
notifications.json
[
  {
    "Notification": {
      "NotificationType": "ACTUAL",
      "ComparisonOperator": "GREATER_THAN",
      "Threshold": 80,
      "ThresholdType": "PERCENTAGE"
    },
    "Subscribers": [{ "SubscriptionType": "EMAIL", "Address": "finops@example.com" }]
  },
  {
    "Notification": {
      "NotificationType": "FORECASTED",
      "ComparisonOperator": "GREATER_THAN",
      "Threshold": 100,
      "ThresholdType": "PERCENTAGE"
    },
    "Subscribers": [{ "SubscriptionType": "EMAIL", "Address": "finops@example.com" }]
  }
]

The usage-type names Cost Explorer returns for CloudWatch (for example ones containing DataProcessing-Bytes for log ingestion, TimedStorage-ByteHrs for log storage, MetricMonitorUsage for custom metrics and AlarmMonitorUsage for alarms, usually prefixed by a Region code) are from memory; read the actual strings from your own output before filtering on them. ⚠ VERIFY The service name in the budget filter (Amazon CloudWatch) differs from the Cost Explorer dimension value (AmazonCloudWatch), and the exact strings should be checked in each console. ⚠ VERIFY

Infrastructure as Code

The stack creates a CloudWatch-scoped monthly budget, an anomaly monitor by service with a subscription, a log group with sensible retention, and an alarm on its ingestion volume. L1 constructs are used for Budgets and Cost Explorer; property names follow the CloudFormation references, compile before relying on them. ⚠ VERIFY

CostGuardrailsStack.cs
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.Budgets;
using Amazon.CDK.AWS.CE;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.CloudWatch.Actions;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.SNS;
using Constructs;

namespace Harbourline.Monitoring.Infra;

public sealed class CostGuardrailsStack : Stack
{
    public CostGuardrailsStack(Construct scope, string id, string finopsEmail, IStackProps? props = null)
        : base(scope, id, props)
    {
        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m11-l14");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");

        var topic = new Topic(this, "FinOpsTopic", new TopicProps { TopicName = "hl-m11-finops-alerts" });

        // --- Budget: CloudWatch only, 20 USD per month, alert at 80% actual and 100% forecast ---
        new CfnBudget(this, "CloudWatchBudget", new CfnBudgetProps
        {
            Budget = new CfnBudget.BudgetDataProperty
            {
                BudgetName = "hl-m11-cloudwatch-monthly",
                BudgetType = "COST",
                TimeUnit = "MONTHLY",
                BudgetLimit = new CfnBudget.SpendProperty { Amount = 20, Unit = "USD" },
                CostFilters = new Dictionary<string, object>
                {
                    ["Service"] = new[] { "Amazon CloudWatch" },
                },
            },
            NotificationsWithSubscribers = new[]
            {
                new CfnBudget.NotificationWithSubscribersProperty
                {
                    Notification = new CfnBudget.NotificationProperty
                    {
                        NotificationType = "ACTUAL",
                        ComparisonOperator = "GREATER_THAN",
                        Threshold = 80,
                        ThresholdType = "PERCENTAGE",
                    },
                    Subscribers = new[]
                    {
                        new CfnBudget.SubscriberProperty { SubscriptionType = "EMAIL", Address = finopsEmail },
                    },
                },
                new CfnBudget.NotificationWithSubscribersProperty
                {
                    Notification = new CfnBudget.NotificationProperty
                    {
                        NotificationType = "FORECASTED",
                        ComparisonOperator = "GREATER_THAN",
                        Threshold = 100,
                        ThresholdType = "PERCENTAGE",
                    },
                    Subscribers = new[]
                    {
                        new CfnBudget.SubscriberProperty { SubscriptionType = "EMAIL", Address = finopsEmail },
                    },
                },
            },
        });

        // --- Cost Anomaly Detection: one monitor across AWS services, daily email ---
        var monitor = new CfnAnomalyMonitor(this, "ServiceMonitor", new CfnAnomalyMonitorProps
        {
            MonitorName = "hl-m11-services",
            MonitorType = "DIMENSIONAL",
            MonitorDimension = "SERVICE",
        });

        new CfnAnomalySubscription(this, "AnomalyAlerts", new CfnAnomalySubscriptionProps
        {
            SubscriptionName = "hl-m11-anomaly-daily",
            Frequency = "DAILY",
            MonitorArnList = new[] { monitor.AttrMonitorArn },
            Subscribers = new[]
            {
                new CfnAnomalySubscription.SubscriberProperty { Type = "EMAIL", Address = finopsEmail },
            },
            // Alert when the total impact is at least 10 USD ⚠ VERIFY: Threshold is documented as deprecated in favour of ThresholdExpression.
            Threshold = 10,
        });

        // --- A log group with deliberate retention, class and tags ---
        var logGroup = new LogGroup(this, "PaymentsLogs", new LogGroupProps
        {
            LogGroupName = "/hl/payments/api",
            Retention = RetentionDays.TWO_WEEKS,
            LogGroupClass = LogGroupClass.STANDARD, // alarms/metric filters need Standard
            RemovalPolicy = RemovalPolicy.DESTROY,  // lab only
        });

        // --- Alarm on ingestion: more than 5 GB per hour from this one group ---
        var incoming = new Metric(new MetricProps
        {
            Namespace = "AWS/Logs",
            MetricName = "IncomingBytes",
            DimensionsMap = new Dictionary<string, string> { ["LogGroupName"] = logGroup.LogGroupName },
            Statistic = "Sum",
            Period = Duration.Hours(1),
        });

        new Alarm(this, "IngestionSpike", new AlarmProps
        {
            AlarmName = "hl-m11-payments-logs-ingestion-spike",
            AlarmDescription = "SEV3: payments log ingestion over 5 GB in an hour. Check log level and new deploys. Runbook: https://wiki.example.com/runbooks/log-volume",
            Metric = incoming,
            Threshold = 5L * 1024 * 1024 * 1024,
            EvaluationPeriods = 1,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
            TreatMissingData = TreatMissingData.NOT_BREACHING,
        }).AddAlarmAction(new SnsAction(topic));
    }
}

Complete code

A weekly cost-hygiene report in C#. It ranks log groups by bytes ingested over the last seven days, estimates their ingestion cost at a price you supply, and lists log groups with no retention. It is read-only by default; with --fix-retention <days> it sets retention on groups that have none, after printing what it will do.

LogCostReport.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
  </PropertyGroup>
  <ItemGroup>
    <!-- Look up exact 4.x versions at https://api.nuget.org/v3-flatcontainer/<id>/index.json -->
    <PackageReference Include="AWSSDK.CloudWatch" Version="4.*" />
    <PackageReference Include="AWSSDK.CloudWatchLogs" Version="4.*" />
  </ItemGroup>
</Project>
Program.cs
using Amazon;
using Amazon.CloudWatch;
using Amazon.CloudWatch.Model;
using Amazon.CloudWatchLogs;
using Amazon.CloudWatchLogs.Model;

// Usage: dotnet run -- [--price 0.50] [--fix-retention 14] [--top 15]
var price = 0.50m;           // USD per GB, Standard class. ⚠ VERIFY on the pricing page
int? fixRetention = null;
var top = 15;

for (var i = 0; i < args.Length - 1; i++)
{
    switch (args[i])
    {
        case "--price" when decimal.TryParse(args[i + 1], System.Globalization.CultureInfo.InvariantCulture, out var p):
            price = p; break;
        case "--fix-retention" when int.TryParse(args[i + 1], out var d):
            fixRetention = d; break;
        case "--top" when int.TryParse(args[i + 1], out var t):
            top = t; break;
    }
}

var region = RegionEndpoint.USEast1;
using var logs = new AmazonCloudWatchLogsClient(region);
using var cw = new AmazonCloudWatchClient(region);

// 1) All log groups (paged).
var groups = new List<LogGroup>();
string? token = null;
do
{
    var res = await logs.DescribeLogGroupsAsync(new DescribeLogGroupsRequest { NextToken = token });
    groups.AddRange(res.LogGroups ?? []);
    token = res.NextToken;
} while (token is not null);

Console.WriteLine($"{groups.Count} log group(s) found.\n");

// 2) Bytes ingested in the last 7 days per group. GetMetricData accepts up to 500 queries per call.
var end = DateTime.UtcNow;
var start = end.AddDays(-7);
var weeklyBytes = new Dictionary<string, double>();

foreach (var batch in groups.Chunk(400))
{
    var queries = batch.Select((g, idx) => new MetricDataQuery
    {
        Id = $"g{idx}",
        ReturnData = true,
        MetricStat = new MetricStat
        {
            Period = 7 * 24 * 3600,
            Stat = "Sum",
            Metric = new Metric
            {
                Namespace = "AWS/Logs",
                MetricName = "IncomingBytes",
                Dimensions = [new Dimension { Name = "LogGroupName", Value = g.LogGroupName }],
            },
        },
    }).ToList();

    var data = await cw.GetMetricDataAsync(new GetMetricDataRequest
    {
        StartTime = start,
        EndTime = end,
        MetricDataQueries = queries,
    });

    foreach (var r in data.MetricDataResults ?? [])
    {
        var idx = int.Parse(r.Id[1..]);
        weeklyBytes[batch[idx].LogGroupName] = (r.Values ?? []).Sum();
    }
}

// 3) Rank, convert to GB and estimated monthly ingestion cost (weekly volume x 30/7).
const double GiB = 1024d * 1024 * 1024;
Console.WriteLine($"Top {top} log groups by ingestion (last 7 days). Price assumed: ${price}/GB\n");
Console.WriteLine($"{"Log group",-52} {"GB/week",9} {"~USD/month",11}  Retention");

foreach (var g in groups
             .OrderByDescending(g => weeklyBytes.GetValueOrDefault(g.LogGroupName))
             .Take(top))
{
    var gb = weeklyBytes.GetValueOrDefault(g.LogGroupName) / GiB;
    var monthly = (decimal)gb * 30m / 7m * price;
    var retention = g.RetentionInDays is int r ? $"{r} d" : "NEVER EXPIRES";
    Console.WriteLine($"{Truncate(g.LogGroupName, 52),-52} {gb,9:F2} {monthly,11:F2}  {retention}");
}

// 4) Groups with no retention.
var noRetention = groups.Where(g => g.RetentionInDays is null).ToList();
Console.WriteLine($"\n{noRetention.Count} group(s) have no retention policy.");

if (fixRetention is int days && noRetention.Count > 0)
{
    Console.WriteLine($"Setting {days}-day retention on {noRetention.Count} group(s). Ctrl+C within 5 seconds to cancel.");
    await Task.Delay(TimeSpan.FromSeconds(5));

    foreach (var g in noRetention)
    {
        await logs.PutRetentionPolicyAsync(new PutRetentionPolicyRequest
        {
            LogGroupName = g.LogGroupName,
            RetentionInDays = days,
        });
        Console.WriteLine($"  set {g.LogGroupName}");
    }
}
else if (noRetention.Count > 0)
{
    Console.WriteLine("Re-run with --fix-retention <days> after agreeing the period with the data owner.");
}

return 0;

static string Truncate(string s, int max) => s.Length <= max ? s : "..." + s[^(max - 3)..];

Code explained line by line

  1. The argument loop uses switch ... when patterns to parse three optional flags without a command-line library. CultureInfo.InvariantCulture makes 0.50 parse the same on every machine.
  2. DescribeLogGroupsAsync is paged, so the do/while follows NextToken until it is null. Missing this is the classic cause of "report shows only 50 groups".
  3. groups.Chunk(400) keeps each GetMetricData call safely under its 500-query limit. ⚠ VERIFY the current limit.
  4. Each query asks for a single data point: Period = 7 * 24 * 3600 with the Sum statistic gives one total for the whole week, which is all the report needs. Smaller periods return more data points and more API cost for no benefit here.
  5. The query Id must begin with a lowercase letter, so g0, g1 and so on. r.Id[1..] parses the index back.
  6. Fully qualified or aliased Metric and Dimension come from Amazon.CloudWatch.Model; no other Metric type is imported, so no ambiguity here.
  7. The monthly estimate multiplies the weekly gigabytes by 30/7 and the price. It estimates ingestion only, not storage or queries.
  8. RetentionInDays is int r uses a pattern to unwrap the nullable. null means never expires, which is the default for new log groups unless you set one.
  9. The destructive part is opt-in and delayed: --fix-retention prints a message and waits five seconds, so a mistaken run can be cancelled. Shortening retention deletes older logs permanently. Agree the period with the data owner first.

Security considerations

⚠ Learning shortcut — not for production

The lab tool runs with your sandbox role and can change retention on every log group in the account. In production, give the cost report a read-only role, apply retention through infrastructure as code (so changes are reviewed), and treat any shortening of retention as a controlled change that needs sign-off from the data owner and, where applicable, the compliance team.

  • Retention is a compliance decision, not only a cost decision. Cutting retention can destroy evidence you are obliged to keep; keeping logs too long can breach data-minimisation expectations under UK GDPR. Cost pressure alone must not decide it.
  • Debug logging is a data risk. Request bodies in logs may include personal data. Raising the log level back to a normal one reduces both cost and exposure.
  • Cost data reveals the architecture. Restrict Cost Explorer and Budgets to billing and FinOps roles.
  • Budgets and anomaly alerts do not stop spending. They notify. If you need hard limits, use quotas, SCPs or automation with human approval.
  • Do not auto-delete log groups. Automation that deletes data on a cost trigger can remove evidence during an incident, when logging is highest.
  • Tag everything. Untagged log groups cannot be attributed, which turns every cost conversation into a hunt.

Banking use case

Harbourline's observability cost policy:

AreaPolicyWhy
Retention14 days for application logs in Standard; 30 days for audit-relevant services; archive evidence to S3 with Object Lock and a lifecycle ruleCost and data minimisation; the evidence period comes from policy, confirmed with Compliance
Log classStandard for logs that feed alarms or EMF; Essentials for rarely read, high-volume logs (flow logs, access logs)Essentials is cheaper but has no metric filters
LevelsInformation in production; Debug only through a time-limited feature flag that expires in 4 hoursPrevents the forgotten debug flag
MetricsDimensions limited to service, environment, route template, status class; never customer, account or request idCardinality creates cost and slows queries
DashboardsOne per service plus the triage dashboard; delete those unused for 90 daysDashboards are billed
CanariesProduction only; 5-minute frequency for critical journeys, 15 minutes elsewhereCost scales with frequency
GuardrailsBudget on CloudWatch, anomaly monitor by service, IncomingBytes alarms on the five noisiest groupsCatch a mistake in a day
ReviewMonthly top-10 log group review by service ownersOwnership

A common result of the first review is a 30 to 60 percent reduction in log volume, achieved without losing any alert, by removing health-check logs, request bodies and repeated retry messages. ⚠ This range is anecdotal; measure your own baseline.

Common mistakes

  • Leaving retention at "never expire". Storage grows forever and old personal data lingers.
  • Debug logging left on after an incident. The largest single cause of surprise logging bills.
  • High-cardinality dimensions. Each unique combination is a separate billed metric: 50,000 customers × 3 metrics is 150,000 metrics.
  • Logging health checks and static assets. Pure noise at high volume.
  • Choosing Essentials, then needing a metric filter. The class cannot be changed; you must create a new log group.
  • Duplicating data. Centralization plus subscription plus S3 export each add cost; replicate only what you need.
  • Running broad Logs Insights queries over months of data repeatedly. You pay per gigabyte scanned; narrow the time range and log groups, and save results.
  • Ignoring third-party API polling. Tools calling GetMetricData or ListMetrics constantly create request charges.
  • Setting a budget with no recipient who reads it. Alerts need an owner.

Troubleshooting

SymptomLikely causeFix
CloudWatch bill jumped overnightIngestion spike (debug level, retry storm) or a new high-cardinality metricCost Explorer daily by usage type; IncomingBytes per group; list-metrics counts
Cost Explorer shows no CloudWatch dataCost Explorer not enabled or data delay (up to about a day)Enable it; wait; check the account is the payer or has access
Budget never alertsService filter string does not match, or notification recipients unconfirmedCheck the filter in the console; confirm email subscription
Anomaly detection silentNot enough history, threshold too high, or subscription frequency dailyWait several days; lower the impact threshold; use SNS for faster alerts
IncomingBytes alarm in INSUFFICIENT_DATALog group received no data in the periodExpected for quiet groups; use treatMissingData = notBreaching
Report shows 0 for a busy groupMetric lag, wrong dimension value or different RegionCheck the Region and exact LogGroupName; wait a few minutes
put-retention-policy rejectedInvalid number of days (only specific values are accepted)Use an allowed value such as 7, 14, 30, 60, 90, 180, 365 ⚠ VERIFY list
Retention set but storage did not fallDeletion of expired data is asynchronousAllow time; check the stored bytes later
Custom metric count keeps growingUnbounded dimension valuesFind the dimension with list-metrics, remove it in the code, wait for old series to age out

Cost and cleanup

ItemLabelEstimate
Cost Explorer API calls from the CLIcreditsA small charge per paginated request, so run it a handful of times ⚠ VERIFY
BudgetfreeThe first budgets are free; check current terms ⚠ VERIFY
Anomaly monitor and subscriptionfreeCost Anomaly Detection has no additional charge ⚠ VERIFY
One log group, one alarm, one SNS topiccreditsPennies
The C# reportfreeReads metrics and log group metadata

Free alternative: read the worked estimate, plug in your own numbers in a spreadsheet and rank which line you would attack first. Run the report tool only if you have an AWS account; it reads the sandbox's own few log groups.

# Remove the lab resources
cdk destroy CostGuardrailsStack --profile learn-sandbox     # or: terraform destroy
aws budgets delete-budget --account-id 123456789012 --budget-name hl-m11-cloudwatch-monthly --profile learn-sandbox --region us-east-1
aws ce list-anomaly-monitors --profile learn-sandbox --region us-east-1       # find the monitor ARN
aws ce delete-anomaly-monitor --monitor-arn <monitor-arn> --profile learn-sandbox --region us-east-1
aws ce list-anomaly-subscriptions --profile learn-sandbox --region us-east-1  # find the subscription ARN
aws ce delete-anomaly-subscription --subscription-arn <subscription-arn> --profile learn-sandbox --region us-east-1

# Confirm nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --profile learn-sandbox --region us-east-1 --query 'ResourceTagMappingList[].ResourceARN'

Exercises

  • 🟢 Guided: Redo the worked estimate for a smaller service: 8 GB/day of Standard logs, 14-day retention, 150 custom metrics, 12 alarms, 1 dashboard (within the free allowance) and no canary. Write the total and the single biggest line.
  • 🟡 Challenge: Extend LogCostReport with a --class summary that lists how many log groups are in Standard and Essentials (use LogGroupClass) and flags any Standard group whose log name contains flow or access as a candidate for Essentials in a new group. Explain why you cannot convert it in place.
  • 🔴 Live problem: Ticket: "CloudWatch costs doubled in October and the owner of the platform account says nothing changed." You have read access to Cost Explorer, CloudTrail and CloudWatch. Describe the order of investigation (usage type, day of the jump, IncomingBytes by group, metric counts, new alarms or canaries, deploys that day, CloudTrail for log-level or configuration changes) and the three most likely root causes with the evidence that would confirm each.

Assignment

Write a "CloudWatch cost policy" for Harbourline: retention by log category with the source of each requirement, log class rules, log level rules, metric dimension rules, dashboard and canary rules, the three guardrails, and a monthly review checklist. Include the worked estimate for your own environment and a one-page explanation for the finance team of why the bill moves with traffic.

Solutions

Knowledge check

Interview questions

  • CloudWatch costs doubled. How do you find the cause? Cost Explorer by usage type and day to find the meter and the date, then IncomingBytes per log group, metric counts per namespace, new alarms and canaries, and deployments and configuration changes that day.
  • How do you reduce logging cost without losing observability? Lower levels, drop noisy lines and bodies, sample high-volume info logs, set retention, use Essentials for rarely read logs, and keep alarms on metrics rather than raw log scans.
  • What is metric cardinality and why does it matter? The number of distinct series created by dimension combinations; each is billed and makes queries slower, so dimensions must come from bounded sets.
  • Who should decide log retention? The data owner with Compliance and the data-protection officer, based on documented requirements; engineering supplies the cost impact.

Certification practice

Completion checklist

  • I can name the main CloudWatch cost drivers and which is usually largest
  • I can estimate a monthly bill from ingestion, metrics, alarms and dashboards
  • I ran or read the cost report and found log groups with no retention
  • I created (or designed) a budget, an anomaly monitor and an ingestion alarm
  • I can explain why retention is a compliance decision as well as a cost decision
  • I ran the cleanup steps and confirmed nothing is still billing

Key takeaways

  • Ingestion and metric cardinality usually dominate the CloudWatch bill.
  • New log groups never expire by default: set retention deliberately, with the data owner's agreement.
  • Choose the log class at creation; it cannot be changed, and Essentials lacks metric filters, subscriptions and EMF.
  • Use IncomingBytes, a CloudWatch-scoped budget and anomaly detection to catch mistakes within a day.
  • Debug logging left on is both a cost and a data-protection problem.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page