Playbook: unexpected AWS bill spike
Find out what caused a cost spike by narrowing from account to service to usage type to resource, stop the spend safely, check for forgotten or looping resources, and set up budgets, anomaly monitors and tag rules so the same mistake is caught within a day.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.CostExplorer 4.x) · AWS CLI v2 · Region us-east-1 (billing data is global). Verified against official sources on 2026-11-18.
Learning objectives
By the end of this playbook you can:
- Run a repeatable narrowing method on a cost spike: account, then service, then usage type, then resource, then the person or change that caused it.
- Recognise the dozen most common spike causes in a bank's AWS estate (NAT data processing, log ingestion, a retry or recursion loop, idle load balancers and public IPv4 addresses, and others) from the shape of the numbers.
- Stop a runaway cost safely: contain without breaking production, and keep the evidence.
- Use Cost Explorer from the console, the CLI and C#, and know its limits (data delay, per-request charge, resource-level data being optional).
- Set up budgets, anomaly monitors, cost allocation tags and a teardown habit so repeats are caught within a day.
- Write a short cost-incident record that finance and audit can read.
Prerequisites
- Free Tier and billing and Cost guardrails, for how billing works and the classic traps
- FinOps foundations, for Cost Explorer, Budgets and Cost Anomaly Detection
- Cost monitoring for CloudWatch cost drivers, and NAT gateways and controlled egress
- Tools: AWS CLI v2 with the
learn-sandboxprofile (billing read access); .NET 10 SDK for the detector, which runs on sample data without an AWS account
Why this matters in banking
At 08:50 on a Monday, Harbourline's finance business partner messages the platform team: the Cost Anomaly Detection email says the shared payments-dev account spent USD 310 on Sunday against a normal USD 45. That is a small number for a bank, which is exactly why teams ignore it. Two weeks later the pattern would have cost USD 3,700, and the budget owner would be asking why nobody acted.
Three things make cost spikes a banking concern and not only a tidiness one:
- A spike is a signal. A sudden jump in data transfer, KMS requests or object reads can mean an exfiltration, a crypto-mining workload on a stolen key, or a runaway job. Treat a surprising bill as a possible security event until you know it is not.
- The cause is often a control that was removed. A VPC endpoint deleted "to save money" sends S3 traffic through a NAT gateway that charges per GB. Cost problems and architecture decisions are linked.
- Finance and audit want an explanation, not a shrug: what happened, how much, who owned it, what you changed. That is a record you can write in 15 minutes if you collected evidence while you investigated.
This playbook assumes you are the engineer asked "why?" and are expected to answer by lunch.
The idea in plain language
A bill is the product of price times quantity across thousands of line items. A spike means a quantity exploded (a loop, a flood of traffic, a forgotten fleet), or a price category changed under you (a resource moved to a more expensive mode, traffic took a metered route). Your job is to find which line item grew, then find who or what made it grow, by narrowing in a fixed order:
| Step | Question | Tool |
|---|---|---|
| 1. Account | Which linked account? | Cost Explorer grouped by linked account |
| 2. Service | Which AWS service? | Group by service |
| 3. Usage type | Which meter inside the service? (for example NatGateway-Bytes, DataTransfer-Regional-Bytes, TimedStorage-ByteHrs) | Group by usage type |
| 4. Resource | Which gateway, bucket, function or log group? | Resource-level Cost Explorer data (an opt-in feature ⚠ VERIFY), tags, or the service's own metrics |
| 5. Cause | Who or what started it, and when? | CloudTrail, deployment history, git log |
Analogy: The electricity bill doubles. You do not start by arguing with the supplier. You walk the building: which floor (account), which kind of appliance (service), which setting (usage type), which exact machine (resource), and who switched it on (cause). The meter alone cannot tell you, so you need the floor plan.
Spend is a stream, so timing is evidence
Costs arrive by day (and by hour if you enable hourly data ⚠ VERIFY). Plot the daily cost for the suspicious service:
- A step up that stays (flat at a higher level) means something was added and left running: a NAT gateway, an idle load balancer, a larger instance, a new log group with no retention.
- A ramp (growing each day) means accumulation: logs, snapshots, S3 versions, a queue backlog.
- A single tall spike then back to normal means a burst: a batch job, a load test, a loop that someone stopped.
- A sawtooth, high on weekdays means a workload pattern, probably not a mistake.
The shape tells you which questions to ask first.
What problem the AWS services solve
| Tool | Use it for | Honest limits |
|---|---|---|
| Cost Explorer | Interactive and API cost and usage by service, account, usage type, Region, tag; forecasts | Data lags by up to about 24 hours; resource-level and hourly data are separate opt-ins ⚠ VERIFY; the API charges per request (about USD 0.01 at last check ⚠ VERIFY) |
| Cost Anomaly Detection | Machine-learning alerts for unusual spend with root-cause hints | Detection can lag up to about a day; needs history; not every charge type is covered (see the FinOps lesson) |
| AWS Budgets | "Tell me when actual or forecast cost crosses X" | Alerts, not stops; updated several times a day |
| Billing and Cost Management data exports (Cost and Usage Report) | Line-item detail including resource IDs, queryable with Athena | Needs setup in advance; covered in Cost and usage reports |
| CloudTrail | Who created or changed the resource that is costing money | Management events show creation; they do not show data volume |
| Service metrics (CloudWatch, S3 Storage Lens, VPC Flow Logs) | The quantity behind the price (bytes, requests, invocations) | Each needs to be enabled or queried per service |
| Cost Optimization Hub, Compute Optimizer, Trusted Advisor | Standing recommendations | Show waste, not what changed last week |
What none of them do is stop spending automatically. Budget actions can, but delayed, and in production a billing alarm should never be able to switch off a payments service without a human decision.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Cost analysis in Azure Cost Management, group by resource group or meter | Cost Explorer grouped by service, usage type or tag | The meter name is the usage type, such as USE1-NatGateway-Bytes; learn to read the prefixes (Region code first) |
| Cost alerts and budgets in Azure | AWS Budgets and Cost Anomaly Detection | Anomaly detection is a separate service you create monitors for |
| Resource groups as a cost boundary | AWS accounts, plus cost allocation tags | An account is the cleanest cost boundary; tags must be activated for billing before they appear |
| Azure Advisor cost recommendations | Cost Optimization Hub and Compute Optimizer | Similar, but you enable them |
| Egress charges for data leaving Azure | Data transfer charges | AWS also charges for traffic between Availability Zones and through NAT gateways, which surprises Azure teams |
| Subscription spending limit on trial accounts | Free plan credits, or Budgets actions | On the Free plan you cannot be charged; on a Paid plan nothing stops spending by default |
| Log Analytics ingestion cost | CloudWatch Logs ingestion (per GB) | Ingestion, not storage, is usually the large number |
Architecture
The investigation funnel
Where the money hides: usage types by suspect
Step-by-step implementation
Minutes 0 to 10: confirm it is real and scope it
- Which view are you looking at? Cost Explorer shows unblended cost by default; Budgets, the invoice and the finance team may use another metric (amortized, net). A one-off annual fee or the end of a credit can look like a spike. Check you compare the same metric and that credits are not hiding a steady cost.
- Which account, which dates? Note the exact start of the change. A daily chart makes it obvious.
- Is it still running? Today's data is partial and delayed. Check the service's own live metrics (for example Lambda invocations, NAT bytes) to see whether it is still going.
- Treat it as a possible security event if the service is unexpected (GPU instances, a Region nobody uses, a new Bedrock model) or the usage looks like data leaving the building. If so, hand it to the incident process now and keep reading here only for evidence.
Minutes 10 to 25: narrow by dimension
In Cost Explorer: filter to the suspicious account and dates, then switch Group by through Service, then Usage type, then Region. At each step write down the dollar amounts and the day the change started. You are looking for the one line item whose daily value changed by roughly the size of the spike.
Typical translation of what you see:
| Usage type you find | What it usually means | First question |
|---|---|---|
NatGateway-Bytes (under EC2-Other) | Traffic processed by a NAT gateway | Did a VPC gateway endpoint for S3 or DynamoDB disappear, or did a workload start pulling a lot of data from the internet? |
NatGateway-Hours, LoadBalancerUsage, PublicIPv4:InUseAddress (and idle) | A resource that is simply running | Who created it, and is it still needed? |
DataTransfer-Regional-Bytes | Traffic between AZs in a Region | New cross-AZ calls, a chatty service, replication |
DataTransfer-Out-Bytes | Data leaving AWS to the internet | Download surge, partner file, exfiltration |
DataProcessing-Bytes or PutLogEvents (CloudWatch) | Log ingestion | A new debug log level, a retry loop that logs each failure |
Requests (KMS) | KMS API calls | S3 without Bucket Keys, tight loops decrypting each item |
Request or Duration-GB-Second (Lambda) | Invocation volume or duration | A recursive trigger, a queue retry storm, a function with a growing timeout |
Requests-Tier1 or Tier2 (S3) | PUT/LIST or GET requests | A job that writes or lists millions of small objects |
TimedStorage-ByteHrs | Stored data growing | Versioning without lifecycle, snapshots, log retention |
ReadRequestUnits, WriteRequestUnits (DynamoDB) | On-demand reads or writes | A scan in a loop, a hot job, a missing index |
Tokens or model usage (Bedrock) | Generative AI usage | An agent loop, a missing token limit, a bulk job |
Minutes 25 to 45: find the resource and the cause
- Group by a tag that identifies the owner (
Project,CostCenter) if the tag is activated for cost allocation. Untagged spend becomes a finding in itself. - Use resource-level data if enabled, or the Cost and Usage Report in Athena for the exact resource IDs of the top line items. ⚠ VERIFY whether resource-level data is enabled in your account before relying on it.
- Ask the service itself about the quantity: NAT gateway bytes (
BytesOutToDestination,BytesOutToSource), Lambda invocations and errors, SQSApproximateAgeOfOldestMessage, S3 request metrics, log groupIncomingBytes. - Find the change. CloudTrail management events for the resource (
CreateNatGateway,CreateFunction,DeleteVpcEndpoints,PutRetentionPolicy) and the deployment and git history for the day the cost changed. In the Harbourline example, the day cost jumped was the day someone deleted the S3 gateway endpoint to test a rollback. - Check for loops. A function that writes to the bucket that triggers it, a message that fails, goes back on the queue, and retries forever, or an EventBridge rule that fires on its own output. The shape is a sharp, sustained jump in requests with a flat business metric.
Minutes 45 to 60: contain without breaking production
Do not delete things in a hurry in a regulated environment. Follow the change process and keep the evidence.
| Situation | Safe containment | Why |
|---|---|---|
| Looping Lambda | Set reserved concurrency to 0 (stops invocations, keeps the function and its config), then fix the trigger | Reversible, no data loss; deleting loses logs and config |
| Poison messages retrying | Pause the consumer or lower its concurrency; check the DLQ and maxReceiveCount | Stops the repeat without deleting messages |
| NAT bytes from S3 or DynamoDB | Add the gateway endpoint back (free) and update route tables | Fixes the cause and removes the cost |
| Idle load balancer, NAT, Elastic IPs | Confirm no traffic, tag as "to delete on date", delete with approval | Avoids breaking an unknown dependency |
| Noisy debug logs | Return the log level to Information and set retention | Fast, and reversible |
| Suspected compromise | Follow the incident process: isolate, rotate keys, preserve CloudTrail | This is a security event first, a cost event second |
Record what you changed, when and who approved it. If you cannot contain safely, escalate to the service owner with the cost per day so the decision is informed.
Minutes 60 to 80: prevent and record
- Add or tighten budgets and anomaly monitors for the account and for the team tag.
- Make the cause impossible or visible: an SCP or policy that blocks the expensive thing in development, a tag policy, a lifecycle rule, a retention requirement in IaC, an alarm on the service metric.
- Write the cost-incident record: dates, amount, cause, owner, containment, root-cause fix, prevention and lessons. If the spend was not intended and large, finance may ask AWS Support about a billing review; whether any adjustment is possible is AWS's decision and is not guaranteed.
Likely causes, ranked by how often they bite
- NAT gateway data processing. S3, DynamoDB or ECR traffic through a NAT gateway, often after a gateway endpoint was removed, or a container image pulled on every task start.
- Idle or forgotten resources. NAT gateways, load balancers, Elastic IPs and public IPv4 addresses, unattached EBS volumes and old snapshots, a test RDS instance, a forgotten Region.
- CloudWatch Logs ingestion. Debug logging left on, a retry loop writing a stack trace per failure, no retention on a new log group.
- Loops. Recursive triggers, message retries without a DLQ, a scheduled job that now overlaps itself.
- Cross-AZ and cross-Region data transfer. Chatty microservices spread across AZs, replication, or a cache in the wrong AZ.
- KMS requests. Not using S3 Bucket Keys, or decrypting per item in a hot loop.
- A bulk or batch job. A reprocessing run, a load test, a migration, an Athena query that scans an unpartitioned table.
- Storage growth. Versioning without lifecycle, forgotten backups, logs kept forever.
- AI usage. Large prompts, an agent loop, no token or concurrency limit.
- Real growth. More customers. Compare with a business metric (payments per day) before blaming a bug; a spike that tracks volume may be fine.
- Security incident. Unauthorised compute in an unused Region, large egress. Check GuardDuty findings and CloudTrail
RunInstancesin unusual Regions.
AWS Console instructions
Cost Explorer: Console → Billing and Cost Management → Cost Explorer → Reports (or New cost and usage report). Set the date range, choose Daily granularity, set Group by to Service, and use Filters for Linked account. Then change Group by to Usage type and add a filter for the suspect service. Use Download CSV to keep the evidence.
Anomalies: Billing and Cost Management → Cost Anomaly Detection → Detected anomalies. Open one to see the root causes (service, account, Region, usage type) and the impact in dollars.
Budgets: Billing and Cost Management → Budgets → open the budget to see actual versus forecast and its alert history.
Resource-level view: Billing and Cost Management → Cost Explorer → Preferences shows whether resource-level data is enabled. ⚠ VERIFY the current option names and any additional charge.
Find idle resources: Console → EC2 → Elastic IPs (unassociated), Volumes (state available), Load Balancers; VPC → NAT gateways; CloudWatch → Log groups sorted by stored bytes.
AWS CLI
# 1) Daily cost by service for the last 14 days (the whole account). Amounts come back as strings.
aws ce get-cost-and-usage \
--time-period Start=2026-11-04,End=2026-11-18 \
--granularity DAILY --metrics UnblendedCost \
--group-by Type=DIMENSION,Key=SERVICE \
--profile learn-sandbox --region us-east-1
# 2) Drill into one service: usage types for EC2-Other (where NAT gateway traffic shows up).
aws ce get-cost-and-usage \
--time-period Start=2026-11-04,End=2026-11-18 \
--granularity DAILY --metrics UnblendedCost \
--filter '{"Dimensions":{"Key":"SERVICE","Values":["EC2 - Other"]}}' \
--group-by Type=DIMENSION,Key=USAGE_TYPE \
--profile learn-sandbox --region us-east-1
# 3) Anomalies Cost Anomaly Detection found in the last 30 days.
aws ce get-anomalies --date-interval StartDate=2026-10-19,EndDate=2026-11-18 \
--profile learn-sandbox --region us-east-1
# 4) Budgets in this account, with actual and forecast spend.
aws budgets describe-budgets --account-id 111122223333 \
--profile learn-sandbox --region us-east-1
# 5) Idle resources: unassociated Elastic IPs, unattached volumes, NAT gateways.
aws ec2 describe-addresses --query 'Addresses[?AssociationId==`null`].[PublicIp,AllocationId]' \
--profile learn-sandbox --region us-east-1
aws ec2 describe-volumes --filters Name=status,Values=available \
--query 'Volumes[].[VolumeId,Size,CreateTime]' --profile learn-sandbox --region us-east-1
aws ec2 describe-nat-gateways --filter Name=state,Values=available \
--query 'NatGateways[].[NatGatewayId,VpcId,CreateTime]' --profile learn-sandbox --region us-east-1
# 6) Who created the NAT gateway, and when? (management events, last 90 days)
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=CreateNatGateway \
--profile learn-sandbox --region us-east-1
# 7) Contain a looping Lambda without deleting it: reserved concurrency 0 stops invocations.
aws lambda put-function-concurrency --function-name hb-statement-renderer \
--reserved-concurrent-executions 0 --profile learn-sandbox --region us-east-1
# Undo after the fix:
aws lambda delete-function-concurrency --function-name hb-statement-renderer \
--profile learn-sandbox --region us-east-1
# 8) Find the log groups that store the most data (largest first).
aws logs describe-log-groups \
--query 'sort_by(logGroups,&storedBytes)[-10:].[logGroupName,storedBytes,retentionInDays]' \
--profile learn-sandbox --region us-east-1Note that every aws ce call is a billed API request. Run a few well-chosen queries, not a loop of them.
Infrastructure as Code
FinOps foundations already shows per-account budgets and a service-level anomaly monitor. The prevention step here adds a team-scoped anomaly monitor (so a spike in one product's tagged spend is flagged even when the account total looks normal) and a budget with a forecast alert.
using Amazon.CDK;
using Amazon.CDK.AWS.Budgets;
using Amazon.CDK.AWS.CE;
using Constructs;
namespace Harbourline.Playbooks;
public sealed class SpikeGuardrailStack : Stack
{
public SpikeGuardrailStack(Construct scope, string id, string alertEmail,
double monthlyUsd, IStackProps? props = null) : base(scope, id, props)
{
Amazon.CDK.Tags.Of(this).Add("Project", "Harbourline");
Amazon.CDK.Tags.Of(this).Add("Environment", "dev");
// A custom monitor that watches only spend tagged Project=payments.
// The tag must be activated as a cost allocation tag first. ⚠ VERIFY the monitor
// specification format against the CloudFormation reference for your CDK version.
var monitor = new CfnAnomalyMonitor(this, "PaymentsMonitor", new CfnAnomalyMonitorProps
{
MonitorName = "hb-payments-tag",
MonitorType = "CUSTOM",
MonitorSpecification =
"{\"Tags\":{\"Key\":\"Project\",\"Values\":[\"payments\"],\"MatchOptions\":[\"EQUALS\"]}}",
});
new CfnAnomalySubscription(this, "PaymentsAnomalies", new CfnAnomalySubscriptionProps
{
SubscriptionName = "hb-payments-daily",
Frequency = "DAILY",
MonitorArnList = new[] { monitor.AttrMonitorArn },
Subscribers = new[]
{
new CfnAnomalySubscription.SubscriberProperty { Type = "EMAIL", Address = alertEmail },
},
ThresholdExpression =
"{\"Dimensions\":{\"Key\":\"ANOMALY_TOTAL_IMPACT_ABSOLUTE\",\"MatchOptions\":[\"GREATER_THAN_OR_EQUAL\"],\"Values\":[\"25\"]}}",
});
// A monthly budget that warns when the FORECAST passes 100 percent.
new CfnBudget(this, "DevBudget", new CfnBudgetProps
{
Budget = new CfnBudget.BudgetDataProperty
{
BudgetName = "hb-dev-monthly",
BudgetType = "COST",
TimeUnit = "MONTHLY",
BudgetLimit = new CfnBudget.SpendProperty { Amount = monthlyUsd, Unit = "USD" },
},
NotificationsWithSubscribers = new[]
{
new CfnBudget.NotificationWithSubscribersProperty
{
Notification = new CfnBudget.NotificationProperty
{
NotificationType = "FORECASTED",
ComparisonOperator = "GREATER_THAN",
Threshold = 100,
ThresholdType = "PERCENTAGE",
},
Subscribers = new[]
{
new CfnBudget.SubscriberProperty { SubscriptionType = "EMAIL", Address = alertEmail },
},
},
},
});
}
}Complete code
A spike detector you can run in a pipeline or from a laptop. It pulls 15 days of daily cost by service (one Cost Explorer request, plus pages if needed), compares the newest full day with the median of the earlier days, and prints services that rose by at least a dollar amount and a ratio. The median ignores a single odd day. In sample mode it uses built-in fictional data, so you can run it without an AWS account. Exit code 1 means something was flagged.
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AWSSDK.CostExplorer" Version="4.*" />
</ItemGroup>
</Project>using System.Globalization;
using Amazon.CostExplorer;
using Amazon.CostExplorer.Model;
// Usage:
// dotnet run -- live [profile] fetch 15 days of daily cost by service (about 1 Cost Explorer request)
// dotnet run -- sample run the detector on built-in fictional data (no AWS needed)
string mode = args.Length > 0 ? args[0] : "sample";
IReadOnlyDictionary<string, decimal[]> series;
if (mode == "live")
{
if (args.Length > 1) Environment.SetEnvironmentVariable("AWS_PROFILE", args[1]);
using var ce = new AmazonCostExplorerClient(Amazon.RegionEndpoint.USEast1);
series = await CostFetcher.DailyByServiceAsync(ce, days: 15);
}
else
{
series = SpikeDetector.SampleData();
}
var findings = SpikeDetector.Detect(series, minDailyIncrease: 5m, minRatio: 2m);
if (findings.Count == 0)
{
Console.WriteLine("No service moved more than the thresholds. Check the budget alerts instead.");
return 0;
}
Console.WriteLine($"{"Service",-34} {"Baseline",10} {"Latest",10} {"Delta",10} {"Ratio",7}");
foreach (var f in findings)
Console.WriteLine($"{f.Service,-34} {f.Baseline,10:F2} {f.Latest,10:F2} {f.Delta,10:F2} {f.Ratio,7:F1}x");
return 1;
public sealed record Finding(string Service, decimal Baseline, decimal Latest, decimal Delta, decimal Ratio);
public static class SpikeDetector
{
// series: service name -> daily unblended cost, oldest first, newest LAST.
// The newest day is compared with the median of all earlier days.
public static List<Finding> Detect(IReadOnlyDictionary<string, decimal[]> series,
decimal minDailyIncrease, decimal minRatio)
{
var findings = new List<Finding>();
foreach (var (service, days) in series)
{
if (days.Length < 4) continue; // not enough history to call anything a spike
decimal latest = days[^1];
decimal baseline = Median(days[..^1]);
decimal delta = latest - baseline;
// A brand-new service has a zero baseline: treat any real spend as a spike.
decimal ratio = baseline == 0m ? (latest > 0m ? 999m : 0m) : latest / baseline;
if (delta >= minDailyIncrease && ratio >= minRatio)
findings.Add(new Finding(service, baseline, latest, delta, ratio));
}
return findings.OrderByDescending(f => f.Delta).ToList();
}
public static decimal Median(decimal[] values)
{
var sorted = values.OrderBy(v => v).ToArray();
int mid = sorted.Length / 2;
return sorted.Length % 2 == 1 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2m;
}
public static IReadOnlyDictionary<string, decimal[]> SampleData() => new Dictionary<string, decimal[]>
{
["Amazon Elastic Compute Cloud - Compute"] = [12.1m, 12.0m, 12.2m, 12.1m, 12.0m, 12.3m, 12.1m],
["Amazon Virtual Private Cloud"] = [1.4m, 1.4m, 1.5m, 1.4m, 1.4m, 1.5m, 38.9m],
["Amazon CloudWatch"] = [2.0m, 2.1m, 2.0m, 2.2m, 2.1m, 2.0m, 3.0m],
["Amazon Kinesis"] = [0m, 0m, 0m, 0m, 0m, 0m, 9.5m],
};
}
public static class CostFetcher
{
// Cost Explorer prices API requests per call, so keep this to one grouped query per run.
public static async Task<IReadOnlyDictionary<string, decimal[]>> DailyByServiceAsync(
IAmazonCostExplorer ce, int days)
{
// End is exclusive. Today's partial data is excluded by ending at today (midnight UTC).
var end = DateTime.UtcNow.Date;
var start = end.AddDays(-days);
var perService = new Dictionary<string, SortedDictionary<DateTime, decimal>>();
string? token = null;
do
{
var resp = await ce.GetCostAndUsageAsync(new GetCostAndUsageRequest
{
TimePeriod = new DateInterval { Start = start.ToString("yyyy-MM-dd"), End = end.ToString("yyyy-MM-dd") },
Granularity = Granularity.DAILY,
Metrics = new List<string> { "UnblendedCost" },
GroupBy = new List<GroupDefinition>
{
new() { Type = GroupDefinitionType.DIMENSION, Key = "SERVICE" }
},
NextPageToken = token
});
// AWS SDK for .NET v4 leaves collections null when empty, so guard every list.
foreach (var day in resp.ResultsByTime ?? new List<ResultByTime>())
{
var date = DateTime.ParseExact(day.TimePeriod.Start, "yyyy-MM-dd", CultureInfo.InvariantCulture);
foreach (var g in day.Groups ?? new List<Group>())
{
string service = g.Keys[0];
// Amounts are strings. Parse with the invariant culture, never the machine's.
decimal amount = decimal.Parse(g.Metrics["UnblendedCost"].Amount, CultureInfo.InvariantCulture);
if (!perService.TryGetValue(service, out var byDate))
perService[service] = byDate = new SortedDictionary<DateTime, decimal>();
byDate[date] = byDate.GetValueOrDefault(date) + amount;
}
}
token = resp.NextPageToken;
} while (!string.IsNullOrEmpty(token));
// Fill missing days with zero so every series has the same length and order.
var allDays = Enumerable.Range(0, days).Select(i => start.AddDays(i)).ToList();
return perService.ToDictionary(
kv => kv.Key,
kv => allDays.Select(d => kv.Value.GetValueOrDefault(d)).ToArray());
}
}Run it on the sample data:
dotnet run -- sampleExpected output:
Service Baseline Latest Delta Ratio
Amazon Virtual Private Cloud 1.40 38.90 37.50 27.8x
Amazon Kinesis 0.00 9.50 9.50 999.0xThe Amazon VPC line (where NAT gateway charges appear in some views) is the Harbourline scenario: the day after a gateway endpoint was removed, traffic to S3 flowed through the NAT gateway. Amazon CloudWatch moved from about 2.0 to 3.0, which is below both thresholds, so it is not flagged. ⚠ VERIFY which service name your own bill uses for NAT gateway charges (they often appear under "EC2 - Other").
Code explained line by line
- Modes.
sampleneeds no credentials and is the safe default;livecreates a Cost Explorer client inus-east-1(the Cost Explorer endpoint is global but served from that Region ⚠ VERIFY) and uses the profile you pass throughAWS_PROFILE. Detect. For each service, the newest day isdays[^1], and the baseline is the median of the earlier days. Two conditions must hold: the dollar increase is at leastminDailyIncrease(to ignore pennies) and the ratio is at leastminRatio(to ignore a large service wobbling by a small percentage).- Zero baseline. A service that had no spend and now has some gets ratio
999, so a brand-new, unexpected service is always reported. That is the case you most want to catch. Median. Sorts a copy and returns the middle value (or the average of the two middle values). A single earlier spike does not shift it the way an average would.DailyByServiceAsync. OneGetCostAndUsagerequest grouped bySERVICEatDAILYgranularity. The end date is exclusive and set to today's midnight so the partial, delayed current day is not used.NextPageTokenloop. Large accounts paginate; each extra page is another billed request, so keep the window short.- Null guards. In AWS SDK for .NET v4, list properties are not pre-initialised, so an empty result is
null, not an empty list. Guard every list you iterate. - Decimal parsing. Cost Explorer returns amounts as strings.
decimal.ParsewithCultureInfo.InvariantCultureavoids a comma-versus-dot bug on a machine with a different locale. - Zero-filling. Days with no cost for a service are absent from the response. Filling them with zero keeps all series aligned, so index
[^1]always means "yesterday". - Exit code.
1when something is flagged, so a scheduled job can fail visibly and notify the team.
Security considerations
⚠ Learning shortcut — not for production
The detector uses the caller's own credentials and prints to the console. In production, run it as a scheduled job with a dedicated read-only role limited to ce:GetCostAndUsage (and nothing else), send findings to a ticket or chat channel, and keep results in the same place as other operational evidence. Do not run Cost Explorer queries in a tight loop; each request is billed.
- A spike can be an attack. Unexpected compute in unused Regions, unusual data egress or a sudden rise in KMS or Bedrock requests can mean stolen credentials. Check GuardDuty findings and CloudTrail for
RunInstancesandCreateAccessKeybefore assuming a mistake. - Billing access is sensitive. Cost data reveals what the bank runs and where. Grant
ce:*andbudgets:*to a small set of roles, and use the management account's billing permissions sparingly. - Containment changes need approval. Stopping a service to save money can create an outage or a regulatory incident (for example, halting a payment batch). Production changes go through the change process with a named approver.
- Preserve evidence. Export the Cost Explorer views, the CloudTrail events and the resource configuration before you delete anything.
- Least privilege for the guardrails themselves. Budget actions that attach IAM policies or stop instances should be used in development accounts only, and be tested before you rely on them.
- No compliance claims. A budget alert is a financial control. It does not demonstrate compliance with any regulation.
Banking use case
The scenario: Harbourline's payments-dev account normally costs about USD 45 a day. On 12 November a developer removed the S3 gateway endpoint while testing a rollback and forgot to add it back. A nightly export job started reading the 400 GB statements dataset through the NAT gateway.
| Day | Cost (USD, fictional) | What changed |
|---|---|---|
| 10 Nov | 44 | Normal |
| 11 Nov | 46 | Normal |
| 12 Nov | 310 | Endpoint deleted; nightly job reads 400 GB through NAT |
| 13 Nov | 312 | Same |
| 14 Nov | 47 | Endpoint restored after the investigation |
The anomaly email arrived on the 13th. The investigation took 40 minutes: Account (payments-dev) → Service (EC2-Other) → Usage type (NatGateway-Bytes) → Resource (the dev VPC's NAT gateway, with BytesOutToDestination flat before the 12th and large after) → Cause (CloudTrail DeleteVpcEndpoints by the developer's role on the 12th). The data processing charge at the rates used elsewhere in this course (about USD 0.045 per GB ⚠ VERIFY) explains roughly USD 18 per run, so the larger jump included repeated runs and retries from a loop in the job. Containment was to add the gateway endpoint back through IaC with approval, and to set the job's concurrency to one.
Prevention: the endpoint is now part of the VPC module so deleting it needs a code change and review; a forecast budget at 120 percent of normal and a daily anomaly monitor for the Project=payments tag both email the team; and a Sunday review looks at the previous week's top five cost changes. The cost-incident record (about half a page) went to finance with the timeline above. Nothing in it names the developer; it is a blameless record of what the system allowed.
Common mistakes
- Fixing the symptom, not the cause. Deleting the NAT gateway stops the cost and the outbound traffic of every private workload with it. Find why the traffic uses NAT.
- Comparing different cost metrics. Unblended, amortized and net views disagree, especially around credits and commitments. Use the same metric as the alert.
- Trusting today's number. Billing data lags, so yesterday is the latest reliable day.
- Ignoring small, steady leaks. A USD 40 a month idle load balancer does not trigger an anomaly alert and never stops; find them by regular review.
- Skipping tags. Untagged spend cannot be attributed. Enforce tags in IaC and activate them for cost allocation.
- Using a budget as a kill switch. Budgets alert. They do not stop spending, and automated actions in production can cause an outage.
- Not checking for security. A bill spike is sometimes the first sign of stolen credentials.
- Running Cost Explorer queries in a loop. Each request costs money; the investigation tool should not become part of the bill.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Cost Explorer shows nothing for today | Billing data delay | Use yesterday and earlier; check service metrics for live data |
| Group by tag shows everything under "No tag" | The tag is not activated as a cost allocation tag, or was applied recently | Activate it in the management account; it applies going forward only |
| Resource IDs are missing in Cost Explorer | Resource-level data is an opt-in feature | Enable it ⚠ VERIFY, or use the Cost and Usage Report in Athena |
AccessDeniedException from ce:GetCostAndUsage | No Cost Explorer permission, or it was never enabled in the account | Grant the read-only cost policy; enable Cost Explorer from the Billing console |
| Anomaly email arrives a day late | Detection runs a few times a day on delayed data | Combine with a service-metric alarm for fast detection of known risks |
| The numbers differ between Cost Explorer and the invoice | Different cost metric, credits, tax, or the current month is still open | Compare the same metric and period; agree the finance view first |
| Budget alert never fires | Threshold type or notification type wrong (actual versus forecast), or email not confirmed | Re-check the notification type; confirm the subscription |
| The detector flags a service every Monday | A weekly pattern, not a spike | Compare with the same weekday, or use a longer baseline |
Cost and cleanup
Cost: free for reading and for the sample. The sample mode needs no AWS account. In live mode, a Cost Explorer API request costs about USD 0.01 ⚠ VERIFY, so running the detector once is a cent or two. Anomaly Detection itself is free; budgets are free for the first two action-enabled budgets per account, with a small daily charge beyond that ⚠ VERIFY. If you deploy the guardrail stack in the sandbox, tear it down the same day.
# Remove the demo budget created for this exercise.
aws budgets delete-budget --account-id 111122223333 --budget-name hb-dev-monthly \
--profile learn-sandbox --region us-east-1
# List and delete the demo anomaly subscription and monitor (ARNs come from the list commands).
aws ce get-anomaly-subscriptions --profile learn-sandbox --region us-east-1
aws ce delete-anomaly-subscription --subscription-arn <subscription-arn> \
--profile learn-sandbox --region us-east-1
aws ce get-anomaly-monitors --profile learn-sandbox --region us-east-1
aws ce delete-anomaly-monitor --monitor-arn <monitor-arn> \
--profile learn-sandbox --region us-east-1
# Or, if you used IaC:
cdk destroy SpikeGuardrailStack # or: terraform destroyConfirm nothing is left: Billing → Budgets and Cost Anomaly Detection show no demo entries.
Exercises
- 🟢 Guided: Run
dotnet run -- sample. Explain why Amazon CloudWatch (2.0 to 3.0) is not flagged and Amazon Kinesis is. ChangeminRatioto1.4and predict what changes before you run it. - 🟡 Challenge: Change the detector to compare the newest day with the median of the same weekday in the earlier weeks, so a weekly batch job is not a false alarm. You need 28 days of data; say what you do for a service that is new.
- 🔴 Live problem: The finance partner writes: "CloudWatch is USD 4,000 higher than last month and nobody knows why." Write the order of questions you ask, the three most likely usage types, which commands you run, what you would do to contain it, and what you tell finance by the end of the day. Do not delete any log group before step four.
Assignment
Create a cost-spike runbook for your portfolio: the narrowing funnel, a table translating usage types into likely causes for your stack, the exact CLI and console steps, a containment table with required approvals, a cost-incident record template and a Sunday review checklist. Include the output of your detector run on sample data.
Solutions
CloudWatch rose by 1.0 from a baseline of 2.05 (the median of 2.0, 2.1, 2.0, 2.2, 2.1, 2.0 is 2.05). The delta is below the dollar threshold of 5, so it is not flagged even though the ratio is about 1.5. Kinesis has a zero baseline and a latest value of 9.5, so its ratio is set to 999 and its delta is above 5: a brand-new service is always worth a look. If you set minRatio to 1.4 you still do not flag CloudWatch, because the delta (about 0.95) is below 5; both conditions must hold. The point of requiring both is to ignore pennies and ignore wobbles in large, steady services.
Group the daily values by DayOfWeek using the series dates (return the dates from the fetcher as well as the amounts). For the newest day, take the median of earlier days with the same weekday, which needs at least four earlier occurrences and so 28 days of data. If a service has fewer than four same-weekday points (it is new), fall back to the all-days median or flag it as "insufficient history" in a separate list, because a new service cannot be called normal. The reason: many bank jobs run on a weekly or monthly rhythm, and comparing Monday with Sunday produces alert fatigue that teaches people to ignore the report.
Knowledge check
Answer: Account, service, usage type, resource, then the person or change that caused it (CloudTrail and deployment history). Starting at the resource is slow because you do not yet know which one; starting at the invoice total shows you nothing about the cause.
Answer: No. Budgets send alerts. Budget actions can attach policies or stop some resources, but only if you configure them, with delay and with the risk of an outage. Production should alert people instead.
Answer: A NAT gateway data processing usage type (NatGateway-Bytes, typically shown under EC2-Other). S3 and DynamoDB traffic that used a free gateway endpoint now goes through the NAT gateway, which charges per GB processed. Idle-hours or storage usage types would not move that suddenly.
Answer: Stolen credentials are often used to start compute (including in unused Regions) or to move data out, and the first visible sign is the bill. Check GuardDuty, CloudTrail RunInstances and egress before you assume a mistake.
Answer: Set its reserved concurrency to 0, which blocks new invocations and keeps the function, configuration and logs. Remove the setting after the trigger is fixed. Deleting loses evidence and configuration; lowering memory does not stop the loop.
Interview questions
- Your AWS bill doubled overnight. What do you do? Confirm the metric and period, narrow by account, service, usage type and resource, check the shape over time, find the change in CloudTrail and the deployment history, contain safely with approval, then set budgets, anomaly monitors and tags to catch the next one.
- What are the most common causes of unexpected AWS cost? NAT gateway data processing, forgotten idle resources, log ingestion, loops and retries, cross-AZ transfer, KMS requests, large batch jobs, and, in some cases, a security incident.
- How do budgets differ from Cost Anomaly Detection? A budget compares against a number you set and can forecast; anomaly detection learns the usual pattern and flags unusual change. A slow 30 percent overspend trips the budget, a sudden spike inside the budget trips the monitor.
- How would you stop an expensive resource in production? Follow the change process: confirm the dependency, choose a reversible action (for example concurrency to 0, a scheduled stop, a tag with a delete date), record who approved it, keep the evidence.
- How do you make cost visible to engineers? Tag in IaC, activate cost allocation tags, give each team a budget and monitor, publish a weekly top-movers view and put cost in the design review.
Certification practice
A company wants an email when the forecasted cost for a month is expected to exceed a set amount. Which service should it use?
A. AWS Trusted Advisor B. AWS Budgets C. AWS CloudTrail D. AWS Config
Answer: B. AWS Budgets supports forecast-based alerts. Trusted Advisor gives recommendations, CloudTrail records API calls, and Config records configuration.
Monthly cost for an account suddenly rises. Cost Explorer grouped by service shows "EC2 - Other" as the cause. Which next step best identifies the exact driver?
A. Group by usage type and filter to EC2 - Other. B. Delete all EC2 instances. C. Switch to a Savings Plan. D. Disable Cost Explorer.
Answer: A. Usage type separates NAT gateway bytes, data transfer, EBS and similar meters. Deleting resources (B) is destructive and unfocused, a Savings Plan (C) changes price not cause, and disabling the tool (D) removes visibility.
Private workloads read large volumes from S3 through a NAT gateway, producing high data processing charges. Which design change most directly reduces them?
A. Move the workloads to public subnets. B. Add an S3 gateway VPC endpoint and route table entries. C. Replace the NAT gateway with a larger instance. D. Enable S3 Transfer Acceleration.
Answer: B. A gateway endpoint carries S3 traffic privately with no hourly or per-GB endpoint charge, removing the NAT data processing for that traffic. Public subnets (A) weaken security, NAT gateways are not sized as instances (C), and Transfer Acceleration (D) is for long-distance uploads and adds cost.
Completion checklist
- I can recite the five-step narrowing order and the tool for each
- I can match three usage types to likely causes
- I ran the detector in sample mode and explained each result
- I know a safe containment action for a loop, an idle resource and a noisy log
- I can say why a spike may be a security event
- I ran the cleanup steps and confirmed no demo budget or monitor remains
Key takeaways
- Narrow in order: account, service, usage type, resource, cause.
- The shape of the daily graph (step, ramp, spike) tells you what to look for.
- NAT gateway bytes, idle resources, log ingestion and loops explain most spikes.
- Contain with reversible, approved changes and keep the evidence.
- Budgets and anomaly monitors alert; only people and design decisions prevent repeats.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Playbook: who accessed this customer record?
Answer a data protection team's access question with evidence: scope the request, pull application audit events, CloudTrail data events and database audit logs, join them by AWS request ID, find the access that bypassed the application, and hand over a hashed, reviewable report.
Interview preparation: cloud engineer
Prepare for a cloud engineer interview by turning your .NET and Azure experience into AWS stories, rehearsing a core bank of IAM, networking, operations and troubleshooting answers, and practising how to think aloud when you do not know the answer.