CloudWatch metrics and custom business metrics with the Embedded Metric Format
Understand namespaces, dimensions, units, statistics and resolution, publish a business metric (failed transfers per minute) from C# as an Embedded Metric Format log line, and price the result before you deploy it.
Assumes: .NET 10 LTS · AWSSDK.CloudWatch 4.1.1.2 · Embedded Metric Format specification (checked 2026-11-18) · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Describe a CloudWatch metric by its namespace, name, dimensions, unit, statistic and period, and explain what a data point is.
- Choose between
PutMetricData, the Embedded Metric Format (EMF) and OpenTelemetry metrics for a custom business metric. - Write a valid EMF document from C# (the lab's
EmfWriter), including multiple dimension sets, and explain the limits the specification imposes. - Predict how many custom metrics (and how much money) a design creates, and spot a dimension that will explode the bill.
- Read a metric back with the CLI and the .NET SDK.
Prerequisites
- Observability fundamentals: cardinality and golden signals.
- CloudWatch Logs for .NET: EMF travels inside a log line, so the log pipeline must already work.
- Tools: AWS CLI v2 (
learn-sandbox), .NET 10 SDK or Docker.
Why this matters in banking
A bank's service-level questions are business questions: "How many transfers failed in the last five minutes?", "Is the payment p99 above our promise?" AWS publishes technical metrics for you (CPU, request count, queue depth), but not "transfers by outcome". If the only way to know the failed-transfer rate is to scan logs, your alarm is slow and expensive. If you instead publish a business metric at the moment each transfer completes, the alarm is cheap, fast and precise.
The danger is the other direction. A well-meaning engineer adds customerId as a dimension "so we can see per-customer error rates". CloudWatch creates one metric per distinct value, a million customers become a million custom metrics, and the next invoice is a surprise. Knowing how metrics are identified and priced is what separates a useful metric from a costly one.
The idea in plain language
A metric is a time-ordered series of numbers. Each number is a data point. CloudWatch identifies one metric by:
| Part | Meaning | Example |
|---|---|---|
| Namespace | A container that keeps your metrics apart from AWS's (AWS/... is reserved) | Harbourline/Payments |
| Metric name | What is measured | TransferCount |
| Dimensions | Up to 30 name/value pairs that identify which series | Service=transfers-api, Environment=learn, Outcome=failed |
Two metrics with the same name but different dimension values are different metrics. A metric with dimensions {Service} and the same metric with {Service, Outcome} are also different metrics, and you only get data in the combinations you published.
Other properties:
- Unit (
Count,Milliseconds,Bytes,Percent,None, and others): documentation and display only; CloudWatch does not convert between them, but an alarm or graph can filter by unit. - Statistic: how data points in one period are combined:
Sum,Average,Minimum,Maximum,SampleCount, and percentiles such asp99. For a counter likeTransferCountyou want Sum; for latency you want p95/p99 (andAverageonly as a trend). - Period: the aggregation window, in seconds (multiples of 60 for standard metrics).
- Resolution: standard (one data point per minute) or high resolution (down to one second, billed higher ⚠ VERIFY). Use high resolution only when you genuinely need sub-minute detection, such as fast-moving payment rails.
CloudWatch keeps metric data for a long time but at decreasing resolution: one-minute data for about 15 days, five-minute data for about 63 days and one-hour data for about 15 months ⚠ VERIFY the exact periods on the CloudWatch concepts page. Sub-minute data is kept for a short period only (a few hours ⚠ VERIFY).
Analogy: A metric is a line on the wall chart in the branch manager's office: "customers waiting" measured every minute. The namespace is which chart book it lives in, the dimensions say which branch and which counter, and the statistic says whether you are plotting the total, the busiest moment, or the slowest 1%. Adding a dimension per customer would be like keeping one wall chart per customer.
Three ways to publish a custom metric
| Method | How it works | Good for | Watch out for |
|---|---|---|---|
PutMetricData API | Your code calls CloudWatch with data points | Batch jobs, scripts, low-volume metrics | An API call (latency, IAM permission, request charge ⚠ VERIFY) in your code path; lost on crash |
| Embedded Metric Format (EMF) | You write a JSON log line with a metadata block; CloudWatch Logs extracts the metrics | High-volume app metrics from ECS, Lambda, EC2 | Needs a Standard-class log group ⚠ VERIFY; log ingestion is billed as well as the metric |
| OpenTelemetry metrics (OTLP) | The OTel SDK exports metrics to a collector or CloudWatch's OTLP endpoint | One instrumentation standard across signals | Newer path; check the current availability, pricing and PromQL-based querying before choosing ⚠ VERIFY |
This lesson focuses on EMF because it needs no extra permissions or calls: the line is written to stdout like any log, and CloudWatch does the rest.
What problem the AWS service solves
CloudWatch metrics give you cheap, fast, aggregated numbers you can graph and alarm on. Service-published metrics are free (or included) for most AWS services; custom metrics are billed.
Prices checked on the CloudWatch pricing page on 2026-11-18 (US East N. Virginia examples; prices vary by Region ⚠ VERIFY before budgeting):
| Item | Price |
|---|---|
| Custom metrics, first 10,000 | USD 0.30 per metric per month |
| Custom metrics, next 240,000 | USD 0.10 per metric per month |
| Free tier | 10 custom metrics and 1 million API requests per month |
| Standard alarm | USD 0.10 per alarm metric per month |
| Log ingestion (Standard class) | USD 0.50 per GB |
The EMF specification's rules (checked on 2026-11-18). These explain most "my metric didn't appear" problems:
- The log event must be a single valid JSON object, with nothing before or after it, and an event is limited to the standard log-event size (the specification says 1 MB).
- The
_awsobject carries aTimestamp(milliseconds since the Unix epoch) and aCloudWatchMetricsarray of metric directives. The timestamp must be no more than 14 days in the past and no more than 2 hours in the future; otherwise CloudWatch silently does not publish the metric. - Each directive has a
Namespace(1 to 255 printable ASCII characters),Dimensions(an array of dimension sets) andMetrics. - A dimension set holds at most 30 keys, and may be empty. Every dimension set creates its own metric; the spec warns that unintentional high-cardinality dimensions such as a request ID will create a metric for every value.
- A log event holds at most 100 metric definitions in total across directives, and a metric value array at most 100 members.
- A metric definition has a
Name, an optionalUnit(defaultNone) and an optionalStorageResolutionof 1 (high) or 60 (standard, the default). - Metric values and dimension values live at the root of the JSON; they cannot be nested objects.
- Delivery is at least once, so a duplicate data point can occasionally occur: do not build billing-grade counts on it.
- Failures of EMF processing are reported as metrics in the
AWS/Logsnamespace of your account, so you can alarm on "EMF stopped working".
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
TelemetryClient.GetMetric("Transfers").TrackValue(1) | An EMF line (or PutMetricData) | No client object: write a JSON line; no SDK call in the request path |
| Custom metrics with dimensions (Azure Monitor custom metrics) | CloudWatch custom metrics with dimensions | Each dimension combination is a separately billed metric |
Metric namespaces such as Microsoft.Web/sites | Namespaces AWS/ApplicationELB for AWS, your own Harbourline/Payments | Pick a company-wide convention early; namespaces cannot be renamed |
| Log-based metrics in Application Insights | Metric filters (pattern on a log group) or EMF | Metric filters count matching lines; EMF carries the value in the line |
| Aggregation types (Avg, Min, Max, Sum, Count) | Statistics (Average, Minimum, Maximum, Sum, SampleCount, percentiles) | Percentiles are first-class statistics |
| Dimension limits per metric | Up to 30 dimensions per metric | The limit that matters is cost, not the count |
Architecture
Security-boundary view. The application needs no CloudWatch permission to publish EMF metrics, because it only writes to stdout; the log pipeline's role does the rest. That is a real security advantage over PutMetricData, which needs cloudwatch:PutMetricData in the application role (and that action cannot be restricted to a specific metric name; you can restrict it by namespace with the cloudwatch:namespace condition key). Anyone who can read the log group can also read the metric values embedded in the lines, so the same data-minimisation rules apply: counts and timings are fine, customer identifiers are not.
How the lab emits its metrics
BankingMetrics.RecordTransfer writes one EMF line per transfer with two metrics (TransferCount, TransferLatency) under two dimension sets: [Service, Environment] and [Service, Environment, Outcome]. A real line from the lab's container (reformatted):
{
"_aws": {
"Timestamp": 1794988813199,
"CloudWatchMetrics": [
{
"Namespace": "Harbourline/Payments",
"Dimensions": [["Service", "Environment"], ["Service", "Environment", "Outcome"]],
"Metrics": [
{ "Name": "TransferCount", "Unit": "Count" },
{ "Name": "TransferLatency", "Unit": "Milliseconds" }
]
}
]
},
"Service": "transfers-api",
"Environment": "learn",
"Outcome": "failed",
"TransferCount": 1,
"TransferLatency": 29.8723,
"correlationId": "01a1edab-ab5a-7c55-bda9-d95e2a2212e4",
"scheme": "FPS",
"level": "Metric"
}Everything outside _aws is a normal log field. The dimension values (Service, Environment, Outcome) and the metric values (TransferCount, TransferLatency) are root members the directive points at. correlationId and scheme are properties: searchable in Logs Insights, but not dimensions, so they create no metrics and cost nothing extra.
Counting the metrics (and the cost)
Per metric name, the two dimension sets produce these series:
| Dimension set | Series | Count |
|---|---|---|
[Service, Environment] | one, all outcomes combined | 1 |
[Service, Environment, Outcome] | one per outcome: succeeded, rejected, failed | 3 |
Two metric names × 4 series = 8 custom metrics for this one service. At USD 0.30 per metric per month the list-price cost is USD 2.40, and after the free tier of 10 custom metrics, zero for the first service ⚠ VERIFY the free-tier applicability to your account. Now add scheme (5 values) as a third dimension: the [Service, Environment, Outcome, Scheme] set adds 15 series per name, so 30 more metrics. The lesson: each dimension set multiplies by the number of value combinations, so add dimension sets deliberately and only when an alarm or dashboard needs that breakdown.
Rule of thumb: a dimension is justified only if you will alarm or graph on it. If you only need to find things, make it a log property.
Step-by-step implementation
Name the metric and its question
"How many transfers fail per minute?" gives TransferCount with an Outcome dimension. "How slow is the tail?" gives TransferLatency with a p99 statistic. Write the question next to the name in code.
Fix the vocabulary
List every legal value of each dimension. The lab's Outcomes array has three entries, and RecordTransfer throws for anything else. That check is what keeps a typo (or an injected value) from creating new billed metrics.
Build the EMF document with validation
EmfWriter.Build checks the namespace, dimension count (30), metric count (100), the unit list, resolution (1 or 60) and that values are finite numbers. Failing in code is cheaper than silently losing data in CloudWatch.
Write the line to stdout, not through the logger
Use Console.Out.WriteLine (the lab's StdoutMetricSink). A logging framework would wrap the line in its own envelope and break the "pure JSON object" rule. Keep the logger for human-oriented events.
Check the timestamp source
Use the event's own time from TimeProvider. A host whose clock is more than two hours ahead, or a buffered replay older than 14 days, produces metrics CloudWatch quietly drops.
Verify in the console and the CLI
After the first line arrives allow a minute or two. Find the metric under CloudWatch → Metrics → All metrics → Harbourline/Payments, and read it back with the CLI. Also check the AWS/Logs namespace for EMF failure metrics.
Price it before you deploy
Count the series (dimension sets × value combinations × metric names), multiply by the price, subtract the free tier, and write the number in the pull request.
AWS Console instructions
Browse metrics: Console → CloudWatch → Metrics → All metrics. Choose the custom namespace Harbourline/Payments, then a dimension combination (for example Service, Environment, Outcome), tick the metrics and use the Graphed metrics tab to set the statistic (Sum for counts, p99 for latency) and period (one minute).
Find EMF problems: in All metrics choose the AWS/Logs namespace and look for the EMF failure metrics (names such as parsing or validation failures; see the EMF monitoring documentation ⚠ VERIFY exact names).
Remember that a metric appears only after its first data point, and that custom metrics cannot be deleted by you: they simply age out when no data arrives ⚠ VERIFY.
AWS CLI
# List the metrics in the namespace: this shows you the series you are paying for
aws cloudwatch list-metrics --namespace Harbourline/Payments --profile learn-sandbox \
--query 'Metrics[].[MetricName,Dimensions[].Value|join(`,`,@)]' --output table
# Sum of failed transfers per minute for the last 15 minutes
aws cloudwatch get-metric-statistics --profile learn-sandbox \
--namespace Harbourline/Payments --metric-name TransferCount \
--dimensions Name=Service,Value=transfers-api Name=Environment,Value=learn Name=Outcome,Value=failed \
--start-time "$(date -u -d '-15 minutes' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
--period 60 --statistics Sum
# p99 latency, using an extended statistic
aws cloudwatch get-metric-statistics --profile learn-sandbox \
--namespace Harbourline/Payments --metric-name TransferLatency \
--dimensions Name=Service,Value=transfers-api Name=Environment,Value=learn \
--start-time "$(date -u -d '-15 minutes' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
--period 60 --extended-statistics p99
# Publish one data point by API (the alternative to EMF)
aws cloudwatch put-metric-data --namespace Harbourline/Payments --metric-name TransferCount \
--dimensions Service=transfers-api,Environment=learn,Outcome=failed --value 1 --unit Count --profile learn-sandboxInfrastructure as Code
Metrics created by EMF need no resource. What you manage as code is what reads them: alarms and dashboards (lessons 5 and 6) and, optionally, a log-group metric filter for lines you cannot change.
using Amazon.CDK.AWS.Logs;
// A metric filter counts log lines that match a JSON pattern: handy for code you cannot edit.
_ = new MetricFilter(this, "ServerErrors", new MetricFilterProps
{
LogGroup = logGroup,
MetricNamespace = "Harbourline/Payments",
MetricName = "ServerErrorLines",
FilterPattern = FilterPattern.Literal("{ $.statusCode >= 500 }"),
MetricValue = "1",
DefaultValue = 0,
});A metric filter counts lines; EMF carries values (latency, amounts) and several metrics per line. Prefer EMF for code you own.
Complete code
The full writer is labs/m11-observe-banking-api/src/Harbourline.Observability/Metrics/EmfWriter.cs. The call site in the API is one line per outcome:
metrics.RecordTransfer("failed", ElapsedMs(), correlationId, "FPS");BankingMetrics builds the document:
public void RecordTransfer(string outcome, double latencyMs, string correlationId, string scheme)
{
if (!Outcomes.Contains(outcome))
throw new ArgumentException($"Outcome must be one of: {string.Join(", ", Outcomes)}.", nameof(outcome));
var doc = EmfWriter.Build(
Namespace,
DimensionSets, // [[Service, Environment], [Service, Environment, Outcome]]
new Dictionary<string, string> { ["Service"] = _service, ["Environment"] = _environment, ["Outcome"] = outcome },
[new EmfMetric(TransferCount, 1, "Count"), new EmfMetric(TransferLatency, latencyMs, "Milliseconds")],
new Dictionary<string, object?> { ["correlationId"] = correlationId, ["scheme"] = scheme, ["level"] = "Metric" },
_clock.GetUtcNow());
_sink.Emit(doc);
}The API-based alternative, for a script or batch job (compiles against AWSSDK.CloudWatch 4.1.1.2):
using Amazon.CloudWatch;
using Amazon.CloudWatch.Model;
using var cw = new AmazonCloudWatchClient();
await cw.PutMetricDataAsync(new PutMetricDataRequest
{
Namespace = "Harbourline/Payments",
MetricData =
[
new MetricDatum
{
MetricName = "TransferCount",
Dimensions =
[
new Dimension { Name = "Service", Value = "transfers-api" },
new Dimension { Name = "Environment", Value = "learn" },
new Dimension { Name = "Outcome", Value = "failed" },
],
Unit = StandardUnit.Count,
Value = 1,
Timestamp = DateTime.UtcNow,
StorageResolution = 60,
},
],
});
// Read it back: Sum per minute over the last 15 minutes.
var stats = await cw.GetMetricStatisticsAsync(new GetMetricStatisticsRequest
{
Namespace = "Harbourline/Payments",
MetricName = "TransferCount",
Dimensions =
[
new Dimension { Name = "Service", Value = "transfers-api" },
new Dimension { Name = "Environment", Value = "learn" },
new Dimension { Name = "Outcome", Value = "failed" },
],
StartTime = DateTime.UtcNow.AddMinutes(-15),
EndTime = DateTime.UtcNow,
Period = 60,
Statistics = [Statistic.Sum],
});
foreach (var point in stats.Datapoints.OrderBy(p => p.Timestamp))
Console.WriteLine($"{point.Timestamp:HH:mm} sum={point.Sum}");Code explained line by line
Outcomes.Contains(outcome)is the cardinality guard. Without it, a code path that passedex.Messageas the outcome would create a new billed metric per exception message.DimensionSetsis a list of lists. Each inner list is one combination CloudWatch publishes. Both inner lists nameOutcomeor not; the dimension values are supplied once, at the root.new EmfMetric(TransferCount, 1, "Count")records a data point of 1 per transfer. CloudWatch sums the ones per minute, giving transfers per minute. (If you had one line per minute with a pre-aggregated count, you would publish that count instead.)latencyMsis a continuous value; per minute CloudWatch computes min, max, sum, count and percentile estimates from the values received, so p99 works with no extra effort.- The
propertiesdictionary becomes ordinary JSON fields.level = "Metric"tags EMF lines so queries for errors can ignore them. _clock.GetUtcNow()supplies_aws.Timestampin milliseconds. UsingTimeProvidermakes the output deterministic in tests (the lab'sEmfWriterTestsuse a fixed instant).- In the SDK sample,
StandardUnit.CountandStatistic.Sumare constant classes in SDK v4, andTimestamp,StartTimeandEndTimeareDateTimevalues in UTC. The collection initialisers ([...]) rely on the SDK v4 list properties being initialised for you.
Security considerations
⚠ Learning shortcut — not for production
The lab runs under one sandbox identity, and put-metric-data is shown with full access to the sandbox account. In production, give an application that must call the API only cloudwatch:PutMetricData with a cloudwatch:namespace condition for its own namespace, and prefer EMF so the application needs no CloudWatch permission at all.
- Metric data is not secret but it is still data. A dimension such as
Outcome=failedis harmless;customerIdis not. Keep identifiers out of metrics. - Control cost as a security issue. A bug or attacker that causes unbounded distinct dimension values (for example from user input) creates unbounded metrics. Validate dimension values against a fixed list; the lab's
BankingMetricsdoes. - IAM scope for
PutMetricData. The action does not support resource-level restrictions, but thecloudwatch:namespacecondition key does limit which namespace the caller may write to. Check the current condition keys ⚠ VERIFY. - Who can read.
cloudwatch:GetMetricData,GetMetricStatisticsandListMetricsare the read actions. Give them to operations roles. - Alarms depend on honest metrics. If an application can falsify its own counts, alarms can be silenced. For safety-critical signals use a measurement taken outside the application (load balancer metrics, canaries) as well.
Banking use case
Harbourline's payments service publishes TransferCount by Outcome and TransferLatency for each transfer. Operations alarm on Outcome=failed (more than five per minute in three of five minutes) and on total traffic falling below one per minute during business hours, and page only when customers are affected. The business dashboard graphs rejected separately: a spike in rejections is often a client-side deployment bug, not a platform incident, and it should reach the product team, not the infrastructure on-call. The settlement scheme (FPS, BACS, CHAPS) is a log property: operations can filter by it in Logs Insights when needed, without paying for 15 extra metrics. Data is fictional.
Common mistakes
- High-cardinality dimensions (
customerId,paymentId,scheme×outcome×endpointall together). Fix: fixed vocabularies; properties for the rest. - Wrapping EMF in the logger. The envelope breaks extraction. Fix: write the raw JSON line to stdout.
- Wrong statistic. Averaging a counter hides spikes. Fix:
Sumfor counts,p95/p99for latency. - Mismatched dimension values between the code and the alarm (
transfers-apivstransfer-api). Fix: one constant shared by the app and the infrastructure code, as the lab's CDK stack mirrors the C# names. - Timestamps outside the window. Replaying old logs produces nothing. Fix: use the event time and check it against the 14-day and 2-hour limits.
- Using
PutMetricDatain a hot path. It adds latency and a permission. Fix: EMF. - Expecting instant data. Metrics lag a minute or more. Fix: design alarms with that lag in mind.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Metric never appears | The line is not a single JSON object, the _aws block is invalid, or the timestamp is out of range | Run aws logs tail and inspect the line; check AWS/Logs EMF failure metrics |
| Metric appears under the wrong dimensions | Dimension names not in the directive, or values missing at the root | Compare the Dimensions array with the root keys |
get-metric-statistics returns no data points | Dimension values differ, period too short, or time range wrong | List metrics first and copy the exact dimension values |
| Bill is high | A high-cardinality dimension or too many dimension sets | list-metrics the namespace and count the series |
| Duplicate counts | EMF delivery is at least once | Do not use for billing; use alarm thresholds that tolerate small duplication |
| Percentile statistic missing | Percentiles need extended statistics, not --statistics | Use --extended-statistics p99 |
Cost and cleanup
At lab volume (tens of transfers) the data cost is negligible. The custom metrics: eight series, USD 2.40 per month at list price, and inside the free tier of 10 custom metrics if nothing else in the account uses it ⚠ VERIFY. Log ingestion for the lines is a few kilobytes. Metrics cannot be deleted; they stop being billed when no data arrives (billing is per metric that received data in the month ⚠ VERIFY), so the clean-up is to stop publishing and delete the log group.
# Stop publishing: destroy the stack and the log group
cd labs/m11-observe-banking-api/infra/cdk && cdk destroy --profile learn-sandbox
aws logs describe-log-groups --log-group-name-prefix /hl-m11 --profile learn-sandbox
aws cloudwatch list-metrics --namespace Harbourline/Payments --profile learn-sandbox # metrics age out on their ownExercises
- 🟢 Guided: Run the lab's
EmfWriterTestsandBankingMetricsTests. Then read the EMF line the container writes (docker compose up,scripts/load.sh,docker compose logs api) and identify: the namespace, both dimension sets, the metric names, one property. - 🟡 Challenge: Add a metric
TransferAmountGbp(unitNone, sum per minute) toBankingMetricsfor successful transfers only, with a unit test. Count the new metric series and the monthly list cost. - 🔴 Live problem: The billing alert shows custom metrics jumped from 8 to 40,000 overnight. A release added
endpoint(taken from the raw request path, which contains account IDs) as a dimension. Write the response: how you confirm the cause withlist-metrics, how you fix the code, what happens to the metrics already created, how you stop it happening again (tests, review rule, an alarm on theAWS/Usageor billing metrics ⚠ VERIFY), and what you tell finance.
Assignment
Add EMF business metrics to a service of your own: a counter by outcome and a latency value, with a validated vocabulary and unit tests that assert the document structure. Document the metric series count and the list price in the README, and add one alarm definition (CDK or Terraform) that reads one of the metrics.
Solutions
Namespace Harbourline/Payments; dimension sets [Service, Environment] and [Service, Environment, Outcome]; metrics TransferCount (Count) and TransferLatency (Milliseconds); properties correlationId, scheme, level. The dimension values (transfers-api, learn, failed) sit at the root next to the metric values. If you can point to each, you can debug any EMF line.
In RecordTransfer, add a parameter for the amount and new EmfMetric("TransferAmountGbp", amount, "None") when outcome == "succeeded". The metric then sits in the same directive and is published under both dimension sets, so there are up to 4 new series (1 for [Service, Environment] and 3 for the Outcome set). Only succeeded ever carries the amount, so in practice 2 series receive data, and billing follows series that receive data ⚠ VERIFY. List cost: 2 to 4 series × USD 0.30 = USD 0.60 to 1.20 per month. In a unit test, record a succeeded transfer and assert the document contains TransferAmountGbp equal to the amount, and that a rejected transfer does not contain the property. Never put the amount in a dimension. This sum is an operational indicator, not a ledger.
Confirm with aws cloudwatch list-metrics --namespace ... --query and count series per dimension name; the new endpoint dimension will dominate. Fix by removing the dimension (or replacing it with the route template such as /v1/accounts/{id}, a fixed small set) and deploy. Existing metrics cannot be deleted; they stop accruing cost after they stop receiving data and age out ⚠ VERIFY. Prevent: unit tests asserting the dimension vocabulary (like Unknown_outcomes_are_refused...), a review rule "no request-derived dimension values", and budget or AWS/Usage alarms. Tell finance the cause, the cost incurred, the fix date and what stops recurrence. This is a classic incident where the cheapest control is a test.
Knowledge check
Answer: a different namespace, metric name or any difference in the set of dimension names and values. Each unique combination is a separate (billed) metric.
Answer: Sum. Average of 1s is always 1; Sum gives failures per minute.
Answer: it only writes a log line to stdout; the log pipeline (the execution role) delivers it to CloudWatch Logs, where the EMF extraction happens. No PutMetricData call is made by the app.
Answer: CloudWatch does not publish the metrics. The specification accepts timestamps up to 14 days in the past and 2 hours in the future.
Answer: 8. For each name: 1 series for [A,B] and 3 for [A,B,C], so 4; times 2 names is 8.
Interview questions
- How do you publish a business metric from ECS without an SDK call? Write an EMF JSON line to stdout; the
awslogsdriver ships it and CloudWatch Logs extracts the metrics. - What is a dimension and why is cardinality a cost issue? A dimension is a name/value pair that identifies a series; each combination is a separate custom metric billed monthly, so unbounded values mean unbounded cost.
- EMF versus a metric filter? EMF carries numeric values and multiple metrics in a line you control; a metric filter counts or extracts values from lines you cannot change.
- How would you alarm on latency? Use p95 or p99 over a one-minute period with M-out-of-N evaluation, not the average.
- What can go wrong with EMF? Invalid JSON or wrapped lines, timestamps outside the window, mismatched dimension values, and unbounded dimensions.
Certification practice
A Lambda function emits EMF logs, but the custom metric never appears in CloudWatch. The log lines are visible. Which is the most likely cause?
A. The log group has no retention.
B. The JSON is malformed or the _aws metadata block is invalid, so extraction fails.
C. The Lambda memory is too low.
D. CloudTrail is off.
Answer: B. EMF requires a valid single JSON object with a correct _aws block; otherwise the line stays an ordinary log. A, C and D do not affect extraction.
A team adds the user ID as a dimension on a request-count metric, and the CloudWatch bill rises sharply. What is the root cause and fix?
A. Standard alarms are expensive; delete them. B. Each distinct user ID creates a separate custom metric; remove the dimension and log the ID instead. C. High-resolution metrics are on by default; disable them. D. The metric namespace is too long.
Answer: B. Dimension values multiply the number of billed metrics. Keep identifiers in logs or traces.
Which approach publishes high-volume application metrics from containers with the least latency and the least IAM exposure?
A. Call PutMetricData from every request.
B. Write Embedded Metric Format JSON to stdout and let the log driver ship it.
C. Poll the application's /metrics endpoint from a Lambda every second.
D. Store counts in DynamoDB and read them with a script.
Answer: B. No call in the request path and no CloudWatch permission in the app. A adds latency and a permission; C and D are slower, costlier and more complex.
Completion checklist
- I can name the parts that identify a CloudWatch metric
- I can write (or read) a valid EMF document and name its limits
- I counted the series and the list cost for the lab's metrics
- I read a metric back with the CLI
- I stopped publishing and removed the lab's log group
Key takeaways
- A metric is identified by namespace, name and the exact dimension combination; each combination is billed.
- EMF turns a log line into metrics with no API call and no application permission, if the line is pure JSON and the timestamp is in range.
- Dimensions are for breakdowns you will alarm or graph on; everything else is a property.
- Use
Sumfor counters and percentiles for latency. - Price the series count before deploying; write the number in the pull request.
Further reading
- Specification: Embedded metric format: the source of the limits quoted here.
- Amazon CloudWatch pricing: custom metric, alarm and log prices.
- Alarms and composite alarms: the next lesson.
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Logs Insights queries: five saved queries that answer real incident questions
Write CloudWatch Logs Insights queries for the questions an on-call engineer asks during an incident (what happened to this request, are customers failing, where is it slow), save them, run them from the CLI and from C#, and keep their cost under control.
CloudWatch alarms and composite alarms: thresholds, missing data and noise reduction
Design alarms that page a human only when customers are affected: choose statistics and periods, use M-out-of-N evaluation, pick the right missing-data policy, add metric math and anomaly detection where they fit, and combine alarms into one composite page.