Log aggregation and error monitoring across accounts
Centralise logs from many AWS accounts without copying everything twice, group thousands of exception lines into a handful of distinct errors, and route the right alerts to the right people with severity-based paths.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Choose between four ways of bringing logs together: cross-account observability, log centralization rules, subscription filters and S3 export, and say what each costs and cannot do.
- Pick the right CloudWatch Logs class (Standard or Essentials, formerly Infrequent Access) for each log group, knowing which features you lose with Essentials.
- Turn a flood of exception lines into a short list of distinct errors with a stable fingerprint, Logs Insights
patternqueries and metric filters. - Design alert routing by severity so that pages are rare and tickets are common, using composite alarms and SNS.
- Build an observability access manager sink and link in CDK and Terraform.
Prerequisites
- The earlier Module 11 lessons: CloudWatch Logs for .NET, Logs Insights queries and Alarms and composite alarms.
- CloudTrail for operations, because audit logs are part of what you aggregate.
- Organisations and multi-account design is helpful background; you need to know what an organisation, a delegated administrator and a log-archive account are.
- Tools: AWS CLI v2 with the
learn-sandboxSSO profile, .NET 10 SDK. A true cross-account lab needs a second account; the free path below works with one.
Why this matters in banking
Harbourline runs payments, accounts, notifications and the data platform in separate AWS accounts, because blast radius and separation of duties matter. That is good design and awful for troubleshooting: a failed transfer touches three accounts, and the engineer on call has to log into each one, remember which log group is where, and run the same query three times.
Two opposite failures follow. Either logs stay scattered and incidents take twice as long, or someone copies everything into one big place, forgets retention, and ends up with a seven-figure logging bill containing every customer's IBAN. A good aggregation design gives responders one place to search, keeps raw logs in the account that produced them where possible, and sends only what is needed to the long-term archive.
Error monitoring has the same shape. A single bad deployment can produce 400,000 identical exception lines in ten minutes. A pager that goes off 400,000 times is useless. You want one alert that says "new error, 400,000 occurrences, started 14:02, first seen in release 2026.11.18".
The idea in plain language
Aggregation answers "where do I look?". Error monitoring answers "what is actually going wrong, and who needs to know?".
For aggregation, think of a bank with many branches. You can (a) give head office a window into each branch's filing cabinet (nothing copied), (b) have each branch photocopy every document to head office (duplication, cost, but one cabinet), (c) have branches mail a copy of specific documents to an external auditor, or (d) box up old files and archive them. CloudWatch offers all four.
For error monitoring, the key idea is fingerprinting: reduce each error to a short, stable identity (exception type plus the place it came from), ignoring things that vary (ids, timestamps, amounts). Group by fingerprint and the 400,000 lines become one error with a count.
Analogy: a hospital triage desk. Ten patients with the same broken arm do not each get a separate doctor's alarm. They are grouped, ranked by severity, and the right specialist is called once.
What problem the AWS service solves
Four ways to aggregate logs
| Approach | What it does | Copies data? | Good for | Sharp edges |
|---|---|---|---|---|
| Cross-account observability (Observability Access Manager, OAM) | A monitoring account queries logs, metrics, traces, Application Signals services and SLOs that live in source accounts | No, it reads in place | Live troubleshooting across accounts | Same Region only; one sink per account per Region; each source can share with up to five monitoring accounts |
| Log centralization rules | Organization-level rules that replicate new log events from many accounts and Regions into a central account | Yes, a copy | Central retention, one place for filters and compliance copies | Only events arriving after the rule exists; extra ingested copy is billed ⚠ VERIFY; needs Organizations trusted access |
| Subscription filters | Stream matching events in near real time to Kinesis Data Streams, Amazon Data Firehose or Lambda (and from there to S3, OpenSearch or a SIEM) | Yes, a stream | Feeding a SIEM, a data lake or custom processing | Standard class only; one account-level filter per Region and up to five per log group; at-least-once delivery means duplicates |
| Export to S3 / scheduled queries | Batch export, or run Logs Insights queries on a schedule and deliver results to S3 or EventBridge | Yes, a batch | Long-term archive, reporting | Not real time |
Cross-account observability has an attractive price: AWS documents no extra cost for sharing logs and metrics or Application Signals, and the first trace copy is free. ⚠ VERIFY on the pricing page before relying on that. You still pay for the underlying ingestion in each source account.
Log classes
| Feature | Standard | Essentials (formerly Infrequent Access) |
|---|---|---|
| Ingestion price | Higher | Lower (the pricing examples show $0.50 vs $0.25 per GB ⚠ VERIFY) |
| Storage and Logs Insights charges | Same | Same |
| Logs Insights | All commands | Most commands |
| Metric filters | Yes | No |
| Subscription filters | Yes | No |
| Embedded Metric Format | Yes | No |
| Live Tail | Yes | No |
| Log anomaly detection | Yes | No |
| Cross-account observability | Yes | Yes |
| Data protection (masking) | Yes | Yes |
Two rules fall out of this. A log group's class cannot change after creation, so choose carefully. And never put application logs you want alarms from into Essentials, because you lose metric filters and EMF. Essentials is for the high-volume, rarely read logs: load balancer access logs you query only after incidents, VPC Flow Logs you keep for forensics. At the time of checking, the old value INFREQUENT_ACCESS is still accepted as input and the console now says Essentials. ⚠ VERIFY the API behaviour change AWS announced for December 2026.
Error grouping tools
- Logs Insights
pattern: clusters log lines by their shape and counts them. Good for a first look at an unknown flood. stats count() byon a field you control: if your logs carryerror.fingerprint, a plainstatsgives exact groups.- Metric filters: turn a matching line into a number you can alarm on (Standard class only).
- Log anomaly detection: learns a baseline for a Standard log group and flags unusual patterns. Useful as a safety net, not a replacement for explicit alarms.
- Application Signals
Faultmetric: counts 5xx and error spans per operation, with no log parsing.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| One Log Analytics workspace for many subscriptions | Cross-account observability monitoring account, or centralization into one account | AWS keeps logs per account unless you link or replicate |
| Workspace-based Application Insights, cross-resource queries | OAM links plus Logs Insights across source accounts | Same-Region constraint for OAM |
| Data export rules to storage or Event Hubs | Subscription filters to Firehose or Kinesis, S3 export | Standard log class required for subscriptions |
| Basic Logs vs Analytics Logs tables | Essentials vs Standard log classes | Class is fixed at log group creation |
| Azure Policy to deploy diagnostic settings everywhere | Organization centralization rules and OAM organisation links | Managed from the organisation management or delegated administrator account |
| Application Insights failures blade (grouped exceptions) | Logs Insights pattern and fingerprint fields, Application Signals faults | You design the fingerprint |
| Action groups (email, SMS, webhook, ITSM) | SNS topics, EventBridge, AWS Chatbot / chat applications integration | Alarm → SNS → subscriber; composite alarms reduce noise |
| Sentinel (SIEM) | Security Lake, OpenSearch, third-party SIEM fed by subscriptions | Out of scope here; see Module 5 |
Architecture
Security-boundary view: OAM gives the monitoring account read access to telemetry in the source accounts. That is powerful, so the sink policy names exactly which accounts or organisation units may link, and which telemetry types they may share. Raw archive writes go to a separate account with no delete rights for the workload accounts.
Step-by-step implementation
Decide what lives where
Classify each log group: hot (alarm sources, Standard, 14-30 days), warm (queried after incidents, Essentials), archive (compliance copies in S3 with a lifecycle policy). Write the classification in a table before you touch the console.
Create the monitoring-account sink
One sink per Region. The sink policy limits who can link: a specific list of accounts, or everyone in your organisation by aws:PrincipalOrgID.
Link each source account
The link states which telemetry types to share. The source account must share no more types than the monitoring account's sink accepts, otherwise link creation fails.
Standardise the fingerprint in code
Every service computes an error.fingerprint and logs it as a structured field. This is the keystone of error grouping.
Create fingerprint-based metrics and alarms
Metric filters on the Standard log groups count total errors and per-severity errors; composite alarms combine "error rate high" with "traffic present" to avoid night-time noise.
Route by severity
Sev1 pages the on-call engineer, Sev2 posts to the team channel and opens a ticket, Sev3 goes to a daily digest. Test every route with set-alarm-state before you trust it.
AWS Console instructions
Cross-account observability.
- In the monitoring account: Console → CloudWatch → Settings → Monitoring account configuration → Configure (wording may vary). Choose which data types to receive (metrics, logs, traces, Application Signals) and which source accounts or organisation units may link.
- Copy the sink ARN or the generated link template or URL.
- In each source account: CloudWatch → Settings → Source account configuration → Link to monitoring account. Choose the data to share, review the label template (for example
payments-prod), and confirm. - In the monitoring account: CloudWatch → Logs → Logs Insights, then choose Select log groups → Browse log groups across linked accounts. Run a query across accounts.
Log centralization. From the organisation management or delegated administrator account: CloudWatch → Settings → Organization → Configure rule. Choose source accounts, source Regions, the destination account and Region, an optional backup Region, which log groups to include, and how to handle customer managed KMS keys. Destination log group names can be templated with ${source.accountId} and ${source.region} to avoid merging unrelated groups. A rule has a health status you can watch.
Alarm routing. CloudWatch → Alarms → Create composite alarm to combine rules; SNS → Topics → Create topic per severity; subscribe the on-call integration to the Sev1 topic only.
AWS CLI
# MONITORING ACCOUNT: create a sink (one per Region) and read its ARN.
aws oam create-sink --name hl-monitoring-sink \
--tags Project=cea-lab,Lab=m11-l10,Environment=learn \
--profile learn-monitoring --region us-east-1
# Allow accounts in the organisation to link, sharing only metrics, logs and traces.
# (Replace o-exampleorgid and SINK_ARN.)
cat > sink-policy.json <<'EOF'
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": "*",
"Action": ["oam:CreateLink", "oam:UpdateLink"],
"Resource": "*",
"Condition": {
"ForAllValues:StringEquals": {
"oam:ResourceTypes": ["AWS::CloudWatch::Metric", "AWS::Logs::LogGroup", "AWS::XRay::Trace"]
},
"StringEquals": { "aws:PrincipalOrgID": "o-exampleorgid" }
}
}]
}
EOF
aws oam put-sink-policy --sink-identifier SINK_ARN \
--policy file://sink-policy.json --profile learn-monitoring --region us-east-1
# SOURCE ACCOUNT: create the link to the sink, sharing logs and metrics.
aws oam create-link \
--label-template '$AccountName' \
--resource-types "AWS::CloudWatch::Metric" "AWS::Logs::LogGroup" \
--sink-identifier SINK_ARN \
--profile learn-payments --region us-east-1
# Check links from either side.
aws oam list-attached-links --sink-identifier SINK_ARN --profile learn-monitoring --region us-east-1
# Logs Insights across accounts: the monitoring account uses log group ARNs.
aws logs start-query \
--log-group-identifiers "arn:aws:logs:us-east-1:111111111111:log-group:/hl/payments/api" \
--start-time $(date -u -d '-1 hour' +%s) --end-time $(date -u +%s) \
--query-string 'fields @timestamp, error.fingerprint, @message
| filter level = "Error"
| stats count() as n, min(@timestamp) as first_seen by error.fingerprint
| sort n desc | limit 10' \
--profile learn-monitoring --region us-east-1
# Cluster an unknown flood into patterns (Logs Insights pattern command).
aws logs start-query --log-group-name /hl/payments/api \
--start-time $(date -u -d '-30 minutes' +%s) --end-time $(date -u +%s) \
--query-string 'filter level = "Error" | pattern @message | sort @sampleCount desc | limit 10' \
--profile learn-sandbox --region us-east-1
# Subscription filter: stream every Error line from the payments log group to Firehose.
aws logs put-subscription-filter \
--log-group-name /hl/payments/api \
--filter-name errors-to-archive \
--filter-pattern '{ $.level = "Error" }' \
--destination-arn arn:aws:firehose:us-east-1:222222222222:deliverystream/hl-log-archive \
--role-arn arn:aws:iam::111111111111:role/hl-logs-to-firehose \
--profile learn-payments --region us-east-1
# Test a route before trusting it: force the alarm into ALARM for a moment.
aws cloudwatch set-alarm-state --alarm-name hl-payments-sev1 \
--state-value ALARM --state-reason "Routing test" \
--profile learn-sandbox --region us-east-1The oam, logs and cloudwatch commands above exist in AWS CLI v2. The sink-policy condition keys follow the Observability Access Manager documentation as I recall it; confirm them against the current OAM policy examples before you apply this policy. ⚠ VERIFY
Infrastructure as Code
The monitoring stack creates the sink and its policy. A second stack (deployed to each source account) creates the link. L1 constructs are used because OAM has no L2. Property names follow the CloudFormation AWS::Oam::Sink and AWS::Oam::Link references; compile before relying on them. ⚠ VERIFY
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.CloudWatch.Actions;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.Oam;
using Amazon.CDK.AWS.SNS;
using Constructs;
namespace Harbourline.Monitoring.Infra;
/// <summary>Deployed to the monitoring account.</summary>
public sealed class MonitoringSinkStack : Stack
{
public MonitoringSinkStack(Construct scope, string id, string organisationId, IStackProps? props = null)
: base(scope, id, props)
{
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m11-l10");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
new CfnSink(this, "Sink", new CfnSinkProps
{
Name = "hl-monitoring-sink",
Policy = new Dictionary<string, object>
{
["Version"] = "2012-10-17",
["Statement"] = new object[]
{
new Dictionary<string, object>
{
["Effect"] = "Allow",
["Principal"] = "*",
["Action"] = new[] { "oam:CreateLink", "oam:UpdateLink" },
["Resource"] = "*",
["Condition"] = new Dictionary<string, object>
{
["ForAllValues:StringEquals"] = new Dictionary<string, object>
{
["oam:ResourceTypes"] = new[]
{
"AWS::CloudWatch::Metric", "AWS::Logs::LogGroup", "AWS::XRay::Trace",
},
},
// Only accounts in OUR organisation may link.
["StringEquals"] = new Dictionary<string, object>
{
["aws:PrincipalOrgID"] = organisationId,
},
},
},
},
},
});
}
}
/// <summary>Deployed to each source account.</summary>
public sealed class SourceLinkStack : Stack
{
public SourceLinkStack(Construct scope, string id, string sinkArn, IStackProps? props = null)
: base(scope, id, props)
{
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m11-l10");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
new CfnLink(this, "Link", new CfnLinkProps
{
LabelTemplate = "$AccountName",
ResourceTypes = new[] { "AWS::CloudWatch::Metric", "AWS::Logs::LogGroup", "AWS::XRay::Trace" },
SinkIdentifier = sinkArn,
});
// Error counting and alarms live next to the log group, in the source account.
var logGroup = new LogGroup(this, "ApiLogs", new LogGroupProps
{
LogGroupName = "/hl/payments/api",
Retention = RetentionDays.TWO_WEEKS,
LogGroupClass = LogGroupClass.STANDARD, // metric filters need Standard
RemovalPolicy = RemovalPolicy.DESTROY,
});
var errors = new MetricFilter(this, "ErrorCount", new MetricFilterProps
{
LogGroup = logGroup,
MetricNamespace = "Harbourline/Payments",
MetricName = "ErrorLines",
MetricValue = "1",
FilterPattern = FilterPattern.Literal("{ $.level = \"Error\" }"),
});
var requests = new MetricFilter(this, "RequestCount", new MetricFilterProps
{
LogGroup = logGroup,
MetricNamespace = "Harbourline/Payments",
MetricName = "RequestLines",
MetricValue = "1",
FilterPattern = FilterPattern.Literal("{ $.category = \"RequestCompleted\" }"),
});
var highErrors = new Alarm(this, "HighErrors", new AlarmProps
{
AlarmName = "hl-payments-errors-high",
Metric = errors.Metric(new MetricOptions { Statistic = "Sum", Period = Duration.Minutes(1) }),
Threshold = 20,
EvaluationPeriods = 3,
DatapointsToAlarm = 3,
TreatMissingData = TreatMissingData.NOT_BREACHING,
});
var hasTraffic = new Alarm(this, "HasTraffic", new AlarmProps
{
AlarmName = "hl-payments-has-traffic",
Metric = requests.Metric(new MetricOptions { Statistic = "Sum", Period = Duration.Minutes(1) }),
Threshold = 10,
EvaluationPeriods = 3,
ComparisonOperator = ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING, // no data at all means no traffic, so not in ALARM
});
// Page only when errors are high AND there is real traffic: ignores a quiet night.
var sev1 = new Topic(this, "Sev1", new TopicProps { TopicName = "hl-sev1-page" });
var composite = new CompositeAlarm(this, "PaymentsSev1", new CompositeAlarmProps
{
CompositeAlarmName = "hl-payments-sev1",
AlarmRule = AlarmRule.AllOf(highErrors, hasTraffic),
ActionsEnabled = true,
});
composite.AddAlarmAction(new SnsAction(sev1));
}
}Complete code
Error fingerprinting in C#. It reduces an exception to a stable identity that is the same across hosts, requests and deployments, and different between genuinely different failures.
using System.Diagnostics;
using System.Security.Cryptography;
using System.Text;
using System.Text.RegularExpressions;
namespace Harbourline.Observability;
/// <summary>
/// Produces a short, stable fingerprint for an exception so that thousands of log
/// lines group into one error. Variable data (ids, numbers, GUIDs, IBANs) is removed.
/// </summary>
public static partial class ErrorFingerprint
{
public static string For(Exception ex)
{
var sb = new StringBuilder();
// Walk the chain: the root cause matters as much as the wrapper.
for (var e = ex; e is not null; e = e.InnerException)
{
sb.Append(e.GetType().FullName).Append('|');
sb.Append(Normalise(e.Message)).Append('|');
}
// The first few application frames locate the failure; library frames change with upgrades.
var frames = new StackTrace(ex, fNeedFileInfo: false).GetFrames()
.Select(f => f.GetMethod())
.Where(m => m?.DeclaringType?.FullName?.StartsWith("Harbourline.", StringComparison.Ordinal) == true)
.Take(3)
.Select(m => $"{m!.DeclaringType!.FullName}.{m.Name}");
sb.Append(string.Join(">", frames));
var hash = SHA256.HashData(Encoding.UTF8.GetBytes(sb.ToString()));
return Convert.ToHexString(hash, 0, 6).ToLowerInvariant(); // 12 hex chars is plenty
}
/// <summary>Replace things that vary between occurrences with placeholders.</summary>
public static string Normalise(string message)
{
var m = GuidPattern().Replace(message, "<guid>");
m = IbanPattern().Replace(m, "<iban>");
m = AccountPattern().Replace(m, "<id>");
m = NumberPattern().Replace(m, "<n>");
return m.Trim();
}
[GeneratedRegex(@"[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}")]
private static partial Regex GuidPattern();
[GeneratedRegex(@"\b[A-Z]{2}\d{2}[A-Z0-9]{10,30}\b")]
private static partial Regex IbanPattern();
[GeneratedRegex(@"\b(ACC|CUS|PAY)-[A-Za-z0-9-]+")]
private static partial Regex AccountPattern();
[GeneratedRegex(@"\d+")]
private static partial Regex NumberPattern();
}using Microsoft.AspNetCore.Diagnostics;
using Microsoft.AspNetCore.Mvc;
namespace Harbourline.Observability;
/// <summary>Logs every unhandled exception once, with a fingerprint, and returns ProblemDetails.</summary>
public sealed partial class GlobalExceptionHandler(ILogger<GlobalExceptionHandler> logger) : IExceptionHandler
{
public async ValueTask<bool> TryHandleAsync(HttpContext ctx, Exception ex, CancellationToken ct)
{
var fingerprint = ErrorFingerprint.For(ex);
var correlationId = ctx.TraceIdentifier;
// The message is a CONSTANT template; variable data goes in structured properties.
// That keeps the message line itself groupable and keeps raw values out of the text.
LogUnhandled(logger, ex, fingerprint, ex.GetType().Name, ctx.Request.Path.Value, correlationId);
ctx.Response.StatusCode = StatusCodes.Status500InternalServerError;
await ctx.Response.WriteAsJsonAsync(new ProblemDetails
{
Status = 500,
Title = "An unexpected error occurred.",
Extensions = { ["correlationId"] = correlationId, ["errorRef"] = fingerprint },
}, ct);
return true; // handled
}
[LoggerMessage(EventId = 5000, Level = LogLevel.Error,
Message = "Unhandled exception {ErrorType} (fingerprint {ErrorFingerprint})")]
private static partial void LogUnhandled(ILogger logger, Exception ex, string errorFingerprint,
string errorType, string? path, string correlationId);
}Registration in Program.cs:
builder.Services.AddExceptionHandler<GlobalExceptionHandler>();
builder.Services.AddProblemDetails();
// ...
app.UseExceptionHandler();Code explained line by line
ErrorFingerprint.Forloops from the outer exception to the innermost, because wrappers such asAggregateExceptionorDbUpdateExceptionhide the real cause.Normalisestrips GUIDs, IBAN-shaped strings, course-style ids and any digits. Without it, "Account ACC-123 not found" and "Account ACC-456 not found" would be different errors.- Only frames in the
Harbourline.namespace feed the hash. Framework and library frames shift with each package upgrade and would split one error into many after every release. SHA256.HashDataplus the first six bytes gives a 12-character hex id: short enough for a dashboard, long enough to avoid collisions in practice.[GeneratedRegex]compiles the patterns at build time, which is faster and trim-safe.IExceptionHandleris the .NET 8+ way to centralise unhandled exceptions.TryHandleAsyncreturnstrueto say it handled the exception.[LoggerMessage]generates allocation-free logging. The message template is constant;errorFingerprint,errorType,pathandcorrelationIdbecome structured properties in the JSON log.- The response exposes
errorRef(the fingerprint) andcorrelationId, so a support agent can quote them and an engineer can find the exact group. - The log message includes no exception message text, only the type and fingerprint. The exception object itself is attached separately (
ex), so the stack trace is still logged, but mask or restrict who can read it, since exception messages sometimes contain account data.
Security considerations
⚠ Learning shortcut — not for production
The lab shares logs, metrics and traces from every account in the organisation, uses an email address on the paging topic and keeps default KMS settings. In production, scope the sink policy to named accounts or organisational units, use customer managed KMS keys with the LogsManaged tag where centralization needs them, run the alert topic through an approved paging integration and review who can query across accounts.
- A monitoring account is a privileged vantage point. Anyone who can run Logs Insights there can read logs from every linked account. Treat it like a production data store, with MFA, short sessions and audited access.
- Logs hold personal data. Aggregation copies or exposes it more widely. Mask at source (data protection policies, structured logging that never records full account numbers), set retention to what your data-protection officer agrees, and remember that replicating logs across Regions or accounts can change data-residency analysis under UK GDPR.
- Encryption. Centralization needs the destination KMS key to carry the
LogsManaged = truetag, per the AWS documentation, and key policies that allow CloudWatch Logs to use it. A wrong key policy silently drops replicated events until you notice the rule's health or error metrics. - Subscription recursion. Never subscribe a log group to a function that logs into the same group. AWS documents recursion prevention for this reason.
- Alert channels are an exfiltration path. An alarm message can contain log text. Do not paste customer data into chat channels.
Banking use case
Harbourline's accounts: payments, accounts, notifications, data-platform, plus monitoring and log-archive.
- Every workload account links to
monitoringby OAM (logs, metrics, traces, Application Signals), so one engineer can query "all errors with correlation id X" across three accounts. - Hot application logs (Standard, 14 days) feed metric filters and alarms in their own accounts.
- Load balancer and VPC Flow logs go to Essentials, queried only for forensics.
- A subscription filter sends
ErrorandWarninglines from payments to a Data Firehose stream inlog-archive, writing compressed objects to an S3 bucket with Object Lock for seven years of retention, because the retention requirement comes from the bank's policy, not from AWS. ⚠ confirm the real period with Compliance. - The alert policy:
| Severity | Example | Route | Response |
|---|---|---|---|
| Sev1 | Payments error rate high with traffic present; SLO fast burn | Page on-call, open incident channel | Immediate |
| Sev2 | New fingerprint with more than 50 occurrences, notification delays | Team channel and ticket | Same business day |
| Sev3 | Known error trending up; Warning-level patterns | Daily digest | Next sprint |
The fingerprint is the thread through all three: the Sev2 rule is "a fingerprint not seen before the current release with more than 50 occurrences in 15 minutes".
Common mistakes
- Copying everything everywhere. Use OAM to read in place; replicate only what compliance requires.
- Choosing Essentials for logs you want to alarm on. No metric filters, no EMF, and the class cannot be changed later.
- Grouping by raw message. Messages with ids never group; log a constant template and put the variable data in fields.
- Paging on a rate with no traffic guard. A 50% error rate at 3 a.m. with two requests is noise.
- Forgetting a destination KMS tag or key policy. Centralization quietly fails to deliver.
- Linking more than the sink accepts. If the source shares more types than the monitoring account allows, link creation fails, and nothing is shared.
- Expecting history. Centralization only handles log data that arrives after the rule is created.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Link creation fails | Source shares types the sink does not allow, or sink policy blocks the account | Align resource types; check aws:PrincipalOrgID and the account list |
| Cross-account log groups missing in Logs Insights | Link not active, or different Region | OAM is per Region; check list-attached-links |
| Metric from a linked account missing | Metric has not emitted since the link was created | Wait for new data points; the console only shows metrics that emit after linking |
| Centralized logs not appearing | Rule unhealthy, KMS tag/policy missing, trusted access off | Check rule health and FailureReason; fix key policy; enable trusted access |
| Subscription filter delivers nothing | Log group is Essentials, or role cannot write to the destination | Use a Standard log group; fix the role; check delivery metrics |
| Subscription stops after an error | Non-retryable destination error such as AccessDenied | Fix permissions; delivery retries after a pause but skipped events are lost |
| Duplicate events downstream | At-least-once delivery | De-duplicate on a log event id in the consumer |
| Fingerprints keep changing | Variable data in messages, or library frames in the hash | Strengthen Normalise; restrict frames to your own namespace |
| Composite alarm never fires | Child alarm in INSUFFICIENT_DATA, wrong missing-data setting | Test each child with set-alarm-state; review treatMissingData |
Cost and cleanup
| Item | Label | Estimate |
|---|---|---|
| OAM sink and link | free | AWS documents no extra cost for sharing logs and metrics ⚠ VERIFY |
| Log ingestion in lab accounts | credits | Pennies at lab volume (Standard about $0.50 per GB in examples ⚠ VERIFY) |
| Centralization copy | credits | An extra ingested copy; skip it in the lab, read the docs |
| Firehose and S3 archive | credits | Pennies at lab volume; delete the stream and bucket afterwards |
| Metric filters, alarms (3 plus composite) | credits | Alarms about $0.10 each, composite about $0.50 per month in the examples ⚠ VERIFY |
Free alternative: use one account. Create an OAM sink and link inside the same account (a single account can be both monitoring and source, but only sends its own data), run the fingerprint code and Logs Insights queries against a single log group, and read the centralization section as a design exercise. Everything else in this lesson works.
# Delete links first (from the source account), then the sink.
aws oam list-links --profile learn-payments --region us-east-1
aws oam delete-link --identifier LINK_ARN --profile learn-payments --region us-east-1
aws oam delete-sink --identifier SINK_ARN --profile learn-monitoring --region us-east-1
# Remove subscription filters, alarms and log groups.
aws logs delete-subscription-filter --log-group-name /hl/payments/api --filter-name errors-to-archive \
--profile learn-payments --region us-east-1
aws cloudwatch delete-alarms --alarm-names hl-payments-sev1 hl-payments-errors-high hl-payments-has-traffic \
--profile learn-payments --region us-east-1
aws logs delete-log-group --log-group-name /hl/payments/api --profile learn-payments --region us-east-1
# Confirm nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-payments --region us-east-1 --query 'ResourceTagMappingList[].ResourceARN'Exercises
- 🟢 Guided: Generate 200 log lines with four different exceptions (vary ids in the messages). Run the Logs Insights
pattern @messagecommand, then compute fingerprints withErrorFingerprint.Forand run astats count() by error.fingerprintquery. Compare the groupings. - 🟡 Challenge: Write a unit test suite for
ErrorFingerprint: same exception with different ids gives the same fingerprint; different exception types give different ones; changing a library frame does not change it. - 🔴 Live problem: Ticket: "The logging bill tripled last month and the compliance team says they cannot find the transfer logs from three weeks ago." You inherit eight accounts, centralization rules and one monitoring account. List what you would check first, which numbers you need from Cost Explorer, and how you would redesign retention and classes without losing evidence.
Assignment
Produce a "logging architecture decision record" for Harbourline: a table of every log source (application, ALB, VPC Flow Logs, CloudTrail, database), its class, retention, aggregation route and owner, with a one-paragraph justification for each choice and a monthly cost estimate for 50 GB per day of total ingestion. State any retention figures you cannot verify as assumptions to confirm with Compliance.
Solutions
The pattern command clusters by shape, so lines that differ only in ids usually collapse into one pattern each: you should see roughly four patterns, possibly more if messages contain unusual tokens. Your own fingerprint query should give exactly four groups, because Normalise removed the ids and the first Harbourline frames identify the origin. If you see more than four fingerprints, find the varying token (a timestamp, a different exception message detail) and add it to Normalise. The lesson: pattern discovery is for exploring unknown logs, fingerprints are for repeatable alerting.
Create three xUnit v3 tests. (1) Throw new InvalidOperationException("Account ACC-123 not found") and ... "ACC-456 ..." from the same method; assert equal fingerprints. (2) Throw an InvalidOperationException and an ArgumentException from the same method; assert different fingerprints. (3) For the library-frame invariance, assert that Normalise is stable and that fingerprints depend only on the first three Harbourline. frames, for instance by throwing from a helper method in a non-Harbourline namespace that wraps the same app call. Keep the tests free of AWS dependencies so they run in CI without an account.
Knowledge check
Answer: CloudWatch cross-account observability (OAM) with a monitoring account and links from the source accounts. Centralization copies data and costs more, a subscription filter streams data out, S3 export is batch.
Answer: Essentials does not support metric filters (nor subscription filters, EMF, Live Tail or anomaly detection), and a log group's class cannot be changed after creation.
Answer: Per the AWS documentation, one account-level subscription filter per Region and up to five filters per log group. ⚠ VERIFY, limits change.
Answer: Framework and library frames change with upgrades and would split one logical error into several after each deployment.
Answer: No. It only processes data that arrives after the rule is created.
Interview questions
- How would you give on-call engineers one place to search logs from twelve accounts? Cross-account observability: a monitoring account with a sink, source accounts linked, scoped by organisation. Add centralization or Firehose only for retention or SIEM requirements.
- How do you stop one bad deployment creating thousands of pages? Group by fingerprint, alarm on rate with a traffic guard, combine alarms in a composite alarm, and route by severity so only customer-impacting conditions page.
- What are the privacy risks of log aggregation? Wider exposure of personal data, cross-Region or cross-account residency changes, and chat channels as an exfiltration path. Mitigate with masking, retention limits, access control and DPO review.
- When would you use a subscription filter? To feed a SIEM, data lake or custom processor in near real time, accepting Standard-class only and at-least-once delivery.
Certification practice
A company has 30 accounts in one Region. Operators want to search application logs and view metrics from all of them in a central account without duplicating data. Which solution is MOST cost-effective?
A. Subscription filters from every log group to a central Kinesis stream. B. CloudWatch cross-account observability with a monitoring account and linked source accounts. C. Daily export of all log groups to S3 and Athena. D. A third-party agent shipping all logs to an external SaaS.
Answer: B. It reads data in place and AWS documents no additional charge for sharing logs and metrics. A and D duplicate data and add pipeline cost; C is batch and not interactive.
An operations team wants all new log events from member accounts in an organisation replicated into one account in another Region for central retention. Which feature fits?
A. CloudWatch Logs centralization rules managed from the organisation management or delegated administrator account. B. VPC peering between accounts. C. An S3 bucket policy. D. A resource-based policy on each log group.
Answer: A. Centralization rules replicate new data across accounts and Regions. B, C and D do not replicate CloudWatch Logs data.
A log group is created in the Essentials (Infrequent Access) class. Later the team wants metric filters and alarms on it. What should they do?
A. Change the class in the console.
B. Create a new Standard log group, point the application to it, and recreate filters there.
C. Use the CLI flag --force.
D. Enable KMS encryption.
Answer: B. A log group's class cannot be changed after creation, and Essentials lacks metric filters. The other options do not exist or do not help.
Completion checklist
- I can name four ways to aggregate logs and what each costs or cannot do
- I can choose between Standard and Essentials log classes and explain why the class is permanent
- I implemented or traced through the fingerprint code
- I can describe a severity-based alert routing policy
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- Read logs in place with cross-account observability; copy only when retention, SIEM or compliance needs it.
- Choose the log class once, carefully: Essentials is cheap but has no metric filters, subscriptions or EMF.
- Group errors by a stable fingerprint, not by raw message text.
- Page on customer impact with a traffic guard; route everything else to tickets and digests.
- Aggregated logs concentrate personal data: mask at source, limit access, agree retention.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
AWS CloudTrail for operations: who changed what, and when
Use CloudTrail event history, a trail delivered to CloudWatch Logs and EventBridge rules to answer 'who changed this security group?' in minutes, alert on risky API calls, and keep audit trails separate from application logs.
Performance bottlenecks in .NET on AWS: thread pool, connection pools, cold starts and N+1
Recognise the four classic .NET performance failures on AWS from their symptoms, prove each with evidence from dotnet-counters, CloudWatch and traces, and apply the right fix.