Skip to Content
Cloud Engineering Academy
11. Monitoring, Logging & Troubleshooting

Log aggregation and error monitoring across accounts

Centralise logs from many AWS accounts without copying everything twice, group thousands of exception lines into a handful of distinct errors, and route the right alerts to the right people with severity-based paths.

advanced⏱ 1 h 40 min🟡 Low costSOA-C03DOP-C02SAP-C02

Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Choose between four ways of bringing logs together: cross-account observability, log centralization rules, subscription filters and S3 export, and say what each costs and cannot do.
  • Pick the right CloudWatch Logs class (Standard or Essentials, formerly Infrequent Access) for each log group, knowing which features you lose with Essentials.
  • Turn a flood of exception lines into a short list of distinct errors with a stable fingerprint, Logs Insights pattern queries and metric filters.
  • Design alert routing by severity so that pages are rare and tickets are common, using composite alarms and SNS.
  • Build an observability access manager sink and link in CDK and Terraform.

Prerequisites

Why this matters in banking

Harbourline runs payments, accounts, notifications and the data platform in separate AWS accounts, because blast radius and separation of duties matter. That is good design and awful for troubleshooting: a failed transfer touches three accounts, and the engineer on call has to log into each one, remember which log group is where, and run the same query three times.

Two opposite failures follow. Either logs stay scattered and incidents take twice as long, or someone copies everything into one big place, forgets retention, and ends up with a seven-figure logging bill containing every customer's IBAN. A good aggregation design gives responders one place to search, keeps raw logs in the account that produced them where possible, and sends only what is needed to the long-term archive.

Error monitoring has the same shape. A single bad deployment can produce 400,000 identical exception lines in ten minutes. A pager that goes off 400,000 times is useless. You want one alert that says "new error, 400,000 occurrences, started 14:02, first seen in release 2026.11.18".

The idea in plain language

Aggregation answers "where do I look?". Error monitoring answers "what is actually going wrong, and who needs to know?".

For aggregation, think of a bank with many branches. You can (a) give head office a window into each branch's filing cabinet (nothing copied), (b) have each branch photocopy every document to head office (duplication, cost, but one cabinet), (c) have branches mail a copy of specific documents to an external auditor, or (d) box up old files and archive them. CloudWatch offers all four.

For error monitoring, the key idea is fingerprinting: reduce each error to a short, stable identity (exception type plus the place it came from), ignoring things that vary (ids, timestamps, amounts). Group by fingerprint and the 400,000 lines become one error with a count.

Analogy: a hospital triage desk. Ten patients with the same broken arm do not each get a separate doctor's alarm. They are grouped, ranked by severity, and the right specialist is called once.

What problem the AWS service solves

Four ways to aggregate logs

ApproachWhat it doesCopies data?Good forSharp edges
Cross-account observability (Observability Access Manager, OAM)A monitoring account queries logs, metrics, traces, Application Signals services and SLOs that live in source accountsNo, it reads in placeLive troubleshooting across accountsSame Region only; one sink per account per Region; each source can share with up to five monitoring accounts
Log centralization rulesOrganization-level rules that replicate new log events from many accounts and Regions into a central accountYes, a copyCentral retention, one place for filters and compliance copiesOnly events arriving after the rule exists; extra ingested copy is billed ⚠ VERIFY; needs Organizations trusted access
Subscription filtersStream matching events in near real time to Kinesis Data Streams, Amazon Data Firehose or Lambda (and from there to S3, OpenSearch or a SIEM)Yes, a streamFeeding a SIEM, a data lake or custom processingStandard class only; one account-level filter per Region and up to five per log group; at-least-once delivery means duplicates
Export to S3 / scheduled queriesBatch export, or run Logs Insights queries on a schedule and deliver results to S3 or EventBridgeYes, a batchLong-term archive, reportingNot real time

Cross-account observability has an attractive price: AWS documents no extra cost for sharing logs and metrics or Application Signals, and the first trace copy is free. ⚠ VERIFY on the pricing page before relying on that. You still pay for the underlying ingestion in each source account.

Log classes

FeatureStandardEssentials (formerly Infrequent Access)
Ingestion priceHigherLower (the pricing examples show $0.50 vs $0.25 per GB ⚠ VERIFY)
Storage and Logs Insights chargesSameSame
Logs InsightsAll commandsMost commands
Metric filtersYesNo
Subscription filtersYesNo
Embedded Metric FormatYesNo
Live TailYesNo
Log anomaly detectionYesNo
Cross-account observabilityYesYes
Data protection (masking)YesYes

Two rules fall out of this. A log group's class cannot change after creation, so choose carefully. And never put application logs you want alarms from into Essentials, because you lose metric filters and EMF. Essentials is for the high-volume, rarely read logs: load balancer access logs you query only after incidents, VPC Flow Logs you keep for forensics. At the time of checking, the old value INFREQUENT_ACCESS is still accepted as input and the console now says Essentials. ⚠ VERIFY the API behaviour change AWS announced for December 2026.

Error grouping tools

  • Logs Insights pattern: clusters log lines by their shape and counts them. Good for a first look at an unknown flood.
  • stats count() by on a field you control: if your logs carry error.fingerprint, a plain stats gives exact groups.
  • Metric filters: turn a matching line into a number you can alarm on (Standard class only).
  • Log anomaly detection: learns a baseline for a Standard log group and flags unusual patterns. Useful as a safety net, not a replacement for explicit alarms.
  • Application Signals Fault metric: counts 5xx and error spans per operation, with no log parsing.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
One Log Analytics workspace for many subscriptionsCross-account observability monitoring account, or centralization into one accountAWS keeps logs per account unless you link or replicate
Workspace-based Application Insights, cross-resource queriesOAM links plus Logs Insights across source accountsSame-Region constraint for OAM
Data export rules to storage or Event HubsSubscription filters to Firehose or Kinesis, S3 exportStandard log class required for subscriptions
Basic Logs vs Analytics Logs tablesEssentials vs Standard log classesClass is fixed at log group creation
Azure Policy to deploy diagnostic settings everywhereOrganization centralization rules and OAM organisation linksManaged from the organisation management or delegated administrator account
Application Insights failures blade (grouped exceptions)Logs Insights pattern and fingerprint fields, Application Signals faultsYou design the fingerprint
Action groups (email, SMS, webhook, ITSM)SNS topics, EventBridge, AWS Chatbot / chat applications integrationAlarm → SNS → subscriber; composite alarms reduce noise
Sentinel (SIEM)Security Lake, OpenSearch, third-party SIEM fed by subscriptionsOut of scope here; see Module 5

Architecture

Security-boundary view: OAM gives the monitoring account read access to telemetry in the source accounts. That is powerful, so the sink policy names exactly which accounts or organisation units may link, and which telemetry types they may share. Raw archive writes go to a separate account with no delete rights for the workload accounts.

Step-by-step implementation

Decide what lives where

Classify each log group: hot (alarm sources, Standard, 14-30 days), warm (queried after incidents, Essentials), archive (compliance copies in S3 with a lifecycle policy). Write the classification in a table before you touch the console.

Create the monitoring-account sink

One sink per Region. The sink policy limits who can link: a specific list of accounts, or everyone in your organisation by aws:PrincipalOrgID.

The link states which telemetry types to share. The source account must share no more types than the monitoring account's sink accepts, otherwise link creation fails.

Standardise the fingerprint in code

Every service computes an error.fingerprint and logs it as a structured field. This is the keystone of error grouping.

Create fingerprint-based metrics and alarms

Metric filters on the Standard log groups count total errors and per-severity errors; composite alarms combine "error rate high" with "traffic present" to avoid night-time noise.

Route by severity

Sev1 pages the on-call engineer, Sev2 posts to the team channel and opens a ticket, Sev3 goes to a daily digest. Test every route with set-alarm-state before you trust it.

AWS Console instructions

Cross-account observability.

  1. In the monitoring account: Console → CloudWatch → Settings → Monitoring account configuration → Configure (wording may vary). Choose which data types to receive (metrics, logs, traces, Application Signals) and which source accounts or organisation units may link.
  2. Copy the sink ARN or the generated link template or URL.
  3. In each source account: CloudWatch → Settings → Source account configuration → Link to monitoring account. Choose the data to share, review the label template (for example payments-prod), and confirm.
  4. In the monitoring account: CloudWatch → Logs → Logs Insights, then choose Select log groups → Browse log groups across linked accounts. Run a query across accounts.

Log centralization. From the organisation management or delegated administrator account: CloudWatch → Settings → Organization → Configure rule. Choose source accounts, source Regions, the destination account and Region, an optional backup Region, which log groups to include, and how to handle customer managed KMS keys. Destination log group names can be templated with ${source.accountId} and ${source.region} to avoid merging unrelated groups. A rule has a health status you can watch.

Alarm routing. CloudWatch → Alarms → Create composite alarm to combine rules; SNS → Topics → Create topic per severity; subscribe the on-call integration to the Sev1 topic only.

AWS CLI

# MONITORING ACCOUNT: create a sink (one per Region) and read its ARN.
aws oam create-sink --name hl-monitoring-sink \
  --tags Project=cea-lab,Lab=m11-l10,Environment=learn \
  --profile learn-monitoring --region us-east-1

# Allow accounts in the organisation to link, sharing only metrics, logs and traces.
# (Replace o-exampleorgid and SINK_ARN.)
cat > sink-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": "*",
    "Action": ["oam:CreateLink", "oam:UpdateLink"],
    "Resource": "*",
    "Condition": {
      "ForAllValues:StringEquals": {
        "oam:ResourceTypes": ["AWS::CloudWatch::Metric", "AWS::Logs::LogGroup", "AWS::XRay::Trace"]
      },
      "StringEquals": { "aws:PrincipalOrgID": "o-exampleorgid" }
    }
  }]
}
EOF
aws oam put-sink-policy --sink-identifier SINK_ARN \
  --policy file://sink-policy.json --profile learn-monitoring --region us-east-1

# SOURCE ACCOUNT: create the link to the sink, sharing logs and metrics.
aws oam create-link \
  --label-template '$AccountName' \
  --resource-types "AWS::CloudWatch::Metric" "AWS::Logs::LogGroup" \
  --sink-identifier SINK_ARN \
  --profile learn-payments --region us-east-1

# Check links from either side.
aws oam list-attached-links --sink-identifier SINK_ARN --profile learn-monitoring --region us-east-1

# Logs Insights across accounts: the monitoring account uses log group ARNs.
aws logs start-query \
  --log-group-identifiers "arn:aws:logs:us-east-1:111111111111:log-group:/hl/payments/api" \
  --start-time $(date -u -d '-1 hour' +%s) --end-time $(date -u +%s) \
  --query-string 'fields @timestamp, error.fingerprint, @message
    | filter level = "Error"
    | stats count() as n, min(@timestamp) as first_seen by error.fingerprint
    | sort n desc | limit 10' \
  --profile learn-monitoring --region us-east-1

# Cluster an unknown flood into patterns (Logs Insights pattern command).
aws logs start-query --log-group-name /hl/payments/api \
  --start-time $(date -u -d '-30 minutes' +%s) --end-time $(date -u +%s) \
  --query-string 'filter level = "Error" | pattern @message | sort @sampleCount desc | limit 10' \
  --profile learn-sandbox --region us-east-1

# Subscription filter: stream every Error line from the payments log group to Firehose.
aws logs put-subscription-filter \
  --log-group-name /hl/payments/api \
  --filter-name errors-to-archive \
  --filter-pattern '{ $.level = "Error" }' \
  --destination-arn arn:aws:firehose:us-east-1:222222222222:deliverystream/hl-log-archive \
  --role-arn arn:aws:iam::111111111111:role/hl-logs-to-firehose \
  --profile learn-payments --region us-east-1

# Test a route before trusting it: force the alarm into ALARM for a moment.
aws cloudwatch set-alarm-state --alarm-name hl-payments-sev1 \
  --state-value ALARM --state-reason "Routing test" \
  --profile learn-sandbox --region us-east-1

The oam, logs and cloudwatch commands above exist in AWS CLI v2. The sink-policy condition keys follow the Observability Access Manager documentation as I recall it; confirm them against the current OAM policy examples before you apply this policy. ⚠ VERIFY

Infrastructure as Code

The monitoring stack creates the sink and its policy. A second stack (deployed to each source account) creates the link. L1 constructs are used because OAM has no L2. Property names follow the CloudFormation AWS::Oam::Sink and AWS::Oam::Link references; compile before relying on them. ⚠ VERIFY

ObservabilityStacks.cs
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.CloudWatch.Actions;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.Oam;
using Amazon.CDK.AWS.SNS;
using Constructs;

namespace Harbourline.Monitoring.Infra;

/// <summary>Deployed to the monitoring account.</summary>
public sealed class MonitoringSinkStack : Stack
{
    public MonitoringSinkStack(Construct scope, string id, string organisationId, IStackProps? props = null)
        : base(scope, id, props)
    {
        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m11-l10");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");

        new CfnSink(this, "Sink", new CfnSinkProps
        {
            Name = "hl-monitoring-sink",
            Policy = new Dictionary<string, object>
            {
                ["Version"] = "2012-10-17",
                ["Statement"] = new object[]
                {
                    new Dictionary<string, object>
                    {
                        ["Effect"] = "Allow",
                        ["Principal"] = "*",
                        ["Action"] = new[] { "oam:CreateLink", "oam:UpdateLink" },
                        ["Resource"] = "*",
                        ["Condition"] = new Dictionary<string, object>
                        {
                            ["ForAllValues:StringEquals"] = new Dictionary<string, object>
                            {
                                ["oam:ResourceTypes"] = new[]
                                {
                                    "AWS::CloudWatch::Metric", "AWS::Logs::LogGroup", "AWS::XRay::Trace",
                                },
                            },
                            // Only accounts in OUR organisation may link.
                            ["StringEquals"] = new Dictionary<string, object>
                            {
                                ["aws:PrincipalOrgID"] = organisationId,
                            },
                        },
                    },
                },
            },
        });
    }
}

/// <summary>Deployed to each source account.</summary>
public sealed class SourceLinkStack : Stack
{
    public SourceLinkStack(Construct scope, string id, string sinkArn, IStackProps? props = null)
        : base(scope, id, props)
    {
        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m11-l10");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");

        new CfnLink(this, "Link", new CfnLinkProps
        {
            LabelTemplate = "$AccountName",
            ResourceTypes = new[] { "AWS::CloudWatch::Metric", "AWS::Logs::LogGroup", "AWS::XRay::Trace" },
            SinkIdentifier = sinkArn,
        });

        // Error counting and alarms live next to the log group, in the source account.
        var logGroup = new LogGroup(this, "ApiLogs", new LogGroupProps
        {
            LogGroupName = "/hl/payments/api",
            Retention = RetentionDays.TWO_WEEKS,
            LogGroupClass = LogGroupClass.STANDARD, // metric filters need Standard
            RemovalPolicy = RemovalPolicy.DESTROY,
        });

        var errors = new MetricFilter(this, "ErrorCount", new MetricFilterProps
        {
            LogGroup = logGroup,
            MetricNamespace = "Harbourline/Payments",
            MetricName = "ErrorLines",
            MetricValue = "1",
            FilterPattern = FilterPattern.Literal("{ $.level = \"Error\" }"),
        });

        var requests = new MetricFilter(this, "RequestCount", new MetricFilterProps
        {
            LogGroup = logGroup,
            MetricNamespace = "Harbourline/Payments",
            MetricName = "RequestLines",
            MetricValue = "1",
            FilterPattern = FilterPattern.Literal("{ $.category = \"RequestCompleted\" }"),
        });

        var highErrors = new Alarm(this, "HighErrors", new AlarmProps
        {
            AlarmName = "hl-payments-errors-high",
            Metric = errors.Metric(new MetricOptions { Statistic = "Sum", Period = Duration.Minutes(1) }),
            Threshold = 20,
            EvaluationPeriods = 3,
            DatapointsToAlarm = 3,
            TreatMissingData = TreatMissingData.NOT_BREACHING,
        });

        var hasTraffic = new Alarm(this, "HasTraffic", new AlarmProps
        {
            AlarmName = "hl-payments-has-traffic",
            Metric = requests.Metric(new MetricOptions { Statistic = "Sum", Period = Duration.Minutes(1) }),
            Threshold = 10,
            EvaluationPeriods = 3,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
            TreatMissingData = TreatMissingData.NOT_BREACHING, // no data at all means no traffic, so not in ALARM
        });

        // Page only when errors are high AND there is real traffic: ignores a quiet night.
        var sev1 = new Topic(this, "Sev1", new TopicProps { TopicName = "hl-sev1-page" });
        var composite = new CompositeAlarm(this, "PaymentsSev1", new CompositeAlarmProps
        {
            CompositeAlarmName = "hl-payments-sev1",
            AlarmRule = AlarmRule.AllOf(highErrors, hasTraffic),
            ActionsEnabled = true,
        });
        composite.AddAlarmAction(new SnsAction(sev1));
    }
}

Complete code

Error fingerprinting in C#. It reduces an exception to a stable identity that is the same across hosts, requests and deployments, and different between genuinely different failures.

ErrorFingerprint.cs
using System.Diagnostics;
using System.Security.Cryptography;
using System.Text;
using System.Text.RegularExpressions;

namespace Harbourline.Observability;

/// <summary>
/// Produces a short, stable fingerprint for an exception so that thousands of log
/// lines group into one error. Variable data (ids, numbers, GUIDs, IBANs) is removed.
/// </summary>
public static partial class ErrorFingerprint
{
    public static string For(Exception ex)
    {
        var sb = new StringBuilder();

        // Walk the chain: the root cause matters as much as the wrapper.
        for (var e = ex; e is not null; e = e.InnerException)
        {
            sb.Append(e.GetType().FullName).Append('|');
            sb.Append(Normalise(e.Message)).Append('|');
        }

        // The first few application frames locate the failure; library frames change with upgrades.
        var frames = new StackTrace(ex, fNeedFileInfo: false).GetFrames()
            .Select(f => f.GetMethod())
            .Where(m => m?.DeclaringType?.FullName?.StartsWith("Harbourline.", StringComparison.Ordinal) == true)
            .Take(3)
            .Select(m => $"{m!.DeclaringType!.FullName}.{m.Name}");

        sb.Append(string.Join(">", frames));

        var hash = SHA256.HashData(Encoding.UTF8.GetBytes(sb.ToString()));
        return Convert.ToHexString(hash, 0, 6).ToLowerInvariant(); // 12 hex chars is plenty
    }

    /// <summary>Replace things that vary between occurrences with placeholders.</summary>
    public static string Normalise(string message)
    {
        var m = GuidPattern().Replace(message, "<guid>");
        m = IbanPattern().Replace(m, "<iban>");
        m = AccountPattern().Replace(m, "<id>");
        m = NumberPattern().Replace(m, "<n>");
        return m.Trim();
    }

    [GeneratedRegex(@"[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}")]
    private static partial Regex GuidPattern();

    [GeneratedRegex(@"\b[A-Z]{2}\d{2}[A-Z0-9]{10,30}\b")]
    private static partial Regex IbanPattern();

    [GeneratedRegex(@"\b(ACC|CUS|PAY)-[A-Za-z0-9-]+")]
    private static partial Regex AccountPattern();

    [GeneratedRegex(@"\d+")]
    private static partial Regex NumberPattern();
}
GlobalExceptionHandler.cs
using Microsoft.AspNetCore.Diagnostics;
using Microsoft.AspNetCore.Mvc;

namespace Harbourline.Observability;

/// <summary>Logs every unhandled exception once, with a fingerprint, and returns ProblemDetails.</summary>
public sealed partial class GlobalExceptionHandler(ILogger<GlobalExceptionHandler> logger) : IExceptionHandler
{
    public async ValueTask<bool> TryHandleAsync(HttpContext ctx, Exception ex, CancellationToken ct)
    {
        var fingerprint = ErrorFingerprint.For(ex);
        var correlationId = ctx.TraceIdentifier;

        // The message is a CONSTANT template; variable data goes in structured properties.
        // That keeps the message line itself groupable and keeps raw values out of the text.
        LogUnhandled(logger, ex, fingerprint, ex.GetType().Name, ctx.Request.Path.Value, correlationId);

        ctx.Response.StatusCode = StatusCodes.Status500InternalServerError;
        await ctx.Response.WriteAsJsonAsync(new ProblemDetails
        {
            Status = 500,
            Title = "An unexpected error occurred.",
            Extensions = { ["correlationId"] = correlationId, ["errorRef"] = fingerprint },
        }, ct);

        return true; // handled
    }

    [LoggerMessage(EventId = 5000, Level = LogLevel.Error,
        Message = "Unhandled exception {ErrorType} (fingerprint {ErrorFingerprint})")]
    private static partial void LogUnhandled(ILogger logger, Exception ex, string errorFingerprint,
        string errorType, string? path, string correlationId);
}

Registration in Program.cs:

Program.cs (excerpt)
builder.Services.AddExceptionHandler<GlobalExceptionHandler>();
builder.Services.AddProblemDetails();
// ...
app.UseExceptionHandler();

Code explained line by line

  1. ErrorFingerprint.For loops from the outer exception to the innermost, because wrappers such as AggregateException or DbUpdateException hide the real cause.
  2. Normalise strips GUIDs, IBAN-shaped strings, course-style ids and any digits. Without it, "Account ACC-123 not found" and "Account ACC-456 not found" would be different errors.
  3. Only frames in the Harbourline. namespace feed the hash. Framework and library frames shift with each package upgrade and would split one error into many after every release.
  4. SHA256.HashData plus the first six bytes gives a 12-character hex id: short enough for a dashboard, long enough to avoid collisions in practice.
  5. [GeneratedRegex] compiles the patterns at build time, which is faster and trim-safe.
  6. IExceptionHandler is the .NET 8+ way to centralise unhandled exceptions. TryHandleAsync returns true to say it handled the exception.
  7. [LoggerMessage] generates allocation-free logging. The message template is constant; errorFingerprint, errorType, path and correlationId become structured properties in the JSON log.
  8. The response exposes errorRef (the fingerprint) and correlationId, so a support agent can quote them and an engineer can find the exact group.
  9. The log message includes no exception message text, only the type and fingerprint. The exception object itself is attached separately (ex), so the stack trace is still logged, but mask or restrict who can read it, since exception messages sometimes contain account data.

Security considerations

⚠ Learning shortcut — not for production

The lab shares logs, metrics and traces from every account in the organisation, uses an email address on the paging topic and keeps default KMS settings. In production, scope the sink policy to named accounts or organisational units, use customer managed KMS keys with the LogsManaged tag where centralization needs them, run the alert topic through an approved paging integration and review who can query across accounts.

  • A monitoring account is a privileged vantage point. Anyone who can run Logs Insights there can read logs from every linked account. Treat it like a production data store, with MFA, short sessions and audited access.
  • Logs hold personal data. Aggregation copies or exposes it more widely. Mask at source (data protection policies, structured logging that never records full account numbers), set retention to what your data-protection officer agrees, and remember that replicating logs across Regions or accounts can change data-residency analysis under UK GDPR.
  • Encryption. Centralization needs the destination KMS key to carry the LogsManaged = true tag, per the AWS documentation, and key policies that allow CloudWatch Logs to use it. A wrong key policy silently drops replicated events until you notice the rule's health or error metrics.
  • Subscription recursion. Never subscribe a log group to a function that logs into the same group. AWS documents recursion prevention for this reason.
  • Alert channels are an exfiltration path. An alarm message can contain log text. Do not paste customer data into chat channels.

Banking use case

Harbourline's accounts: payments, accounts, notifications, data-platform, plus monitoring and log-archive.

  • Every workload account links to monitoring by OAM (logs, metrics, traces, Application Signals), so one engineer can query "all errors with correlation id X" across three accounts.
  • Hot application logs (Standard, 14 days) feed metric filters and alarms in their own accounts.
  • Load balancer and VPC Flow logs go to Essentials, queried only for forensics.
  • A subscription filter sends Error and Warning lines from payments to a Data Firehose stream in log-archive, writing compressed objects to an S3 bucket with Object Lock for seven years of retention, because the retention requirement comes from the bank's policy, not from AWS. ⚠ confirm the real period with Compliance.
  • The alert policy:
SeverityExampleRouteResponse
Sev1Payments error rate high with traffic present; SLO fast burnPage on-call, open incident channelImmediate
Sev2New fingerprint with more than 50 occurrences, notification delaysTeam channel and ticketSame business day
Sev3Known error trending up; Warning-level patternsDaily digestNext sprint

The fingerprint is the thread through all three: the Sev2 rule is "a fingerprint not seen before the current release with more than 50 occurrences in 15 minutes".

Common mistakes

  • Copying everything everywhere. Use OAM to read in place; replicate only what compliance requires.
  • Choosing Essentials for logs you want to alarm on. No metric filters, no EMF, and the class cannot be changed later.
  • Grouping by raw message. Messages with ids never group; log a constant template and put the variable data in fields.
  • Paging on a rate with no traffic guard. A 50% error rate at 3 a.m. with two requests is noise.
  • Forgetting a destination KMS tag or key policy. Centralization quietly fails to deliver.
  • Linking more than the sink accepts. If the source shares more types than the monitoring account allows, link creation fails, and nothing is shared.
  • Expecting history. Centralization only handles log data that arrives after the rule is created.

Troubleshooting

SymptomLikely causeFix
Link creation failsSource shares types the sink does not allow, or sink policy blocks the accountAlign resource types; check aws:PrincipalOrgID and the account list
Cross-account log groups missing in Logs InsightsLink not active, or different RegionOAM is per Region; check list-attached-links
Metric from a linked account missingMetric has not emitted since the link was createdWait for new data points; the console only shows metrics that emit after linking
Centralized logs not appearingRule unhealthy, KMS tag/policy missing, trusted access offCheck rule health and FailureReason; fix key policy; enable trusted access
Subscription filter delivers nothingLog group is Essentials, or role cannot write to the destinationUse a Standard log group; fix the role; check delivery metrics
Subscription stops after an errorNon-retryable destination error such as AccessDeniedFix permissions; delivery retries after a pause but skipped events are lost
Duplicate events downstreamAt-least-once deliveryDe-duplicate on a log event id in the consumer
Fingerprints keep changingVariable data in messages, or library frames in the hashStrengthen Normalise; restrict frames to your own namespace
Composite alarm never firesChild alarm in INSUFFICIENT_DATA, wrong missing-data settingTest each child with set-alarm-state; review treatMissingData

Cost and cleanup

ItemLabelEstimate
OAM sink and linkfreeAWS documents no extra cost for sharing logs and metrics ⚠ VERIFY
Log ingestion in lab accountscreditsPennies at lab volume (Standard about $0.50 per GB in examples ⚠ VERIFY)
Centralization copycreditsAn extra ingested copy; skip it in the lab, read the docs
Firehose and S3 archivecreditsPennies at lab volume; delete the stream and bucket afterwards
Metric filters, alarms (3 plus composite)creditsAlarms about $0.10 each, composite about $0.50 per month in the examples ⚠ VERIFY

Free alternative: use one account. Create an OAM sink and link inside the same account (a single account can be both monitoring and source, but only sends its own data), run the fingerprint code and Logs Insights queries against a single log group, and read the centralization section as a design exercise. Everything else in this lesson works.

# Delete links first (from the source account), then the sink.
aws oam list-links --profile learn-payments --region us-east-1
aws oam delete-link --identifier LINK_ARN --profile learn-payments --region us-east-1
aws oam delete-sink --identifier SINK_ARN --profile learn-monitoring --region us-east-1

# Remove subscription filters, alarms and log groups.
aws logs delete-subscription-filter --log-group-name /hl/payments/api --filter-name errors-to-archive \
  --profile learn-payments --region us-east-1
aws cloudwatch delete-alarms --alarm-names hl-payments-sev1 hl-payments-errors-high hl-payments-has-traffic \
  --profile learn-payments --region us-east-1
aws logs delete-log-group --log-group-name /hl/payments/api --profile learn-payments --region us-east-1

# Confirm nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --profile learn-payments --region us-east-1 --query 'ResourceTagMappingList[].ResourceARN'

Exercises

  • 🟢 Guided: Generate 200 log lines with four different exceptions (vary ids in the messages). Run the Logs Insights pattern @message command, then compute fingerprints with ErrorFingerprint.For and run a stats count() by error.fingerprint query. Compare the groupings.
  • 🟡 Challenge: Write a unit test suite for ErrorFingerprint: same exception with different ids gives the same fingerprint; different exception types give different ones; changing a library frame does not change it.
  • 🔴 Live problem: Ticket: "The logging bill tripled last month and the compliance team says they cannot find the transfer logs from three weeks ago." You inherit eight accounts, centralization rules and one monitoring account. List what you would check first, which numbers you need from Cost Explorer, and how you would redesign retention and classes without losing evidence.

Assignment

Produce a "logging architecture decision record" for Harbourline: a table of every log source (application, ALB, VPC Flow Logs, CloudTrail, database), its class, retention, aggregation route and owner, with a one-paragraph justification for each choice and a monthly cost estimate for 50 GB per day of total ingestion. State any retention figures you cannot verify as assumptions to confirm with Compliance.

Solutions

Knowledge check

Interview questions

  • How would you give on-call engineers one place to search logs from twelve accounts? Cross-account observability: a monitoring account with a sink, source accounts linked, scoped by organisation. Add centralization or Firehose only for retention or SIEM requirements.
  • How do you stop one bad deployment creating thousands of pages? Group by fingerprint, alarm on rate with a traffic guard, combine alarms in a composite alarm, and route by severity so only customer-impacting conditions page.
  • What are the privacy risks of log aggregation? Wider exposure of personal data, cross-Region or cross-account residency changes, and chat channels as an exfiltration path. Mitigate with masking, retention limits, access control and DPO review.
  • When would you use a subscription filter? To feed a SIEM, data lake or custom processor in near real time, accepting Standard-class only and at-least-once delivery.

Certification practice

Completion checklist

  • I can name four ways to aggregate logs and what each costs or cannot do
  • I can choose between Standard and Essentials log classes and explain why the class is permanent
  • I implemented or traced through the fingerprint code
  • I can describe a severity-based alert routing policy
  • I ran the cleanup steps and confirmed nothing is still billing

Key takeaways

  • Read logs in place with cross-account observability; copy only when retention, SIEM or compliance needs it.
  • Choose the log class once, carefully: Essentials is cheap but has no metric filters, subscriptions or EMF.
  • Group errors by a stable fingerprint, not by raw message text.
  • Page on customer impact with a traffic guard; route everything else to tickets and digests.
  • Aggregated logs concentrate personal data: mask at source, limit access, agree retention.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page