Skip to Content
Cloud Engineering Academy
15. Multi-Account Governance & FinOps

Centralised logging: an organisation trail, a log archive account and retention you can defend

Design where every account's audit logs go, who can write, read and delete them, how long they are kept and what that costs, then build an organisation trail into an encrypted, Object Lock protected log archive bucket and verify it with a tested check.

advanced⏱ 1 h 30 min🟡 Low costSCS-C03SAP-C03SOA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.CloudTrail 4.1.x) · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform 1.14 / AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • List the log sources a bank must centralise and say, for each, where it lands and what protects it.
  • Design the log archive account so that producers can write, auditors can read, and nobody can delete or rewrite.
  • Create an organisation trail that covers every account and Region, delivers to a KMS-encrypted S3 bucket with Object Lock, and has log file validation on.
  • Choose retention periods with the records, legal and privacy teams, and explain why governance mode is for the lab and compliance mode needs a decision.
  • Estimate the cost of centralised logging, including the parts that surprise people (data events, KMS requests, extra trail copies).
  • Run a tested .NET check that tells you whether the organisation trail is healthy, and tear the lab down cleanly.

Prerequisites

Why this matters in banking

Imagine an attacker who steals an engineer's session in payments-prod. The first thing a competent attacker does is stop the evidence. If CloudTrail logs live in the same account, in a bucket the engineer's role can administer, the attacker deletes them and the bank cannot answer the regulator's first three questions: what did they do, which data did they touch, when did it start.

Centralised logging answers that structurally. Logs leave the account as they are written, travel to an account the workload cannot reach, land in storage that cannot be altered for a set period, and are checkable for tampering. Supervisors and auditors also ask the dull questions: how long do you keep it, who can read it, how do you know it is complete. This lesson gives you answers you can demonstrate rather than assert.

The idea in plain language

Think of logging as a letterbox into a sealed records room. Every account has a slot (services deliver logs on its behalf). The room is in a different building (the log archive account). Staff can post letters but cannot open the room. Auditors have a key to read, not to remove. A tamper-evident seal on every bundle (log file validation) tells you if anything was swapped. Nobody, not even the bank's most senior engineer, can shred a letter before its retention date (Object Lock).

Analogy: a bank's branch posts its daily ledger copies through a slot to head-office archives. The branch cannot retrieve or burn them. If the branch manager claims "that day's ledger never existed", the archive's signed digest proves otherwise.

What to centralise

SourceProduced byLands inNotes
Management events (API activity)CloudTrail organisation trailLog archive S3 bucketThe audit trail. First copy of management events is free per trail ⚠ VERIFY current pricing
Data events (S3 object, Lambda invoke, DynamoDB item)CloudTrail, advanced event selectorsSame bucket, or a separate trail and bucketCharged per event; enable only for sensitive resources
Configuration history and snapshotsAWS Config delivery channelLog archive bucket (or a dedicated one)Records what changed; see Config conformance packs
VPC Flow LogsEach VPCLog archive bucket or central network accountVolume can be large; sample or filter
Load balancer, S3 server access, CloudFront logsThe servicePer-service bucket in log archiveOften forgotten
DNS query logsRoute 53 ResolverLog archive bucket or CloudWatch LogsUseful for exfiltration hunting
WAF logsAWS WAFLog archive bucket via FirehoseName format restrictions apply ⚠ VERIFY
Findings exportGuardDuty / Security HubS3 in the security tooling or log archive accountCovered in Centralised security tooling
Application logsYour .NET servicesCloudWatch Logs, then forwardedDifferent retention; may contain personal data

CloudWatch Logs can also be centralised across accounts. AWS has been adding organisation-level log centralisation features; check the current CloudWatch documentation before designing around one ⚠ VERIFY.

What problem the AWS service solves

CloudTrail by itself records API activity per account. An organisation trail, created in the management account or by a CloudTrail delegated administrator, applies to every account in the organisation, including accounts added later, with a copy of the trail appearing in each member account automatically. Member accounts can see the trail but cannot stop it, delete it or change what it logs. That is the key property: the control sits above the account.

Facts worth knowing (from the CloudTrail user guide, checked 2026-11-18):

  • The console creates multi-Region organisation trails; the CLI lets you choose single or multi-Region.
  • Logs go to one S3 bucket, with a path that includes the organisation ID and each account ID (AWSLogs/<org-id>/<account-id>/...).
  • Event history in each account's console shows only that account's events (last 90 days of management events). Organisation-wide analysis needs the S3 logs (Athena) or CloudTrail Lake ⚠ VERIFY availability and pricing before relying on Lake.
  • If an organisation trail's bucket policy, SNS policy or KMS permission is wrong, CloudTrail still creates the member-account trails, but delivery fails. Check get-trail-status.
  • Removing an account from the organisation deletes its copy of the trail and stops logging for it. Logs already delivered stay in the bucket.
  • If the management account that created the trail stops being the management account, the trail becomes a normal trail.

Limits and honest costs

  • Opt-in Regions need care: a multi-Region trail whose home Region is an opt-in Region only logs member accounts that enabled that Region.
  • Cost drivers: the first copy of management events is free; extra trails and data events cost per event ⚠ VERIFY; S3 storage; KMS requests (use S3 Bucket Keys); Athena scans when you query; cross-Region replication if you replicate.
  • Object Lock is hard to undo. In compliance mode no one, not even the root user, can shorten retention or delete a locked version until it expires. Choose it deliberately.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Activity Log with diagnostic settings sent to a central Log Analytics workspace or storage accountCloudTrail organisation trail to the log archive bucketAWS logs are files in S3 you query later; Azure defaults to a workspace you query immediately
Immutable blob storage (WORM) policyS3 Object LockObject Lock must be enabled at bucket creation or enabled later with care ⚠ VERIFY; compliance mode is not reversible
Separate "logging" subscriptionLog archive accountSame idea; AWS accounts are a stronger wall
Management group diagnostic settings via Azure PolicyOrganisation trail (no per-account setup)One trail covers all accounts; no policy needed to roll it out
Microsoft Sentinel ingesting logsSecurity Lake, OpenSearch, a SIEM, or AthenaNot covered here; Security Lake can normalise logs to OCSF ⚠ VERIFY
Log Analytics retention (30 days default, up to years)S3 lifecycle and Object Lock retentionYou manage tiers and deletion yourself

Architecture

The security-boundary view: who can do what to the archive.

PrincipalWriteReadDelete or alter
AWS services (CloudTrail, Config)Yes, only for the right trail ARN and prefix (bucket policy conditions)NoNo
Workload engineersNoNoNo
Auditor role (read-only permission set)NoYes, via Athena/S3 readNo
Security analystsNoYesNo
Log archive administratorsPolicy and key managementYesNo: Object Lock, plus a deny in the bucket policy
Break-glass roleYes, rarelyYesOnly per governance-mode bypass or never, depending on mode

Step-by-step implementation

Agree retention with the people who decide

Records management, legal, privacy and security decide retention. Engineers implement it. Capture it in a table (see Banking use case) with the reason for each period. Logs contain personal data (user IDs, IP addresses, resource names), so under UK GDPR you need a justification for how long you keep them. Do not state a legal conclusion yourself.

Create the log archive account

In the Security OU (lesson Organizations structure). Give it its own e-mail distribution list, MFA on root, no workloads. Create IAM Identity Center permission sets: LogArchiveReadOnly for auditors and LogArchiveAdmin for a tiny group.

Create the KMS key in the log archive account

Symmetric, customer managed, automatic rotation on. The key policy lets the account's administrators manage it, lets cloudtrail.amazonaws.com call kms:GenerateDataKey* conditioned on your trail's ARN, and lets the auditor role kms:Decrypt. No workload account principal gets a statement.

Create the bucket

Object Lock enabled at creation (this also enables versioning), Block Public Access on, SSE-KMS with the key and S3 Bucket Keys, TLS-only bucket policy. Default retention: governance mode, one day in the lab. Add a lifecycle rule to move older logs to a cheaper class and to expire after the retention you agreed.

Write the bucket policy

Allow s3:GetBucketAcl on the bucket and s3:PutObject on AWSLogs/<org-id>/*, both conditioned with aws:SourceArn equal to the trail and s3:x-amz-acl equal to bucket-owner-full-control for writes. Deny non-TLS requests. Do not add any delete permission.

Enable trusted access and create the organisation trail

In the management account, enable trusted access for CloudTrail, then create a multi-Region organisation trail with management events (read and write), log file validation on, the KMS key and the bucket. Optionally delegate CloudTrail administration to the security tooling account.

Protect it with the guardrails

Attach the baseline SCP (the ProtectAuditTrail statement) to all OUs. Add an alert for any StopLogging, DeleteTrail, UpdateTrail, PutEventSelectors, bucket policy change and ScheduleKeyDeletion on the key. See the lab for the tested SCP.

Verify and keep the evidence

Run the health check below, then aws cloudtrail validate-logs over a window, and file the output. Run both on a schedule.

Test a retrieval as the auditor

Sign in with the auditor permission set, find yesterday's events for one account, and decrypt a file. An archive nobody has ever read is an assumption, not a control.

AWS Console instructions

  1. Management account → CloudTrail → Trails → Create trail. Name hl-org-trail. Tick Enable for all accounts in my organization. Choose the existing bucket (in the log archive account, by name) and the existing KMS key (by ARN). Enable Log file validation. Keep Management events (Read and Write). Create.
  2. Log archive account → S3 → bucket → Properties. Check Object Lock (default retention), Versioning, Default encryption and Block Public Access. Permissions shows the bucket policy.
  3. Any member account → CloudTrail → Trails. You see hl-org-trail marked as an organisation trail; try to stop it and notice you cannot.
  4. Management account → CloudTrail → Trails → hl-org-trail shows delivery status and any latest delivery error.

AWS CLI

# 1. Allow CloudTrail to work across the organisation (management account)
aws organizations enable-aws-service-access --service-principal cloudtrail.amazonaws.com --profile learn-management

# 2. Create the organisation trail (the bucket and key already exist in the log archive account)
aws cloudtrail create-trail --name hl-org-trail \
  --s3-bucket-name hl-org-logs-<log-archive-account-id>-us-east-1 \
  --kms-key-id arn:aws:kms:us-east-1:<log-archive-account-id>:key/<key-id> \
  --is-organization-trail --is-multi-region-trail --include-global-service-events \
  --enable-log-file-validation --profile learn-management

# 3. Start logging (a new trail does not log until you do)
aws cloudtrail start-logging --name hl-org-trail --profile learn-management

# 4. Is it healthy? Look at IsLogging and LatestDeliveryError
aws cloudtrail get-trail-status --name hl-org-trail --profile learn-management

# 5. Prove nothing was tampered with over the last day (add --verbose to see each file)
aws cloudtrail validate-logs --trail-arn arn:aws:cloudtrail:us-east-1:<mgmt-account-id>:trail/hl-org-trail \
  --start-time "$(date -u -d '1 day ago' +%Y-%m-%dT%H:%M:%SZ)" --profile learn-management

# 6. Check the archive bucket's lock and encryption (log archive account)
aws s3api get-object-lock-configuration --bucket hl-org-logs-<log-archive-account-id>-us-east-1 --profile learn-log-archive
aws s3api get-bucket-encryption --bucket hl-org-logs-<log-archive-account-id>-us-east-1 --profile learn-log-archive

# 7. Optional: delegate CloudTrail administration to the security tooling account (⚠ VERIFY command name)
aws cloudtrail register-organization-delegated-admin --member-account-id <security-tooling-account-id> --profile learn-management

validate-logs can only validate logs for a trail whose digest files exist, and the machine running it needs permission to read the digest and log files from the bucket. Run it from an auditor role in the log archive account, not from the management account.

Infrastructure as Code

The Terraform below passed terraform validate on 1.14 with AWS provider v6; the C# compiled against Amazon.CDK.Lib 2.277.0. Neither has been deployed from this course.

terraform {
  required_version = ">= 1.10"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

variable "log_archive_profile" {
  type    = string
  default = "learn-log-archive"
}

variable "management_profile" {
  type    = string
  default = "learn-management"
}

variable "trail_name" {
  type    = string
  default = "hl-org-trail"
}

provider "aws" {
  alias   = "management"
  region  = "us-east-1"
  profile = var.management_profile
}

provider "aws" {
  alias   = "log_archive"
  region  = "us-east-1"
  profile = var.log_archive_profile
}

data "aws_caller_identity" "mgmt" {
  provider = aws.management
}

data "aws_caller_identity" "log" {
  provider = aws.log_archive
}

data "aws_organizations_organization" "this" {
  provider = aws.management
}

locals {
  trail_arn   = "arn:aws:cloudtrail:us-east-1:${data.aws_caller_identity.mgmt.account_id}:trail/${var.trail_name}"
  bucket_name = "hl-org-logs-${data.aws_caller_identity.log.account_id}-us-east-1"
  org_id      = data.aws_organizations_organization.this.id
}

# ---- LOG ARCHIVE ACCOUNT -------------------------------------------------

data "aws_iam_policy_document" "key" {
  provider = aws.log_archive

  statement {
    sid       = "AccountAdministration"
    actions   = ["kms:*"]
    resources = ["*"]
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.log.account_id}:root"]
    }
  }

  statement {
    sid       = "AllowCloudTrailToEncrypt"
    actions   = ["kms:GenerateDataKey*"]
    resources = ["*"]
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = [local.trail_arn]
    }
  }
}

resource "aws_kms_key" "logs" {
  provider            = aws.log_archive
  description         = "Harbourline organisation log key"
  enable_key_rotation = true
  policy              = data.aws_iam_policy_document.key.json
}

resource "aws_s3_bucket" "logs" {
  provider            = aws.log_archive
  bucket              = local.bucket_name
  object_lock_enabled = true
}

resource "aws_s3_bucket_versioning" "logs" {
  provider = aws.log_archive
  bucket   = aws_s3_bucket.logs.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_object_lock_configuration" "logs" {
  provider = aws.log_archive
  bucket   = aws_s3_bucket.logs.id
  rule {
    default_retention {
      mode = "GOVERNANCE" # LAB shortcut. Production: COMPLIANCE for your records-management period.
      days = 1
    }
  }
  depends_on = [aws_s3_bucket_versioning.logs]
}

resource "aws_s3_bucket_server_side_encryption_configuration" "logs" {
  provider = aws.log_archive
  bucket   = aws_s3_bucket.logs.id
  rule {
    bucket_key_enabled = true
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.logs.arn
    }
  }
}

resource "aws_s3_bucket_public_access_block" "logs" {
  provider                = aws.log_archive
  bucket                  = aws_s3_bucket.logs.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

data "aws_iam_policy_document" "bucket" {
  provider = aws.log_archive

  statement {
    sid       = "AllowCloudTrailAclCheck"
    actions   = ["s3:GetBucketAcl"]
    resources = [aws_s3_bucket.logs.arn]
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = [local.trail_arn]
    }
  }

  statement {
    sid       = "AllowCloudTrailWrite"
    actions   = ["s3:PutObject"]
    resources = ["${aws_s3_bucket.logs.arn}/AWSLogs/${local.org_id}/*"]
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "s3:x-amz-acl"
      values   = ["bucket-owner-full-control"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = [local.trail_arn]
    }
  }

  statement {
    sid       = "DenyInsecureTransport"
    effect    = "Deny"
    actions   = ["s3:*"]
    resources = [aws_s3_bucket.logs.arn, "${aws_s3_bucket.logs.arn}/*"]
    principals {
      type        = "*"
      identifiers = ["*"]
    }
    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }
}

resource "aws_s3_bucket_policy" "logs" {
  provider   = aws.log_archive
  bucket     = aws_s3_bucket.logs.id
  policy     = data.aws_iam_policy_document.bucket.json
  depends_on = [aws_s3_bucket_public_access_block.logs]
}

resource "aws_s3_bucket_lifecycle_configuration" "logs" {
  provider = aws.log_archive
  bucket   = aws_s3_bucket.logs.id
  rule {
    id     = "tier-and-expire"
    status = "Enabled"
    filter {}
    transition {
      days          = 90
      storage_class = "GLACIER_IR"
    }
    expiration {
      days = 2555 # about 7 years: a placeholder your records-management policy must replace
    }
  }
  depends_on = [aws_s3_bucket_versioning.logs]
}

# ---- MANAGEMENT ACCOUNT --------------------------------------------------

resource "aws_cloudtrail" "org" {
  provider                      = aws.management
  name                          = var.trail_name
  s3_bucket_name                = aws_s3_bucket.logs.id
  kms_key_id                    = aws_kms_key.logs.arn
  is_organization_trail         = true
  is_multi_region_trail         = true
  include_global_service_events = true
  enable_log_file_validation    = true

  depends_on = [aws_s3_bucket_policy.logs]
}

Complete code

A health check for the organisation trail. It exits 0 when healthy and 1 otherwise, so you can run it daily from a pipeline. It compiles against AWSSDK.CloudTrail 4.1.x on .NET 10.

dotnet new console -n TrailCheck
cd TrailCheck
dotnet add package AWSSDK.CloudTrail
Program.cs
using Amazon.CloudTrail;
using Amazon.CloudTrail.Model;

// Does the organisation have a healthy organisation trail? Exit code 0 = yes, 1 = problems.
// Run in the MANAGEMENT account (or the CloudTrail delegated administrator).
var trail = new AmazonCloudTrailClient();
var problems = new List<string>();

var described = await trail.DescribeTrailsAsync(new DescribeTrailsRequest { IncludeShadowTrails = false });
var orgTrails = (described.TrailList ?? []).Where(t => t.IsOrganizationTrail == true).ToList();

if (orgTrails.Count == 0) problems.Add("No organisation trail exists.");

foreach (var t in orgTrails)
{
    Console.WriteLine($"Trail {t.Name} -> s3://{t.S3BucketName}");

    if (t.IsMultiRegionTrail != true) problems.Add($"{t.Name}: not multi-Region, so other Regions are blind.");
    if (t.LogFileValidationEnabled != true) problems.Add($"{t.Name}: log file validation is off, so tampering cannot be detected.");
    if (string.IsNullOrEmpty(t.KmsKeyId)) problems.Add($"{t.Name}: logs are not encrypted with a customer managed KMS key.");

    var status = await trail.GetTrailStatusAsync(new GetTrailStatusRequest { Name = t.TrailARN });
    if (status.IsLogging != true) problems.Add($"{t.Name}: logging is OFF.");
    if (!string.IsNullOrEmpty(status.LatestDeliveryError)) problems.Add($"{t.Name}: last delivery error: {status.LatestDeliveryError}");
}

foreach (var p in problems) Console.WriteLine($"PROBLEM  {p}");
Console.WriteLine(problems.Count == 0 ? "OK: organisation trail looks healthy." : $"{problems.Count} problem(s).");
return problems.Count == 0 ? 0 : 1;

Code explained line by line

  1. DescribeTrails with IncludeShadowTrails = false returns trails owned by this account. Shadow trails are replicated copies of multi-Region trails in other Regions; excluding them avoids duplicates.
  2. Where(t => t.IsOrganizationTrail == true) keeps organisation trails. IsOrganizationTrail is a nullable bool in SDK v4, so compare with == true.
  3. Three configuration checks follow: multi-Region (otherwise attackers use an unwatched Region), log file validation (otherwise you cannot prove integrity), and a KMS key (otherwise you rely on S3-managed keys only, which your policy may not allow).
  4. GetTrailStatus is the runtime check. IsLogging can be false even when the trail exists, for example after a StopLogging call. LatestDeliveryError is non-empty when the bucket policy, KMS key policy or Region is wrong.
  5. The program prints each problem and returns a non-zero exit code. Wire the result to an alert, not a dashboard nobody opens.
  6. It uses the default credential chain. Run it with AWS_PROFILE=learn-management; no keys are in code.

Security considerations

⚠ Learning shortcut — not for production

The lab uses Object Lock in governance mode for one day so you can clean up. Governance mode lets a principal with s3:BypassGovernanceRetention delete locked objects, which means the archive is not truly immutable. Production uses compliance mode with your records-management retention, after a risk and legal sign-off, and tests the whole design in a throwaway account first because compliance mode cannot be undone. The lab also runs the Terraform from your laptop; a bank runs it from a pipeline role.

  • The log archive account is tier-0. Few administrators, hardware MFA, no workloads, alerts on every change to the bucket policy, key policy, Object Lock configuration and lifecycle rules.
  • Bucket policy discipline. Allow only the service principals and conditions you need. Add aws:SourceArn and aws:SourceOrgID style conditions where supported to prevent the confused-deputy problem ⚠ VERIFY exact condition keys for each service.
  • Key policy discipline. Keep decrypt rights narrow. An auditor who cannot decrypt cannot audit. An engineer who can decrypt can read other teams' activity.
  • Privacy. Logs contain personal data (principal names, IP addresses, sometimes request parameters). Apply data minimisation, access logging and retention with your privacy team. Do not log secrets or full payloads in application logs.
  • Integrity is not completeness. Validation proves that delivered files were not altered. It does not prove that all events were delivered or that every service was logged. Combine it with the health check and coverage reviews.
  • Cross-Region. A single-Region archive is a single point of failure for evidence. Consider replicating to a second Region or account with Object Lock; replication has its own rules ⚠ VERIFY.
  • Do not claim compliance. "Logs are immutable and retained 7 years" is a design fact. Whether it satisfies a regulator or the card schemes' requirements is a judgement for your compliance team and, for PCI DSS, the assessor.

Banking use case

Harbourline's retention table (fictional; the numbers are placeholders that records management and legal must set):

LogHot (S3 Standard)Cooler tierDeleted afterReason
CloudTrail management events90 daysGlacier Instant Retrieval7 years ⚠ VERIFY with legalFinancial services record-keeping and investigation windows
CloudTrail data events (sensitive buckets only)90 daysGlacier Instant Retrieval2 yearsNarrower scope, high volume
Config history1 yearGlacier Instant Retrieval7 years ⚠ VERIFYEvidence of configuration over time
VPC Flow Logs30 daysNone13 monthsInvestigations; storage minimisation
Application logs30 days (CloudWatch)None90 daysMay contain personal data

For card-related systems, PCI DSS audit log retention requirements apply (at least 12 months of history with the most recent three months immediately available, in the version current at the time) ⚠ VERIFY the requirement text for your PCI DSS version with the assessor.

An incident drill: the security team simulates a compromised role in payments-prod that tries to stop CloudTrail. The SCP blocks the call (ProtectAuditTrail), the denied attempt itself appears in the organisation trail in the log archive, an EventBridge rule pages the SOC, and the analyst reconstructs the role's last six hours with Athena in the security tooling account. The bank rehearses this quarterly and keeps the evidence.

Common mistakes

  • Logs in the same account as the workload. Deliverable to attackers.
  • Forgetting start-logging after create-trail with the CLI, so no events are recorded.
  • Wrong bucket policy path. Organisation trails write under AWSLogs/<org-id>/; a policy that only allows AWSLogs/<account-id>/ fails silently with a delivery error.
  • KMS key in the management account. The log archive should own the key so the management account is not a dependency of the evidence store.
  • Enabling data events for everything. Cost explodes and the signal drowns. Scope them.
  • Choosing compliance-mode Object Lock in a lab. You cannot delete the bucket for the whole retention period.
  • Never testing a read. The first time someone reads the archive should not be during an investigation.
  • Retention "forever". Cost, privacy and discovery risk. Decide a period and expire.

Troubleshooting

SymptomLikely causeFix
InsufficientS3BucketPolicyException on create-trailBucket policy does not allow CloudTrail or has the wrong pathCompare with the policy above; use the organisation ID path
InsufficientEncryptionPolicyExceptionThe KMS key policy lacks the CloudTrail statement or the trail ARN condition does not matchFix the key policy; confirm the trail name and Region in the ARN
get-trail-status shows LatestDeliveryErrorBucket policy, key policy or Object Lock default retention issueRead the error; fix; wait for the next delivery
Member account cannot see events of other accountsEvent history is per accountQuery the S3 logs with Athena from the security tooling account
validate-logs says digest files missingLog file validation was off at the time, or the time window is before the trail existedCheck LogFileValidationEnabled and the window
Cannot delete the bucketObject Lock retention still active on versionsWait, or (governance mode) delete versions with --bypass-governance-retention and the right permission
AccessDenied for the auditor reading logsKMS decrypt missing on the key policyAdd the auditor role to the key policy
Unexpected Cost Explorer line for CloudTrailExtra trails, or data eventsList trails and event selectors; remove duplicates

Cost and cleanup

Cost label: credits. With one organisation trail the first copy of management events is free ⚠ VERIFY. Expected lab cost is cents: a little S3 storage, KMS requests (reduced by Bucket Keys) and one KMS key (about USD 1 per month while it exists ⚠ VERIFY). Data events are the trap: charged per event ⚠ VERIFY the current rate. Do not enable them in the lab.

Teardown, in this order:

# 1. Stop new logs, then remove the trail (management account)
aws cloudtrail stop-logging --name hl-org-trail --profile learn-management
aws cloudtrail delete-trail --name hl-org-trail --profile learn-management

# 2. Empty the bucket including versions. Governance-mode locks need the bypass flag AND the s3:BypassGovernanceRetention permission.
#    (If the lock is still active and you lack the permission, wait one day.) Repeat until empty.
aws s3api list-object-versions --bucket hl-org-logs-<log-archive-account-id>-us-east-1 --profile learn-log-archive \
  --query "{Objects: Versions[].{Key:Key,VersionId:VersionId}}" --output json > versions.json
aws s3api delete-objects --bucket hl-org-logs-<log-archive-account-id>-us-east-1 --delete file://versions.json \
  --bypass-governance-retention --profile learn-log-archive

# 3. Destroy the rest, or delete the bucket by hand
terraform destroy

# 4. Schedule the KMS key for deletion (7 is the minimum waiting period)
aws kms schedule-key-deletion --key-id <key-id> --pending-window-in-days 7 --profile learn-log-archive

# 5. Check that nothing tagged for the lab remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab --region us-east-1 --profile learn-log-archive

Also delete delete markers (list-object-versions returns DeleteMarkers separately) if the bucket will not delete. Disable trusted access for CloudTrail only if nothing else depends on it.

Exercises

  • 🟢 Guided: Create the bucket and key with Terraform in your log archive (or sandbox) account, set governance mode for one day, and run get-object-lock-configuration and get-bucket-encryption. Run the health check against a normal trail first to see the "not an organisation trail" problem.
  • 🟡 Challenge: Write an EventBridge rule (as Terraform or CDK) that matches StopLogging, DeleteTrail, UpdateTrail and PutEventSelectors events and sends them to an SNS topic. State where it must be deployed (every account? one?) and why.
  • 🔴 Live problem: An auditor asks you to prove that no events from payments-prod between 03:00 and 05:00 last Tuesday were altered, and to list every AssumeRole into the break-glass role in that window. Write the steps, the commands and the evidence you would hand over.

Assignment

Add docs/logging-design.md to your portfolio: the source table, the retention table with reasons and owners, the bucket and key policy design, the SCP protection and alerting plan, and the output of the health check and validate-logs (account IDs redacted).

Solutions

Knowledge check

Interview questions

  • How do you make audit logs tamper-resistant? A separate log archive account, an organisation trail (so member accounts cannot disable it), S3 Object Lock, SSE-KMS with a key the workloads cannot touch, log file validation, SCPs on the trail and bucket actions, and alerts on changes.
  • How long should we keep logs? Ask records management, legal and privacy; set per log type with a stated reason; expire at the end. Do not keep forever.
  • What are the cost traps in centralised logging? Data events, extra trail copies, KMS requests without Bucket Keys, flow log volume, Athena full scans.
  • How do you know the logging pipeline is working? A scheduled health check (trail logging, no delivery errors), regular validate-logs, coverage checks that each account and Region appears in the bucket, and a periodic auditor read test.

Certification practice

Completion checklist

  • I can list at least six log sources and where each should land
  • I created the bucket and key, and verified Object Lock and encryption
  • I ran the health check and understand each problem it reports
  • I can explain governance versus compliance mode and why the lab uses governance
  • I wrote the retention table with owners and reasons
  • I ran the teardown and confirmed no billing resources remain

Key takeaways

  • Logs must leave the account, land where workloads cannot reach, and resist deletion.
  • Organisation trails cover every account and cannot be stopped by members.
  • Retention is a business, legal and privacy decision, with a reason for every period.
  • Data events and KMS requests are the cost traps; scope them.
  • Test reads and run validation on a schedule; an unread archive is an assumption.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page