Centralised logging: an organisation trail, a log archive account and retention you can defend
Design where every account's audit logs go, who can write, read and delete them, how long they are kept and what that costs, then build an organisation trail into an encrypted, Object Lock protected log archive bucket and verify it with a tested check.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.CloudTrail 4.1.x) · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform 1.14 / AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- List the log sources a bank must centralise and say, for each, where it lands and what protects it.
- Design the log archive account so that producers can write, auditors can read, and nobody can delete or rewrite.
- Create an organisation trail that covers every account and Region, delivers to a KMS-encrypted S3 bucket with Object Lock, and has log file validation on.
- Choose retention periods with the records, legal and privacy teams, and explain why governance mode is for the lab and compliance mode needs a decision.
- Estimate the cost of centralised logging, including the parts that surprise people (data events, KMS requests, extra trail copies).
- Run a tested .NET check that tells you whether the organisation trail is healthy, and tear the lab down cleanly.
Prerequisites
- CloudTrail for banks: how CloudTrail works, event types, digest files. This lesson is the organisation-level design; it assumes you have read that one.
- S3 protection, Object Lock and Macie and Audit logging and evidence collection.
- AWS Organizations structure: you need a Security OU and a log archive account.
- Guardrails with SCPs and RCPs: the
ProtectAuditTrailstatement protects what you build here. - Tools: AWS CLI v2, SSO profiles for the management and log-archive accounts, Terraform 1.10+ or CDK, .NET 10 SDK.
Why this matters in banking
Imagine an attacker who steals an engineer's session in payments-prod. The first thing a competent attacker does is stop the evidence. If CloudTrail logs live in the same account, in a bucket the engineer's role can administer, the attacker deletes them and the bank cannot answer the regulator's first three questions: what did they do, which data did they touch, when did it start.
Centralised logging answers that structurally. Logs leave the account as they are written, travel to an account the workload cannot reach, land in storage that cannot be altered for a set period, and are checkable for tampering. Supervisors and auditors also ask the dull questions: how long do you keep it, who can read it, how do you know it is complete. This lesson gives you answers you can demonstrate rather than assert.
The idea in plain language
Think of logging as a letterbox into a sealed records room. Every account has a slot (services deliver logs on its behalf). The room is in a different building (the log archive account). Staff can post letters but cannot open the room. Auditors have a key to read, not to remove. A tamper-evident seal on every bundle (log file validation) tells you if anything was swapped. Nobody, not even the bank's most senior engineer, can shred a letter before its retention date (Object Lock).
Analogy: a bank's branch posts its daily ledger copies through a slot to head-office archives. The branch cannot retrieve or burn them. If the branch manager claims "that day's ledger never existed", the archive's signed digest proves otherwise.
What to centralise
| Source | Produced by | Lands in | Notes |
|---|---|---|---|
| Management events (API activity) | CloudTrail organisation trail | Log archive S3 bucket | The audit trail. First copy of management events is free per trail ⚠ VERIFY current pricing |
| Data events (S3 object, Lambda invoke, DynamoDB item) | CloudTrail, advanced event selectors | Same bucket, or a separate trail and bucket | Charged per event; enable only for sensitive resources |
| Configuration history and snapshots | AWS Config delivery channel | Log archive bucket (or a dedicated one) | Records what changed; see Config conformance packs |
| VPC Flow Logs | Each VPC | Log archive bucket or central network account | Volume can be large; sample or filter |
| Load balancer, S3 server access, CloudFront logs | The service | Per-service bucket in log archive | Often forgotten |
| DNS query logs | Route 53 Resolver | Log archive bucket or CloudWatch Logs | Useful for exfiltration hunting |
| WAF logs | AWS WAF | Log archive bucket via Firehose | Name format restrictions apply ⚠ VERIFY |
| Findings export | GuardDuty / Security Hub | S3 in the security tooling or log archive account | Covered in Centralised security tooling |
| Application logs | Your .NET services | CloudWatch Logs, then forwarded | Different retention; may contain personal data |
CloudWatch Logs can also be centralised across accounts. AWS has been adding organisation-level log centralisation features; check the current CloudWatch documentation before designing around one ⚠ VERIFY.
What problem the AWS service solves
CloudTrail by itself records API activity per account. An organisation trail, created in the management account or by a CloudTrail delegated administrator, applies to every account in the organisation, including accounts added later, with a copy of the trail appearing in each member account automatically. Member accounts can see the trail but cannot stop it, delete it or change what it logs. That is the key property: the control sits above the account.
Facts worth knowing (from the CloudTrail user guide, checked 2026-11-18):
- The console creates multi-Region organisation trails; the CLI lets you choose single or multi-Region.
- Logs go to one S3 bucket, with a path that includes the organisation ID and each account ID (
AWSLogs/<org-id>/<account-id>/...). - Event history in each account's console shows only that account's events (last 90 days of management events). Organisation-wide analysis needs the S3 logs (Athena) or CloudTrail Lake ⚠ VERIFY availability and pricing before relying on Lake.
- If an organisation trail's bucket policy, SNS policy or KMS permission is wrong, CloudTrail still creates the member-account trails, but delivery fails. Check
get-trail-status. - Removing an account from the organisation deletes its copy of the trail and stops logging for it. Logs already delivered stay in the bucket.
- If the management account that created the trail stops being the management account, the trail becomes a normal trail.
Limits and honest costs
- Opt-in Regions need care: a multi-Region trail whose home Region is an opt-in Region only logs member accounts that enabled that Region.
- Cost drivers: the first copy of management events is free; extra trails and data events cost per event ⚠ VERIFY; S3 storage; KMS requests (use S3 Bucket Keys); Athena scans when you query; cross-Region replication if you replicate.
- Object Lock is hard to undo. In compliance mode no one, not even the root user, can shorten retention or delete a locked version until it expires. Choose it deliberately.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Activity Log with diagnostic settings sent to a central Log Analytics workspace or storage account | CloudTrail organisation trail to the log archive bucket | AWS logs are files in S3 you query later; Azure defaults to a workspace you query immediately |
| Immutable blob storage (WORM) policy | S3 Object Lock | Object Lock must be enabled at bucket creation or enabled later with care ⚠ VERIFY; compliance mode is not reversible |
| Separate "logging" subscription | Log archive account | Same idea; AWS accounts are a stronger wall |
| Management group diagnostic settings via Azure Policy | Organisation trail (no per-account setup) | One trail covers all accounts; no policy needed to roll it out |
| Microsoft Sentinel ingesting logs | Security Lake, OpenSearch, a SIEM, or Athena | Not covered here; Security Lake can normalise logs to OCSF ⚠ VERIFY |
| Log Analytics retention (30 days default, up to years) | S3 lifecycle and Object Lock retention | You manage tiers and deletion yourself |
Architecture
The security-boundary view: who can do what to the archive.
| Principal | Write | Read | Delete or alter |
|---|---|---|---|
| AWS services (CloudTrail, Config) | Yes, only for the right trail ARN and prefix (bucket policy conditions) | No | No |
| Workload engineers | No | No | No |
| Auditor role (read-only permission set) | No | Yes, via Athena/S3 read | No |
| Security analysts | No | Yes | No |
| Log archive administrators | Policy and key management | Yes | No: Object Lock, plus a deny in the bucket policy |
| Break-glass role | Yes, rarely | Yes | Only per governance-mode bypass or never, depending on mode |
Step-by-step implementation
Agree retention with the people who decide
Records management, legal, privacy and security decide retention. Engineers implement it. Capture it in a table (see Banking use case) with the reason for each period. Logs contain personal data (user IDs, IP addresses, resource names), so under UK GDPR you need a justification for how long you keep them. Do not state a legal conclusion yourself.
Create the log archive account
In the Security OU (lesson Organizations structure). Give it its own e-mail distribution list, MFA on root, no workloads. Create IAM Identity Center permission sets: LogArchiveReadOnly for auditors and LogArchiveAdmin for a tiny group.
Create the KMS key in the log archive account
Symmetric, customer managed, automatic rotation on. The key policy lets the account's administrators manage it, lets cloudtrail.amazonaws.com call kms:GenerateDataKey* conditioned on your trail's ARN, and lets the auditor role kms:Decrypt. No workload account principal gets a statement.
Create the bucket
Object Lock enabled at creation (this also enables versioning), Block Public Access on, SSE-KMS with the key and S3 Bucket Keys, TLS-only bucket policy. Default retention: governance mode, one day in the lab. Add a lifecycle rule to move older logs to a cheaper class and to expire after the retention you agreed.
Write the bucket policy
Allow s3:GetBucketAcl on the bucket and s3:PutObject on AWSLogs/<org-id>/*, both conditioned with aws:SourceArn equal to the trail and s3:x-amz-acl equal to bucket-owner-full-control for writes. Deny non-TLS requests. Do not add any delete permission.
Enable trusted access and create the organisation trail
In the management account, enable trusted access for CloudTrail, then create a multi-Region organisation trail with management events (read and write), log file validation on, the KMS key and the bucket. Optionally delegate CloudTrail administration to the security tooling account.
Protect it with the guardrails
Attach the baseline SCP (the ProtectAuditTrail statement) to all OUs. Add an alert for any StopLogging, DeleteTrail, UpdateTrail, PutEventSelectors, bucket policy change and ScheduleKeyDeletion on the key. See the lab for the tested SCP.
Verify and keep the evidence
Run the health check below, then aws cloudtrail validate-logs over a window, and file the output. Run both on a schedule.
Test a retrieval as the auditor
Sign in with the auditor permission set, find yesterday's events for one account, and decrypt a file. An archive nobody has ever read is an assumption, not a control.
AWS Console instructions
- Management account → CloudTrail → Trails → Create trail. Name
hl-org-trail. Tick Enable for all accounts in my organization. Choose the existing bucket (in the log archive account, by name) and the existing KMS key (by ARN). Enable Log file validation. Keep Management events (Read and Write). Create. - Log archive account → S3 → bucket → Properties. Check Object Lock (default retention), Versioning, Default encryption and Block Public Access. Permissions shows the bucket policy.
- Any member account → CloudTrail → Trails. You see
hl-org-trailmarked as an organisation trail; try to stop it and notice you cannot. - Management account → CloudTrail → Trails → hl-org-trail shows delivery status and any latest delivery error.
AWS CLI
# 1. Allow CloudTrail to work across the organisation (management account)
aws organizations enable-aws-service-access --service-principal cloudtrail.amazonaws.com --profile learn-management
# 2. Create the organisation trail (the bucket and key already exist in the log archive account)
aws cloudtrail create-trail --name hl-org-trail \
--s3-bucket-name hl-org-logs-<log-archive-account-id>-us-east-1 \
--kms-key-id arn:aws:kms:us-east-1:<log-archive-account-id>:key/<key-id> \
--is-organization-trail --is-multi-region-trail --include-global-service-events \
--enable-log-file-validation --profile learn-management
# 3. Start logging (a new trail does not log until you do)
aws cloudtrail start-logging --name hl-org-trail --profile learn-management
# 4. Is it healthy? Look at IsLogging and LatestDeliveryError
aws cloudtrail get-trail-status --name hl-org-trail --profile learn-management
# 5. Prove nothing was tampered with over the last day (add --verbose to see each file)
aws cloudtrail validate-logs --trail-arn arn:aws:cloudtrail:us-east-1:<mgmt-account-id>:trail/hl-org-trail \
--start-time "$(date -u -d '1 day ago' +%Y-%m-%dT%H:%M:%SZ)" --profile learn-management
# 6. Check the archive bucket's lock and encryption (log archive account)
aws s3api get-object-lock-configuration --bucket hl-org-logs-<log-archive-account-id>-us-east-1 --profile learn-log-archive
aws s3api get-bucket-encryption --bucket hl-org-logs-<log-archive-account-id>-us-east-1 --profile learn-log-archive
# 7. Optional: delegate CloudTrail administration to the security tooling account (⚠ VERIFY command name)
aws cloudtrail register-organization-delegated-admin --member-account-id <security-tooling-account-id> --profile learn-managementvalidate-logs can only validate logs for a trail whose digest files exist, and the machine running it needs permission to read the digest and log files from the bucket. Run it from an auditor role in the log archive account, not from the management account.
Infrastructure as Code
The Terraform below passed terraform validate on 1.14 with AWS provider v6; the C# compiled against Amazon.CDK.Lib 2.277.0. Neither has been deployed from this course.
terraform {
required_version = ">= 1.10"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 6.0"
}
}
}
variable "log_archive_profile" {
type = string
default = "learn-log-archive"
}
variable "management_profile" {
type = string
default = "learn-management"
}
variable "trail_name" {
type = string
default = "hl-org-trail"
}
provider "aws" {
alias = "management"
region = "us-east-1"
profile = var.management_profile
}
provider "aws" {
alias = "log_archive"
region = "us-east-1"
profile = var.log_archive_profile
}
data "aws_caller_identity" "mgmt" {
provider = aws.management
}
data "aws_caller_identity" "log" {
provider = aws.log_archive
}
data "aws_organizations_organization" "this" {
provider = aws.management
}
locals {
trail_arn = "arn:aws:cloudtrail:us-east-1:${data.aws_caller_identity.mgmt.account_id}:trail/${var.trail_name}"
bucket_name = "hl-org-logs-${data.aws_caller_identity.log.account_id}-us-east-1"
org_id = data.aws_organizations_organization.this.id
}
# ---- LOG ARCHIVE ACCOUNT -------------------------------------------------
data "aws_iam_policy_document" "key" {
provider = aws.log_archive
statement {
sid = "AccountAdministration"
actions = ["kms:*"]
resources = ["*"]
principals {
type = "AWS"
identifiers = ["arn:aws:iam::${data.aws_caller_identity.log.account_id}:root"]
}
}
statement {
sid = "AllowCloudTrailToEncrypt"
actions = ["kms:GenerateDataKey*"]
resources = ["*"]
principals {
type = "Service"
identifiers = ["cloudtrail.amazonaws.com"]
}
condition {
test = "StringEquals"
variable = "aws:SourceArn"
values = [local.trail_arn]
}
}
}
resource "aws_kms_key" "logs" {
provider = aws.log_archive
description = "Harbourline organisation log key"
enable_key_rotation = true
policy = data.aws_iam_policy_document.key.json
}
resource "aws_s3_bucket" "logs" {
provider = aws.log_archive
bucket = local.bucket_name
object_lock_enabled = true
}
resource "aws_s3_bucket_versioning" "logs" {
provider = aws.log_archive
bucket = aws_s3_bucket.logs.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_object_lock_configuration" "logs" {
provider = aws.log_archive
bucket = aws_s3_bucket.logs.id
rule {
default_retention {
mode = "GOVERNANCE" # LAB shortcut. Production: COMPLIANCE for your records-management period.
days = 1
}
}
depends_on = [aws_s3_bucket_versioning.logs]
}
resource "aws_s3_bucket_server_side_encryption_configuration" "logs" {
provider = aws.log_archive
bucket = aws_s3_bucket.logs.id
rule {
bucket_key_enabled = true
apply_server_side_encryption_by_default {
sse_algorithm = "aws:kms"
kms_master_key_id = aws_kms_key.logs.arn
}
}
}
resource "aws_s3_bucket_public_access_block" "logs" {
provider = aws.log_archive
bucket = aws_s3_bucket.logs.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
data "aws_iam_policy_document" "bucket" {
provider = aws.log_archive
statement {
sid = "AllowCloudTrailAclCheck"
actions = ["s3:GetBucketAcl"]
resources = [aws_s3_bucket.logs.arn]
principals {
type = "Service"
identifiers = ["cloudtrail.amazonaws.com"]
}
condition {
test = "StringEquals"
variable = "aws:SourceArn"
values = [local.trail_arn]
}
}
statement {
sid = "AllowCloudTrailWrite"
actions = ["s3:PutObject"]
resources = ["${aws_s3_bucket.logs.arn}/AWSLogs/${local.org_id}/*"]
principals {
type = "Service"
identifiers = ["cloudtrail.amazonaws.com"]
}
condition {
test = "StringEquals"
variable = "s3:x-amz-acl"
values = ["bucket-owner-full-control"]
}
condition {
test = "StringEquals"
variable = "aws:SourceArn"
values = [local.trail_arn]
}
}
statement {
sid = "DenyInsecureTransport"
effect = "Deny"
actions = ["s3:*"]
resources = [aws_s3_bucket.logs.arn, "${aws_s3_bucket.logs.arn}/*"]
principals {
type = "*"
identifiers = ["*"]
}
condition {
test = "Bool"
variable = "aws:SecureTransport"
values = ["false"]
}
}
}
resource "aws_s3_bucket_policy" "logs" {
provider = aws.log_archive
bucket = aws_s3_bucket.logs.id
policy = data.aws_iam_policy_document.bucket.json
depends_on = [aws_s3_bucket_public_access_block.logs]
}
resource "aws_s3_bucket_lifecycle_configuration" "logs" {
provider = aws.log_archive
bucket = aws_s3_bucket.logs.id
rule {
id = "tier-and-expire"
status = "Enabled"
filter {}
transition {
days = 90
storage_class = "GLACIER_IR"
}
expiration {
days = 2555 # about 7 years: a placeholder your records-management policy must replace
}
}
depends_on = [aws_s3_bucket_versioning.logs]
}
# ---- MANAGEMENT ACCOUNT --------------------------------------------------
resource "aws_cloudtrail" "org" {
provider = aws.management
name = var.trail_name
s3_bucket_name = aws_s3_bucket.logs.id
kms_key_id = aws_kms_key.logs.arn
is_organization_trail = true
is_multi_region_trail = true
include_global_service_events = true
enable_log_file_validation = true
depends_on = [aws_s3_bucket_policy.logs]
}Complete code
A health check for the organisation trail. It exits 0 when healthy and 1 otherwise, so you can run it daily from a pipeline. It compiles against AWSSDK.CloudTrail 4.1.x on .NET 10.
dotnet new console -n TrailCheck
cd TrailCheck
dotnet add package AWSSDK.CloudTrailusing Amazon.CloudTrail;
using Amazon.CloudTrail.Model;
// Does the organisation have a healthy organisation trail? Exit code 0 = yes, 1 = problems.
// Run in the MANAGEMENT account (or the CloudTrail delegated administrator).
var trail = new AmazonCloudTrailClient();
var problems = new List<string>();
var described = await trail.DescribeTrailsAsync(new DescribeTrailsRequest { IncludeShadowTrails = false });
var orgTrails = (described.TrailList ?? []).Where(t => t.IsOrganizationTrail == true).ToList();
if (orgTrails.Count == 0) problems.Add("No organisation trail exists.");
foreach (var t in orgTrails)
{
Console.WriteLine($"Trail {t.Name} -> s3://{t.S3BucketName}");
if (t.IsMultiRegionTrail != true) problems.Add($"{t.Name}: not multi-Region, so other Regions are blind.");
if (t.LogFileValidationEnabled != true) problems.Add($"{t.Name}: log file validation is off, so tampering cannot be detected.");
if (string.IsNullOrEmpty(t.KmsKeyId)) problems.Add($"{t.Name}: logs are not encrypted with a customer managed KMS key.");
var status = await trail.GetTrailStatusAsync(new GetTrailStatusRequest { Name = t.TrailARN });
if (status.IsLogging != true) problems.Add($"{t.Name}: logging is OFF.");
if (!string.IsNullOrEmpty(status.LatestDeliveryError)) problems.Add($"{t.Name}: last delivery error: {status.LatestDeliveryError}");
}
foreach (var p in problems) Console.WriteLine($"PROBLEM {p}");
Console.WriteLine(problems.Count == 0 ? "OK: organisation trail looks healthy." : $"{problems.Count} problem(s).");
return problems.Count == 0 ? 0 : 1;Code explained line by line
DescribeTrailswithIncludeShadowTrails = falsereturns trails owned by this account. Shadow trails are replicated copies of multi-Region trails in other Regions; excluding them avoids duplicates.Where(t => t.IsOrganizationTrail == true)keeps organisation trails.IsOrganizationTrailis a nullable bool in SDK v4, so compare with== true.- Three configuration checks follow: multi-Region (otherwise attackers use an unwatched Region), log file validation (otherwise you cannot prove integrity), and a KMS key (otherwise you rely on S3-managed keys only, which your policy may not allow).
GetTrailStatusis the runtime check.IsLoggingcan be false even when the trail exists, for example after aStopLoggingcall.LatestDeliveryErroris non-empty when the bucket policy, KMS key policy or Region is wrong.- The program prints each problem and returns a non-zero exit code. Wire the result to an alert, not a dashboard nobody opens.
- It uses the default credential chain. Run it with
AWS_PROFILE=learn-management; no keys are in code.
Security considerations
⚠ Learning shortcut — not for production
The lab uses Object Lock in governance mode for one day so you can clean up. Governance mode lets a principal with s3:BypassGovernanceRetention delete locked objects, which means the archive is not truly immutable. Production uses compliance mode with your records-management retention, after a risk and legal sign-off, and tests the whole design in a throwaway account first because compliance mode cannot be undone. The lab also runs the Terraform from your laptop; a bank runs it from a pipeline role.
- The log archive account is tier-0. Few administrators, hardware MFA, no workloads, alerts on every change to the bucket policy, key policy, Object Lock configuration and lifecycle rules.
- Bucket policy discipline. Allow only the service principals and conditions you need. Add
aws:SourceArnandaws:SourceOrgIDstyle conditions where supported to prevent the confused-deputy problem ⚠ VERIFY exact condition keys for each service. - Key policy discipline. Keep decrypt rights narrow. An auditor who cannot decrypt cannot audit. An engineer who can decrypt can read other teams' activity.
- Privacy. Logs contain personal data (principal names, IP addresses, sometimes request parameters). Apply data minimisation, access logging and retention with your privacy team. Do not log secrets or full payloads in application logs.
- Integrity is not completeness. Validation proves that delivered files were not altered. It does not prove that all events were delivered or that every service was logged. Combine it with the health check and coverage reviews.
- Cross-Region. A single-Region archive is a single point of failure for evidence. Consider replicating to a second Region or account with Object Lock; replication has its own rules ⚠ VERIFY.
- Do not claim compliance. "Logs are immutable and retained 7 years" is a design fact. Whether it satisfies a regulator or the card schemes' requirements is a judgement for your compliance team and, for PCI DSS, the assessor.
Banking use case
Harbourline's retention table (fictional; the numbers are placeholders that records management and legal must set):
| Log | Hot (S3 Standard) | Cooler tier | Deleted after | Reason |
|---|---|---|---|---|
| CloudTrail management events | 90 days | Glacier Instant Retrieval | 7 years ⚠ VERIFY with legal | Financial services record-keeping and investigation windows |
| CloudTrail data events (sensitive buckets only) | 90 days | Glacier Instant Retrieval | 2 years | Narrower scope, high volume |
| Config history | 1 year | Glacier Instant Retrieval | 7 years ⚠ VERIFY | Evidence of configuration over time |
| VPC Flow Logs | 30 days | None | 13 months | Investigations; storage minimisation |
| Application logs | 30 days (CloudWatch) | None | 90 days | May contain personal data |
For card-related systems, PCI DSS audit log retention requirements apply (at least 12 months of history with the most recent three months immediately available, in the version current at the time) ⚠ VERIFY the requirement text for your PCI DSS version with the assessor.
An incident drill: the security team simulates a compromised role in payments-prod that tries to stop CloudTrail. The SCP blocks the call (ProtectAuditTrail), the denied attempt itself appears in the organisation trail in the log archive, an EventBridge rule pages the SOC, and the analyst reconstructs the role's last six hours with Athena in the security tooling account. The bank rehearses this quarterly and keeps the evidence.
Common mistakes
- Logs in the same account as the workload. Deliverable to attackers.
- Forgetting
start-loggingaftercreate-trailwith the CLI, so no events are recorded. - Wrong bucket policy path. Organisation trails write under
AWSLogs/<org-id>/; a policy that only allowsAWSLogs/<account-id>/fails silently with a delivery error. - KMS key in the management account. The log archive should own the key so the management account is not a dependency of the evidence store.
- Enabling data events for everything. Cost explodes and the signal drowns. Scope them.
- Choosing compliance-mode Object Lock in a lab. You cannot delete the bucket for the whole retention period.
- Never testing a read. The first time someone reads the archive should not be during an investigation.
- Retention "forever". Cost, privacy and discovery risk. Decide a period and expire.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
InsufficientS3BucketPolicyException on create-trail | Bucket policy does not allow CloudTrail or has the wrong path | Compare with the policy above; use the organisation ID path |
InsufficientEncryptionPolicyException | The KMS key policy lacks the CloudTrail statement or the trail ARN condition does not match | Fix the key policy; confirm the trail name and Region in the ARN |
get-trail-status shows LatestDeliveryError | Bucket policy, key policy or Object Lock default retention issue | Read the error; fix; wait for the next delivery |
| Member account cannot see events of other accounts | Event history is per account | Query the S3 logs with Athena from the security tooling account |
validate-logs says digest files missing | Log file validation was off at the time, or the time window is before the trail existed | Check LogFileValidationEnabled and the window |
| Cannot delete the bucket | Object Lock retention still active on versions | Wait, or (governance mode) delete versions with --bypass-governance-retention and the right permission |
AccessDenied for the auditor reading logs | KMS decrypt missing on the key policy | Add the auditor role to the key policy |
| Unexpected Cost Explorer line for CloudTrail | Extra trails, or data events | List trails and event selectors; remove duplicates |
Cost and cleanup
Cost label: credits. With one organisation trail the first copy of management events is free ⚠ VERIFY. Expected lab cost is cents: a little S3 storage, KMS requests (reduced by Bucket Keys) and one KMS key (about USD 1 per month while it exists ⚠ VERIFY). Data events are the trap: charged per event ⚠ VERIFY the current rate. Do not enable them in the lab.
Teardown, in this order:
# 1. Stop new logs, then remove the trail (management account)
aws cloudtrail stop-logging --name hl-org-trail --profile learn-management
aws cloudtrail delete-trail --name hl-org-trail --profile learn-management
# 2. Empty the bucket including versions. Governance-mode locks need the bypass flag AND the s3:BypassGovernanceRetention permission.
# (If the lock is still active and you lack the permission, wait one day.) Repeat until empty.
aws s3api list-object-versions --bucket hl-org-logs-<log-archive-account-id>-us-east-1 --profile learn-log-archive \
--query "{Objects: Versions[].{Key:Key,VersionId:VersionId}}" --output json > versions.json
aws s3api delete-objects --bucket hl-org-logs-<log-archive-account-id>-us-east-1 --delete file://versions.json \
--bypass-governance-retention --profile learn-log-archive
# 3. Destroy the rest, or delete the bucket by hand
terraform destroy
# 4. Schedule the KMS key for deletion (7 is the minimum waiting period)
aws kms schedule-key-deletion --key-id <key-id> --pending-window-in-days 7 --profile learn-log-archive
# 5. Check that nothing tagged for the lab remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab --region us-east-1 --profile learn-log-archiveAlso delete delete markers (list-object-versions returns DeleteMarkers separately) if the bucket will not delete. Disable trusted access for CloudTrail only if nothing else depends on it.
Exercises
- 🟢 Guided: Create the bucket and key with Terraform in your log archive (or sandbox) account, set governance mode for one day, and run
get-object-lock-configurationandget-bucket-encryption. Run the health check against a normal trail first to see the "not an organisation trail" problem. - 🟡 Challenge: Write an EventBridge rule (as Terraform or CDK) that matches
StopLogging,DeleteTrail,UpdateTrailandPutEventSelectorsevents and sends them to an SNS topic. State where it must be deployed (every account? one?) and why. - 🔴 Live problem: An auditor asks you to prove that no events from
payments-prodbetween 03:00 and 05:00 last Tuesday were altered, and to list everyAssumeRoleinto the break-glass role in that window. Write the steps, the commands and the evidence you would hand over.
Assignment
Add docs/logging-design.md to your portfolio: the source table, the retention table with reasons and owners, the bucket and key policy design, the SCP protection and alerting plan, and the output of the health check and validate-logs (account IDs redacted).
Solutions
get-object-lock-configuration should return ObjectLockEnabled: Enabled with a default retention of GOVERNANCE and 1 day. get-bucket-encryption shows aws:kms with your key ARN and BucketKeyEnabled: true. The health check against a normal trail prints "No organisation trail exists." because IsOrganizationTrail is false: it is a useful negative test. After you create the organisation trail the check should print OK.
CloudTrail management events are delivered to EventBridge in each Region and account where they occur (the default event bus). Option 1: deploy the rule in every account and Region through a StackSet, forwarding to a central bus in the security tooling account. Option 2: with an organisation trail, query the S3 logs with a scheduled Athena query, which is cheaper but slower (minutes rather than seconds). Banks usually choose option 1 for the four calls above because speed matters, and option 2 for broad hunting. Match source = aws.cloudtrail with detail-type = AWS API Call via CloudTrail and detail.eventName in the four names ⚠ VERIFY event shape against your Region.
Steps: (1) run validate-logs for the trail ARN with --start-time and --end-time covering that window (and the hour either side), with --account-id if supported ⚠ VERIFY the flag, and save the output; (2) query with Athena over the CloudTrail table for eventName = 'AssumeRole' and requestParameters containing the break-glass role ARN, restricted by eventTime and recipientAccountId = <payments-prod>; (3) show the Object Lock configuration and the bucket policy that deny deletes; (4) show that the SCP and alert for StopLogging exist and the health check history shows IsLogging true throughout. Hand over the query text, results, the validation output and screenshots or CLI output of the controls, with a note on the limits (validation proves integrity of delivered files, not that every event was delivered).
Knowledge check
Answer: So that principals who administer the workload accounts cannot delete or alter the evidence, supporting segregation of duties and tamper resistance.
Answer: No. Member accounts can see it but cannot delete it, turn logging off or change its settings. Only the management account (or a delegated administrator) can.
Answer: That delivered log files have not been modified or deleted since delivery (using signed digest files). It does not prove that every event was delivered.
Answer: In governance mode, principals with a special permission can bypass or shorten retention. In compliance mode nobody, including the root user, can delete the locked object version or shorten retention until it expires.
Answer: The bucket policy does not allow CloudTrail on the organisation-ID path; the KMS key policy does not allow kms:GenerateDataKey* for the trail; logging was never started; the trail's Region is an opt-in Region and members have not enabled it.
Interview questions
- How do you make audit logs tamper-resistant? A separate log archive account, an organisation trail (so member accounts cannot disable it), S3 Object Lock, SSE-KMS with a key the workloads cannot touch, log file validation, SCPs on the trail and bucket actions, and alerts on changes.
- How long should we keep logs? Ask records management, legal and privacy; set per log type with a stated reason; expire at the end. Do not keep forever.
- What are the cost traps in centralised logging? Data events, extra trail copies, KMS requests without Bucket Keys, flow log volume, Athena full scans.
- How do you know the logging pipeline is working? A scheduled health check (trail logging, no delivery errors), regular
validate-logs, coverage checks that each account and Region appears in the bucket, and a periodic auditor read test.
Certification practice
A security team needs CloudTrail logs from all current and future accounts, delivered to a central bucket that account administrators cannot modify. Which solution meets this with the least operational overhead?
A) Create a trail in each account with a CloudFormation StackSet. B) Create an organisation trail from the management account (or a delegated administrator) delivering to a bucket in a dedicated log archive account. C) Use CloudWatch Logs subscription filters in every account. D) Enable S3 replication from each account's bucket.
Answer: B. An organisation trail automatically covers new accounts and cannot be modified by member accounts. A needs per-account trails that account administrators can alter. C and D are not the primary audit-trail mechanism and add operational work.
A bank must keep CloudTrail logs unmodifiable for seven years and show regulators that no one, including administrators, could delete them. What should the solution include?
A) S3 versioning only.
B) S3 Object Lock in compliance mode with a seven-year default retention, in a dedicated account.
C) An S3 lifecycle rule that transitions to Glacier.
D) An IAM policy denying s3:DeleteObject to administrators.
Answer: B. Compliance mode prevents deletion or retention changes by any principal until retention expires. A allows deletion of versions. C is about cost, not immutability. D can be edited by whoever administers IAM.
An organisation trail shows LatestDeliveryError: InsufficientS3BucketPolicyException. What should the operator check first?
A) The bucket policy allows CloudTrail to write under the organisation ID path with the correct aws:SourceArn.
B) The IAM Identity Center permission sets.
C) The Cost Explorer filters.
D) The VPC route tables.
Answer: A. Delivery errors of this kind mean the bucket policy does not allow the trail to write. The other options are unrelated to CloudTrail delivery.
Completion checklist
- I can list at least six log sources and where each should land
- I created the bucket and key, and verified Object Lock and encryption
- I ran the health check and understand each problem it reports
- I can explain governance versus compliance mode and why the lab uses governance
- I wrote the retention table with owners and reasons
- I ran the teardown and confirmed no billing resources remain
Key takeaways
- Logs must leave the account, land where workloads cannot reach, and resist deletion.
- Organisation trails cover every account and cannot be stopped by members.
- Retention is a business, legal and privacy decision, with a reason for every period.
- Data events and KMS requests are the cost traps; scope them.
- Test reads and run validation on a schedule; an unread archive is an assumption.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Lab 15.1: design, test and attach organisation guardrails
Describe Harbourline's organisation as a file, prove with 50 tests that its guardrails block what they should and allow what they must, then attach them to your sandbox OU with Terraform and watch a real explicit deny.
Centralised security tooling: delegated administrators, auto-enable and a Config aggregator
Run GuardDuty, Security Hub, AWS Config and IAM Access Analyzer for the whole organisation from a dedicated security tooling account, keep the management account out of daily security work, and prove coverage with tested checks.