Skip to Content
Cloud Engineering Academy
5. Security for Banking & Finance

CloudTrail for banks: organisation trail, integrity validation and an immutable log archive

Build the audit trail a bank can defend in front of an auditor: an organisation-wide CloudTrail trail delivered to a locked-down log-archive account, KMS-encrypted, protected with S3 Object Lock, and provably untampered thanks to log file integrity validation.

intermediate⏱ 2 h 🟢 FreeSCS-C03SAA-C03DVA-C03CLF-C02

Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain what CloudTrail records — management events, data events, network activity events and Insights events — and which of them cost money.
  • Design an organisation trail that delivers every account's events to a separate log-archive account, encrypted with a KMS customer managed key and protected by S3 Object Lock.
  • Turn on log file integrity validation, explain how hourly digest files chain together, and validate logs with aws cloudtrail validate-logs.
  • Write a .NET 10 tool that verifies log-file hashes and the digest chain offline, the way an auditor or forensic investigator would.
  • Query CloudTrail logs with Amazon Athena, and explain why CloudTrail Lake is not used for new designs on this platform.

Prerequisites

Why this matters in banking

A Harbourline engineer's access key leaks on a public code-sharing site on a Friday evening. On Monday the security team must answer the regulator's and the board's questions: What did the attacker do? Which customer data did they touch? Can you prove your logs are complete and unaltered?

Without a well-designed trail, the honest answers are bad:

  • "We only had the 90-day Event history in the console, in one Region, and the attacker used eu-west-2."
  • "The trail's bucket was in the same account the attacker compromised, and they called StopLogging and deleted objects."
  • "We logged S3 management events but not data events, so we can't say which customer files were read."
  • "We can't prove nobody edited the logs."

Auditors (internal audit, external auditors, PCI QSAs) ask for audit trails of administrative access that are complete, protected from modification, retained for a defined period and reviewed. A bank's records-management policy usually sets retention in years. CloudTrail plus a log-archive account, Object Lock and integrity validation is the standard AWS answer to all four. As always, whether it meets a specific obligation is a decision for the bank's compliance and legal teams.

The idea in plain language

Almost everything you do in AWS — in the console, the CLI, an SDK or CDK — is an API call. AWS CloudTrail records those API calls as events: who called (userIdentity), what (eventName), on which resource, when, from where (sourceIPAddress), and whether it succeeded (errorCode).

  • Event history is on by default: 90 days of management events per Region, viewable and searchable in the console, free. It cannot be extended or protected.
  • A trail is a configuration that continuously delivers events as compressed JSON files to an S3 bucket (and optionally CloudWatch Logs). Trails are what you keep for years.
  • An organisation trail, created in the management account (or a delegated administrator account), automatically covers every account in AWS Organizations, including new ones, and member accounts can't change or delete it.
  • Log file integrity validation makes CloudTrail write a digest file every hour listing the SHA-256 hash of each log file it delivered, signed with a CloudTrail private key. Each digest also contains the signature of the previous one, forming a chain. If anyone edits, deletes or inserts a log file — or deletes a digest — validation detects it.

Analogy: CloudTrail is the bank's CCTV system. Event history is the monitor in the guard's room that only shows the last 90 days. A trail is the recorder that copies footage to tapes. The log-archive account is an off-site vault the branch staff can't open, Object Lock is the time lock on that vault, and integrity validation is a tamper-evident seal on every tape, with each seal's number written on the next one.

What problem the AWS service solves

Event categoryWhat it recordsDefaultPrice (checked 2026-11-18)
Management events (control plane)Creating, changing and deleting resources: CreateUser, PutBucketPolicy, AuthorizeSecurityGroupIngress, console sign-inRecorded in event history; included in a trail by defaultFirst copy delivered to S3 is free; additional copies USD 2.00 per 100,000 events
Data events (data plane)Operations on data: S3 GetObject/PutObject, DynamoDB item reads/writes, Lambda Invoke, and many moreOff — you must select themUSD 0.10 per 100,000 events
Network activity eventsAPI calls made through VPC endpoints, including calls denied by endpoint policiesOffUSD 0.10 per 100,000 events
Insights eventsUnusual API call volume or error rates compared with a baselineOffUSD 0.35 per 100,000 management events analysed per insight type
CloudWatch Logs deliveryCopy of events to a log group for metric filters and alarmsOffUSD 0.25 per GB delivered, plus CloudWatch Logs storage

Read vs write: management and data events can be filtered to read-only, write-only or both. Advanced event selectors let you filter precisely, for example "S3 data events only for buckets whose ARN starts with arn:aws:s3:::hl-prod-customer-docs".

Limitations to be honest about:

  • Delivery is not instant. CloudTrail typically delivers log files within about 5 minutes of an API call, but there is no hard guarantee ⚠ VERIFY current wording. For real-time detection use EventBridge rules or GuardDuty (see Security Hub, GuardDuty, Inspector and Detective).
  • Not everything is a CloudTrail event. Operating-system logins, SQL queries inside RDS, application-level actions (a customer viewing a statement) and network packets are not. You need OS, database, application and VPC Flow Logs as well.
  • Data events can be very expensive at banking volume. A busy S3 bucket can generate billions of GetObject events. Select them deliberately.
  • CloudTrail Lake is closed to new customers (since 2026-05-31). It remains available to existing customers, but new designs on this platform use trails into S3, queried with Athena, plus CloudWatch for alerting.
  • Integrity validation detects tampering; it doesn't prevent it. Prevention comes from the separate account, bucket policy, SCPs and Object Lock.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Activity Log (subscription-level control plane)CloudTrail management events / event historyEvent history keeps 90 days per Region; trails keep data for as long as you store it in S3
Diagnostic settings exporting Activity Log to a storage accountTrail delivering to S3One organisation trail covers all accounts and Regions
Resource logs (e.g. Storage read/write logs)CloudTrail data eventsOpt-in per resource type, priced per event
Entra ID sign-in and audit logsCloudTrail ConsoleLogin events and IAM Identity Center eventsIdentity Center sign-ins appear as CloudTrail events from signin.amazonaws.com / sso.amazonaws.com ⚠ VERIFY event sources in your Region
Immutable blob storage (time-based retention, legal hold)S3 Object Lock (governance or compliance mode, legal hold)Object Lock must be enabled on the bucket; compliance mode can't be shortened even by root
Log Analytics KQL queriesAthena SQL over S3, or CloudWatch Logs InsightsAthena charges per TB scanned; partition projection keeps scans small
Microsoft Sentinel ingesting Activity LogSIEM ingesting from S3 (SQS notifications) or CloudWatch LogsAWS doesn't hash-chain Azure logs for you; CloudTrail's digest chain has no direct Azure equivalent

Architecture

Security boundaries:

  1. The trail is defined in the management account (or a delegated administrator), so workload admins can't modify it.
  2. Logs are stored in the log-archive account. Nobody in a workload account has write or delete permission there; the bucket policy only lets the CloudTrail service write.
  3. Object Lock stops deletion and overwriting for the retention period, even by administrators of the log-archive account (compliance mode) or by all but specially permitted users (governance mode).
  4. SSE-KMS with a customer managed key means reading logs needs both S3 and KMS permissions — and every decrypt is itself logged.
  5. Digest files prove integrity to a third party.

How the digest chain works

Each digest is signed (SHA-256 with RSA, using a per-Region CloudTrail key). The string that is signed includes the digest's end time, its S3 path, the SHA-256 of its own contents and the previous digest's signature. Changing any log file breaks its hash; deleting a digest breaks the chain; and only CloudTrail holds the private key, so nobody can forge a replacement digest. Empty digests are still delivered each hour, proving that no logs existed for that hour.

Step-by-step implementation

Create (or designate) the log-archive account

In AWS Organizations, keep a dedicated Log Archive account in a Security OU (Control Tower's landing zone creates one for you). Access is limited to a small group of security engineers and auditors, through IAM Identity Center permission sets that are read-only by default.

Create the KMS key in the log-archive account

A symmetric customer managed key with automatic rotation on. The key policy allows cloudtrail.amazonaws.com to kms:GenerateDataKey* with a condition on the encryption context (kms:EncryptionContext:aws:cloudtrail:arn) matching your organisation's trails, and allows a named auditor role to kms:Decrypt. No workload account principal gets access.

Create the bucket with Object Lock

Enable Object Lock at creation (it also enables versioning), Block Public Access, SSE-KMS with the key above and S3 Bucket Keys (fewer KMS calls, lower cost). Set a default retention: in production, compliance mode with your records-management period (for example 7 years); in the lab, governance mode for 1 day. Add lifecycle rules to move older logs to cheaper storage classes.

Write the bucket policy

Allow CloudTrail s3:GetBucketAcl on the bucket and s3:PutObject on AWSLogs/<org-id>/* (organisation trails use the org ID in the path), with conditions on aws:SourceArn so only your trail can write. Deny any request without TLS (aws:SecureTransport = false). Deny s3:DeleteObject* and s3:PutBucketObjectLockConfiguration to everyone except a break-glass role.

Create the organisation trail

In the management account (or delegated admin): multi-Region, organisation trail, management events read and write, log file validation on, KMS key from the log-archive account, delivery to the log-archive bucket. Add data events only where they matter (see the Banking use case).

Protect the trail with an SCP

Attach an SCP to all OUs that denies cloudtrail:StopLogging, cloudtrail:DeleteTrail, cloudtrail:UpdateTrail and cloudtrail:PutEventSelectors except for a named security automation role. Organisation trails are already protected from member accounts, but the SCP also stops anyone in a member account from creating confusion with their own trails.

Alert on attempts to blind you

EventBridge rules (or CloudWatch metric filters) for StopLogging, DeleteTrail, UpdateTrail, PutBucketPolicy on the log bucket, ScheduleKeyDeletion/DisableKey on the trail key, and root-user activity. Route to the SOC.

Validate regularly and prove it

Run aws cloudtrail validate-logs on a schedule (for example daily, from a security automation account), store the output as evidence, and alert on any invalid or missing file. Auditors will ask for this evidence — see AWS Config for related posture evidence.

AWS Console instructions

Single-account lab trail (free):

  1. Console → CloudTrail → Trails → Create trail. Name hl-m05-audit-trail.
  2. Storage location: create a new S3 bucket hl-m05-cloudtrail-<account-id>. For the lab, keep SSE-KMS off (SSE-S3 is free) or choose a new KMS key if you have credits.
  3. Log file validation: Enabled (it's on by default in the console — leave it).
  4. CloudWatch Logs: optional; skip for the free version.
  5. Tags: Project=cea-lab, Lab=m05-l17, Environment=learn, Owner=<you>.
  6. Choose log events: Management events — Read and Write. Leave data, Insights and network activity events off for the lab.
  7. Create trail. The trail is multi-Region by default in the console.

Look at events: CloudTrail → Event history, filter by Event name = CreateTrail. Open the event and read userIdentity — that's you, via your SSO role session.

Object Lock (lab): S3 → bucket → Properties → Object Lock → Edit — enabling Object Lock on an existing bucket is supported, but test it on a new bucket first. Choose Governance, 1 day. Never choose Compliance in a lab.

Organisation trail (design only for most learners): in the management account, CloudTrail → Trails → Create trail → Enable for all accounts in my organization.

AWS CLI

PROFILE=learn-sandbox
ACCOUNT=$(aws sts get-caller-identity --query Account --output text --profile $PROFILE)
BUCKET=hl-m05-cloudtrail-$ACCOUNT

# 1. Bucket with Object Lock (enables versioning automatically) and Block Public Access
aws s3api create-bucket --bucket $BUCKET --object-lock-enabled-for-bucket --profile $PROFILE
aws s3api put-public-access-block --bucket $BUCKET --profile $PROFILE \
  --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

# 2. Default retention: GOVERNANCE for 1 day (lab only!)
aws s3api put-object-lock-configuration --bucket $BUCKET --profile $PROFILE \
  --object-lock-configuration '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":1}}}'

# 3. Bucket policy that lets only this trail write (see bucket-policy.json below)
aws s3api put-bucket-policy --bucket $BUCKET --policy file://bucket-policy.json --profile $PROFILE

# 4. Multi-Region trail with integrity validation
aws cloudtrail create-trail --name hl-m05-audit-trail --s3-bucket-name $BUCKET \
  --is-multi-region-trail --enable-log-file-validation \
  --tags-list Key=Project,Value=cea-lab Key=Lab,Value=m05-l17 Key=Environment,Value=learn Key=Owner,Value=you \
  --profile $PROFILE
aws cloudtrail start-logging --name hl-m05-audit-trail --profile $PROFILE

# 5. Check health: IsLogging, latest delivery and digest delivery errors
aws cloudtrail get-trail-status --name hl-m05-audit-trail --profile $PROFILE \
  --query "{Logging:IsLogging,LastDelivery:LatestDeliveryTime,LastDigest:LatestDigestDeliveryTime,DigestError:LatestDigestDeliveryError}"

# 6. Add S3 data events for ONE sensitive bucket only (advanced event selectors)
aws cloudtrail put-event-selectors --trail-name hl-m05-audit-trail --profile $PROFILE \
  --advanced-event-selectors '[
    {"Name":"Management events","FieldSelectors":[{"Field":"eventCategory","Equals":["Management"]}]},
    {"Name":"Customer docs reads and writes","FieldSelectors":[
      {"Field":"eventCategory","Equals":["Data"]},
      {"Field":"resources.type","Equals":["AWS::S3::Object"]},
      {"Field":"resources.ARN","StartsWith":["arn:aws:s3:::hl-m05-customer-docs-'"$ACCOUNT"'/"]}]}
  ]'

# 7. Validate integrity (wait at least an hour after starting so a digest exists)
aws cloudtrail validate-logs \
  --trail-arn arn:aws:cloudtrail:us-east-1:$ACCOUNT:trail/hl-m05-audit-trail \
  --start-time 2026-11-18T00:00:00Z --verbose --profile $PROFILE

# 8. Quick look-ups in the 90-day event history (no trail needed)
aws cloudtrail lookup-events --max-results 5 --profile $PROFILE \
  --lookup-attributes AttributeKey=EventName,AttributeValue=ConsoleLogin
bucket-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AWSCloudTrailAclCheck",
      "Effect": "Allow",
      "Principal": { "Service": "cloudtrail.amazonaws.com" },
      "Action": "s3:GetBucketAcl",
      "Resource": "arn:aws:s3:::hl-m05-cloudtrail-111122223333",
      "Condition": { "StringEquals": { "aws:SourceArn": "arn:aws:cloudtrail:us-east-1:111122223333:trail/hl-m05-audit-trail" } }
    },
    {
      "Sid": "AWSCloudTrailWrite",
      "Effect": "Allow",
      "Principal": { "Service": "cloudtrail.amazonaws.com" },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::hl-m05-cloudtrail-111122223333/AWSLogs/111122223333/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control",
          "aws:SourceArn": "arn:aws:cloudtrail:us-east-1:111122223333:trail/hl-m05-audit-trail"
        }
      }
    },
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": ["arn:aws:s3:::hl-m05-cloudtrail-111122223333", "arn:aws:s3:::hl-m05-cloudtrail-111122223333/*"],
      "Condition": { "Bool": { "aws:SecureTransport": "false" } }
    }
  ]
}

For an organisation trail, the write resource becomes arn:aws:s3:::<bucket>/AWSLogs/<o-org-id>/* (plus the management account's own AWSLogs/<management-account-id>/* path), and aws:SourceArn names the trail in the management account.

Expected validate-logs output (abridged):

Validating log files for trail arn:aws:cloudtrail:us-east-1:111122223333:trail/hl-m05-audit-trail between 2026-11-18T00:00:00Z and 2026-11-18T15:10:02Z

Digest file    s3://hl-m05-cloudtrail-111122223333/AWSLogs/111122223333/CloudTrail-Digest/us-east-1/2026/11/18/...T140102Z.json.gz    valid
Log file       s3://hl-m05-cloudtrail-111122223333/AWSLogs/111122223333/CloudTrail/us-east-1/2026/11/18/..._20261118T1355Z_abc.json.gz    valid

Results requested for 2026-11-18T00:00:00Z to 2026-11-18T15:10:02Z
Results found for 2026-11-18T13:01:02Z to 2026-11-18T15:01:02Z:

3/3 digest files valid
12/12 log files valid

If a log file was modified you see INVALID: hash value doesn't match; if one was deleted, INVALID: not found.

Querying with Athena

-- One-off: table over the trail with partition projection (no crawlers, no MSCK REPAIR)
CREATE EXTERNAL TABLE hl_cloudtrail (
  eventversion STRING,
  useridentity STRUCT<type:STRING, principalid:STRING, arn:STRING, accountid:STRING,
                      invokedby:STRING, accesskeyid:STRING, username:STRING,
                      sessioncontext:STRUCT<attributes:STRUCT<mfaauthenticated:STRING, creationdate:STRING>,
                      sessionissuer:STRUCT<type:STRING, principalid:STRING, arn:STRING, accountid:STRING, username:STRING>>>,
  eventtime STRING,
  eventsource STRING,
  eventname STRING,
  awsregion STRING,
  sourceipaddress STRING,
  useragent STRING,
  errorcode STRING,
  errormessage STRING,
  requestparameters STRING,
  responseelements STRING,
  requestid STRING,
  eventid STRING,
  readonly STRING,
  eventtype STRING,
  recipientaccountid STRING
)
PARTITIONED BY (region STRING, day STRING)
ROW FORMAT SERDE 'com.amazon.emr.hive.serde.CloudTrailSerde'
STORED AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION 's3://hl-m05-cloudtrail-111122223333/AWSLogs/111122223333/CloudTrail/'
TBLPROPERTIES (
  'projection.enabled'='true',
  'projection.region.type'='enum',
  'projection.region.values'='us-east-1,eu-west-2',
  'projection.day.type'='date',
  'projection.day.range'='2026/01/01,NOW',
  'projection.day.format'='yyyy/MM/dd',
  'projection.day.interval'='1',
  'projection.day.interval.unit'='DAYS',
  'storage.location.template'='s3://hl-m05-cloudtrail-111122223333/AWSLogs/111122223333/CloudTrail/${region}/${day}'
);

-- Everything one access key did in the last 7 days (incident question #1)
SELECT eventtime, eventsource, eventname, awsregion, sourceipaddress, errorcode
FROM hl_cloudtrail
WHERE day >= date_format(current_date - interval '7' day, '%Y/%m/%d')
  AND useridentity.accesskeyid = 'AKIAIOSFODNN7EXAMPLE'
ORDER BY eventtime;

-- Who changed security groups or bucket policies yesterday? (change-review evidence)
SELECT eventtime, useridentity.arn, eventname, requestparameters
FROM hl_cloudtrail
WHERE day = date_format(current_date - interval '1' day, '%Y/%m/%d')
  AND eventname IN ('AuthorizeSecurityGroupIngress','PutBucketPolicy','DeleteBucketPolicy','PutBucketPublicAccessBlock');

Always filter on the day partition: Athena charges per TB scanned (USD 5 per TB in us-east-1 ⚠ VERIFY), and a bank's multi-year trail is large.

Infrastructure as Code

The stacks below build the single-account lab version: bucket with Object Lock (governance, 1 day), bucket policy, and a multi-Region trail with validation. The organisation-trail differences are noted in comments.

AuditTrailStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.CloudTrail;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.S3;
using Constructs;

namespace Harbourline.Security.Infra;

public sealed class AuditTrailStack : Stack
{
    public AuditTrailStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        Tags.SetTag("Project", "cea-lab");
        Tags.SetTag("Lab", "m05-l17");
        Tags.SetTag("Environment", "learn");

        const string trailName = "hl-m05-audit-trail";
        var trailArn = $"arn:aws:cloudtrail:{Region}:{Account}:trail/{trailName}";

        var bucket = new Bucket(this, "TrailBucket", new BucketProps
        {
            BucketName = $"hl-m05-cloudtrail-{Account}",
            BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
            Encryption = BucketEncryption.S3_MANAGED,    // production: KMS key in the log-archive account
            EnforceSSL = true,
            Versioned = true,
            ObjectLockEnabled = true,
            // LEARNING SHORTCUT: governance mode, 1 day. Production: compliance mode, records-management period.
            ObjectLockDefaultRetention = ObjectLockRetention.Governance(Duration.Days(1)),
            LifecycleRules =
            [
                new LifecycleRule { Transitions = [new Transition { StorageClass = StorageClass.GLACIER_INSTANT_RETRIEVAL, TransitionAfter = Duration.Days(90) }] },
            ],
            RemovalPolicy = RemovalPolicy.RETAIN, // locked objects can't be force-deleted anyway
        });

        bucket.AddToResourcePolicy(new PolicyStatement(new PolicyStatementProps
        {
            Sid = "AWSCloudTrailAclCheck",
            Principals = [new ServicePrincipal("cloudtrail.amazonaws.com")],
            Actions = ["s3:GetBucketAcl"],
            Resources = [bucket.BucketArn],
            Conditions = new Dictionary<string, object> { ["StringEquals"] = new Dictionary<string, string> { ["aws:SourceArn"] = trailArn } },
        }));
        bucket.AddToResourcePolicy(new PolicyStatement(new PolicyStatementProps
        {
            Sid = "AWSCloudTrailWrite",
            Principals = [new ServicePrincipal("cloudtrail.amazonaws.com")],
            Actions = ["s3:PutObject"],
            // Organisation trail: $"{bucket.BucketArn}/AWSLogs/{orgId}/*"
            Resources = [$"{bucket.BucketArn}/AWSLogs/{Account}/*"],
            Conditions = new Dictionary<string, object>
            {
                ["StringEquals"] = new Dictionary<string, string>
                {
                    ["s3:x-amz-acl"] = "bucket-owner-full-control",
                    ["aws:SourceArn"] = trailArn,
                },
            },
        }));

        var trail = new CfnTrail(this, "Trail", new CfnTrailProps
        {
            TrailName = trailName,
            S3BucketName = bucket.BucketName,
            IsLogging = true,
            IsMultiRegionTrail = true,
            IncludeGlobalServiceEvents = true,
            EnableLogFileValidation = true,
            // IsOrganizationTrail = true,   // only in the management or delegated admin account
            // KmsKeyId = "arn:aws:kms:us-east-1:<log-archive>:key/<id>",
            EventSelectors = new[]
            {
                new CfnTrail.EventSelectorProperty { IncludeManagementEvents = true, ReadWriteType = "All" },
            },
        });
        trail.Node.AddDependency(bucket.Policy!);

        _ = new CfnOutput(this, "TrailArn", new CfnOutputProps { Value = trail.AttrArn });
    }
}

The organisation-wide SCP that protects trails:

scp-protect-cloudtrail.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ProtectCloudTrail",
      "Effect": "Deny",
      "Action": [
        "cloudtrail:StopLogging",
        "cloudtrail:DeleteTrail",
        "cloudtrail:UpdateTrail",
        "cloudtrail:PutEventSelectors"
      ],
      "Resource": "*",
      "Condition": {
        "ArnNotLike": { "aws:PrincipalArn": "arn:aws:iam::*:role/hl-security-automation" }
      }
    }
  ]
}

Complete code

aws cloudtrail validate-logs needs network access to your bucket. During a forensic investigation, or for an auditor who receives a copy of the logs, you want to validate offline. This .NET 10 console app takes a folder produced by aws s3 sync s3://<bucket> ./evidence and:

  1. reads every digest file;
  2. checks that each log file listed in a digest exists locally and that the SHA-256 of its uncompressed content matches hashValue;
  3. checks the chain: each digest's previousDigestHashValue must equal the SHA-256 of the uncompressed previous digest file;
  4. optionally verifies each digest's RSA signature, if you saved the CloudTrail public keys (aws cloudtrail list-public-keys) and the current digest signatures beforehand.

It needs no AWS SDK, so it runs on an air-gapped forensic workstation.

TrailIntegrity/TrailIntegrity.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <RootNamespace>Harbourline.Security.TrailIntegrity</RootNamespace>
  </PropertyGroup>
</Project>
TrailIntegrity/DigestModels.cs
using System.Text.Json.Serialization;

namespace Harbourline.Security.TrailIntegrity;

// Fields from the CloudTrail digest file format that validation needs.
public sealed record DigestFile(
    [property: JsonPropertyName("digestStartTime")] string DigestStartTime,
    [property: JsonPropertyName("digestEndTime")] string DigestEndTime,
    [property: JsonPropertyName("digestS3Bucket")] string DigestS3Bucket,
    [property: JsonPropertyName("digestS3Object")] string DigestS3Object,
    [property: JsonPropertyName("digestPublicKeyFingerprint")] string DigestPublicKeyFingerprint,
    [property: JsonPropertyName("previousDigestS3Object")] string? PreviousDigestS3Object,
    [property: JsonPropertyName("previousDigestHashValue")] string? PreviousDigestHashValue,
    [property: JsonPropertyName("previousDigestSignature")] string? PreviousDigestSignature,
    [property: JsonPropertyName("logFiles")] List<DigestLogFile> LogFiles);

public sealed record DigestLogFile(
    [property: JsonPropertyName("s3Object")] string S3Object,
    [property: JsonPropertyName("hashValue")] string HashValue,
    [property: JsonPropertyName("hashAlgorithm")] string HashAlgorithm);

public enum Severity { Ok, Warning, Invalid }

public sealed record Finding(Severity Severity, string Subject, string Message);
TrailIntegrity/TrailValidator.cs
using System.IO.Compression;
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;

namespace Harbourline.Security.TrailIntegrity;

public sealed class TrailValidator(string evidenceRoot)
{
    // Optional: fingerprint -> PKCS#1 DER public key bytes, saved from `aws cloudtrail list-public-keys`.
    public Dictionary<string, byte[]> PublicKeys { get; } = new(StringComparer.OrdinalIgnoreCase);

    // Optional: digest S3 key -> hex signature of the NEWEST digest (from its x-amz-meta-signature metadata).
    public Dictionary<string, string> KnownSignatures { get; } = new();

    public IReadOnlyList<Finding> Validate()
    {
        var findings = new List<Finding>();
        var digests = Directory.EnumerateFiles(evidenceRoot, "*CloudTrail-Digest*.json.gz", SearchOption.AllDirectories)
            .Select(path => (Path: path, Raw: ReadGzip(path)))
            .Select(d => (d.Path, d.Raw, Digest: JsonSerializer.Deserialize<DigestFile>(d.Raw)!))
            .OrderBy(d => d.Digest.DigestEndTime, StringComparer.Ordinal)
            .ToList();

        if (digests.Count == 0)
        {
            findings.Add(new(Severity.Invalid, evidenceRoot, "No digest files found"));
            return findings;
        }

        var byKey = digests.ToDictionary(d => d.Digest.DigestS3Object, StringComparer.Ordinal);

        foreach (var (path, raw, digest) in digests)
        {
            // 1. The digest must sit at the path it claims (detects files moved or renamed).
            if (!ToLocal(digest.DigestS3Object).Equals(Path.GetFullPath(path), StringComparison.OrdinalIgnoreCase))
                findings.Add(new(Severity.Invalid, path, "Digest is not at the location recorded inside it"));

            // 2. Every log file it lists must exist and match its hash.
            foreach (var log in digest.LogFiles)
                findings.Add(CheckLogFile(log));

            // 3. Chain: the previous digest must exist and hash to the recorded value.
            if (digest.PreviousDigestS3Object is null)
            {
                findings.Add(new(Severity.Warning, digest.DigestS3Object, "Starting digest (validation was enabled or restarted here)"));
            }
            else if (!byKey.TryGetValue(digest.PreviousDigestS3Object, out var prev))
            {
                var outsideRange = string.CompareOrdinal(digest.DigestEndTime, digests[0].Digest.DigestEndTime) == 0;
                findings.Add(new(outsideRange ? Severity.Warning : Severity.Invalid, digest.PreviousDigestS3Object,
                    outsideRange ? "Previous digest is outside the evidence set" : "Previous digest is MISSING — chain broken"));
            }
            else if (!Hex(SHA256.HashData(prev.Raw)).Equals(digest.PreviousDigestHashValue, StringComparison.OrdinalIgnoreCase))
            {
                findings.Add(new(Severity.Invalid, digest.PreviousDigestS3Object, "Previous digest content does not match the hash recorded in the next digest"));
            }

            // 4. Signature, when we have both the key and the signature.
            var signature = KnownSignatures.GetValueOrDefault(digest.DigestS3Object)
                ?? digests.FirstOrDefault(n => n.Digest.PreviousDigestS3Object == digest.DigestS3Object).Digest?.PreviousDigestSignature;
            if (signature is not null && PublicKeys.TryGetValue(digest.DigestPublicKeyFingerprint, out var der))
            {
                var ok = VerifySignature(digest, raw, signature, digest.PreviousDigestSignature, der);
                findings.Add(new(ok ? Severity.Ok : Severity.Invalid, digest.DigestS3Object, ok ? "Signature valid" : "Signature INVALID"));
            }
        }

        return findings;
    }

    private Finding CheckLogFile(DigestLogFile log)
    {
        var local = ToLocal(log.S3Object);
        if (!File.Exists(local)) return new(Severity.Invalid, log.S3Object, "Log file listed in digest is MISSING");
        if (log.HashAlgorithm != "SHA-256") return new(Severity.Warning, log.S3Object, $"Unexpected algorithm {log.HashAlgorithm}");

        var actual = Hex(SHA256.HashData(ReadGzip(local)));
        return actual.Equals(log.HashValue, StringComparison.OrdinalIgnoreCase)
            ? new(Severity.Ok, log.S3Object, "Hash matches")
            : new(Severity.Invalid, log.S3Object, $"Hash MISMATCH (expected {log.HashValue[..12]}…, got {actual[..12]}…)");
    }

    // Data-signing string: end time \n bucket/key \n hex(sha256(digest)) \n previous signature (hex)
    public static bool VerifySignature(DigestFile d, byte[] rawDigest, string signatureHex, string? previousSignatureHex, byte[] pkcs1Der)
    {
        var toSign = $"{d.DigestEndTime}\n{d.DigestS3Bucket}/{d.DigestS3Object}\n{Hex(SHA256.HashData(rawDigest))}\n{previousSignatureHex ?? "null"}";
        using var rsa = RSA.Create();
        rsa.ImportRSAPublicKey(pkcs1Der, out _);
        return rsa.VerifyData(Encoding.UTF8.GetBytes(toSign), Convert.FromHexString(signatureHex),
            HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
    }

    private string ToLocal(string s3Key)
    {
        // `aws s3 sync s3://bucket ./evidence` preserves keys as relative paths.
        var full = Path.GetFullPath(Path.Combine(evidenceRoot, s3Key.Replace('/', Path.DirectorySeparatorChar)));
        if (!full.StartsWith(Path.GetFullPath(evidenceRoot), StringComparison.OrdinalIgnoreCase))
            throw new InvalidDataException($"Key escapes evidence root: {s3Key}");
        return full;
    }

    public static byte[] ReadGzip(string path)
    {
        using var file = File.OpenRead(path);
        using var gz = new GZipStream(file, CompressionMode.Decompress);
        using var ms = new MemoryStream();
        gz.CopyTo(ms);
        return ms.ToArray();
    }

    public static string Hex(byte[] bytes) => Convert.ToHexStringLower(bytes);
}
TrailIntegrity/Program.cs
using System.Text.Json;
using Harbourline.Security.TrailIntegrity;

if (args.Length < 1)
{
    Console.Error.WriteLine("Usage: TrailIntegrity <evidence-folder> [public-keys.json] [signatures.json]");
    return 2;
}

var validator = new TrailValidator(args[0]);

// public-keys.json = output of: aws cloudtrail list-public-keys --start-time ... --end-time ...
if (args.Length > 1)
{
    using var doc = JsonDocument.Parse(File.ReadAllText(args[1]));
    foreach (var key in doc.RootElement.GetProperty("PublicKeyList").EnumerateArray())
        validator.PublicKeys[key.GetProperty("Fingerprint").GetString()!] = Convert.FromBase64String(key.GetProperty("Value").GetString()!);
}

// signatures.json = { "<digest key>": "<hex from x-amz-meta-signature>" } for the newest digest(s)
if (args.Length > 2)
{
    var sigs = JsonSerializer.Deserialize<Dictionary<string, string>>(File.ReadAllText(args[2]))!;
    foreach (var (k, v) in sigs) validator.KnownSignatures[k] = v;
}

var findings = validator.Validate();
foreach (var f in findings.Where(f => f.Severity != Severity.Ok))
    Console.WriteLine($"{f.Severity,-8} {f.Subject}\n         {f.Message}");

var ok = findings.Count(f => f.Severity == Severity.Ok);
var invalid = findings.Count(f => f.Severity == Severity.Invalid);
Console.WriteLine($"\n{ok} checks passed, {invalid} invalid, {findings.Count(f => f.Severity == Severity.Warning)} warnings");
return invalid == 0 ? 0 : 1;

Run it:

aws s3 sync s3://hl-m05-cloudtrail-111122223333/AWSLogs ./evidence/AWSLogs --profile learn-sandbox
aws cloudtrail list-public-keys --start-time 2026-11-18T00:00:00Z --profile learn-sandbox > public-keys.json
dotnet run --project TrailIntegrity -- ./evidence public-keys.json

Then simulate tampering on your local copy (never the bucket): decompress one log file, change a sourceIPAddress, recompress, and run again. You should see Hash MISMATCH for that file and exit code 1.

Code explained line by line

  1. DigestFile / DigestLogFile — records for only the fields validation needs; System.Text.Json ignores the rest, so new fields AWS adds won't break the tool.
  2. previousDigest* properties are nullable: the starting digest (when validation was first enabled or re-enabled) has nulls there.
  3. EnumerateFiles(..., "*CloudTrail-Digest*.json.gz", AllDirectories) — digest files live under CloudTrail-Digest/<region>/yyyy/MM/dd/; log files live under CloudTrail/, so the pattern separates them.
  4. ReadGzip — CloudTrail hashes the uncompressed content of both log and digest files, so every hash is computed after decompression.
  5. OrderBy(DigestEndTime) — ISO 8601 UTC strings sort correctly as plain strings; we process the chain oldest-first.
  6. The "location recorded inside it" check mirrors AWS's guidance: a digest copied to a different path should not validate, otherwise an attacker could swap a genuine digest from another hour into place.
  7. CheckLogFile — missing log files are as serious as modified ones: deleting the log that recorded DeleteBucket is a classic cover-up.
  8. The chain check compares previousDigestHashValue with the SHA-256 of the previous digest's uncompressed bytes. A deleted or altered digest breaks the chain.
  9. outsideRange — the oldest digest in your evidence set naturally points at a predecessor you didn't download; that's a warning, not tampering.
  10. Signatures: the newest digest's signature lives only in S3 object metadata (x-amz-meta-signature), so it must be captured separately; every older digest's signature is in the next digest's previousDigestSignature field — the code finds it there.
  11. VerifySignature rebuilds the documented data-signing string (end time, bucket/key, hex SHA-256 of the digest, previous signature) and verifies SHA-256 with RSA PKCS#1 v1.5 padding. The "null" fallback for a starting digest is our assumption ⚠ VERIFY against a real starting digest.
  12. ImportRSAPublicKey — ListPublicKeys returns the key as DER in PKCS#1 format; the CLI shows it base64-encoded, so Program.cs decodes base64 first.
  13. ToLocal refuses keys that escape the evidence root (../), because the digest content is input you are, by definition, not yet trusting.
  14. Convert.ToHexStringLower — CloudTrail writes lower-case hex; comparisons are case-insensitive anyway.
  15. Exit code 0/1 lets you run the tool in a scheduled job and alert on failure.

Security considerations

⚠ Learning shortcut — not for production

The lab trail writes to a bucket in the same account, uses SSE-S3 instead of a KMS customer managed key, uses Object Lock governance mode for 1 day, and has no SCP. In production: an organisation trail delivering to the log-archive account, SSE-KMS with a dedicated key whose policy lets only CloudTrail encrypt and named auditor roles decrypt, Object Lock compliance mode with the retention your records-management policy sets (agreed with legal and compliance), an SCP protecting trails, alarms on StopLogging/DeleteTrail/key changes, and scheduled validation with stored evidence.

  • Separate the duties. People who administer workloads must not administer the log archive. Use separate Identity Center permission sets, and require two people for any change to the log bucket or key policy.
  • CloudTrail logs contain personal data. Source IPs, usernames and request parameters can identify people. UK GDPR still applies: document the purpose, restrict access, and define retention. Immutable logs and the right to erasure pull in different directions — the UK GDPR lesson in Part 6 covers how banks handle that tension with legal advice.
  • Don't log secrets. CloudTrail records request parameters of management calls; most services redact sensitive fields, but your own code must never put secrets in resource names, tags or descriptions.
  • KMS key deletion is a log-destruction attack. If someone schedules deletion of the trail's KMS key, every log becomes unreadable after the waiting period. Alert on ScheduleKeyDeletion and DisableKey for that key, and restrict those actions in the key policy.
  • Compliance-mode retention is a financial commitment. You'll pay for storage for the full period and can't shorten it. Get sign-off before enabling it.
  • Validate from somewhere else. Run validation from a security automation account, not from the account being audited.

Banking use case

Harbourline's audit-logging standard, written by security and approved by compliance, says:

RequirementImplementation
All administrative actions in every account and Region are recordedOrganisation trail, multi-Region, management events read + write
Access to customer documents and card-token data is recordedS3 data events for hl-prod-customer-docs-* and hl-prod-card-vault-* only; DynamoDB data events for the token table
Logs are protected from modification and deletionLog-archive account; bucket policy; SCP; Object Lock compliance mode with the retention period from the records-management policy
Integrity is provableValidation enabled; daily validate-logs run by security automation; results stored as evidence
Logs are reviewedEventBridge rules for high-risk events to the SOC; monthly Athena reports of IAM and network changes reviewed by the change board
Cost is controlledNo data events on high-volume, low-sensitivity buckets (static assets, Athena results); lifecycle to Glacier Instant Retrieval after 90 days

Rough cost reasoning (fictional numbers): 120 accounts generate about 400 million management events a month — free for the first copy. Data events for the sensitive buckets: about 300 million per month × USD 0.10 per 100,000 = USD 300 per month. Turning on data events for all S3 buckets would have been around 40 billion events — USD 40,000 a month. That difference is why the standard names specific buckets.

Common mistakes

  • Relying on event history → only 90 days, one Region at a time, no integrity proof → create a trail on day one.
  • Trail bucket in the workload account → an attacker with admin deletes the evidence → deliver to a separate log-archive account.
  • Turning validation off and on again → the chain breaks, and there are no digests for the gap → keep it on; if it was ever off, record why.
  • Data events for everything → five-figure monthly bills → select buckets and tables by sensitivity with advanced event selectors.
  • Compliance-mode Object Lock in a sandbox → you pay to keep test logs for years → governance mode and short retention in labs.
  • Bucket policy without aws:SourceArn → any trail in any account could write into your bucket ("confused deputy") → add the condition.
  • Forgetting the KMS key policy → trail creation fails with InsufficientEncryptionPolicyException → grant CloudTrail GenerateDataKey* with the encryption-context condition.

Troubleshooting

SymptomLikely causeFix
InsufficientS3BucketPolicyException on create-trailBucket policy missing CloudTrail statements, wrong path or wrong aws:SourceArnCompare with the policy above; for org trails use the AWSLogs/<org-id>/* path
InsufficientEncryptionPolicyExceptionKMS key policy doesn't allow CloudTrailAdd kms:GenerateDataKey* for cloudtrail.amazonaws.com with the encryption-context condition
No files in the bucket after 15 minutesTrail not logging, or no API activityget-trail-status: check IsLogging and LatestDeliveryError
validate-logs says no digest files foundLess than an hour since validation started, or wrong --start-timeWait for the first hourly digest; widen the time range
validate-logs reports INVALID: not found for a log fileSomeone deleted it, or a lifecycle rule expired itCheck S3 versioning/Object Lock and lifecycle rules; treat unexplained deletions as an incident
LatestDigestDeliveryError populatedBucket policy or KMS change blocked digest deliveryFix the policy; CloudTrail retries; the chain may temporarily look broken
Athena query scans terabytesMissing partition filterAlways filter on day (and region); use partition projection
Org trail not visible in a member accountExpected; member accounts see it read-onlyManage it from the management or delegated admin account

Cost and cleanup

Item (checked 2026-11-18)PriceLab exposure
First copy of management events to S3FreeFree
Data eventsUSD 0.10 per 100,000Only if you add the optional data-event selector; pennies
S3 storage (Standard)About USD 0.023 per GB-monthA few MB — under USD 0.01
AthenaPer TB scanned ⚠ VERIFY current rateA few MB scanned — effectively free (10 MB minimum per query)
KMS customer managed key (optional)USD 1 per month, proratedSkip for the free version

Teardown. Objects protected by governance-mode retention can be deleted only by a principal with s3:BypassGovernanceRetention, using the bypass header — or you can simply wait one day for retention to expire.

PROFILE=learn-sandbox
ACCOUNT=$(aws sts get-caller-identity --query Account --output text --profile $PROFILE)
BUCKET=hl-m05-cloudtrail-$ACCOUNT

# 1. Stop and delete the trail
aws cloudtrail stop-logging --name hl-m05-audit-trail --profile $PROFILE
aws cloudtrail delete-trail --name hl-m05-audit-trail --profile $PROFILE

# 2. After retention expires (1 day), delete all object versions, then the bucket.
#    (Or, with s3:BypassGovernanceRetention, add --bypass-governance-retention to delete-objects.)
aws s3api list-object-versions --bucket $BUCKET --profile $PROFILE \
  --query '{Objects: [Versions, DeleteMarkers][][].{Key: Key, VersionId: VersionId}}' --output json > versions.json
aws s3api delete-objects --bucket $BUCKET --delete file://versions.json --profile $PROFILE
aws s3api delete-bucket --bucket $BUCKET --profile $PROFILE

# 3. Athena: drop the table
#    DROP TABLE hl_cloudtrail;

# 4. Confirm nothing remains
aws cloudtrail describe-trails --query "trailList[?Name=='hl-m05-audit-trail']" --profile $PROFILE
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --query "ResourceTagMappingList[].ResourceARN" --profile $PROFILE

delete-objects accepts at most 1,000 keys per call; repeat if you have more.

Exercises

  • 🟢 Guided: Create the lab trail with validation, wait an hour, run validate-logs, sync the logs locally and run TrailIntegrity. Then tamper with one local log file and one local digest file (separately) and record what each tool reports.
  • 🟡 Challenge: Write an EventBridge rule (CLI or IaC) that sends an SNS email when StopLogging, DeleteTrail or UpdateTrail is called, or when PutEventSelectors is called on any trail. Test it by calling StopLogging on your lab trail and starting it again.
  • 🔴 Live problem: An internal auditor emails: "For the Payments prod account, please provide evidence that all IAM changes between 1 July and 30 September were logged, that the logs are complete and unaltered, and a list of those changes with who approved them." Plan what you'll produce, from which sources, and what you'll say about the parts CloudTrail can't prove.

Assignment

Add an "audit-trail design" to your portfolio: the organisation-trail architecture diagram, the bucket policy, KMS key policy and SCP as files, a decision table for which data events Harbourline records (with cost estimates), the Athena DDL and three investigation queries, and the TrailIntegrity tool with a short README showing a tamper test.

Solutions

Knowledge check

Interview questions

  • "How would you design audit logging for a multi-account bank on AWS?" — Organisation trail, multi-Region, management events, selective data events; delivered to a log-archive account with SSE-KMS, Object Lock and a strict bucket policy; validation on; SCP protecting trails; EventBridge alerts on tampering; Athena or SIEM for analysis; documented retention agreed with compliance.
  • "How do you prove to an auditor that CloudTrail logs weren't altered?" — Log file integrity validation: hourly signed digests with hashes of every log file, chained by signatures. Run validate-logs (or an offline validator) and present the results along with the configuration showing validation was continuously on.
  • "What would you check first if you suspect an access key was compromised?" — Athena query on useridentity.accesskeyid across all Regions for the relevant period, focusing on write events, IAM changes, new access keys, AssumeRole calls and data access; then follow assumed-role sessions using the session issuer.
  • "Why not enable S3 data events on every bucket?" — Cost and noise. Data events are priced per event; at bank scale, logging every object read in high-volume buckets is extremely expensive. Classify buckets and log the sensitive ones.

Certification practice

Completion checklist

  • I can explain management, data, network activity and Insights events, and which cost money
  • I created a multi-Region trail with log file validation and saw logs and digests arrive
  • I ran validate-logs and the offline TrailIntegrity tool, and detected a tampered local copy
  • I can describe the log-archive account pattern, the bucket policy and the protective SCP
  • I queried CloudTrail logs with Athena using partition filters
  • I deleted the trail and bucket (after retention) and confirmed nothing tagged remains

Key takeaways

  • Event history isn't an audit trail. Create an organisation trail, multi-Region, delivering to a separate log-archive account.
  • Management events (first copy) are free; data events are where cost and much of the investigative value lie — choose them by data sensitivity.
  • Integrity validation makes tampering detectable through hourly signed, chained digests; Object Lock, bucket policies, SCPs and KMS make it hard in the first place.
  • CloudTrail answers who did what to which AWS resource, when and from where. It doesn't cover OS, database or application actions, and it doesn't record approvals.
  • CloudTrail Lake is closed to new customers; use S3 plus Athena for analysis and EventBridge or CloudWatch for alerting.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page