Skip to Content
Cloud Engineering Academy
14. Advanced Banking Architecture

Audit and compliance architecture

Design a tamper-evident audit trail with CloudTrail, a locked log archive, application audit records and evidence collection, answer who-accessed-what questions with Athena, and verify a hash-chained audit log in C#.

advanced⏱ 2 h 30 min🟢 FreeSAP-C03SCS-C03SAA-C03

Assumes: .NET 10 LTS (file-based apps) · AWS SDK for .NET v4 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Separate the layers of audit evidence: AWS control plane, AWS data plane, application, network and configuration.
  • Design a log-archive account with an organisation trail, log-file validation, KMS encryption and S3 Object Lock.
  • Build an evidence catalogue that maps each control to its evidence source, query, retention and owner.
  • Answer "who accessed this account's data, and when?" using CloudTrail data events, application audit records and Athena.
  • Explain why a hash-chained audit log is tamper-evident, and run a C# verifier that finds a forged or deleted record.
  • Avoid closed services (CloudTrail Lake, Audit Manager) and use the supported alternatives, marking every compliance statement as needing review.

Prerequisites

Why this matters in banking

Audits and incidents ask the same question in different forms: what happened, who did it, and can you prove the record is complete and unaltered? The prompts sound like:

  • A regulator: "Show every access to customer account records by support staff in March."
  • Internal audit: "Who changed the production payment configuration on 3 October, and who approved it?"
  • A security incident: "Was this IAM role used from outside the UK between 02:00 and 04:00?"
  • A data-protection request: "Which staff viewed this customer's file?"

If logs are scattered, editable by administrators, or deleted after 30 days, the honest answer is "we do not know". That is itself a finding. Audit architecture is about making the right answers cheap, fast and credible.

Educational content, not compliance advice

Which events must be logged, for how long, who may read them, and how they must be protected is set by your regulators, PCI DSS (where card data is in scope), contracts and your data-protection obligations ⚠ VERIFY every retention period and requirement locally. Audit logs themselves contain personal data (user identities, IP addresses), so they need their own access control and retention decisions. Nothing here claims that a design is compliant.

The idea in plain language

An audit trail has four properties. If one is missing, it is not evidence:

PropertyMeaningMechanism
CompleteEvery relevant action is recordedOrganisation-wide trails, data events for sensitive stores, application audit events
AttributableEach record names a real actorFederated identity, no shared accounts, role session names tied to people
Tamper-evidentChanges or deletions are detectableLog-file validation, hash chains, Object Lock, separate account
RetrievableYou can query it when askedAthena over partitioned logs, tested queries, defined retention

Analogy: a ship's logbook. Pages are numbered, entries are in ink with the officer's name, nothing is torn out, and the book goes into the safe in a different building each month. Anyone can see if a page is missing.

Layers of evidence

LayerQuestion it answersAWS source
Control planeWho changed a resource or permission?CloudTrail management events
Data planeWho read or wrote this object or item?CloudTrail data events for S3, DynamoDB and others (opt-in, billed)
IdentityWho signed in, with what MFA, from where?IAM Identity Center and IdP logs, CloudTrail sign-in events
NetworkWhat talked to what?VPC Flow Logs, DNS query logs, firewall logs
ConfigurationWas this resource compliant on this date?AWS Config history and conformance packs
FindingsWhat did detection services see?GuardDuty, Security Hub (the unified service and Security Hub CSPM have different names; use the right one) ⚠ VERIFY
ApplicationWhich staff member viewed which customer's account, and why?Your own audit events

The application layer is the one teams forget. CloudTrail will say a role read a DynamoDB item; it cannot say which human support agent did it on a customer's behalf, or the case reference that justified it. Only your application knows.

What problem the AWS services solve

NeedServiceHonest limits
Record API activityCloudTrail (organisation trail)Management events are on by default in the event history for 90 days; a trail delivers them to S3 for retention. Data events cost extra and can be high volume
Prove logs are unalteredCloudTrail log-file validationValidates digest files signed by AWS; proves integrity of what was delivered, not that nothing was never logged
Immutable storageS3 Object Lock in a log-archive accountCompliance mode cannot be shortened; test retention before applying
Query historyAthena over CloudTrail logs in S3Needs partitioning (or partition projection) to stay affordable
Continuous compliance stateAWS Config with conformance packsConfig evaluates configuration, not application behaviour
Evidence workflowsYour own evidence catalogue plus S3AWS Audit Manager no longer onboards new accounts (from 2026-04-30); build the catalogue yourself ⚠ VERIFY current status
SQL over trails inside CloudTrailCloudTrail LakeClosed to new customers since 2026-05-31: use CloudTrail to S3 plus Athena

Two availability notes: do not design a new platform on CloudTrail Lake or Audit Manager. Re-check both statuses before relying on this text ⚠ VERIFY.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Activity LogCloudTrail management eventsActivity Log is per subscription; CloudTrail can be organisation-wide
Diagnostic settings to a storage accountTrail to S3, service logs to S3 or CloudWatch LogsMany services need their own log enablement
Microsoft Purview AuditApplication audit events plus CloudTrailPurview covers Microsoft 365; your own apps need their own audit design
Azure Policy compliance stateAWS Config rules and conformance packsConfig keeps a history timeline per resource
Immutable blob storageS3 Object LockGovernance versus Compliance modes
Azure Monitor Log Analytics (KQL)CloudWatch Logs Insights, AthenaAthena queries S3 in place using SQL
Microsoft Defender for Cloud regulatory dashboardsSecurity Hub standards, Config conformance packsStandards are checklists of automated controls, not certification

Architecture

Security-boundary view

The people and roles who can do things in workload accounts cannot delete or alter what the log-archive account holds. That separation is the heart of the design.

The 15-section design

Sections 1 to 3: requirements

SectionContent
1 BusinessProvide a credible, queryable record of who did what to systems and customer data, across all Harbourline AWS accounts, to support audits, investigations and data-subject requests
2 FunctionalF1 Record all management events organisation-wide. F2 Record data events for evidence and customer-data stores. F3 Record application-level audit events with a reason or case ID. F4 Validate log integrity. F5 Provide who-accessed-what queries. F6 Alert on attempts to disable logging. F7 Map controls to evidence
3 Non-functionalCompleteness: no gaps longer than 15 minutes, detected within 5. Query: a 30-day who-accessed-what query returns in under 60 seconds. Retention: 7 years for evidence logs ⚠ VERIFY; shorter for verbose network logs. Integrity: any record change or deletion detectable. Availability of audit query: 99.5%

Section 5: service selection

ChoiceWhyRejected alternativeTrade-off
Organisation trail to a log-archive accountComplete coverage, separate from workload adminsPer-account trails in each accountOne more account; strong separation
Object Lock (Compliance) on the log bucketImmutable for the retention periodVersioning onlyCannot undo a mistaken long retention
Athena on S3 logsCheap, serverless, SQLCloudTrail LakeClosed to new customers; Athena needs partitioning discipline
Application audit events with a hash chainTamper evidence at the application layerPlain database tableSlightly more code
Config conformance packs and Security HubContinuous control evidenceSpreadsheetsSetup effort; they check configuration only

Sections 6 and 7: network and security

  • Network. Log delivery uses AWS service principals, not your VPC. Analysts query through Athena over the console or CLI, using VPC endpoints from restricted networks (VPC endpoints).
  • Identity. Auditors use read-only SSO roles with MFA; nobody has standing write access to the log bucket. Break-glass roles exist, are alarmed and reviewed.
  • Keys. A KMS key in the log-archive account. The key policy allows CloudTrail to encrypt and a small audit role to decrypt; deletion is blocked by SCP and key-deletion waiting periods.
  • Guardrails. An SCP denies cloudtrail:StopLogging, cloudtrail:DeleteTrail, cloudtrail:UpdateTrail (except for a named role) and organizations:LeaveOrganization.
  • Data protection. Logs hold identities and IP addresses. Limit read access, set retention per log type, and agree with the data-protection officer how data-subject access requests treat logs ⚠ VERIFY.
  • Clock integrity. Use Amazon Time Sync; investigations rely on timestamps.

Sections 8 to 10: API, data and flow

  • Audit query API (internal). GET /audit/accounts/{accountId}/access?from=&to= returning access events with actor, action, outcome and case reference. Requires a privileged role, and the query itself is audited.
  • Application audit record.
FieldExample
seq, prevHash, hashChain fields
at2026-11-18T09:05:00Z
actorrole/support-agent-7 plus staff ID
actionViewAccount, ExportStatement
resourceACC-0000000001
outcomeallowed, denied
caseRefCASE-48211
  • Flow. (1) Service performs an action. (2) It writes an audit record in the same transaction as the action where possible. (3) The record goes to the audit bus. (4) Delivered to the log-archive bucket. (5) Athena queries partitions by date. (6) A periodic job verifies the hash chain and stores its head hash in the evidence bucket.

Section 11: failure scenarios

FailureDetectionAutomatic responseImpactManual step
Trail stopped or deletedConfig rule and EventBridge alert on StopLoggingSCP prevents; alert if it happens anywayGap in evidenceRestore, document the gap
Log delivery fails (bucket policy or KMS error)CloudTrail delivery error metricAlarmDelivery delayed (CloudTrail retries within limits)Fix permissions
Insider edits application audit tableHash-chain verifier failsAlert; compare with archived head hashesIntegrity incidentInvestigate
KMS key disabled or scheduled for deletionKey-state alarmSCP and waiting periodLogs unreadableCancel deletion
Data-event volume explodes costCost anomaly alarmNoneBill spikeNarrow selectors, sample
Logs missing for an investigation periodGap detection jobAlertEvidence gapReconstruct from other sources, record the gap
Wrong retention appliedChange reviewNoneOver-retention or early expiryCorrect going forward; compliance review
Query too slow or costlyAthena workgroup metricsScan limitDelayPartition, convert to Parquet, narrow date range

Sections 12 to 15

  • Scalability. Log volume grows with accounts and data events. Partition by date and account; convert hot logs to Parquet; set per-log-type retention. Reserve data events for stores that need them.
  • Cost. Management events delivered by one trail are free to start; data events, extra trails, S3 storage, Athena scans, KMS requests and Config items bill ⚠ VERIFY prices. The largest surprise is usually S3 data events on busy buckets. Estimate events per day times price, per store.
  • Implementation. Phase 1 log-archive account, KMS, bucket with Object Lock. Phase 2 organisation trail with validation. Phase 3 SCP guardrails and alerts. Phase 4 Athena tables and tested queries. Phase 5 application audit events with hash chain. Phase 6 evidence catalogue and an audit rehearsal.
  • Compliance touchpoints.
AreaWhere it touchesReviewing teamEvidence
Logging and monitoring requirements (for example PCI DSS requirement 10) ⚠ VERIFYTrail scope, retention, reviewSecurity, QSA/assessorControl mapping, sample queries
Records retentionRetention periods per log typeCompliance, legalRetention schedule
Data protectionPersonal data in logs, access to logsData protection officerAccess review, DPIA entry
Segregation of dutiesWho can alter logsInternal controlRole design
Audit readinessEvidence catalogueInternal auditRehearsal outcome

Step-by-step implementation

Create the log-archive account and bucket

Encrypted with a customer-managed key, Block Public Access on, versioning on, Object Lock enabled at creation. Use Governance mode and a short retention in the sandbox; the production retention is a compliance decision.

Create an organisation trail with validation

Multi-Region, management events on, log-file validation on, encrypted, delivering to the archive bucket. Add data-event selectors only for the stores that need them.

Add guardrails

An SCP denying changes to logging, plus alerts (EventBridge rules) for StopLogging, DeleteTrail, PutBucketPolicy on the log bucket and ScheduleKeyDeletion.

Build and test the queries

Create Athena tables with partition projection and write the standing queries (who accessed an account, who changed a role, root usage). Test them against a known event you create.

Add application audit events with a chain

Write audit records for every access to customer data, including the reason or case reference. Verify the chain on a schedule and archive the head hash.

Write the evidence catalogue and rehearse an audit

Each control lists its evidence source, the query, the owner and the retention. Pick five controls and answer them live in 15 minutes.

AWS Console instructions

  • Console → CloudTrail → Trails → Create trail. Name it, choose Create new S3 bucket or select the archive bucket, enable Log file validation, choose KMS encryption, select Multi-Region trail and, in an organisation management account, Enable for all accounts in my organization.
  • Event type step: keep Management events on; add Data events for chosen S3 buckets or DynamoDB tables only.
  • Console → Athena → Query editor to run saved queries against the CloudTrail table.
  • Console → AWS Config → Conformance packs to deploy a pack and view compliance history per resource.

AWS CLI

# Create an encrypted, multi-Region trail with log-file validation.
aws cloudtrail create-trail --name hl-m14-audit-trail \
  --s3-bucket-name hl-m14-audit-logs \
  --is-multi-region-trail --enable-log-file-validation \
  --kms-key-id alias/hl-m14-audit \
  --profile learn-sandbox --region us-east-1
aws cloudtrail start-logging --name hl-m14-audit-trail --profile learn-sandbox --region us-east-1

# Record data events for one evidence bucket only (not all buckets).
aws cloudtrail put-event-selectors --trail-name hl-m14-audit-trail \
  --advanced-event-selectors '[{"Name":"Evidence bucket data events","FieldSelectors":[{"Field":"eventCategory","Equals":["Data"]},{"Field":"resources.type","Equals":["AWS::S3::Object"]},{"Field":"resources.ARN","StartsWith":["arn:aws:s3:::hl-m14-evidence-staging/"]}]}]' \
  --profile learn-sandbox --region us-east-1

# Prove the delivered logs are intact over a time range.
aws cloudtrail validate-logs \
  --trail-arn arn:aws:cloudtrail:us-east-1:111122223333:trail/hl-m14-audit-trail \
  --start-time 2026-11-17T00:00:00Z --end-time 2026-11-18T00:00:00Z \
  --profile learn-sandbox --region us-east-1

# Quick look at recent control-plane events (90-day event history).
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=StopLogging \
  --max-results 5 --profile learn-sandbox --region us-east-1

Example Athena query for "who read objects in the evidence bucket" (table name and column details depend on how you created the CloudTrail table ⚠ VERIFY):

SELECT eventtime, useridentity.arn AS actor, eventname,
       json_extract_scalar(requestparameters, '$.key') AS object_key, sourceipaddress
FROM cloudtrail_logs
WHERE eventsource = 's3.amazonaws.com'
  AND eventname IN ('GetObject', 'HeadObject')
  AND json_extract_scalar(requestparameters, '$.bucketName') = 'hl-m14-evidence-staging'
  AND eventtime >= '2026-11-01T00:00:00Z'
ORDER BY eventtime DESC
LIMIT 100;

Infrastructure as Code

AuditTrailStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.CloudTrail;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.S3;
using Constructs;

namespace Harbourline.Architecture.Audit;

public class AuditTrailStack : Stack
{
    public AuditTrailStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        var key = new Key(this, "AuditKey", new KeyProps
        {
            Alias = "alias/hl-m14-audit",
            EnableKeyRotation = true,
        });

        var logs = new Bucket(this, "AuditLogs", new BucketProps
        {
            Encryption = BucketEncryption.KMS,
            EncryptionKey = key,
            BucketKeyEnabled = true,
            BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
            EnforceSSL = true,
            Versioned = true,
            ObjectLockEnabled = true,
            // Learning shortcut: Governance + 1 day. Production retention is a compliance decision.
            ObjectLockDefaultRetention = ObjectLockRetention.Governance(Duration.Days(1)),
            RemovalPolicy = RemovalPolicy.RETAIN,
        });

        var evidenceBucket = new Bucket(this, "EvidenceStaging", new BucketProps
        {
            Encryption = BucketEncryption.S3_MANAGED,
            BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
            EnforceSSL = true,
            RemovalPolicy = RemovalPolicy.DESTROY,
            AutoDeleteObjects = true,
        });

        var trail = new Trail(this, "AuditTrail", new TrailProps
        {
            TrailName = "hl-m14-audit-trail",
            Bucket = logs,
            EncryptionKey = key,
            EnableFileValidation = true,
            IsMultiRegionTrail = true,
            IncludeGlobalServiceEvents = true,
            ManagementEvents = ReadWriteType.ALL,
        });

        // Data events for one evidence bucket only: broad data-event logging is a cost trap.
        trail.AddS3EventSelector(
            [new S3EventSelector { Bucket = evidenceBucket }],
            new AddEventSelectorOptions { ReadWriteType = ReadWriteType.ALL });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m14-l15");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        Amazon.CDK.Tags.Of(this).Add("Owner", "learner");
    }
}

This is an account-level trail for learning. An organisation trail is created from the management (or delegated administrator) account with IsOrganizationTrail = true.

Complete code

CloudTrail protects AWS API activity. For your own application audit records, a hash chain gives the same tamper evidence: each record includes the hash of the one before it, so editing, reordering or deleting any record breaks the chain from that point on. The program also shows the who-accessed-what query.

AuditChain.cs
// Run with: dotnet run AuditChain.cs
// A hash-chained audit trail: each record commits to the one before it, so editing or
// deleting history is detectable. Also shows a "who accessed what" query.
using System.Security.Cryptography;
using System.Text;

var chain = new List<AuditRecord>();
void Add(string at, string actor, string action, string resource, string outcome) =>
    chain.Add(AuditRecord.Next(chain.LastOrDefault(), DateTimeOffset.Parse(at), actor, action, resource, outcome));

Add("2026-11-18T09:00:00Z", "role/support-agent-7", "ViewAccount", "ACC-0000000001", "allowed");
Add("2026-11-18T09:05:00Z", "role/support-agent-7", "ViewAccount", "ACC-0000000002", "allowed");
Add("2026-11-18T09:10:00Z", "role/ops-admin", "FreezeAccount", "ACC-0000000001", "allowed");
Add("2026-11-18T09:12:00Z", "role/support-agent-9", "ViewAccount", "ACC-0000000001", "denied");
Add("2026-11-18T09:20:00Z", "role/support-agent-7", "ExportStatement", "ACC-0000000001", "allowed");

Console.WriteLine("Who touched ACC-0000000001?");
foreach (var r in chain.Where(r => r.Resource == "ACC-0000000001"))
    Console.WriteLine($"  #{r.Seq} {r.At:HH:mm} {r.Actor} {r.Action} -> {r.Outcome}");

Console.WriteLine("Verify intact chain : " + AuditChain.Verify(chain));

// An insider rewrites a denied attempt as allowed, keeping that record's own hash consistent.
var forged = new List<AuditRecord>(chain);
var original = forged[3];
forged[3] = AuditRecord.Next(forged[2], original.At, original.Actor, original.Action, original.Resource, "allowed");
Console.WriteLine("Verify forged record: " + AuditChain.Verify(forged));

var truncated = chain.Where(r => r.Seq != 2).ToList();   // a record is deleted
Console.WriteLine("Verify deleted row  : " + AuditChain.Verify(truncated));
return 0;

public sealed record AuditRecord(long Seq, DateTimeOffset At, string Actor, string Action, string Resource,
    string Outcome, string PrevHash, string Hash)
{
    public const string GenesisHash = "0000000000000000000000000000000000000000000000000000000000000000";

    public static AuditRecord Next(AuditRecord? previous, DateTimeOffset at, string actor, string action,
        string resource, string outcome)
    {
        var seq = (previous?.Seq ?? 0) + 1;
        var prevHash = previous?.Hash ?? GenesisHash;
        return new AuditRecord(seq, at, actor, action, resource, outcome, prevHash,
            Compute(seq, at, actor, action, resource, outcome, prevHash));
    }

    public static string Compute(long seq, DateTimeOffset at, string actor, string action, string resource,
        string outcome, string prevHash)
    {
        var text = string.Join('|', prevHash, seq, at.ToUniversalTime().ToString("O"), actor, action, resource, outcome);
        return Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(text))).ToLowerInvariant();
    }
}

public static class AuditChain
{
    public static string Verify(IReadOnlyList<AuditRecord> records)
    {
        var prevHash = AuditRecord.GenesisHash;
        long expectedSeq = 1;
        foreach (var r in records)
        {
            if (r.Seq != expectedSeq) return $"BROKEN: sequence gap before #{r.Seq} (expected #{expectedSeq})";
            if (r.PrevHash != prevHash) return $"BROKEN: #{r.Seq} does not link to the record before it";
            if (AuditRecord.Compute(r.Seq, r.At, r.Actor, r.Action, r.Resource, r.Outcome, r.PrevHash) != r.Hash)
                return $"BROKEN: #{r.Seq} content does not match its hash";
            prevHash = r.Hash;
            expectedSeq++;
        }
        return $"OK ({records.Count} records, head {prevHash[..12]}...)";
    }
}

Expected output (the head hash will match this run only if the inputs are identical):

Who touched ACC-0000000001?
  #1 09:00 role/support-agent-7 ViewAccount -> allowed
  #3 09:10 role/ops-admin FreezeAccount -> allowed
  #4 09:12 role/support-agent-9 ViewAccount -> denied
  #5 09:20 role/support-agent-7 ExportStatement -> allowed
Verify intact chain : OK (5 records, head 582b3188b16c...)
Verify forged record: BROKEN: #5 does not link to the record before it
Verify deleted row  : BROKEN: sequence gap before #3 (expected #2)

Code explained line by line

  1. AuditRecord is an immutable record holding the business facts, the previous record's hash (PrevHash) and its own Hash.
  2. GenesisHash is a fixed all-zero string that the first record uses as its "previous" value.
  3. Next builds the following record: sequence number is previous plus one, PrevHash is the previous record's Hash, and Hash is computed over all the fields including PrevHash. That inclusion is what chains the records.
  4. Compute joins fields with | and hashes the UTF-8 bytes with SHA-256. The time is converted to UTC in round-trip format so the same instant always hashes the same.
  5. Verify walks the list and checks three things for each record: the sequence has no gap, PrevHash equals the previous record's hash, and recomputing the hash from the content matches the stored Hash.
  6. The forgery replaces record 4 with a new one whose own hash is internally consistent (the attacker recomputed it). The chain still breaks: record 5 stores the old hash of record 4, so it no longer links. Verification fails at #5.
  7. The deletion leaves a sequence gap, reported at the first record after the hole.
  8. The Where query at the top is the "who touched this account" answer. In production, this is an Athena or database query over archived audit events, and the run itself is recorded.
  9. One caveat that matters: an attacker who can rewrite every record from the point of change and recompute all later hashes can produce a consistent chain. The defence is to anchor the head hash outside the attacker's reach, for example by writing it to an Object Lock bucket in the log-archive account each hour, or signing it with a KMS key whose use is separately controlled ⚠ VERIFY design with your security team.

Security considerations

⚠ Learning shortcut — not for production

The lab trail lives in the same account as the workload, uses Governance-mode Object Lock for one day and the demo chain is in memory. Production uses a separate log-archive account, a retention set by compliance, SCP guardrails against disabling logs, the chain persisted in a durable store with head hashes anchored off-box, and a tested process for what happens when verification fails.

  • Separate the writers from the keepers. Workload administrators must not be able to alter or delete the archive.
  • Audit the auditors. Reading logs is itself an auditable act.
  • Log denials as well as successes. A burst of denied is often the first sign of probing.
  • Do not log secrets or full personal data. Log identifiers and reasons, not card numbers or statement contents.
  • Protect the key. If the KMS key is deleted, the archive is unreadable. Guard it with SCPs and long deletion windows.
  • Alert on logging changes. StopLogging, DeleteTrail, bucket policy changes and key changes must page someone.

Banking use case

A customer submits a data-subject access request and asks who at Harbourline has viewed their account in the last 12 months. The privacy team raises a ticket; an investigator with the audit role runs the standing "access to account" query, which joins application audit events (human actor, case reference) with CloudTrail data events (system access) for the account's items. The result lists 14 views by 3 staff members, each with a case reference. Separately, a quarterly internal-audit sample asks for evidence that nobody disabled logging: the investigator shows the Config compliance history for the trail rule and the CloudTrail digest validation output. Both answers come from a rehearsed evidence catalogue, not from a scramble.

Common mistakes

  • Relying on the 90-day event history. It is not an archive. Deliver to S3 with retention.
  • Logging everything. Blanket S3 data events on busy buckets create large bills and noise. Select what matters.
  • Admins can delete the log bucket. Then the evidence is only as strong as your weakest admin.
  • No application-layer audit. You can prove a role read data but not which human or why.
  • Untested queries. The first run should not be during the audit.
  • Designing around closed services. Use S3 plus Athena instead of CloudTrail Lake, and build your own evidence catalogue instead of depending on Audit Manager ⚠ VERIFY.
  • Claiming compliance from tooling. Config rules and Security Hub standards check configuration; they do not certify anything.

Troubleshooting

SymptomLikely causeFix
InsufficientS3BucketPolicyException creating a trailBucket policy does not allow CloudTrailAdd the two CloudTrail statements with the correct account path
Trail exists but no logs in S3Logging not started, or KMS key policy blocks CloudTrailRun start-logging; fix the key policy
validate-logs reports invalid or missing digestsTime range before validation was enabled, or logs altered/missingCheck trail start date; investigate any gap as an incident
Athena query scans huge data and is slowNo partition filterUse partition projection and filter by date and account
Hash-chain verification fails after a restoreRecords restored out of order or partiallyRestore in sequence order; compare with the anchored head hash
Cost spike after enabling data eventsSelector covers busy bucketsNarrow to the evidence stores; use sampling where acceptable
Cannot delete the log bucket during cleanupObject Lock retention activeWait for retention, or use the bypass permission in Governance mode

Cost and cleanup

The C# chain demo is free. The optional trail and buckets cost little for a few hours: management events on a first trail are free to start; the data-event selector, KMS requests and S3 storage are small at lab volumes ⚠ VERIFY. Object Lock delays deletion by the retention period.

# Stop and delete the trail first.
aws cloudtrail stop-logging --name hl-m14-audit-trail --profile learn-sandbox --region us-east-1
aws cloudtrail delete-trail --name hl-m14-audit-trail --profile learn-sandbox --region us-east-1

# Governance-mode objects need the bypass flag (role needs s3:BypassGovernanceRetention).
aws s3api list-object-versions --bucket hl-m14-audit-logs --profile learn-sandbox --region us-east-1
aws s3api delete-object --bucket hl-m14-audit-logs --key <key> --version-id <id> \
  --bypass-governance-retention --profile learn-sandbox --region us-east-1

cdk destroy --profile learn-sandbox
terraform destroy

# Confirm nothing tagged for the course remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --profile learn-sandbox --region us-east-1

Exercises

  • 🟢 Guided: run AuditChain.cs, then change record 3's actor in memory without recomputing its hash and predict the verification message.
  • 🟡 Challenge: write the evidence catalogue entry (control, evidence source, query, owner, retention, review frequency) for "no one can disable logging in production" using Config, SCP and CloudTrail evidence.
  • 🔴 Live problem: an investigator reports a 40-minute gap in application audit events during an incident. Describe how you would establish whether logs were lost, suppressed or never written, which sources you would cross-check, and how you would record the gap honestly in the incident report.

Assignment

Add architecture/audit-and-evidence/ to your portfolio: the 15-section document, a security-boundary diagram showing the log-archive account, five evidence-catalogue entries with tested Athena queries, and the audit chain with unit tests for forgery, deletion and reorder.

Solutions

Knowledge check

Interview questions

  • How do you make audit logs trustworthy? Complete coverage, named actors, separate account with Object Lock, log-file validation, hash chains for app records, alerts on tampering attempts.
  • How do you answer "who accessed this customer's data"? Application audit events for the human and case, CloudTrail data events for system access, joined by account and time, with the query rehearsed.
  • What are the cost traps in audit logging? Data events on busy stores, verbose network logs retained too long, unpartitioned Athena scans.
  • What do you do if logging is disabled? Alert, restore, record the gap honestly, and review why the SCP or process failed.
  • Can you say a system is compliant because Config shows green? No. Config checks configuration; compliance is assessed by qualified people over controls, processes and evidence.

Certification practice

Completion checklist

  • I can list the layers of audit evidence and name the AWS source for each
  • I ran the audit chain and explained the forged and deleted results
  • My design uses a separate log-archive account and avoids closed services
  • I wrote an evidence-catalogue entry with a tested query
  • I ran cleanup and confirmed nothing tagged Project=cea-lab remains

Key takeaways

  • Audit evidence must be complete, attributable, tamper-evident and retrievable.
  • Keep logs in a separate account with Object Lock, validation, a protected key and SCP guardrails.
  • Application audit events (who and why) complement CloudTrail (which role and when).
  • Use S3 plus Athena instead of CloudTrail Lake, and your own evidence catalogue instead of Audit Manager.
  • Retention, personal data in logs and compliance claims need specialist review.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page