Audit and compliance architecture
Design a tamper-evident audit trail with CloudTrail, a locked log archive, application audit records and evidence collection, answer who-accessed-what questions with Athena, and verify a hash-chained audit log in C#.
Assumes: .NET 10 LTS (file-based apps) · AWS SDK for .NET v4 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Separate the layers of audit evidence: AWS control plane, AWS data plane, application, network and configuration.
- Design a log-archive account with an organisation trail, log-file validation, KMS encryption and S3 Object Lock.
- Build an evidence catalogue that maps each control to its evidence source, query, retention and owner.
- Answer "who accessed this account's data, and when?" using CloudTrail data events, application audit records and Athena.
- Explain why a hash-chained audit log is tamper-evident, and run a C# verifier that finds a forged or deleted record.
- Avoid closed services (CloudTrail Lake, Audit Manager) and use the supported alternatives, marking every compliance statement as needing review.
Prerequisites
- CloudTrail for banks and Audit logging and evidence collection
- S3 protection, Object Lock and Macie
- AWS Config and conformance packs
- Transaction integrity and non-repudiation
- Regulatory reporting architecture for immutable evidence storage
- Tools: .NET 10 SDK or Docker for the audit-chain demo
Why this matters in banking
Audits and incidents ask the same question in different forms: what happened, who did it, and can you prove the record is complete and unaltered? The prompts sound like:
- A regulator: "Show every access to customer account records by support staff in March."
- Internal audit: "Who changed the production payment configuration on 3 October, and who approved it?"
- A security incident: "Was this IAM role used from outside the UK between 02:00 and 04:00?"
- A data-protection request: "Which staff viewed this customer's file?"
If logs are scattered, editable by administrators, or deleted after 30 days, the honest answer is "we do not know". That is itself a finding. Audit architecture is about making the right answers cheap, fast and credible.
Educational content, not compliance advice
Which events must be logged, for how long, who may read them, and how they must be protected is set by your regulators, PCI DSS (where card data is in scope), contracts and your data-protection obligations ⚠ VERIFY every retention period and requirement locally. Audit logs themselves contain personal data (user identities, IP addresses), so they need their own access control and retention decisions. Nothing here claims that a design is compliant.
The idea in plain language
An audit trail has four properties. If one is missing, it is not evidence:
| Property | Meaning | Mechanism |
|---|---|---|
| Complete | Every relevant action is recorded | Organisation-wide trails, data events for sensitive stores, application audit events |
| Attributable | Each record names a real actor | Federated identity, no shared accounts, role session names tied to people |
| Tamper-evident | Changes or deletions are detectable | Log-file validation, hash chains, Object Lock, separate account |
| Retrievable | You can query it when asked | Athena over partitioned logs, tested queries, defined retention |
Analogy: a ship's logbook. Pages are numbered, entries are in ink with the officer's name, nothing is torn out, and the book goes into the safe in a different building each month. Anyone can see if a page is missing.
Layers of evidence
| Layer | Question it answers | AWS source |
|---|---|---|
| Control plane | Who changed a resource or permission? | CloudTrail management events |
| Data plane | Who read or wrote this object or item? | CloudTrail data events for S3, DynamoDB and others (opt-in, billed) |
| Identity | Who signed in, with what MFA, from where? | IAM Identity Center and IdP logs, CloudTrail sign-in events |
| Network | What talked to what? | VPC Flow Logs, DNS query logs, firewall logs |
| Configuration | Was this resource compliant on this date? | AWS Config history and conformance packs |
| Findings | What did detection services see? | GuardDuty, Security Hub (the unified service and Security Hub CSPM have different names; use the right one) ⚠ VERIFY |
| Application | Which staff member viewed which customer's account, and why? | Your own audit events |
The application layer is the one teams forget. CloudTrail will say a role read a DynamoDB item; it cannot say which human support agent did it on a customer's behalf, or the case reference that justified it. Only your application knows.
What problem the AWS services solve
| Need | Service | Honest limits |
|---|---|---|
| Record API activity | CloudTrail (organisation trail) | Management events are on by default in the event history for 90 days; a trail delivers them to S3 for retention. Data events cost extra and can be high volume |
| Prove logs are unaltered | CloudTrail log-file validation | Validates digest files signed by AWS; proves integrity of what was delivered, not that nothing was never logged |
| Immutable storage | S3 Object Lock in a log-archive account | Compliance mode cannot be shortened; test retention before applying |
| Query history | Athena over CloudTrail logs in S3 | Needs partitioning (or partition projection) to stay affordable |
| Continuous compliance state | AWS Config with conformance packs | Config evaluates configuration, not application behaviour |
| Evidence workflows | Your own evidence catalogue plus S3 | AWS Audit Manager no longer onboards new accounts (from 2026-04-30); build the catalogue yourself ⚠ VERIFY current status |
| SQL over trails inside CloudTrail | CloudTrail Lake | Closed to new customers since 2026-05-31: use CloudTrail to S3 plus Athena |
Two availability notes: do not design a new platform on CloudTrail Lake or Audit Manager. Re-check both statuses before relying on this text ⚠ VERIFY.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Activity Log | CloudTrail management events | Activity Log is per subscription; CloudTrail can be organisation-wide |
| Diagnostic settings to a storage account | Trail to S3, service logs to S3 or CloudWatch Logs | Many services need their own log enablement |
| Microsoft Purview Audit | Application audit events plus CloudTrail | Purview covers Microsoft 365; your own apps need their own audit design |
| Azure Policy compliance state | AWS Config rules and conformance packs | Config keeps a history timeline per resource |
| Immutable blob storage | S3 Object Lock | Governance versus Compliance modes |
| Azure Monitor Log Analytics (KQL) | CloudWatch Logs Insights, Athena | Athena queries S3 in place using SQL |
| Microsoft Defender for Cloud regulatory dashboards | Security Hub standards, Config conformance packs | Standards are checklists of automated controls, not certification |
Architecture
Security-boundary view
The people and roles who can do things in workload accounts cannot delete or alter what the log-archive account holds. That separation is the heart of the design.
The 15-section design
Sections 1 to 3: requirements
| Section | Content |
|---|---|
| 1 Business | Provide a credible, queryable record of who did what to systems and customer data, across all Harbourline AWS accounts, to support audits, investigations and data-subject requests |
| 2 Functional | F1 Record all management events organisation-wide. F2 Record data events for evidence and customer-data stores. F3 Record application-level audit events with a reason or case ID. F4 Validate log integrity. F5 Provide who-accessed-what queries. F6 Alert on attempts to disable logging. F7 Map controls to evidence |
| 3 Non-functional | Completeness: no gaps longer than 15 minutes, detected within 5. Query: a 30-day who-accessed-what query returns in under 60 seconds. Retention: 7 years for evidence logs ⚠ VERIFY; shorter for verbose network logs. Integrity: any record change or deletion detectable. Availability of audit query: 99.5% |
Section 5: service selection
| Choice | Why | Rejected alternative | Trade-off |
|---|---|---|---|
| Organisation trail to a log-archive account | Complete coverage, separate from workload admins | Per-account trails in each account | One more account; strong separation |
| Object Lock (Compliance) on the log bucket | Immutable for the retention period | Versioning only | Cannot undo a mistaken long retention |
| Athena on S3 logs | Cheap, serverless, SQL | CloudTrail Lake | Closed to new customers; Athena needs partitioning discipline |
| Application audit events with a hash chain | Tamper evidence at the application layer | Plain database table | Slightly more code |
| Config conformance packs and Security Hub | Continuous control evidence | Spreadsheets | Setup effort; they check configuration only |
Sections 6 and 7: network and security
- Network. Log delivery uses AWS service principals, not your VPC. Analysts query through Athena over the console or CLI, using VPC endpoints from restricted networks (VPC endpoints).
- Identity. Auditors use read-only SSO roles with MFA; nobody has standing write access to the log bucket. Break-glass roles exist, are alarmed and reviewed.
- Keys. A KMS key in the log-archive account. The key policy allows CloudTrail to encrypt and a small audit role to decrypt; deletion is blocked by SCP and key-deletion waiting periods.
- Guardrails. An SCP denies
cloudtrail:StopLogging,cloudtrail:DeleteTrail,cloudtrail:UpdateTrail(except for a named role) andorganizations:LeaveOrganization. - Data protection. Logs hold identities and IP addresses. Limit read access, set retention per log type, and agree with the data-protection officer how data-subject access requests treat logs ⚠ VERIFY.
- Clock integrity. Use Amazon Time Sync; investigations rely on timestamps.
Sections 8 to 10: API, data and flow
- Audit query API (internal).
GET /audit/accounts/{accountId}/access?from=&to=returning access events with actor, action, outcome and case reference. Requires a privileged role, and the query itself is audited. - Application audit record.
| Field | Example |
|---|---|
seq, prevHash, hash | Chain fields |
at | 2026-11-18T09:05:00Z |
actor | role/support-agent-7 plus staff ID |
action | ViewAccount, ExportStatement |
resource | ACC-0000000001 |
outcome | allowed, denied |
caseRef | CASE-48211 |
- Flow. (1) Service performs an action. (2) It writes an audit record in the same transaction as the action where possible. (3) The record goes to the audit bus. (4) Delivered to the log-archive bucket. (5) Athena queries partitions by date. (6) A periodic job verifies the hash chain and stores its head hash in the evidence bucket.
Section 11: failure scenarios
| Failure | Detection | Automatic response | Impact | Manual step |
|---|---|---|---|---|
| Trail stopped or deleted | Config rule and EventBridge alert on StopLogging | SCP prevents; alert if it happens anyway | Gap in evidence | Restore, document the gap |
| Log delivery fails (bucket policy or KMS error) | CloudTrail delivery error metric | Alarm | Delivery delayed (CloudTrail retries within limits) | Fix permissions |
| Insider edits application audit table | Hash-chain verifier fails | Alert; compare with archived head hashes | Integrity incident | Investigate |
| KMS key disabled or scheduled for deletion | Key-state alarm | SCP and waiting period | Logs unreadable | Cancel deletion |
| Data-event volume explodes cost | Cost anomaly alarm | None | Bill spike | Narrow selectors, sample |
| Logs missing for an investigation period | Gap detection job | Alert | Evidence gap | Reconstruct from other sources, record the gap |
| Wrong retention applied | Change review | None | Over-retention or early expiry | Correct going forward; compliance review |
| Query too slow or costly | Athena workgroup metrics | Scan limit | Delay | Partition, convert to Parquet, narrow date range |
Sections 12 to 15
- Scalability. Log volume grows with accounts and data events. Partition by date and account; convert hot logs to Parquet; set per-log-type retention. Reserve data events for stores that need them.
- Cost. Management events delivered by one trail are free to start; data events, extra trails, S3 storage, Athena scans, KMS requests and Config items bill ⚠ VERIFY prices. The largest surprise is usually S3 data events on busy buckets. Estimate events per day times price, per store.
- Implementation. Phase 1 log-archive account, KMS, bucket with Object Lock. Phase 2 organisation trail with validation. Phase 3 SCP guardrails and alerts. Phase 4 Athena tables and tested queries. Phase 5 application audit events with hash chain. Phase 6 evidence catalogue and an audit rehearsal.
- Compliance touchpoints.
| Area | Where it touches | Reviewing team | Evidence |
|---|---|---|---|
| Logging and monitoring requirements (for example PCI DSS requirement 10) ⚠ VERIFY | Trail scope, retention, review | Security, QSA/assessor | Control mapping, sample queries |
| Records retention | Retention periods per log type | Compliance, legal | Retention schedule |
| Data protection | Personal data in logs, access to logs | Data protection officer | Access review, DPIA entry |
| Segregation of duties | Who can alter logs | Internal control | Role design |
| Audit readiness | Evidence catalogue | Internal audit | Rehearsal outcome |
Step-by-step implementation
Create the log-archive account and bucket
Encrypted with a customer-managed key, Block Public Access on, versioning on, Object Lock enabled at creation. Use Governance mode and a short retention in the sandbox; the production retention is a compliance decision.
Create an organisation trail with validation
Multi-Region, management events on, log-file validation on, encrypted, delivering to the archive bucket. Add data-event selectors only for the stores that need them.
Add guardrails
An SCP denying changes to logging, plus alerts (EventBridge rules) for StopLogging, DeleteTrail, PutBucketPolicy on the log bucket and ScheduleKeyDeletion.
Build and test the queries
Create Athena tables with partition projection and write the standing queries (who accessed an account, who changed a role, root usage). Test them against a known event you create.
Add application audit events with a chain
Write audit records for every access to customer data, including the reason or case reference. Verify the chain on a schedule and archive the head hash.
Write the evidence catalogue and rehearse an audit
Each control lists its evidence source, the query, the owner and the retention. Pick five controls and answer them live in 15 minutes.
AWS Console instructions
- Console → CloudTrail → Trails → Create trail. Name it, choose Create new S3 bucket or select the archive bucket, enable Log file validation, choose KMS encryption, select Multi-Region trail and, in an organisation management account, Enable for all accounts in my organization.
- Event type step: keep Management events on; add Data events for chosen S3 buckets or DynamoDB tables only.
- Console → Athena → Query editor to run saved queries against the CloudTrail table.
- Console → AWS Config → Conformance packs to deploy a pack and view compliance history per resource.
AWS CLI
# Create an encrypted, multi-Region trail with log-file validation.
aws cloudtrail create-trail --name hl-m14-audit-trail \
--s3-bucket-name hl-m14-audit-logs \
--is-multi-region-trail --enable-log-file-validation \
--kms-key-id alias/hl-m14-audit \
--profile learn-sandbox --region us-east-1
aws cloudtrail start-logging --name hl-m14-audit-trail --profile learn-sandbox --region us-east-1
# Record data events for one evidence bucket only (not all buckets).
aws cloudtrail put-event-selectors --trail-name hl-m14-audit-trail \
--advanced-event-selectors '[{"Name":"Evidence bucket data events","FieldSelectors":[{"Field":"eventCategory","Equals":["Data"]},{"Field":"resources.type","Equals":["AWS::S3::Object"]},{"Field":"resources.ARN","StartsWith":["arn:aws:s3:::hl-m14-evidence-staging/"]}]}]' \
--profile learn-sandbox --region us-east-1
# Prove the delivered logs are intact over a time range.
aws cloudtrail validate-logs \
--trail-arn arn:aws:cloudtrail:us-east-1:111122223333:trail/hl-m14-audit-trail \
--start-time 2026-11-17T00:00:00Z --end-time 2026-11-18T00:00:00Z \
--profile learn-sandbox --region us-east-1
# Quick look at recent control-plane events (90-day event history).
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=StopLogging \
--max-results 5 --profile learn-sandbox --region us-east-1Example Athena query for "who read objects in the evidence bucket" (table name and column details depend on how you created the CloudTrail table ⚠ VERIFY):
SELECT eventtime, useridentity.arn AS actor, eventname,
json_extract_scalar(requestparameters, '$.key') AS object_key, sourceipaddress
FROM cloudtrail_logs
WHERE eventsource = 's3.amazonaws.com'
AND eventname IN ('GetObject', 'HeadObject')
AND json_extract_scalar(requestparameters, '$.bucketName') = 'hl-m14-evidence-staging'
AND eventtime >= '2026-11-01T00:00:00Z'
ORDER BY eventtime DESC
LIMIT 100;Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.CloudTrail;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.S3;
using Constructs;
namespace Harbourline.Architecture.Audit;
public class AuditTrailStack : Stack
{
public AuditTrailStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
var key = new Key(this, "AuditKey", new KeyProps
{
Alias = "alias/hl-m14-audit",
EnableKeyRotation = true,
});
var logs = new Bucket(this, "AuditLogs", new BucketProps
{
Encryption = BucketEncryption.KMS,
EncryptionKey = key,
BucketKeyEnabled = true,
BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
EnforceSSL = true,
Versioned = true,
ObjectLockEnabled = true,
// Learning shortcut: Governance + 1 day. Production retention is a compliance decision.
ObjectLockDefaultRetention = ObjectLockRetention.Governance(Duration.Days(1)),
RemovalPolicy = RemovalPolicy.RETAIN,
});
var evidenceBucket = new Bucket(this, "EvidenceStaging", new BucketProps
{
Encryption = BucketEncryption.S3_MANAGED,
BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
EnforceSSL = true,
RemovalPolicy = RemovalPolicy.DESTROY,
AutoDeleteObjects = true,
});
var trail = new Trail(this, "AuditTrail", new TrailProps
{
TrailName = "hl-m14-audit-trail",
Bucket = logs,
EncryptionKey = key,
EnableFileValidation = true,
IsMultiRegionTrail = true,
IncludeGlobalServiceEvents = true,
ManagementEvents = ReadWriteType.ALL,
});
// Data events for one evidence bucket only: broad data-event logging is a cost trap.
trail.AddS3EventSelector(
[new S3EventSelector { Bucket = evidenceBucket }],
new AddEventSelectorOptions { ReadWriteType = ReadWriteType.ALL });
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m14-l15");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "learner");
}
}This is an account-level trail for learning. An organisation trail is created from the management (or delegated administrator) account with IsOrganizationTrail = true.
Complete code
CloudTrail protects AWS API activity. For your own application audit records, a hash chain gives the same tamper evidence: each record includes the hash of the one before it, so editing, reordering or deleting any record breaks the chain from that point on. The program also shows the who-accessed-what query.
// Run with: dotnet run AuditChain.cs
// A hash-chained audit trail: each record commits to the one before it, so editing or
// deleting history is detectable. Also shows a "who accessed what" query.
using System.Security.Cryptography;
using System.Text;
var chain = new List<AuditRecord>();
void Add(string at, string actor, string action, string resource, string outcome) =>
chain.Add(AuditRecord.Next(chain.LastOrDefault(), DateTimeOffset.Parse(at), actor, action, resource, outcome));
Add("2026-11-18T09:00:00Z", "role/support-agent-7", "ViewAccount", "ACC-0000000001", "allowed");
Add("2026-11-18T09:05:00Z", "role/support-agent-7", "ViewAccount", "ACC-0000000002", "allowed");
Add("2026-11-18T09:10:00Z", "role/ops-admin", "FreezeAccount", "ACC-0000000001", "allowed");
Add("2026-11-18T09:12:00Z", "role/support-agent-9", "ViewAccount", "ACC-0000000001", "denied");
Add("2026-11-18T09:20:00Z", "role/support-agent-7", "ExportStatement", "ACC-0000000001", "allowed");
Console.WriteLine("Who touched ACC-0000000001?");
foreach (var r in chain.Where(r => r.Resource == "ACC-0000000001"))
Console.WriteLine($" #{r.Seq} {r.At:HH:mm} {r.Actor} {r.Action} -> {r.Outcome}");
Console.WriteLine("Verify intact chain : " + AuditChain.Verify(chain));
// An insider rewrites a denied attempt as allowed, keeping that record's own hash consistent.
var forged = new List<AuditRecord>(chain);
var original = forged[3];
forged[3] = AuditRecord.Next(forged[2], original.At, original.Actor, original.Action, original.Resource, "allowed");
Console.WriteLine("Verify forged record: " + AuditChain.Verify(forged));
var truncated = chain.Where(r => r.Seq != 2).ToList(); // a record is deleted
Console.WriteLine("Verify deleted row : " + AuditChain.Verify(truncated));
return 0;
public sealed record AuditRecord(long Seq, DateTimeOffset At, string Actor, string Action, string Resource,
string Outcome, string PrevHash, string Hash)
{
public const string GenesisHash = "0000000000000000000000000000000000000000000000000000000000000000";
public static AuditRecord Next(AuditRecord? previous, DateTimeOffset at, string actor, string action,
string resource, string outcome)
{
var seq = (previous?.Seq ?? 0) + 1;
var prevHash = previous?.Hash ?? GenesisHash;
return new AuditRecord(seq, at, actor, action, resource, outcome, prevHash,
Compute(seq, at, actor, action, resource, outcome, prevHash));
}
public static string Compute(long seq, DateTimeOffset at, string actor, string action, string resource,
string outcome, string prevHash)
{
var text = string.Join('|', prevHash, seq, at.ToUniversalTime().ToString("O"), actor, action, resource, outcome);
return Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(text))).ToLowerInvariant();
}
}
public static class AuditChain
{
public static string Verify(IReadOnlyList<AuditRecord> records)
{
var prevHash = AuditRecord.GenesisHash;
long expectedSeq = 1;
foreach (var r in records)
{
if (r.Seq != expectedSeq) return $"BROKEN: sequence gap before #{r.Seq} (expected #{expectedSeq})";
if (r.PrevHash != prevHash) return $"BROKEN: #{r.Seq} does not link to the record before it";
if (AuditRecord.Compute(r.Seq, r.At, r.Actor, r.Action, r.Resource, r.Outcome, r.PrevHash) != r.Hash)
return $"BROKEN: #{r.Seq} content does not match its hash";
prevHash = r.Hash;
expectedSeq++;
}
return $"OK ({records.Count} records, head {prevHash[..12]}...)";
}
}Expected output (the head hash will match this run only if the inputs are identical):
Who touched ACC-0000000001?
#1 09:00 role/support-agent-7 ViewAccount -> allowed
#3 09:10 role/ops-admin FreezeAccount -> allowed
#4 09:12 role/support-agent-9 ViewAccount -> denied
#5 09:20 role/support-agent-7 ExportStatement -> allowed
Verify intact chain : OK (5 records, head 582b3188b16c...)
Verify forged record: BROKEN: #5 does not link to the record before it
Verify deleted row : BROKEN: sequence gap before #3 (expected #2)Code explained line by line
AuditRecordis an immutable record holding the business facts, the previous record's hash (PrevHash) and its ownHash.GenesisHashis a fixed all-zero string that the first record uses as its "previous" value.Nextbuilds the following record: sequence number is previous plus one,PrevHashis the previous record'sHash, andHashis computed over all the fields includingPrevHash. That inclusion is what chains the records.Computejoins fields with|and hashes the UTF-8 bytes with SHA-256. The time is converted to UTC in round-trip format so the same instant always hashes the same.Verifywalks the list and checks three things for each record: the sequence has no gap,PrevHashequals the previous record's hash, and recomputing the hash from the content matches the storedHash.- The forgery replaces record 4 with a new one whose own hash is internally consistent (the attacker recomputed it). The chain still breaks: record 5 stores the old hash of record 4, so it no longer links. Verification fails at #5.
- The deletion leaves a sequence gap, reported at the first record after the hole.
- The
Wherequery at the top is the "who touched this account" answer. In production, this is an Athena or database query over archived audit events, and the run itself is recorded. - One caveat that matters: an attacker who can rewrite every record from the point of change and recompute all later hashes can produce a consistent chain. The defence is to anchor the head hash outside the attacker's reach, for example by writing it to an Object Lock bucket in the log-archive account each hour, or signing it with a KMS key whose use is separately controlled ⚠ VERIFY design with your security team.
Security considerations
⚠ Learning shortcut — not for production
The lab trail lives in the same account as the workload, uses Governance-mode Object Lock for one day and the demo chain is in memory. Production uses a separate log-archive account, a retention set by compliance, SCP guardrails against disabling logs, the chain persisted in a durable store with head hashes anchored off-box, and a tested process for what happens when verification fails.
- Separate the writers from the keepers. Workload administrators must not be able to alter or delete the archive.
- Audit the auditors. Reading logs is itself an auditable act.
- Log denials as well as successes. A burst of
deniedis often the first sign of probing. - Do not log secrets or full personal data. Log identifiers and reasons, not card numbers or statement contents.
- Protect the key. If the KMS key is deleted, the archive is unreadable. Guard it with SCPs and long deletion windows.
- Alert on logging changes.
StopLogging,DeleteTrail, bucket policy changes and key changes must page someone.
Banking use case
A customer submits a data-subject access request and asks who at Harbourline has viewed their account in the last 12 months. The privacy team raises a ticket; an investigator with the audit role runs the standing "access to account" query, which joins application audit events (human actor, case reference) with CloudTrail data events (system access) for the account's items. The result lists 14 views by 3 staff members, each with a case reference. Separately, a quarterly internal-audit sample asks for evidence that nobody disabled logging: the investigator shows the Config compliance history for the trail rule and the CloudTrail digest validation output. Both answers come from a rehearsed evidence catalogue, not from a scramble.
Common mistakes
- Relying on the 90-day event history. It is not an archive. Deliver to S3 with retention.
- Logging everything. Blanket S3 data events on busy buckets create large bills and noise. Select what matters.
- Admins can delete the log bucket. Then the evidence is only as strong as your weakest admin.
- No application-layer audit. You can prove a role read data but not which human or why.
- Untested queries. The first run should not be during the audit.
- Designing around closed services. Use S3 plus Athena instead of CloudTrail Lake, and build your own evidence catalogue instead of depending on Audit Manager ⚠ VERIFY.
- Claiming compliance from tooling. Config rules and Security Hub standards check configuration; they do not certify anything.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
InsufficientS3BucketPolicyException creating a trail | Bucket policy does not allow CloudTrail | Add the two CloudTrail statements with the correct account path |
| Trail exists but no logs in S3 | Logging not started, or KMS key policy blocks CloudTrail | Run start-logging; fix the key policy |
validate-logs reports invalid or missing digests | Time range before validation was enabled, or logs altered/missing | Check trail start date; investigate any gap as an incident |
| Athena query scans huge data and is slow | No partition filter | Use partition projection and filter by date and account |
| Hash-chain verification fails after a restore | Records restored out of order or partially | Restore in sequence order; compare with the anchored head hash |
| Cost spike after enabling data events | Selector covers busy buckets | Narrow to the evidence stores; use sampling where acceptable |
| Cannot delete the log bucket during cleanup | Object Lock retention active | Wait for retention, or use the bypass permission in Governance mode |
Cost and cleanup
The C# chain demo is free. The optional trail and buckets cost little for a few hours: management events on a first trail are free to start; the data-event selector, KMS requests and S3 storage are small at lab volumes ⚠ VERIFY. Object Lock delays deletion by the retention period.
# Stop and delete the trail first.
aws cloudtrail stop-logging --name hl-m14-audit-trail --profile learn-sandbox --region us-east-1
aws cloudtrail delete-trail --name hl-m14-audit-trail --profile learn-sandbox --region us-east-1
# Governance-mode objects need the bypass flag (role needs s3:BypassGovernanceRetention).
aws s3api list-object-versions --bucket hl-m14-audit-logs --profile learn-sandbox --region us-east-1
aws s3api delete-object --bucket hl-m14-audit-logs --key <key> --version-id <id> \
--bypass-governance-retention --profile learn-sandbox --region us-east-1
cdk destroy --profile learn-sandbox
terraform destroy
# Confirm nothing tagged for the course remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1Exercises
- 🟢 Guided: run
AuditChain.cs, then change record 3's actor in memory without recomputing its hash and predict the verification message. - 🟡 Challenge: write the evidence catalogue entry (control, evidence source, query, owner, retention, review frequency) for "no one can disable logging in production" using Config, SCP and CloudTrail evidence.
- 🔴 Live problem: an investigator reports a 40-minute gap in application audit events during an incident. Describe how you would establish whether logs were lost, suppressed or never written, which sources you would cross-check, and how you would record the gap honestly in the incident report.
Assignment
Add architecture/audit-and-evidence/ to your portfolio: the 15-section document, a security-boundary diagram showing the log-archive account, five evidence-catalogue entries with tested Athena queries, and the audit chain with unit tests for forgery, deletion and reorder.
Solutions
Changing the actor on record 3 without recomputing its hash leaves the stored Hash inconsistent with the content, so verification reports BROKEN: #3 content does not match its hash. If the attacker also recomputes record 3's hash, the break moves to record 4, because its PrevHash points at the old value. Why: each record commits to its predecessor, so a local edit must be propagated through every later record to stay hidden.
Control: production trails cannot be disabled. Evidence sources: (1) SCP text denying StopLogging and DeleteTrail; (2) Config rule cloud-trail-enabled compliance history; (3) CloudTrail query for StopLogging events with zero results, or with an approved exception. Query: the Athena statement filtering eventname IN ('StopLogging','DeleteTrail','UpdateTrail'). Owner: security operations. Retention: as for audit evidence ⚠ VERIFY. Review: monthly. Why: an auditor wants a preventive control, a detective control and proof that both worked.
Cross-check: the chain's sequence numbers (a gap proves deletion; no gap suggests nothing was written), the source services' own logs (load balancer, API Gateway, CloudTrail for the same period), queue and bus metrics (were audit events published and delivered?), and deployment records (did a release or outage cover the period?). Record in the incident report what is known, what is not, and the controls that failed (for example, audit write was asynchronous with no outbox). Never reconstruct records and present them as originals.
Knowledge check
Answer: It is short-term and not an immutable archive. Deliver trails to S3 with the retention your policy requires and protect that bucket.
Answer: That delivered log files and digests have not been altered or deleted since delivery. It does not prove that every event was logged in the first place.
Answer: So that administrators of workload accounts cannot delete or modify the evidence, even if they are compromised.
Answer: Each record stores the previous record's hash and a sequence number, so removing a record creates a gap and the next record no longer links.
Answer: CloudTrail delivering to S3, queried with Athena (with partitioning). CloudTrail Lake is closed to new customers ⚠ VERIFY.
Interview questions
- How do you make audit logs trustworthy? Complete coverage, named actors, separate account with Object Lock, log-file validation, hash chains for app records, alerts on tampering attempts.
- How do you answer "who accessed this customer's data"? Application audit events for the human and case, CloudTrail data events for system access, joined by account and time, with the query rehearsed.
- What are the cost traps in audit logging? Data events on busy stores, verbose network logs retained too long, unpartitioned Athena scans.
- What do you do if logging is disabled? Alert, restore, record the gap honestly, and review why the SCP or process failed.
- Can you say a system is compliant because Config shows green? No. Config checks configuration; compliance is assessed by qualified people over controls, processes and evidence.
Certification practice
A bank must ensure that account administrators cannot delete CloudTrail logs. Which design is best?
A. Store logs in the same account with a bucket policy. B. Deliver an organisation trail to a bucket in a separate log-archive account with Object Lock, log-file validation and an SCP preventing trail changes. C. Email logs to the security team weekly. D. Keep logs only in CloudTrail event history.
Answer: B. It separates duties and prevents deletion. A can be changed by local admins, C is not a controlled archive, and D expires after 90 days.
A new design needs SQL queries over years of CloudTrail data. Which approach is recommended given current service availability?
A. Build on CloudTrail Lake. B. Deliver trails to S3 and query with Athena using partition projection. C. Copy events by hand into spreadsheets. D. Use only the console event history.
Answer: B. CloudTrail Lake is closed to new customers ⚠ VERIFY, and S3 plus Athena is cheap and flexible. C and D do not scale or retain.
Which feature proves that CloudTrail log files were not modified after delivery?
A. S3 Transfer Acceleration. B. CloudTrail log-file integrity validation with digest files. C. S3 Intelligent-Tiering. D. VPC Flow Logs.
Answer: B. Digest files signed by CloudTrail allow validation. The others are unrelated to integrity.
Completion checklist
- I can list the layers of audit evidence and name the AWS source for each
- I ran the audit chain and explained the forged and deleted results
- My design uses a separate log-archive account and avoids closed services
- I wrote an evidence-catalogue entry with a tested query
- I ran cleanup and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- Audit evidence must be complete, attributable, tamper-evident and retrievable.
- Keep logs in a separate account with Object Lock, validation, a protected key and SCP guardrails.
- Application audit events (who and why) complement CloudTrail (which role and when).
- Use S3 plus Athena instead of CloudTrail Lake, and your own evidence catalogue instead of Audit Manager.
- Retention, personal data in logs and compliance claims need specialist review.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Event-driven banking architecture
Design domain events, per-domain EventBridge buses, schema governance and safe replay for a bank, keep personal data out of events, and check event contracts and duplicate delivery with a runnable C# validator.
Disaster-recovery architecture for important business services
Map important business services and their impact tolerances to backup-and-restore, pilot-light, warm-standby or active-active designs, plan for cyber recovery and dependencies, and check the mapping with a runnable C# tolerance checker.