Playbook: who accessed this customer record?
Answer a data protection team's access question with evidence: scope the request, pull application audit events, CloudTrail data events and database audit logs, join them by AWS request ID, find the access that bypassed the application, and hand over a hashed, reviewable report.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · Amazon Athena engine v3 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this playbook you can:
- Turn a vague "who looked at this customer?" request into a scoped, authorised evidence question: subject identifier, time window, systems in scope and who may see the answer.
- Explain the four evidence layers (application audit log, CloudTrail data events, database audit, network and identity logs) and what each can and cannot tell you.
- Explain why an AWS role session identifies a service, not a person, and what the application must record so a human can be named.
- Query CloudTrail data events with Athena for one customer's objects and items, and join them to application audit events by AWS request ID.
- Spot the dangerous finding: access that appears in CloudTrail but not in the application audit log.
- Package the result as a hashed report, record that you ran the investigation, and hand it to the data protection team without drawing conclusions you cannot support.
Prerequisites
- CloudTrail for operations, for reading a CloudTrail record and the event types
- CloudTrail for banks and Audit logging and evidence collection, for organisation trails, log integrity and the evidence mindset
- UK GDPR and data residency and Incident response
- Tools: AWS CLI v2 with the
learn-sandboxprofile (read-only use); .NET 10 SDK for the correlation tool, which runs on sample files with no AWS account
Why this matters in banking
On a Monday morning the data protection officer (DPO) forwards a complaint. A Harbourline customer, cust-4471 in our fictional data, says a relative who works at the bank knew details of her recent statement. She asks who has looked at her account in the last six weeks. Nobody knows. The Customer Service tool shows the cases agents opened, but the relative says they never opened a case. The compliance team asks you, the cloud engineer, one question: can you show every access to this customer's data, by whom, and why?
Three things go wrong in teams that have not prepared:
- The answer is "we do not know", because object-level logging was never switched on, or the logs expired after 30 days.
- The answer names the wrong thing. CloudTrail says
hb-statements-taskread the file 40 times, which is the application's role. It does not say which employee clicked the button. - The investigation itself leaks. Someone runs ad hoc queries on production, pastes a customer ID into a chat, and now the evidence has been handled informally and the person under suspicion hears about it before HR does.
This playbook is about the method and the evidence design. It is not legal advice. What counts as an access log the customer is entitled to see, what must be disclosed, to whom and when are decisions for your DPO and legal team (UK GDPR is the reference flavour). Your job is to produce accurate, complete, clearly bounded evidence, and to say honestly where it has gaps.
The idea in plain language
"Who accessed this record?" is really four questions stacked on top of each other:
| Question | Where the answer lives |
|---|---|
| Which person asked the system to show the data, and for what purpose? | The application audit log: end-user identity, action, purpose or case reference, outcome |
| Which AWS identity actually read the data? | CloudTrail data events: the role session, source IP, object key or table key, request ID |
| Did anyone read the data without going through the application? | CloudTrail data events whose role is a human or support role, plus database audit logs |
| Could anyone change or delete the evidence? | Log integrity validation, Object Lock, separate log-archive account |
Analogy: The bank's branch has a customer-records room. The door badge reader (CloudTrail) records which badge opened the door, but the clerk at the counter (the application) is the one who knows which customer asked and why. If you only have badge logs, you know a clerk's badge was used. If you only have the clerk's notebook, you cannot prove nobody slipped in through the back door. You need both books, and you need to check that the entries in one match the other.
The "last hop" problem
CloudTrail identifies the IAM principal that called the AWS API. In a typical Harbourline service, every request to S3 or DynamoDB is made by the ECS task role, whatever employee is logged into the web interface. So for thousands of reads, userIdentity is the same role session. The human is invisible to AWS.
There are two ways to bring the human back:
- Record the end-user identity in the application audit log and join it to CloudTrail with a shared key. This is the main method and the one you must design before you need it.
- Use identity-propagating or per-user credentials for the few systems where access by an individual must be visible directly in AWS logs (for example, support staff reading a bucket through IAM Identity Center permission sets, where the role session name carries the user's name). This is the exception, not the default.
The shared key can be as simple as the AWS request ID. Every AWS API response carries one, the AWS SDK for .NET exposes it, and CloudTrail records it as requestID. If the application writes that ID into its audit event, the join is exact. ⚠ VERIFY for each service you rely on that the request ID visible in the SDK response matches the requestID field in the CloudTrail data event; test it once in a sandbox and record the result in your runbook. When that is not possible, you fall back to a time window plus role plus object key, which is weaker, and your report must say so.
What problem the AWS services solve
AWS gives you the raw material, but most of it is off by default or short-lived:
| Source | What it shows | Default | Sharp edges |
|---|---|---|---|
| CloudTrail management events | Who created, changed or deleted resources, who assumed which role | On; 90 days in event history | Does not show reads of S3 objects or DynamoDB items |
| CloudTrail data events | S3 object-level calls (GetObject, PutObject, DeleteObject), DynamoDB item-level calls, Lambda invokes and more | Off | Priced per 100,000 events ⚠ VERIFY current rate; high-volume buckets get expensive; you must select which resources to log |
| Application audit events | End-user, action, subject, purpose, outcome | Only if you built them | Must be tamper-evident and must not contain raw personal data |
| Database audit | Queries and logins: pgaudit for PostgreSQL, SQL Server Audit, Aurora Database Activity Streams | Off | Not in CloudTrail; each engine has its own switch and overhead ⚠ VERIFY engine and version support |
| S3 server access logs | Best-effort request records for a bucket | Off | Delivered with delay and on a best-effort basis, so not complete evidence on their own |
| VPC Flow Logs, ALB access logs | Connections and HTTP requests | Off | Show that something talked to something, not which record |
| IAM Identity Center / sign-in events | Who signed in to the console or CLI with which permission set | On in CloudTrail management events | Shows sign-in, not what they then read |
CloudTrail Lake is not an option for new accounts — verified 2026-11-17
AWS CloudTrail Lake closed to new customers on 2026-05-31, so this playbook uses a trail delivering to S3, queried with Amazon Athena, plus an optional copy into CloudWatch Logs. If your bank already uses CloudTrail Lake, the same questions work there with SQL, but do not design a new capability around it. AWS Systems Manager Incident Manager is also closed to new customers; use your bank's own incident tooling.
Honest limits to tell the DPO up front:
- CloudTrail delivers log files to S3 typically within minutes, not in real time ⚠ VERIFY the current typical delay.
- You can only show what was logged while logging was on. If data events were enabled on 1 October, you cannot reconstruct September from CloudTrail.
- Database content changes, exports through an engine-specific tool and anything outside AWS (laptops, email) are outside this evidence unless you have separate logs.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Activity Log | CloudTrail management events | Both record control-plane calls. CloudTrail also records role assumptions and console sign-ins |
| Azure Storage analytics logs, Azure Monitor resource logs for Storage | CloudTrail S3 data events (and S3 server access logs) | Data events are chosen per bucket or prefix and billed per event |
| Microsoft Purview audit, SQL auditing to Log Analytics | Application audit table, pgaudit or SQL Server Audit, Aurora Database Activity Streams | Auditing the database is an engine feature you switch on, not a platform default |
| Entra ID sign-in logs plus Conditional Access | IAM Identity Center events in CloudTrail, plus your IdP's own logs | The identity provider (for example Entra ID) holds the person; CloudTrail holds the role session |
| Log Analytics workspace with KQL | Athena (SQL) over S3, or CloudWatch Logs Insights | Athena charges per data scanned, so partition by date and project columns |
| Azure Policy audit effect | AWS Config rules and Security Hub controls | Shows configuration state, not who read data |
| Immutable blob storage for logs | S3 Object Lock in a log-archive account | Compliance mode cannot be shortened, even by the account root user |
Architecture
The four evidence layers
Arrows 1 to 4 are the normal path: the person is known to the application and the AWS read is made by the task role. Arrow 6 is the case you are hunting for: a human with their own AWS access reading data directly. It leaves no application audit event, so it shows up in CloudTrail with a person's name in the role session and nothing on the other side to match.
Joining the two books
Step-by-step implementation
Minutes 0 to 10: scope, authority and containment of the investigation
Before you query anything:
- Who is asking, and are they entitled to ask? The request should come from the DPO, compliance or HR through a ticket (a fictional
CASE-9930here), not a chat message. A manager asking "just check on this person" is a red flag. Write down who asked, when and why. - Define the subject. Use the pseudonymous internal ID (
cust-4471), never the customer's name or account number, in queries, tickets and chats. - Define the window. Start and end in UTC. Add a buffer on each side for clock skew and delivery delay.
- List systems in scope. Application, S3 buckets, DynamoDB tables, RDS databases, support tooling, exports to analytics.
- Limit who sees the result. Agree who gets the findings. If the concern is that an employee misused access, the employee must not be warned by a broad distribution list.
- Preserve first. Check retention. If any source is close to expiry, copy it (to the evidence bucket) before you analyse it. Do not change anything in production.
Do not draw conclusions. At this stage you are collecting facts for a person with the authority to decide what they mean.
Minutes 10 to 30: check that the evidence exists
Ask what you will be able to show, before the clock runs:
- Is there a trail with data events for the statements bucket and the customers table, and since when? (
aws cloudtrail get-event-selectors.) - Is the trail delivering, and is log file validation on? (
aws cloudtrail get-trail-status.) - What is the retention of the application audit log and of the trail's S3 bucket?
- Does the application audit event contain the end-user ID, subject, purpose, outcome and AWS request IDs?
Write the answers into the evidence record. A gap is a finding, not an embarrassment to hide: "data events for the statements bucket were enabled from 2026-10-01; reads before that date cannot be shown from CloudTrail."
Minutes 30 to 60: pull the application and AWS records
- Application audit events for the subject in the window: extract to a file, with an export query that is itself logged.
- CloudTrail data events for the subject's object prefix and DynamoDB key: query with Athena (see the CLI section) and export the result.
- Management events for the same window: anyone who changed bucket policies, trails, event selectors or IAM roles that grant access to this data. Disabling or narrowing a trail during the period is its own finding.
- Database audit where the subject's data also lives in RDS: query the
pgauditor SQL Server audit stream for the subject's key.
Minutes 60 to 80: correlate and classify
Run the correlation tool below. Every access ends up in one of three groups:
| Group | Meaning | What to do |
|---|---|---|
| Matched | Application event and CloudTrail event agree (by request ID, or weakly by time) | Check each has a plausible purpose or case reference |
| Trail-only | CloudTrail shows a read of the subject's data and the application has no record | The important group. A support engineer, a script, a leaked key, or an application path that forgot to log. Escalate per the incident process |
| App-only | The application says it served the data but no AWS read is visible | Served from cache, a data event gap, or logging enabled later than the access. Explain, do not ignore |
A denied application event (outcome denied) is evidence too: someone tried and was refused.
Minutes 80 to 90: report, hash, record
Write the report as facts only: what was searched, which sources, what period, what was found, what could not be determined and why. Hash it, store it in the evidence bucket with Object Lock, and record in the case: who ran which queries, when, from which role. Hand it to the DPO. Leave interpretation, disclosure to the customer and any HR or regulatory step to them.
Likely causes of an incomplete answer, ranked
When the evidence does not answer the question, work down this list:
- Data events were not enabled for that bucket or table during the window.
- The application logs the role but not the end user (or logs the end user's display name rather than a stable ID).
- Logs expired before the request arrived (short retention).
- The access was through a path the application audit does not cover (batch job, export, admin tool, support console).
- The object keys do not contain a subject ID, so you cannot find a customer's objects in CloudTrail without joining to a lookup table.
- Time zones and clock skew made a time-window join miss a pair.
- The trail had a delivery gap (check the digest files and
get-trail-status).
AWS Console instructions
Check that data events are on: Console → CloudTrail → Trails → choose the organisation or account trail → Data events section. Look for an S3 entry covering the statements bucket and a DynamoDB entry for the customers table. If the page shows "Advanced event selectors" with eventCategory = Data, expand them to read the exact resource filters.
Check delivery and integrity: the same trail page shows Log file validation (Enabled), the destination bucket and the Latest delivery time.
Look at management events for the last 90 days: Console → CloudTrail → Event history. Filter by event name (for example PutEventSelectors, StopLogging, PutBucketPolicy) to see if anyone changed the logging itself. Event history shows management events only, not data events.
Run the data-event query: Console → Athena → choose the workgroup hb-security-investigations → Query editor. The workgroup should enforce an output location in the evidence bucket and a per-query data-scanned limit.
AWS CLI
# 1) Which events does the trail record? Look for Data events on the statements bucket
# and the customers table. Advanced selectors are the current format.
aws cloudtrail get-event-selectors --trail-name hb-org-trail --profile learn-sandbox
# 2) Is the trail delivering and is anything stuck? LatestDeliveryError should be absent.
aws cloudtrail get-trail-status --name hb-org-trail --profile learn-sandbox
# 3) Did anyone touch the logging configuration in the window? (management events, 90 days)
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=PutEventSelectors \
--start-time 2026-10-01T00:00:00Z --end-time 2026-11-18T00:00:00Z \
--profile learn-sandbox
# 4) Start an Athena query in the locked-down investigations workgroup.
# Keep the SQL in a file so the exact query becomes part of the evidence.
aws athena start-query-execution \
--work-group hb-security-investigations \
--query-string file://who-accessed.sql \
--query-execution-context Database=security_logs \
--profile learn-sandbox
# 5) Check the query state, then fetch results (QueryExecutionId comes from step 4).
aws athena get-query-execution --query-execution-id 11111111-2222-3333-4444-555555555555 \
--profile learn-sandbox
aws athena get-query-results --query-execution-id 11111111-2222-3333-4444-555555555555 \
--profile learn-sandbox
# 6) Confirm log file integrity for the period (digest validation).
aws cloudtrail validate-logs \
--trail-arn arn:aws:cloudtrail:us-east-1:111122223333:trail/hb-org-trail \
--start-time 2026-10-01T00:00:00Z --end-time 2026-11-18T00:00:00Z \
--profile learn-sandboxThe SQL for step 4 (who-accessed.sql). It assumes a CloudTrail table named cloudtrail_logs with partition projection on date; the Athena table definition for CloudTrail is in the audit evidence lesson. ⚠ VERIFY column names against your own table definition before relying on the query.
-- Every data event that touched cust-4471's objects or items, 1 Oct to 18 Nov 2026 (UTC).
-- The partition filter keeps the scan small, and therefore cheap.
SELECT
eventtime,
eventname,
useridentity.arn AS principal,
useridentity.sessioncontext.sessionissuer.username AS role_name,
sourceipaddress,
requestid,
json_extract_scalar(requestparameters, '$.bucketName') AS bucket,
json_extract_scalar(requestparameters, '$.key') AS object_key,
errorcode
FROM cloudtrail_logs
WHERE date >= '2026/10/01' AND date <= '2026/11/18'
AND eventsource IN ('s3.amazonaws.com', 'dynamodb.amazonaws.com')
AND ( json_extract_scalar(requestparameters, '$.key') LIKE 'statements/cust-4471/%'
OR requestparameters LIKE '%"cust-4471"%')
ORDER BY eventtime;Infrastructure as Code
The trail needs data events for specific resources. Select the statements bucket prefix and the customers table, not "all S3 data events".
using Amazon.CDK;
using Amazon.CDK.AWS.CloudTrail;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.S3;
using Constructs;
namespace Harbourline.Playbooks;
public class CustomerAccessLoggingStack : Stack
{
public CustomerAccessLoggingStack(Construct scope, string id, IBucket statements,
ITable customers, IBucket logArchive, IStackProps? props = null) : base(scope, id, props)
{
var trail = new Trail(this, "CustomerDataTrail", new TrailProps
{
Bucket = logArchive, // Object Lock bucket in the log-archive account
EnableFileValidation = true,
IncludeGlobalServiceEvents = false, // management events are covered by the org trail
ManagementEvents = ReadWriteType.NONE // this trail is only for customer-data access
});
// S3 object-level events for the statements prefix only, reads and writes.
trail.AddS3EventSelector(new[]
{
new S3EventSelector { Bucket = statements, ObjectPrefix = "statements/" }
}, new AddEventSelectorOptions { ReadWriteType = ReadWriteType.ALL });
// DynamoDB item-level events for the customers table.
trail.AddEventSelector(DataResourceType.DYNAMODB_TABLE,
new[] { customers.TableArn },
new AddEventSelectorOptions { ReadWriteType = ReadWriteType.ALL });
Amazon.CDK.Tags.Of(this).Add("Project", "Harbourline");
Amazon.CDK.Tags.Of(this).Add("DataClassification", "evidence");
}
}Check the property names against your versions
The CDK Trail helpers and the Terraform advanced_event_selector block exist in current releases, but construct and argument names move between versions. ⚠ VERIFY both snippets against the CDK and Terraform provider versions you pin, and run terraform validate or dotnet build before applying.
Complete code
The correlation tool reads the application audit log (JSON lines) and a CloudTrail log file, filters both to one subject, joins them, and writes a hashed report. It has no AWS dependency, so you can run it on exported files and on the sample data below. The exit code is 1 when there are trail-only accesses, so a pipeline or a runbook step can stop and escalate.
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>
</Project>using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
// Usage: dotnet run -- <subject-id> <app-audit.jsonl> <cloudtrail-export.json> <report.json>
if (args.Length != 4)
{
Console.Error.WriteLine("usage: AccessTrail <subject-id> <app-audit.jsonl> <cloudtrail-export.json> <report.json>");
return 2;
}
string subject = args[0];
var appEvents = AccessTrailBuilder.ReadAppEvents(File.ReadLines(args[1]), subject);
var trailEvents = AccessTrailBuilder.ReadTrailEvents(File.ReadAllText(args[2]), subject);
var report = AccessTrailBuilder.Build(subject, appEvents, trailEvents, TimeSpan.FromSeconds(5));
// Hash the exact bytes we write, so the hash file matches the report file byte for byte.
var bytes = Encoding.UTF8.GetBytes(JsonSerializer.Serialize(report, new JsonSerializerOptions { WriteIndented = true }));
File.WriteAllBytes(args[3], bytes);
string hash = Convert.ToHexString(SHA256.HashData(bytes)).ToLowerInvariant();
File.WriteAllText(args[3] + ".sha256", hash + " " + Path.GetFileName(args[3]) + "\n");
Console.WriteLine($"Subject {subject}: {report.Matched.Count} matched, " +
$"{report.TrailOnly.Count} trail-only (bypassed the app?), " +
$"{report.AppOnly.Count} app-only (no AWS call seen).");
Console.WriteLine($"SHA-256 {hash}");
return report.TrailOnly.Count > 0 ? 1 : 0;
public sealed record AppEvent(
DateTimeOffset Timestamp, string CorrelationId, string Actor, string Action,
string Subject, string? Purpose, string Outcome, IReadOnlyList<string> AwsRequestIds);
public sealed record TrailEvent(
DateTimeOffset Time, string EventName, string Principal, string? SourceIp,
string? RequestId, string? Resource);
public sealed record MatchedAccess(AppEvent App, TrailEvent Trail, string Basis);
public sealed record AccessReport(
string Subject,
IReadOnlyList<MatchedAccess> Matched,
IReadOnlyList<TrailEvent> TrailOnly,
IReadOnlyList<AppEvent> AppOnly);
public static class AccessTrailBuilder
{
public static List<AppEvent> ReadAppEvents(IEnumerable<string> lines, string subject)
{
var result = new List<AppEvent>();
foreach (var line in lines)
{
if (string.IsNullOrWhiteSpace(line)) continue;
using var doc = JsonDocument.Parse(line);
var r = doc.RootElement;
if (!string.Equals(Str(r, "subject"), subject, StringComparison.Ordinal)) continue;
var ids = new List<string>();
if (r.TryGetProperty("awsRequestIds", out var arr) && arr.ValueKind == JsonValueKind.Array)
foreach (var id in arr.EnumerateArray())
if (id.GetString() is { Length: > 0 } s) ids.Add(s);
result.Add(new AppEvent(
DateTimeOffset.Parse(Str(r, "ts")!), Str(r, "correlationId") ?? "", Str(r, "actor") ?? "unknown",
Str(r, "action") ?? "", subject, Str(r, "purpose"), Str(r, "outcome") ?? "", ids));
}
return result.OrderBy(e => e.Timestamp).ToList();
}
// Reads a CloudTrail log file ({"Records":[...]}) and keeps data events whose
// resource (bucket/key or table/key) mentions the subject's pseudonymous id.
public static List<TrailEvent> ReadTrailEvents(string json, string subject)
{
var result = new List<TrailEvent>();
using var doc = JsonDocument.Parse(json);
if (!doc.RootElement.TryGetProperty("Records", out var records)) return result;
foreach (var rec in records.EnumerateArray())
{
string? resource = DescribeResource(rec);
if (resource is null || !resource.Contains(subject, StringComparison.Ordinal)) continue;
string principal = "unknown";
if (rec.TryGetProperty("userIdentity", out var ui))
principal = Str(ui, "arn") ?? Str(ui, "principalId") ?? "unknown";
result.Add(new TrailEvent(
DateTimeOffset.Parse(Str(rec, "eventTime")!), Str(rec, "eventName") ?? "",
principal, Str(rec, "sourceIPAddress"), Str(rec, "requestID"), resource));
}
return result.OrderBy(e => e.Time).ToList();
}
public static AccessReport Build(string subject, IReadOnlyList<AppEvent> app,
IReadOnlyList<TrailEvent> trail, TimeSpan window)
{
var matched = new List<MatchedAccess>();
var usedTrail = new HashSet<TrailEvent>();
var unmatchedApp = new List<AppEvent>();
foreach (var a in app)
{
// Strong join: the application recorded the AWS request id it received.
var byId = trail.FirstOrDefault(t => t.RequestId is not null
&& a.AwsRequestIds.Contains(t.RequestId) && !usedTrail.Contains(t));
if (byId is not null)
{
usedTrail.Add(byId);
matched.Add(new MatchedAccess(a, byId, "request-id"));
continue;
}
// Weak join: same moment (within the window). Say so in the report.
var byTime = trail.FirstOrDefault(t => !usedTrail.Contains(t)
&& (t.Time - a.Timestamp).Duration() <= window);
if (byTime is not null)
{
usedTrail.Add(byTime);
matched.Add(new MatchedAccess(a, byTime, "time-window"));
continue;
}
unmatchedApp.Add(a);
}
var trailOnly = trail.Where(t => !usedTrail.Contains(t)).ToList();
return new AccessReport(subject, matched, trailOnly, unmatchedApp);
}
private static string? DescribeResource(JsonElement rec)
{
if (!rec.TryGetProperty("requestParameters", out var p) || p.ValueKind != JsonValueKind.Object)
return null;
string? bucket = Str(p, "bucketName");
string? key = Str(p, "key");
if (bucket is not null && key is not null) return $"s3://{bucket}/{key}";
string? table = Str(p, "tableName");
if (table is not null && p.TryGetProperty("key", out var k)) return $"dynamodb:{table}/{k.GetRawText()}";
return null;
}
private static string? Str(JsonElement e, string name) =>
e.ValueKind == JsonValueKind.Object && e.TryGetProperty(name, out var v) && v.ValueKind == JsonValueKind.String
? v.GetString() : null;
}Sample inputs (all fictional). Save them next to the project:
{"ts":"2026-11-03T09:14:22Z","correlationId":"c-001","actor":"emp-7781","action":"ViewStatement","subject":"cust-4471","purpose":"CASE-9921","outcome":"allowed","awsRequestIds":["REQ-A1"]}
{"ts":"2026-11-03T09:20:05Z","correlationId":"c-002","actor":"emp-7781","action":"ViewProfile","subject":"cust-4471","purpose":"CASE-9921","outcome":"allowed","awsRequestIds":[]}
{"ts":"2026-11-04T16:02:41Z","correlationId":"c-003","actor":"emp-1203","action":"ViewStatement","subject":"cust-9999","purpose":"CASE-0001","outcome":"allowed","awsRequestIds":["REQ-Z9"]}
{"ts":"2026-11-05T11:45:00Z","correlationId":"c-004","actor":"emp-3310","action":"ViewStatement","subject":"cust-4471","purpose":null,"outcome":"denied","awsRequestIds":[]}{"Records":[
{"eventTime":"2026-11-03T09:14:22Z","eventName":"GetObject","sourceIPAddress":"10.0.12.34","requestID":"REQ-A1","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/hb-statements-task/0f3c9d2a"},"requestParameters":{"bucketName":"hb-statements-111122223333","key":"statements/cust-4471/2026-10.pdf"}},
{"eventTime":"2026-11-03T09:20:06Z","eventName":"GetItem","sourceIPAddress":"10.0.12.34","requestID":"REQ-B2","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/hb-statements-task/0f3c9d2a"},"requestParameters":{"tableName":"hb-customers","key":{"pk":{"S":"cust-4471"}}}},
{"eventTime":"2026-11-07T22:31:09Z","eventName":"GetObject","sourceIPAddress":"203.0.113.50","requestID":"REQ-C3","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/hb-support-readonly/alex.example"},"requestParameters":{"bucketName":"hb-statements-111122223333","key":"statements/cust-4471/2026-09.pdf"}},
{"eventTime":"2026-11-07T22:31:10Z","eventName":"ListBuckets","userIdentity":{"arn":"x"},"requestParameters":null}
]}Run it:
dotnet run -- cust-4471 app-audit.jsonl cloudtrail-export.json report.jsonExpected output (the hash will match when you run the exact files above, byte for byte):
Subject cust-4471: 2 matched, 1 trail-only (bypassed the app?), 1 app-only (no AWS call seen).
SHA-256 <64 hex characters>Reading the result: the first access matched exactly by request ID; the second matched only by time, so the report marks it time-window (a weaker claim); the 7 November read by hb-support-readonly at 22:31 from a non-VPC address has no application record and is the trail-only finding; and the denied attempt on 5 November is application-only, because a denied request never reached S3.
Code explained line by line
- Argument check. The tool takes the subject, the two input files and an output path. A wrong count returns exit code 2 so a script does not treat "no arguments" as "no findings".
ReadAppEvents. Reads JSON lines, keeps only events for the subject and keeps the AWS request IDs the application recorded. UsingJsonDocumentmeans one bad line throws a clear parse error instead of silently dropping data.ReadTrailEvents. CloudTrail files wrap events in{"Records":[...]}.DescribeResourceturns S3 (bucketNamepluskey) and DynamoDB (tableNamepluskey) parameters into a readable resource string and drops events that are not data events (such asListBuckets).- Subject filter. A CloudTrail event is kept only if its resource string contains the pseudonymous subject ID. This is why object keys that include
cust-4471are useful, and why you must choose a pseudonymous ID rather than an account number. Buildstrong join. For each application event, look for a CloudTrail event with one of the AWS request IDs the application recorded.usedTrailmakes sure one CloudTrail event is not matched twice.Buildweak join. If there is no request-ID match, accept a CloudTrail event within a five-second window, and recordtime-windowas the basis. A weak join is an inference; say so.- Leftovers become the findings. CloudTrail events nobody claimed are
TrailOnly; application events nothing matched areAppOnly. - Hashing. The report is written as bytes and the SHA-256 of the same bytes goes into a
.sha256file, so the reviewer can verify the report was not edited after generation. Hashing gives tamper evidence; to prevent change, store the files in an Object Lock bucket. - Exit code.
1means "there is something to escalate". Pipelines and runbook scripts can branch on it.
Security considerations
⚠ Learning shortcut — not for production
The sample reads local files and uses a five-second time window for the weak join. In production, run the query and the correlation in a locked-down investigation account with a read-only role, store results in an Object Lock bucket, and treat every investigation as a ticketed, logged activity. A fixed five-second window is also too crude for busy systems; prefer request IDs, and tune the window to your measured clock skew and processing delay.
- The evidence is personal data. A report that lists employee identities and a customer's activity is itself sensitive. Restrict access, encrypt with a dedicated KMS key, and set a retention period agreed with the DPO.
- No raw PII in logs. Use pseudonymous IDs in audit events and object keys. If keys carry account numbers or names, the logs leak what they are meant to protect.
- Separate duties. The people who run the service should not control the log-archive account or be able to disable the trail. Alert on
StopLogging,PutEventSelectors,DeleteTrailand bucket policy changes. - Query access is access. Give investigators a role that can run queries in one workgroup and read the evidence bucket, and nothing else. Log their Athena queries (they are in CloudTrail).
- Sensitive subject. If the person of interest is an employee, involve HR and legal before anyone broadens access to the report.
- Never claim compliance. The report supports your bank's answer; whether it satisfies a regulator, a subject access request or an employment process is for your legal and compliance functions.
Banking use case
Harbourline's DPO opens CASE-9930 for cust-4471. The cloud team confirms data events on the statements bucket and customers table have been on since 1 October, runs the Athena query and the correlation tool, and finds the pattern shown in the sample: two normal accesses for a documented case, one denied attempt, and one read of a September statement at 22:31 by a support role session named after an individual from an address outside the bank's VPN range. The application never saw it.
The team does not conclude misconduct. They report: the access exists, who the role session was issued to (from the Identity Center sign-in record in CloudTrail, in the same window), the object read, the source IP, and that the read bypassed the application audit trail. The DPO and HR decide what happens next. In parallel the engineering team acts on the control gap: support staff should not hold a standing role with direct read access to customer statements, and any break-glass access should require a ticket reference and an alert. The follow-up ticket also adds the AWS request ID to every application audit event so the next investigation is exact.
Common mistakes
- Starting the investigation before authority is confirmed. Why: urgency and curiosity. Avoid: require a ticket from the DPO, compliance or HR before the first query.
- Assuming CloudTrail shows who read the data. Why: it shows the role. Avoid: record the end user in the application audit log and join by request ID.
- Assuming data events are on. Why: management events are on by default and look similar. Avoid: check
get-event-selectorsfirst and record when logging began. - Searching by name or account number. Why: that is what the complaint contains. Avoid: translate once to a pseudonymous ID, and keep the lookup restricted.
- Treating a missing match as "no access". Why: absence of a log is not absence of an action. Avoid: state the coverage (period, systems) explicitly in the report.
- Editing the evidence. Why: tidying a spreadsheet. Avoid: work on copies, hash the originals and the report, store in Object Lock.
- Reporting conclusions. Why: wanting to be helpful. Avoid: report facts and gaps; let the DPO and HR interpret.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Athena returns no rows for a customer who was definitely viewed | Data events were not enabled for the bucket or table, or the trail's S3 prefix or partition projection range is wrong | Check get-event-selectors; check the table's date range; widen the partition filter |
| Athena query scans many GB and costs more than expected | No partition filter, SELECT *, large JSON column read | Filter on the partition column, select only the columns you need, set a workgroup data-scanned limit |
HIVE_BAD_DATA or JSON parse errors in Athena | A row with an unexpected shape, or the wrong SerDe in the table definition | Use the CloudTrail table definition from the audit lesson; ⚠ VERIFY SerDe and input format for your engine version |
| Every row shows the same role and no person | Normal for service-mediated access | Join to the application audit log by request ID |
| Request IDs never match | The application did not record them, or the field is a different ID for that service | Test one call in the sandbox, compare the SDK response ID with the CloudTrail requestID; fall back to the time window and say so |
| Events appear hours late | Delivery delay, or a large file still being written | Re-run after the delay; check get-trail-status for delivery errors |
validate-logs reports invalid or missing digests | Log tampering, delivery gap, or a trail stopped during the window | Escalate as a security incident; check StopLogging events |
| App-only events with allowed outcome but no AWS read | Cache hit, replica read path, or logging started later | Explain the read path in the report; check the cache and its own audit |
Cost and cleanup
Cost: free for the sample. The correlation tool runs locally on sample files. Nothing needs an AWS account.
If you reproduce this in the sandbox: CloudTrail data events are priced per 100,000 events (about $0.10 at the last check) ⚠ VERIFY, so a few thousand test calls cost cents. Athena charges per data scanned (a few cents for sample logs) and S3 charges for the stored logs. Delete anything you create the same day.
# Delete the demo trail and stop its data events (only if you created it for this exercise).
aws cloudtrail delete-trail --name hb-customer-data-trail --profile learn-sandbox
# Empty and remove the demo log bucket (never do this to a real log-archive bucket).
aws s3 rm s3://hb-demo-trail-logs-111122223333 --recursive --profile learn-sandbox
aws s3 rb s3://hb-demo-trail-logs-111122223333 --profile learn-sandbox
# Remove local sample outputs.
rm -f report.json report.json.sha256Confirm nothing still bills: Console → CloudTrail → Trails (none left from this exercise) and Billing → Cost Explorer by service tomorrow.
Exercises
- 🟢 Guided: Run the correlation tool on the sample files. Explain, for each of the four events, which group it landed in and why. Change one application event so its request ID is
REQ-A1but its timestamp is off by an hour, and confirm it still matches by request ID. - 🟡 Challenge: Extend the tool to read the application events'
purposeand flag any allowed event with no purpose. Add a "denied attempts" section to the report that lists repeated denials by the same actor. - 🔴 Live problem: The DPO says: "A customer's relative may have looked at her account over the last three months. We enabled data events two weeks ago." Draft the message you send back: what you can show, what you cannot, what you will do now to close the gap, and who needs to be in the loop. Do not speculate about the relative.
Assignment
Write a one-page "customer access investigation" runbook for your portfolio: request intake and authority, scope definition, the four evidence layers with the exact commands for your stack, the correlation method, a report template, the retention and Object Lock settings, and a list of gaps your design would need to close. Keep all data fictional.
Solutions
Event 1 (GetObject, REQ-A1) is matched by request ID: the application recorded REQ-A1. Event 2 (GetItem, REQ-B2) is matched by time window (the application event at 09:20:05 and the AWS event at 09:20:06, and the application did not record a request ID), so the report labels it as a weaker inference. Event 3 (the 22:31 GetObject by hb-support-readonly) has no application event, so it is trail-only and the tool exits with code 1. Event 4 (the denied attempt on 5 November) is app-only: the application refused it before calling AWS, so there is nothing in CloudTrail. If you shift the first application event's timestamp by an hour but keep REQ-A1, it still matches, because the request-ID join does not depend on clocks. That is why you design for request IDs.
Add a string? Purpose check when building the report: after Build, produce app.Where(a => a.Outcome == "allowed" && string.IsNullOrWhiteSpace(a.Purpose)) as a MissingPurpose list. For denied attempts, group app.Where(a => a.Outcome == "denied") by Actor and keep groups with a count of 2 or more. Both lists belong in the hashed report. The reason: a legitimate access has a business purpose, and repeated denials show someone probing for access they do not have. Neither proves wrongdoing, so the report states the pattern and leaves judgement to the DPO and HR.
Knowledge check
Answer: The ECS task role makes every AWS call on behalf of all users, so CloudTrail identifies the service, not the human. The application audit log must record the end-user identity and ideally the AWS request ID so the two logs can be joined. Mentioning "enable more CloudTrail" is wrong: more CloudTrail does not add the person.
Answer: Data events, which are off by default. Management events (on by default, 90 days in event history) record control-plane calls such as creating buckets or assuming roles, not object reads.
Answer: Record it as a trail-only finding, preserve the evidence, check who the role session was issued to (Identity Center sign-in events), and escalate through the incident or data protection process. Do not conclude misconduct and do not contact the individual yourself. The finding shows access bypassed the application; interpretation belongs to the DPO and HR.
Answer: Keys and logs are widely readable by operations staff and tools. If they contain names or account numbers, the log becomes a second copy of personal data. A pseudonymous ID lets you search for one customer's data without exposing who they are, with the lookup kept in a restricted place.
Answer: Hash the exact bytes (SHA-256), store the hash with the report, and keep both in an S3 bucket with Object Lock so they cannot be replaced. For the source logs, enable CloudTrail log file validation and run validate-logs. A hash alone shows tampering only if the hash itself is protected.
Interview questions
- How would you find out who accessed a customer's data in AWS? Check the evidence exists first (data events, retention), then combine application audit events (the person and purpose) with CloudTrail data events (the AWS reads), join by request ID, and list accesses that exist in one source and not the other.
- Why isn't CloudTrail enough on its own? It identifies the IAM principal, which is usually a service role. It does not know the human, the purpose, or reads that never reached AWS APIs, such as cached data.
- What would you change so the next investigation is easier? Add the end-user ID and AWS request ID to every audit event, enable data events for sensitive resources, use pseudonymous IDs in keys, set retention that matches policy and alert on any change to logging.
- How do you handle an investigation into an employee? Authority and a ticket first, restrict distribution, work from preserved evidence, report facts and gaps, and leave conclusions to HR and the DPO.
- What are the limits of the evidence? Only what was logged while logging was on, delivery delays, no coverage of data taken off-platform, and weak joins when request IDs are missing.
Certification practice
A security team must prove which objects in a bucket were read by a specific application role last month. Event history in the console shows no GetObject calls. What is the most likely reason?
A. Event history only shows events from the last 24 hours.
B. GetObject is a data event, and data events are not recorded unless enabled on a trail, and event history does not show them.
C. The role must be an IAM user for CloudTrail to record it.
D. CloudTrail records only write operations.
Answer: B. Object-level reads are data events, which are off by default and not shown in event history. A is wrong (event history covers 90 days of management events), C is wrong (role sessions are recorded), and D is wrong (CloudTrail records reads and writes; read-only filtering is optional).
An operations team wants to query a year of CloudTrail records for one resource at low cost, without running a database. Which approach fits?
A. Keep logs only in event history and filter by resource name. B. Deliver a trail to S3 and query it with Athena using a partitioned table. C. Export the console page to a spreadsheet each day. D. Enable CloudTrail Lake in a new account.
Answer: B. A trail to S3 gives retention you control and Athena queries it in place, with partitioning keeping scan costs low. Event history keeps 90 days (A), spreadsheets are neither complete nor tamper-evident (C), and CloudTrail Lake is closed to new customers (D).
A DevOps team wants an alert if anyone narrows or stops the logging of customer-data access. What should they do?
A. Review the trail configuration monthly.
B. Create an EventBridge rule or metric filter alarm on CloudTrail management events such as StopLogging, DeleteTrail and PutEventSelectors, notifying the security team.
C. Disable all IAM write permissions.
D. Move the trail to another Region.
Answer: B. Changes to logging are themselves recorded as management events, so a rule or alarm on those event names gives near-real-time detection. Monthly review (A) is too slow, removing all write permissions (C) is impractical, and moving the trail (D) does not detect changes.
Completion checklist
- I can explain why a role session identifies the service and not the person
- I can list the four evidence layers and one gap in each
- I ran the correlation tool and explained all four result groups
- I can describe how to confirm data events and log integrity before trusting a result
- I wrote down who should and should not see an investigation like this
- I ran the cleanup steps and confirmed no demo trail or bucket remains
Key takeaways
- Establish authority, subject, window and distribution before you query.
- CloudTrail names the role. The application audit log must name the person and record the AWS request ID.
- Data events are off by default, and you can only show what was logged while logging was on.
- The dangerous finding is a CloudTrail read with no application event.
- Report facts and gaps, hash the report, store it immutably and leave interpretation to the DPO and HR.
Further reading
- What is AWS CloudTrail?
- Logging data events
- Querying AWS CloudTrail logs with Amazon Athena
- Previous playbook: Key rotation without downtime
- Next playbook: Unexpected AWS bill spike
- Related: Module 17 overview
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Playbook: key rotation without downtime
Rotate KMS keys, database secrets, API keys and signing keys without an outage by using overlap periods: KMS automatic rotation versus manual key swap, Secrets Manager staging labels and alternating users, and clients that refresh credentials on failure.
Playbook: unexpected AWS bill spike
Find out what caused a cost spike by narrowing from account to service to usage type to resource, stop the spend safely, check for forgotten or looping resources, and set up budgets, anomaly monitors and tag rules so the same mistake is caught within a day.