Skip to Content
Cloud Engineering Academy
17. Daily-Work Playbooks & Career

Playbook: who accessed this customer record?

Answer a data protection team's access question with evidence: scope the request, pull application audit events, CloudTrail data events and database audit logs, join them by AWS request ID, find the access that bypassed the application, and hand over a hashed, reviewable report.

advanced⏱ 1 h 30 min🟢 FreeSCS-C03SOA-C03DOP-C02

Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · Amazon Athena engine v3 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this playbook you can:

  • Turn a vague "who looked at this customer?" request into a scoped, authorised evidence question: subject identifier, time window, systems in scope and who may see the answer.
  • Explain the four evidence layers (application audit log, CloudTrail data events, database audit, network and identity logs) and what each can and cannot tell you.
  • Explain why an AWS role session identifies a service, not a person, and what the application must record so a human can be named.
  • Query CloudTrail data events with Athena for one customer's objects and items, and join them to application audit events by AWS request ID.
  • Spot the dangerous finding: access that appears in CloudTrail but not in the application audit log.
  • Package the result as a hashed report, record that you ran the investigation, and hand it to the data protection team without drawing conclusions you cannot support.

Prerequisites

Why this matters in banking

On a Monday morning the data protection officer (DPO) forwards a complaint. A Harbourline customer, cust-4471 in our fictional data, says a relative who works at the bank knew details of her recent statement. She asks who has looked at her account in the last six weeks. Nobody knows. The Customer Service tool shows the cases agents opened, but the relative says they never opened a case. The compliance team asks you, the cloud engineer, one question: can you show every access to this customer's data, by whom, and why?

Three things go wrong in teams that have not prepared:

  1. The answer is "we do not know", because object-level logging was never switched on, or the logs expired after 30 days.
  2. The answer names the wrong thing. CloudTrail says hb-statements-task read the file 40 times, which is the application's role. It does not say which employee clicked the button.
  3. The investigation itself leaks. Someone runs ad hoc queries on production, pastes a customer ID into a chat, and now the evidence has been handled informally and the person under suspicion hears about it before HR does.

This playbook is about the method and the evidence design. It is not legal advice. What counts as an access log the customer is entitled to see, what must be disclosed, to whom and when are decisions for your DPO and legal team (UK GDPR is the reference flavour). Your job is to produce accurate, complete, clearly bounded evidence, and to say honestly where it has gaps.

The idea in plain language

"Who accessed this record?" is really four questions stacked on top of each other:

QuestionWhere the answer lives
Which person asked the system to show the data, and for what purpose?The application audit log: end-user identity, action, purpose or case reference, outcome
Which AWS identity actually read the data?CloudTrail data events: the role session, source IP, object key or table key, request ID
Did anyone read the data without going through the application?CloudTrail data events whose role is a human or support role, plus database audit logs
Could anyone change or delete the evidence?Log integrity validation, Object Lock, separate log-archive account

Analogy: The bank's branch has a customer-records room. The door badge reader (CloudTrail) records which badge opened the door, but the clerk at the counter (the application) is the one who knows which customer asked and why. If you only have badge logs, you know a clerk's badge was used. If you only have the clerk's notebook, you cannot prove nobody slipped in through the back door. You need both books, and you need to check that the entries in one match the other.

The "last hop" problem

CloudTrail identifies the IAM principal that called the AWS API. In a typical Harbourline service, every request to S3 or DynamoDB is made by the ECS task role, whatever employee is logged into the web interface. So for thousands of reads, userIdentity is the same role session. The human is invisible to AWS.

There are two ways to bring the human back:

  • Record the end-user identity in the application audit log and join it to CloudTrail with a shared key. This is the main method and the one you must design before you need it.
  • Use identity-propagating or per-user credentials for the few systems where access by an individual must be visible directly in AWS logs (for example, support staff reading a bucket through IAM Identity Center permission sets, where the role session name carries the user's name). This is the exception, not the default.

The shared key can be as simple as the AWS request ID. Every AWS API response carries one, the AWS SDK for .NET exposes it, and CloudTrail records it as requestID. If the application writes that ID into its audit event, the join is exact. ⚠ VERIFY for each service you rely on that the request ID visible in the SDK response matches the requestID field in the CloudTrail data event; test it once in a sandbox and record the result in your runbook. When that is not possible, you fall back to a time window plus role plus object key, which is weaker, and your report must say so.

What problem the AWS services solve

AWS gives you the raw material, but most of it is off by default or short-lived:

SourceWhat it showsDefaultSharp edges
CloudTrail management eventsWho created, changed or deleted resources, who assumed which roleOn; 90 days in event historyDoes not show reads of S3 objects or DynamoDB items
CloudTrail data eventsS3 object-level calls (GetObject, PutObject, DeleteObject), DynamoDB item-level calls, Lambda invokes and moreOffPriced per 100,000 events ⚠ VERIFY current rate; high-volume buckets get expensive; you must select which resources to log
Application audit eventsEnd-user, action, subject, purpose, outcomeOnly if you built themMust be tamper-evident and must not contain raw personal data
Database auditQueries and logins: pgaudit for PostgreSQL, SQL Server Audit, Aurora Database Activity StreamsOffNot in CloudTrail; each engine has its own switch and overhead ⚠ VERIFY engine and version support
S3 server access logsBest-effort request records for a bucketOffDelivered with delay and on a best-effort basis, so not complete evidence on their own
VPC Flow Logs, ALB access logsConnections and HTTP requestsOffShow that something talked to something, not which record
IAM Identity Center / sign-in eventsWho signed in to the console or CLI with which permission setOn in CloudTrail management eventsShows sign-in, not what they then read

CloudTrail Lake is not an option for new accounts — verified 2026-11-17

AWS CloudTrail Lake closed to new customers on 2026-05-31, so this playbook uses a trail delivering to S3, queried with Amazon Athena, plus an optional copy into CloudWatch Logs. If your bank already uses CloudTrail Lake, the same questions work there with SQL, but do not design a new capability around it. AWS Systems Manager Incident Manager is also closed to new customers; use your bank's own incident tooling.

Honest limits to tell the DPO up front:

  • CloudTrail delivers log files to S3 typically within minutes, not in real time ⚠ VERIFY the current typical delay.
  • You can only show what was logged while logging was on. If data events were enabled on 1 October, you cannot reconstruct September from CloudTrail.
  • Database content changes, exports through an engine-specific tool and anything outside AWS (laptops, email) are outside this evidence unless you have separate logs.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Activity LogCloudTrail management eventsBoth record control-plane calls. CloudTrail also records role assumptions and console sign-ins
Azure Storage analytics logs, Azure Monitor resource logs for StorageCloudTrail S3 data events (and S3 server access logs)Data events are chosen per bucket or prefix and billed per event
Microsoft Purview audit, SQL auditing to Log AnalyticsApplication audit table, pgaudit or SQL Server Audit, Aurora Database Activity StreamsAuditing the database is an engine feature you switch on, not a platform default
Entra ID sign-in logs plus Conditional AccessIAM Identity Center events in CloudTrail, plus your IdP's own logsThe identity provider (for example Entra ID) holds the person; CloudTrail holds the role session
Log Analytics workspace with KQLAthena (SQL) over S3, or CloudWatch Logs InsightsAthena charges per data scanned, so partition by date and project columns
Azure Policy audit effectAWS Config rules and Security Hub controlsShows configuration state, not who read data
Immutable blob storage for logsS3 Object Lock in a log-archive accountCompliance mode cannot be shortened, even by the account root user

Architecture

The four evidence layers

Arrows 1 to 4 are the normal path: the person is known to the application and the AWS read is made by the task role. Arrow 6 is the case you are hunting for: a human with their own AWS access reading data directly. It leaves no application audit event, so it shows up in CloudTrail with a person's name in the role session and nothing on the other side to match.

Joining the two books

Step-by-step implementation

Minutes 0 to 10: scope, authority and containment of the investigation

Before you query anything:

  1. Who is asking, and are they entitled to ask? The request should come from the DPO, compliance or HR through a ticket (a fictional CASE-9930 here), not a chat message. A manager asking "just check on this person" is a red flag. Write down who asked, when and why.
  2. Define the subject. Use the pseudonymous internal ID (cust-4471), never the customer's name or account number, in queries, tickets and chats.
  3. Define the window. Start and end in UTC. Add a buffer on each side for clock skew and delivery delay.
  4. List systems in scope. Application, S3 buckets, DynamoDB tables, RDS databases, support tooling, exports to analytics.
  5. Limit who sees the result. Agree who gets the findings. If the concern is that an employee misused access, the employee must not be warned by a broad distribution list.
  6. Preserve first. Check retention. If any source is close to expiry, copy it (to the evidence bucket) before you analyse it. Do not change anything in production.

Do not draw conclusions. At this stage you are collecting facts for a person with the authority to decide what they mean.

Minutes 10 to 30: check that the evidence exists

Ask what you will be able to show, before the clock runs:

  • Is there a trail with data events for the statements bucket and the customers table, and since when? (aws cloudtrail get-event-selectors.)
  • Is the trail delivering, and is log file validation on? (aws cloudtrail get-trail-status.)
  • What is the retention of the application audit log and of the trail's S3 bucket?
  • Does the application audit event contain the end-user ID, subject, purpose, outcome and AWS request IDs?

Write the answers into the evidence record. A gap is a finding, not an embarrassment to hide: "data events for the statements bucket were enabled from 2026-10-01; reads before that date cannot be shown from CloudTrail."

Minutes 30 to 60: pull the application and AWS records

  1. Application audit events for the subject in the window: extract to a file, with an export query that is itself logged.
  2. CloudTrail data events for the subject's object prefix and DynamoDB key: query with Athena (see the CLI section) and export the result.
  3. Management events for the same window: anyone who changed bucket policies, trails, event selectors or IAM roles that grant access to this data. Disabling or narrowing a trail during the period is its own finding.
  4. Database audit where the subject's data also lives in RDS: query the pgaudit or SQL Server audit stream for the subject's key.

Minutes 60 to 80: correlate and classify

Run the correlation tool below. Every access ends up in one of three groups:

GroupMeaningWhat to do
MatchedApplication event and CloudTrail event agree (by request ID, or weakly by time)Check each has a plausible purpose or case reference
Trail-onlyCloudTrail shows a read of the subject's data and the application has no recordThe important group. A support engineer, a script, a leaked key, or an application path that forgot to log. Escalate per the incident process
App-onlyThe application says it served the data but no AWS read is visibleServed from cache, a data event gap, or logging enabled later than the access. Explain, do not ignore

A denied application event (outcome denied) is evidence too: someone tried and was refused.

Minutes 80 to 90: report, hash, record

Write the report as facts only: what was searched, which sources, what period, what was found, what could not be determined and why. Hash it, store it in the evidence bucket with Object Lock, and record in the case: who ran which queries, when, from which role. Hand it to the DPO. Leave interpretation, disclosure to the customer and any HR or regulatory step to them.

Likely causes of an incomplete answer, ranked

When the evidence does not answer the question, work down this list:

  1. Data events were not enabled for that bucket or table during the window.
  2. The application logs the role but not the end user (or logs the end user's display name rather than a stable ID).
  3. Logs expired before the request arrived (short retention).
  4. The access was through a path the application audit does not cover (batch job, export, admin tool, support console).
  5. The object keys do not contain a subject ID, so you cannot find a customer's objects in CloudTrail without joining to a lookup table.
  6. Time zones and clock skew made a time-window join miss a pair.
  7. The trail had a delivery gap (check the digest files and get-trail-status).

AWS Console instructions

Check that data events are on: Console → CloudTrail → Trails → choose the organisation or account trail → Data events section. Look for an S3 entry covering the statements bucket and a DynamoDB entry for the customers table. If the page shows "Advanced event selectors" with eventCategory = Data, expand them to read the exact resource filters.

Check delivery and integrity: the same trail page shows Log file validation (Enabled), the destination bucket and the Latest delivery time.

Look at management events for the last 90 days: Console → CloudTrail → Event history. Filter by event name (for example PutEventSelectors, StopLogging, PutBucketPolicy) to see if anyone changed the logging itself. Event history shows management events only, not data events.

Run the data-event query: Console → Athena → choose the workgroup hb-security-investigations → Query editor. The workgroup should enforce an output location in the evidence bucket and a per-query data-scanned limit.

AWS CLI

# 1) Which events does the trail record? Look for Data events on the statements bucket
#    and the customers table. Advanced selectors are the current format.
aws cloudtrail get-event-selectors --trail-name hb-org-trail --profile learn-sandbox

# 2) Is the trail delivering and is anything stuck? LatestDeliveryError should be absent.
aws cloudtrail get-trail-status --name hb-org-trail --profile learn-sandbox

# 3) Did anyone touch the logging configuration in the window? (management events, 90 days)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=PutEventSelectors \
  --start-time 2026-10-01T00:00:00Z --end-time 2026-11-18T00:00:00Z \
  --profile learn-sandbox

# 4) Start an Athena query in the locked-down investigations workgroup.
#    Keep the SQL in a file so the exact query becomes part of the evidence.
aws athena start-query-execution \
  --work-group hb-security-investigations \
  --query-string file://who-accessed.sql \
  --query-execution-context Database=security_logs \
  --profile learn-sandbox

# 5) Check the query state, then fetch results (QueryExecutionId comes from step 4).
aws athena get-query-execution --query-execution-id 11111111-2222-3333-4444-555555555555 \
  --profile learn-sandbox
aws athena get-query-results --query-execution-id 11111111-2222-3333-4444-555555555555 \
  --profile learn-sandbox

# 6) Confirm log file integrity for the period (digest validation).
aws cloudtrail validate-logs \
  --trail-arn arn:aws:cloudtrail:us-east-1:111122223333:trail/hb-org-trail \
  --start-time 2026-10-01T00:00:00Z --end-time 2026-11-18T00:00:00Z \
  --profile learn-sandbox

The SQL for step 4 (who-accessed.sql). It assumes a CloudTrail table named cloudtrail_logs with partition projection on date; the Athena table definition for CloudTrail is in the audit evidence lesson. ⚠ VERIFY column names against your own table definition before relying on the query.

-- Every data event that touched cust-4471's objects or items, 1 Oct to 18 Nov 2026 (UTC).
-- The partition filter keeps the scan small, and therefore cheap.
SELECT
  eventtime,
  eventname,
  useridentity.arn                      AS principal,
  useridentity.sessioncontext.sessionissuer.username AS role_name,
  sourceipaddress,
  requestid,
  json_extract_scalar(requestparameters, '$.bucketName') AS bucket,
  json_extract_scalar(requestparameters, '$.key')         AS object_key,
  errorcode
FROM cloudtrail_logs
WHERE date >= '2026/10/01' AND date <= '2026/11/18'
  AND eventsource IN ('s3.amazonaws.com', 'dynamodb.amazonaws.com')
  AND (   json_extract_scalar(requestparameters, '$.key') LIKE 'statements/cust-4471/%'
       OR requestparameters LIKE '%"cust-4471"%')
ORDER BY eventtime;

Infrastructure as Code

The trail needs data events for specific resources. Select the statements bucket prefix and the customers table, not "all S3 data events".

using Amazon.CDK;
using Amazon.CDK.AWS.CloudTrail;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.S3;
using Constructs;

namespace Harbourline.Playbooks;

public class CustomerAccessLoggingStack : Stack
{
    public CustomerAccessLoggingStack(Construct scope, string id, IBucket statements,
        ITable customers, IBucket logArchive, IStackProps? props = null) : base(scope, id, props)
    {
        var trail = new Trail(this, "CustomerDataTrail", new TrailProps
        {
            Bucket = logArchive,                  // Object Lock bucket in the log-archive account
            EnableFileValidation = true,
            IncludeGlobalServiceEvents = false,   // management events are covered by the org trail
            ManagementEvents = ReadWriteType.NONE // this trail is only for customer-data access
        });

        // S3 object-level events for the statements prefix only, reads and writes.
        trail.AddS3EventSelector(new[]
        {
            new S3EventSelector { Bucket = statements, ObjectPrefix = "statements/" }
        }, new AddEventSelectorOptions { ReadWriteType = ReadWriteType.ALL });

        // DynamoDB item-level events for the customers table.
        trail.AddEventSelector(DataResourceType.DYNAMODB_TABLE,
            new[] { customers.TableArn },
            new AddEventSelectorOptions { ReadWriteType = ReadWriteType.ALL });

        Amazon.CDK.Tags.Of(this).Add("Project", "Harbourline");
        Amazon.CDK.Tags.Of(this).Add("DataClassification", "evidence");
    }
}

Check the property names against your versions

The CDK Trail helpers and the Terraform advanced_event_selector block exist in current releases, but construct and argument names move between versions. ⚠ VERIFY both snippets against the CDK and Terraform provider versions you pin, and run terraform validate or dotnet build before applying.

Complete code

The correlation tool reads the application audit log (JSON lines) and a CloudTrail log file, filters both to one subject, joins them, and writes a hashed report. It has no AWS dependency, so you can run it on exported files and on the sample data below. The exit code is 1 when there are trail-only accesses, so a pipeline or a runbook step can stop and escalate.

AccessTrail.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>
</Project>
Program.cs
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;

// Usage: dotnet run -- <subject-id> <app-audit.jsonl> <cloudtrail-export.json> <report.json>
if (args.Length != 4)
{
    Console.Error.WriteLine("usage: AccessTrail <subject-id> <app-audit.jsonl> <cloudtrail-export.json> <report.json>");
    return 2;
}

string subject = args[0];
var appEvents = AccessTrailBuilder.ReadAppEvents(File.ReadLines(args[1]), subject);
var trailEvents = AccessTrailBuilder.ReadTrailEvents(File.ReadAllText(args[2]), subject);
var report = AccessTrailBuilder.Build(subject, appEvents, trailEvents, TimeSpan.FromSeconds(5));

// Hash the exact bytes we write, so the hash file matches the report file byte for byte.
var bytes = Encoding.UTF8.GetBytes(JsonSerializer.Serialize(report, new JsonSerializerOptions { WriteIndented = true }));
File.WriteAllBytes(args[3], bytes);
string hash = Convert.ToHexString(SHA256.HashData(bytes)).ToLowerInvariant();
File.WriteAllText(args[3] + ".sha256", hash + "  " + Path.GetFileName(args[3]) + "\n");

Console.WriteLine($"Subject {subject}: {report.Matched.Count} matched, " +
                  $"{report.TrailOnly.Count} trail-only (bypassed the app?), " +
                  $"{report.AppOnly.Count} app-only (no AWS call seen).");
Console.WriteLine($"SHA-256 {hash}");
return report.TrailOnly.Count > 0 ? 1 : 0;

public sealed record AppEvent(
    DateTimeOffset Timestamp, string CorrelationId, string Actor, string Action,
    string Subject, string? Purpose, string Outcome, IReadOnlyList<string> AwsRequestIds);

public sealed record TrailEvent(
    DateTimeOffset Time, string EventName, string Principal, string? SourceIp,
    string? RequestId, string? Resource);

public sealed record MatchedAccess(AppEvent App, TrailEvent Trail, string Basis);

public sealed record AccessReport(
    string Subject,
    IReadOnlyList<MatchedAccess> Matched,
    IReadOnlyList<TrailEvent> TrailOnly,
    IReadOnlyList<AppEvent> AppOnly);

public static class AccessTrailBuilder
{
    public static List<AppEvent> ReadAppEvents(IEnumerable<string> lines, string subject)
    {
        var result = new List<AppEvent>();
        foreach (var line in lines)
        {
            if (string.IsNullOrWhiteSpace(line)) continue;
            using var doc = JsonDocument.Parse(line);
            var r = doc.RootElement;
            if (!string.Equals(Str(r, "subject"), subject, StringComparison.Ordinal)) continue;

            var ids = new List<string>();
            if (r.TryGetProperty("awsRequestIds", out var arr) && arr.ValueKind == JsonValueKind.Array)
                foreach (var id in arr.EnumerateArray())
                    if (id.GetString() is { Length: > 0 } s) ids.Add(s);

            result.Add(new AppEvent(
                DateTimeOffset.Parse(Str(r, "ts")!), Str(r, "correlationId") ?? "", Str(r, "actor") ?? "unknown",
                Str(r, "action") ?? "", subject, Str(r, "purpose"), Str(r, "outcome") ?? "", ids));
        }
        return result.OrderBy(e => e.Timestamp).ToList();
    }

    // Reads a CloudTrail log file ({"Records":[...]}) and keeps data events whose
    // resource (bucket/key or table/key) mentions the subject's pseudonymous id.
    public static List<TrailEvent> ReadTrailEvents(string json, string subject)
    {
        var result = new List<TrailEvent>();
        using var doc = JsonDocument.Parse(json);
        if (!doc.RootElement.TryGetProperty("Records", out var records)) return result;

        foreach (var rec in records.EnumerateArray())
        {
            string? resource = DescribeResource(rec);
            if (resource is null || !resource.Contains(subject, StringComparison.Ordinal)) continue;

            string principal = "unknown";
            if (rec.TryGetProperty("userIdentity", out var ui))
                principal = Str(ui, "arn") ?? Str(ui, "principalId") ?? "unknown";

            result.Add(new TrailEvent(
                DateTimeOffset.Parse(Str(rec, "eventTime")!), Str(rec, "eventName") ?? "",
                principal, Str(rec, "sourceIPAddress"), Str(rec, "requestID"), resource));
        }
        return result.OrderBy(e => e.Time).ToList();
    }

    public static AccessReport Build(string subject, IReadOnlyList<AppEvent> app,
                                     IReadOnlyList<TrailEvent> trail, TimeSpan window)
    {
        var matched = new List<MatchedAccess>();
        var usedTrail = new HashSet<TrailEvent>();
        var unmatchedApp = new List<AppEvent>();

        foreach (var a in app)
        {
            // Strong join: the application recorded the AWS request id it received.
            var byId = trail.FirstOrDefault(t => t.RequestId is not null
                && a.AwsRequestIds.Contains(t.RequestId) && !usedTrail.Contains(t));
            if (byId is not null)
            {
                usedTrail.Add(byId);
                matched.Add(new MatchedAccess(a, byId, "request-id"));
                continue;
            }
            // Weak join: same moment (within the window). Say so in the report.
            var byTime = trail.FirstOrDefault(t => !usedTrail.Contains(t)
                && (t.Time - a.Timestamp).Duration() <= window);
            if (byTime is not null)
            {
                usedTrail.Add(byTime);
                matched.Add(new MatchedAccess(a, byTime, "time-window"));
                continue;
            }
            unmatchedApp.Add(a);
        }

        var trailOnly = trail.Where(t => !usedTrail.Contains(t)).ToList();
        return new AccessReport(subject, matched, trailOnly, unmatchedApp);
    }

    private static string? DescribeResource(JsonElement rec)
    {
        if (!rec.TryGetProperty("requestParameters", out var p) || p.ValueKind != JsonValueKind.Object)
            return null;
        string? bucket = Str(p, "bucketName");
        string? key = Str(p, "key");
        if (bucket is not null && key is not null) return $"s3://{bucket}/{key}";
        string? table = Str(p, "tableName");
        if (table is not null && p.TryGetProperty("key", out var k)) return $"dynamodb:{table}/{k.GetRawText()}";
        return null;
    }

    private static string? Str(JsonElement e, string name) =>
        e.ValueKind == JsonValueKind.Object && e.TryGetProperty(name, out var v) && v.ValueKind == JsonValueKind.String
            ? v.GetString() : null;
}

Sample inputs (all fictional). Save them next to the project:

app-audit.jsonl
{"ts":"2026-11-03T09:14:22Z","correlationId":"c-001","actor":"emp-7781","action":"ViewStatement","subject":"cust-4471","purpose":"CASE-9921","outcome":"allowed","awsRequestIds":["REQ-A1"]}
{"ts":"2026-11-03T09:20:05Z","correlationId":"c-002","actor":"emp-7781","action":"ViewProfile","subject":"cust-4471","purpose":"CASE-9921","outcome":"allowed","awsRequestIds":[]}
{"ts":"2026-11-04T16:02:41Z","correlationId":"c-003","actor":"emp-1203","action":"ViewStatement","subject":"cust-9999","purpose":"CASE-0001","outcome":"allowed","awsRequestIds":["REQ-Z9"]}
{"ts":"2026-11-05T11:45:00Z","correlationId":"c-004","actor":"emp-3310","action":"ViewStatement","subject":"cust-4471","purpose":null,"outcome":"denied","awsRequestIds":[]}
cloudtrail-export.json
{"Records":[
{"eventTime":"2026-11-03T09:14:22Z","eventName":"GetObject","sourceIPAddress":"10.0.12.34","requestID":"REQ-A1","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/hb-statements-task/0f3c9d2a"},"requestParameters":{"bucketName":"hb-statements-111122223333","key":"statements/cust-4471/2026-10.pdf"}},
{"eventTime":"2026-11-03T09:20:06Z","eventName":"GetItem","sourceIPAddress":"10.0.12.34","requestID":"REQ-B2","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/hb-statements-task/0f3c9d2a"},"requestParameters":{"tableName":"hb-customers","key":{"pk":{"S":"cust-4471"}}}},
{"eventTime":"2026-11-07T22:31:09Z","eventName":"GetObject","sourceIPAddress":"203.0.113.50","requestID":"REQ-C3","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/hb-support-readonly/alex.example"},"requestParameters":{"bucketName":"hb-statements-111122223333","key":"statements/cust-4471/2026-09.pdf"}},
{"eventTime":"2026-11-07T22:31:10Z","eventName":"ListBuckets","userIdentity":{"arn":"x"},"requestParameters":null}
]}

Run it:

dotnet run -- cust-4471 app-audit.jsonl cloudtrail-export.json report.json

Expected output (the hash will match when you run the exact files above, byte for byte):

Subject cust-4471: 2 matched, 1 trail-only (bypassed the app?), 1 app-only (no AWS call seen).
SHA-256 <64 hex characters>

Reading the result: the first access matched exactly by request ID; the second matched only by time, so the report marks it time-window (a weaker claim); the 7 November read by hb-support-readonly at 22:31 from a non-VPC address has no application record and is the trail-only finding; and the denied attempt on 5 November is application-only, because a denied request never reached S3.

Code explained line by line

  1. Argument check. The tool takes the subject, the two input files and an output path. A wrong count returns exit code 2 so a script does not treat "no arguments" as "no findings".
  2. ReadAppEvents. Reads JSON lines, keeps only events for the subject and keeps the AWS request IDs the application recorded. Using JsonDocument means one bad line throws a clear parse error instead of silently dropping data.
  3. ReadTrailEvents. CloudTrail files wrap events in {"Records":[...]}. DescribeResource turns S3 (bucketName plus key) and DynamoDB (tableName plus key) parameters into a readable resource string and drops events that are not data events (such as ListBuckets).
  4. Subject filter. A CloudTrail event is kept only if its resource string contains the pseudonymous subject ID. This is why object keys that include cust-4471 are useful, and why you must choose a pseudonymous ID rather than an account number.
  5. Build strong join. For each application event, look for a CloudTrail event with one of the AWS request IDs the application recorded. usedTrail makes sure one CloudTrail event is not matched twice.
  6. Build weak join. If there is no request-ID match, accept a CloudTrail event within a five-second window, and record time-window as the basis. A weak join is an inference; say so.
  7. Leftovers become the findings. CloudTrail events nobody claimed are TrailOnly; application events nothing matched are AppOnly.
  8. Hashing. The report is written as bytes and the SHA-256 of the same bytes goes into a .sha256 file, so the reviewer can verify the report was not edited after generation. Hashing gives tamper evidence; to prevent change, store the files in an Object Lock bucket.
  9. Exit code. 1 means "there is something to escalate". Pipelines and runbook scripts can branch on it.

Security considerations

⚠ Learning shortcut — not for production

The sample reads local files and uses a five-second time window for the weak join. In production, run the query and the correlation in a locked-down investigation account with a read-only role, store results in an Object Lock bucket, and treat every investigation as a ticketed, logged activity. A fixed five-second window is also too crude for busy systems; prefer request IDs, and tune the window to your measured clock skew and processing delay.

  • The evidence is personal data. A report that lists employee identities and a customer's activity is itself sensitive. Restrict access, encrypt with a dedicated KMS key, and set a retention period agreed with the DPO.
  • No raw PII in logs. Use pseudonymous IDs in audit events and object keys. If keys carry account numbers or names, the logs leak what they are meant to protect.
  • Separate duties. The people who run the service should not control the log-archive account or be able to disable the trail. Alert on StopLogging, PutEventSelectors, DeleteTrail and bucket policy changes.
  • Query access is access. Give investigators a role that can run queries in one workgroup and read the evidence bucket, and nothing else. Log their Athena queries (they are in CloudTrail).
  • Sensitive subject. If the person of interest is an employee, involve HR and legal before anyone broadens access to the report.
  • Never claim compliance. The report supports your bank's answer; whether it satisfies a regulator, a subject access request or an employment process is for your legal and compliance functions.

Banking use case

Harbourline's DPO opens CASE-9930 for cust-4471. The cloud team confirms data events on the statements bucket and customers table have been on since 1 October, runs the Athena query and the correlation tool, and finds the pattern shown in the sample: two normal accesses for a documented case, one denied attempt, and one read of a September statement at 22:31 by a support role session named after an individual from an address outside the bank's VPN range. The application never saw it.

The team does not conclude misconduct. They report: the access exists, who the role session was issued to (from the Identity Center sign-in record in CloudTrail, in the same window), the object read, the source IP, and that the read bypassed the application audit trail. The DPO and HR decide what happens next. In parallel the engineering team acts on the control gap: support staff should not hold a standing role with direct read access to customer statements, and any break-glass access should require a ticket reference and an alert. The follow-up ticket also adds the AWS request ID to every application audit event so the next investigation is exact.

Common mistakes

  • Starting the investigation before authority is confirmed. Why: urgency and curiosity. Avoid: require a ticket from the DPO, compliance or HR before the first query.
  • Assuming CloudTrail shows who read the data. Why: it shows the role. Avoid: record the end user in the application audit log and join by request ID.
  • Assuming data events are on. Why: management events are on by default and look similar. Avoid: check get-event-selectors first and record when logging began.
  • Searching by name or account number. Why: that is what the complaint contains. Avoid: translate once to a pseudonymous ID, and keep the lookup restricted.
  • Treating a missing match as "no access". Why: absence of a log is not absence of an action. Avoid: state the coverage (period, systems) explicitly in the report.
  • Editing the evidence. Why: tidying a spreadsheet. Avoid: work on copies, hash the originals and the report, store in Object Lock.
  • Reporting conclusions. Why: wanting to be helpful. Avoid: report facts and gaps; let the DPO and HR interpret.

Troubleshooting

SymptomLikely causeFix
Athena returns no rows for a customer who was definitely viewedData events were not enabled for the bucket or table, or the trail's S3 prefix or partition projection range is wrongCheck get-event-selectors; check the table's date range; widen the partition filter
Athena query scans many GB and costs more than expectedNo partition filter, SELECT *, large JSON column readFilter on the partition column, select only the columns you need, set a workgroup data-scanned limit
HIVE_BAD_DATA or JSON parse errors in AthenaA row with an unexpected shape, or the wrong SerDe in the table definitionUse the CloudTrail table definition from the audit lesson; ⚠ VERIFY SerDe and input format for your engine version
Every row shows the same role and no personNormal for service-mediated accessJoin to the application audit log by request ID
Request IDs never matchThe application did not record them, or the field is a different ID for that serviceTest one call in the sandbox, compare the SDK response ID with the CloudTrail requestID; fall back to the time window and say so
Events appear hours lateDelivery delay, or a large file still being writtenRe-run after the delay; check get-trail-status for delivery errors
validate-logs reports invalid or missing digestsLog tampering, delivery gap, or a trail stopped during the windowEscalate as a security incident; check StopLogging events
App-only events with allowed outcome but no AWS readCache hit, replica read path, or logging started laterExplain the read path in the report; check the cache and its own audit

Cost and cleanup

Cost: free for the sample. The correlation tool runs locally on sample files. Nothing needs an AWS account.

If you reproduce this in the sandbox: CloudTrail data events are priced per 100,000 events (about $0.10 at the last check) ⚠ VERIFY, so a few thousand test calls cost cents. Athena charges per data scanned (a few cents for sample logs) and S3 charges for the stored logs. Delete anything you create the same day.

# Delete the demo trail and stop its data events (only if you created it for this exercise).
aws cloudtrail delete-trail --name hb-customer-data-trail --profile learn-sandbox

# Empty and remove the demo log bucket (never do this to a real log-archive bucket).
aws s3 rm s3://hb-demo-trail-logs-111122223333 --recursive --profile learn-sandbox
aws s3 rb s3://hb-demo-trail-logs-111122223333 --profile learn-sandbox

# Remove local sample outputs.
rm -f report.json report.json.sha256

Confirm nothing still bills: Console → CloudTrail → Trails (none left from this exercise) and Billing → Cost Explorer by service tomorrow.

Exercises

  • 🟢 Guided: Run the correlation tool on the sample files. Explain, for each of the four events, which group it landed in and why. Change one application event so its request ID is REQ-A1 but its timestamp is off by an hour, and confirm it still matches by request ID.
  • 🟡 Challenge: Extend the tool to read the application events' purpose and flag any allowed event with no purpose. Add a "denied attempts" section to the report that lists repeated denials by the same actor.
  • 🔴 Live problem: The DPO says: "A customer's relative may have looked at her account over the last three months. We enabled data events two weeks ago." Draft the message you send back: what you can show, what you cannot, what you will do now to close the gap, and who needs to be in the loop. Do not speculate about the relative.

Assignment

Write a one-page "customer access investigation" runbook for your portfolio: request intake and authority, scope definition, the four evidence layers with the exact commands for your stack, the correlation method, a report template, the retention and Object Lock settings, and a list of gaps your design would need to close. Keep all data fictional.

Solutions

Knowledge check

Interview questions

  • How would you find out who accessed a customer's data in AWS? Check the evidence exists first (data events, retention), then combine application audit events (the person and purpose) with CloudTrail data events (the AWS reads), join by request ID, and list accesses that exist in one source and not the other.
  • Why isn't CloudTrail enough on its own? It identifies the IAM principal, which is usually a service role. It does not know the human, the purpose, or reads that never reached AWS APIs, such as cached data.
  • What would you change so the next investigation is easier? Add the end-user ID and AWS request ID to every audit event, enable data events for sensitive resources, use pseudonymous IDs in keys, set retention that matches policy and alert on any change to logging.
  • How do you handle an investigation into an employee? Authority and a ticket first, restrict distribution, work from preserved evidence, report facts and gaps, and leave conclusions to HR and the DPO.
  • What are the limits of the evidence? Only what was logged while logging was on, delivery delays, no coverage of data taken off-platform, and weak joins when request IDs are missing.

Certification practice

Completion checklist

  • I can explain why a role session identifies the service and not the person
  • I can list the four evidence layers and one gap in each
  • I ran the correlation tool and explained all four result groups
  • I can describe how to confirm data events and log integrity before trusting a result
  • I wrote down who should and should not see an investigation like this
  • I ran the cleanup steps and confirmed no demo trail or bucket remains

Key takeaways

  • Establish authority, subject, window and distribution before you query.
  • CloudTrail names the role. The application audit log must name the person and record the AWS request ID.
  • Data events are off by default, and you can only show what was logged while logging was on.
  • The dangerous finding is a CloudTrail read with no application event.
  • Report facts and gaps, hash the report, store it immutably and leave interpretation to the DPO and HR.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page