Skip to Content
Cloud Engineering Academy
7. Data Engineering & ETL Pipelines

Secure pipeline design: encryption, least privilege, private networking and PII protection

Design a data pipeline that is safe by construction: a KMS key per data domain, one narrow role per job, private network paths, PII tokenised and masked inside the job, and an audit trail you can show a reviewer.

advanced⏱ 2 h 🟢 FreeDEA-C01SCS-C03SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.S3 4.1.1, AWSSDK.SecretsManager 4.1.1.2) · AWS CDK v2 (C#) · AWS Glue 5.0 · AWS KMS · Amazon Macie · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain what must be encrypted in a Glue and S3 pipeline (data in S3, job bookmarks, logs, query results, secrets and streams) and which key protects each.
  • Split a pipeline into roles with different powers (ingest, transform, analyst, reporting, administrator) and justify every permission on the Glue job role in the project stack.
  • Decide when a Glue job needs a VPC network connection, which VPC endpoints it then needs, and what they cost.
  • Choose between masking, tokenising, hashing and encrypting a personal-data column, and explain why a plain SHA-256 of an IBAN is not protection.
  • Run a small C# audit tool that checks the pipeline's buckets and secret against the baseline, and list what you would still need from a security review.

Prerequisites

Why this matters in banking

Harbourline's pipeline holds three kinds of dangerous data: raw files with full counterparty IBANs and names, a quarantine zone that by design keeps the raw values of the rows that failed, and a tokenisation key that can turn tokens back into IBANs if someone guesses them. A reviewer will ask six questions, and you should be able to answer each with a setting or a line of code, not a promise:

  1. Who can read raw data, and can an analyst reach it by a side door?
  2. Does every copy of the data sit encrypted under a key that we control, and who can use that key?
  3. What happens if a Glue job role is stolen: how much can it read, write and delete?
  4. Does any of this travel over the public internet?
  5. Where does personal data stop? Does the curated zone still contain the raw IBAN?
  6. Can you prove, for one file, who processed it, when and with what result?

A pipeline built only to "work" answers these with "it depends". A pipeline designed to be secure answers them from the infrastructure code.

The idea in plain language

Think of the pipeline as a cash-in-transit route. The van (the Glue job) collects from one place (raw), drives a fixed route and delivers to others (curated, quarantine, gold). A secure design has four layers:

  • Locked boxes (encryption). Everything that rests is encrypted, and the keys are held by a separate team from the drivers.
  • Named keys for named doors (least privilege). The van's key opens exactly the doors on its route, and the raw door is open for reading only.
  • A private road (network). Where policy says traffic must not touch the public internet, the route uses private paths.
  • Redaction at the source (data protection). The most sensitive field is replaced by a token before the data leaves the loading bay, so the later zones never hold it.

Analogy: a bank does not make the cashier who counts the money also hold the vault's master key, and it does not leave a customer's full card number on the receipt. Encryption keys, privileges and redaction are the same habits applied to files.

What problem the AWS service solves

No single AWS service "secures a pipeline". The controls are spread over several, and the design job is knowing which setting does what:

ControlAWS featureWhat it protects againstHonest limitation
Encryption at restS3 default encryption with SSE-KMS and a customer managed key; Glue security configuration; encrypted CloudWatch log groups, SNS, SQS, DynamoDB, KinesisLost disks, stray access to storage, an over-broad S3 policy that cannot use the keyAnyone allowed to use the key and read the object still reads plaintext. Encryption is a second lock, not authorisation
Key separationKMS key policy with distinct administrators and users; key per domainOne compromised role decrypting everythingA customer managed key costs about USD 1 per month plus API calls ⚠ VERIFY current price
Least privilegeIAM roles per function, resource-scoped policies, permission boundariesA stolen job role used for wider damageNeeds maintenance; wildcards creep back
Private networkingS3 gateway endpoint, interface endpoints (KMS, Secrets Manager, STS, Glue), Glue network connectionTraffic over the public internet, data exfiltration to unknown endpointsInterface endpoints bill per hour per AZ; a mis-built network makes jobs time out
PII protectionTokenise, mask or drop in the job; Lake Formation filters downstreamAnalysts and reports seeing raw personal dataPseudonymised data can still be personal data under UK GDPR
DiscoveryAmazon MaciePersonal data in places you did not expectSampling and per-GB costs; findings are leads, not proof
EvidenceCloudTrail, Step Functions history, the run's audit record, S3 access logs"Prove who did what"Logs holding PII are a new risk. Log ids and counts, not rows

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Key Vault (keys) with customer-managed keys on storageAWS KMS customer managed key on S3, Glue, Athena resultsKey administration and key use are separate permissions in the key policy, and the key policy is mandatory, not optional
Key Vault secretsAWS Secrets ManagerEncrypted by a KMS key; automatic rotation needs a rotation Lambda
Managed identity for a Data Factory pipelineIAM role assumed by the Glue jobThe role is the identity; no stored credential. The role is passed with iam:PassRole
Private Endpoint / Private Link on storageS3 gateway endpoint (free) or interface endpoints (hourly charge)S3 uses a gateway endpoint in route tables, not a private IP in your subnet
Azure Policy "deny public network access"S3 Block Public Access, SCPs, bucket policy conditionsAccount-level and bucket-level switches
Purview sensitivity scansAmazon MacieScans S3 only; ties into EventBridge and Security Hub
Always Encrypted / dynamic data masking in SQL ServerTokenise or mask in the pipeline; Lake Formation column and row filtersThere is no engine-wide masking function; you design it into the job

Architecture

Security-boundary view. There are five trust zones. The partner zone can only call one HTTPS route and write under one prefix. The pipeline zone (Glue role, Step Functions) reads raw, writes the other zones and reads one secret. The analyst zone reaches curated data only through Athena and Lake Formation, and has no S3 permission on the zone buckets. The key zone (KMS key policy, Secrets Manager) is administered by a different role from the pipeline. The audit zone (CloudTrail, the audit record) is write-once from the pipeline's point of view: the job can add evidence but not erase it.

Step-by-step implementation

Threat-model the pipeline before configuring anything

List what can go wrong, then map each risk to a control. This table is the one the project uses.

ThreatExampleControl in the project
A hostile or broken fileA partner sends an oversized line, a wrong header, or a text field starting with =Header check rejects the file; row-level reason codes; size limits at the API (1,000 records per request)
Over-privileged jobA stolen Glue role deletes raw evidenceRaw is GrantRead only; versioned buckets; no s3:DeleteObject on raw
Analyst side doorAmazonS3ReadOnlyAccess attached to an analyst roleLake Formation filters plus no direct S3 permission (previous lesson)
Key abuseSomeone decrypts everything with the data keyKey policy separates administrators from users; CloudTrail alarm on key policy changes
Secret leakageThe HMAC key printed in a job logKey fetched at run time, never logged; logs encrypted; only counts and ids are logged
PII in the wrong zoneRaw IBAN in curated or in a Step Functions logRaw IBAN dropped in the job; execution data excluded from state-machine logs
Replay or double processingThe same file uploaded twiceRun id from bucket, key and version id; audit record checked first
Silent tamperingA raw file overwrittenVersioning on raw and quarantine; the run records the version id it read

One KMS key per data domain, with separated duties

The project uses one customer managed key, alias/hl-m07-data, for every zone, the secret, the Athena results, the log groups and the streaming resources. A real bank would use one key per domain and sensitivity (for example raw-pii, curated, logs), so that revoking one does not stop every pipeline. The key policy decides who may administer the key and who may use it. A user in the use list can encrypt and decrypt; an administrator can change the policy and schedule deletion, but should not be able to decrypt data.

Turn on encryption everywhere data is written

Walk through each place data lands and set the key explicitly. The stack does all of these, and the audit tool in this lesson checks the S3 ones:

  • Each zone bucket: SSE-KMS with the customer managed key, S3 Bucket Keys on (fewer KMS calls and lower cost), Block Public Access on, TLS-only policy.
  • The Glue security configuration: S3 encryption, CloudWatch Logs encryption and job bookmark encryption, all with the key. Bookmark state is pipeline metadata that names files, so encrypt it.
  • The Athena workgroup: results encrypted with SSE-KMS, and the setting enforced so users cannot switch it off.
  • Secrets Manager, SNS, SQS, DynamoDB and Kinesis: the same key.
  • CloudWatch log groups: the key policy needs a statement for the Logs service principal, otherwise the log group cannot be encrypted.

Give each job its own narrow role

Create one role per function, never one shared "pipeline role". Grant on resources, not on *. The project's Glue role has exactly these permissions:

PermissionResourceWhy
Read objectsraw bucketIt never rewrites evidence
Read and write objectscurated, quarantine, gold bucketsIts three output zones
Read one secrethl-m07/tokenisation-keyThe HMAC key
Encrypt and decryptthe data keyAll zones are SSE-KMS
logs:AssociateKmsKey/aws-glue/* log groupsNeeded when the security configuration encrypts job logs
glue:PublishDataQuality*The action has no resource-level ARN (a documented reason for a nag suppression)
Managed policy AWSGlueServiceRoleservice baselineCatalog, logs and Glue operations; the policy is broad on some resources ⚠ VERIFY its current contents

The ingest Lambda is the opposite shape: write-only to incoming/api/*, no read, no list.

Decide whether the job needs a VPC

A Glue job that only moves S3 data to S3 needs no VPC setup, and AWS's guidance says so explicitly. A VPC network connection is needed when the job must reach a resource inside your VPC (a JDBC database, for example) or when policy demands that data never use a public path. Then:

  • an S3 gateway endpoint in the route tables is required for S3 access from inside the VPC;
  • the job needs a route to KMS (a NAT gateway or a KMS interface endpoint) when a security configuration is attached, otherwise it fails with a KMS timeout;
  • the same pattern applies to Secrets Manager, STS and the Glue service endpoints ⚠ VERIFY the exact list for Glue 5.0 in your Region;
  • the connection's security group needs a self-referencing rule so the job's network interfaces can talk to each other.

The project does not deploy a VPC to keep the lab cheap. The "Infrastructure as code" section shows the design you would add.

Remove personal data in the job, not afterwards

The job replaces the counterparty IBAN with two derived columns: a keyed token for joins and counts, and a masked display form. The raw IBAN column does not exist in curated. Decide column by column using the table below.

TechniqueOutput exampleReversible?Joinable?Use when
Drop the column(nothing)NoNoYou do not need it downstream. This is the best control
MaskGB45**************3821No (information is destroyed)NoHumans need to recognise an account in a report
Keyed token (HMAC-SHA-256)tok_a117af47...Only by guessing with the keyYes, the same input gives the same tokenAnalysts must count or join on the counterparty
Plain hash (SHA-256, no key)9f86d081...Yes in practice: IBANs are structured and enumerable, so an attacker hashes candidatesYesNever for low-entropy identifiers
Encrypt (KMS or envelope)ciphertextYes, by key holdersNot without decryptingAnother system must get the real value back, under control
Vault tokenisation (a lookup service)random tokenOnly through the vaultDepends on designStrict separation, with a service that logs every lookup

Write evidence as you go

Each run writes one audit record last, containing the actor (the caller identity), the source file with its version id, the counts and the reconciliation status. Add the platform's own logs: CloudTrail management events (always), CloudTrail data events for the raw and quarantine buckets (they cost extra, so scope them to those two), Step Functions execution history on Standard workflows, and S3 server access logs to a separate bucket. Keep personal data out of every log: the state machine logs exclude execution data, and the Lambda and Glue code log counts and ids only.

Add discovery as a safety net

Amazon Macie can scan your S3 buckets for personal data you did not plan for: an analyst uploading a spreadsheet to the gold bucket, or a column that should have been dropped. Enable it on raw, quarantine and gold, add a custom data identifier for the invented HLBK IBAN prefix if you want to see how it works, and route findings to EventBridge. A new account gets a 30-day free trial of Macie's bucket monitoring, but sensitive-data discovery jobs are not included in that trial ⚠ VERIFY current terms before enabling.

AWS Console instructions

  1. KMS → Customer managed keys → alias/hl-m07-data → Key policy. Confirm there is one statement for key administrators and one for key users, and that the CloudWatch Logs service statement is present. Key rotation tab: automatic rotation on.
  2. S3 → each hl-m07-* bucket → Properties → Default encryption. It should read SSE-KMS with the key above and Bucket Key: Enabled. Permissions → Block Public Access (all four on) and the bucket policy with the aws:SecureTransport deny.
  3. Glue → Security configurations → hl-m07-glue-security. S3, CloudWatch and job bookmark encryption all show the key.
  4. IAM → Roles → hl-m07-glue-job-role → Permissions. Expand each policy. Use Access Advisor to see which services it actually used in the last runs, and trim what it never touched.
  5. Secrets Manager → hl-m07/tokenisation-key. Encryption key is the customer managed key. There is no rotation schedule, on purpose (see below).
  6. VPC → Endpoints → Create endpoint (only for the optional private design): service com.amazonaws.us-east-1.s3, type Gateway, select the private route tables.
  7. Macie → Get started (optional), then S3 buckets to see the inventory with encryption and public-access status.

AWS CLI

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)

# 1. Is the raw bucket encrypted with the customer managed key and a Bucket Key?
aws s3api get-bucket-encryption --bucket "hl-m07-raw-${ACCOUNT}"

# 2. Block Public Access status (all four values should be true).
aws s3api get-public-access-block --bucket "hl-m07-raw-${ACCOUNT}"

# 3. Who can use the data key, and is rotation on?
aws kms get-key-policy --key-id alias/hl-m07-data --policy-name default --query Policy --output text
aws kms get-key-rotation-status --key-id alias/hl-m07-data

# 4. What does the Glue security configuration encrypt?
aws glue get-security-configuration --name hl-m07-glue-security

# 5. Which permissions does the Glue role really have? (inline and attached)
aws iam list-attached-role-policies --role-name hl-m07-glue-job-role
aws iam list-role-policies --role-name hl-m07-glue-job-role

# 6. Which IAM principals could read the raw bucket? Simulate one analyst-style action.
aws iam simulate-principal-policy \
  --policy-source-arn "arn:aws:iam::${ACCOUNT}:role/hl-m07-analyst-uk" \
  --action-names s3:GetObject \
  --resource-arns "arn:aws:s3:::hl-m07-curated-${ACCOUNT}/transactions/*" \
  --query "EvaluationResults[].EvalDecision"
# Expected after Lab 7.5: "implicitDeny": the analyst has no direct S3 read on curated.

# 7. Was the tokenisation secret ever read by someone unexpected? (CloudTrail, last 90 days)
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=GetSecretValue \
  --query "Events[].{time:EventTime,user:Username}" --output table

Infrastructure as Code

The project stack (labs/m07-transaction-pipeline/infra/cdk) already defines the key, buckets, security configuration, role and secret. The tabs below show the additions for a private network path and a locked-down quarantine bucket.

using Amazon.CDK;
using Amazon.CDK.AWS.EC2;
using Amazon.CDK.AWS.Glue;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.S3;

// Inside a Stack. "vpc" is an existing Vpc with private subnets; "glueRole", "quarantine" are from DataLakeStack.

// 1. Gateway endpoint for S3: no hourly charge, routes S3 traffic privately.
vpc.AddGatewayEndpoint("S3", new GatewayVpcEndpointOptions { Service = GatewayVpcEndpointAwsService.S3 });

// 2. Interface endpoints the job needs when it has a security configuration and a secret.
var endpointSg = new SecurityGroup(this, "EndpointSg", new SecurityGroupProps { Vpc = vpc, AllowAllOutbound = false });
foreach (var (id, service) in new (string, InterfaceVpcEndpointAwsService)[]
         {
             ("Kms", InterfaceVpcEndpointAwsService.KMS),
             ("Secrets", InterfaceVpcEndpointAwsService.SECRETS_MANAGER),
             ("Sts", InterfaceVpcEndpointAwsService.STS),
         })
{
    vpc.AddInterfaceEndpoint(id, new InterfaceVpcEndpointOptions
    {
        Service = service,
        PrivateDnsEnabled = true,
        SecurityGroups = [endpointSg],
    });
}

// 3. Glue NETWORK connection with a self-referencing security group rule.
var jobSg = new SecurityGroup(this, "GlueJobSg", new SecurityGroupProps { Vpc = vpc, AllowAllOutbound = true });
jobSg.AddIngressRule(jobSg, Port.AllTraffic(), "Glue worker to worker");
endpointSg.AddIngressRule(jobSg, Port.Tcp(443), "Glue job to interface endpoints");

var subnet = vpc.PrivateSubnets[0];
_ = new CfnConnection(this, "GlueNetwork", new CfnConnectionProps
{
    CatalogId = Account,
    ConnectionInput = new CfnConnection.ConnectionInputProperty
    {
        Name = "hl-m07-private-network",
        ConnectionType = "NETWORK",
        PhysicalConnectionRequirements = new CfnConnection.PhysicalConnectionRequirementsProperty
        {
            SubnetId = subnet.SubnetId,
            AvailabilityZone = subnet.AvailabilityZone,
            SecurityGroupIdList = [jobSg.SecurityGroupId],
        },
    },
});

// 4. Quarantine holds RAW values: deny everyone except the pipeline role and a break-glass role.
quarantine.AddToResourcePolicy(new PolicyStatement(new PolicyStatementProps
{
    Effect = Effect.DENY,
    Principals = [new AnyPrincipal()],
    Actions = ["s3:GetObject", "s3:ListBucket"],
    Resources = [quarantine.BucketArn, quarantine.ArnForObjects("*")],
    Conditions = new Dictionary<string, object>
    {
        ["ArnNotLike"] = new Dictionary<string, object>
        {
            ["aws:PrincipalArn"] = new[] { glueRole.RoleArn, $"arn:aws:iam::{Account}:role/hl-break-glass-*" },
        },
    },
}));

Attach the connection to the job with Connections = new CfnJob.ConnectionsListProperty { Connections = ["hl-m07-private-network"] } on the CfnJob.

Complete code

PipelineAudit is a small console tool that checks the five zone buckets and the tokenisation secret against the baseline above and exits with code 1 if anything fails, so it can run on a schedule or in a deployment pipeline. It was built against AWS SDK for .NET v4 (AWSSDK.S3 4.1.1, AWSSDK.SecretsManager 4.1.1.2). It only reads configuration; it never reads object data.

PipelineAudit.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="AWSSDK.S3" Version="4.1.1" />
    <PackageReference Include="AWSSDK.SecretsManager" Version="4.1.1.2" />
    <!-- Needed at run time for the IAM Identity Center (SSO) profile learn-sandbox. -->
    <PackageReference Include="AWSSDK.SSO" Version="4.1.0.2" />
    <PackageReference Include="AWSSDK.SSOOIDC" Version="4.1.0.2" />
  </ItemGroup>
</Project>
Program.cs
// dotnet run -- <account-id>      (AWS_PROFILE=learn-sandbox, after "aws sso login")
using System.Net;
using System.Text.Json;
using Amazon.S3;
using Amazon.S3.Model;
using Amazon.SecretsManager;
using Amazon.SecretsManager.Model;

if (args.Length != 1 || args[0].Length != 12)
{
    Console.Error.WriteLine("usage: PipelineAudit <12-digit account id>");
    return 2;
}

var account = args[0];
string[] zones = ["raw", "curated", "quarantine", "gold", "athena-results"];
using var s3 = new AmazonS3Client();
var failures = 0;

foreach (var zone in zones)
{
    var bucket = $"hl-m07-{zone}-{account}";
    var findings = new List<string>();
    try
    {
        // 1. Default encryption must be SSE-KMS with a customer managed key, using S3 Bucket Keys.
        var enc = await s3.GetBucketEncryptionAsync(new GetBucketEncryptionRequest { BucketName = bucket });
        var rule = enc.ServerSideEncryptionConfiguration.ServerSideEncryptionRules.FirstOrDefault();
        var byDefault = rule?.ServerSideEncryptionByDefault;
        if (byDefault?.ServerSideEncryptionAlgorithm != ServerSideEncryptionMethod.AWSKMS)
        {
            findings.Add("default encryption is not SSE-KMS");
        }
        else if (string.IsNullOrEmpty(byDefault.ServerSideEncryptionKeyManagementServiceKeyId))
        {
            findings.Add("SSE-KMS without an explicit key (AWS managed key in use)");
        }

        if (rule?.BucketKeyEnabled != true)
        {
            findings.Add("S3 Bucket Key is off (more KMS calls, higher cost)");
        }

        // 2. All four Block Public Access switches on.
        var pab = (await s3.GetPublicAccessBlockAsync(new GetPublicAccessBlockRequest { BucketName = bucket })).PublicAccessBlockConfiguration;
        if (pab.BlockPublicAcls != true || pab.IgnorePublicAcls != true || pab.BlockPublicPolicy != true || pab.RestrictPublicBuckets != true)
        {
            findings.Add("Block Public Access is not fully on");
        }

        // 3. A bucket policy that denies requests that do not use TLS.
        try
        {
            var policy = (await s3.GetBucketPolicyAsync(new GetBucketPolicyRequest { BucketName = bucket })).Policy;
            using var doc = JsonDocument.Parse(policy);
            var denyInsecure = doc.RootElement.GetProperty("Statement").EnumerateArray().Any(st =>
                st.GetProperty("Effect").GetString() == "Deny" &&
                st.TryGetProperty("Condition", out var c) && c.GetRawText().Contains("aws:SecureTransport", StringComparison.OrdinalIgnoreCase));
            if (!denyInsecure)
            {
                findings.Add("no Deny statement on aws:SecureTransport");
            }
        }
        catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.NotFound)
        {
            findings.Add("no bucket policy at all");
        }

        // 4. Raw and quarantine keep evidence: versioning must be on.
        if (zone is "raw" or "quarantine")
        {
            var versioning = await s3.GetBucketVersioningAsync(new GetBucketVersioningRequest { BucketName = bucket });
            if (versioning.VersioningConfig.Status != VersionStatus.Enabled)
            {
                findings.Add("versioning is not enabled");
            }
        }
    }
    catch (AmazonS3Exception ex)
    {
        findings.Add($"could not inspect: {ex.ErrorCode}");
    }

    Console.WriteLine($"{(findings.Count == 0 ? "PASS" : "FAIL")}  {bucket}");
    foreach (var f in findings)
    {
        Console.WriteLine($"      - {f}");
    }

    failures += findings.Count == 0 ? 0 : 1;
}

// 5. The tokenisation secret must exist and use a customer managed key.
using var secrets = new AmazonSecretsManagerClient();
try
{
    var secret = await secrets.DescribeSecretAsync(new DescribeSecretRequest { SecretId = "hl-m07/tokenisation-key" });
    var ok = !string.IsNullOrEmpty(secret.KmsKeyId);
    Console.WriteLine($"{(ok ? "PASS" : "FAIL")}  secret hl-m07/tokenisation-key (KMS key: {secret.KmsKeyId ?? "aws/secretsmanager default"}, rotation: {(secret.RotationEnabled == true ? "on" : "manual")})");
    failures += ok ? 0 : 1;
}
catch (ResourceNotFoundException)
{
    Console.WriteLine("FAIL  secret hl-m07/tokenisation-key does not exist");
    failures++;
}

Console.WriteLine(failures == 0 ? "All checks passed." : $"{failures} item(s) need attention.");
return failures == 0 ? 0 : 1;

The lab already ships the production-shaped privacy code: src/Harbourline.DataPipeline.Core/Privacy/PiiProtector.cs (C#) and glue/src/harbourline_etl/masking.py (Python) produce identical tokens for the same IBAN, which the test suites assert.

Code explained line by line

  1. zones and the bucket naming — the stack names buckets hl-m07-<zone>-<account id>, so the tool needs only the account id.
  2. GetBucketEncryptionAsync — reads the default encryption rule. An empty key id means S3 would use the AWS managed key, which you cannot restrict with your own key policy, so the tool flags it.
  3. BucketKeyEnabled — S3 Bucket Keys let S3 reuse a bucket-level data key instead of calling KMS for every object, which cuts KMS request cost. A missing flag is a cost finding, not a leak.
  4. GetPublicAccessBlockAsync — the four switches are nullable in SDK v4, so the code compares with != true, which also catches a missing value.
  5. Bucket policy check — the JSON is parsed and the tool looks for a Deny whose condition mentions aws:SecureTransport. A bucket with no policy raises a 404, which the code turns into a finding instead of a crash.
  6. Versioning on raw and quarantine only — these two hold evidence and raw values. Curated and gold are rebuildable from raw.
  7. DescribeSecretAsync — returns metadata and never the secret value, so the tool needs no permission to read the key itself. Only secretsmanager:DescribeSecret is required.
  8. The exit code — 1 if anything failed, so a scheduled job or a deployment gate can stop on it.

Permissions for the tool: s3:GetEncryptionConfiguration, s3:GetBucketPublicAccessBlock, s3:GetBucketPolicy, s3:GetBucketVersioning on the five buckets, and secretsmanager:DescribeSecret on the secret.

Security considerations

⚠ Learning shortcut — not for production

The lab uses one KMS key for every zone, the secret, the logs and the stream, and sets RemovalPolicy.DESTROY with autoDeleteObjects so cdk destroy leaves nothing behind. A bank uses separate keys per domain and sensitivity, retains the keys and the raw and quarantine buckets, adds a permission boundary to every pipeline role, runs the jobs in a private VPC with endpoint policies, enables CloudTrail data events and Macie in a central security account, and sends alarms on key policy changes. The tokenisation key is a single Secrets Manager secret read by the job role; in production the key would be in a hardware-backed service with its own access reviews.

  • Encryption is not access control. A role that can call kms:Decrypt and s3:GetObject reads plaintext. Keep key use lists as short as the S3 permission lists.
  • Do not make the tokenisation key rotate automatically. Tokens are deterministic: the same IBAN must give the same token across runs so counts and joins keep working. Rotating the key changes every token. A rotation is therefore a planned release: create the new key version, re-tokenise the curated zone from raw, switch downstream queries, then retire the old key. Put a version in the token prefix (for example tok1_) so old and new tokens can coexist during the change. This is why the stack suppresses the Secrets Manager rotation rule with a written reason.
  • The quarantine zone is raw data. It exists so an operator can fix and replay rows, so it holds the original IBANs and names. It needs the raw zone's protection, a short retention, and no analyst access.
  • Plain hashes are not tokens. A SHA-256 of an IBAN can be reversed by hashing every plausible IBAN for a bank code. The keyed HMAC makes guessing useless without the key.
  • Never log rows. Log run ids, counts and reason codes. Step Functions IncludeExecutionData = false keeps the payload out of state-machine logs.
  • Text fields from partners are untrusted input. If a report is ever exported to a spreadsheet, a counterparty name starting with =, + or @ can run as a formula. Neutralise text fields on export. Negative amounts legitimately start with -, so apply the rule to text columns only.
  • Least privilege for people too. The person who can change the key policy should not be the person who runs the pipeline.
  • Endpoint policies. A gateway endpoint can carry a policy limiting which buckets are reachable through it. That is a strong control against copying data to someone else's bucket ⚠ VERIFY the policy syntax in the S3 gateway endpoint documentation before using it.

Compliance review required

Tokenised and masked data can still be personal data under UK GDPR if it can be linked back to a person by any means reasonably likely to be used, including by whoever holds the key. Retention periods, erasure handling for raw, quarantine and curated copies, and whether Macie scanning is allowed on a bucket are decisions for your data-protection officer, security and compliance teams. Nothing in this lesson makes a system compliant with GDPR, UK GDPR or PCI DSS.

Banking use case

Harbourline's partner bureau uploads a payroll file at 06:00. The raw file is stored unchanged with its version id. The Glue job (read-only on raw) validates every row, writes the accepted rows with the IBAN replaced by a keyed token and a masked form, and writes the rejected rows with reason codes to quarantine. The finance analyst sees only GBP rows without free-text columns through Lake Formation. The fraud investigator, through a controlled process, can ask the security team to resolve a token: the team holds the key, looks the IBAN up in raw, and the lookup is logged. A quarterly job runs the audit tool and attaches its output to the access review. When a regulator asks, "was the raw account number ever visible to analysts?", the answer is a list of settings and a query over the curated schema, not a statement of intent.

Common mistakes

  • One role for the whole pipeline → a bug or leak in any step gets the powers of every step. Create a role per job.
  • Granting s3:* on * "to unblock the job" → the exception becomes permanent. Use the job's real prefixes, then trim with Access Advisor.
  • Encrypting the data bucket but not the bookmarks, logs or query results → personal data persists unencrypted in the side channels. Walk the whole list.
  • Hashing an identifier and calling it anonymised → it is trivially reversible for low-entropy values.
  • Putting the tokenisation key in an environment variable, a job parameter or the repository → job parameters show in the console and logs. Fetch it from Secrets Manager at run time.
  • Building a VPC for an S3-to-S3 job → extra cost and new failure modes for no gain. Add a network connection only when you have a reason.
  • Forgetting the KMS route in a VPC job → the job hangs and fails with a KMS timeout.
  • Logging a rejected row "for debugging" → you have copied personal data into a log group with a different retention and audience.

Troubleshooting

SymptomLikely causeFix
Glue job in a VPC times out at startNo route to S3 or KMSAdd the S3 gateway endpoint and a KMS endpoint or NAT gateway; check the job's subnet route table
AccessDeniedException from KMS on writeRole not in the key policy, or key disabledAdd kms:GenerateDataKey* and Decrypt for the role; check the key state is Enabled
Job cannot write encrypted CloudWatch logsKey policy lacks the Logs service statementAdd the logs.us-east-1.amazonaws.com principal statement
AccessDenied reading the secretRole lacks GetSecretValue, or lacks key use on the secret's keyGrant both the secret and the KMS permissions
Audit tool: "could not inspect: AccessDenied"Your role lacks one of the read-only S3 configuration permissionsAdd the four S3 read actions on the five buckets
Audit tool: "SSE-KMS without an explicit key"Bucket uses the AWS managed key (alias aws/s3)Set the customer managed key on the bucket's default encryption
Interface endpoint exists but calls still time outPrivate DNS off, or endpoint security group blocks 443 from the jobTurn on private DNS; allow TCP 443 from the job's security group
Macie reports no findings on a file you know has IBANsObject type or sampling settings exclude it, or the IBAN pattern is not coveredCreate a custom data identifier and run a targeted discovery job

Cost and cleanup

Reading this lesson costs nothing. The audit tool costs fractions of a cent in API calls. If you deploy the project stack to run it, the costs are the project's (see the project page). The optional private design adds real money while it exists:

ItemApproximate cost (us-east-1)
S3 gateway endpointNo hourly charge ⚠ VERIFY
Each interface endpointUSD 0.01 per hour per AZ, plus USD 0.01 per GB processed (first tier)
Three interface endpoints in two AZs for one day3 × 2 × 24 × 0.01 = about USD 1.44
NAT gateway (alternative)Hourly charge plus per-GB processing ⚠ VERIFY, usually more than the endpoints for this pattern
Customer managed KMS keyAbout USD 1 per month, prorated; deleted after the waiting period
Macie30-day trial for bucket monitoring ⚠ VERIFY; sensitive-data discovery is billed per GB scanned
# If you created the optional endpoints, delete them the same day.
aws ec2 describe-vpc-endpoints --query "VpcEndpoints[?Tags[?Key=='Project'&&Value=='cea-lab']].VpcEndpointId" --output text
aws ec2 delete-vpc-endpoints --vpc-endpoint-ids vpce-0123456789abcdef0   # replace with your ids
# Disable Macie if you enabled it only for this exercise.
aws macie2 disable-macie
# Anything still tagged for the module?
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab

Exercises

  • 🟢 Guided: Run the audit tool against the deployed project stack (or read its logic) and write down, for each of the five buckets, the four settings it checks and the AWS Console page where you would see the same setting.
  • 🟡 Challenge: Change the token prefix to include a key version (tok1_), update the C# and Python tokenisers, and make the existing tests pass again. Then write the runbook for rotating to key version 2 without breaking joins.
  • 🔴 Live problem: The quarantine bucket was found readable by an analyst group through a broad managed policy, and it contains raw IBANs for the last 14 days. Decide what to do in the first hour, what evidence to preserve, who to tell, and which controls you would add so it cannot recur.

Assignment

Write a one-page pipeline security baseline for your own portfolio repository: the threat table, the KMS key design (how many keys and why), the role list with permissions, the PII technique chosen per column, the logging and retention plan, and the output of the audit tool run against your deployment. Mark every decision that needs sign-off from a data-protection officer.

Solutions

Knowledge check

Interview questions

  • "How would you secure a Glue ETL pipeline that processes customer data?" — KMS customer managed keys with separated duties, a Glue security configuration, one narrow role per job with read-only on raw, TLS-only and non-public buckets, secrets from Secrets Manager, personal data tokenised or dropped in the job, Lake Formation for consumers, CloudTrail and an audit record per run, and Macie as a safety net.
  • "When do Glue jobs need a VPC?" — When they reach VPC resources or policy forbids a public path. Then add an S3 gateway endpoint, a route to KMS and Secrets Manager (endpoints or NAT) and a self-referencing security group.
  • "Mask, tokenise, hash or encrypt?" — Drop if possible; mask for display; keyed token when analysts must join; encrypt when someone must get the real value back under control; never a plain hash of a low-entropy value.
  • "How would you rotate the key used for tokens?" — As a planned re-tokenisation, with versioned tokens so both can coexist, a reconciliation of counts before and after, and a defined retirement date for the old key.

Certification practice

Completion checklist

  • I can name every place the pipeline stores data and the key that protects each
  • I can justify each permission on the Glue job role
  • I can explain when a Glue job needs a VPC connection and which endpoints it then needs
  • I can choose between drop, mask, token, hash and encrypt, and defend the choice
  • I ran or read the audit tool and know what it does not check
  • I deleted any endpoints or Macie settings I created for practice

Key takeaways

  • Secure a pipeline in layers: encryption with separated key duties, one narrow role per job, private paths only where needed, and personal data removed in the job.
  • Encryption is a second lock. Authorisation (IAM, bucket policy, Lake Formation) decides who reads.
  • Use a keyed token, not a plain hash, and treat key rotation as a planned re-tokenisation.
  • The quarantine zone holds raw values, so protect it like the raw zone.
  • Evidence must not become a leak: log ids and counts, keep rows out.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page