Secure pipeline design: encryption, least privilege, private networking and PII protection
Design a data pipeline that is safe by construction: a KMS key per data domain, one narrow role per job, private network paths, PII tokenised and masked inside the job, and an audit trail you can show a reviewer.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.S3 4.1.1, AWSSDK.SecretsManager 4.1.1.2) · AWS CDK v2 (C#) · AWS Glue 5.0 · AWS KMS · Amazon Macie · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain what must be encrypted in a Glue and S3 pipeline (data in S3, job bookmarks, logs, query results, secrets and streams) and which key protects each.
- Split a pipeline into roles with different powers (ingest, transform, analyst, reporting, administrator) and justify every permission on the Glue job role in the project stack.
- Decide when a Glue job needs a VPC network connection, which VPC endpoints it then needs, and what they cost.
- Choose between masking, tokenising, hashing and encrypting a personal-data column, and explain why a plain SHA-256 of an IBAN is not protection.
- Run a small C# audit tool that checks the pipeline's buckets and secret against the baseline, and list what you would still need from a security review.
Prerequisites
- Module 7 overview, S3 data lake design, PySpark in Glue for beginners
- Validation and data quality and Governance with Lake Formation, because this lesson completes the same story from the producer's side
- Module 5: Security for KMS key policies, IAM and bucket policies
- Tools: AWS CLI v2 (
learn-sandbox), .NET 10 SDK or Docker; the project stack deployed only if you want to run the audit tool for real
Why this matters in banking
Harbourline's pipeline holds three kinds of dangerous data: raw files with full counterparty IBANs and names, a quarantine zone that by design keeps the raw values of the rows that failed, and a tokenisation key that can turn tokens back into IBANs if someone guesses them. A reviewer will ask six questions, and you should be able to answer each with a setting or a line of code, not a promise:
- Who can read raw data, and can an analyst reach it by a side door?
- Does every copy of the data sit encrypted under a key that we control, and who can use that key?
- What happens if a Glue job role is stolen: how much can it read, write and delete?
- Does any of this travel over the public internet?
- Where does personal data stop? Does the curated zone still contain the raw IBAN?
- Can you prove, for one file, who processed it, when and with what result?
A pipeline built only to "work" answers these with "it depends". A pipeline designed to be secure answers them from the infrastructure code.
The idea in plain language
Think of the pipeline as a cash-in-transit route. The van (the Glue job) collects from one place (raw), drives a fixed route and delivers to others (curated, quarantine, gold). A secure design has four layers:
- Locked boxes (encryption). Everything that rests is encrypted, and the keys are held by a separate team from the drivers.
- Named keys for named doors (least privilege). The van's key opens exactly the doors on its route, and the raw door is open for reading only.
- A private road (network). Where policy says traffic must not touch the public internet, the route uses private paths.
- Redaction at the source (data protection). The most sensitive field is replaced by a token before the data leaves the loading bay, so the later zones never hold it.
Analogy: a bank does not make the cashier who counts the money also hold the vault's master key, and it does not leave a customer's full card number on the receipt. Encryption keys, privileges and redaction are the same habits applied to files.
What problem the AWS service solves
No single AWS service "secures a pipeline". The controls are spread over several, and the design job is knowing which setting does what:
| Control | AWS feature | What it protects against | Honest limitation |
|---|---|---|---|
| Encryption at rest | S3 default encryption with SSE-KMS and a customer managed key; Glue security configuration; encrypted CloudWatch log groups, SNS, SQS, DynamoDB, Kinesis | Lost disks, stray access to storage, an over-broad S3 policy that cannot use the key | Anyone allowed to use the key and read the object still reads plaintext. Encryption is a second lock, not authorisation |
| Key separation | KMS key policy with distinct administrators and users; key per domain | One compromised role decrypting everything | A customer managed key costs about USD 1 per month plus API calls ⚠ VERIFY current price |
| Least privilege | IAM roles per function, resource-scoped policies, permission boundaries | A stolen job role used for wider damage | Needs maintenance; wildcards creep back |
| Private networking | S3 gateway endpoint, interface endpoints (KMS, Secrets Manager, STS, Glue), Glue network connection | Traffic over the public internet, data exfiltration to unknown endpoints | Interface endpoints bill per hour per AZ; a mis-built network makes jobs time out |
| PII protection | Tokenise, mask or drop in the job; Lake Formation filters downstream | Analysts and reports seeing raw personal data | Pseudonymised data can still be personal data under UK GDPR |
| Discovery | Amazon Macie | Personal data in places you did not expect | Sampling and per-GB costs; findings are leads, not proof |
| Evidence | CloudTrail, Step Functions history, the run's audit record, S3 access logs | "Prove who did what" | Logs holding PII are a new risk. Log ids and counts, not rows |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Key Vault (keys) with customer-managed keys on storage | AWS KMS customer managed key on S3, Glue, Athena results | Key administration and key use are separate permissions in the key policy, and the key policy is mandatory, not optional |
| Key Vault secrets | AWS Secrets Manager | Encrypted by a KMS key; automatic rotation needs a rotation Lambda |
| Managed identity for a Data Factory pipeline | IAM role assumed by the Glue job | The role is the identity; no stored credential. The role is passed with iam:PassRole |
| Private Endpoint / Private Link on storage | S3 gateway endpoint (free) or interface endpoints (hourly charge) | S3 uses a gateway endpoint in route tables, not a private IP in your subnet |
| Azure Policy "deny public network access" | S3 Block Public Access, SCPs, bucket policy conditions | Account-level and bucket-level switches |
| Purview sensitivity scans | Amazon Macie | Scans S3 only; ties into EventBridge and Security Hub |
| Always Encrypted / dynamic data masking in SQL Server | Tokenise or mask in the pipeline; Lake Formation column and row filters | There is no engine-wide masking function; you design it into the job |
Architecture
Security-boundary view. There are five trust zones. The partner zone can only call one HTTPS route and write under one prefix. The pipeline zone (Glue role, Step Functions) reads raw, writes the other zones and reads one secret. The analyst zone reaches curated data only through Athena and Lake Formation, and has no S3 permission on the zone buckets. The key zone (KMS key policy, Secrets Manager) is administered by a different role from the pipeline. The audit zone (CloudTrail, the audit record) is write-once from the pipeline's point of view: the job can add evidence but not erase it.
Step-by-step implementation
Threat-model the pipeline before configuring anything
List what can go wrong, then map each risk to a control. This table is the one the project uses.
| Threat | Example | Control in the project |
|---|---|---|
| A hostile or broken file | A partner sends an oversized line, a wrong header, or a text field starting with = | Header check rejects the file; row-level reason codes; size limits at the API (1,000 records per request) |
| Over-privileged job | A stolen Glue role deletes raw evidence | Raw is GrantRead only; versioned buckets; no s3:DeleteObject on raw |
| Analyst side door | AmazonS3ReadOnlyAccess attached to an analyst role | Lake Formation filters plus no direct S3 permission (previous lesson) |
| Key abuse | Someone decrypts everything with the data key | Key policy separates administrators from users; CloudTrail alarm on key policy changes |
| Secret leakage | The HMAC key printed in a job log | Key fetched at run time, never logged; logs encrypted; only counts and ids are logged |
| PII in the wrong zone | Raw IBAN in curated or in a Step Functions log | Raw IBAN dropped in the job; execution data excluded from state-machine logs |
| Replay or double processing | The same file uploaded twice | Run id from bucket, key and version id; audit record checked first |
| Silent tampering | A raw file overwritten | Versioning on raw and quarantine; the run records the version id it read |
One KMS key per data domain, with separated duties
The project uses one customer managed key, alias/hl-m07-data, for every zone, the secret, the Athena results, the log groups and the streaming resources. A real bank would use one key per domain and sensitivity (for example raw-pii, curated, logs), so that revoking one does not stop every pipeline. The key policy decides who may administer the key and who may use it. A user in the use list can encrypt and decrypt; an administrator can change the policy and schedule deletion, but should not be able to decrypt data.
Turn on encryption everywhere data is written
Walk through each place data lands and set the key explicitly. The stack does all of these, and the audit tool in this lesson checks the S3 ones:
- Each zone bucket: SSE-KMS with the customer managed key, S3 Bucket Keys on (fewer KMS calls and lower cost), Block Public Access on, TLS-only policy.
- The Glue security configuration: S3 encryption, CloudWatch Logs encryption and job bookmark encryption, all with the key. Bookmark state is pipeline metadata that names files, so encrypt it.
- The Athena workgroup: results encrypted with SSE-KMS, and the setting enforced so users cannot switch it off.
- Secrets Manager, SNS, SQS, DynamoDB and Kinesis: the same key.
- CloudWatch log groups: the key policy needs a statement for the Logs service principal, otherwise the log group cannot be encrypted.
Give each job its own narrow role
Create one role per function, never one shared "pipeline role". Grant on resources, not on *. The project's Glue role has exactly these permissions:
| Permission | Resource | Why |
|---|---|---|
| Read objects | raw bucket | It never rewrites evidence |
| Read and write objects | curated, quarantine, gold buckets | Its three output zones |
| Read one secret | hl-m07/tokenisation-key | The HMAC key |
| Encrypt and decrypt | the data key | All zones are SSE-KMS |
logs:AssociateKmsKey | /aws-glue/* log groups | Needed when the security configuration encrypts job logs |
glue:PublishDataQuality | * | The action has no resource-level ARN (a documented reason for a nag suppression) |
Managed policy AWSGlueServiceRole | service baseline | Catalog, logs and Glue operations; the policy is broad on some resources ⚠ VERIFY its current contents |
The ingest Lambda is the opposite shape: write-only to incoming/api/*, no read, no list.
Decide whether the job needs a VPC
A Glue job that only moves S3 data to S3 needs no VPC setup, and AWS's guidance says so explicitly. A VPC network connection is needed when the job must reach a resource inside your VPC (a JDBC database, for example) or when policy demands that data never use a public path. Then:
- an S3 gateway endpoint in the route tables is required for S3 access from inside the VPC;
- the job needs a route to KMS (a NAT gateway or a KMS interface endpoint) when a security configuration is attached, otherwise it fails with a KMS timeout;
- the same pattern applies to Secrets Manager, STS and the Glue service endpoints ⚠ VERIFY the exact list for Glue 5.0 in your Region;
- the connection's security group needs a self-referencing rule so the job's network interfaces can talk to each other.
The project does not deploy a VPC to keep the lab cheap. The "Infrastructure as code" section shows the design you would add.
Remove personal data in the job, not afterwards
The job replaces the counterparty IBAN with two derived columns: a keyed token for joins and counts, and a masked display form. The raw IBAN column does not exist in curated. Decide column by column using the table below.
| Technique | Output example | Reversible? | Joinable? | Use when |
|---|---|---|---|---|
| Drop the column | (nothing) | No | No | You do not need it downstream. This is the best control |
| Mask | GB45**************3821 | No (information is destroyed) | No | Humans need to recognise an account in a report |
| Keyed token (HMAC-SHA-256) | tok_a117af47... | Only by guessing with the key | Yes, the same input gives the same token | Analysts must count or join on the counterparty |
| Plain hash (SHA-256, no key) | 9f86d081... | Yes in practice: IBANs are structured and enumerable, so an attacker hashes candidates | Yes | Never for low-entropy identifiers |
| Encrypt (KMS or envelope) | ciphertext | Yes, by key holders | Not without decrypting | Another system must get the real value back, under control |
| Vault tokenisation (a lookup service) | random token | Only through the vault | Depends on design | Strict separation, with a service that logs every lookup |
Write evidence as you go
Each run writes one audit record last, containing the actor (the caller identity), the source file with its version id, the counts and the reconciliation status. Add the platform's own logs: CloudTrail management events (always), CloudTrail data events for the raw and quarantine buckets (they cost extra, so scope them to those two), Step Functions execution history on Standard workflows, and S3 server access logs to a separate bucket. Keep personal data out of every log: the state machine logs exclude execution data, and the Lambda and Glue code log counts and ids only.
Add discovery as a safety net
Amazon Macie can scan your S3 buckets for personal data you did not plan for: an analyst uploading a spreadsheet to the gold bucket, or a column that should have been dropped. Enable it on raw, quarantine and gold, add a custom data identifier for the invented HLBK IBAN prefix if you want to see how it works, and route findings to EventBridge. A new account gets a 30-day free trial of Macie's bucket monitoring, but sensitive-data discovery jobs are not included in that trial ⚠ VERIFY current terms before enabling.
AWS Console instructions
- KMS → Customer managed keys →
alias/hl-m07-data→ Key policy. Confirm there is one statement for key administrators and one for key users, and that the CloudWatch Logs service statement is present. Key rotation tab: automatic rotation on. - S3 → each
hl-m07-*bucket → Properties → Default encryption. It should read SSE-KMS with the key above and Bucket Key: Enabled. Permissions → Block Public Access (all four on) and the bucket policy with theaws:SecureTransportdeny. - Glue → Security configurations →
hl-m07-glue-security. S3, CloudWatch and job bookmark encryption all show the key. - IAM → Roles →
hl-m07-glue-job-role→ Permissions. Expand each policy. Use Access Advisor to see which services it actually used in the last runs, and trim what it never touched. - Secrets Manager →
hl-m07/tokenisation-key. Encryption key is the customer managed key. There is no rotation schedule, on purpose (see below). - VPC → Endpoints → Create endpoint (only for the optional private design): service
com.amazonaws.us-east-1.s3, type Gateway, select the private route tables. - Macie → Get started (optional), then S3 buckets to see the inventory with encryption and public-access status.
AWS CLI
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
# 1. Is the raw bucket encrypted with the customer managed key and a Bucket Key?
aws s3api get-bucket-encryption --bucket "hl-m07-raw-${ACCOUNT}"
# 2. Block Public Access status (all four values should be true).
aws s3api get-public-access-block --bucket "hl-m07-raw-${ACCOUNT}"
# 3. Who can use the data key, and is rotation on?
aws kms get-key-policy --key-id alias/hl-m07-data --policy-name default --query Policy --output text
aws kms get-key-rotation-status --key-id alias/hl-m07-data
# 4. What does the Glue security configuration encrypt?
aws glue get-security-configuration --name hl-m07-glue-security
# 5. Which permissions does the Glue role really have? (inline and attached)
aws iam list-attached-role-policies --role-name hl-m07-glue-job-role
aws iam list-role-policies --role-name hl-m07-glue-job-role
# 6. Which IAM principals could read the raw bucket? Simulate one analyst-style action.
aws iam simulate-principal-policy \
--policy-source-arn "arn:aws:iam::${ACCOUNT}:role/hl-m07-analyst-uk" \
--action-names s3:GetObject \
--resource-arns "arn:aws:s3:::hl-m07-curated-${ACCOUNT}/transactions/*" \
--query "EvaluationResults[].EvalDecision"
# Expected after Lab 7.5: "implicitDeny": the analyst has no direct S3 read on curated.
# 7. Was the tokenisation secret ever read by someone unexpected? (CloudTrail, last 90 days)
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=GetSecretValue \
--query "Events[].{time:EventTime,user:Username}" --output tableInfrastructure as Code
The project stack (labs/m07-transaction-pipeline/infra/cdk) already defines the key, buckets, security configuration, role and secret. The tabs below show the additions for a private network path and a locked-down quarantine bucket.
using Amazon.CDK;
using Amazon.CDK.AWS.EC2;
using Amazon.CDK.AWS.Glue;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.S3;
// Inside a Stack. "vpc" is an existing Vpc with private subnets; "glueRole", "quarantine" are from DataLakeStack.
// 1. Gateway endpoint for S3: no hourly charge, routes S3 traffic privately.
vpc.AddGatewayEndpoint("S3", new GatewayVpcEndpointOptions { Service = GatewayVpcEndpointAwsService.S3 });
// 2. Interface endpoints the job needs when it has a security configuration and a secret.
var endpointSg = new SecurityGroup(this, "EndpointSg", new SecurityGroupProps { Vpc = vpc, AllowAllOutbound = false });
foreach (var (id, service) in new (string, InterfaceVpcEndpointAwsService)[]
{
("Kms", InterfaceVpcEndpointAwsService.KMS),
("Secrets", InterfaceVpcEndpointAwsService.SECRETS_MANAGER),
("Sts", InterfaceVpcEndpointAwsService.STS),
})
{
vpc.AddInterfaceEndpoint(id, new InterfaceVpcEndpointOptions
{
Service = service,
PrivateDnsEnabled = true,
SecurityGroups = [endpointSg],
});
}
// 3. Glue NETWORK connection with a self-referencing security group rule.
var jobSg = new SecurityGroup(this, "GlueJobSg", new SecurityGroupProps { Vpc = vpc, AllowAllOutbound = true });
jobSg.AddIngressRule(jobSg, Port.AllTraffic(), "Glue worker to worker");
endpointSg.AddIngressRule(jobSg, Port.Tcp(443), "Glue job to interface endpoints");
var subnet = vpc.PrivateSubnets[0];
_ = new CfnConnection(this, "GlueNetwork", new CfnConnectionProps
{
CatalogId = Account,
ConnectionInput = new CfnConnection.ConnectionInputProperty
{
Name = "hl-m07-private-network",
ConnectionType = "NETWORK",
PhysicalConnectionRequirements = new CfnConnection.PhysicalConnectionRequirementsProperty
{
SubnetId = subnet.SubnetId,
AvailabilityZone = subnet.AvailabilityZone,
SecurityGroupIdList = [jobSg.SecurityGroupId],
},
},
});
// 4. Quarantine holds RAW values: deny everyone except the pipeline role and a break-glass role.
quarantine.AddToResourcePolicy(new PolicyStatement(new PolicyStatementProps
{
Effect = Effect.DENY,
Principals = [new AnyPrincipal()],
Actions = ["s3:GetObject", "s3:ListBucket"],
Resources = [quarantine.BucketArn, quarantine.ArnForObjects("*")],
Conditions = new Dictionary<string, object>
{
["ArnNotLike"] = new Dictionary<string, object>
{
["aws:PrincipalArn"] = new[] { glueRole.RoleArn, $"arn:aws:iam::{Account}:role/hl-break-glass-*" },
},
},
}));Attach the connection to the job with Connections = new CfnJob.ConnectionsListProperty { Connections = ["hl-m07-private-network"] } on the CfnJob.
Complete code
PipelineAudit is a small console tool that checks the five zone buckets and the tokenisation secret against the baseline above and exits with code 1 if anything fails, so it can run on a schedule or in a deployment pipeline. It was built against AWS SDK for .NET v4 (AWSSDK.S3 4.1.1, AWSSDK.SecretsManager 4.1.1.2). It only reads configuration; it never reads object data.
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AWSSDK.S3" Version="4.1.1" />
<PackageReference Include="AWSSDK.SecretsManager" Version="4.1.1.2" />
<!-- Needed at run time for the IAM Identity Center (SSO) profile learn-sandbox. -->
<PackageReference Include="AWSSDK.SSO" Version="4.1.0.2" />
<PackageReference Include="AWSSDK.SSOOIDC" Version="4.1.0.2" />
</ItemGroup>
</Project>// dotnet run -- <account-id> (AWS_PROFILE=learn-sandbox, after "aws sso login")
using System.Net;
using System.Text.Json;
using Amazon.S3;
using Amazon.S3.Model;
using Amazon.SecretsManager;
using Amazon.SecretsManager.Model;
if (args.Length != 1 || args[0].Length != 12)
{
Console.Error.WriteLine("usage: PipelineAudit <12-digit account id>");
return 2;
}
var account = args[0];
string[] zones = ["raw", "curated", "quarantine", "gold", "athena-results"];
using var s3 = new AmazonS3Client();
var failures = 0;
foreach (var zone in zones)
{
var bucket = $"hl-m07-{zone}-{account}";
var findings = new List<string>();
try
{
// 1. Default encryption must be SSE-KMS with a customer managed key, using S3 Bucket Keys.
var enc = await s3.GetBucketEncryptionAsync(new GetBucketEncryptionRequest { BucketName = bucket });
var rule = enc.ServerSideEncryptionConfiguration.ServerSideEncryptionRules.FirstOrDefault();
var byDefault = rule?.ServerSideEncryptionByDefault;
if (byDefault?.ServerSideEncryptionAlgorithm != ServerSideEncryptionMethod.AWSKMS)
{
findings.Add("default encryption is not SSE-KMS");
}
else if (string.IsNullOrEmpty(byDefault.ServerSideEncryptionKeyManagementServiceKeyId))
{
findings.Add("SSE-KMS without an explicit key (AWS managed key in use)");
}
if (rule?.BucketKeyEnabled != true)
{
findings.Add("S3 Bucket Key is off (more KMS calls, higher cost)");
}
// 2. All four Block Public Access switches on.
var pab = (await s3.GetPublicAccessBlockAsync(new GetPublicAccessBlockRequest { BucketName = bucket })).PublicAccessBlockConfiguration;
if (pab.BlockPublicAcls != true || pab.IgnorePublicAcls != true || pab.BlockPublicPolicy != true || pab.RestrictPublicBuckets != true)
{
findings.Add("Block Public Access is not fully on");
}
// 3. A bucket policy that denies requests that do not use TLS.
try
{
var policy = (await s3.GetBucketPolicyAsync(new GetBucketPolicyRequest { BucketName = bucket })).Policy;
using var doc = JsonDocument.Parse(policy);
var denyInsecure = doc.RootElement.GetProperty("Statement").EnumerateArray().Any(st =>
st.GetProperty("Effect").GetString() == "Deny" &&
st.TryGetProperty("Condition", out var c) && c.GetRawText().Contains("aws:SecureTransport", StringComparison.OrdinalIgnoreCase));
if (!denyInsecure)
{
findings.Add("no Deny statement on aws:SecureTransport");
}
}
catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.NotFound)
{
findings.Add("no bucket policy at all");
}
// 4. Raw and quarantine keep evidence: versioning must be on.
if (zone is "raw" or "quarantine")
{
var versioning = await s3.GetBucketVersioningAsync(new GetBucketVersioningRequest { BucketName = bucket });
if (versioning.VersioningConfig.Status != VersionStatus.Enabled)
{
findings.Add("versioning is not enabled");
}
}
}
catch (AmazonS3Exception ex)
{
findings.Add($"could not inspect: {ex.ErrorCode}");
}
Console.WriteLine($"{(findings.Count == 0 ? "PASS" : "FAIL")} {bucket}");
foreach (var f in findings)
{
Console.WriteLine($" - {f}");
}
failures += findings.Count == 0 ? 0 : 1;
}
// 5. The tokenisation secret must exist and use a customer managed key.
using var secrets = new AmazonSecretsManagerClient();
try
{
var secret = await secrets.DescribeSecretAsync(new DescribeSecretRequest { SecretId = "hl-m07/tokenisation-key" });
var ok = !string.IsNullOrEmpty(secret.KmsKeyId);
Console.WriteLine($"{(ok ? "PASS" : "FAIL")} secret hl-m07/tokenisation-key (KMS key: {secret.KmsKeyId ?? "aws/secretsmanager default"}, rotation: {(secret.RotationEnabled == true ? "on" : "manual")})");
failures += ok ? 0 : 1;
}
catch (ResourceNotFoundException)
{
Console.WriteLine("FAIL secret hl-m07/tokenisation-key does not exist");
failures++;
}
Console.WriteLine(failures == 0 ? "All checks passed." : $"{failures} item(s) need attention.");
return failures == 0 ? 0 : 1;The lab already ships the production-shaped privacy code: src/Harbourline.DataPipeline.Core/Privacy/PiiProtector.cs (C#) and glue/src/harbourline_etl/masking.py (Python) produce identical tokens for the same IBAN, which the test suites assert.
Code explained line by line
zonesand the bucket naming — the stack names bucketshl-m07-<zone>-<account id>, so the tool needs only the account id.GetBucketEncryptionAsync— reads the default encryption rule. An empty key id means S3 would use the AWS managed key, which you cannot restrict with your own key policy, so the tool flags it.BucketKeyEnabled— S3 Bucket Keys let S3 reuse a bucket-level data key instead of calling KMS for every object, which cuts KMS request cost. A missing flag is a cost finding, not a leak.GetPublicAccessBlockAsync— the four switches are nullable in SDK v4, so the code compares with!= true, which also catches a missing value.- Bucket policy check — the JSON is parsed and the tool looks for a
Denywhose condition mentionsaws:SecureTransport. A bucket with no policy raises a 404, which the code turns into a finding instead of a crash. - Versioning on raw and quarantine only — these two hold evidence and raw values. Curated and gold are rebuildable from raw.
DescribeSecretAsync— returns metadata and never the secret value, so the tool needs no permission to read the key itself. Onlysecretsmanager:DescribeSecretis required.- The exit code —
1if anything failed, so a scheduled job or a deployment gate can stop on it.
Permissions for the tool: s3:GetEncryptionConfiguration, s3:GetBucketPublicAccessBlock, s3:GetBucketPolicy, s3:GetBucketVersioning on the five buckets, and secretsmanager:DescribeSecret on the secret.
Security considerations
⚠ Learning shortcut — not for production
The lab uses one KMS key for every zone, the secret, the logs and the stream, and sets RemovalPolicy.DESTROY with autoDeleteObjects so cdk destroy leaves nothing behind. A bank uses separate keys per domain and sensitivity, retains the keys and the raw and quarantine buckets, adds a permission boundary to every pipeline role, runs the jobs in a private VPC with endpoint policies, enables CloudTrail data events and Macie in a central security account, and sends alarms on key policy changes. The tokenisation key is a single Secrets Manager secret read by the job role; in production the key would be in a hardware-backed service with its own access reviews.
- Encryption is not access control. A role that can call
kms:Decryptands3:GetObjectreads plaintext. Keep key use lists as short as the S3 permission lists. - Do not make the tokenisation key rotate automatically. Tokens are deterministic: the same IBAN must give the same token across runs so counts and joins keep working. Rotating the key changes every token. A rotation is therefore a planned release: create the new key version, re-tokenise the curated zone from raw, switch downstream queries, then retire the old key. Put a version in the token prefix (for example
tok1_) so old and new tokens can coexist during the change. This is why the stack suppresses the Secrets Manager rotation rule with a written reason. - The quarantine zone is raw data. It exists so an operator can fix and replay rows, so it holds the original IBANs and names. It needs the raw zone's protection, a short retention, and no analyst access.
- Plain hashes are not tokens. A SHA-256 of an IBAN can be reversed by hashing every plausible IBAN for a bank code. The keyed HMAC makes guessing useless without the key.
- Never log rows. Log run ids, counts and reason codes. Step Functions
IncludeExecutionData = falsekeeps the payload out of state-machine logs. - Text fields from partners are untrusted input. If a report is ever exported to a spreadsheet, a counterparty name starting with
=,+or@can run as a formula. Neutralise text fields on export. Negative amounts legitimately start with-, so apply the rule to text columns only. - Least privilege for people too. The person who can change the key policy should not be the person who runs the pipeline.
- Endpoint policies. A gateway endpoint can carry a policy limiting which buckets are reachable through it. That is a strong control against copying data to someone else's bucket ⚠ VERIFY the policy syntax in the S3 gateway endpoint documentation before using it.
Compliance review required
Tokenised and masked data can still be personal data under UK GDPR if it can be linked back to a person by any means reasonably likely to be used, including by whoever holds the key. Retention periods, erasure handling for raw, quarantine and curated copies, and whether Macie scanning is allowed on a bucket are decisions for your data-protection officer, security and compliance teams. Nothing in this lesson makes a system compliant with GDPR, UK GDPR or PCI DSS.
Banking use case
Harbourline's partner bureau uploads a payroll file at 06:00. The raw file is stored unchanged with its version id. The Glue job (read-only on raw) validates every row, writes the accepted rows with the IBAN replaced by a keyed token and a masked form, and writes the rejected rows with reason codes to quarantine. The finance analyst sees only GBP rows without free-text columns through Lake Formation. The fraud investigator, through a controlled process, can ask the security team to resolve a token: the team holds the key, looks the IBAN up in raw, and the lookup is logged. A quarterly job runs the audit tool and attaches its output to the access review. When a regulator asks, "was the raw account number ever visible to analysts?", the answer is a list of settings and a query over the curated schema, not a statement of intent.
Common mistakes
- One role for the whole pipeline → a bug or leak in any step gets the powers of every step. Create a role per job.
- Granting
s3:*on*"to unblock the job" → the exception becomes permanent. Use the job's real prefixes, then trim with Access Advisor. - Encrypting the data bucket but not the bookmarks, logs or query results → personal data persists unencrypted in the side channels. Walk the whole list.
- Hashing an identifier and calling it anonymised → it is trivially reversible for low-entropy values.
- Putting the tokenisation key in an environment variable, a job parameter or the repository → job parameters show in the console and logs. Fetch it from Secrets Manager at run time.
- Building a VPC for an S3-to-S3 job → extra cost and new failure modes for no gain. Add a network connection only when you have a reason.
- Forgetting the KMS route in a VPC job → the job hangs and fails with a KMS timeout.
- Logging a rejected row "for debugging" → you have copied personal data into a log group with a different retention and audience.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Glue job in a VPC times out at start | No route to S3 or KMS | Add the S3 gateway endpoint and a KMS endpoint or NAT gateway; check the job's subnet route table |
AccessDeniedException from KMS on write | Role not in the key policy, or key disabled | Add kms:GenerateDataKey* and Decrypt for the role; check the key state is Enabled |
| Job cannot write encrypted CloudWatch logs | Key policy lacks the Logs service statement | Add the logs.us-east-1.amazonaws.com principal statement |
AccessDenied reading the secret | Role lacks GetSecretValue, or lacks key use on the secret's key | Grant both the secret and the KMS permissions |
| Audit tool: "could not inspect: AccessDenied" | Your role lacks one of the read-only S3 configuration permissions | Add the four S3 read actions on the five buckets |
| Audit tool: "SSE-KMS without an explicit key" | Bucket uses the AWS managed key (alias aws/s3) | Set the customer managed key on the bucket's default encryption |
| Interface endpoint exists but calls still time out | Private DNS off, or endpoint security group blocks 443 from the job | Turn on private DNS; allow TCP 443 from the job's security group |
| Macie reports no findings on a file you know has IBANs | Object type or sampling settings exclude it, or the IBAN pattern is not covered | Create a custom data identifier and run a targeted discovery job |
Cost and cleanup
Reading this lesson costs nothing. The audit tool costs fractions of a cent in API calls. If you deploy the project stack to run it, the costs are the project's (see the project page). The optional private design adds real money while it exists:
| Item | Approximate cost (us-east-1) |
|---|---|
| S3 gateway endpoint | No hourly charge ⚠ VERIFY |
| Each interface endpoint | USD 0.01 per hour per AZ, plus USD 0.01 per GB processed (first tier) |
| Three interface endpoints in two AZs for one day | 3 × 2 × 24 × 0.01 = about USD 1.44 |
| NAT gateway (alternative) | Hourly charge plus per-GB processing ⚠ VERIFY, usually more than the endpoints for this pattern |
| Customer managed KMS key | About USD 1 per month, prorated; deleted after the waiting period |
| Macie | 30-day trial for bucket monitoring ⚠ VERIFY; sensitive-data discovery is billed per GB scanned |
# If you created the optional endpoints, delete them the same day.
aws ec2 describe-vpc-endpoints --query "VpcEndpoints[?Tags[?Key=='Project'&&Value=='cea-lab']].VpcEndpointId" --output text
aws ec2 delete-vpc-endpoints --vpc-endpoint-ids vpce-0123456789abcdef0 # replace with your ids
# Disable Macie if you enabled it only for this exercise.
aws macie2 disable-macie
# Anything still tagged for the module?
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-labExercises
- 🟢 Guided: Run the audit tool against the deployed project stack (or read its logic) and write down, for each of the five buckets, the four settings it checks and the AWS Console page where you would see the same setting.
- 🟡 Challenge: Change the token prefix to include a key version (
tok1_), update the C# and Python tokenisers, and make the existing tests pass again. Then write the runbook for rotating to key version 2 without breaking joins. - 🔴 Live problem: The quarantine bucket was found readable by an analyst group through a broad managed policy, and it contains raw IBANs for the last 14 days. Decide what to do in the first hour, what evidence to preserve, who to tell, and which controls you would add so it cannot recur.
Assignment
Write a one-page pipeline security baseline for your own portfolio repository: the threat table, the KMS key design (how many keys and why), the role list with permissions, the PII technique chosen per column, the logging and retention plan, and the output of the audit tool run against your deployment. Mark every decision that needs sign-off from a data-protection officer.
Solutions
For each bucket the tool checks: (1) default encryption is SSE-KMS with a named customer managed key, (2) the S3 Bucket Key is enabled, (3) all four Block Public Access switches are on, (4) a bucket policy denies requests without TLS; raw and quarantine also need versioning. In the console these are on S3 → bucket → Properties (encryption, versioning) and Permissions (Block Public Access, bucket policy). The reason for each: (1) keeps the key under your control and revocable, (2) controls KMS cost, (3) prevents accidental exposure, (4) prevents plaintext transport, and versioning preserves evidence and lets you identify exactly which version a run read.
In masking.py return prefix + digest[:32] with prefix = "tok1_", and in PiiProtector.Tokenise use prefix = "tok1_". Update the expected token in the C# test (tok_a117af47... becomes tok1_a117af47...) and the Python test that checks startswith("tok_"). Runbook: (1) create the new key version in Secrets Manager and give the job role the new secret id; (2) deploy a job that writes tok2_ tokens; (3) re-run raw files for the retained period into a new curated prefix; (4) switch Athena views and reports to the new table and run the reconciliation view to compare counts per day; (5) keep both versions queryable for an agreed period; (6) retire the old table and, after the retention window, the old key. The version in the prefix lets analysts see at a glance which key produced a token.
First hour: remove the exposure (detach the broad policy or add an explicit deny on the quarantine bucket for the analyst group), without deleting anything. Preserve evidence: S3 server access logs and CloudTrail data events for the bucket (if data events were not on, say so, because that limits what you can prove), the IAM policy version history and the list of group members. Find who read what with the logs and the Access Analyzer findings. Tell the security lead and the data-protection officer immediately, because whether this is a reportable personal-data breach is their decision with a legal clock. Add controls: the explicit deny shown in this lesson, a permission boundary on analyst roles, Macie on the bucket, an alarm on policy changes to it, and a scheduled run of the audit tool. Shorten quarantine retention so there is less to expose.
Knowledge check
Answer: No. A job that does not need VPC resources runs without extra network configuration. A network connection becomes necessary for VPC resources such as a JDBC database or when policy requires a private path, and then it needs an S3 gateway endpoint and a route to KMS.
Answer: IBANs have a known structure and a small set of valid bank codes, so an attacker can compute the hash for every candidate and match. An HMAC with a secret key gives the same join-ability without that weakness, as long as the key stays secret.
Answer: Data written to S3, CloudWatch Logs written by the job, and job bookmarks. It does not encrypt your source data, Athena results or secrets; those are set elsewhere.
Answer: The tokens are deterministic. A new key changes every token, which breaks joins and historical counts. Rotation has to be a planned re-tokenisation from raw, ideally with a key version in the token.
Answer: No. It also holds raw personal data, so it needs narrow read access (a bucket policy limited to the pipeline and a break-glass role), a short retention period, versioning for evidence and monitoring. Encryption does not stop a principal who is allowed to use the key and read the object.
Interview questions
- "How would you secure a Glue ETL pipeline that processes customer data?" — KMS customer managed keys with separated duties, a Glue security configuration, one narrow role per job with read-only on raw, TLS-only and non-public buckets, secrets from Secrets Manager, personal data tokenised or dropped in the job, Lake Formation for consumers, CloudTrail and an audit record per run, and Macie as a safety net.
- "When do Glue jobs need a VPC?" — When they reach VPC resources or policy forbids a public path. Then add an S3 gateway endpoint, a route to KMS and Secrets Manager (endpoints or NAT) and a self-referencing security group.
- "Mask, tokenise, hash or encrypt?" — Drop if possible; mask for display; keyed token when analysts must join; encrypt when someone must get the real value back under control; never a plain hash of a low-entropy value.
- "How would you rotate the key used for tokens?" — As a planned re-tokenisation, with versioned tokens so both can coexist, a reconciliation of counts before and after, and a defined retirement date for the old key.
Certification practice
A Glue ETL job writes Parquet to S3, writes logs to CloudWatch Logs and stores job bookmarks. The security team requires all three to be encrypted with a customer managed KMS key. What is the most direct way?
A. Enable default encryption on the S3 bucket only.
B. Create an AWS Glue security configuration that specifies SSE-KMS for S3, CloudWatch Logs and job bookmarks, and attach it to the job.
C. Pass --encryption-type sse-kms as a job parameter.
D. Run the job inside a VPC.
Answer: B. The security configuration covers all three outputs. A leaves logs and bookmarks. C is not the documented mechanism and the security configuration overrides such settings. D controls networking, not encryption.
A company wants the team that administers a KMS key to be unable to decrypt the data the key protects, while a pipeline role can decrypt. How should the key policy be designed?
A. Give the administrators kms:* on the key.
B. Give administrators management actions only, give the pipeline role Decrypt and GenerateDataKey, and do not grant administrators the usage actions.
C. Use an AWS managed key so nobody can administer it.
D. Share the key policy by IAM user in each account.
Answer: B. Key policies allow separate sets of management and usage permissions. A lets administrators decrypt. C removes your control and key policy. D does not separate duties.
Instances in private subnets with no internet access must read an S3 bucket without traffic leaving the AWS network, at the lowest cost. What should the architect use?
A. A NAT gateway. B. An S3 gateway VPC endpoint and route table entries. C. An interface endpoint for S3 in every Availability Zone. D. A public IP on each instance.
Answer: B. The S3 gateway endpoint has no hourly charge and keeps traffic on the AWS network. A works but costs hourly plus data processing and routes via the internet gateway path. C works but bills per endpoint-hour. D makes the instances public.
Completion checklist
- I can name every place the pipeline stores data and the key that protects each
- I can justify each permission on the Glue job role
- I can explain when a Glue job needs a VPC connection and which endpoints it then needs
- I can choose between drop, mask, token, hash and encrypt, and defend the choice
- I ran or read the audit tool and know what it does not check
- I deleted any endpoints or Macie settings I created for practice
Key takeaways
- Secure a pipeline in layers: encryption with separated key duties, one narrow role per job, private paths only where needed, and personal data removed in the job.
- Encryption is a second lock. Authorisation (IAM, bucket policy, Lake Formation) decides who reads.
- Use a keyed token, not a plain hash, and treat key rotation as a planned re-tokenisation.
- The quarantine zone holds raw values, so protect it like the raw zone.
- Evidence must not become a leak: log ids and counts, keep rows out.
Further reading
- Encrypting data written by AWS Glue
- Setting up network access to data stores (AWS Glue)
- What is Amazon Macie?
- AWS PrivateLink pricing
- Previous lesson: Governance, lineage and cataloguing with Lake Formation
- Next lesson: Reporting with Amazon Quick Sight
- Project: Banking transaction processing pipeline
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Governance, lineage and cataloguing with Lake Formation
Govern Harbourline's data lake: register S3 locations with AWS Lake Formation, replace coarse IAM access with table, column, row and tag-based permissions, give a UK analyst only GBP rows without free-text columns, share safely across accounts, and record ownership and lineage.
Reporting with Amazon Quick Sight: datasets, SPICE and row-level security
Turn the pipeline's gold tables into a reconciliation dashboard: connect Quick Sight to Athena, choose SPICE or direct query, restrict rows per group with a permissions dataset, and know the free alternative when you do not want a paid seat.