S3 data lake design: zones, partitions, naming, lifecycle and encryption
Design Harbourline's S3 data lake with raw/bronze, curated/silver and consumption/gold zones, Hive-style partitions, predictable naming, lifecycle rules and KMS encryption, then deploy it with CDK (C#) or Terraform and land a file from C#.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.S3 4.1.1) · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain the zone model (raw/bronze → curated/silver → consumption/gold, plus quarantine) and who may read each zone.
- Design S3 key layouts with Hive-style partitions (
year=2026/month=11/day=16/) that let Athena and Spark skip data, and avoid the two classic traps: over-partitioning and small files. - Choose one bucket per zone vs one bucket with prefixes, and justify it with security and lifecycle arguments.
- Configure encryption (SSE-KMS with a customer managed key and Bucket Keys), versioning, Block Public Access, TLS-only policies and lifecycle rules for each zone.
- Deploy the lake with CDK (C#) or Terraform, and land a raw file from C# with a checksum, KMS encryption and a no-overwrite condition.
Prerequisites
- Data engineering for .NET developers
- Module 5: Security, especially the S3 protection and KMS lessons
- Tools: AWS CLI v2 with the
learn-sandboxSSO profile, .NET 10 SDK, and either the CDK CLI or Terraform 1.14
Why this matters in banking
A data lake becomes a "data swamp" in about six months if nobody designs it. Typical findings at a bank's internal audit:
- One bucket called
analytics-datawith 400 top-level prefixes, no owner, no classification, and a mix of raw customer extracts and finished reports. Analysts were givens3:GetObjecton the whole bucket "to keep it simple". - Two copies of the same day's transactions in different folders because a job was re-run with a different path, so reports disagree depending on which folder they read.
- Raw files kept for ever "just in case" — including personal data the bank no longer has a lawful reason to retain under UK GDPR's storage-limitation principle.
- Partitioned by
account_id, which produced millions of tiny folders; every Athena query spent most of its time listing objects.
Good lake design prevents all four. It is cheap to get right on day one and very expensive to fix later, because every job and every query depends on the paths.
The idea in plain language
Amazon S3 stores objects under keys inside buckets. There are no real folders: transactions/year=2026/month=11/day=16/part-0000.parquet is a single key, and the slashes are just characters. Tools display prefixes as folders, and query engines interpret name=value segments as partition columns. That convention is called Hive-style partitioning.
A data lake is a set of buckets with rules:
- Zones say how refined and how trusted the data is.
- Key layouts say where each dataset lives and how it is partitioned.
- Policies say who can read and write each zone.
- Lifecycle says how long data lives and in which storage class.
- Encryption says which key protects it, and therefore who can decrypt it.
Analogy: Think of a bank's records building. The post room (raw) receives every envelope and stamps it with the date and sender, but never opens or alters it. The processing floor (curated) works on checked, standardised copies with personal details redacted where possible. The reading room (consumption) holds finished summaries that many staff may consult. A locked exceptions cage (quarantine) holds the envelopes that failed checks, with a note explaining why. Each room has its own key, its own visitor list and its own shredding schedule.
What problem the AWS service solves
S3 gives you effectively unlimited, eleven-nines-durable storage that every AWS analytics service can read: Glue, Athena, Redshift Spectrum, EMR, SageMaker, Quick Sight (through Athena). Storage is separated from compute, so you can keep years of history cheaply and pay for compute only when you query.
What S3 does not give you, and what you must design:
- No schema. S3 does not know that a file is a table. The Glue Data Catalog adds that (see Glue Data Catalog and crawlers).
- No transactions across objects. Writing 50 Parquet files for a day is 50 separate PUTs. A reader can see some of them mid-write. Iceberg tables solve this (see Lake vs warehouse vs lakehouse); for plain files you write to a staging prefix and switch.
- No atomic rename. "Moving" a prefix is copy plus delete, object by object. Design so you never need to rename.
- Request-rate limits per prefix. S3 supports at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, and scales by adding prefixes. Lakes rarely hit this, but very hot single prefixes can see
503 Slow Downduring scale-up. - Cost of many small objects. Every object costs a request to write and to read, and engines pay a fixed overhead per file. Aim for analytical files of roughly 128 MB–1 GB each in curated and gold.
The zone model
| Zone | Also called | Contents | Format | Who writes | Who reads | Typical retention (learning values) |
|---|---|---|---|---|---|---|
| raw | bronze, landing | Source files exactly as received | As received: CSV, JSONL, XML, SFTP batch files | Ingestion only (Transfer Family, API, DMS, Firehose) | Pipeline role only | Agreed with records management; lab: 90 days then expire |
| quarantine | rejects, errors | Rejected rows or whole files plus reason codes | JSONL or CSV with reason_code | Pipeline role | Pipeline + data-operations role | Short: 90 days |
| curated | silver, conformed, trusted | Validated, typed, de-duplicated, masked/tokenised records | Parquet (or Iceberg), partitioned | Pipeline role | Analysts (through Lake Formation), downstream jobs | Years, per data-retention policy |
| consumption | gold, presentation, mart | Aggregates and business views: daily summary, reconciliation, suspicious-activity signals | Parquet / Iceberg | Pipeline (Athena CTAS, Glue) | Business users, Quick Sight | Per report retention |
Some teams add a sandbox zone for analysts' own experiments, with automatic expiry after a few days.
One bucket per zone, or one bucket with prefixes?
| One bucket per zone (Harbourline's choice) | One bucket, prefix per zone | |
|---|---|---|
| Access control | Bucket policy and KMS key per zone: simple, auditable | Prefix conditions in every policy: easy to get wrong |
| Encryption keys | Different CMK per zone: raw key can be denied to analysts outright | One default key; per-prefix keys need explicit headers |
| Lifecycle | Rules per bucket: simple | Prefix filters per rule: workable |
| Blast radius | Misconfiguration affects one zone | Affects everything |
| Bucket count | Grows with zones × environments (default quota is 10,000 general purpose buckets per account ⚠ VERIFY current value) | Fewer buckets |
Harbourline uses one bucket per zone per environment, named hl-m07-<zone>-<account-id>. Bucket names are global, so the account ID makes them unique. In production the pattern would be hl-<env>-lake-<zone>-<account-id>-<region>.
Key layout and partitioning
The canonical layouts used throughout this module:
s3://hl-m07-raw-<account-id>/
transactions/source=sftp/ingest_date=2026-11-16/HLPG_PTN0042_PAYMENTS_20261116_001.csv
transactions/source=api/ingest_date=2026-11-16/batch-01a1c3f2.jsonl
transactions/source=core/ingest_date=2026-11-16/eod_transactions_20261116.csv
s3://hl-m07-quarantine-<account-id>/
transactions/ingest_date=2026-11-16/run_id=<run-id>/rejected.jsonl
s3://hl-m07-curated-<account-id>/
transactions/year=2026/month=11/day=16/part-00000-<run-id>.snappy.parquet
s3://hl-m07-gold-<account-id>/
daily_summary/year=2026/month=11/day=16/part-00000.parquet
reconciliation/year=2026/month=11/day=16/part-00000.parquetDesign rules behind it:
- Raw is partitioned by arrival (
ingest_date), because that is what you know when a file lands. You do not open the file to decide where to put it. - Curated and gold are partitioned by business date (
booking_datesplit intoyear/month/day), because that is how people query: "October's transactions". This matchesCURATED_COLUMNSinlabs/m07-transaction-pipeline/glue/src/harbourline_etl/transactions.py. - Low-cardinality partition keys only. Date is ideal.
account_id(thousands to millions of values) is not: it creates millions of tiny files and slows queries. Use sorting or Iceberg bucketing for high-cardinality columns instead. - Hive-style
name=valuesegments so engines discover partition columns automatically. - Lower-case, no spaces, ASCII only in keys. Avoid characters that need URL encoding.
- Dataset name first (
transactions/,payments/), so each dataset can have its own catalogue table, lifecycle and permissions. - Run ID in curated file names, so you can trace any file to the run that wrote it, and delete exactly one run's output if needed.
year/month/day or dt=2026-11-16?
Both work. Separate year, month, day columns make "all of 2026" and "every November" easy to prune. A single dt string makes ranges like dt BETWEEN '2026-10-28' AND '2026-11-03' simpler and needs fewer partition values. Pick one per dataset and never mix. This module uses year/month/day because the lab's Glue job already writes it.
Lifecycle
S3 Lifecycle moves or deletes objects on a schedule. For a lake, the most useful rules are:
| Rule | Why |
|---|---|
| Abort incomplete multipart uploads after 7 days | Failed large uploads otherwise leave invisible, billed parts |
| Expire noncurrent versions after N days | Versioning protects against mistakes, but old versions cost money |
| Raw: transition to S3 Standard-IA after 30 days, S3 Glacier Flexible Retrieval after 90 days | Raw is rarely read after the first week |
| Quarantine: expire after 90 days | Rejected rows should be resolved quickly, not kept for ever |
| Athena results: expire after 7 days | Query results pile up and may contain personal data |
| Curated/gold: keep in Standard, or use S3 Intelligent-Tiering | Access patterns are unpredictable; Intelligent-Tiering moves cold objects automatically for a small monitoring fee per object |
Prices in us-east-1 (checked 2026-11-18, first 50 TB tier): S3 Standard about USD 0.023 per GB-month, Standard-IA USD 0.0125, Glacier Instant Retrieval USD 0.004, Glacier Flexible Retrieval USD 0.0036, Glacier Deep Archive USD 0.00099. Standard-IA and the Glacier classes have minimum storage durations (30, 90, 90 and 180 days) and Standard-IA has a 128 KB minimum billable object size, so do not transition small or short-lived files.
Retention is a compliance decision
The expiry periods in this lesson are learning values. How long Harbourline must keep transaction records (for example under anti-money-laundering record-keeping rules) and how soon it must delete personal data it no longer needs (UK GDPR storage limitation) are decisions for the records-management and data-protection teams. The pipeline implements their decision; it does not make it.
Encryption
All new S3 objects are encrypted at rest with SSE-S3 by default. For a bank's lake, use SSE-KMS with a customer managed key (CMK) per zone, and turn on S3 Bucket Keys so S3 does not call KMS for every object:
- The raw key's policy allows only the ingestion and pipeline roles to decrypt. An analyst with a mistaken
s3:GetObjectgrant on raw still cannot read the data. - Each CMK costs USD 1 per month plus requests; Bucket Keys reduce request costs dramatically for busy buckets.
- Glue, Athena and Lambda roles need
kms:Decrypt(to read) andkms:GenerateDataKey(to write) on the relevant keys. - Athena query results and Glue job temporary data also need encryption (covered in the Athena lesson).
Encryption in transit: deny any request that is not TLS with an aws:SecureTransport condition in every bucket policy.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| ADLS Gen2 storage account with containers | S3 buckets | Bucket names are global; no storage-account wrapper |
| Hierarchical namespace (real directories, atomic rename) | Flat key space; prefixes | No atomic directory rename; design so you never need one |
| Container per zone (bronze/silver/gold) | Bucket per zone | Same idea; bucket policies and KMS keys give per-zone boundaries |
| Azure RBAC + ACLs on directories | IAM policies, bucket policies, Lake Formation | Prefer Lake Formation for table-level grants rather than path-level IAM for analysts |
| Lifecycle management policies (hot/cool/cold/archive) | S3 Lifecycle (Standard, Standard-IA, Glacier classes, Intelligent-Tiering) | Minimum storage durations differ per class |
| Customer-managed keys in Key Vault | AWS KMS customer managed keys | Key policy is a resource policy that also gates every decrypt |
| Event Grid blob-created events | S3 Event Notifications or EventBridge | Turn on "send to EventBridge" to route events to Step Functions with rules |
Architecture
Security-boundary view:
Step-by-step implementation
Write down the lake contract
Before creating anything, record in your design note: zones, bucket names, dataset prefixes, partition keys per zone, KMS key per zone, readers and writers per zone, lifecycle per zone. The tables above are Harbourline's version. Getting this reviewed by the data owner and security before deployment is cheaper than any migration.
Create the KMS keys and buckets with IaC
Use the CDK or Terraform code in Infrastructure as Code. It creates two CMKs (raw and curated), five buckets (raw, quarantine, curated, gold, athena-results) with Block Public Access, ACLs disabled, versioning on data buckets, SSE-KMS with Bucket Keys, TLS-only policies, lifecycle rules, EventBridge notifications on raw, and the mandatory tags.
Land a file in raw from C#
Run the uploader in Complete code to put the SFTP sample (datasets/samples/sftp/HLPG_PTN0042_PAYMENTS_20261116_001.csv) at transactions/source=sftp/ingest_date=2026-11-16/. It sends a SHA-256 checksum, uses the raw KMS key, adds lineage metadata, and refuses to overwrite an existing key.
Prove the controls
- Upload the same file again: expect
412 Precondition Failed(the no-overwrite condition). - Try an HTTP (non-TLS) request with
--endpoint-url http://s3.us-east-1.amazonaws.com: expectAccessDenied. - Check the object's encryption:
aws s3api head-objectshowsaws:kms, your key ARN andBucketKeyEnabled: true.
Tear it down the same day
Follow Cost and cleanup. Versioned buckets need all versions removed before deletion.
AWS Console instructions
- Console → KMS → Customer managed keys → Create key: symmetric, encrypt and decrypt. Alias
hl-m07-raw. Key administrators: your SSO admin role. Key users: leave empty for now (the pipeline role is added later). Repeat forhl-m07-curated. Add the four mandatory tags. - Console → S3 → Create bucket: name
hl-m07-raw-<account-id>, Regionus-east-1, Object Ownership ACLs disabled, Block all public access on, Bucket Versioning Enable, Default encryption SSE-KMS, choosealias/hl-m07-raw, Bucket Key: Enable. Add tags. - On the new bucket: Properties → Amazon EventBridge → Edit → On.
- Management → Create lifecycle rule: name
raw-tiering, apply to all objects; transition current versions to Standard-IA after 30 days and Glacier Flexible Retrieval after 90 days; expire current versions after 120 days (learning value); permanently delete noncurrent versions after 30 days; delete expired delete markers and incomplete multipart uploads after 7 days. - Permissions → Bucket policy: paste the TLS-only policy from the CLI section.
- Repeat for quarantine, curated, gold and athena-results with the lifecycle values from the table. The CLI and IaC routes are faster and repeatable; use the console once to learn where settings live.
AWS CLI
# Variables used throughout the module
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
RAW=hl-m07-raw-$ACCOUNT
TAGS='TagSet=[{Key=Project,Value=cea-lab},{Key=Lab,Value=m07-transaction-pipeline},{Key=Environment,Value=learn},{Key=Owner,Value=your-name}]'
# 1. A customer managed key for the raw zone, with an alias
RAW_KEY=$(aws kms create-key --description "hl-m07 raw zone" \
--tags TagKey=Project,TagValue=cea-lab TagKey=Lab,TagValue=m07-transaction-pipeline \
TagKey=Environment,TagValue=learn TagKey=Owner,TagValue=your-name \
--query KeyMetadata.Arn --output text)
aws kms create-alias --alias-name alias/hl-m07-raw --target-key-id "$RAW_KEY"
# 2. The bucket (us-east-1 needs no LocationConstraint)
aws s3api create-bucket --bucket "$RAW"
aws s3api put-bucket-tagging --bucket "$RAW" --tagging "$TAGS"
# 3. Block Public Access, ACLs off, versioning
aws s3api put-public-access-block --bucket "$RAW" --public-access-block-configuration \
BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
aws s3api put-bucket-ownership-controls --bucket "$RAW" \
--ownership-controls 'Rules=[{ObjectOwnership=BucketOwnerEnforced}]'
aws s3api put-bucket-versioning --bucket "$RAW" --versioning-configuration Status=Enabled
# 4. Default encryption: SSE-KMS with the CMK and a Bucket Key
aws s3api put-bucket-encryption --bucket "$RAW" --server-side-encryption-configuration \
"{\"Rules\":[{\"ApplyServerSideEncryptionByDefault\":{\"SSEAlgorithm\":\"aws:kms\",\"KMSMasterKeyID\":\"$RAW_KEY\"},\"BucketKeyEnabled\":true}]}"
# 5. TLS-only bucket policy
cat > tls-only.json <<EOF
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "DenyInsecureTransport",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": ["arn:aws:s3:::$RAW", "arn:aws:s3:::$RAW/*"],
"Condition": { "Bool": { "aws:SecureTransport": "false" } }
}]
}
EOF
aws s3api put-bucket-policy --bucket "$RAW" --policy file://tls-only.json
# 6. Lifecycle rules for raw
cat > raw-lifecycle.json <<'EOF'
{
"Rules": [{
"ID": "raw-tiering",
"Status": "Enabled",
"Filter": {},
"Transitions": [
{ "Days": 30, "StorageClass": "STANDARD_IA" },
{ "Days": 90, "StorageClass": "GLACIER" }
],
"Expiration": { "Days": 120 },
"NoncurrentVersionExpiration": { "NoncurrentDays": 30 },
"AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
}]
}
EOF
aws s3api put-bucket-lifecycle-configuration --bucket "$RAW" --lifecycle-configuration file://raw-lifecycle.json
# 7. Send object events to EventBridge (the pipeline trigger in later lessons)
aws s3api put-bucket-notification-configuration --bucket "$RAW" \
--notification-configuration '{"EventBridgeConfiguration":{}}'
# 8. Land the SFTP sample file under its raw key
aws s3 cp datasets/samples/sftp/HLPG_PTN0042_PAYMENTS_20261116_001.csv \
"s3://$RAW/transactions/source=sftp/ingest_date=2026-11-16/HLPG_PTN0042_PAYMENTS_20261116_001.csv" \
--checksum-algorithm SHA256
# 9. Inspect: encryption, key, bucket key
aws s3api head-object --bucket "$RAW" \
--key transactions/source=sftp/ingest_date=2026-11-16/HLPG_PTN0042_PAYMENTS_20261116_001.csv \
--query '{SSE:ServerSideEncryption,Key:SSEKMSKeyId,BucketKey:BucketKeyEnabled}'The GLACIER storage-class name in lifecycle rules means S3 Glacier Flexible Retrieval. Repeat steps 2–6 for the other buckets with their own lifecycle values, or use the IaC below.
Infrastructure as Code
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.S3;
using Constructs;
namespace Harbourline.DataLake.Infra;
public sealed class LakeStack : Stack
{
public Bucket Raw { get; }
public Bucket Quarantine { get; }
public Bucket Curated { get; }
public Bucket Gold { get; }
public Bucket AthenaResults { get; }
public LakeStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
var rawKey = new Key(this, "RawKey", new KeyProps
{
Alias = "alias/hl-m07-raw",
Description = "hl-m07 raw and quarantine zones",
EnableKeyRotation = true,
RemovalPolicy = RemovalPolicy.DESTROY, // learning only: production keys are retained
PendingWindow = Duration.Days(7),
});
var curatedKey = new Key(this, "CuratedKey", new KeyProps
{
Alias = "alias/hl-m07-curated",
Description = "hl-m07 curated, gold and Athena results",
EnableKeyRotation = true,
RemovalPolicy = RemovalPolicy.DESTROY,
PendingWindow = Duration.Days(7),
});
Raw = ZoneBucket("Raw", "raw", rawKey, versioned: true, eventBridge: true, new ILifecycleRule[]
{
new LifecycleRule
{
Id = "raw-tiering",
Transitions = new ITransition[]
{
new Transition { StorageClass = StorageClass.INFREQUENT_ACCESS, TransitionAfter = Duration.Days(30) },
new Transition { StorageClass = StorageClass.GLACIER, TransitionAfter = Duration.Days(90) },
},
Expiration = Duration.Days(120),
NoncurrentVersionExpiration = Duration.Days(30),
AbortIncompleteMultipartUploadAfter = Duration.Days(7),
},
});
Quarantine = ZoneBucket("Quarantine", "quarantine", rawKey, versioned: true, eventBridge: false, new ILifecycleRule[]
{
new LifecycleRule { Id = "quarantine-expiry", Expiration = Duration.Days(90),
NoncurrentVersionExpiration = Duration.Days(7), AbortIncompleteMultipartUploadAfter = Duration.Days(7) },
});
Curated = ZoneBucket("Curated", "curated", curatedKey, versioned: true, eventBridge: false, new ILifecycleRule[]
{
new LifecycleRule { Id = "curated-housekeeping",
NoncurrentVersionExpiration = Duration.Days(30), AbortIncompleteMultipartUploadAfter = Duration.Days(7) },
});
Gold = ZoneBucket("Gold", "gold", curatedKey, versioned: true, eventBridge: false, new ILifecycleRule[]
{
new LifecycleRule { Id = "gold-housekeeping",
NoncurrentVersionExpiration = Duration.Days(30), AbortIncompleteMultipartUploadAfter = Duration.Days(7) },
});
AthenaResults = ZoneBucket("AthenaResults", "athena-results", curatedKey, versioned: false, eventBridge: false, new ILifecycleRule[]
{
new LifecycleRule { Id = "results-expiry", Expiration = Duration.Days(7),
AbortIncompleteMultipartUploadAfter = Duration.Days(1) },
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m07-transaction-pipeline");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "your-name");
}
private Bucket ZoneBucket(string id, string zone, IKey key, bool versioned, bool eventBridge, ILifecycleRule[] rules) =>
new(this, id, new BucketProps
{
BucketName = $"hl-m07-{zone}-{Aws.ACCOUNT_ID}",
BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
ObjectOwnership = ObjectOwnership.BUCKET_OWNER_ENFORCED,
Encryption = BucketEncryption.KMS,
EncryptionKey = key,
BucketKeyEnabled = true,
EnforceSSL = true,
MinimumTLSVersion = 1.2,
Versioned = versioned,
EventBridgeEnabled = eventBridge,
LifecycleRules = rules,
// Learning only: lets `cdk destroy` empty and delete the bucket.
RemovalPolicy = RemovalPolicy.DESTROY,
AutoDeleteObjects = true,
});
}using Amazon.CDK;
using Cdklabs.CdkNag;
using Harbourline.DataLake.Infra;
var app = new App();
_ = new LakeStack(app, "HlM07Lake", new StackProps
{
Env = new Amazon.CDK.Environment { Region = "us-east-1" },
});
Aspects.Of(app).Add(new AwsSolutionsChecks());
app.Synth();cdk-nag will flag the buckets for missing server access logs (AwsSolutions-S1). Either add a dedicated log bucket (production) or suppress with a documented reason for the lab. AutoDeleteObjects deploys a small custom-resource Lambda; that is fine for a sandbox and never appropriate for a bank's raw zone.
Complete code
A small C# console app that lands a file in the raw zone the way an ingestion service should: correct key, KMS encryption, end-to-end checksum, lineage metadata, and no overwrite.
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<RootNamespace>Harbourline.DataLake.RawLander</RootNamespace>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AWSSDK.S3" Version="4.1.1" />
</ItemGroup>
</Project>using System.Net;
using Amazon.S3;
using Amazon.S3.Model;
namespace Harbourline.DataLake.RawLander;
public static class LakeKeys
{
// Raw is partitioned by arrival date and source; the file name is kept exactly as received.
public static string RawKey(string dataset, string source, DateOnly ingestDate, string fileName)
{
if (fileName.Contains('/') || fileName.Contains('\\'))
throw new ArgumentException("File name must not contain path separators", nameof(fileName));
return $"{dataset}/source={source.ToLowerInvariant()}/ingest_date={ingestDate:yyyy-MM-dd}/{fileName}";
}
// Curated is partitioned by business date; the run id makes each file traceable.
public static string CuratedPrefix(string dataset, DateOnly bookingDate) =>
$"{dataset}/year={bookingDate:yyyy}/month={bookingDate:MM}/day={bookingDate:dd}/";
}
public static class Program
{
public static async Task<int> Main(string[] args)
{
if (args.Length != 3)
{
Console.Error.WriteLine("usage: RawLander <bucket> <source: sftp|api|core> <file>");
return 2;
}
var (bucket, source, path) = (args[0], args[1], args[2]);
var ingestDate = DateOnly.FromDateTime(DateTime.UtcNow);
var key = LakeKeys.RawKey("transactions", source, ingestDate, Path.GetFileName(path));
var correlationId = Guid.CreateVersion7().ToString();
// Credentials come from the SSO profile (AWS_PROFILE=learn-sandbox); never from code.
using var s3 = new AmazonS3Client(Amazon.RegionEndpoint.USEast1);
var request = new PutObjectRequest
{
BucketName = bucket,
Key = key,
FilePath = path,
// Fail with 412 if the key already exists: raw files are write-once.
IfNoneMatch = "*",
// S3 verifies this checksum on receipt; the value is stored with the object.
ChecksumAlgorithm = ChecksumAlgorithm.SHA256,
// Bucket default encryption already applies the CMK; we state it explicitly
// so a misconfigured bucket fails loudly instead of using another key.
ServerSideEncryptionMethod = ServerSideEncryptionMethod.AWSKMS,
ServerSideEncryptionKeyManagementServiceKeyId = "alias/hl-m07-raw",
BucketKeyEnabled = true,
};
// Lineage metadata travels with the object (x-amz-meta-*). No personal data here.
request.Metadata.Add("source-system", source);
request.Metadata.Add("correlation-id", correlationId);
request.Metadata.Add("original-file-name", Path.GetFileName(path));
try
{
var response = await s3.PutObjectAsync(request);
Console.WriteLine($"Landed s3://{bucket}/{key}");
Console.WriteLine($" version {response.VersionId}");
Console.WriteLine($" sha256 {response.ChecksumSHA256}");
Console.WriteLine($" kms key {response.ServerSideEncryptionKeyManagementServiceKeyId}");
Console.WriteLine($" corr id {correlationId}");
Console.WriteLine($" curated partition for 2026-11-16 would be: {LakeKeys.CuratedPrefix("transactions", new DateOnly(2026, 11, 16))}");
return 0;
}
catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.PreconditionFailed)
{
Console.Error.WriteLine($"Refused: s3://{bucket}/{key} already exists. Raw files are never overwritten.");
return 3;
}
}
}Run it:
aws sso login --profile learn-sandbox
export AWS_PROFILE=learn-sandbox
dotnet run --project RawLander -- "hl-m07-raw-$ACCOUNT" sftp datasets/samples/sftp/HLPG_PTN0042_PAYMENTS_20261116_001.csv
# run again: expect "Refused: ... already exists"Code explained line by line
LakeKeys.RawKey(...)— key construction lives in one place. Every ingestion path (SFTP, API, core extract) calls it, so layouts never drift. Unit-test it like any other pure function.- The path-separator check — a file name such as
../../curated/x.csvmust not let a partner write outside its prefix. Treat partner file names as untrusted input. ingest_date={ingestDate:yyyy-MM-dd}— raw is partitioned by arrival date in UTC, not by the business date inside the file.CuratedPrefix(...)— business-date partitioning with zero-padded month and day (month=11,day=06). Consistent padding matters:month=6andmonth=06are different partitions.Guid.CreateVersion7()— a time-ordered correlation ID (.NET 9+), matching the UUID v7 convention for IDs in this course.new AmazonS3Client(Amazon.RegionEndpoint.USEast1)— credentials are resolved from the environment (the SSO profile). In Lambda or Glue they come from the execution role.IfNoneMatch = "*"— an S3 conditional write: the PUT succeeds only if no object with that key exists. This makes raw write-once at the API level, even with versioning on.ChecksumAlgorithm = ChecksumAlgorithm.SHA256— the SDK computes a SHA-256 and S3 rejects the upload if the bytes arriving differ. The checksum is stored and can be retrieved later as evidence of integrity.ServerSideEncryptionMethod.AWSKMSand the key alias — states the expected key. Combined with a bucket policy that denies other keys, a wrong configuration fails rather than silently encrypting with a different key.Metadata.Add(...)— user-defined metadata becomesx-amz-meta-*headers. Use it for lineage (source, correlation ID), never for personal data: metadata is visible to anyone withHeadObject.catch (AmazonS3Exception ex) when (... PreconditionFailed)— the 412 is an expected business outcome (duplicate delivery), handled explicitly with a distinct exit code so the orchestrator can treat it as "already landed".
Security considerations
⚠ Learning shortcut — not for production
The CDK and Terraform code sets RemovalPolicy.DESTROY, AutoDeleteObjects and force_destroy = true so you can tear the lab down in one command, and it uses no server access logging. In production, raw and curated buckets are retained on stack deletion, KMS keys are retained, deletion is protected by SCPs and (for records) Object Lock, server access logs or CloudTrail data events are on, and lifecycle values come from the records-management policy.
- Separate keys per zone let you deny raw decryption to everyone except ingestion and pipeline roles, which is a stronger control than path-based IAM.
- Deny wrong keys with a bucket policy statement on
s3:x-amz-server-side-encryption-aws-kms-key-idusingStringNotEqualsIfExists(see the Module 5 S3 lesson for whyIfExistsmatters). - Private access path. Glue and Lambda in private subnets reach S3 through a gateway VPC endpoint; add
aws:SourceVpceconditions for the pipeline role in production. - Partner isolation. Each SFTP partner writes only to its own prefix (
source=sftp/partner=PTN0042/); Transfer Family logical directories enforce this. - Quarantine is as sensitive as raw, because rejected rows are raw rows.
- Don't put personal data in keys or metadata. Keys appear in logs, CloudTrail and inventory reports.
transactions/customer=Jane-Smith/is a data leak.
Banking use case
Harbourline's Data Platform team owns the lake. The agreed design:
- Accounts: one data-lake account per environment, separate from application accounts. Producers write cross-account into raw via bucket policies scoped to their prefix.
- Raw: written by Transfer Family (partner files), the Transactions API's batch exporter, and DMS (SQL Server CDC). Read only by the pipeline role. Expired after the period agreed with records management; a copy of selected record types goes to an Object Lock bucket in a records account.
- Curated: Parquet, partitioned by booking date, IBANs tokenised with a keyed HMAC, counterparty names kept (they are business names in most rows; the DPO reviewed this). Analysts read through Athena with Lake Formation column and row filters.
- Gold: the daily summary, the ledger reconciliation and suspicious-activity signals. Finance reads gold through Quick Sight.
- Every object carries
correlation-idmetadata; every run writes an audit record with counts and sums.
Common mistakes
- Partitioning by a high-cardinality column (
account_id,transaction_id) → millions of tiny files, slow listing, slow queries → partition by date; sort or bucket by account inside files. - Inconsistent partition formats (
month=6vsmonth=06,2026-11-16vs20261116) → duplicate partitions and missed data → generate keys from one function. - One bucket for everything → policies with prefix conditions that nobody can review → bucket (and key) per zone.
- Writing curated output into the raw bucket → raw is no longer "as received" → separate buckets, and deny
PutObjecton raw to the pipeline role. - Transitioning tiny files to Standard-IA → minimum object size and duration charges exceed the savings → only transition large, cold objects; consider Intelligent-Tiering.
- Versioning without noncurrent expiry → silent cost growth on overwritten partitions → always pair versioning with a noncurrent-version rule.
- Forgetting incomplete multipart uploads → invisible billed storage → abort after 7 days on every bucket.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
BucketAlreadyExists | Bucket names are global; someone else owns that name | Keep the account ID in the name |
AccessDenied on PutObject with the KMS key | Caller lacks kms:GenerateDataKey on the CMK | Add the role to the key policy (key users) and to its IAM policy |
AccessDenied on GetObject though IAM allows S3 | Missing kms:Decrypt on the zone key | Grant decrypt on that key; check CloudTrail for the KMS denial |
412 Precondition Failed | Object already exists and IfNoneMatch: * was sent | Expected for duplicates; use a new file name if it is a genuine new delivery |
InvalidRequest mentioning checksum | Checksum header and body disagree | The file changed during upload or a proxy altered it; retry from the original |
| Athena returns no rows for a new partition | Partition not registered in the catalogue | Run a crawler, MSCK REPAIR TABLE, ALTER TABLE ADD PARTITION, or use partition projection (next lessons) |
cdk destroy fails on a bucket | Objects or versions remain and auto-delete is off | Empty all versions first (see cleanup) |
| Lifecycle rule "does nothing" | Rules run asynchronously, roughly once a day | Wait 24–48 hours; check the rule's filter |
503 Slow Down during a big backfill | Request rate on a single prefix scaling up | Spread writes across partitions, retry with back-off (the SDK does this) |
Cost and cleanup
Expected cost for this lesson if you tear down the same day:
- S3 storage and requests for a few KB: effectively USD 0.00.
- Two KMS customer managed keys: USD 1 per key per month, prorated per hour, so a few hours costs about USD 0.01. Scheduled deletion (7-day minimum window) stops further charges ⚠ VERIFY that keys pending deletion are not billed.
- EventBridge: events from S3 to the default bus are free to publish; no rules are created in this lesson.
Total: under USD 0.05. Keep the buckets if you continue to the next lessons the same day; otherwise remove them.
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
# IaC route
cdk destroy HlM07Lake # or: terraform destroy
# CLI route: empty every version, then delete each bucket
for ZONE in raw quarantine curated gold athena-results; do
B=hl-m07-$ZONE-$ACCOUNT
aws s3api head-bucket --bucket "$B" 2>/dev/null || continue
aws s3api delete-objects --bucket "$B" --delete "$(aws s3api list-object-versions --bucket "$B" \
--query '{Objects: Versions[].{Key:Key,VersionId:VersionId}, Quiet: `true`}' --output json)" 2>/dev/null
aws s3api delete-objects --bucket "$B" --delete "$(aws s3api list-object-versions --bucket "$B" \
--query '{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}, Quiet: `true`}' --output json)" 2>/dev/null
aws s3api delete-bucket --bucket "$B"
done
# KMS keys: schedule deletion after removing aliases
for A in hl-m07-raw hl-m07-curated; do
K=$(aws kms describe-key --key-id alias/$A --query KeyMetadata.KeyId --output text 2>/dev/null) || continue
aws kms delete-alias --alias-name alias/$A
aws kms schedule-key-deletion --key-id "$K" --pending-window-in-days 7
done
# Confirm nothing tagged for the lab remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--query "ResourceTagMappingList[].ResourceARN"The delete-objects calls handle up to 1,000 versions each; for larger buckets, repeat or use the IaC destroy.
Exercises
- 🟢 Guided: Deploy the five buckets with CDK or Terraform. Land the SFTP sample with the C# lander, prove the 412 on a second upload, and show
aws:kmsandBucketKeyEnabled: truewithhead-object. - 🟡 Challenge: Add a bucket policy statement to raw that denies
s3:PutObjectunless the request usesalias/hl-m07-raw's key ARN (StringNotEqualsIfExists), and a statement denyings3:PutObjectto any principal except a role namedhl-m07-ingest(ArnNotLikeonaws:PrincipalArn). Prove both with CLI calls. - 🔴 Live problem: An analyst asks for "all transactions for account ACC-1000000003 for the last 3 years, fast". A colleague proposes adding
account_id=as a partition level in curated. Write the reply: what happens to file counts and query performance, and what you would do instead.
Assignment
Add a Lake layout section to your design note: a table of zones, bucket names, KMS keys, readers, writers, partition keys and lifecycle; plus the LakeKeys class with xUnit tests for zero padding, path-traversal rejection and source normalisation. Commit both to your portfolio.
Solutions
Evidence to keep: cdk deploy or terraform apply output listing five buckets; the lander's first output with a version ID, SHA-256 and the key ARN; the second run's Refused: ... already exists (exit code 3); and head-object showing {"SSE": "aws:kms", "Key": "arn:aws:kms:us-east-1:...:key/...", "BucketKey": true}. Why: each item proves one control (layout, integrity, write-once, encryption) instead of asserting it.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DenyWrongKmsKey",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::hl-m07-raw-ACCOUNT/*",
"Condition": { "StringNotEqualsIfExists": {
"s3:x-amz-server-side-encryption-aws-kms-key-id": "arn:aws:kms:us-east-1:ACCOUNT:key/RAW-KEY-ID" } }
},
{
"Sid": "OnlyIngestRoleWrites",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::hl-m07-raw-ACCOUNT/*",
"Condition": { "ArnNotLike": { "aws:PrincipalArn": [
"arn:aws:iam::ACCOUNT:role/hl-m07-ingest",
"arn:aws:iam::ACCOUNT:role/aws-reserved/sso.amazonaws.com/*AdministratorAccess*" ] } }
}
]
}Keep the TLS statement too. Test: aws s3 cp x.csv s3://.../x.csv --sse aws:kms --sse-kms-key-id alias/aws/s3 is denied; the lander (which names the correct key) succeeds only when run as hl-m07-ingest. The key condition compares against the key ARN, not the alias, which is a common surprise. The admin exception stops you locking yourself out in the lab; in production it would be a break-glass role. Why: these statements turn the lake contract into enforced policy.
Adding account_id under day multiplies partitions by the number of active accounts (Harbourline's medium dataset has thousands): each daily partition explodes into thousands of folders holding a few KB each. Writes get slower, the catalogue holds millions of partitions, and most queries (which filter on date, not account) get slower because engines list and open many small files. Instead: keep date partitions; sort by account_id within each file so Parquet row-group statistics let Athena skip row groups; or move curated transactions to an Iceberg table with bucket(16, account_id) partitioning, which groups accounts into a fixed number of buckets. For a "fast single-account history" API, serve it from an operational store (the Accounts API's database or DynamoDB), not from the lake.
Knowledge check
Answer: When a file lands you only know when and from where it arrived; you should not open it to route it. After validation you know each row's business date, and analysts filter by business date, so curated is partitioned that way to enable pruning.
Answer: The key policy becomes a second, independent gate. Even if an IAM or bucket policy mistakenly allows an analyst to read raw objects, they cannot decrypt without kms:Decrypt on the raw key.
Answer: S3 derives a short-lived bucket-level key from the CMK and uses it for many objects, cutting KMS request volume and cost. CloudTrail then shows the bucket ARN as encryption context.
Answer: Abort incomplete multipart uploads (for example after 7 days). If versioned, also expire noncurrent versions.
Answer: High cardinality creates huge numbers of tiny files and partitions, which slows writes, catalogue operations and most queries. Partition by date and sort or bucket by account instead.
Interview questions
- "How would you structure a data lake for a bank?" — Zones as separate buckets (raw, quarantine, curated, gold), a CMK per sensitivity level, Hive-style partitions by business date in curated, raw kept as received and partitioned by arrival, Block Public Access and TLS-only policies, lifecycle per zone driven by records-management policy, Lake Formation for analyst access, and a data-lake account separate from application accounts.
- "How do you stop the lake becoming a swamp?" — Documented layout and naming generated by shared code, a catalogue entry and owner for every dataset, data contracts enforced in the pipeline, quarantine instead of silent drops, lifecycle expiry, and periodic review using S3 Inventory and Storage Lens.
- "How big should Parquet files be?" — Roughly 128 MB to 1 GB for analytical scans; compact small files produced by streaming or frequent micro-batches.
Certification practice
A data engineer stores 3 years of card transactions in S3 as Parquet. Most Athena queries filter on transaction date and sometimes on merchant category (about 600 values). The current layout partitions by customer_id (2 million values), and queries are slow. Which layout improves performance MOST?
A. Partition by customer_id and merchant_category.
B. Partition by year, month, day; keep merchant_category as a regular column.
C. Store all data in a single unpartitioned prefix.
D. Convert the files to CSV so they can be read faster.
Answer: B. Date partitions match the dominant filter and keep cardinality low; Parquet column statistics help with merchant category. A increases cardinality further. C removes pruning. D removes columnar benefits and increases bytes scanned.
Analysts must never read raw files that contain unmasked personal data, even if someone grants them broad S3 permissions by mistake. What provides the STRONGEST safeguard?
A. Tell analysts not to open the raw prefix. B. Encrypt raw objects with SSE-S3. C. Encrypt raw objects with a dedicated customer managed KMS key whose key policy allows decryption only by ingestion and pipeline roles. D. Enable S3 versioning on the raw bucket.
Answer: C. The key policy is an independent control on every read. A is not a control. B: SSE-S3 decryption is transparent to anyone with S3 read permission. D protects against deletion, not reading.
Raw transaction files are read heavily for 7 days, occasionally for 60 days, and must then be kept for 1 year with rare access and retrieval within hours acceptable. Which lifecycle configuration is MOST cost-effective?
A. Keep in S3 Standard for 1 year. B. Transition to S3 Standard-IA after 30 days and to S3 Glacier Flexible Retrieval after 90 days; expire after 365 days. C. Transition to S3 Glacier Deep Archive after 1 day. D. Transition to S3 One Zone-IA immediately.
Answer: B. It matches access patterns and retrieval tolerance. A overpays. C makes the 60-day occasional reads slow and incurs early-deletion charges if objects are removed earlier than the class minimum. D reduces resilience for the only copy of raw evidence and still bills Standard-IA-style retrieval for the hot first week.
Completion checklist
- I can draw the four zones and say who reads and writes each
- I deployed the buckets with Block Public Access, ACLs off, SSE-KMS with Bucket Keys, TLS-only and lifecycle rules
- I landed a file from C# with a checksum and saw the no-overwrite condition refuse a duplicate
- I explained why curated is partitioned by date and not by account
- I emptied and deleted every bucket and scheduled the keys for deletion
Key takeaways
- A lake is buckets plus rules: zones, key layout, policies, lifecycle and encryption.
- Raw is unchanged and partitioned by arrival; curated and gold are typed, masked and partitioned by business date.
- Use low-cardinality Hive-style partitions and reasonably large files; never partition by account or customer.
- A bucket and a CMK per zone give clear, auditable boundaries; Bucket Keys keep KMS affordable.
- Lifecycle rules, including multipart cleanup and noncurrent-version expiry, are part of the design, and retention values come from compliance.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Data engineering for .NET developers
Learn the vocabulary and the core trade-offs of data engineering (batch vs stream, ETL vs ELT, idempotent loads, lineage) by building a tiny end-of-day pipeline in C# over Harbourline's fictional transactions.
File formats: CSV vs JSON vs Parquet, compression and schema evolution
Measure how CSV, JSON Lines and Parquet store the same Harbourline transactions, understand row groups, column statistics and compression codecs, handle schema evolution safely, and see why Parquet makes Athena faster and cheaper.