Skip to Content
Cloud Engineering Academy
7. Data Engineering & ETL Pipelines

S3 data lake design: zones, partitions, naming, lifecycle and encryption

Design Harbourline's S3 data lake with raw/bronze, curated/silver and consumption/gold zones, Hive-style partitions, predictable naming, lifecycle rules and KMS encryption, then deploy it with CDK (C#) or Terraform and land a file from C#.

intermediate⏱ 2 h 🟡 Low costDEA-C01SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.S3 4.1.1) · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain the zone model (raw/bronze → curated/silver → consumption/gold, plus quarantine) and who may read each zone.
  • Design S3 key layouts with Hive-style partitions (year=2026/month=11/day=16/) that let Athena and Spark skip data, and avoid the two classic traps: over-partitioning and small files.
  • Choose one bucket per zone vs one bucket with prefixes, and justify it with security and lifecycle arguments.
  • Configure encryption (SSE-KMS with a customer managed key and Bucket Keys), versioning, Block Public Access, TLS-only policies and lifecycle rules for each zone.
  • Deploy the lake with CDK (C#) or Terraform, and land a raw file from C# with a checksum, KMS encryption and a no-overwrite condition.

Prerequisites

Why this matters in banking

A data lake becomes a "data swamp" in about six months if nobody designs it. Typical findings at a bank's internal audit:

  • One bucket called analytics-data with 400 top-level prefixes, no owner, no classification, and a mix of raw customer extracts and finished reports. Analysts were given s3:GetObject on the whole bucket "to keep it simple".
  • Two copies of the same day's transactions in different folders because a job was re-run with a different path, so reports disagree depending on which folder they read.
  • Raw files kept for ever "just in case" — including personal data the bank no longer has a lawful reason to retain under UK GDPR's storage-limitation principle.
  • Partitioned by account_id, which produced millions of tiny folders; every Athena query spent most of its time listing objects.

Good lake design prevents all four. It is cheap to get right on day one and very expensive to fix later, because every job and every query depends on the paths.

The idea in plain language

Amazon S3 stores objects under keys inside buckets. There are no real folders: transactions/year=2026/month=11/day=16/part-0000.parquet is a single key, and the slashes are just characters. Tools display prefixes as folders, and query engines interpret name=value segments as partition columns. That convention is called Hive-style partitioning.

A data lake is a set of buckets with rules:

  1. Zones say how refined and how trusted the data is.
  2. Key layouts say where each dataset lives and how it is partitioned.
  3. Policies say who can read and write each zone.
  4. Lifecycle says how long data lives and in which storage class.
  5. Encryption says which key protects it, and therefore who can decrypt it.

Analogy: Think of a bank's records building. The post room (raw) receives every envelope and stamps it with the date and sender, but never opens or alters it. The processing floor (curated) works on checked, standardised copies with personal details redacted where possible. The reading room (consumption) holds finished summaries that many staff may consult. A locked exceptions cage (quarantine) holds the envelopes that failed checks, with a note explaining why. Each room has its own key, its own visitor list and its own shredding schedule.

What problem the AWS service solves

S3 gives you effectively unlimited, eleven-nines-durable storage that every AWS analytics service can read: Glue, Athena, Redshift Spectrum, EMR, SageMaker, Quick Sight (through Athena). Storage is separated from compute, so you can keep years of history cheaply and pay for compute only when you query.

What S3 does not give you, and what you must design:

  • No schema. S3 does not know that a file is a table. The Glue Data Catalog adds that (see Glue Data Catalog and crawlers).
  • No transactions across objects. Writing 50 Parquet files for a day is 50 separate PUTs. A reader can see some of them mid-write. Iceberg tables solve this (see Lake vs warehouse vs lakehouse); for plain files you write to a staging prefix and switch.
  • No atomic rename. "Moving" a prefix is copy plus delete, object by object. Design so you never need to rename.
  • Request-rate limits per prefix. S3 supports at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, and scales by adding prefixes. Lakes rarely hit this, but very hot single prefixes can see 503 Slow Down during scale-up.
  • Cost of many small objects. Every object costs a request to write and to read, and engines pay a fixed overhead per file. Aim for analytical files of roughly 128 MB–1 GB each in curated and gold.

The zone model

ZoneAlso calledContentsFormatWho writesWho readsTypical retention (learning values)
rawbronze, landingSource files exactly as receivedAs received: CSV, JSONL, XML, SFTP batch filesIngestion only (Transfer Family, API, DMS, Firehose)Pipeline role onlyAgreed with records management; lab: 90 days then expire
quarantinerejects, errorsRejected rows or whole files plus reason codesJSONL or CSV with reason_codePipeline rolePipeline + data-operations roleShort: 90 days
curatedsilver, conformed, trustedValidated, typed, de-duplicated, masked/tokenised recordsParquet (or Iceberg), partitionedPipeline roleAnalysts (through Lake Formation), downstream jobsYears, per data-retention policy
consumptiongold, presentation, martAggregates and business views: daily summary, reconciliation, suspicious-activity signalsParquet / IcebergPipeline (Athena CTAS, Glue)Business users, Quick SightPer report retention

Some teams add a sandbox zone for analysts' own experiments, with automatic expiry after a few days.

One bucket per zone, or one bucket with prefixes?

One bucket per zone (Harbourline's choice)One bucket, prefix per zone
Access controlBucket policy and KMS key per zone: simple, auditablePrefix conditions in every policy: easy to get wrong
Encryption keysDifferent CMK per zone: raw key can be denied to analysts outrightOne default key; per-prefix keys need explicit headers
LifecycleRules per bucket: simplePrefix filters per rule: workable
Blast radiusMisconfiguration affects one zoneAffects everything
Bucket countGrows with zones × environments (default quota is 10,000 general purpose buckets per account ⚠ VERIFY current value)Fewer buckets

Harbourline uses one bucket per zone per environment, named hl-m07-<zone>-<account-id>. Bucket names are global, so the account ID makes them unique. In production the pattern would be hl-<env>-lake-<zone>-<account-id>-<region>.

Key layout and partitioning

The canonical layouts used throughout this module:

s3://hl-m07-raw-<account-id>/
  transactions/source=sftp/ingest_date=2026-11-16/HLPG_PTN0042_PAYMENTS_20261116_001.csv
  transactions/source=api/ingest_date=2026-11-16/batch-01a1c3f2.jsonl
  transactions/source=core/ingest_date=2026-11-16/eod_transactions_20261116.csv

s3://hl-m07-quarantine-<account-id>/
  transactions/ingest_date=2026-11-16/run_id=<run-id>/rejected.jsonl

s3://hl-m07-curated-<account-id>/
  transactions/year=2026/month=11/day=16/part-00000-<run-id>.snappy.parquet

s3://hl-m07-gold-<account-id>/
  daily_summary/year=2026/month=11/day=16/part-00000.parquet
  reconciliation/year=2026/month=11/day=16/part-00000.parquet

Design rules behind it:

  1. Raw is partitioned by arrival (ingest_date), because that is what you know when a file lands. You do not open the file to decide where to put it.
  2. Curated and gold are partitioned by business date (booking_date split into year/month/day), because that is how people query: "October's transactions". This matches CURATED_COLUMNS in labs/m07-transaction-pipeline/glue/src/harbourline_etl/transactions.py.
  3. Low-cardinality partition keys only. Date is ideal. account_id (thousands to millions of values) is not: it creates millions of tiny files and slows queries. Use sorting or Iceberg bucketing for high-cardinality columns instead.
  4. Hive-style name=value segments so engines discover partition columns automatically.
  5. Lower-case, no spaces, ASCII only in keys. Avoid characters that need URL encoding.
  6. Dataset name first (transactions/, payments/), so each dataset can have its own catalogue table, lifecycle and permissions.
  7. Run ID in curated file names, so you can trace any file to the run that wrote it, and delete exactly one run's output if needed.

year/month/day or dt=2026-11-16?

Both work. Separate year, month, day columns make "all of 2026" and "every November" easy to prune. A single dt string makes ranges like dt BETWEEN '2026-10-28' AND '2026-11-03' simpler and needs fewer partition values. Pick one per dataset and never mix. This module uses year/month/day because the lab's Glue job already writes it.

Lifecycle

S3 Lifecycle moves or deletes objects on a schedule. For a lake, the most useful rules are:

RuleWhy
Abort incomplete multipart uploads after 7 daysFailed large uploads otherwise leave invisible, billed parts
Expire noncurrent versions after N daysVersioning protects against mistakes, but old versions cost money
Raw: transition to S3 Standard-IA after 30 days, S3 Glacier Flexible Retrieval after 90 daysRaw is rarely read after the first week
Quarantine: expire after 90 daysRejected rows should be resolved quickly, not kept for ever
Athena results: expire after 7 daysQuery results pile up and may contain personal data
Curated/gold: keep in Standard, or use S3 Intelligent-TieringAccess patterns are unpredictable; Intelligent-Tiering moves cold objects automatically for a small monitoring fee per object

Prices in us-east-1 (checked 2026-11-18, first 50 TB tier): S3 Standard about USD 0.023 per GB-month, Standard-IA USD 0.0125, Glacier Instant Retrieval USD 0.004, Glacier Flexible Retrieval USD 0.0036, Glacier Deep Archive USD 0.00099. Standard-IA and the Glacier classes have minimum storage durations (30, 90, 90 and 180 days) and Standard-IA has a 128 KB minimum billable object size, so do not transition small or short-lived files.

Retention is a compliance decision

The expiry periods in this lesson are learning values. How long Harbourline must keep transaction records (for example under anti-money-laundering record-keeping rules) and how soon it must delete personal data it no longer needs (UK GDPR storage limitation) are decisions for the records-management and data-protection teams. The pipeline implements their decision; it does not make it.

Encryption

All new S3 objects are encrypted at rest with SSE-S3 by default. For a bank's lake, use SSE-KMS with a customer managed key (CMK) per zone, and turn on S3 Bucket Keys so S3 does not call KMS for every object:

  • The raw key's policy allows only the ingestion and pipeline roles to decrypt. An analyst with a mistaken s3:GetObject grant on raw still cannot read the data.
  • Each CMK costs USD 1 per month plus requests; Bucket Keys reduce request costs dramatically for busy buckets.
  • Glue, Athena and Lambda roles need kms:Decrypt (to read) and kms:GenerateDataKey (to write) on the relevant keys.
  • Athena query results and Glue job temporary data also need encryption (covered in the Athena lesson).

Encryption in transit: deny any request that is not TLS with an aws:SecureTransport condition in every bucket policy.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
ADLS Gen2 storage account with containersS3 bucketsBucket names are global; no storage-account wrapper
Hierarchical namespace (real directories, atomic rename)Flat key space; prefixesNo atomic directory rename; design so you never need one
Container per zone (bronze/silver/gold)Bucket per zoneSame idea; bucket policies and KMS keys give per-zone boundaries
Azure RBAC + ACLs on directoriesIAM policies, bucket policies, Lake FormationPrefer Lake Formation for table-level grants rather than path-level IAM for analysts
Lifecycle management policies (hot/cool/cold/archive)S3 Lifecycle (Standard, Standard-IA, Glacier classes, Intelligent-Tiering)Minimum storage durations differ per class
Customer-managed keys in Key VaultAWS KMS customer managed keysKey policy is a resource policy that also gates every decrypt
Event Grid blob-created eventsS3 Event Notifications or EventBridgeTurn on "send to EventBridge" to route events to Step Functions with rules

Architecture

Security-boundary view:

Step-by-step implementation

Write down the lake contract

Before creating anything, record in your design note: zones, bucket names, dataset prefixes, partition keys per zone, KMS key per zone, readers and writers per zone, lifecycle per zone. The tables above are Harbourline's version. Getting this reviewed by the data owner and security before deployment is cheaper than any migration.

Create the KMS keys and buckets with IaC

Use the CDK or Terraform code in Infrastructure as Code. It creates two CMKs (raw and curated), five buckets (raw, quarantine, curated, gold, athena-results) with Block Public Access, ACLs disabled, versioning on data buckets, SSE-KMS with Bucket Keys, TLS-only policies, lifecycle rules, EventBridge notifications on raw, and the mandatory tags.

Land a file in raw from C#

Run the uploader in Complete code to put the SFTP sample (datasets/samples/sftp/HLPG_PTN0042_PAYMENTS_20261116_001.csv) at transactions/source=sftp/ingest_date=2026-11-16/. It sends a SHA-256 checksum, uses the raw KMS key, adds lineage metadata, and refuses to overwrite an existing key.

Prove the controls

  • Upload the same file again: expect 412 Precondition Failed (the no-overwrite condition).
  • Try an HTTP (non-TLS) request with --endpoint-url http://s3.us-east-1.amazonaws.com: expect AccessDenied.
  • Check the object's encryption: aws s3api head-object shows aws:kms, your key ARN and BucketKeyEnabled: true.

Tear it down the same day

Follow Cost and cleanup. Versioned buckets need all versions removed before deletion.

AWS Console instructions

  1. Console → KMS → Customer managed keys → Create key: symmetric, encrypt and decrypt. Alias hl-m07-raw. Key administrators: your SSO admin role. Key users: leave empty for now (the pipeline role is added later). Repeat for hl-m07-curated. Add the four mandatory tags.
  2. Console → S3 → Create bucket: name hl-m07-raw-<account-id>, Region us-east-1, Object Ownership ACLs disabled, Block all public access on, Bucket Versioning Enable, Default encryption SSE-KMS, choose alias/hl-m07-raw, Bucket Key: Enable. Add tags.
  3. On the new bucket: Properties → Amazon EventBridge → Edit → On.
  4. Management → Create lifecycle rule: name raw-tiering, apply to all objects; transition current versions to Standard-IA after 30 days and Glacier Flexible Retrieval after 90 days; expire current versions after 120 days (learning value); permanently delete noncurrent versions after 30 days; delete expired delete markers and incomplete multipart uploads after 7 days.
  5. Permissions → Bucket policy: paste the TLS-only policy from the CLI section.
  6. Repeat for quarantine, curated, gold and athena-results with the lifecycle values from the table. The CLI and IaC routes are faster and repeatable; use the console once to learn where settings live.

AWS CLI

# Variables used throughout the module
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
RAW=hl-m07-raw-$ACCOUNT
TAGS='TagSet=[{Key=Project,Value=cea-lab},{Key=Lab,Value=m07-transaction-pipeline},{Key=Environment,Value=learn},{Key=Owner,Value=your-name}]'

# 1. A customer managed key for the raw zone, with an alias
RAW_KEY=$(aws kms create-key --description "hl-m07 raw zone" \
  --tags TagKey=Project,TagValue=cea-lab TagKey=Lab,TagValue=m07-transaction-pipeline \
         TagKey=Environment,TagValue=learn TagKey=Owner,TagValue=your-name \
  --query KeyMetadata.Arn --output text)
aws kms create-alias --alias-name alias/hl-m07-raw --target-key-id "$RAW_KEY"

# 2. The bucket (us-east-1 needs no LocationConstraint)
aws s3api create-bucket --bucket "$RAW"
aws s3api put-bucket-tagging --bucket "$RAW" --tagging "$TAGS"

# 3. Block Public Access, ACLs off, versioning
aws s3api put-public-access-block --bucket "$RAW" --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
aws s3api put-bucket-ownership-controls --bucket "$RAW" \
  --ownership-controls 'Rules=[{ObjectOwnership=BucketOwnerEnforced}]'
aws s3api put-bucket-versioning --bucket "$RAW" --versioning-configuration Status=Enabled

# 4. Default encryption: SSE-KMS with the CMK and a Bucket Key
aws s3api put-bucket-encryption --bucket "$RAW" --server-side-encryption-configuration \
  "{\"Rules\":[{\"ApplyServerSideEncryptionByDefault\":{\"SSEAlgorithm\":\"aws:kms\",\"KMSMasterKeyID\":\"$RAW_KEY\"},\"BucketKeyEnabled\":true}]}"

# 5. TLS-only bucket policy
cat > tls-only.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DenyInsecureTransport",
    "Effect": "Deny",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": ["arn:aws:s3:::$RAW", "arn:aws:s3:::$RAW/*"],
    "Condition": { "Bool": { "aws:SecureTransport": "false" } }
  }]
}
EOF
aws s3api put-bucket-policy --bucket "$RAW" --policy file://tls-only.json

# 6. Lifecycle rules for raw
cat > raw-lifecycle.json <<'EOF'
{
  "Rules": [{
    "ID": "raw-tiering",
    "Status": "Enabled",
    "Filter": {},
    "Transitions": [
      { "Days": 30, "StorageClass": "STANDARD_IA" },
      { "Days": 90, "StorageClass": "GLACIER" }
    ],
    "Expiration": { "Days": 120 },
    "NoncurrentVersionExpiration": { "NoncurrentDays": 30 },
    "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
  }]
}
EOF
aws s3api put-bucket-lifecycle-configuration --bucket "$RAW" --lifecycle-configuration file://raw-lifecycle.json

# 7. Send object events to EventBridge (the pipeline trigger in later lessons)
aws s3api put-bucket-notification-configuration --bucket "$RAW" \
  --notification-configuration '{"EventBridgeConfiguration":{}}'

# 8. Land the SFTP sample file under its raw key
aws s3 cp datasets/samples/sftp/HLPG_PTN0042_PAYMENTS_20261116_001.csv \
  "s3://$RAW/transactions/source=sftp/ingest_date=2026-11-16/HLPG_PTN0042_PAYMENTS_20261116_001.csv" \
  --checksum-algorithm SHA256

# 9. Inspect: encryption, key, bucket key
aws s3api head-object --bucket "$RAW" \
  --key transactions/source=sftp/ingest_date=2026-11-16/HLPG_PTN0042_PAYMENTS_20261116_001.csv \
  --query '{SSE:ServerSideEncryption,Key:SSEKMSKeyId,BucketKey:BucketKeyEnabled}'

The GLACIER storage-class name in lifecycle rules means S3 Glacier Flexible Retrieval. Repeat steps 2–6 for the other buckets with their own lifecycle values, or use the IaC below.

Infrastructure as Code

infra/LakeStack.cs
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.S3;
using Constructs;

namespace Harbourline.DataLake.Infra;

public sealed class LakeStack : Stack
{
    public Bucket Raw { get; }
    public Bucket Quarantine { get; }
    public Bucket Curated { get; }
    public Bucket Gold { get; }
    public Bucket AthenaResults { get; }

    public LakeStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        var rawKey = new Key(this, "RawKey", new KeyProps
        {
            Alias = "alias/hl-m07-raw",
            Description = "hl-m07 raw and quarantine zones",
            EnableKeyRotation = true,
            RemovalPolicy = RemovalPolicy.DESTROY, // learning only: production keys are retained
            PendingWindow = Duration.Days(7),
        });
        var curatedKey = new Key(this, "CuratedKey", new KeyProps
        {
            Alias = "alias/hl-m07-curated",
            Description = "hl-m07 curated, gold and Athena results",
            EnableKeyRotation = true,
            RemovalPolicy = RemovalPolicy.DESTROY,
            PendingWindow = Duration.Days(7),
        });

        Raw = ZoneBucket("Raw", "raw", rawKey, versioned: true, eventBridge: true, new ILifecycleRule[]
        {
            new LifecycleRule
            {
                Id = "raw-tiering",
                Transitions = new ITransition[]
                {
                    new Transition { StorageClass = StorageClass.INFREQUENT_ACCESS, TransitionAfter = Duration.Days(30) },
                    new Transition { StorageClass = StorageClass.GLACIER, TransitionAfter = Duration.Days(90) },
                },
                Expiration = Duration.Days(120),
                NoncurrentVersionExpiration = Duration.Days(30),
                AbortIncompleteMultipartUploadAfter = Duration.Days(7),
            },
        });

        Quarantine = ZoneBucket("Quarantine", "quarantine", rawKey, versioned: true, eventBridge: false, new ILifecycleRule[]
        {
            new LifecycleRule { Id = "quarantine-expiry", Expiration = Duration.Days(90),
                NoncurrentVersionExpiration = Duration.Days(7), AbortIncompleteMultipartUploadAfter = Duration.Days(7) },
        });

        Curated = ZoneBucket("Curated", "curated", curatedKey, versioned: true, eventBridge: false, new ILifecycleRule[]
        {
            new LifecycleRule { Id = "curated-housekeeping",
                NoncurrentVersionExpiration = Duration.Days(30), AbortIncompleteMultipartUploadAfter = Duration.Days(7) },
        });

        Gold = ZoneBucket("Gold", "gold", curatedKey, versioned: true, eventBridge: false, new ILifecycleRule[]
        {
            new LifecycleRule { Id = "gold-housekeeping",
                NoncurrentVersionExpiration = Duration.Days(30), AbortIncompleteMultipartUploadAfter = Duration.Days(7) },
        });

        AthenaResults = ZoneBucket("AthenaResults", "athena-results", curatedKey, versioned: false, eventBridge: false, new ILifecycleRule[]
        {
            new LifecycleRule { Id = "results-expiry", Expiration = Duration.Days(7),
                AbortIncompleteMultipartUploadAfter = Duration.Days(1) },
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m07-transaction-pipeline");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        Amazon.CDK.Tags.Of(this).Add("Owner", "your-name");
    }

    private Bucket ZoneBucket(string id, string zone, IKey key, bool versioned, bool eventBridge, ILifecycleRule[] rules) =>
        new(this, id, new BucketProps
        {
            BucketName = $"hl-m07-{zone}-{Aws.ACCOUNT_ID}",
            BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
            ObjectOwnership = ObjectOwnership.BUCKET_OWNER_ENFORCED,
            Encryption = BucketEncryption.KMS,
            EncryptionKey = key,
            BucketKeyEnabled = true,
            EnforceSSL = true,
            MinimumTLSVersion = 1.2,
            Versioned = versioned,
            EventBridgeEnabled = eventBridge,
            LifecycleRules = rules,
            // Learning only: lets `cdk destroy` empty and delete the bucket.
            RemovalPolicy = RemovalPolicy.DESTROY,
            AutoDeleteObjects = true,
        });
}
infra/Program.cs
using Amazon.CDK;
using Cdklabs.CdkNag;
using Harbourline.DataLake.Infra;

var app = new App();
_ = new LakeStack(app, "HlM07Lake", new StackProps
{
    Env = new Amazon.CDK.Environment { Region = "us-east-1" },
});
Aspects.Of(app).Add(new AwsSolutionsChecks());
app.Synth();

cdk-nag will flag the buckets for missing server access logs (AwsSolutions-S1). Either add a dedicated log bucket (production) or suppress with a documented reason for the lab. AutoDeleteObjects deploys a small custom-resource Lambda; that is fine for a sandbox and never appropriate for a bank's raw zone.

Complete code

A small C# console app that lands a file in the raw zone the way an ingestion service should: correct key, KMS encryption, end-to-end checksum, lineage metadata, and no overwrite.

RawLander/RawLander.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <RootNamespace>Harbourline.DataLake.RawLander</RootNamespace>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="AWSSDK.S3" Version="4.1.1" />
  </ItemGroup>
</Project>
RawLander/Program.cs
using System.Net;
using Amazon.S3;
using Amazon.S3.Model;

namespace Harbourline.DataLake.RawLander;

public static class LakeKeys
{
    // Raw is partitioned by arrival date and source; the file name is kept exactly as received.
    public static string RawKey(string dataset, string source, DateOnly ingestDate, string fileName)
    {
        if (fileName.Contains('/') || fileName.Contains('\\'))
            throw new ArgumentException("File name must not contain path separators", nameof(fileName));
        return $"{dataset}/source={source.ToLowerInvariant()}/ingest_date={ingestDate:yyyy-MM-dd}/{fileName}";
    }

    // Curated is partitioned by business date; the run id makes each file traceable.
    public static string CuratedPrefix(string dataset, DateOnly bookingDate) =>
        $"{dataset}/year={bookingDate:yyyy}/month={bookingDate:MM}/day={bookingDate:dd}/";
}

public static class Program
{
    public static async Task<int> Main(string[] args)
    {
        if (args.Length != 3)
        {
            Console.Error.WriteLine("usage: RawLander <bucket> <source: sftp|api|core> <file>");
            return 2;
        }

        var (bucket, source, path) = (args[0], args[1], args[2]);
        var ingestDate = DateOnly.FromDateTime(DateTime.UtcNow);
        var key = LakeKeys.RawKey("transactions", source, ingestDate, Path.GetFileName(path));
        var correlationId = Guid.CreateVersion7().ToString();

        // Credentials come from the SSO profile (AWS_PROFILE=learn-sandbox); never from code.
        using var s3 = new AmazonS3Client(Amazon.RegionEndpoint.USEast1);

        var request = new PutObjectRequest
        {
            BucketName = bucket,
            Key = key,
            FilePath = path,
            // Fail with 412 if the key already exists: raw files are write-once.
            IfNoneMatch = "*",
            // S3 verifies this checksum on receipt; the value is stored with the object.
            ChecksumAlgorithm = ChecksumAlgorithm.SHA256,
            // Bucket default encryption already applies the CMK; we state it explicitly
            // so a misconfigured bucket fails loudly instead of using another key.
            ServerSideEncryptionMethod = ServerSideEncryptionMethod.AWSKMS,
            ServerSideEncryptionKeyManagementServiceKeyId = "alias/hl-m07-raw",
            BucketKeyEnabled = true,
        };
        // Lineage metadata travels with the object (x-amz-meta-*). No personal data here.
        request.Metadata.Add("source-system", source);
        request.Metadata.Add("correlation-id", correlationId);
        request.Metadata.Add("original-file-name", Path.GetFileName(path));

        try
        {
            var response = await s3.PutObjectAsync(request);
            Console.WriteLine($"Landed s3://{bucket}/{key}");
            Console.WriteLine($"  version   {response.VersionId}");
            Console.WriteLine($"  sha256    {response.ChecksumSHA256}");
            Console.WriteLine($"  kms key   {response.ServerSideEncryptionKeyManagementServiceKeyId}");
            Console.WriteLine($"  corr id   {correlationId}");
            Console.WriteLine($"  curated partition for 2026-11-16 would be: {LakeKeys.CuratedPrefix("transactions", new DateOnly(2026, 11, 16))}");
            return 0;
        }
        catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.PreconditionFailed)
        {
            Console.Error.WriteLine($"Refused: s3://{bucket}/{key} already exists. Raw files are never overwritten.");
            return 3;
        }
    }
}

Run it:

aws sso login --profile learn-sandbox
export AWS_PROFILE=learn-sandbox
dotnet run --project RawLander -- "hl-m07-raw-$ACCOUNT" sftp datasets/samples/sftp/HLPG_PTN0042_PAYMENTS_20261116_001.csv
# run again: expect "Refused: ... already exists"

Code explained line by line

  1. LakeKeys.RawKey(...) — key construction lives in one place. Every ingestion path (SFTP, API, core extract) calls it, so layouts never drift. Unit-test it like any other pure function.
  2. The path-separator check — a file name such as ../../curated/x.csv must not let a partner write outside its prefix. Treat partner file names as untrusted input.
  3. ingest_date={ingestDate:yyyy-MM-dd} — raw is partitioned by arrival date in UTC, not by the business date inside the file.
  4. CuratedPrefix(...) — business-date partitioning with zero-padded month and day (month=11, day=06). Consistent padding matters: month=6 and month=06 are different partitions.
  5. Guid.CreateVersion7() — a time-ordered correlation ID (.NET 9+), matching the UUID v7 convention for IDs in this course.
  6. new AmazonS3Client(Amazon.RegionEndpoint.USEast1) — credentials are resolved from the environment (the SSO profile). In Lambda or Glue they come from the execution role.
  7. IfNoneMatch = "*" — an S3 conditional write: the PUT succeeds only if no object with that key exists. This makes raw write-once at the API level, even with versioning on.
  8. ChecksumAlgorithm = ChecksumAlgorithm.SHA256 — the SDK computes a SHA-256 and S3 rejects the upload if the bytes arriving differ. The checksum is stored and can be retrieved later as evidence of integrity.
  9. ServerSideEncryptionMethod.AWSKMS and the key alias — states the expected key. Combined with a bucket policy that denies other keys, a wrong configuration fails rather than silently encrypting with a different key.
  10. Metadata.Add(...) — user-defined metadata becomes x-amz-meta-* headers. Use it for lineage (source, correlation ID), never for personal data: metadata is visible to anyone with HeadObject.
  11. catch (AmazonS3Exception ex) when (... PreconditionFailed) — the 412 is an expected business outcome (duplicate delivery), handled explicitly with a distinct exit code so the orchestrator can treat it as "already landed".

Security considerations

⚠ Learning shortcut — not for production

The CDK and Terraform code sets RemovalPolicy.DESTROY, AutoDeleteObjects and force_destroy = true so you can tear the lab down in one command, and it uses no server access logging. In production, raw and curated buckets are retained on stack deletion, KMS keys are retained, deletion is protected by SCPs and (for records) Object Lock, server access logs or CloudTrail data events are on, and lifecycle values come from the records-management policy.

  • Separate keys per zone let you deny raw decryption to everyone except ingestion and pipeline roles, which is a stronger control than path-based IAM.
  • Deny wrong keys with a bucket policy statement on s3:x-amz-server-side-encryption-aws-kms-key-id using StringNotEqualsIfExists (see the Module 5 S3 lesson for why IfExists matters).
  • Private access path. Glue and Lambda in private subnets reach S3 through a gateway VPC endpoint; add aws:SourceVpce conditions for the pipeline role in production.
  • Partner isolation. Each SFTP partner writes only to its own prefix (source=sftp/partner=PTN0042/); Transfer Family logical directories enforce this.
  • Quarantine is as sensitive as raw, because rejected rows are raw rows.
  • Don't put personal data in keys or metadata. Keys appear in logs, CloudTrail and inventory reports. transactions/customer=Jane-Smith/ is a data leak.

Banking use case

Harbourline's Data Platform team owns the lake. The agreed design:

  • Accounts: one data-lake account per environment, separate from application accounts. Producers write cross-account into raw via bucket policies scoped to their prefix.
  • Raw: written by Transfer Family (partner files), the Transactions API's batch exporter, and DMS (SQL Server CDC). Read only by the pipeline role. Expired after the period agreed with records management; a copy of selected record types goes to an Object Lock bucket in a records account.
  • Curated: Parquet, partitioned by booking date, IBANs tokenised with a keyed HMAC, counterparty names kept (they are business names in most rows; the DPO reviewed this). Analysts read through Athena with Lake Formation column and row filters.
  • Gold: the daily summary, the ledger reconciliation and suspicious-activity signals. Finance reads gold through Quick Sight.
  • Every object carries correlation-id metadata; every run writes an audit record with counts and sums.

Common mistakes

  • Partitioning by a high-cardinality column (account_id, transaction_id) → millions of tiny files, slow listing, slow queries → partition by date; sort or bucket by account inside files.
  • Inconsistent partition formats (month=6 vs month=06, 2026-11-16 vs 20261116) → duplicate partitions and missed data → generate keys from one function.
  • One bucket for everything → policies with prefix conditions that nobody can review → bucket (and key) per zone.
  • Writing curated output into the raw bucket → raw is no longer "as received" → separate buckets, and deny PutObject on raw to the pipeline role.
  • Transitioning tiny files to Standard-IA → minimum object size and duration charges exceed the savings → only transition large, cold objects; consider Intelligent-Tiering.
  • Versioning without noncurrent expiry → silent cost growth on overwritten partitions → always pair versioning with a noncurrent-version rule.
  • Forgetting incomplete multipart uploads → invisible billed storage → abort after 7 days on every bucket.

Troubleshooting

SymptomLikely causeFix
BucketAlreadyExistsBucket names are global; someone else owns that nameKeep the account ID in the name
AccessDenied on PutObject with the KMS keyCaller lacks kms:GenerateDataKey on the CMKAdd the role to the key policy (key users) and to its IAM policy
AccessDenied on GetObject though IAM allows S3Missing kms:Decrypt on the zone keyGrant decrypt on that key; check CloudTrail for the KMS denial
412 Precondition FailedObject already exists and IfNoneMatch: * was sentExpected for duplicates; use a new file name if it is a genuine new delivery
InvalidRequest mentioning checksumChecksum header and body disagreeThe file changed during upload or a proxy altered it; retry from the original
Athena returns no rows for a new partitionPartition not registered in the catalogueRun a crawler, MSCK REPAIR TABLE, ALTER TABLE ADD PARTITION, or use partition projection (next lessons)
cdk destroy fails on a bucketObjects or versions remain and auto-delete is offEmpty all versions first (see cleanup)
Lifecycle rule "does nothing"Rules run asynchronously, roughly once a dayWait 24–48 hours; check the rule's filter
503 Slow Down during a big backfillRequest rate on a single prefix scaling upSpread writes across partitions, retry with back-off (the SDK does this)

Cost and cleanup

Expected cost for this lesson if you tear down the same day:

  • S3 storage and requests for a few KB: effectively USD 0.00.
  • Two KMS customer managed keys: USD 1 per key per month, prorated per hour, so a few hours costs about USD 0.01. Scheduled deletion (7-day minimum window) stops further charges ⚠ VERIFY that keys pending deletion are not billed.
  • EventBridge: events from S3 to the default bus are free to publish; no rules are created in this lesson.

Total: under USD 0.05. Keep the buckets if you continue to the next lessons the same day; otherwise remove them.

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)

# IaC route
cdk destroy HlM07Lake            # or: terraform destroy

# CLI route: empty every version, then delete each bucket
for ZONE in raw quarantine curated gold athena-results; do
  B=hl-m07-$ZONE-$ACCOUNT
  aws s3api head-bucket --bucket "$B" 2>/dev/null || continue
  aws s3api delete-objects --bucket "$B" --delete "$(aws s3api list-object-versions --bucket "$B" \
     --query '{Objects: Versions[].{Key:Key,VersionId:VersionId}, Quiet: `true`}' --output json)" 2>/dev/null
  aws s3api delete-objects --bucket "$B" --delete "$(aws s3api list-object-versions --bucket "$B" \
     --query '{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}, Quiet: `true`}' --output json)" 2>/dev/null
  aws s3api delete-bucket --bucket "$B"
done

# KMS keys: schedule deletion after removing aliases
for A in hl-m07-raw hl-m07-curated; do
  K=$(aws kms describe-key --key-id alias/$A --query KeyMetadata.KeyId --output text 2>/dev/null) || continue
  aws kms delete-alias --alias-name alias/$A
  aws kms schedule-key-deletion --key-id "$K" --pending-window-in-days 7
done

# Confirm nothing tagged for the lab remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --query "ResourceTagMappingList[].ResourceARN"

The delete-objects calls handle up to 1,000 versions each; for larger buckets, repeat or use the IaC destroy.

Exercises

  • 🟢 Guided: Deploy the five buckets with CDK or Terraform. Land the SFTP sample with the C# lander, prove the 412 on a second upload, and show aws:kms and BucketKeyEnabled: true with head-object.
  • 🟡 Challenge: Add a bucket policy statement to raw that denies s3:PutObject unless the request uses alias/hl-m07-raw's key ARN (StringNotEqualsIfExists), and a statement denying s3:PutObject to any principal except a role named hl-m07-ingest (ArnNotLike on aws:PrincipalArn). Prove both with CLI calls.
  • 🔴 Live problem: An analyst asks for "all transactions for account ACC-1000000003 for the last 3 years, fast". A colleague proposes adding account_id= as a partition level in curated. Write the reply: what happens to file counts and query performance, and what you would do instead.

Assignment

Add a Lake layout section to your design note: a table of zones, bucket names, KMS keys, readers, writers, partition keys and lifecycle; plus the LakeKeys class with xUnit tests for zero padding, path-traversal rejection and source normalisation. Commit both to your portfolio.

Solutions

Knowledge check

Interview questions

  • "How would you structure a data lake for a bank?" — Zones as separate buckets (raw, quarantine, curated, gold), a CMK per sensitivity level, Hive-style partitions by business date in curated, raw kept as received and partitioned by arrival, Block Public Access and TLS-only policies, lifecycle per zone driven by records-management policy, Lake Formation for analyst access, and a data-lake account separate from application accounts.
  • "How do you stop the lake becoming a swamp?" — Documented layout and naming generated by shared code, a catalogue entry and owner for every dataset, data contracts enforced in the pipeline, quarantine instead of silent drops, lifecycle expiry, and periodic review using S3 Inventory and Storage Lens.
  • "How big should Parquet files be?" — Roughly 128 MB to 1 GB for analytical scans; compact small files produced by streaming or frequent micro-batches.

Certification practice

Completion checklist

  • I can draw the four zones and say who reads and writes each
  • I deployed the buckets with Block Public Access, ACLs off, SSE-KMS with Bucket Keys, TLS-only and lifecycle rules
  • I landed a file from C# with a checksum and saw the no-overwrite condition refuse a duplicate
  • I explained why curated is partitioned by date and not by account
  • I emptied and deleted every bucket and scheduled the keys for deletion

Key takeaways

  • A lake is buckets plus rules: zones, key layout, policies, lifecycle and encryption.
  • Raw is unchanged and partitioned by arrival; curated and gold are typed, masked and partitioned by business date.
  • Use low-cardinality Hive-style partitions and reasonably large files; never partition by account or customer.
  • A bucket and a CMK per zone give clear, auditable boundaries; Bucket Keys keep KMS affordable.
  • Lifecycle rules, including multipart cleanup and noncurrent-version expiry, are part of the design, and retention values come from compliance.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page