Skip to Content
Cloud Engineering Academy
5. Security for Banking & Finance

S3 protection: Block Public Access, bucket policies, Object Lock and Macie

Lock an S3 bucket down so it cannot be made public, accepts only TLS and the bank's KMS key, keeps WORM copies of records with Object Lock, and is scanned for personal data by Amazon Macie.

intermediate⏱ 2 h 🟡 Low costSCS-C03SAA-C03DVA-C03CLF-C02

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.S3 4.1.1, AWSSDK.Macie2 4.1.0.2) · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-17.

Learning objectives

By the end of this lesson you can:

  • Explain the layers that decide whether an S3 object can be read: Block Public Access, Object Ownership, bucket policy, IAM, KMS key policy, VPC endpoint policy, and organisation policies.
  • Turn on Block Public Access at account and organisation level and explain why it overrides everything else.
  • Write a bucket policy that requires TLS, requires the bank's KMS key and restricts access to a VPC endpoint, without breaking default encryption.
  • Choose between SSE-S3, SSE-KMS, DSSE-KMS and (now blocked by default) SSE-C, and use S3 Bucket Keys to cut KMS cost.
  • Use versioning, Object Lock (governance vs compliance, retention, legal hold) and know which settings are irreversible.
  • Find personal data with Amazon Macie using managed and custom data identifiers, and read findings from C#.

Prerequisites

Why this matters in banking

Public S3 buckets have been behind some of the best-known cloud data leaks: a misconfigured bucket with customer records, found by a researcher scanning bucket names. In a bank the impact is regulatory (UK GDPR breach notification within 72 hours to the ICO where required), financial and reputational.

The reverse problem matters too. Regulators and auditors expect certain records (trade confirmations, audit logs, statements) to be kept unaltered for years. "We have backups" is not enough when an insider with admin rights, or ransomware using stolen credentials, can delete both the data and the backups. S3 Object Lock gives you write once, read many (WORM) storage that even the account's root user cannot shorten in compliance mode.

Finally, you cannot protect what you do not know you have. Macie answers the question every data protection officer asks: "Which buckets contain personal data, and how much?"

The idea in plain language

S3 stores objects (files) in buckets. By default only the account that owns the bucket can access it. Data leaks happen when someone adds access: a bucket policy with "Principal": "*", an object ACL that says "public-read", or an over-broad IAM policy.

AWS gives you several independent locks. Any one of them can deny access, and Block Public Access can override the others:

  1. Block Public Access (BPA) — a master switch that refuses public ACLs and public policies, at account, organisation or bucket level.
  2. Object Ownership: bucket owner enforced — turns ACLs off completely, so permissions come only from policies.
  3. Bucket policy — a resource policy on the bucket, for example "deny anything not over TLS".
  4. IAM policies — what each role may do.
  5. KMS key policy — if objects are encrypted with a customer managed key, readers also need permission to use the key.
  6. Object Lock — even someone with full access cannot delete or overwrite a locked object version until its retention date.

Analogy: The bucket is a safe-deposit room. Block Public Access is the rule "this room never has a street door", whatever the branch manager says. The bucket policy is the guard at the room's door checking that you arrived by the secure corridor (TLS) and carry the right key ring (KMS key). Object Lock is the time-lock on individual boxes: once set, not even the bank manager can open the box early. Macie is the auditor who walks the room and lists which boxes contain passports and card details.

What problem the AWS services solve

Defaults you already get (and their dates)

DefaultSinceWhat it means
Block Public Access on and ACLs disabled for new bucketsApril 2023New buckets cannot be made public by accident; ACLs are ignored
Default encryption with SSE-S3 for all new objectsJanuary 2023Every new object is encrypted at rest, at no cost
SSE-C disabled for new write requests on new general purpose bucketsApril 2026Customer-provided keys must be deliberately re-enabled with PutBucketEncryption; existing buckets in accounts with no SSE-C objects were also switched

Defaults are not controls. An administrator can still switch BPA off on a bucket, and a later change can still add a broad policy. Banks therefore enforce BPA centrally and detect drift.

Block Public Access levels

LevelHowWho can override
Bucketput-public-access-block on the bucketAnyone with s3:PutBucketPublicAccessBlock
Accounts3control put-public-access-blockAnyone with s3:PutAccountPublicAccessBlock in that account
OrganisationAn AWS Organizations S3 policy (launched November 2025) attached to the root, an OU or accountsOnly the management account / delegated policy administrators; member accounts cannot change their account-level setting while the policy applies

The four settings are BlockPublicAcls, IgnorePublicAcls, BlockPublicPolicy and RestrictPublicBuckets. Turn on all four unless an account is a deliberate, reviewed exception (for example a static website account behind CloudFront, which should use origin access control and still needs no public bucket).

Server-side encryption options

OptionKey held byWho can audit key useCostBank use
SSE-S3S3No per-request key auditFreeNon-sensitive data, logs where separation is not needed
SSE-KMS (AWS managed key aws/s3)KMS, AWS-managed policyCloudTrail shows KMS callsKMS request chargesRarely; you cannot restrict the key policy
SSE-KMS (customer managed key)KMS, your key policyCloudTrail + your key policy decides who decryptsUSD 1/key/month + requests (reduce with Bucket Keys)Default for Confidential and Restricted data
DSSE-KMSTwo layers of encryption with KMSAs aboveHigherWhere a policy explicitly requires dual-layer encryption
SSE-CYou send the key on every requestYour own systemFree in S3, expensive operationallyAvoid; now blocked by default on new buckets

S3 Bucket Keys let S3 use a short-lived bucket-level key derived from your KMS key, instead of calling KMS for every object. This reduces KMS request traffic (and cost) dramatically for busy buckets. CloudTrail then shows the bucket ARN, not the object ARN, as the encryption context, which matters if you wrote key policies that check per-object context.

Versioning and Object Lock

  • Versioning keeps every version of an object; a delete adds a delete marker. It protects against overwrite and accidental deletion, but someone with s3:DeleteObjectVersion can still remove versions.
  • MFA delete requires the root user's MFA code to permanently delete versions or change versioning. It can only be configured by the root user via the CLI/API, which makes it awkward to operate; most banks prefer Object Lock plus separate backup accounts.
  • Object Lock (requires versioning) puts a retention period or legal hold on each object version:
Governance modeCompliance modeLegal hold
Who can delete/overwrite the version earlyPrincipals with s3:BypassGovernanceRetention who send x-amz-bypass-governance-retention: trueNobody, including root. Only closing the AWS account deletes itAnyone with s3:PutObjectLegalHold can remove the hold
Can the retention be shortenedYes, by bypass holdersNo; it can only be extendedNo end date at all
Typical useTesting, internal policies, ransomware protection with a break-glass roleRegulatory WORM records (for example records that a regulation says must be unalterable)Litigation or investigation: "keep until we say otherwise"

You can set a default retention on the bucket, set retention per object version at upload, use variable retention (an event hold whose clock starts when you release it, for example "7 years after account closure"), and cap allowed retention values with the s3:object-lock-remaining-retention-days condition key. S3 Object Lock has been assessed by Cohasset Associates against SEC 17a-4, CFTC and FINRA rules; that assessment does not mean your use of it meets UK or EU record-keeping rules — your compliance team decides.

Irreversible: compliance mode in a lab account

An object version locked in compliance mode cannot be deleted by anyone until the retain-until date, and you pay to store it until then. A 10-year compliance lock in a sandbox account is a 10-year bill. In this course always use governance mode with a retention of one day or less.

Amazon Macie

Macie discovers sensitive data in S3 using managed data identifiers (built-in detectors) and custom data identifiers (your regular expressions with optional keywords and match distance). It also inventories buckets and raises policy findings when a bucket becomes public, unencrypted or shared outside the account.

Two ways to scan:

  • Automated sensitive data discovery — continuous sampling across your estate, producing a sensitivity score per bucket. It is on by default when Macie is first enabled for an account ⚠ VERIFY for organisation member accounts.
  • Sensitive data discovery jobs — targeted one-time or scheduled jobs over chosen buckets, with full inspection.

Managed identifiers relevant to Harbourline include UK_NATIONAL_INSURANCE_NUMBER, UK_PASSPORT_NUMBER, UK_DRIVERS_LICENSE, UK_PHONE_NUMBER, UK_BANK_ACCOUNT_NUMBER (IBANs and BBANs), UK_TAX_IDENTIFICATION_NUMBER, CREDIT_CARD_NUMBER, CREDIT_CARD_SECURITY_CODE, NAME, ADDRESS, DATE_OF_BIRTH and credential detectors such as AWS_CREDENTIALS and PKCS. Harbourline-specific identifiers such as customer IDs (CUS- + 8 digits) need custom identifiers.

Pricing (us-east-1, checked 2026-11-17): USD 0.10 per bucket per month for inventory and monitoring; automated discovery USD 0.01 per 100,000 objects monitored plus USD 1 per GB inspected; discovery jobs USD 1 per GB inspected. A 30-day free trial covers bucket monitoring and automated discovery (up to 150 GB inspected), but not targeted discovery jobs.

Limitations: Macie only reads S3 (not RDS or DynamoDB directly); it needs kms:Decrypt on customer managed keys to inspect encrypted objects; detection is probabilistic, so false positives and negatives happen; and findings are kept for 90 days in Macie, so export them (EventBridge, Security Hub) if you need them longer.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Storage account "Allow Blob public access = Disabled"S3 Block Public Access (bucket/account/organisation)Four separate settings; can be enforced organisation-wide with an Organizations S3 policy
Shared Key access disabled / Entra ID-only authObject Ownership "bucket owner enforced" (ACLs off) + IAM/bucket policiesACLs were S3's legacy permission system; turn them off
Storage account network rules / private endpointsVPC endpoints + aws:SourceVpce in bucket policyRestriction is expressed in policy, not as a network setting on the bucket
"Secure transfer required"Bucket policy denying aws:SecureTransport = falseYou write it yourself (or CDK EnforceSSL)
Storage Service Encryption with customer-managed keys in Key VaultSSE-KMS with a customer managed KMS keyKMS key policy is a second authorization gate on every read
Immutable blob storage (time-based retention, legal hold, locked policy)S3 Object Lock (governance/compliance, legal hold)Very close match; compliance mode = locked time-based policy
Microsoft Purview data map / sensitive info typesAmazon Macie managed and custom data identifiersMacie is S3-only
Defender for Storage (malware/sensitive-data threat detection)GuardDuty S3 Protection / Malware Protection for S3 + MacieDetection and classification are separate services on AWS

Architecture

Security-boundary view — how an S3 GetObject is decided:

Object Lock is not on the read path; it guards writes and deletes of locked versions.

  • Security of the cloud (AWS): durability of S3, the encryption implementation, the HSMs behind KMS, enforcement of Object Lock semantics.
  • Security in the cloud (Harbourline): turning on BPA everywhere, writing the bucket and key policies, choosing retention modes and periods, scanning with Macie and acting on findings, and deciding legally how long to keep data.

Step-by-step implementation

Lock public access at account level

Even if you only own one sandbox account, set account-level BPA. In an Organization, use an Organizations S3 policy so member accounts cannot undo it (shown in Infrastructure as Code).

Create the records bucket with Object Lock enabled

Object Lock can be enabled at creation or later on an existing versioned bucket. Turning it on is permanent for the bucket (you cannot disable Object Lock afterwards), but with no default retention it does nothing until you lock an object.

Set default encryption to your KMS key with a Bucket Key, and block SSE-C explicitly

Use the KMS key from the KMS lesson or create alias/hl-m05-records. Default encryption means callers do not need to send encryption headers.

Attach the bucket policy

Three statements: deny non-TLS requests, deny uploads that ask for a different encryption type or key, and deny data access from outside your VPC endpoint (except for administrators). Use the IfExists condition operators so uploads that rely on default encryption are not rejected.

Write and lock a record from C#

Run the console app below: it uploads a fictional statement with SSE-KMS and a SHA-256 checksum, applies a governance-mode retention of 2 hours, shows that a version-specific delete is refused, and lists Macie findings.

Scan with Macie

Enable Macie (free trial), create a custom data identifier for Harbourline customer IDs, upload the synthetic customers.csv sample from datasets/samples/csv/, and run a one-time job on the bucket. Expect findings for names, emails, dates of birth, UK phone numbers and your CUS- identifier. A one-time job over a 1 MB file costs well under one cent.

AWS Console instructions

Account-level BPA: Console → S3 → left menu Block Public Access settings for this account → Edit → tick Block all public access → Save changes → type confirm.

Create the bucket: S3 → Create bucket → General purpose → name hl-m05-records-<account-id> → Object Ownership: ACLs disabled (recommended) → Block all public access: on → Bucket Versioning: Enable → Default encryption: SSE-KMS → choose your key → Bucket Key: Enable → Advanced settings → Object Lock: Enable and acknowledge → add the four tags → Create bucket. Do not configure a default retention for the lab.

Bucket policy: bucket → Permissions → Bucket policy → Edit → paste the policy from the CLI section → Save changes. The policy editor runs IAM Access Analyzer validation and shows warnings.

Object Lock retention on one object: bucket → object → Versions toggle on → select the version → Properties → Object Lock retention → Edit → Governance mode → retain-until date within the next day → Save changes.

Macie: Console → Amazon Macie → Get started → Enable Macie. Then Custom data identifiers → Create → name hl-customer-id, regex CUS-[0-9]{8}, keywords customer_id, customer, maximum match distance 50 → Test with customer_id: CUS-00010001 → Submit. Then Jobs → Create job → select the bucket → One-time job → select managed identifiers Recommended and your custom identifier → Submit. Findings appear under Findings after the job completes (minutes for a small bucket).

AWS CLI

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text --profile learn-sandbox)
BUCKET="hl-m05-records-${ACCOUNT_ID}"
KEY_ARN=$(aws kms describe-key --key-id alias/hl-m05-records --query KeyMetadata.Arn --output text --profile learn-sandbox)

# 1. Account-level Block Public Access (all four settings)
aws s3control put-public-access-block --account-id "$ACCOUNT_ID" \
  --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true \
  --profile learn-sandbox

# 2. Bucket with Object Lock enabled (us-east-1 needs no LocationConstraint)
aws s3api create-bucket --bucket "$BUCKET" --object-lock-enabled-for-bucket --profile learn-sandbox
aws s3api put-bucket-tagging --bucket "$BUCKET" --profile learn-sandbox --tagging \
  'TagSet=[{Key=Project,Value=cea-lab},{Key=Lab,Value=m05-s3},{Key=Environment,Value=learn},{Key=Owner,Value=alex}]'

# 3. ACLs off and BPA on for the bucket (defaults, but make it explicit and auditable)
aws s3api put-bucket-ownership-controls --bucket "$BUCKET" \
  --ownership-controls 'Rules=[{ObjectOwnership=BucketOwnerEnforced}]' --profile learn-sandbox
aws s3api put-public-access-block --bucket "$BUCKET" \
  --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true \
  --profile learn-sandbox

# 4. Default encryption: our CMK + Bucket Key, and explicitly block SSE-C
aws s3api put-bucket-encryption --bucket "$BUCKET" --profile learn-sandbox \
  --server-side-encryption-configuration "{
    \"Rules\": [{
      \"ApplyServerSideEncryptionByDefault\": {\"SSEAlgorithm\": \"aws:kms\", \"KMSMasterKeyID\": \"${KEY_ARN}\"},
      \"BucketKeyEnabled\": true,
      \"BlockedEncryptionTypes\": {\"EncryptionType\": [\"SSE-C\"]}
    }]}"

BlockedEncryptionTypes is new (2026); if your CLI version rejects it, upgrade the CLI or drop that line, since new buckets block SSE-C by default anyway ⚠ VERIFY exact JSON shape against the PutBucketEncryption API reference.

bucket-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": ["arn:aws:s3:::BUCKET", "arn:aws:s3:::BUCKET/*"],
      "Condition": { "Bool": { "aws:SecureTransport": "false" } }
    },
    {
      "Sid": "DenyWrongEncryptionType",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::BUCKET/*",
      "Condition": { "StringNotEqualsIfExists": { "s3:x-amz-server-side-encryption": "aws:kms" } }
    },
    {
      "Sid": "DenyWrongKmsKey",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::BUCKET/*",
      "Condition": { "StringNotEqualsIfExists": { "s3:x-amz-server-side-encryption-aws-kms-key-id": "KEY_ARN" } }
    },
    {
      "Sid": "DenyDataAccessOutsideVpcEndpoint",
      "Effect": "Deny",
      "Principal": "*",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:DeleteObjectVersion"],
      "Resource": "arn:aws:s3:::BUCKET/*",
      "Condition": {
        "StringNotEquals": { "aws:SourceVpce": "VPCE_ID" },
        "ArnNotLike": { "aws:PrincipalArn": [
          "arn:aws:iam::ACCOUNT_ID:role/aws-reserved/sso.amazonaws.com/*AWSReservedSSO_SandboxAdmin_*"
        ] }
      }
    }
  ]
}
# Fill in placeholders and apply. Skip the VPC endpoint statement if you have no endpoint yet,
# or you will lock yourself out of object operations from your laptop.
sed -e "s|BUCKET|${BUCKET}|g" -e "s|KEY_ARN|${KEY_ARN}|g" -e "s|ACCOUNT_ID|${ACCOUNT_ID}|g" \
    -e "s|VPCE_ID|${VPCE_ID:-vpce-00000000000000000}|g" bucket-policy.json > /tmp/policy.json
aws s3api put-bucket-policy --bucket "$BUCKET" --policy file:///tmp/policy.json --profile learn-sandbox

# 5. Prove TLS enforcement: an http:// endpoint call is denied
aws s3api head-bucket --bucket "$BUCKET" --endpoint-url http://s3.us-east-1.amazonaws.com --profile learn-sandbox \
  || echo "Denied as expected (no TLS)"

# 6. Upload with a governance-mode lock for 2 hours (date in UTC)
UNTIL=$(date -u -d '+2 hours' +%Y-%m-%dT%H:%M:%SZ)
aws s3api put-object --bucket "$BUCKET" --key statements/2026/10/ACC-1000000001.json \
  --body statement.json --checksum-algorithm SHA256 \
  --object-lock-mode GOVERNANCE --object-lock-retain-until-date "$UNTIL" --profile learn-sandbox

# 7. Inspect the lock
aws s3api get-object-retention --bucket "$BUCKET" --key statements/2026/10/ACC-1000000001.json --profile learn-sandbox

# 8. Place and remove a legal hold
aws s3api put-object-legal-hold --bucket "$BUCKET" --key statements/2026/10/ACC-1000000001.json \
  --legal-hold Status=ON --profile learn-sandbox
aws s3api put-object-legal-hold --bucket "$BUCKET" --key statements/2026/10/ACC-1000000001.json \
  --legal-hold Status=OFF --profile learn-sandbox

Macie:

# Enable Macie (starts the 30-day trial in this Region)
aws macie2 enable-macie --profile learn-sandbox

# Custom identifier for Harbourline customer IDs
CDI_ID=$(aws macie2 create-custom-data-identifier --name hl-customer-id \
  --regex 'CUS-[0-9]{8}' --keywords customer_id customer --maximum-match-distance 50 \
  --query customDataIdentifierId --output text --profile learn-sandbox)

# Test the identifier against sample text before using it
aws macie2 test-custom-data-identifier --regex 'CUS-[0-9]{8}' \
  --keywords customer_id --sample-text 'customer_id,CUS-00010001' --profile learn-sandbox

# Upload synthetic customers and run a one-time job on the bucket
aws s3 cp ../../datasets/samples/csv/customers.csv "s3://${BUCKET}/macie-test/customers.csv" --profile learn-sandbox
aws macie2 create-classification-job --job-type ONE_TIME --name hl-m05-customers-scan \
  --custom-data-identifier-ids "$CDI_ID" \
  --s3-job-definition "{\"bucketDefinitions\":[{\"accountId\":\"${ACCOUNT_ID}\",\"buckets\":[\"${BUCKET}\"]}]}" \
  --tags Project=cea-lab,Lab=m05-s3,Environment=learn,Owner=alex \
  --profile learn-sandbox

# List high-severity sensitive data findings
aws macie2 list-findings --finding-criteria '{"criterion":{"category":{"eq":["CLASSIFICATION"]}}}' \
  --profile learn-sandbox

Macie must be able to decrypt the objects: add a key policy statement allowing the Macie service-linked role (AWSServiceRoleForAmazonMacie) kms:Decrypt on the records key, or Macie reports the objects as unclassifiable.

Infrastructure as Code

RecordsBucketStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.S3;
using Constructs;

namespace Harbourline.Security.Infra;

public sealed class RecordsBucketStack : Stack
{
    public RecordsBucketStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        var key = new Key(this, "RecordsKey", new KeyProps
        {
            Alias = "alias/hl-m05-records",
            EnableKeyRotation = true,
            PendingWindow = Duration.Days(7),
            RemovalPolicy = RemovalPolicy.DESTROY          // learning shortcut
        });

        var bucket = new Bucket(this, "Records", new BucketProps
        {
            BucketName = $"hl-m05-records-{Account}",
            BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
            ObjectOwnership = ObjectOwnership.BUCKET_OWNER_ENFORCED,
            Encryption = BucketEncryption.KMS,
            EncryptionKey = key,
            BucketKeyEnabled = true,
            EnforceSSL = true,                              // adds the aws:SecureTransport deny
            MinimumTLSVersion = 1.2,                        // denies TLS below 1.2 via s3:TlsVersion
            Versioned = true,
            ObjectLockEnabled = true,
            // ObjectLockDefaultRetention = ObjectLockRetention.Governance(Duration.Days(1)),  // only if needed; never compliance in labs
            RemovalPolicy = RemovalPolicy.RETAIN            // locked objects cannot be auto-deleted anyway
        });

        // Deny uploads that request a different key (IfExists keeps default encryption working)
        bucket.AddToResourcePolicy(new PolicyStatement(new PolicyStatementProps
        {
            Sid = "DenyWrongKmsKey",
            Effect = Effect.DENY,
            Principals = new IPrincipal[] { new AnyPrincipal() },
            Actions = new[] { "s3:PutObject" },
            Resources = new[] { bucket.ArnForObjects("*") },
            Conditions = new Dictionary<string, object>
            {
                ["StringNotEqualsIfExists"] = new Dictionary<string, object>
                {
                    ["s3:x-amz-server-side-encryption-aws-kms-key-id"] = key.KeyArn
                }
            }
        }));

        // Let Macie read encrypted objects
        key.AddToResourcePolicy(new PolicyStatement(new PolicyStatementProps
        {
            Sid = "AllowMacieDecrypt",
            Principals = new IPrincipal[] { new ArnPrincipal($"arn:aws:iam::{Account}:role/aws-service-role/macie.amazonaws.com/AWSServiceRoleForAmazonMacie") },
            Actions = new[] { "kms:Decrypt" },
            Resources = new[] { "*" }
        }));

        foreach (var (k, v) in new Dictionary<string, string>
                 { ["Project"] = "cea-lab", ["Lab"] = "m05-s3", ["Environment"] = "learn", ["Owner"] = "alex" })
            Amazon.CDK.Tags.Of(this).Add(k, v);
    }
}

The Macie service-linked role must exist (it is created when you enable Macie) before this key policy statement deploys, or KMS rejects the unknown principal. cdk-nag rule AwsSolutions-S1 will also ask for server access logs; add a log bucket or suppress with a written reason.

Complete code

A console app that writes a locked record, proves it cannot be deleted, and reads Macie findings.

Harbourline.Security.S3Records.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <RootNamespace>Harbourline.Security.S3Records</RootNamespace>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="AWSSDK.S3" Version="4.1.1" />
    <PackageReference Include="AWSSDK.Macie2" Version="4.1.0.2" />
    <!-- Needed at runtime to resolve IAM Identity Center (SSO) profiles -->
    <PackageReference Include="AWSSDK.SSO" Version="4.1.0.2" />
    <PackageReference Include="AWSSDK.SSOOIDC" Version="4.1.0.2" />
  </ItemGroup>
</Project>
Program.cs
using System.Net;
using System.Text;
using System.Text.Json;
using Amazon;
using Amazon.Macie2;
using Amazon.Macie2.Model;
using Amazon.Runtime.Credentials;
using Amazon.S3;
using Amazon.S3.Model;

var bucket = args.Length > 0 ? args[0] : throw new ArgumentException("Usage: dotnet run -- <bucket> <kms-key-arn>");
var kmsKeyArn = args.Length > 1 ? args[1] : throw new ArgumentException("Missing KMS key ARN");

var credentials = DefaultAWSCredentialsIdentityResolver.GetCredentials(); // honours AWS_PROFILE=learn-sandbox
var region = RegionEndpoint.USEast1;
using var s3 = new AmazonS3Client(credentials, region);
using var macie = new AmazonMacie2Client(credentials, region);

// 1. Build a fictional statement document (synthetic data only)
var statement = new
{
    accountId = "ACC-1000000001",
    customerId = "CUS-00010001",
    period = "2026-10",
    currency = "GBP",
    closingBalance = 1523.47m,
    generatedAt = DateTime.UtcNow
};
var body = JsonSerializer.Serialize(statement);
var key = $"statements/2026/10/{statement.accountId}.json";

// 2. Upload with SSE-KMS, a Bucket Key, a SHA-256 checksum and a governance lock
var put = await s3.PutObjectAsync(new PutObjectRequest
{
    BucketName = bucket,
    Key = key,
    ContentBody = body,
    ContentType = "application/json",
    ServerSideEncryptionMethod = ServerSideEncryptionMethod.AWSKMS,
    ServerSideEncryptionKeyManagementServiceKeyId = kmsKeyArn,
    BucketKeyEnabled = true,
    ChecksumAlgorithm = ChecksumAlgorithm.SHA256,
    ObjectLockMode = ObjectLockMode.Governance,
    ObjectLockRetainUntilDate = DateTime.UtcNow.AddHours(2)   // lab: hours, never years
});
Console.WriteLine($"Stored {key} version {put.VersionId} sha256={put.ChecksumSHA256} sse={put.ServerSideEncryptionMethod}");

// 3. Read the retention back
var retention = await s3.GetObjectRetentionAsync(new GetObjectRetentionRequest
{
    BucketName = bucket, Key = key, VersionId = put.VersionId
});
Console.WriteLine($"Lock: {retention.Retention.Mode} until {retention.Retention.RetainUntilDate:O}");

// 4. Try to permanently delete the locked version WITHOUT bypass: must fail
try
{
    await s3.DeleteObjectAsync(new DeleteObjectRequest { BucketName = bucket, Key = key, VersionId = put.VersionId });
    Console.WriteLine("UNEXPECTED: locked version was deleted. Check Object Lock configuration!");
}
catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.Forbidden)
{
    Console.WriteLine($"Delete refused as expected: {ex.ErrorCode}");
}

// 5. Verify integrity on read: ask S3 to validate the stored checksum
using (var get = await s3.GetObjectAsync(new GetObjectRequest
{
    BucketName = bucket, Key = key, VersionId = put.VersionId, ChecksumMode = ChecksumMode.ENABLED
}))
using (var reader = new StreamReader(get.ResponseStream, Encoding.UTF8))
{
    var roundTrip = await reader.ReadToEndAsync();
    Console.WriteLine($"Read back {roundTrip.Length} chars, checksum validated by SDK: {get.ChecksumSHA256}");
}

// 6. Macie: list recent sensitive-data (classification) findings and summarise them
var list = await macie.ListFindingsAsync(new ListFindingsRequest
{
    FindingCriteria = new FindingCriteria
    {
        Criterion = new Dictionary<string, CriterionAdditionalProperties>
        {
            ["category"] = new() { Eq = new List<string> { "CLASSIFICATION" } }
        }
    },
    SortCriteria = new SortCriteria { AttributeName = "updatedAt", OrderBy = OrderBy.DESC },
    MaxResults = 20
});

if (list.FindingIds is { Count: > 0 })
{
    var details = await macie.GetFindingsAsync(new GetFindingsRequest { FindingIds = list.FindingIds });
    foreach (var f in details.Findings)
    {
        var obj = f.ResourcesAffected?.S3Object?.Key;
        Console.WriteLine($"[{f.Severity?.Description}] {f.Type} s3://{f.ResourcesAffected?.S3Bucket?.Name}/{obj}");
        foreach (var item in f.ClassificationDetails?.Result?.SensitiveData ?? new List<SensitiveDataItem>())
            foreach (var d in item.Detections ?? new List<DefaultDetection>())
                Console.WriteLine($"    {item.Category}: {d.Type} x{d.Count}");
        foreach (var c in f.ClassificationDetails?.Result?.CustomDataIdentifiers?.Detections ?? new List<CustomDetection>())
            Console.WriteLine($"    CUSTOM: {c.Name} x{c.Count}");
    }
}
else
{
    Console.WriteLine("No Macie classification findings yet (job still running, or Macie not enabled).");
}

Run:

export AWS_PROFILE=learn-sandbox
aws sso login
dotnet run -- "$BUCKET" "$KEY_ARN"

Expected output (shape):

Stored statements/2026/10/ACC-1000000001.json version 3HL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY sha256=Vx1… sse=aws:kms
Lock: GOVERNANCE until 2026-11-17T14:05:12.0000000Z
Delete refused as expected: AccessDenied
Read back 162 chars, checksum validated by SDK: Vx1…
[High] SensitiveData:S3Object/Personal s3://hl-m05-records-111122223333/macie-test/customers.csv
    PERSONAL_INFORMATION: NAME x31
    PERSONAL_INFORMATION: DATE_OF_BIRTH x31
    CUSTOM: hl-customer-id x31

Code explained line by line

  1. DefaultAWSCredentialsIdentityResolver.GetCredentials() — SDK v4's credential chain; with AWS_PROFILE=learn-sandbox it uses your Identity Center session. No keys in code.
  2. new AmazonS3Client(credentials, region) and new AmazonMacie2Client(...) — two clients in us-east-1. In an ASP.NET Core app you would register them with AddAWSService<IAmazonS3>() instead.
  3. The anonymous statement object — synthetic data in the canonical schema; decimal for money.
  4. ServerSideEncryptionMethod.AWSKMS + ServerSideEncryptionKeyManagementServiceKeyId — asks S3 to encrypt with our CMK. The bucket policy would deny a different key. Sending it explicitly documents intent; default encryption would apply the same key if omitted.
  5. BucketKeyEnabled = true — use the bucket-level key to reduce KMS calls.
  6. ChecksumAlgorithm = ChecksumAlgorithm.SHA256 — the SDK computes a SHA-256 of the body, S3 verifies it on arrival and stores it. Object Lock uploads must carry an integrity checksum (Content-MD5 or an x-amz-checksum-* header); this satisfies it.
  7. ObjectLockMode.Governance and ObjectLockRetainUntilDate — lock this version for two hours. Governance mode means a principal with s3:BypassGovernanceRetention can still remove it, which is what makes the lab safely reversible.
  8. put.VersionId — Object Lock protects versions, not keys. Always keep the version ID of regulated records in your own database.
  9. GetObjectRetentionAsync — reads the lock back as evidence (auditors like seeing this in automated tests).
  10. DeleteObjectAsync with VersionId and no bypass — a permanent delete of a locked version returns 403 AccessDenied. A delete without a version ID would succeed by adding a delete marker; the locked version remains.
  11. ChecksumMode = ChecksumMode.ENABLED on GetObject — the SDK validates the stored checksum as it reads, detecting corruption in transit.
  12. ListFindingsAsync with category = CLASSIFICATION — sensitive data findings (as opposed to POLICY findings about bucket settings), newest first.
  13. GetFindingsAsync — returns details; ClassificationDetails.Result.SensitiveData groups detections by category, CustomDataIdentifiers.Detections shows our hl-customer-id matches. Counts only; Macie does not return the sensitive values here (sample retrieval is a separate, permission-controlled feature).

Security considerations

⚠ Learning shortcut — not for production

The lab gives your SandboxAdmin role an exception in the VPC-endpoint deny, keeps retention to hours in governance mode, and uses the same account for the data and for Macie. In production: run Macie from a delegated administrator in the security tooling account, store regulated records in a dedicated account, use compliance mode only after records-management and legal sign-off on retention periods, allow s3:BypassGovernanceRetention to nobody except a monitored break-glass role (deny it in an SCP for everyone else), and alarm on any PutBucketPolicy, PutBucketPublicAccessBlock or PutObjectRetention call.

  • Use IfExists in encryption conditions. StringNotEquals on a key that is absent from the request evaluates to true, so a plain deny would reject uploads that rely on default encryption. StringNotEqualsIfExists only bites when the caller explicitly asks for the wrong encryption.
  • The KMS key policy is part of the access path. A bucket that is accidentally readable by another account is still protected if that account cannot use the key. Conversely, granting kms:Decrypt broadly undoes the benefit.
  • Object Lock is not a backup. It protects against deletion and overwrite, not against Region loss or a mis-designed retention. Combine with replication (S3 Replication can replicate lock settings) or AWS Backup in another account (Backup security and key rotation).
  • Right to erasure tension. Compliance-mode WORM storage of personal data conflicts with UK GDPR erasure requests unless a legal obligation to retain applies. Keep personal data out of WORM stores where possible, or store it encrypted with a per-customer key you can destroy (crypto-shredding). See UK GDPR and data residency. This is a decision for your DPO and legal team, not for engineers alone.
  • Macie sees your data. It needs kms:Decrypt on your keys and stores sample metadata. Restrict who can view findings and enable sensitive data sample retrieval only for the privacy team.
  • Detect drift. AWS Config managed rules such as s3-bucket-public-read-prohibited, s3-bucket-ssl-requests-only, s3-bucket-server-side-encryption-enabled and s3-default-encryption-kms produce continuous evidence (AWS Config). IAM Access Analyzer reports buckets shared outside your organisation.

Banking use case

Harbourline must keep customer statements and payment instruction records for the period its records-retention policy specifies (a policy set by compliance, typically referencing regulatory and tax requirements). The design:

  • A dedicated records account, bucket hl-records-statements-prod, Object Lock compliance mode with a default retention approved by the records-management committee, KMS CMK per data domain, Bucket Keys on.
  • Writers (the statements batch job) can only PutObject through a VPC endpoint; nobody has DeleteObjectVersion. Readers (customer service app) have GetObject on specific prefixes and kms:Decrypt with an encryption-context condition.
  • An Organizations S3 policy enforces Block Public Access in every account; an SCP denies s3:BypassGovernanceRetention and s3:PutBucketObjectLockConfiguration except to the records-admin break-glass role.
  • Macie automated discovery runs across all accounts from the security tooling account. A finding of UK_NATIONAL_INSURANCE_NUMBER in a bucket tagged DataClassification=Internal raises a ticket to the data owner within 24 hours.
  • Legal holds are placed by the legal team's role when litigation starts, and removed only with legal approval, recorded in the ticketing system.

Common mistakes

  • Turning BPA off "temporarily" to share a file → data exposed and forgotten → share with presigned URLs of short duration, or CloudFront with origin access control.
  • Forgetting the KMS key in cross-account access → AccessDenied that looks like an S3 problem → grant kms:Decrypt in the key policy (and IAM) for the reader.
  • Compliance mode in test → a bill you cannot stop → governance mode, short retention.
  • Assuming Object Lock covers new uploads automatically → without a default retention, new objects are not locked → set a default retention or lock at upload.
  • Plain StringNotEquals for encryption headers → default-encrypted uploads denied → use IfExists.
  • Running Macie without key access → objects silently skipped → add the Macie service-linked role to the key policy and check "unclassifiable objects" stats.
  • Running full Macie jobs on large data lakes → USD 1 per GB adds up → rely on automated discovery sampling and target jobs at high-risk buckets.

Troubleshooting

SymptomLikely causeFix
AccessDenied on PutObject after adding the bucket policyClient explicitly sent a different KMS key or AES256Remove the header or send the correct key ARN
AccessDenied on GetObject though IAM allows itMissing kms:Decrypt on the CMK, or VPC endpoint denyCheck the KMS key policy; check aws:SourceVpce; use CloudTrail to see whether S3 or KMS denied
InvalidRequest: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parametersNo checksum on an Object Lock uploadSet ChecksumAlgorithm (SDK) or --checksum-algorithm (CLI)
InvalidRequest when setting retentionBucket does not have Object Lock enabled, or versioning suspendedEnable Object Lock on the bucket (requires versioning)
Delete of a locked object "succeeds"You deleted without a version ID; a delete marker was addedList versions: the locked version is still there
Cannot remove governance retentionMissing s3:BypassGovernanceRetention or the bypass headerAdd the permission to a break-glass role; CLI --bypass-governance-retention
put-public-access-block succeeds but settings revert / are ignoredOrganisation-level S3 policy overrides account settingsChange the organisation policy (management account) if the exception is approved
Macie job finds nothing in customers.csvJob still running, KMS decrypt denied, or file larger/format unsupportedCheck job status and the bucket's "unclassifiable" counts; fix key policy
Macie custom identifier misses CUS-00010001Keyword not within maximum match distanceIncrease distance or remove keyword requirement; test with test-custom-data-identifier

Cost and cleanup

Expected cost: S3 storage and requests for a few KB: under USD 0.01. KMS CMK: USD 1 per month prorated, plus requests (reuse the key from the KMS lesson if you still have it). Macie: free during the 30-day trial for monitoring and automated discovery; the one-time job on one small CSV is about USD 0.000001 per KB ≈ free, billed at USD 1 per GB. Total: well under USD 0.10 if you clean up the same day.

# 1. Wait until the 2-hour governance lock expires, or bypass it (needs s3:BypassGovernanceRetention)
aws s3api list-object-versions --bucket "$BUCKET" --query '{V:Versions[].{K:Key,Id:VersionId},M:DeleteMarkers[].{K:Key,Id:VersionId}}' --profile learn-sandbox
aws s3api delete-object --bucket "$BUCKET" --key statements/2026/10/ACC-1000000001.json \
  --version-id "<version-id>" --bypass-governance-retention --profile learn-sandbox
# repeat for every version and delete marker (or use a small loop over list-object-versions output)

# 2. Delete the bucket
aws s3api delete-bucket --bucket "$BUCKET" --profile learn-sandbox

# 3. Macie: delete the custom identifier, then disable Macie (deletes Macie findings and config in this Region)
aws macie2 delete-custom-data-identifier --id "$CDI_ID" --profile learn-sandbox
aws macie2 disable-macie --profile learn-sandbox

# 4. KMS key: schedule deletion (7-day minimum) if you created one for this lesson
aws kms schedule-key-deletion --key-id alias/hl-m05-records --pending-window-in-days 7 --profile learn-sandbox

# 5. Confirm nothing remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --query "ResourceTagMappingList[].ResourceARN" --profile learn-sandbox

Keep account-level Block Public Access on; it costs nothing.

Exercises

  • 🟢 Guided: Follow the steps: account BPA, Object Lock bucket with CMK default encryption, the bucket policy (without the VPC endpoint statement), the C# app, and a Macie job on customers.csv. Record the output of each step.
  • 🟡 Challenge: Add a bucket policy statement that denies setting a retention longer than 1 day in the sandbox (s3:object-lock-remaining-retention-days), and an SCP-style IAM deny for s3:BypassGovernanceRetention for every principal except a role named hl-m05-records-breakglass. Prove both with CLI calls.
  • 🔴 Live problem: Macie raises a High finding: UK_NATIONAL_INSURANCE_NUMBER x4,812 in s3://hl-analytics-exports-prod/ad-hoc/2026-11/ — a bucket tagged DataClassification=Internal, owned by the analytics team, with versioning on but no Object Lock. The object was written yesterday by a data scientist's role. Decide the immediate actions, who you involve, and the longer-term fix.

Assignment

Create a reusable "Restricted data bucket" baseline: a CDK construct (C#) or Terraform module that always enables BPA, ACLs off, CMK + Bucket Key, TLS 1.2 minimum, versioning, optional Object Lock (governance by default), access logging and Macie key access. Include a README listing which Config rules evidence each setting. Commit it to your portfolio.

Solutions

Knowledge check

Interview questions

  • "How do you make sure no S3 bucket in the bank ever becomes public?" — Organisation-level Block Public Access via an Organizations S3 policy, SCP denying changes to BPA settings, ACLs disabled, Config rules and Access Analyzer for detection, Macie policy findings, and an exception process for the rare static-content account (which still uses CloudFront with origin access control rather than public buckets).
  • "How would you store records that must not be altered for seven years?" — Dedicated account, S3 Object Lock compliance mode with a default retention approved by records management, CMK encryption, replication or AWS Backup to a second account/Region, legal hold process, and evidence (retention configuration, Config rules). Mention the GDPR tension for personal data and crypto-shredding.
  • "What is the risk of SSE-C?" — The bank must manage and send keys on every request; key loss means data loss, and keys travel in headers. S3 now blocks SSE-C on new buckets by default; SSE-KMS with a CMK gives auditability and policy control.
  • "What does Macie not do?" — It does not scan databases, does not prevent writes, and its detection is probabilistic; it complements data classification at source and DLP controls.

Certification practice

Completion checklist

  • I can list the six layers that decide S3 access and which ones can deny
  • I enabled account-level BPA and created an Object Lock bucket with CMK default encryption
  • My bucket policy enforces TLS and the correct KMS key using IfExists
  • I locked an object in governance mode and saw a permanent delete refused
  • I ran a Macie job with a custom identifier and read findings from C#
  • I removed all versions, deleted the bucket, disabled Macie and scheduled key deletion

Key takeaways

  • Block Public Access overrides public policies and ACLs; enforce it at organisation level and turn ACLs off.
  • Write bucket policies for TLS, key choice and network origin, using IfExists so default encryption keeps working.
  • SSE-KMS with a customer managed key puts a second gate (the key policy) on every read; Bucket Keys keep it affordable.
  • Object Lock compliance mode is truly irreversible; use governance mode and short retention in labs, and get legal sign-off for real retention periods.
  • Macie tells you where personal data actually is; route its findings to data owners and act on them.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page