Skip to Content
Cloud Engineering Academy
4. API Development & Integration

SWIFT and file-based integration: SFTP with AWS Transfer Family, PGP and checksums

Exchange batch payment and statement files with partners safely: understand where Swift and scheme file channels fit, run SFTP into S3 with AWS Transfer Family (or a free local SFTP server), encrypt and decrypt with OpenPGP in C#, verify SHA-256 manifests and S3 checksums, and process each file exactly once.

intermediate⏱ 2 h 30 min🔴 Paid — optionalSAA-C03SCS-C03DVA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.S3 4.1.x) · SSH.NET 2026.0.1 · BouncyCastle.Cryptography 2.7.0 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Cost: AWS Transfer Family bills per hour while the endpoint exists

An SFTP endpoint costs about USD 0.30 per hour per enabled protocol (us-east-1, verified 2026-11-18), roughly USD 216 per month if left running, plus USD 0.04 per GB transferred. Stopping a server does not stop the hourly charge; only deleting it does ⚠ VERIFY on the pricing page. The free path in this lesson uses an SFTP server in Docker on your laptop with the same C# code. If you do deploy Transfer Family, deploy and delete it within one hour (about USD 0.30 plus pennies).

Learning objectives

By the end of this lesson you can:

  • Explain why banks still exchange files (batch payments, statements, reconciliation, regulatory reports), and where Swift FileAct, scheme gateways and partner SFTP fit, at a high level.
  • Design an SFTP-to-S3 integration with AWS Transfer Family: endpoint types (public, VPC internal, VPC internet-facing with Elastic IPs), service-managed users with SSH keys, logical home directories, scoped IAM roles, security policies and logging.
  • Encrypt and decrypt files with OpenPGP in C# using BouncyCastle, and know when to use Transfer Family managed workflows to decrypt instead.
  • Verify file integrity with SHA-256 manifests and S3 additional checksums, and process each file exactly once with quarantine, idempotency and immutable storage.
  • Run the whole flow free against a local SFTP server in Docker.

Prerequisites

Why this matters in banking

Every weekday at 06:00, Northwind Payroll Ltd (fictional) uploads a salary file for 4,812 employees to Harbourline Bank. One Thursday the upload is interrupted at 70%, Northwind's script retries, and Harbourline's poller picks up both the partial file and the complete one. The partial file fails parsing halfway, after 3,100 payments were already created; the complete file then creates all 4,812. Some employees are paid twice, others once, and the reconciliation team spends two days unpicking it.

In another bank, a partner's SFTP password was shared in an email thread, and an attacker uploaded a payment file with modified beneficiary IBANs. It was processed because "it came through the secure channel".

File-based integration is old, unglamorous and still carries enormous value: bulk payments, statements, card settlement, direct debits and regulatory returns. Getting authentication, integrity, completeness and exactly-once processing right is the job.

The idea in plain language

A file integration has four questions to answer:

  1. Who sent it? Authentication of the channel (SSH keys for SFTP, certificates for AS2, the network's own authentication for Swift) and, ideally, a signature on the file itself.
  2. Is it complete and unaltered? A checksum (SHA-256) supplied separately, compared with what you received; PGP integrity protection; size and record counts in trailers.
  3. Can anyone else read it? Encryption in transit (SSH/TLS) and, for many partners, encryption of the file itself with PGP, so it stays protected in every folder and mailbox it passes through.
  4. Will we process it exactly once? A "file is ready" signal, a dedupe key (checksum plus partner and file name), immutable storage of the original and a quarantine area for anything suspicious.

Where Swift fits. Swift is the bank-to-bank network. Messages travel as FIN (MT) or ISO 20022 (MX) messages; FileAct is Swift's service for transferring files between institutions. Banks connect through Swift interfaces (Alliance Access/Gateway) run on their own infrastructure, a service bureau, or Swift-hosted options such as Alliance Lite2 ⚠ VERIFY current product names and hosting options. On AWS, you usually build the integration around the Swift connection: files and messages flow between your payment hub and the Swift interface through S3, queues or SFTP, under Swift's Customer Security Programme (CSP) controls, which your Swift security officer owns. This lesson focuses on the SFTP pattern because it is the most common partner channel and the same controls apply everywhere.

Analogy: Think of a secure document drop at a bank branch. SFTP is the locked drop box with a key per courier (SSH key). PGP is the sealed, tamper-evident envelope inside the box, which only the intended recipient can open. The checksum manifest is the courier's signed delivery note listing exactly what is in the envelope. Exactly-once processing is the clerk who stamps each envelope "processed" with its reference number and refuses to process a second envelope with the same number.

What problem the AWS services solve

NeedAWS building blockHonest limitations
Managed SFTP/FTPS/AS2 endpoint writing to S3 or EFSAWS Transfer Family serverHourly charge per protocol while it exists; you still own users, keys, partner onboarding
Fixed IPs partners can allowlist, plus source-IP filteringTransfer Family VPC internet-facing endpoint with Elastic IPs and security groupsNeeds a VPC with public subnets; up to three AZs
Per-partner folder isolationLogical home directories and a per-user IAM role or session policyMappings are per user; automate them in IaC
Decrypt PGP on uploadTransfer Family managed workflows with a decrypt step (keys in Secrets Manager)Charged per GB decrypted; keys must be stored under a specific secret naming scheme
Push files to partners' SFTP serversTransfer Family SFTP connectorsCharged per call and per GB
Event-driven processingS3 event notifications or EventBridge → SQS → Lambda/ECSYou implement dedupe and ordering
Malware scanningGuardDuty Malware Protection for S3 (tags objects with a scan result)Per-GB and per-object charges; scan before processing
Immutable originalsS3 versioning + Object LockRetention cannot be shortened in compliance mode

Transfer Family itself does not know what a "payment file" is. Validation, dedupe and business processing are your code.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Blob Storage SFTP supportAWS Transfer Family SFTP with S3 domainTransfer Family is a separate service with its own endpoint and hourly price; supports FTPS, AS2, connectors and workflows
SFTP server on a Windows VM (e.g. with OpenSSH)Transfer Family, or SFTP on EC2 if you need full controlNo patching or HA to build with Transfer Family
Logic Apps SFTP connector / BizTalk file adaptersTransfer Family SFTP connectors, Lambda/ECS processorsWrite the processing logic in .NET
Event Grid BlobCreatedS3 event notifications / EventBridgeSame idea
Defender for Storage malware scanningGuardDuty Malware Protection for S3Results as object tags and findings
Gpg4win / BouncyCastle in .NETBouncyCastle in .NET (same library), or the managed workflow decrypt stepCode transfers unchanged

Architecture

Security-boundary view.

  • Partner zone: authenticated only by SSH public key (no passwords), from allowlisted source IPs; can list and write only /in, read only /out.
  • Landing zone: S3 prefix where nothing is trusted yet. The processor never reads a file until malware scanning and the manifest check pass.
  • Processing zone: private subnets; IAM role can read landing, write archive/quarantine/out, read one Secrets Manager secret (PGP private key) and decrypt with one KMS key.
  • Archive: Object Lock; nobody (including the processor) can delete originals during the retention period.

Step-by-step implementation

Agree the file contract with the partner

Write it down before any infrastructure: file naming (PAYROLL_YYYYMMDD_NNN.csv.pgp), format (CSV, ISO 20022 pain.001, Bacs Standard 18 and so on), encryption (PGP to Harbourline's public key, signed with the partner's key), the manifest (<file>.sha256 containing the SHA-256 and file name, uploaded after the data file), cut-off times, and the acknowledgement Harbourline sends back. The manifest-after-data rule is what solves the "partial file" problem: you only process when the manifest exists and matches.

Create the Transfer Family server

Choose SFTP only (each extra protocol is billed). Choose the endpoint type:

  • Public: simplest; AWS-managed IPs that can change, no security groups. Fine for experiments.
  • VPC, internal: reachable only from your network (VPN/Direct Connect); for internal systems such as a core banking batch server.
  • VPC, internet-facing: Elastic IPs (stable addresses partners can allowlist in their firewalls) and security groups (you allowlist partner IPs).

Pick a current security policy (the default is TransferSecurityPolicy-2024-01), enable CloudWatch logging with a logging role, and keep identity service-managed (SSH keys stored by Transfer Family) unless you need a custom identity provider.

Create one user per partner with a jailed home

For each partner: an IAM role that allows s3:PutObject on partners/<name>/in/*, s3:GetObject on partners/<name>/out/*, and s3:ListBucket limited by prefix; a logical home directory mapping / → /bucket/partners/<name>; and the partner's SSH public key. Never generate keys for partners; they send you public keys and keep private keys themselves.

Scan, then signal "ready"

Enable GuardDuty Malware Protection for the landing bucket (or prefix). Route "scan result = no threats found" events (or the object tag) to the processing queue, so nothing is processed before scanning. Files that fail scanning go to quarantine automatically.

Verify the manifest and checksums

When <file>.sha256 arrives, read it, stream the data file from S3 and compute SHA-256; compare in constant time. Additionally, upload to the archive with S3 additional checksums (ChecksumAlgorithm = SHA256 with the value you computed), so S3 itself verifies the bytes it stores.

Decrypt and verify the signature

Decrypt with Harbourline's PGP private key (in Secrets Manager, encrypted with KMS) and, if the contract requires, verify the partner's PGP signature. Use the integrity check (modification detection) that modern PGP encryption includes, and reject files without it.

Process exactly once

Compute the dedupe key: partner + SHA-256 of the encrypted file. Insert it into a DynamoDB table with a conditional write (attribute_not_exists). If it already exists, it is a duplicate: acknowledge, do not process. Then validate the content (record counts, totals, IBAN checksums, EndToEndId uniqueness), create payments idempotently, archive the original and write an acknowledgement file to /out.

AWS Console instructions

  • Server: Console → AWS Transfer Family → Servers → Create server → protocols SFTP → identity provider Service managed → endpoint VPC hosted, access Internet Facing, choose VPC, up to three public subnets and an Elastic IP per subnet, and a security group allowing TCP 22 from partner CIDRs only → domain Amazon S3 → CloudWatch logging: create or choose a logging role → Cryptographic algorithm options: keep the latest security policy → add tags → Create server.
  • User: server → Users → Add user → user name northwind → role hl-m04-sftp-northwind → Home directory: bucket and folder partners/northwind, tick Restricted (this creates a logical / mapping) → paste the partner's SSH public key → Add.
  • Workflow (optional): Workflows → Create workflow → add a Decrypt file step with destination decrypted/${transfer:UserName}/ → attach to the server under Managed workflows with an execution role.
  • Delete: server → Actions → Delete. Deletion, not stopping, ends billing.

AWS CLI

PROFILE=learn-sandbox
REGION=us-east-1
BUCKET=hl-m04-files-111122223333

# Bucket for landing/archive (private, versioned)
aws s3api create-bucket --bucket $BUCKET --profile $PROFILE --region $REGION
aws s3api put-public-access-block --bucket $BUCKET \
  --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true \
  --profile $PROFILE --region $REGION
aws s3api put-bucket-versioning --bucket $BUCKET --versioning-configuration Status=Enabled --profile $PROFILE --region $REGION

# SFTP server (PUBLIC endpoint keeps the lab simple and avoids Elastic IP charges)
SERVER_ID=$(aws transfer create-server \
  --protocols SFTP --domain S3 --endpoint-type PUBLIC \
  --identity-provider-type SERVICE_MANAGED \
  --security-policy-name TransferSecurityPolicy-2024-01 \
  --logging-role arn:aws:iam::111122223333:role/hl-m04-transfer-logging \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m04-l21 Key=Environment,Value=learn Key=Owner,Value=you \
  --query ServerId --output text --profile $PROFILE --region $REGION)
echo "Server: $SERVER_ID   (billing has started)"

# Partner user jailed to partners/northwind with a logical home directory and an SSH public key
aws transfer create-user --server-id $SERVER_ID --user-name northwind \
  --role arn:aws:iam::111122223333:role/hl-m04-sftp-northwind \
  --home-directory-type LOGICAL \
  --home-directory-mappings "[{\"Entry\":\"/\",\"Target\":\"/$BUCKET/partners/northwind\"}]" \
  --ssh-public-key-body "$(cat ~/.ssh/hl_northwind_ed25519.pub)" \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m04-l21 \
  --profile $PROFILE --region $REGION

# Connect as the partner and upload (endpoint hostname format: SERVERID.server.transfer.REGION.amazonaws.com)
sftp -i ~/.ssh/hl_northwind_ed25519 northwind@$SERVER_ID.server.transfer.$REGION.amazonaws.com <<'EOF'
mkdir in
put PAYROLL_20261118_001.csv.pgp in/
put PAYROLL_20261118_001.csv.pgp.sha256 in/
ls -l in
EOF

# Verify what landed in S3, including the stored SHA-256 checksum if one was supplied
aws s3api head-object --bucket $BUCKET --key partners/northwind/in/PAYROLL_20261118_001.csv.pgp \
  --checksum-mode ENABLED --profile $PROFILE --region $REGION

# DELETE the server when done (stop does not end billing)
aws transfer delete-server --server-id $SERVER_ID --profile $PROFILE --region $REGION

The IAM role for the user must trust transfer.amazonaws.com. Its policy (least privilege):

hl-m04-sftp-northwind-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOwnFolder",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::hl-m04-files-111122223333",
      "Condition": { "StringLike": { "s3:prefix": ["partners/northwind/*", "partners/northwind"] } }
    },
    {
      "Sid": "UploadToIn",
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": "arn:aws:s3:::hl-m04-files-111122223333/partners/northwind/in/*"
    },
    {
      "Sid": "DownloadFromOut",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:GetObjectVersion"],
      "Resource": "arn:aws:s3:::hl-m04-files-111122223333/partners/northwind/out/*"
    }
  ]
}

Partners cannot delete or overwrite processed files because there is no s3:DeleteObject and the processor moves files out of in/ after archiving. SFTP clients sometimes need s3:GetObject on in/ to resume uploads; add it only if the partner's client requires it.

Infrastructure as Code

infra/cdk/PartnerSftpStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.S3;
using Amazon.CDK.AWS.Transfer;
using Constructs;

namespace Harbourline.PartnerGateway.Infra;

public sealed class PartnerSftpStack : Stack
{
    public PartnerSftpStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        var bucket = new Bucket(this, "Files", new BucketProps
        {
            BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
            Encryption = BucketEncryption.KMS_MANAGED,
            EnforceSSL = true,
            Versioned = true,
            RemovalPolicy = RemovalPolicy.DESTROY,
            AutoDeleteObjects = true // lab only
        });

        var loggingRole = new Role(this, "TransferLogging", new RoleProps
        {
            AssumedBy = new ServicePrincipal("transfer.amazonaws.com"),
            ManagedPolicies = [ManagedPolicy.FromAwsManagedPolicyName("service-role/AWSTransferLoggingAccess")]
        });

        var server = new CfnServer(this, "Sftp", new CfnServerProps
        {
            Protocols = ["SFTP"],
            Domain = "S3",
            EndpointType = "PUBLIC",
            IdentityProviderType = "SERVICE_MANAGED",
            SecurityPolicyName = "TransferSecurityPolicy-2024-01",
            LoggingRole = loggingRole.RoleArn
        });

        var partnerRole = new Role(this, "NorthwindRole", new RoleProps
        {
            AssumedBy = new ServicePrincipal("transfer.amazonaws.com")
        });
        partnerRole.AddToPolicy(new PolicyStatement(new PolicyStatementProps
        {
            Actions = ["s3:ListBucket"],
            Resources = [bucket.BucketArn],
            Conditions = new Dictionary<string, object>
            {
                ["StringLike"] = new Dictionary<string, object> { ["s3:prefix"] = new[] { "partners/northwind/*", "partners/northwind" } }
            }
        }));
        partnerRole.AddToPolicy(new PolicyStatement(new PolicyStatementProps
        {
            Actions = ["s3:PutObject"],
            Resources = [bucket.ArnForObjects("partners/northwind/in/*")]
        }));
        partnerRole.AddToPolicy(new PolicyStatement(new PolicyStatementProps
        {
            Actions = ["s3:GetObject", "s3:GetObjectVersion"],
            Resources = [bucket.ArnForObjects("partners/northwind/out/*")]
        }));
        bucket.EncryptionKey?.GrantEncryptDecrypt(partnerRole);

        new CfnUser(this, "Northwind", new CfnUserProps
        {
            ServerId = server.AttrServerId,
            UserName = "northwind",
            Role = partnerRole.RoleArn,
            HomeDirectoryType = "LOGICAL",
            HomeDirectoryMappings = new object[]
            {
                new CfnUser.HomeDirectoryMapEntryProperty { Entry = "/", Target = $"/{bucket.BucketName}/partners/northwind" }
            },
            SshPublicKeys = [System.IO.File.ReadAllText("keys/hl_northwind_ed25519.pub").Trim()]
        });

        new CfnOutput(this, "SftpHost", new CfnOutputProps
        {
            Value = $"{server.AttrServerId}.server.transfer.{Region}.amazonaws.com"
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m04-l21");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        Amazon.CDK.Tags.Of(this).Add("Owner", "you");
    }
}

BucketEncryption.KMS_MANAGED uses the AWS managed aws/s3 key, so EncryptionKey is null and no grant is needed; with a customer managed key, the grant line gives the partner role permission to use it.

Complete code

A .NET 10 console app with three commands:

  • local (default, no network): generate a throwaway PGP key pair, encrypt a fictional payroll CSV, write a SHA-256 manifest, then verify and decrypt it, and show that a tampered file is rejected.
  • sftp-pull: download files and manifests from an SFTP server (local Docker or Transfer Family) with host-key pinning.
  • s3-archive: upload a verified file to S3 with an S3-verified SHA-256 checksum and KMS encryption.
src/Harbourline.PartnerGateway.Files/Harbourline.PartnerGateway.Files.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="AWSSDK.S3" Version="4.1.1" />
    <PackageReference Include="BouncyCastle.Cryptography" Version="2.7.0" />
    <PackageReference Include="SSH.NET" Version="2026.0.1" />
  </ItemGroup>
</Project>
src/Harbourline.PartnerGateway.Files/Program.cs
using System.Security.Cryptography;
using System.Text;
using Amazon.S3;
using Amazon.S3.Model;
using Org.BouncyCastle.Bcpg;
using Org.BouncyCastle.Bcpg.OpenPgp;
using Org.BouncyCastle.Crypto.Generators;
using Org.BouncyCastle.Crypto.Parameters;
using Org.BouncyCastle.Security;
using Renci.SshNet;

var command = args.FirstOrDefault() ?? "local";
var work = Directory.CreateTempSubdirectory("hl-files-");

switch (command)
{
    case "local":
        RunLocal(work.FullName);
        break;
    case "sftp-pull":
        // dotnet run -- sftp-pull localhost 2222 northwind ~/.ssh/hl_northwind_ed25519 <SHA256 host key fingerprint>
        SftpPull(host: args[1], port: int.Parse(args[2]), user: args[3], keyPath: args[4], pinnedFingerprint: args[5], target: work.FullName);
        break;
    case "s3-archive":
        // dotnet run -- s3-archive <bucket> <local file>   (uses the learn-sandbox profile via AWS_PROFILE)
        await ArchiveToS3Async(bucket: args[1], path: args[2]);
        break;
    default:
        Console.Error.WriteLine("Commands: local | sftp-pull | s3-archive");
        return 1;
}
return 0;

static void RunLocal(string dir)
{
    // 1. Harbourline's PGP key pair (in production: generated once, private key in Secrets Manager)
    var bank = Pgp.GenerateRsaKeyPair();

    // 2. Partner side: encrypt the payroll file to the bank's public key and write a manifest
    var csv = """
        end_to_end_id,creditor_name,creditor_iban,amount,currency
        SALARY-NOV-2026-0001,Alex Example,GB33BUKB20201555555555,2150.00,GBP
        SALARY-NOV-2026-0002,Sam Sample,GB82WEST12345698765432,1987.45,GBP
        """;
    var encryptedPath = Path.Combine(dir, "PAYROLL_20261118_001.csv.pgp");
    using (var output = File.Create(encryptedPath))
        Pgp.Encrypt(new MemoryStream(Encoding.UTF8.GetBytes(csv)), output, bank.PublicKey, "PAYROLL_20261118_001.csv");
    Manifest.Write(encryptedPath);
    Console.WriteLine($"Partner wrote {Path.GetFileName(encryptedPath)} ({new FileInfo(encryptedPath).Length} bytes) + manifest");

    // 3. Bank side: verify manifest BEFORE decrypting, then decrypt
    Process(encryptedPath, bank.PrivateKey);

    // 4. Tamper with one byte and try again
    var bytes = File.ReadAllBytes(encryptedPath);
    bytes[^10] ^= 0xFF;
    File.WriteAllBytes(encryptedPath, bytes);
    Process(encryptedPath, bank.PrivateKey);
}

static void Process(string encryptedPath, PgpPrivateKey key)
{
    if (!Manifest.Verify(encryptedPath, out var reason))
    {
        Console.WriteLine($"REJECTED (quarantine): {reason}");
        return;
    }
    using var input = File.OpenRead(encryptedPath);
    using var plain = new MemoryStream();
    try
    {
        Pgp.Decrypt(input, plain, key);
    }
    catch (PgpException ex)
    {
        Console.WriteLine($"REJECTED (quarantine): PGP error {ex.Message}");
        return;
    }
    var lines = Encoding.UTF8.GetString(plain.ToArray()).Split('\n', StringSplitOptions.RemoveEmptyEntries);
    Console.WriteLine($"ACCEPTED: {lines.Length - 1} payment records, dedupe key {Manifest.Sha256Hex(encryptedPath)[..16]}…");
}

static void SftpPull(string host, int port, string user, string keyPath, string pinnedFingerprint, string target)
{
    using var key = new PrivateKeyFile(keyPath);
    using var client = new SftpClient(host, port, user, key);
    client.HostKeyReceived += (_, e) =>
    {
        // Pin the server's host key: refuse to talk to anything else (prevents man-in-the-middle).
        e.CanTrust = string.Equals(e.FingerPrintSHA256, pinnedFingerprint, StringComparison.Ordinal);
        if (!e.CanTrust) Console.Error.WriteLine($"Host key mismatch: got {e.FingerPrintSHA256}");
    };
    client.Connect();

    // Only pick up data files whose manifest has also arrived (manifest is uploaded last).
    var names = client.ListDirectory("/in").Where(f => f.IsRegularFile).Select(f => f.Name).ToHashSet();
    foreach (var data in names.Where(n => n.EndsWith(".pgp", StringComparison.Ordinal) && names.Contains(n + ".sha256")))
    {
        foreach (var name in new[] { data, data + ".sha256" })
        {
            using var fs = File.Create(Path.Combine(target, name));
            client.DownloadFile($"/in/{name}", fs);
        }
        var ok = Manifest.Verify(Path.Combine(target, data), out var reason);
        Console.WriteLine($"{data}: {(ok ? "manifest OK" : "REJECTED " + reason)}");
    }
    client.Disconnect();
}

static async Task ArchiveToS3Async(string bucket, string path)
{
    var hashB64 = Convert.ToBase64String(Manifest.Sha256(path));
    using var s3 = new AmazonS3Client(Amazon.RegionEndpoint.USEast1);
    await using var stream = File.OpenRead(path);
    var response = await s3.PutObjectAsync(new PutObjectRequest
    {
        BucketName = bucket,
        Key = $"archive/northwind/{DateTime.UtcNow:yyyy/MM/dd}/{Path.GetFileName(path)}",
        InputStream = stream,
        ChecksumAlgorithm = ChecksumAlgorithm.SHA256,
        ChecksumSHA256 = hashB64,                         // S3 rejects the upload if the bytes don't match
        ServerSideEncryptionMethod = ServerSideEncryptionMethod.AWSKMS
    });
    Console.WriteLine($"Archived, version {response.VersionId}, S3 checksum {response.ChecksumSHA256}");
}

// ===================== Manifest (SHA-256) =====================
public static class Manifest
{
    public static byte[] Sha256(string path)
    {
        using var fs = File.OpenRead(path);
        return SHA256.HashData(fs);                    // streams; constant memory
    }

    public static string Sha256Hex(string path) => Convert.ToHexStringLower(Sha256(path));

    // Format: "<hex>  <filename>\n", the same as sha256sum output, so partners can produce it with standard tools.
    public static void Write(string path) =>
        File.WriteAllText(path + ".sha256", $"{Sha256Hex(path)}  {Path.GetFileName(path)}\n");

    public static bool Verify(string path, out string reason)
    {
        var manifestPath = path + ".sha256";
        if (!File.Exists(manifestPath)) { reason = "manifest missing (file may be incomplete)"; return false; }

        var parts = File.ReadAllText(manifestPath).Trim().Split("  ", 2);
        if (parts.Length != 2 || parts[1] != Path.GetFileName(path)) { reason = "manifest malformed or names another file"; return false; }

        byte[] expected;
        try { expected = Convert.FromHexString(parts[0]); }
        catch (FormatException) { reason = "manifest hash is not hex"; return false; }

        if (!CryptographicOperations.FixedTimeEquals(expected, Sha256(path))) { reason = "SHA-256 mismatch"; return false; }
        reason = "ok";
        return true;
    }
}

// ===================== OpenPGP with BouncyCastle =====================
public static class Pgp
{
    public static PgpKeyPair GenerateRsaKeyPair()
    {
        var gen = new RsaKeyPairGenerator();
        gen.Init(new RsaKeyGenerationParameters(Org.BouncyCastle.Math.BigInteger.ValueOf(0x10001), new SecureRandom(), 3072, 80));
        return new PgpKeyPair(PublicKeyAlgorithmTag.RsaGeneral, gen.GenerateKeyPair(), DateTime.UtcNow);
    }

    public static void Encrypt(Stream plain, Stream output, PgpPublicKey recipient, string fileName)
    {
        var encGen = new PgpEncryptedDataGenerator(SymmetricKeyAlgorithmTag.Aes256, withIntegrityPacket: true, new SecureRandom());
        encGen.AddMethod(recipient);
        using var encOut = encGen.Open(output, new byte[1 << 16]);
        var compGen = new PgpCompressedDataGenerator(CompressionAlgorithmTag.Zip);
        using var compOut = compGen.Open(encOut);
        var litGen = new PgpLiteralDataGenerator();
        using var litOut = litGen.Open(compOut, PgpLiteralData.Binary, fileName, DateTime.UtcNow, new byte[1 << 16]);
        plain.CopyTo(litOut);
    }

    public static void Decrypt(Stream encrypted, Stream output, PgpPrivateKey key)
    {
        var factory = new PgpObjectFactory(PgpUtilities.GetDecoderStream(encrypted));
        var first = factory.NextPgpObject();
        var list = first as PgpEncryptedDataList ?? factory.NextPgpObject() as PgpEncryptedDataList
                   ?? throw new PgpException("No encrypted data found");

        var data = list.GetEncryptedDataObjects().OfType<PgpPublicKeyEncryptedData>()
                       .FirstOrDefault(d => d.KeyId == key.KeyId)
                   ?? throw new PgpException("File was not encrypted to our key");

        if (!data.IsIntegrityProtected()) throw new PgpException("Refusing file without integrity protection");

        using (var clear = data.GetDataStream(key))
        {
            var message = new PgpObjectFactory(clear).NextPgpObject();
            if (message is PgpCompressedData compressed)
                message = new PgpObjectFactory(compressed.GetDataStream()).NextPgpObject();
            if (message is not PgpLiteralData literal) throw new PgpException("Unexpected PGP content (signed-only or unknown)");
            literal.GetInputStream().CopyTo(output);
        }

        if (!data.Verify()) throw new PgpException("Integrity check failed: file modified");
    }
}

Run it locally:

cd src/Harbourline.PartnerGateway.Files
dotnet run
# Partner wrote PAYROLL_20261118_001.csv.pgp (… bytes) + manifest
# ACCEPTED: 2 payment records, dedupe key 3f1c9a…
# REJECTED (quarantine): SHA-256 mismatch

Free local SFTP server. Run an OpenSSH-based SFTP server in Docker and use the same sftp-pull command:

# Create a key pair for the fictional partner
ssh-keygen -t ed25519 -f ~/.ssh/hl_northwind_ed25519 -N "" -C "northwind@example.com"

# atmoz/sftp: user northwind, no password, uid 1001, folder "in"; public key mounted for key auth
docker run --rm -d --name hl-sftp -p 2222:22 \
  -v ~/.ssh/hl_northwind_ed25519.pub:/home/northwind/.ssh/keys/id.pub:ro \
  atmoz/sftp:alpine northwind::1001::in

# Get the server's host key fingerprint to pin (SHA256, base64)
ssh-keyscan -p 2222 localhost 2>/dev/null | ssh-keygen -lf - -E sha256

# Upload as the partner, then pull as the bank
sftp -P 2222 -i ~/.ssh/hl_northwind_ed25519 northwind@localhost
dotnet run -- sftp-pull localhost 2222 northwind ~/.ssh/hl_northwind_ed25519 <fingerprint-without-SHA256:-prefix>
docker rm -f hl-sftp

The atmoz/sftp image and its user:pass:uid:gid:dir argument format come from the image's documentation ⚠ VERIFY the tag and options on Docker Hub. SSH.NET's FingerPrintSHA256 is base64 without the SHA256: prefix that ssh-keygen prints.

Code explained line by line

  1. Directory.CreateTempSubdirectory gives each run an isolated working folder; never process partner files in a shared directory.
  2. Pgp.GenerateRsaKeyPair creates a 3,072-bit RSA key for the demo. In production, the bank generates its key once (often with a key ceremony), publishes the public key to partners and stores the private key in Secrets Manager encrypted with a KMS key.
  3. PgpEncryptedDataGenerator(..., withIntegrityPacket: true, ...) encrypts with AES-256 and adds a modification detection code. Without it, PGP ciphertext can be altered undetectably.
  4. The literal → compressed → encrypted layering is the standard OpenPGP message structure. using declarations dispose in reverse order, which finishes the literal packet first, then compression, then encryption.
  5. Manifest.Write uses the sha256sum format (hex, two spaces, file name) so partners can create it with standard tools.
  6. Manifest.Verify checks, in order: manifest present (completeness signal), manifest names this file (prevents a valid manifest for file A being used for file B), hash parses, hash matches in constant time.
  7. Process verifies the manifest before decrypting: never spend CPU on, or expose your private key to, a file you have not checked.
  8. Pgp.Decrypt finds the encrypted packet for our key ID, refuses data without integrity protection, decompresses if needed, and only accepts a literal data packet.
  9. data.Verify() runs after the whole stream has been read, because the integrity check covers all the data. A tampered file throws here (or earlier, while parsing).
  10. SftpPull pins the server's host key fingerprint through HostKeyReceived. Without pinning, SSH.NET would trust any host key and a man-in-the-middle could impersonate the server.
  11. The download loop only picks data files whose .sha256 manifest exists, which is how the "partial file" problem from the opening scenario is avoided.
  12. ArchiveToS3Async sends ChecksumAlgorithm.SHA256 and the precomputed ChecksumSHA256; S3 calculates the checksum of what it received and rejects the request if they differ. The checksum is stored with the object and can be retrieved later with --checksum-mode ENABLED.
  13. ServerSideEncryptionMethod.AWSKMS encrypts at rest with KMS; the bucket's default encryption should enforce this anyway.

Security considerations

⚠ Learning shortcut — not for production

The demo generates the bank's PGP key pair in memory on every run and never signs or verifies partner signatures. In production: the private key lives in Secrets Manager (or an HSM-backed solution), only the processor role can read it, keys have documented expiry and rotation with overlap, and the contract requires partners to sign files so you can verify who created them, not just who uploaded them.

⚠ Learning shortcut — not for production

The CLI and IaC examples use a PUBLIC endpoint and AutoDeleteObjects/force_destroy for easy teardown. Production partner gateways use a VPC internet-facing endpoint with Elastic IPs and security groups restricted to partner CIDRs, and archive buckets with Object Lock that cannot be force-deleted.

  • Keys, not passwords. Service-managed users with SSH public keys only. Rotate keys with overlap (users can have several keys). Remove keys when partner staff leave.
  • Least-privilege per partner. One IAM role (or session policy) per partner, limited to its prefix; no delete permission.
  • Transport security policy. Keep the server on a current TransferSecurityPolicy-* to exclude weak ciphers and key exchange algorithms; review when AWS publishes new policies.
  • Integrity beyond the channel. SSH protects the transfer only. PGP integrity, manifests and S3 checksums protect the file at rest and across hops.
  • Scan before processing. Treat every file as hostile until malware scanning and validation pass.
  • Logging. Transfer Family logs connections and file operations to CloudWatch; keep them for investigations, without logging file contents.
  • Swift. If files feed a Swift interface, the Swift CSP controls (segregated secure zone, operator authentication, integrity of transactions) apply to that environment; involve your Swift security officer ⚠ VERIFY current CSP control set.
  • Personal data. Payroll and payment files are full of personal data: encrypt, minimise retention outside the archive, restrict access, and include them in your records of processing.

Banking use case

Harbourline's Partner Gateway (fictional) handles 40 corporate customers and 6 service providers:

  • Inbound: payroll and supplier-payment files (CSV and pain.001), PGP-encrypted and signed, with SHA-256 manifests. GuardDuty scans; the .NET processor on ECS verifies, decrypts, validates (record count and total in the trailer must match), creates payments idempotently and writes a pain.002 acknowledgement to /out within 5 minutes of the cut-off.
  • Outbound: end-of-day camt.053 statements for each corporate customer, PGP-encrypted to the customer's public key, delivered to their /out folder; for customers who host their own SFTP, SFTP connectors push the file instead.
  • Card settlement: a daily settlement file from the card processor arrives over a VPC internal endpoint across Direct Connect.
  • Controls: dedupe table on partner + SHA-256, Object Lock archive for seven years ⚠ VERIFY your records retention policy, alarms on "file expected but not received by cut-off" (a scheduled check) and on any quarantine event.

Common mistakes

  • Processing on "file created" without a completeness signal. Partial uploads get processed. Use manifests or a rename-on-complete convention.
  • Deduping on file name. Partners reuse names, or resend with a new name. Dedupe on content hash plus partner.
  • Stopping instead of deleting the server. Billing continues.
  • One IAM role for all partners. One partner can read another's files.
  • Accepting password authentication "just for this partner".
  • Not pinning host keys in outbound SFTP clients.
  • PGP without integrity protection or signatures. Encryption alone does not prove origin or integrity.
  • Keeping decrypted files in the landing bucket. Decrypt in memory or to a separate, tightly controlled prefix, and delete after processing.

Troubleshooting

SymptomLikely causeFix
Permission denied (publickey)Wrong key, key not added to user, or wrong user nameCheck the user's SSH keys in Transfer Family; test with sftp -v
Login works but ls fails / access deniedRole missing s3:ListBucket or prefix condition wrongFix the s3:prefix condition to include the home folder
Upload fails with access denied on KMSBucket uses a customer managed key the user role can't useGrant kms:GenerateDataKey and kms:Decrypt to the partner role
Partner can't connect from their networkSecurity group or partner firewallAllow partner CIDR on TCP 22; give partner your Elastic IPs
Integrity check failedFile corrupted or tampered, or uploaded in ASCII mode by a legacy clientAsk the partner to resend in binary mode; compare manifests
File was not encrypted to our keyPartner used an old or wrong public keySend the current public key and its fingerprint through an authenticated channel
Workflow decrypt step failsSecret not found under the expected name, or anonymous recipientStore key per Transfer Family naming rules; encrypt with an explicit recipient
Same file processed twiceDedupe on name only, or no conditional writeConditional PutItem on partner + SHA-256

Cost and cleanup

  • Free path: the local commands, Docker SFTP server and PGP code cost nothing.
  • Transfer Family (paid): USD 0.30 per protocol-hour while the server exists + USD 0.04/GB. One-hour lab: about USD 0.30. Managed workflow decryption: USD 0.10/GB. SFTP connectors: per call plus per GB ⚠ VERIFY current connector pricing. VPC internet-facing endpoints also use Elastic IPs, which are charged per hour for public IPv4 addresses.
  • GuardDuty Malware Protection for S3: charged per GB scanned and per object evaluated; skip it in the lab.

Teardown:

aws transfer list-servers --profile learn-sandbox --region us-east-1
aws transfer delete-server --server-id s-EXAMPLE1234567890 --profile learn-sandbox --region us-east-1
aws s3 rm s3://hl-m04-files-111122223333 --recursive --profile learn-sandbox
# Versioned bucket: delete all versions and delete markers (console "Empty") before:
aws s3api delete-bucket --bucket hl-m04-files-111122223333 --profile learn-sandbox
aws iam delete-role-policy --role-name hl-m04-sftp-northwind --policy-name northwind --profile learn-sandbox
aws iam delete-role --role-name hl-m04-sftp-northwind --profile learn-sandbox
docker rm -f hl-sftp
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --query 'ResourceTagMappingList[].ResourceARN' --profile learn-sandbox --region us-east-1

Confirm list-servers returns an empty list: that is the line item that bills.

Exercises

  • 🟢 Guided: Run dotnet run and observe accepted and rejected outcomes. Then delete the .sha256 file before processing and confirm the reason "manifest missing".
  • 🟡 Challenge: Add PGP signing by the partner and signature verification by the bank: generate a second key pair for Northwind, sign with PgpSignatureGenerator (one-pass signature) inside the encrypted message, and reject files signed by any other key.
  • 🔴 Live problem: A partner says: "We uploaded PAYROLL_20261120_001 at 05:52, but you say you never received it." Your landing bucket has no such object; Transfer Family logs show a successful login at 05:51 from the partner's IP and an OPEN of /in/PAYROLL_20261120_001.csv.pgp but no CLOSE. Explain what probably happened, what evidence you would gather, and how to make this situation visible automatically next time.

Assignment

Write a one-page partner file contract template for Harbourline (naming, format, encryption and signing, manifest, cut-offs, acknowledgements, resend rules, key rotation, contacts) and a matching onboarding checklist (user, role, home mapping, key fingerprint verification call, test file, go-live). Add the Terraform for one partner user to your portfolio repository.

Solutions

Knowledge check

Interview questions

  • "How do you make batch file processing exactly-once?" — Completeness signal (manifest), dedupe key on partner plus content hash with a conditional write, idempotent downstream operations keyed by EndToEndId, immutable archive, quarantine for failures.
  • "Why PGP if SFTP is already encrypted?" — SFTP protects only the transfer; PGP protects the file at rest across every hop and, with signatures, proves who created it.
  • "How would you onboard a new SFTP partner on AWS?" — Contract, partner sends SSH public key and IPs, IaC creates user, role and home mapping, security group rule, verify key fingerprints out-of-band, exchange PGP public keys, test file, go-live with monitoring.
  • "What does Transfer Family cost and how do you control it?" — Per protocol-hour while the server exists plus per-GB; enable only needed protocols, share one server across partners, delete non-production servers.
  • "Where does Swift fit with AWS?" — The Swift connection is a controlled environment under CSP; AWS hosts the payment hub and integration around it, exchanging messages and files with the Swift interface through controlled channels.

Certification practice

Completion checklist

  • I can explain the four questions of any file integration (who, intact, confidential, once)
  • I can choose a Transfer Family endpoint type and explain the billing trap
  • I ran the PGP + manifest demo and saw a tampered file rejected
  • I ran (or understand) the local SFTP flow with host-key pinning
  • I can describe dedupe on partner + SHA-256 with a conditional write
  • I deleted any Transfer Family server and confirmed nothing tagged Project=cea-lab remains

Key takeaways

  • Files remain core to banking; the risks are completeness, integrity, origin and double processing.
  • Transfer Family gives managed SFTP into S3 with per-partner jails; it bills hourly until deleted.
  • PGP protects files beyond the channel; manifests and S3 checksums prove the bytes; signatures prove the author.
  • Process only scanned, verified files, dedupe on content hash, and archive originals immutably.
  • Swift and scheme connections have their own security regimes; AWS hosts the integration around them.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page