SWIFT and file-based integration: SFTP with AWS Transfer Family, PGP and checksums
Exchange batch payment and statement files with partners safely: understand where Swift and scheme file channels fit, run SFTP into S3 with AWS Transfer Family (or a free local SFTP server), encrypt and decrypt with OpenPGP in C#, verify SHA-256 manifests and S3 checksums, and process each file exactly once.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.S3 4.1.x) · SSH.NET 2026.0.1 · BouncyCastle.Cryptography 2.7.0 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Cost: AWS Transfer Family bills per hour while the endpoint exists
An SFTP endpoint costs about USD 0.30 per hour per enabled protocol (us-east-1, verified 2026-11-18), roughly USD 216 per month if left running, plus USD 0.04 per GB transferred. Stopping a server does not stop the hourly charge; only deleting it does ⚠ VERIFY on the pricing page. The free path in this lesson uses an SFTP server in Docker on your laptop with the same C# code. If you do deploy Transfer Family, deploy and delete it within one hour (about USD 0.30 plus pennies).
Learning objectives
By the end of this lesson you can:
- Explain why banks still exchange files (batch payments, statements, reconciliation, regulatory reports), and where Swift FileAct, scheme gateways and partner SFTP fit, at a high level.
- Design an SFTP-to-S3 integration with AWS Transfer Family: endpoint types (public, VPC internal, VPC internet-facing with Elastic IPs), service-managed users with SSH keys, logical home directories, scoped IAM roles, security policies and logging.
- Encrypt and decrypt files with OpenPGP in C# using BouncyCastle, and know when to use Transfer Family managed workflows to decrypt instead.
- Verify file integrity with SHA-256 manifests and S3 additional checksums, and process each file exactly once with quarantine, idempotency and immutable storage.
- Run the whole flow free against a local SFTP server in Docker.
Prerequisites
- ISO 20022 message handling — many of the files you exchange are pain.001 or camt.053
- Calling partner APIs safely — the same partner-risk thinking applies to files
- S3 protection, Object Lock and Macie
- Module 2: Networking — VPCs, Elastic IPs, security groups
- Tools: .NET 10 SDK, Docker Desktop, an SSH key pair (
ssh-keygen), AWS CLI v2 with thelearn-sandboxprofile (only for the optional deploy)
Why this matters in banking
Every weekday at 06:00, Northwind Payroll Ltd (fictional) uploads a salary file for 4,812 employees to Harbourline Bank. One Thursday the upload is interrupted at 70%, Northwind's script retries, and Harbourline's poller picks up both the partial file and the complete one. The partial file fails parsing halfway, after 3,100 payments were already created; the complete file then creates all 4,812. Some employees are paid twice, others once, and the reconciliation team spends two days unpicking it.
In another bank, a partner's SFTP password was shared in an email thread, and an attacker uploaded a payment file with modified beneficiary IBANs. It was processed because "it came through the secure channel".
File-based integration is old, unglamorous and still carries enormous value: bulk payments, statements, card settlement, direct debits and regulatory returns. Getting authentication, integrity, completeness and exactly-once processing right is the job.
The idea in plain language
A file integration has four questions to answer:
- Who sent it? Authentication of the channel (SSH keys for SFTP, certificates for AS2, the network's own authentication for Swift) and, ideally, a signature on the file itself.
- Is it complete and unaltered? A checksum (SHA-256) supplied separately, compared with what you received; PGP integrity protection; size and record counts in trailers.
- Can anyone else read it? Encryption in transit (SSH/TLS) and, for many partners, encryption of the file itself with PGP, so it stays protected in every folder and mailbox it passes through.
- Will we process it exactly once? A "file is ready" signal, a dedupe key (checksum plus partner and file name), immutable storage of the original and a quarantine area for anything suspicious.
Where Swift fits. Swift is the bank-to-bank network. Messages travel as FIN (MT) or ISO 20022 (MX) messages; FileAct is Swift's service for transferring files between institutions. Banks connect through Swift interfaces (Alliance Access/Gateway) run on their own infrastructure, a service bureau, or Swift-hosted options such as Alliance Lite2 ⚠ VERIFY current product names and hosting options. On AWS, you usually build the integration around the Swift connection: files and messages flow between your payment hub and the Swift interface through S3, queues or SFTP, under Swift's Customer Security Programme (CSP) controls, which your Swift security officer owns. This lesson focuses on the SFTP pattern because it is the most common partner channel and the same controls apply everywhere.
Analogy: Think of a secure document drop at a bank branch. SFTP is the locked drop box with a key per courier (SSH key). PGP is the sealed, tamper-evident envelope inside the box, which only the intended recipient can open. The checksum manifest is the courier's signed delivery note listing exactly what is in the envelope. Exactly-once processing is the clerk who stamps each envelope "processed" with its reference number and refuses to process a second envelope with the same number.
What problem the AWS services solve
| Need | AWS building block | Honest limitations |
|---|---|---|
| Managed SFTP/FTPS/AS2 endpoint writing to S3 or EFS | AWS Transfer Family server | Hourly charge per protocol while it exists; you still own users, keys, partner onboarding |
| Fixed IPs partners can allowlist, plus source-IP filtering | Transfer Family VPC internet-facing endpoint with Elastic IPs and security groups | Needs a VPC with public subnets; up to three AZs |
| Per-partner folder isolation | Logical home directories and a per-user IAM role or session policy | Mappings are per user; automate them in IaC |
| Decrypt PGP on upload | Transfer Family managed workflows with a decrypt step (keys in Secrets Manager) | Charged per GB decrypted; keys must be stored under a specific secret naming scheme |
| Push files to partners' SFTP servers | Transfer Family SFTP connectors | Charged per call and per GB |
| Event-driven processing | S3 event notifications or EventBridge → SQS → Lambda/ECS | You implement dedupe and ordering |
| Malware scanning | GuardDuty Malware Protection for S3 (tags objects with a scan result) | Per-GB and per-object charges; scan before processing |
| Immutable originals | S3 versioning + Object Lock | Retention cannot be shortened in compliance mode |
Transfer Family itself does not know what a "payment file" is. Validation, dedupe and business processing are your code.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Blob Storage SFTP support | AWS Transfer Family SFTP with S3 domain | Transfer Family is a separate service with its own endpoint and hourly price; supports FTPS, AS2, connectors and workflows |
| SFTP server on a Windows VM (e.g. with OpenSSH) | Transfer Family, or SFTP on EC2 if you need full control | No patching or HA to build with Transfer Family |
| Logic Apps SFTP connector / BizTalk file adapters | Transfer Family SFTP connectors, Lambda/ECS processors | Write the processing logic in .NET |
Event Grid BlobCreated | S3 event notifications / EventBridge | Same idea |
| Defender for Storage malware scanning | GuardDuty Malware Protection for S3 | Results as object tags and findings |
| Gpg4win / BouncyCastle in .NET | BouncyCastle in .NET (same library), or the managed workflow decrypt step | Code transfers unchanged |
Architecture
Security-boundary view.
- Partner zone: authenticated only by SSH public key (no passwords), from allowlisted source IPs; can list and write only
/in, read only/out. - Landing zone: S3 prefix where nothing is trusted yet. The processor never reads a file until malware scanning and the manifest check pass.
- Processing zone: private subnets; IAM role can read landing, write archive/quarantine/out, read one Secrets Manager secret (PGP private key) and decrypt with one KMS key.
- Archive: Object Lock; nobody (including the processor) can delete originals during the retention period.
Step-by-step implementation
Agree the file contract with the partner
Write it down before any infrastructure: file naming (PAYROLL_YYYYMMDD_NNN.csv.pgp), format (CSV, ISO 20022 pain.001, Bacs Standard 18 and so on), encryption (PGP to Harbourline's public key, signed with the partner's key), the manifest (<file>.sha256 containing the SHA-256 and file name, uploaded after the data file), cut-off times, and the acknowledgement Harbourline sends back. The manifest-after-data rule is what solves the "partial file" problem: you only process when the manifest exists and matches.
Create the Transfer Family server
Choose SFTP only (each extra protocol is billed). Choose the endpoint type:
- Public: simplest; AWS-managed IPs that can change, no security groups. Fine for experiments.
- VPC, internal: reachable only from your network (VPN/Direct Connect); for internal systems such as a core banking batch server.
- VPC, internet-facing: Elastic IPs (stable addresses partners can allowlist in their firewalls) and security groups (you allowlist partner IPs).
Pick a current security policy (the default is TransferSecurityPolicy-2024-01), enable CloudWatch logging with a logging role, and keep identity service-managed (SSH keys stored by Transfer Family) unless you need a custom identity provider.
Create one user per partner with a jailed home
For each partner: an IAM role that allows s3:PutObject on partners/<name>/in/*, s3:GetObject on partners/<name>/out/*, and s3:ListBucket limited by prefix; a logical home directory mapping / → /bucket/partners/<name>; and the partner's SSH public key. Never generate keys for partners; they send you public keys and keep private keys themselves.
Scan, then signal "ready"
Enable GuardDuty Malware Protection for the landing bucket (or prefix). Route "scan result = no threats found" events (or the object tag) to the processing queue, so nothing is processed before scanning. Files that fail scanning go to quarantine automatically.
Verify the manifest and checksums
When <file>.sha256 arrives, read it, stream the data file from S3 and compute SHA-256; compare in constant time. Additionally, upload to the archive with S3 additional checksums (ChecksumAlgorithm = SHA256 with the value you computed), so S3 itself verifies the bytes it stores.
Decrypt and verify the signature
Decrypt with Harbourline's PGP private key (in Secrets Manager, encrypted with KMS) and, if the contract requires, verify the partner's PGP signature. Use the integrity check (modification detection) that modern PGP encryption includes, and reject files without it.
Process exactly once
Compute the dedupe key: partner + SHA-256 of the encrypted file. Insert it into a DynamoDB table with a conditional write (attribute_not_exists). If it already exists, it is a duplicate: acknowledge, do not process. Then validate the content (record counts, totals, IBAN checksums, EndToEndId uniqueness), create payments idempotently, archive the original and write an acknowledgement file to /out.
AWS Console instructions
- Server: Console → AWS Transfer Family → Servers → Create server → protocols SFTP → identity provider Service managed → endpoint VPC hosted, access Internet Facing, choose VPC, up to three public subnets and an Elastic IP per subnet, and a security group allowing TCP 22 from partner CIDRs only → domain Amazon S3 → CloudWatch logging: create or choose a logging role → Cryptographic algorithm options: keep the latest security policy → add tags → Create server.
- User: server → Users → Add user → user name
northwind→ rolehl-m04-sftp-northwind→ Home directory: bucket and folderpartners/northwind, tick Restricted (this creates a logical/mapping) → paste the partner's SSH public key → Add. - Workflow (optional): Workflows → Create workflow → add a Decrypt file step with destination
decrypted/${transfer:UserName}/→ attach to the server under Managed workflows with an execution role. - Delete: server → Actions → Delete. Deletion, not stopping, ends billing.
AWS CLI
PROFILE=learn-sandbox
REGION=us-east-1
BUCKET=hl-m04-files-111122223333
# Bucket for landing/archive (private, versioned)
aws s3api create-bucket --bucket $BUCKET --profile $PROFILE --region $REGION
aws s3api put-public-access-block --bucket $BUCKET \
--public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true \
--profile $PROFILE --region $REGION
aws s3api put-bucket-versioning --bucket $BUCKET --versioning-configuration Status=Enabled --profile $PROFILE --region $REGION
# SFTP server (PUBLIC endpoint keeps the lab simple and avoids Elastic IP charges)
SERVER_ID=$(aws transfer create-server \
--protocols SFTP --domain S3 --endpoint-type PUBLIC \
--identity-provider-type SERVICE_MANAGED \
--security-policy-name TransferSecurityPolicy-2024-01 \
--logging-role arn:aws:iam::111122223333:role/hl-m04-transfer-logging \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m04-l21 Key=Environment,Value=learn Key=Owner,Value=you \
--query ServerId --output text --profile $PROFILE --region $REGION)
echo "Server: $SERVER_ID (billing has started)"
# Partner user jailed to partners/northwind with a logical home directory and an SSH public key
aws transfer create-user --server-id $SERVER_ID --user-name northwind \
--role arn:aws:iam::111122223333:role/hl-m04-sftp-northwind \
--home-directory-type LOGICAL \
--home-directory-mappings "[{\"Entry\":\"/\",\"Target\":\"/$BUCKET/partners/northwind\"}]" \
--ssh-public-key-body "$(cat ~/.ssh/hl_northwind_ed25519.pub)" \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m04-l21 \
--profile $PROFILE --region $REGION
# Connect as the partner and upload (endpoint hostname format: SERVERID.server.transfer.REGION.amazonaws.com)
sftp -i ~/.ssh/hl_northwind_ed25519 northwind@$SERVER_ID.server.transfer.$REGION.amazonaws.com <<'EOF'
mkdir in
put PAYROLL_20261118_001.csv.pgp in/
put PAYROLL_20261118_001.csv.pgp.sha256 in/
ls -l in
EOF
# Verify what landed in S3, including the stored SHA-256 checksum if one was supplied
aws s3api head-object --bucket $BUCKET --key partners/northwind/in/PAYROLL_20261118_001.csv.pgp \
--checksum-mode ENABLED --profile $PROFILE --region $REGION
# DELETE the server when done (stop does not end billing)
aws transfer delete-server --server-id $SERVER_ID --profile $PROFILE --region $REGIONThe IAM role for the user must trust transfer.amazonaws.com. Its policy (least privilege):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListOwnFolder",
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::hl-m04-files-111122223333",
"Condition": { "StringLike": { "s3:prefix": ["partners/northwind/*", "partners/northwind"] } }
},
{
"Sid": "UploadToIn",
"Effect": "Allow",
"Action": ["s3:PutObject"],
"Resource": "arn:aws:s3:::hl-m04-files-111122223333/partners/northwind/in/*"
},
{
"Sid": "DownloadFromOut",
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:GetObjectVersion"],
"Resource": "arn:aws:s3:::hl-m04-files-111122223333/partners/northwind/out/*"
}
]
}Partners cannot delete or overwrite processed files because there is no s3:DeleteObject and the processor moves files out of in/ after archiving. SFTP clients sometimes need s3:GetObject on in/ to resume uploads; add it only if the partner's client requires it.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.S3;
using Amazon.CDK.AWS.Transfer;
using Constructs;
namespace Harbourline.PartnerGateway.Infra;
public sealed class PartnerSftpStack : Stack
{
public PartnerSftpStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
var bucket = new Bucket(this, "Files", new BucketProps
{
BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
Encryption = BucketEncryption.KMS_MANAGED,
EnforceSSL = true,
Versioned = true,
RemovalPolicy = RemovalPolicy.DESTROY,
AutoDeleteObjects = true // lab only
});
var loggingRole = new Role(this, "TransferLogging", new RoleProps
{
AssumedBy = new ServicePrincipal("transfer.amazonaws.com"),
ManagedPolicies = [ManagedPolicy.FromAwsManagedPolicyName("service-role/AWSTransferLoggingAccess")]
});
var server = new CfnServer(this, "Sftp", new CfnServerProps
{
Protocols = ["SFTP"],
Domain = "S3",
EndpointType = "PUBLIC",
IdentityProviderType = "SERVICE_MANAGED",
SecurityPolicyName = "TransferSecurityPolicy-2024-01",
LoggingRole = loggingRole.RoleArn
});
var partnerRole = new Role(this, "NorthwindRole", new RoleProps
{
AssumedBy = new ServicePrincipal("transfer.amazonaws.com")
});
partnerRole.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
Actions = ["s3:ListBucket"],
Resources = [bucket.BucketArn],
Conditions = new Dictionary<string, object>
{
["StringLike"] = new Dictionary<string, object> { ["s3:prefix"] = new[] { "partners/northwind/*", "partners/northwind" } }
}
}));
partnerRole.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
Actions = ["s3:PutObject"],
Resources = [bucket.ArnForObjects("partners/northwind/in/*")]
}));
partnerRole.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
Actions = ["s3:GetObject", "s3:GetObjectVersion"],
Resources = [bucket.ArnForObjects("partners/northwind/out/*")]
}));
bucket.EncryptionKey?.GrantEncryptDecrypt(partnerRole);
new CfnUser(this, "Northwind", new CfnUserProps
{
ServerId = server.AttrServerId,
UserName = "northwind",
Role = partnerRole.RoleArn,
HomeDirectoryType = "LOGICAL",
HomeDirectoryMappings = new object[]
{
new CfnUser.HomeDirectoryMapEntryProperty { Entry = "/", Target = $"/{bucket.BucketName}/partners/northwind" }
},
SshPublicKeys = [System.IO.File.ReadAllText("keys/hl_northwind_ed25519.pub").Trim()]
});
new CfnOutput(this, "SftpHost", new CfnOutputProps
{
Value = $"{server.AttrServerId}.server.transfer.{Region}.amazonaws.com"
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m04-l21");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "you");
}
}BucketEncryption.KMS_MANAGED uses the AWS managed aws/s3 key, so EncryptionKey is null and no grant is needed; with a customer managed key, the grant line gives the partner role permission to use it.
Complete code
A .NET 10 console app with three commands:
local(default, no network): generate a throwaway PGP key pair, encrypt a fictional payroll CSV, write a SHA-256 manifest, then verify and decrypt it, and show that a tampered file is rejected.sftp-pull: download files and manifests from an SFTP server (local Docker or Transfer Family) with host-key pinning.s3-archive: upload a verified file to S3 with an S3-verified SHA-256 checksum and KMS encryption.
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AWSSDK.S3" Version="4.1.1" />
<PackageReference Include="BouncyCastle.Cryptography" Version="2.7.0" />
<PackageReference Include="SSH.NET" Version="2026.0.1" />
</ItemGroup>
</Project>using System.Security.Cryptography;
using System.Text;
using Amazon.S3;
using Amazon.S3.Model;
using Org.BouncyCastle.Bcpg;
using Org.BouncyCastle.Bcpg.OpenPgp;
using Org.BouncyCastle.Crypto.Generators;
using Org.BouncyCastle.Crypto.Parameters;
using Org.BouncyCastle.Security;
using Renci.SshNet;
var command = args.FirstOrDefault() ?? "local";
var work = Directory.CreateTempSubdirectory("hl-files-");
switch (command)
{
case "local":
RunLocal(work.FullName);
break;
case "sftp-pull":
// dotnet run -- sftp-pull localhost 2222 northwind ~/.ssh/hl_northwind_ed25519 <SHA256 host key fingerprint>
SftpPull(host: args[1], port: int.Parse(args[2]), user: args[3], keyPath: args[4], pinnedFingerprint: args[5], target: work.FullName);
break;
case "s3-archive":
// dotnet run -- s3-archive <bucket> <local file> (uses the learn-sandbox profile via AWS_PROFILE)
await ArchiveToS3Async(bucket: args[1], path: args[2]);
break;
default:
Console.Error.WriteLine("Commands: local | sftp-pull | s3-archive");
return 1;
}
return 0;
static void RunLocal(string dir)
{
// 1. Harbourline's PGP key pair (in production: generated once, private key in Secrets Manager)
var bank = Pgp.GenerateRsaKeyPair();
// 2. Partner side: encrypt the payroll file to the bank's public key and write a manifest
var csv = """
end_to_end_id,creditor_name,creditor_iban,amount,currency
SALARY-NOV-2026-0001,Alex Example,GB33BUKB20201555555555,2150.00,GBP
SALARY-NOV-2026-0002,Sam Sample,GB82WEST12345698765432,1987.45,GBP
""";
var encryptedPath = Path.Combine(dir, "PAYROLL_20261118_001.csv.pgp");
using (var output = File.Create(encryptedPath))
Pgp.Encrypt(new MemoryStream(Encoding.UTF8.GetBytes(csv)), output, bank.PublicKey, "PAYROLL_20261118_001.csv");
Manifest.Write(encryptedPath);
Console.WriteLine($"Partner wrote {Path.GetFileName(encryptedPath)} ({new FileInfo(encryptedPath).Length} bytes) + manifest");
// 3. Bank side: verify manifest BEFORE decrypting, then decrypt
Process(encryptedPath, bank.PrivateKey);
// 4. Tamper with one byte and try again
var bytes = File.ReadAllBytes(encryptedPath);
bytes[^10] ^= 0xFF;
File.WriteAllBytes(encryptedPath, bytes);
Process(encryptedPath, bank.PrivateKey);
}
static void Process(string encryptedPath, PgpPrivateKey key)
{
if (!Manifest.Verify(encryptedPath, out var reason))
{
Console.WriteLine($"REJECTED (quarantine): {reason}");
return;
}
using var input = File.OpenRead(encryptedPath);
using var plain = new MemoryStream();
try
{
Pgp.Decrypt(input, plain, key);
}
catch (PgpException ex)
{
Console.WriteLine($"REJECTED (quarantine): PGP error {ex.Message}");
return;
}
var lines = Encoding.UTF8.GetString(plain.ToArray()).Split('\n', StringSplitOptions.RemoveEmptyEntries);
Console.WriteLine($"ACCEPTED: {lines.Length - 1} payment records, dedupe key {Manifest.Sha256Hex(encryptedPath)[..16]}…");
}
static void SftpPull(string host, int port, string user, string keyPath, string pinnedFingerprint, string target)
{
using var key = new PrivateKeyFile(keyPath);
using var client = new SftpClient(host, port, user, key);
client.HostKeyReceived += (_, e) =>
{
// Pin the server's host key: refuse to talk to anything else (prevents man-in-the-middle).
e.CanTrust = string.Equals(e.FingerPrintSHA256, pinnedFingerprint, StringComparison.Ordinal);
if (!e.CanTrust) Console.Error.WriteLine($"Host key mismatch: got {e.FingerPrintSHA256}");
};
client.Connect();
// Only pick up data files whose manifest has also arrived (manifest is uploaded last).
var names = client.ListDirectory("/in").Where(f => f.IsRegularFile).Select(f => f.Name).ToHashSet();
foreach (var data in names.Where(n => n.EndsWith(".pgp", StringComparison.Ordinal) && names.Contains(n + ".sha256")))
{
foreach (var name in new[] { data, data + ".sha256" })
{
using var fs = File.Create(Path.Combine(target, name));
client.DownloadFile($"/in/{name}", fs);
}
var ok = Manifest.Verify(Path.Combine(target, data), out var reason);
Console.WriteLine($"{data}: {(ok ? "manifest OK" : "REJECTED " + reason)}");
}
client.Disconnect();
}
static async Task ArchiveToS3Async(string bucket, string path)
{
var hashB64 = Convert.ToBase64String(Manifest.Sha256(path));
using var s3 = new AmazonS3Client(Amazon.RegionEndpoint.USEast1);
await using var stream = File.OpenRead(path);
var response = await s3.PutObjectAsync(new PutObjectRequest
{
BucketName = bucket,
Key = $"archive/northwind/{DateTime.UtcNow:yyyy/MM/dd}/{Path.GetFileName(path)}",
InputStream = stream,
ChecksumAlgorithm = ChecksumAlgorithm.SHA256,
ChecksumSHA256 = hashB64, // S3 rejects the upload if the bytes don't match
ServerSideEncryptionMethod = ServerSideEncryptionMethod.AWSKMS
});
Console.WriteLine($"Archived, version {response.VersionId}, S3 checksum {response.ChecksumSHA256}");
}
// ===================== Manifest (SHA-256) =====================
public static class Manifest
{
public static byte[] Sha256(string path)
{
using var fs = File.OpenRead(path);
return SHA256.HashData(fs); // streams; constant memory
}
public static string Sha256Hex(string path) => Convert.ToHexStringLower(Sha256(path));
// Format: "<hex> <filename>\n", the same as sha256sum output, so partners can produce it with standard tools.
public static void Write(string path) =>
File.WriteAllText(path + ".sha256", $"{Sha256Hex(path)} {Path.GetFileName(path)}\n");
public static bool Verify(string path, out string reason)
{
var manifestPath = path + ".sha256";
if (!File.Exists(manifestPath)) { reason = "manifest missing (file may be incomplete)"; return false; }
var parts = File.ReadAllText(manifestPath).Trim().Split(" ", 2);
if (parts.Length != 2 || parts[1] != Path.GetFileName(path)) { reason = "manifest malformed or names another file"; return false; }
byte[] expected;
try { expected = Convert.FromHexString(parts[0]); }
catch (FormatException) { reason = "manifest hash is not hex"; return false; }
if (!CryptographicOperations.FixedTimeEquals(expected, Sha256(path))) { reason = "SHA-256 mismatch"; return false; }
reason = "ok";
return true;
}
}
// ===================== OpenPGP with BouncyCastle =====================
public static class Pgp
{
public static PgpKeyPair GenerateRsaKeyPair()
{
var gen = new RsaKeyPairGenerator();
gen.Init(new RsaKeyGenerationParameters(Org.BouncyCastle.Math.BigInteger.ValueOf(0x10001), new SecureRandom(), 3072, 80));
return new PgpKeyPair(PublicKeyAlgorithmTag.RsaGeneral, gen.GenerateKeyPair(), DateTime.UtcNow);
}
public static void Encrypt(Stream plain, Stream output, PgpPublicKey recipient, string fileName)
{
var encGen = new PgpEncryptedDataGenerator(SymmetricKeyAlgorithmTag.Aes256, withIntegrityPacket: true, new SecureRandom());
encGen.AddMethod(recipient);
using var encOut = encGen.Open(output, new byte[1 << 16]);
var compGen = new PgpCompressedDataGenerator(CompressionAlgorithmTag.Zip);
using var compOut = compGen.Open(encOut);
var litGen = new PgpLiteralDataGenerator();
using var litOut = litGen.Open(compOut, PgpLiteralData.Binary, fileName, DateTime.UtcNow, new byte[1 << 16]);
plain.CopyTo(litOut);
}
public static void Decrypt(Stream encrypted, Stream output, PgpPrivateKey key)
{
var factory = new PgpObjectFactory(PgpUtilities.GetDecoderStream(encrypted));
var first = factory.NextPgpObject();
var list = first as PgpEncryptedDataList ?? factory.NextPgpObject() as PgpEncryptedDataList
?? throw new PgpException("No encrypted data found");
var data = list.GetEncryptedDataObjects().OfType<PgpPublicKeyEncryptedData>()
.FirstOrDefault(d => d.KeyId == key.KeyId)
?? throw new PgpException("File was not encrypted to our key");
if (!data.IsIntegrityProtected()) throw new PgpException("Refusing file without integrity protection");
using (var clear = data.GetDataStream(key))
{
var message = new PgpObjectFactory(clear).NextPgpObject();
if (message is PgpCompressedData compressed)
message = new PgpObjectFactory(compressed.GetDataStream()).NextPgpObject();
if (message is not PgpLiteralData literal) throw new PgpException("Unexpected PGP content (signed-only or unknown)");
literal.GetInputStream().CopyTo(output);
}
if (!data.Verify()) throw new PgpException("Integrity check failed: file modified");
}
}Run it locally:
cd src/Harbourline.PartnerGateway.Files
dotnet run
# Partner wrote PAYROLL_20261118_001.csv.pgp (… bytes) + manifest
# ACCEPTED: 2 payment records, dedupe key 3f1c9a…
# REJECTED (quarantine): SHA-256 mismatchFree local SFTP server. Run an OpenSSH-based SFTP server in Docker and use the same sftp-pull command:
# Create a key pair for the fictional partner
ssh-keygen -t ed25519 -f ~/.ssh/hl_northwind_ed25519 -N "" -C "northwind@example.com"
# atmoz/sftp: user northwind, no password, uid 1001, folder "in"; public key mounted for key auth
docker run --rm -d --name hl-sftp -p 2222:22 \
-v ~/.ssh/hl_northwind_ed25519.pub:/home/northwind/.ssh/keys/id.pub:ro \
atmoz/sftp:alpine northwind::1001::in
# Get the server's host key fingerprint to pin (SHA256, base64)
ssh-keyscan -p 2222 localhost 2>/dev/null | ssh-keygen -lf - -E sha256
# Upload as the partner, then pull as the bank
sftp -P 2222 -i ~/.ssh/hl_northwind_ed25519 northwind@localhost
dotnet run -- sftp-pull localhost 2222 northwind ~/.ssh/hl_northwind_ed25519 <fingerprint-without-SHA256:-prefix>
docker rm -f hl-sftpThe atmoz/sftp image and its user:pass:uid:gid:dir argument format come from the image's documentation ⚠ VERIFY the tag and options on Docker Hub. SSH.NET's FingerPrintSHA256 is base64 without the SHA256: prefix that ssh-keygen prints.
Code explained line by line
Directory.CreateTempSubdirectorygives each run an isolated working folder; never process partner files in a shared directory.Pgp.GenerateRsaKeyPaircreates a 3,072-bit RSA key for the demo. In production, the bank generates its key once (often with a key ceremony), publishes the public key to partners and stores the private key in Secrets Manager encrypted with a KMS key.PgpEncryptedDataGenerator(..., withIntegrityPacket: true, ...)encrypts with AES-256 and adds a modification detection code. Without it, PGP ciphertext can be altered undetectably.- The literal → compressed → encrypted layering is the standard OpenPGP message structure.
usingdeclarations dispose in reverse order, which finishes the literal packet first, then compression, then encryption. Manifest.Writeuses thesha256sumformat (hex, two spaces, file name) so partners can create it with standard tools.Manifest.Verifychecks, in order: manifest present (completeness signal), manifest names this file (prevents a valid manifest for file A being used for file B), hash parses, hash matches in constant time.Processverifies the manifest before decrypting: never spend CPU on, or expose your private key to, a file you have not checked.Pgp.Decryptfinds the encrypted packet for our key ID, refuses data without integrity protection, decompresses if needed, and only accepts a literal data packet.data.Verify()runs after the whole stream has been read, because the integrity check covers all the data. A tampered file throws here (or earlier, while parsing).SftpPullpins the server's host key fingerprint throughHostKeyReceived. Without pinning, SSH.NET would trust any host key and a man-in-the-middle could impersonate the server.- The download loop only picks data files whose
.sha256manifest exists, which is how the "partial file" problem from the opening scenario is avoided. ArchiveToS3AsyncsendsChecksumAlgorithm.SHA256and the precomputedChecksumSHA256; S3 calculates the checksum of what it received and rejects the request if they differ. The checksum is stored with the object and can be retrieved later with--checksum-mode ENABLED.ServerSideEncryptionMethod.AWSKMSencrypts at rest with KMS; the bucket's default encryption should enforce this anyway.
Security considerations
⚠ Learning shortcut — not for production
The demo generates the bank's PGP key pair in memory on every run and never signs or verifies partner signatures. In production: the private key lives in Secrets Manager (or an HSM-backed solution), only the processor role can read it, keys have documented expiry and rotation with overlap, and the contract requires partners to sign files so you can verify who created them, not just who uploaded them.
⚠ Learning shortcut — not for production
The CLI and IaC examples use a PUBLIC endpoint and AutoDeleteObjects/force_destroy for easy teardown. Production partner gateways use a VPC internet-facing endpoint with Elastic IPs and security groups restricted to partner CIDRs, and archive buckets with Object Lock that cannot be force-deleted.
- Keys, not passwords. Service-managed users with SSH public keys only. Rotate keys with overlap (users can have several keys). Remove keys when partner staff leave.
- Least-privilege per partner. One IAM role (or session policy) per partner, limited to its prefix; no delete permission.
- Transport security policy. Keep the server on a current
TransferSecurityPolicy-*to exclude weak ciphers and key exchange algorithms; review when AWS publishes new policies. - Integrity beyond the channel. SSH protects the transfer only. PGP integrity, manifests and S3 checksums protect the file at rest and across hops.
- Scan before processing. Treat every file as hostile until malware scanning and validation pass.
- Logging. Transfer Family logs connections and file operations to CloudWatch; keep them for investigations, without logging file contents.
- Swift. If files feed a Swift interface, the Swift CSP controls (segregated secure zone, operator authentication, integrity of transactions) apply to that environment; involve your Swift security officer ⚠ VERIFY current CSP control set.
- Personal data. Payroll and payment files are full of personal data: encrypt, minimise retention outside the archive, restrict access, and include them in your records of processing.
Banking use case
Harbourline's Partner Gateway (fictional) handles 40 corporate customers and 6 service providers:
- Inbound: payroll and supplier-payment files (CSV and pain.001), PGP-encrypted and signed, with SHA-256 manifests. GuardDuty scans; the .NET processor on ECS verifies, decrypts, validates (record count and total in the trailer must match), creates payments idempotently and writes a
pain.002acknowledgement to/outwithin 5 minutes of the cut-off. - Outbound: end-of-day
camt.053statements for each corporate customer, PGP-encrypted to the customer's public key, delivered to their/outfolder; for customers who host their own SFTP, SFTP connectors push the file instead. - Card settlement: a daily settlement file from the card processor arrives over a VPC internal endpoint across Direct Connect.
- Controls: dedupe table on partner + SHA-256, Object Lock archive for seven years ⚠ VERIFY your records retention policy, alarms on "file expected but not received by cut-off" (a scheduled check) and on any quarantine event.
Common mistakes
- Processing on "file created" without a completeness signal. Partial uploads get processed. Use manifests or a rename-on-complete convention.
- Deduping on file name. Partners reuse names, or resend with a new name. Dedupe on content hash plus partner.
- Stopping instead of deleting the server. Billing continues.
- One IAM role for all partners. One partner can read another's files.
- Accepting password authentication "just for this partner".
- Not pinning host keys in outbound SFTP clients.
- PGP without integrity protection or signatures. Encryption alone does not prove origin or integrity.
- Keeping decrypted files in the landing bucket. Decrypt in memory or to a separate, tightly controlled prefix, and delete after processing.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Permission denied (publickey) | Wrong key, key not added to user, or wrong user name | Check the user's SSH keys in Transfer Family; test with sftp -v |
Login works but ls fails / access denied | Role missing s3:ListBucket or prefix condition wrong | Fix the s3:prefix condition to include the home folder |
| Upload fails with access denied on KMS | Bucket uses a customer managed key the user role can't use | Grant kms:GenerateDataKey and kms:Decrypt to the partner role |
| Partner can't connect from their network | Security group or partner firewall | Allow partner CIDR on TCP 22; give partner your Elastic IPs |
Integrity check failed | File corrupted or tampered, or uploaded in ASCII mode by a legacy client | Ask the partner to resend in binary mode; compare manifests |
File was not encrypted to our key | Partner used an old or wrong public key | Send the current public key and its fingerprint through an authenticated channel |
| Workflow decrypt step fails | Secret not found under the expected name, or anonymous recipient | Store key per Transfer Family naming rules; encrypt with an explicit recipient |
| Same file processed twice | Dedupe on name only, or no conditional write | Conditional PutItem on partner + SHA-256 |
Cost and cleanup
- Free path: the local commands, Docker SFTP server and PGP code cost nothing.
- Transfer Family (paid): USD 0.30 per protocol-hour while the server exists + USD 0.04/GB. One-hour lab: about USD 0.30. Managed workflow decryption: USD 0.10/GB. SFTP connectors: per call plus per GB ⚠ VERIFY current connector pricing. VPC internet-facing endpoints also use Elastic IPs, which are charged per hour for public IPv4 addresses.
- GuardDuty Malware Protection for S3: charged per GB scanned and per object evaluated; skip it in the lab.
Teardown:
aws transfer list-servers --profile learn-sandbox --region us-east-1
aws transfer delete-server --server-id s-EXAMPLE1234567890 --profile learn-sandbox --region us-east-1
aws s3 rm s3://hl-m04-files-111122223333 --recursive --profile learn-sandbox
# Versioned bucket: delete all versions and delete markers (console "Empty") before:
aws s3api delete-bucket --bucket hl-m04-files-111122223333 --profile learn-sandbox
aws iam delete-role-policy --role-name hl-m04-sftp-northwind --policy-name northwind --profile learn-sandbox
aws iam delete-role --role-name hl-m04-sftp-northwind --profile learn-sandbox
docker rm -f hl-sftp
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--query 'ResourceTagMappingList[].ResourceARN' --profile learn-sandbox --region us-east-1Confirm list-servers returns an empty list: that is the line item that bills.
Exercises
- 🟢 Guided: Run
dotnet runand observe accepted and rejected outcomes. Then delete the.sha256file before processing and confirm the reason "manifest missing". - 🟡 Challenge: Add PGP signing by the partner and signature verification by the bank: generate a second key pair for Northwind, sign with
PgpSignatureGenerator(one-pass signature) inside the encrypted message, and reject files signed by any other key. - 🔴 Live problem: A partner says: "We uploaded PAYROLL_20261120_001 at 05:52, but you say you never received it." Your landing bucket has no such object; Transfer Family logs show a successful login at 05:51 from the partner's IP and an
OPENof/in/PAYROLL_20261120_001.csv.pgpbut noCLOSE. Explain what probably happened, what evidence you would gather, and how to make this situation visible automatically next time.
Assignment
Write a one-page partner file contract template for Harbourline (naming, format, encryption and signing, manifest, cut-offs, acknowledgements, resend rules, key rotation, contacts) and a matching onboarding checklist (user, role, home mapping, key fingerprint verification call, test file, go-live). Add the Terraform for one partner user to your portfolio repository.
Solutions
Add File.Delete(encryptedPath + ".sha256"); before the first Process call. The output becomes REJECTED (quarantine): manifest missing (file may be incomplete). Why: the manifest is the partner's statement that the upload finished; without it the bank cannot distinguish a complete file from a partial one, so it must wait (and alarm if the cut-off passes).
In Encrypt, after opening the compressed stream, create var sigGen = new PgpSignatureGenerator(partnerKey.PublicKey.Algorithm, HashAlgorithmTag.Sha256); sigGen.InitSign(PgpSignature.BinaryDocument, partnerKey.PrivateKey);, write sigGen.GenerateOnePassVersion(false).Encode(compOut), then the literal data while calling sigGen.Update(buffer, 0, read) for each chunk, and finally sigGen.Generate().Encode(compOut). In Decrypt, expect a PgpOnePassSignatureList first, call ops.InitVerify(partnerPublicKey) (checking ops.KeyId matches the partner's key), feed the literal data to ops.Update as you copy it, read the trailing PgpSignatureList and call ops.Verify(sigList[0]). Reject if the key ID is unknown or verification fails. Why: encryption proves only that the file was meant for you; the signature proves the partner produced it, which defeats the "attacker uploaded through a stolen SFTP key" scenario unless the signing key was also stolen ⚠ VERIFY exact BouncyCastle method signatures in 2.7.0.
An OPEN without CLOSE means the upload started but the session ended before the file was complete (network drop, client timeout). With S3 as the backend, an incomplete SFTP upload does not produce a visible object, so nothing landed. Evidence: Transfer Family CloudWatch logs for that session (connection, OPEN, any ERROR, disconnect time), the partner's client logs, and S3 server access or CloudTrail data events for the prefix. Make it visible: a scheduled check at cut-off that alarms when an expected file (by partner and date pattern) is missing; a metric filter on sessions with OPEN but no CLOSE; and an acknowledgement contract so partners alarm when they don't receive the bank's ack within N minutes.
Knowledge check
Answer: A VPC-hosted, internet-facing endpoint with Elastic IP addresses and security groups. Public endpoints have no security groups and their IPs can change.
Answer: Yes. Charges accrue per protocol-hour while the server exists; only deleting it stops the charge.
Answer: Its arrival signals that the data file is complete, and its hash lets you verify the exact bytes. Processing only files with a matching manifest prevents processing partial uploads.
Answer: S3 computes the checksum of the received bytes and rejects the upload if it doesn't match the value you supplied; the checksum is stored and retrievable for later integrity checks.
Answer: A logical home directory mapping / to their prefix, plus an IAM role (or session policy) granting access only to that prefix, with no delete permission.
Interview questions
- "How do you make batch file processing exactly-once?" — Completeness signal (manifest), dedupe key on partner plus content hash with a conditional write, idempotent downstream operations keyed by
EndToEndId, immutable archive, quarantine for failures. - "Why PGP if SFTP is already encrypted?" — SFTP protects only the transfer; PGP protects the file at rest across every hop and, with signatures, proves who created it.
- "How would you onboard a new SFTP partner on AWS?" — Contract, partner sends SSH public key and IPs, IaC creates user, role and home mapping, security group rule, verify key fingerprints out-of-band, exchange PGP public keys, test file, go-live with monitoring.
- "What does Transfer Family cost and how do you control it?" — Per protocol-hour while the server exists plus per-GB; enable only needed protocols, share one server across partners, delete non-production servers.
- "Where does Swift fit with AWS?" — The Swift connection is a controlled environment under CSP; AWS hosts the payment hub and integration around it, exchanging messages and files with the Swift interface through controlled channels.
Certification practice
A bank must receive files from partners over SFTP into Amazon S3. Partners require fixed IP addresses to allowlist, and the bank must accept connections only from known partner IP ranges. Which solution meets these requirements with the LEAST operational overhead?
A. An EC2 instance running OpenSSH with an Elastic IP and a security group.
B. An AWS Transfer Family SFTP server with a VPC-hosted, internet-facing endpoint using Elastic IP addresses and a security group allowing partner CIDRs.
C. An AWS Transfer Family server with a public endpoint and a bucket policy restricting aws:SourceIp.
D. Amazon S3 presigned URLs sent to partners by email.
Answer: B. Managed SFTP with stable Elastic IPs and security-group allowlisting. A works but requires patching and HA. C: public endpoints have no fixed IPs, and the S3 request source is Transfer Family, not the partner. D is not SFTP and leaks URLs.
Partners upload PGP-encrypted files to an AWS Transfer Family SFTP server. The bank wants files decrypted automatically on upload without managing servers, with private keys protected and access audited. What should the bank do?
A. Store the PGP private key in the Lambda environment variables and decrypt in Lambda. B. Use a Transfer Family managed workflow with a decrypt step, storing the PGP private key in AWS Secrets Manager under the required naming convention, with the workflow execution role allowed to read only that secret. C. Ask partners to stop encrypting files because SFTP is encrypted. D. Store the private key in the S3 bucket next to the files.
Answer: B. Managed decryption with keys in Secrets Manager, least-privilege access and CloudTrail auditing. A exposes the key to anyone who can read the function configuration. C loses at-rest protection across hops. D places the key with the data it protects.
A developer uploads verified payment files to S3 with the AWS SDK and must guarantee that the stored object is byte-for-byte identical to the file the application checked. What should the developer do?
A. Compare the object's ETag with an MD5 computed locally for all uploads.
B. Supply ChecksumAlgorithm = SHA256 and the precomputed ChecksumSHA256 in the PutObject request.
C. Enable S3 Transfer Acceleration.
D. Enable versioning on the bucket.
Answer: B. S3 validates the supplied SHA-256 against the received bytes and stores it. A is unreliable: ETags are not MD5 for multipart or KMS-encrypted objects. C speeds transfer. D keeps versions but does not verify content.
Completion checklist
- I can explain the four questions of any file integration (who, intact, confidential, once)
- I can choose a Transfer Family endpoint type and explain the billing trap
- I ran the PGP + manifest demo and saw a tampered file rejected
- I ran (or understand) the local SFTP flow with host-key pinning
- I can describe dedupe on partner + SHA-256 with a conditional write
- I deleted any Transfer Family server and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- Files remain core to banking; the risks are completeness, integrity, origin and double processing.
- Transfer Family gives managed SFTP into S3 with per-partner jails; it bills hourly until deleted.
- PGP protects files beyond the channel; manifests and S3 checksums prove the bytes; signatures prove the author.
- Process only scanned, verified files, dedupe on content hash, and archive originals immutably.
- Swift and scheme connections have their own security regimes; AWS hosts the integration around them.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
ISO 20022 message handling: pain.001, pacs.008 and camt.053 in C#
Read, build and validate ISO 20022 payment messages safely in .NET: understand the structure of pain.001, pacs.008 and camt.053, validate in three layers (schema, scheme rules, business rules), generate C# classes from XSDs, parse statements into Harbourline's transaction model, and run it all on AWS with S3, SQS and Lambda.
Integrating with core banking and mainframes: hybrid networks, private APIs and MQ messaging
Connect cloud services to an on-premises core banking system safely: choose between Site-to-Site VPN and Direct Connect, expose and consume private APIs, use request/reply messaging with Amazon MQ (or RabbitMQ locally), and translate COBOL copybook records with EBCDIC and packed decimals into a clean C# domain model behind an anti-corruption layer.