Skip to Content
Cloud Engineering Academy
7. Data Engineering & ETL Pipelines

Lightweight transforms with Lambda: C# and Python functions for small files

Validate Harbourline's partner SFTP payment batches with an S3-triggered Lambda function in C# (.NET 10) or Python, including trailer control totals, IBAN checks and quarantine, and learn exactly where Lambda's limits make Glue the better tool.

intermediate⏱ 2 h 🟢 FreeDEA-C01DVA-C03SAA-C03

Assumes: .NET 10 LTS (Lambda runtime dotnet10) · Amazon.Lambda.Core 3.4.0 · Amazon.Lambda.S3Events 4.0.0 · Amazon.Lambda.Serialization.SystemTextJson 3.0.1 · AWSSDK.S3 4.1.1 · Python 3.13 runtime · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Decide between Lambda and Glue (or Athena SQL) for a transform, using file size, run time, memory, concurrency and cost.
  • Build an S3-triggered Lambda function in C# that validates a partner payment batch file: header, detail lines, trailer record count and control sum, IBAN check digits and duplicates, all-or-nothing.
  • Write the equivalent function in Python and compare the two.
  • Make the function idempotent, safe against recursive invocation loops, and observable with structured logs.
  • Deploy it with CDK (C#) or Terraform, test it locally for free, and know its exact cost.

Prerequisites

Why this matters in banking

Every evening, Harbourline's partner bureau Saltmarsh Payroll Services Ltd (fictional) drops a payroll file on the Partner Gateway SFTP server: one header, about 40 detail lines (one per employee payment) and a trailer with the record count and control sum. The bank must not send any of those payments unless the file is complete and internally consistent. If the trailer says 40 payments totalling £89,281.75 and the bank processes 41 payments or a different total, someone gets paid twice or not at all, and the bureau's client has a very bad payday.

The file is a few kilobytes. Starting a Spark cluster for it would take longer than the validation itself and cost more. A Lambda function that runs in milliseconds as soon as the file lands is the right tool, and the same pattern works for many small banking feeds: FX rate files, sanctions-list deltas, card-scheme fee tables, partner acknowledgements.

The idea in plain language

AWS Lambda runs a function when an event happens and bills only for the time it runs. For data engineering, the event is usually "a file arrived in S3". Lambda reads the file, checks or converts it, writes results somewhere else, and finishes.

The trade-off is simple: Lambda is excellent for small, quick, per-file work and poor for large, joined, multi-file work. It has a hard limit of 15 minutes per invocation, up to 10,240 MB of memory, up to 10,240 MB of /tmp disk, and processes one event on one machine. Spark (Glue) spreads big work across many machines but takes time to start.

Analogy: Lambda is the cashier who checks each envelope as it comes through the slot: open, count, compare with the slip, stamp "accepted" or "returned". Glue is the overnight back-office team that processes the whole day's mountain of paperwork together. You would not call in the overnight team for one envelope, and you would not ask the cashier to reconcile a million transactions.

What problem the AWS service solves

Lambda removes servers and schedulers from small transforms: no cluster, no polling loop, no VM to patch. Events from S3 (directly or through EventBridge), SQS, Kinesis and others invoke it; it scales out automatically (default 1,000 concurrent executions per Region per account); and you pay per request and per GB-second.

Limits that decide the design (checked 2026-11-18)

LimitValueWhy it matters for data work
Timeout900 seconds (15 minutes) maximumLarge files or slow downstream calls can hit it; design for minutes, not hours
Memory128 MB to 10,240 MB (CPU scales with memory; about 1 vCPU at 1,769 MB)More memory also means more CPU
/tmp ephemeral storage512 MB to 10,240 MBSpill space for conversions; billed above 512 MB
Invocation payload6 MB synchronous request/response; 1 MB asynchronousS3 events carry only the key, so file size is not limited by payload; read the file with GetObject
Deployment package50 MB zipped upload, 250 MB unzipped (including layers); container images up to 10 GBBig libraries (pyarrow, pandas) push you to layers or container images
Concurrency1,000 per Region by default (new accounts may start lower)A burst of 5,000 files throttles; use SQS between S3 and Lambda for back-pressure
Environment variables4 KB totalPut configuration in Parameter Store or Secrets Manager

Lambda or Glue?

QuestionLambdaGlue (Spark)
Input size per runKB to a few hundred MBHundreds of MB to TB
Run timeMilliseconds to a few minutes (hard 15-minute cap)Minutes to hours
Start-upMilliseconds (warm) to about a second (cold)Tens of seconds to minutes
Joins across many files or tablesAwkwardNatural
LanguagesC#, Python, Java, Node.js, othersPython (PySpark), Scala
Cost at small scaleFractions of a cent; free tier covers most labs1-minute minimum × 2 DPUs ≈ USD 0.015 per run minimum
Cost at large scaleGrows with GB-seconds; can exceed Glue for long CPU-heavy workEfficient per GB processed
Catalogue, bookmarks, Data QualityBuild it yourselfBuilt in

Rules of thumb Harbourline uses:

  • Per-file checks, routing, small conversions, notifications → Lambda.
  • Anything joining or aggregating across files, anything over about 1 GB, anything needing Spark → Glue.
  • Set-based SQL over data already in the lake → Athena CTAS/INSERT INTO (see Amazon Athena).
  • When in doubt, measure: a Lambda that routinely runs more than a few minutes is a Glue job in disguise.

Pricing (us-east-1, checked 2026-11-18)

  • USD 0.20 per 1 million requests.
  • USD 0.0000166667 per GB-second on x86_64; about 20% less on arm64 (Graviton).
  • Always-free tier: 1 million requests and 400,000 GB-seconds per month.
  • /tmp above 512 MB is billed per GB-second.

Validating one 4 KB batch file at 512 MB for 300 ms costs 0.5 GB × 0.3 s = 0.15 GB-seconds ≈ USD 0.0000025 plus the request: effectively free, and inside the free tier anyway.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Functions with a Blob triggerLambda with an S3 event notification or EventBridge ruleS3 events are push-based; no polling of the container
Event Grid BlobCreated → FunctionS3 → EventBridge → LambdaEventBridge rules filter on bucket, key prefix and suffix
Functions Consumption plan timeoutLambda 15-minute maximumDurable orchestration lives in Step Functions (or Lambda durable functions)
SSIS script task on a small fileLambda functionEvent-driven, per file, no SQL Agent schedule
ILogger structured logs to App InsightsLambda logs to CloudWatch Logs (JSON format option)Use Powertools for AWS Lambda (.NET) for structured logging and metrics
Function app settingsEnvironment variables (4 KB), Parameter Store, Secrets ManagerKeep secrets out of environment variables

Architecture

The function reads from raw and writes only to curated and quarantine, so its output can never trigger it again. That design choice, not luck, prevents recursive loops.

Security-boundary view: the function's role may GetObject only under transactions/source=sftp/ in raw, PutObject only under payment_batches/ in curated and quarantine, and use the two zone keys. It has no network access beyond AWS APIs (no VPC needed; add a VPC with S3 gateway endpoint in production if policy requires private paths).

Step-by-step implementation

Write the validator as a pure class

Put the rules in BatchValidator (no AWS types), so you can unit-test it against the committed samples: the clean file must be accepted, and the messy file must be rejected with exactly the defects listed in HLPG_PTN0042_PAYMENTS_20261116_001_messy_defects.csv.

Wrap it in a Lambda handler

The handler receives an S3Event, reads each object (by version ID, so it validates exactly the bytes that triggered it), calls the validator and writes one output: accepted lines to curated, or an issues report to quarantine.

Test locally (free)

Run the xUnit tests with dotnet test, or in Docker. No AWS account needed.

Deploy and trigger

Deploy with CDK or Terraform. Copy the clean and messy sample files to the raw prefix; watch the outputs and logs.

Prove idempotency

Upload the same file again (a new S3 version). The function produces the same output key with the same content: re-processing is harmless. Then upload a renamed copy and see a second output, which is why the batch ID (from the header) is checked downstream for duplicates.

AWS Console instructions

  1. Console → Lambda → Create function → Author from scratch: name hl-m07-sftp-batch-validator; runtime .NET 10 (C#/F#/PowerShell); architecture arm64; execution role: existing role hl-m07-sftp-batch-validator-role (from IaC) or create one and attach the policy from the IaC tab.
  2. Code → Upload from → .zip file: the output of dotnet lambda package (or dotnet publish zipped). Runtime settings → Edit → Handler: SftpBatch::Harbourline.DataLake.SftpBatch.Function::HandleAsync.
  3. Configuration → General: memory 512 MB, timeout 60 seconds, ephemeral storage 512 MB.
  4. Configuration → Environment variables: CURATED_BUCKET, QUARANTINE_BUCKET.
  5. Configuration → Asynchronous invocation: retry attempts 2; maximum event age 1 hour; On-failure destination → SQS queue hl-m07-sftp-batch-dlq.
  6. Configuration → Monitoring and operations tools → Logging configuration: log format JSON, application log level INFO; log group retention 14 days.
  7. Add trigger → S3: bucket hl-m07-raw-<account-id>; event types All object create events; prefix transactions/source=sftp/; suffix .csv. Acknowledge the recursive-invocation warning only because the function never writes to this bucket.
  8. Configuration → Concurrency and recursion detection: reserved concurrency 10 (protects downstream and your bill); recursive loop detection Terminate (the default).

AWS CLI

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
RAW=hl-m07-raw-$ACCOUNT

# Package the C# function (Amazon.Lambda.Tools: dotnet tool install -g Amazon.Lambda.Tools)
dotnet lambda package --project-location SftpBatch --configuration Release \
  --function-architecture arm64 --output-package sftp-batch.zip

# Trigger it: land the clean and the messy partner file
aws s3 cp datasets/samples/sftp/HLPG_PTN0042_PAYMENTS_20261116_001.csv \
  "s3://$RAW/transactions/source=sftp/ingest_date=2026-11-16/HLPG_PTN0042_PAYMENTS_20261116_001.csv"
aws s3 cp datasets/samples/sftp/HLPG_PTN0042_PAYMENTS_20261116_001_messy.csv \
  "s3://$RAW/transactions/source=sftp/ingest_date=2026-11-16/HLPG_PTN0042_PAYMENTS_20261116_002.csv"

# Read the function's JSON summaries from the last 10 minutes
aws logs tail /aws/lambda/hl-m07-sftp-batch-validator --since 10m --format short

# Look at the outputs
aws s3 ls "s3://hl-m07-curated-$ACCOUNT/payment_batches/" --recursive
aws s3 ls "s3://hl-m07-quarantine-$ACCOUNT/payment_batches/" --recursive
aws s3 cp "s3://hl-m07-quarantine-$ACCOUNT/payment_batches/ingest_date=2026-11-16/HLPG_PTN0042_PAYMENTS_20261116_002.csv.issues.json" - | head -40

# Invoke directly with a hand-made S3 event (useful for testing without uploads)
cat > event.json <<EOF
{"Records":[{"eventSource":"aws:s3","s3":{"bucket":{"name":"$RAW"},
 "object":{"key":"transactions/source%3Dsftp/ingest_date%3D2026-11-16/HLPG_PTN0042_PAYMENTS_20261116_001.csv"}}}]}
EOF
aws lambda invoke --function-name hl-m07-sftp-batch-validator \
  --cli-binary-format raw-in-base64-out --payload file://event.json response.json

# Metrics: invocations, errors, duration, recursive invocations dropped
aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Duration \
  --dimensions Name=FunctionName,Value=hl-m07-sftp-batch-validator \
  --start-time "$(date -u -d '-1 hour' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
  --period 3600 --statistics Average Maximum

Note that keys in S3 events are URL-encoded (= becomes %3D, spaces become +); the function decodes them.

Infrastructure as Code

infra/SftpBatchStack.cs
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.Lambda.Destinations;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.S3;
using Amazon.CDK.AWS.S3.Notifications;
using Amazon.CDK.AWS.SQS;
using Constructs;

namespace Harbourline.DataLake.Infra;

public sealed class SftpBatchStack : Stack
{
    public SftpBatchStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        var account = Aws.ACCOUNT_ID;
        var raw = Bucket.FromBucketName(this, "Raw", $"hl-m07-raw-{account}");
        var curated = Bucket.FromBucketName(this, "Curated", $"hl-m07-curated-{account}");
        var quarantine = Bucket.FromBucketName(this, "Quarantine", $"hl-m07-quarantine-{account}");
        var rawKey = Key.FromLookup(this, "RawKey", new KeyLookupOptions { AliasName = "alias/hl-m07-raw" });
        var curatedKey = Key.FromLookup(this, "CuratedKey", new KeyLookupOptions { AliasName = "alias/hl-m07-curated" });

        var dlq = new Queue(this, "Dlq", new QueueProps
        {
            QueueName = "hl-m07-sftp-batch-dlq",
            Encryption = QueueEncryption.SQS_MANAGED,
            EnforceSSL = true,
            RetentionPeriod = Duration.Days(4),
        });

        var logGroup = new LogGroup(this, "Logs", new LogGroupProps
        {
            LogGroupName = "/aws/lambda/hl-m07-sftp-batch-validator",
            Retention = RetentionDays.TWO_WEEKS,
            RemovalPolicy = RemovalPolicy.DESTROY,
        });

        var fn = new Function(this, "Validator", new FunctionProps
        {
            FunctionName = "hl-m07-sftp-batch-validator",
            Runtime = new Runtime("dotnet10", RuntimeFamily.DOTNET_CORE),
            Architecture = Architecture.ARM_64,
            Handler = "SftpBatch::Harbourline.DataLake.SftpBatch.Function::HandleAsync",
            Code = Code.FromAsset("../SftpBatch/bin/Release/net10.0/publish"),
            MemorySize = 512,
            Timeout = Duration.Seconds(60),
            ReservedConcurrentExecutions = 10,
            LogGroup = logGroup,
            LoggingFormat = LoggingFormat.JSON,
            RetryAttempts = 2,
            MaxEventAge = Duration.Hours(1),
            OnFailure = new SqsDestination(dlq),
            Environment = new Dictionary<string, string>
            {
                ["CURATED_BUCKET"] = curated.BucketName,
                ["QUARANTINE_BUCKET"] = quarantine.BucketName,
            },
        });

        // Least privilege: read only the SFTP prefix, write only payment_batches/.
        fn.AddToRolePolicy(new PolicyStatement(new PolicyStatementProps
        {
            Actions = new[] { "s3:GetObject", "s3:GetObjectVersion" },
            Resources = new[] { raw.ArnForObjects("transactions/source=sftp/*") },
        }));
        fn.AddToRolePolicy(new PolicyStatement(new PolicyStatementProps
        {
            Actions = new[] { "s3:PutObject" },
            Resources = new[] { curated.ArnForObjects("payment_batches/*"), quarantine.ArnForObjects("payment_batches/*") },
        }));
        rawKey.GrantEncryptDecrypt(fn);       // raw + quarantine
        curatedKey.GrantEncrypt(fn);          // write curated

        raw.AddEventNotification(EventType.OBJECT_CREATED, new LambdaDestination(fn),
            new NotificationKeyFilter { Prefix = "transactions/source=sftp/", Suffix = ".csv" });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m07-transaction-pipeline");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        Amazon.CDK.Tags.Of(this).Add("Owner", "your-name");
    }
}

Publish before cdk deploy: dotnet publish SftpBatch -c Release -r linux-arm64 --self-contained false -o SftpBatch/bin/Release/net10.0/publish. Adding a notification to an imported bucket deploys a small custom resource that calls PutBucketNotificationConfiguration; it merges with existing configuration such as the EventBridge setting ⚠ VERIFY on your CDK version, so check the bucket's notification configuration afterwards.

Complete code

C#: project file

SftpBatch/SftpBatch.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <RootNamespace>Harbourline.DataLake.SftpBatch</RootNamespace>
    <AssemblyName>SftpBatch</AssemblyName>
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
    <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Amazon.Lambda.Core" Version="3.4.0" />
    <PackageReference Include="Amazon.Lambda.S3Events" Version="4.0.0" />
    <PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="3.0.1" />
    <PackageReference Include="AWSSDK.S3" Version="4.1.1" />
  </ItemGroup>
</Project>

C#: the validator (no AWS dependencies)

SftpBatch/BatchValidator.cs
using System.Globalization;
using System.Text;
using System.Text.RegularExpressions;

namespace Harbourline.DataLake.SftpBatch;

public sealed record BatchHeader(string BatchId, string PartnerId, string DebtorAccountId,
                                 string Currency, DateOnly RequestedExecutionDate);

public sealed record BatchLine(int LineNumber, string EndToEndId, string CreditorName,
                               string CreditorIbanMasked, decimal Amount, string Scheme, string Reference);

public sealed record Issue(int FileLine, string Code, string Message);

public sealed record BatchResult(BatchHeader? Header, IReadOnlyList<BatchLine> Lines, IReadOnlyList<Issue> Issues,
                                 int? DeclaredCount, decimal? DeclaredSum, decimal ActualSum)
{
    // All-or-nothing: a payroll batch is either entirely valid or returned to the partner.
    public bool Accepted => Issues.Count == 0;
}

public static partial class BatchValidator
{
    private static readonly HashSet<string> Schemes = ["FPS", "BACS", "CHAPS", "SEPA_CT", "SEPA_INST"];
    private static readonly HashSet<string> Currencies = ["GBP", "EUR"];

    [GeneratedRegex(@"^\d{1,16}\.\d{2}$")] private static partial Regex PositiveAmount();
    [GeneratedRegex(@"^ACC-\d{10}$")] private static partial Regex AccountId();

    public static BatchResult Validate(TextReader reader)
    {
        var issues = new List<Issue>();
        var lines = new List<BatchLine>();
        var seenEndToEnd = new HashSet<string>();
        BatchHeader? header = null;
        int? declaredCount = null;
        decimal? declaredSum = null;
        var actualSum = 0m;
        var detailCount = 0;
        var fileLine = 0;
        var trailerSeen = false;

        while (reader.ReadLine() is { } text)
        {
            fileLine++;
            if (text.Length == 0) continue;
            var f = Csv.Split(text);

            if (trailerSeen)
            {
                issues.Add(new(fileLine, "AFTER_TRAILER", "content after the trailer record"));
                continue;
            }

            switch (f[0])
            {
                case "H" when fileLine == 1:
                    header = ParseHeader(f, fileLine, issues);
                    break;
                case "D" when header is not null || fileLine > 1:
                    detailCount++;
                    if (ParseDetail(f, fileLine, issues, seenEndToEnd) is { } line)
                    {
                        lines.Add(line);
                        actualSum += line.Amount;
                    }
                    break;
                case "T":
                    trailerSeen = true;
                    if (f.Length != 3 || !int.TryParse(f[1], NumberStyles.None, CultureInfo.InvariantCulture, out var count)
                        || !decimal.TryParse(f[2], NumberStyles.AllowDecimalPoint, CultureInfo.InvariantCulture, out var sum))
                    {
                        issues.Add(new(fileLine, "TRAILER_INVALID", "trailer must be T,record_count,control_sum"));
                        break;
                    }
                    declaredCount = count;
                    declaredSum = sum;
                    break;
                default:
                    issues.Add(new(fileLine, "RECORD_TYPE", $"unexpected record type '{f[0]}' on line {fileLine}"));
                    break;
            }
        }

        if (header is null) issues.Add(new(1, "HEADER_MISSING", "first record must be the header (H)"));
        if (!trailerSeen) issues.Add(new(fileLine, "TRAILER_MISSING", "file has no trailer (T): possibly truncated"));
        if (declaredCount is { } dc && dc != detailCount)
            issues.Add(new(fileLine, "TRAILER_COUNT", $"trailer declares {dc} records, file has {detailCount}"));
        if (declaredSum is { } ds && ds != actualSum)
            issues.Add(new(fileLine, "TRAILER_SUM", $"trailer control sum {ds:0.00} differs from valid lines' sum {actualSum:0.00}"));

        return new BatchResult(header, lines, issues, declaredCount, declaredSum, actualSum);
    }

    private static BatchHeader? ParseHeader(string[] f, int fileLine, List<Issue> issues)
    {
        // H,batch_id,partner_id,partner_name,created_at,debtor_account_id,debtor_iban,currency,requested_execution_date
        if (f.Length != 9) { issues.Add(new(fileLine, "HEADER_INVALID", "header must have 9 fields")); return null; }
        var ok = true;
        if (!AccountId().IsMatch(f[5])) { issues.Add(new(fileLine, "INVALID_ID", "debtor_account_id")); ok = false; }
        if (!Iban.IsValid(f[6])) { issues.Add(new(fileLine, "INVALID_IBAN", "debtor_iban fails mod-97")); ok = false; }
        if (!Currencies.Contains(f[7])) { issues.Add(new(fileLine, "INVALID_CURRENCY", $"currency '{f[7]}'")); ok = false; }
        if (!DateOnly.TryParseExact(f[8], "yyyy-MM-dd", CultureInfo.InvariantCulture, DateTimeStyles.None, out var execDate))
        {
            issues.Add(new(fileLine, "INVALID_DATE", $"requested_execution_date '{f[8]}' is not ISO yyyy-MM-dd"));
            ok = false;
        }
        return ok ? new BatchHeader(f[1], f[2], f[5], f[7], execDate) : new BatchHeader(f[1], f[2], f[5], f[7], default);
    }

    private static BatchLine? ParseDetail(string[] f, int fileLine, List<Issue> issues, HashSet<string> seen)
    {
        // D,line_number,end_to_end_id,creditor_name,creditor_iban,amount,scheme,reference
        if (f.Length != 8) { issues.Add(new(fileLine, "DETAIL_INVALID", "detail must have 8 fields")); return null; }
        var before = issues.Count;
        if (!seen.Add(f[2])) issues.Add(new(fileLine, "DUPLICATE", $"end_to_end_id {f[2]} repeated"));
        if (!Iban.IsValid(f[4])) issues.Add(new(fileLine, "INVALID_IBAN", "creditor_iban fails mod-97"));
        if (!PositiveAmount().IsMatch(f[5])) issues.Add(new(fileLine, "INVALID_AMOUNT", $"amount '{f[5]}' must be like 1234.56"));
        if (!Schemes.Contains(f[6])) issues.Add(new(fileLine, "INVALID_CODE", $"scheme '{f[6]}'"));
        if (issues.Count != before) return null;

        return new BatchLine(int.Parse(f[1], CultureInfo.InvariantCulture), f[2], f[3], Iban.Mask(f[4]),
                             decimal.Parse(f[5], CultureInfo.InvariantCulture), f[6], f[7]);
    }
}

public static class Iban
{
    public static bool IsValid(string value)
    {
        var s = value.Replace(" ", "").ToUpperInvariant();
        if (s.Length is < 15 or > 34 || !s.All(char.IsAsciiLetterOrDigit)) return false;
        var remainder = 0;
        foreach (var ch in s[4..] + s[..4])
        {
            var digits = char.IsAsciiLetter(ch) ? (ch - 'A' + 10).ToString(CultureInfo.InvariantCulture) : ch.ToString();
            foreach (var d in digits) remainder = (remainder * 10 + (d - '0')) % 97;
        }
        return remainder == 1;
    }

    public static string Mask(string iban) => iban.Length <= 8 ? new string('*', iban.Length)
        : string.Concat(iban.AsSpan(0, 4), new string('*', iban.Length - 8), iban.AsSpan(iban.Length - 4));
}

public static class Csv
{
    public static string[] Split(string line)
    {
        var fields = new List<string>();
        var sb = new StringBuilder();
        var quoted = false;
        for (var i = 0; i < line.Length; i++)
        {
            var c = line[i];
            if (quoted && c == '"' && i + 1 < line.Length && line[i + 1] == '"') { sb.Append('"'); i++; }
            else if (c == '"') quoted = !quoted;
            else if (c == ',' && !quoted) { fields.Add(sb.ToString()); sb.Clear(); }
            else sb.Append(c);
        }
        fields.Add(sb.ToString());
        return fields.ToArray();
    }
}

C#: the Lambda handler

SftpBatch/Function.cs
using System.Net;
using System.Text;
using System.Text.Json;
using Amazon.Lambda.Core;
using Amazon.Lambda.S3Events;
using Amazon.S3;
using Amazon.S3.Model;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace Harbourline.DataLake.SftpBatch;

public sealed class Function
{
    private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
    private readonly IAmazonS3 _s3;
    private readonly string _curatedBucket;
    private readonly string _quarantineBucket;

    // Used by Lambda: one client per execution environment, reused across invocations.
    public Function() : this(new AmazonS3Client(),
        Environment.GetEnvironmentVariable("CURATED_BUCKET") ?? throw new InvalidOperationException("CURATED_BUCKET not set"),
        Environment.GetEnvironmentVariable("QUARANTINE_BUCKET") ?? throw new InvalidOperationException("QUARANTINE_BUCKET not set"))
    {
    }

    // Used by tests: inject a fake IAmazonS3.
    public Function(IAmazonS3 s3, string curatedBucket, string quarantineBucket)
    {
        _s3 = s3;
        _curatedBucket = curatedBucket;
        _quarantineBucket = quarantineBucket;
    }

    public async Task HandleAsync(S3Event s3Event, ILambdaContext context)
    {
        foreach (var record in s3Event.Records)
        {
            var bucket = record.S3.Bucket.Name;
            var key = WebUtility.UrlDecode(record.S3.Object.Key);   // keys arrive URL-encoded
            var versionId = record.S3.Object.VersionId;

            if (bucket == _curatedBucket || bucket == _quarantineBucket)
                throw new InvalidOperationException("Refusing to process an output bucket (recursion guard).");

            using var response = await _s3.GetObjectAsync(new GetObjectRequest
            {
                BucketName = bucket,
                Key = key,
                VersionId = string.IsNullOrEmpty(versionId) ? null : versionId,  // exactly the bytes that triggered us
            });
            using var reader = new StreamReader(response.ResponseStream, new UTF8Encoding(false, throwOnInvalidBytes: true));
            var result = BatchValidator.Validate(reader);

            var fileName = Path.GetFileName(key);
            var ingestDate = key.Split('/').FirstOrDefault(p => p.StartsWith("ingest_date=", StringComparison.Ordinal)) ?? "ingest_date=unknown";
            string outputKey;

            if (result.Accepted)
            {
                // Deterministic key from the input: re-processing the same file overwrites with identical content.
                outputKey = $"payment_batches/partner={result.Header!.PartnerId}/execution_date={result.Header.RequestedExecutionDate:yyyy-MM-dd}/{result.Header.BatchId}.jsonl";
                var body = string.Join('\n', result.Lines.Select(l => JsonSerializer.Serialize(new
                {
                    batchId = result.Header.BatchId,
                    l.LineNumber, l.EndToEndId, l.CreditorName, l.CreditorIbanMasked, l.Amount, l.Scheme, l.Reference,
                    debtorAccountId = result.Header.DebtorAccountId,
                    currency = result.Header.Currency,
                    sourceKey = key,
                    sourceVersionId = versionId,
                }, Json))) + "\n";
                await PutAsync(_curatedBucket, outputKey, body, "application/x-ndjson");
            }
            else
            {
                outputKey = $"payment_batches/{ingestDate}/{fileName}.issues.json";
                var report = JsonSerializer.Serialize(new
                {
                    sourceBucket = bucket, sourceKey = key, sourceVersionId = versionId,
                    batchId = result.Header?.BatchId,
                    result.DeclaredCount, result.DeclaredSum, result.ActualSum,
                    issues = result.Issues,
                }, Json);
                await PutAsync(_quarantineBucket, outputKey, report, "application/json");
            }

            // One structured summary per file: counts and codes only, never names or IBANs.
            context.Logger.LogInformation(JsonSerializer.Serialize(new
            {
                message = "batch validated",
                sourceKey = key,
                accepted = result.Accepted,
                lines = result.Lines.Count,
                declaredCount = result.DeclaredCount,
                declaredSum = result.DeclaredSum,
                actualSum = result.ActualSum,
                issueCodes = result.Issues.Select(i => i.Code).Distinct(),
                outputKey,
                requestId = context.AwsRequestId,
            }, Json));
        }
    }

    private Task PutAsync(string bucket, string key, string body, string contentType) =>
        _s3.PutObjectAsync(new PutObjectRequest
        {
            BucketName = bucket,
            Key = key,
            ContentBody = body,
            ContentType = contentType,
            ChecksumAlgorithm = ChecksumAlgorithm.SHA256,
        });
}

C#: tests (xUnit v3 on Microsoft.Testing.Platform)

SftpBatch.Tests/SftpBatch.Tests.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <OutputType>Exe</OutputType>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="xunit.v3" Version="4.0.3" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="../SftpBatch/SftpBatch.csproj" />
    <None Include="../../datasets/samples/sftp/*.csv" Link="samples/%(Filename)%(Extension)" CopyToOutputDirectory="PreserveNewest" />
  </ItemGroup>
</Project>
SftpBatch.Tests/BatchValidatorTests.cs
using Harbourline.DataLake.SftpBatch;

namespace SftpBatch.Tests;

public class BatchValidatorTests
{
    private static BatchResult Load(string file) =>
        BatchValidator.Validate(new StreamReader(Path.Combine(AppContext.BaseDirectory, "samples", file)));

    [Fact]
    public void Clean_partner_file_is_accepted_and_matches_its_trailer()
    {
        var result = Load("HLPG_PTN0042_PAYMENTS_20261116_001.csv");

        Assert.True(result.Accepted);
        Assert.Equal(40, result.Lines.Count);
        Assert.Equal(40, result.DeclaredCount);
        Assert.Equal(89281.75m, result.ActualSum);
        Assert.Equal(new DateOnly(2026, 11, 18), result.Header!.RequestedExecutionDate);
        Assert.All(result.Lines, l => Assert.Contains("*", l.CreditorIbanMasked));
    }

    [Fact]
    public void Messy_partner_file_is_rejected_with_every_injected_defect()
    {
        var result = Load("HLPG_PTN0042_PAYMENTS_20261116_001_messy.csv");
        var codes = result.Issues.Select(i => i.Code).ToHashSet();

        Assert.False(result.Accepted);
        Assert.Contains("INVALID_DATE", codes);     // header date 18/11/2026
        Assert.Contains("INVALID_IBAN", codes);     // line 3
        Assert.Contains("INVALID_AMOUNT", codes);   // line 4: "1,467.65"
        Assert.Contains("DUPLICATE", codes);        // line 6 repeats line 5
        Assert.Contains("TRAILER_COUNT", codes);    // 41 details vs trailer 40
    }

    [Fact]
    public void Truncated_file_without_trailer_is_rejected()
    {
        var result = BatchValidator.Validate(new StringReader(
            "H,HLPG-1,PTN0042,Saltmarsh,2026-11-16T21:30:00Z,ACC-1000000018,GB62HLBK99441212488220,GBP,2026-11-18\n" +
            "D,1,E2E1,Test Person,GB09VMHY88522720202824,10.00,BACS,REF\n"));

        Assert.Contains(result.Issues, i => i.Code == "TRAILER_MISSING");
    }
}

The tests reference the committed samples with a relative path; adjust ../../datasets/... to where your project sits in the repository.

Python: the same function

sftp_batch_validator.py
"""Python 3.13 Lambda: validate a Harbourline partner payment batch (header/detail/trailer)."""
import csv
import io
import json
import os
import re
import urllib.parse
from decimal import Decimal

import boto3

s3 = boto3.client("s3")                      # created once per execution environment
CURATED = os.environ["CURATED_BUCKET"]
QUARANTINE = os.environ["QUARANTINE_BUCKET"]
AMOUNT = re.compile(r"^\d{1,16}\.\d{2}$")
SCHEMES = {"FPS", "BACS", "CHAPS", "SEPA_CT", "SEPA_INST"}


def iban_ok(value: str) -> bool:
    s = value.replace(" ", "").upper()
    if not (15 <= len(s) <= 34 and s.isalnum()):
        return False
    digits = "".join(str(int(ch, 36)) for ch in s[4:] + s[:4])   # 'A' -> 10 ... 'Z' -> 35
    return int(digits) % 97 == 1


def validate(text: str) -> dict:
    issues, lines, seen, header, trailer, total = [], [], set(), None, None, Decimal("0")
    rows = list(csv.reader(io.StringIO(text)))
    for n, f in enumerate(rows, start=1):
        kind = f[0] if f else ""
        if kind == "H" and n == 1:
            header = f
            if not re.match(r"^\d{4}-\d{2}-\d{2}$", f[8]):
                issues.append({"line": n, "code": "INVALID_DATE"})
        elif kind == "D":
            problems = []
            if f[2] in seen: problems.append("DUPLICATE")
            seen.add(f[2])
            if not iban_ok(f[4]): problems.append("INVALID_IBAN")
            if not AMOUNT.match(f[5]): problems.append("INVALID_AMOUNT")
            if f[6] not in SCHEMES: problems.append("INVALID_CODE")
            issues += [{"line": n, "code": c} for c in problems]
            if not problems:
                total += Decimal(f[5])
                lines.append({"lineNumber": int(f[1]), "endToEndId": f[2], "amount": f[5], "scheme": f[6],
                              "creditorIbanMasked": f[4][:4] + "*" * (len(f[4]) - 8) + f[4][-4:]})
        elif kind == "T":
            trailer = f
    details = sum(1 for f in rows if f and f[0] == "D")
    if trailer is None:
        issues.append({"line": len(rows), "code": "TRAILER_MISSING"})
    else:
        if int(trailer[1]) != details: issues.append({"line": len(rows), "code": "TRAILER_COUNT"})
        if Decimal(trailer[2]) != total: issues.append({"line": len(rows), "code": "TRAILER_SUM"})
    return {"header": header, "lines": lines, "issues": issues, "actual_sum": str(total)}


def handler(event, context):
    for record in event["Records"]:
        bucket = record["s3"]["bucket"]["name"]
        key = urllib.parse.unquote_plus(record["s3"]["object"]["key"])
        version = record["s3"]["object"].get("versionId")
        args = {"Bucket": bucket, "Key": key} | ({"VersionId": version} if version else {})
        text = s3.get_object(**args)["Body"].read().decode("utf-8")     # strict UTF-8
        result = validate(text)
        name = key.rsplit("/", 1)[-1]
        if not result["issues"]:
            h = result["header"]
            out_key = f"payment_batches/partner={h[2]}/execution_date={h[8]}/{h[1]}.jsonl"
            body = "".join(json.dumps(line | {"batchId": h[1], "sourceKey": key}) + "\n" for line in result["lines"])
            s3.put_object(Bucket=CURATED, Key=out_key, Body=body.encode(), ChecksumAlgorithm="SHA256")
        else:
            out_key = f"payment_batches/issues/{name}.issues.json"
            report = {"sourceKey": key, "sourceVersionId": version, "issues": result["issues"]}
            s3.put_object(Bucket=QUARANTINE, Key=out_key, Body=json.dumps(report).encode(), ChecksumAlgorithm="SHA256")
        print(json.dumps({"message": "batch validated", "sourceKey": key, "accepted": not result["issues"],
                          "lines": len(result["lines"]), "outputKey": out_key}))

Code explained line by line

Validator

  1. BatchResult.Accepted => Issues.Count == 0 — all-or-nothing. For payroll, paying 39 of 40 employees is worse than returning the file; the partner fixes and resends.
  2. [GeneratedRegex(...)] — compile-time regexes (.NET 7+): faster cold starts than runtime-compiled regexes, which matters in Lambda.
  3. switch (f[0]) with case "H" when fileLine == 1 — the header must be the first record. Record-type routing is the core of header/detail/trailer formats used across banking (BACS, many SFTP feeds).
  4. trailerSeen and AFTER_TRAILER — anything after the trailer suggests two files concatenated or tampering.
  5. TRAILER_MISSING — the most important check for file transfers: a missing trailer usually means a truncated upload.
  6. TRAILER_COUNT and TRAILER_SUM — the control totals. The sum compares against valid lines only, so any rejected line also shows up as a sum break, which is what operations expects to see.
  7. NumberStyles.None for the trailer count and AllowDecimalPoint for the sum — reject signs, spaces and thousands separators.
  8. seen.Add(f[2]) — duplicate end-to-end IDs inside one file.
  9. Iban.IsValid — ISO 13616 mod-97, processing one digit at a time to avoid big integers (same algorithm as harbourline_etl/iban.py and the lab's C# Iban class).
  10. Iban.Mask — curated output never contains the full creditor IBAN. The lab's pipeline adds a keyed token for joins.

Handler

  1. [assembly: LambdaSerializer(...)] — tells the .NET runtime how to deserialise the S3Event JSON.
  2. Two constructors — Lambda uses the parameterless one; tests inject a fake IAmazonS3. Clients are created once per execution environment and reused across invocations.
  3. WebUtility.UrlDecode(record.S3.Object.Key) — S3 event keys are URL-encoded; forgetting this causes NoSuchKey for any key with =, spaces or special characters.
  4. The bucket check — a second recursion guard: the function refuses to process its own output buckets even if someone misconfigures a trigger. Lambda's recursive loop detection (on by default) also stops S3–Lambda loops after about 16 invocations in a chain, but you should never rely on it.
  5. VersionId = ... — read exactly the version that triggered the event, not whatever is current by the time the function runs.
  6. new UTF8Encoding(false, throwOnInvalidBytes: true) — strict decoding: a mis-encoded partner file fails loudly (and goes to the on-failure destination after retries) instead of silently producing � characters.
  7. Deterministic outputKey — from the batch ID and execution date. Reprocessing the same file writes the same object: idempotent. A second, different file with the same batch ID overwrites it, which is why the downstream payment step must also check batch IDs it has already submitted.
  8. The quarantine report — contains codes, line numbers and totals, not the rejected content. Raw already holds the original file.
  9. context.Logger.LogInformation(JsonSerializer.Serialize(...)) — one JSON log line per file with counts and codes. With the function's log format set to JSON, CloudWatch Logs Insights can query fields directly.
  10. ChecksumAlgorithm.SHA256 on output — integrity evidence for downstream consumers.

Python version

  1. boto3.client("s3") at module level — created once per execution environment, like the C# field.
  2. int(ch, 36) — converts A–Z to 10–35 in one call; Python's arbitrary-precision integers make the mod-97 check a single expression.
  3. urllib.parse.unquote_plus — the Python equivalent of UrlDecode.
  4. csv.reader handles the quoted "1,467.65" field in the messy file correctly; the amount regex then rejects it.

Security considerations

⚠ Learning shortcut — not for production

The function writes masked IBANs but no keyed token, has no VPC, and the lab uses a local SFTP container (labs/m07-transaction-pipeline/local/docker-compose.yml, password authentication) instead of AWS Transfer Family. In production: Transfer Family with SSH keys per partner and logical directories per partner prefix; a VPC-attached function with an S3 gateway endpoint if network policy requires it; tokenisation with a key from Secrets Manager; code signing for Lambda deployments; and alarms on quarantine counts.

  • Least privilege by prefix. Read only transactions/source=sftp/*; write only payment_batches/*. The function cannot read other partners' or other sources' files.
  • Recursion. Never write to the bucket and prefix that trigger the function. Keep recursive loop detection set to Terminate.
  • Untrusted input. Partner files can be malformed or malicious (huge lines, wrong encodings, formula injection such as =HYPERLINK(...) in names that later opens in Excel). Validate shape strictly; if files are exported to spreadsheets later, neutralise leading =, +, -, @.
  • Logs. Log counts and codes, not names, IBANs or amounts per person. Set log retention (14 days here).
  • Reserved concurrency. Caps the blast radius of a flood of files (or a loop) on both your bill and downstream systems.

Banking use case

Harbourline's Partner Gateway flow for payroll bureaus:

  1. Partner uploads HLPG_<partner>_PAYMENTS_<date>_<seq>.csv over SFTP; Transfer Family writes it to raw/transactions/source=sftp/partner=<id>/ingest_date=.../.
  2. This Lambda validates it within a second. Accepted batches become JSON Lines in curated; the Payments Service picks them up (EventBridge rule on the curated prefix) and creates one payment per line with the end-to-end ID as idempotency key.
  3. Rejected batches produce an issues report; an EventBridge rule on the quarantine prefix notifies the partner-operations team, who contact the bureau. Nothing is paid from a rejected file.
  4. The nightly reconciliation (Glue/Athena) compares accepted batches, submitted payments and settled payments by batch ID and control sum.

Volumes are small (hundreds of files per day, each a few KB to a few MB), so Lambda costs pennies per month, well inside the free tier.

Common mistakes

  • Writing output to the triggering bucket/prefix → infinite loop and a surprise bill → separate buckets or prefixes, plus a code guard.
  • Not URL-decoding keys → NoSuchKey for keys with = or spaces → decode first.
  • Reading the "latest" object instead of the event's version → validating different bytes from those that arrived → use VersionId.
  • Loading large files into memory → out-of-memory or timeouts → stream line by line; switch to Glue for large files.
  • Partial acceptance of batch files → some payments sent, some not → all-or-nothing for batches with control totals.
  • No failure destination → failed async events vanish after retries → on-failure destination (SQS) and an alarm.
  • Creating SDK clients inside the handler → slower invocations → create once per execution environment.

Troubleshooting

SymptomLikely causeFix
Function never runs after uploadNotification filter mismatch (prefix/suffix), or missing lambda:InvokeFunction permission for S3Check get-bucket-notification-configuration; add the permission
NoSuchKeyKey not URL-decodedWebUtility.UrlDecode / unquote_plus
AccessDenied on GetObjectMissing s3:GetObjectVersion when using version IDs, or kms:DecryptAdd both
Task timed out after 60.00 secondsFile larger than expected, or slow S3 writesRaise timeout modestly; stream; move big files to Glue
DecoderFallbackException / UnicodeDecodeErrorPartner sent Latin-1Expected: event goes to the on-failure destination; agree UTF-8 in the contract or repair with a warning
RecursiveInvocationsDropped metric above 0Output written to the trigger locationFix the output path; check the trigger filter
Throttles during month-endReserved concurrency or account concurrency reachedPut SQS between S3 and Lambda; raise reserved concurrency deliberately
EventBridge rules for raw stopped firing after deploying the triggerBucket notification configuration replaced without EventBridgeConfigurationInclude eventbridge = true (Terraform) or re-enable EventBridge on the bucket

Cost and cleanup

Expected cost: free. A few invocations at 512 MB for under a second each are well inside the monthly free tier (1 million requests, 400,000 GB-seconds). Even outside it, 100 files cost about USD 0.0003. SQS queue with no messages: free. CloudWatch Logs for a few KB: effectively free. Unit tests run locally at no cost.

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)

cdk destroy HlM07SftpBatch            # or: terraform destroy
# CLI route:
aws lambda delete-function --function-name hl-m07-sftp-batch-validator
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m07-sftp-batch-dlq --query QueueUrl --output text)"
aws logs delete-log-group --log-group-name /aws/lambda/hl-m07-sftp-batch-validator
aws iam delete-role-policy --role-name hl-m07-sftp-batch-validator-role --policy-name sftp-batch
aws iam delete-role --role-name hl-m07-sftp-batch-validator-role

# Restore the raw bucket's notification configuration to EventBridge only
aws s3api put-bucket-notification-configuration --bucket "hl-m07-raw-$ACCOUNT" \
  --notification-configuration '{"EventBridgeConfiguration":{}}'

# Outputs
aws s3 rm "s3://hl-m07-curated-$ACCOUNT/payment_batches/" --recursive
aws s3 rm "s3://hl-m07-quarantine-$ACCOUNT/payment_batches/" --recursive

aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --query "ResourceTagMappingList[].ResourceARN"

Exercises

  • 🟢 Guided: Run the three xUnit tests. Then deploy, upload both sample files, and capture the JSON log line and the output object for each.
  • 🟡 Challenge: Add a rule that the header's debtor_account_id must belong to a list of approved debtor accounts per partner (store the list in Parameter Store as a StringList and cache it for 5 minutes in the function). Return UNAUTHORISED_DEBTOR when it does not match.
  • 🔴 Live problem: At 23:58 on the last working day of the month, the bureau uploads a 180 MB file with 1.2 million salary lines for a new large client. The function times out at 60 seconds, retries twice, and the event lands in the DLQ. Payroll must be submitted by 06:00. What do you do tonight, and what do you change for next month?

Assignment

Add the SftpBatch function and its tests to your portfolio repository with a README covering: the record layout, the rules and reason codes, the all-or-nothing decision, the recursion design, the Lambda-vs-Glue threshold you would set, and the exact monthly cost estimate at 300 files per day.

Solutions

Knowledge check

Interview questions

  • "Lambda or Glue for ETL?" — Lambda for small, per-file, low-latency transforms and routing; Glue for large, multi-file, joined or long-running work. Measure size and duration and set a threshold; route large files to Glue automatically.
  • "How do you make an S3-triggered function idempotent?" — Deterministic output keys from input identity, conditional writes where needed, process the event's version ID, and downstream de-duplication on business keys (batch ID, end-to-end ID).
  • "How do you handle failures in asynchronous Lambda invocations?" — Retries (0–2), maximum event age, an on-failure destination (SQS, SNS, EventBridge or another Lambda), alarms on it, and a replay procedure.
  • "What do you log from a data function?" — Structured JSON with counts, codes, keys and request IDs; never personal data.

Certification practice

Completion checklist

  • I can state Lambda's timeout, memory, /tmp and payload limits and when they push work to Glue
  • My validator passes the clean, messy and truncated-file tests
  • I deployed the function with least-privilege prefix permissions and an on-failure destination
  • I proved the function cannot trigger itself and that re-processing is idempotent
  • I deleted the function, queue, role and log group and restored the bucket notification

Key takeaways

  • Lambda is the right tool for small, per-file, low-latency transforms; Glue for large, joined or long-running work.
  • Put rules in a pure class or module, unit-test against the sample files, and keep the handler thin.
  • Batch files with trailers are all-or-nothing: check record count and control sum before anything is paid.
  • Design out recursion (separate output buckets/prefixes), read the event's version, and write deterministic outputs.
  • Lambda at these volumes is effectively free; the real costs are mistakes such as loops and silent failures.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page