Lightweight transforms with Lambda: C# and Python functions for small files
Validate Harbourline's partner SFTP payment batches with an S3-triggered Lambda function in C# (.NET 10) or Python, including trailer control totals, IBAN checks and quarantine, and learn exactly where Lambda's limits make Glue the better tool.
Assumes: .NET 10 LTS (Lambda runtime dotnet10) · Amazon.Lambda.Core 3.4.0 · Amazon.Lambda.S3Events 4.0.0 · Amazon.Lambda.Serialization.SystemTextJson 3.0.1 · AWSSDK.S3 4.1.1 · Python 3.13 runtime · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Decide between Lambda and Glue (or Athena SQL) for a transform, using file size, run time, memory, concurrency and cost.
- Build an S3-triggered Lambda function in C# that validates a partner payment batch file: header, detail lines, trailer record count and control sum, IBAN check digits and duplicates, all-or-nothing.
- Write the equivalent function in Python and compare the two.
- Make the function idempotent, safe against recursive invocation loops, and observable with structured logs.
- Deploy it with CDK (C#) or Terraform, test it locally for free, and know its exact cost.
Prerequisites
- S3 data lake design (buckets, keys, KMS keys)
- PySpark in Glue for beginners (so you can compare)
- Module 3: .NET on AWS for Lambda basics with .NET
- Tools: .NET 10 SDK (or Docker), AWS CLI v2 (
learn-sandbox), CDK CLI or Terraform
Why this matters in banking
Every evening, Harbourline's partner bureau Saltmarsh Payroll Services Ltd (fictional) drops a payroll file on the Partner Gateway SFTP server: one header, about 40 detail lines (one per employee payment) and a trailer with the record count and control sum. The bank must not send any of those payments unless the file is complete and internally consistent. If the trailer says 40 payments totalling £89,281.75 and the bank processes 41 payments or a different total, someone gets paid twice or not at all, and the bureau's client has a very bad payday.
The file is a few kilobytes. Starting a Spark cluster for it would take longer than the validation itself and cost more. A Lambda function that runs in milliseconds as soon as the file lands is the right tool, and the same pattern works for many small banking feeds: FX rate files, sanctions-list deltas, card-scheme fee tables, partner acknowledgements.
The idea in plain language
AWS Lambda runs a function when an event happens and bills only for the time it runs. For data engineering, the event is usually "a file arrived in S3". Lambda reads the file, checks or converts it, writes results somewhere else, and finishes.
The trade-off is simple: Lambda is excellent for small, quick, per-file work and poor for large, joined, multi-file work. It has a hard limit of 15 minutes per invocation, up to 10,240 MB of memory, up to 10,240 MB of /tmp disk, and processes one event on one machine. Spark (Glue) spreads big work across many machines but takes time to start.
Analogy: Lambda is the cashier who checks each envelope as it comes through the slot: open, count, compare with the slip, stamp "accepted" or "returned". Glue is the overnight back-office team that processes the whole day's mountain of paperwork together. You would not call in the overnight team for one envelope, and you would not ask the cashier to reconcile a million transactions.
What problem the AWS service solves
Lambda removes servers and schedulers from small transforms: no cluster, no polling loop, no VM to patch. Events from S3 (directly or through EventBridge), SQS, Kinesis and others invoke it; it scales out automatically (default 1,000 concurrent executions per Region per account); and you pay per request and per GB-second.
Limits that decide the design (checked 2026-11-18)
| Limit | Value | Why it matters for data work |
|---|---|---|
| Timeout | 900 seconds (15 minutes) maximum | Large files or slow downstream calls can hit it; design for minutes, not hours |
| Memory | 128 MB to 10,240 MB (CPU scales with memory; about 1 vCPU at 1,769 MB) | More memory also means more CPU |
/tmp ephemeral storage | 512 MB to 10,240 MB | Spill space for conversions; billed above 512 MB |
| Invocation payload | 6 MB synchronous request/response; 1 MB asynchronous | S3 events carry only the key, so file size is not limited by payload; read the file with GetObject |
| Deployment package | 50 MB zipped upload, 250 MB unzipped (including layers); container images up to 10 GB | Big libraries (pyarrow, pandas) push you to layers or container images |
| Concurrency | 1,000 per Region by default (new accounts may start lower) | A burst of 5,000 files throttles; use SQS between S3 and Lambda for back-pressure |
| Environment variables | 4 KB total | Put configuration in Parameter Store or Secrets Manager |
Lambda or Glue?
| Question | Lambda | Glue (Spark) |
|---|---|---|
| Input size per run | KB to a few hundred MB | Hundreds of MB to TB |
| Run time | Milliseconds to a few minutes (hard 15-minute cap) | Minutes to hours |
| Start-up | Milliseconds (warm) to about a second (cold) | Tens of seconds to minutes |
| Joins across many files or tables | Awkward | Natural |
| Languages | C#, Python, Java, Node.js, others | Python (PySpark), Scala |
| Cost at small scale | Fractions of a cent; free tier covers most labs | 1-minute minimum × 2 DPUs ≈ USD 0.015 per run minimum |
| Cost at large scale | Grows with GB-seconds; can exceed Glue for long CPU-heavy work | Efficient per GB processed |
| Catalogue, bookmarks, Data Quality | Build it yourself | Built in |
Rules of thumb Harbourline uses:
- Per-file checks, routing, small conversions, notifications → Lambda.
- Anything joining or aggregating across files, anything over about 1 GB, anything needing Spark → Glue.
- Set-based SQL over data already in the lake → Athena CTAS/
INSERT INTO(see Amazon Athena). - When in doubt, measure: a Lambda that routinely runs more than a few minutes is a Glue job in disguise.
Pricing (us-east-1, checked 2026-11-18)
- USD 0.20 per 1 million requests.
- USD 0.0000166667 per GB-second on x86_64; about 20% less on arm64 (Graviton).
- Always-free tier: 1 million requests and 400,000 GB-seconds per month.
/tmpabove 512 MB is billed per GB-second.
Validating one 4 KB batch file at 512 MB for 300 ms costs 0.5 GB × 0.3 s = 0.15 GB-seconds ≈ USD 0.0000025 plus the request: effectively free, and inside the free tier anyway.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Functions with a Blob trigger | Lambda with an S3 event notification or EventBridge rule | S3 events are push-based; no polling of the container |
Event Grid BlobCreated → Function | S3 → EventBridge → Lambda | EventBridge rules filter on bucket, key prefix and suffix |
| Functions Consumption plan timeout | Lambda 15-minute maximum | Durable orchestration lives in Step Functions (or Lambda durable functions) |
| SSIS script task on a small file | Lambda function | Event-driven, per file, no SQL Agent schedule |
ILogger structured logs to App Insights | Lambda logs to CloudWatch Logs (JSON format option) | Use Powertools for AWS Lambda (.NET) for structured logging and metrics |
| Function app settings | Environment variables (4 KB), Parameter Store, Secrets Manager | Keep secrets out of environment variables |
Architecture
The function reads from raw and writes only to curated and quarantine, so its output can never trigger it again. That design choice, not luck, prevents recursive loops.
Security-boundary view: the function's role may GetObject only under transactions/source=sftp/ in raw, PutObject only under payment_batches/ in curated and quarantine, and use the two zone keys. It has no network access beyond AWS APIs (no VPC needed; add a VPC with S3 gateway endpoint in production if policy requires private paths).
Step-by-step implementation
Write the validator as a pure class
Put the rules in BatchValidator (no AWS types), so you can unit-test it against the committed samples: the clean file must be accepted, and the messy file must be rejected with exactly the defects listed in HLPG_PTN0042_PAYMENTS_20261116_001_messy_defects.csv.
Wrap it in a Lambda handler
The handler receives an S3Event, reads each object (by version ID, so it validates exactly the bytes that triggered it), calls the validator and writes one output: accepted lines to curated, or an issues report to quarantine.
Test locally (free)
Run the xUnit tests with dotnet test, or in Docker. No AWS account needed.
Deploy and trigger
Deploy with CDK or Terraform. Copy the clean and messy sample files to the raw prefix; watch the outputs and logs.
Prove idempotency
Upload the same file again (a new S3 version). The function produces the same output key with the same content: re-processing is harmless. Then upload a renamed copy and see a second output, which is why the batch ID (from the header) is checked downstream for duplicates.
AWS Console instructions
- Console → Lambda → Create function → Author from scratch: name
hl-m07-sftp-batch-validator; runtime .NET 10 (C#/F#/PowerShell); architecture arm64; execution role: existing rolehl-m07-sftp-batch-validator-role(from IaC) or create one and attach the policy from the IaC tab. - Code → Upload from → .zip file: the output of
dotnet lambda package(ordotnet publishzipped). Runtime settings → Edit → Handler:SftpBatch::Harbourline.DataLake.SftpBatch.Function::HandleAsync. - Configuration → General: memory 512 MB, timeout 60 seconds, ephemeral storage 512 MB.
- Configuration → Environment variables:
CURATED_BUCKET,QUARANTINE_BUCKET. - Configuration → Asynchronous invocation: retry attempts 2; maximum event age 1 hour; On-failure destination → SQS queue
hl-m07-sftp-batch-dlq. - Configuration → Monitoring and operations tools → Logging configuration: log format JSON, application log level INFO; log group retention 14 days.
- Add trigger → S3: bucket
hl-m07-raw-<account-id>; event types All object create events; prefixtransactions/source=sftp/; suffix.csv. Acknowledge the recursive-invocation warning only because the function never writes to this bucket. - Configuration → Concurrency and recursion detection: reserved concurrency 10 (protects downstream and your bill); recursive loop detection Terminate (the default).
AWS CLI
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
RAW=hl-m07-raw-$ACCOUNT
# Package the C# function (Amazon.Lambda.Tools: dotnet tool install -g Amazon.Lambda.Tools)
dotnet lambda package --project-location SftpBatch --configuration Release \
--function-architecture arm64 --output-package sftp-batch.zip
# Trigger it: land the clean and the messy partner file
aws s3 cp datasets/samples/sftp/HLPG_PTN0042_PAYMENTS_20261116_001.csv \
"s3://$RAW/transactions/source=sftp/ingest_date=2026-11-16/HLPG_PTN0042_PAYMENTS_20261116_001.csv"
aws s3 cp datasets/samples/sftp/HLPG_PTN0042_PAYMENTS_20261116_001_messy.csv \
"s3://$RAW/transactions/source=sftp/ingest_date=2026-11-16/HLPG_PTN0042_PAYMENTS_20261116_002.csv"
# Read the function's JSON summaries from the last 10 minutes
aws logs tail /aws/lambda/hl-m07-sftp-batch-validator --since 10m --format short
# Look at the outputs
aws s3 ls "s3://hl-m07-curated-$ACCOUNT/payment_batches/" --recursive
aws s3 ls "s3://hl-m07-quarantine-$ACCOUNT/payment_batches/" --recursive
aws s3 cp "s3://hl-m07-quarantine-$ACCOUNT/payment_batches/ingest_date=2026-11-16/HLPG_PTN0042_PAYMENTS_20261116_002.csv.issues.json" - | head -40
# Invoke directly with a hand-made S3 event (useful for testing without uploads)
cat > event.json <<EOF
{"Records":[{"eventSource":"aws:s3","s3":{"bucket":{"name":"$RAW"},
"object":{"key":"transactions/source%3Dsftp/ingest_date%3D2026-11-16/HLPG_PTN0042_PAYMENTS_20261116_001.csv"}}}]}
EOF
aws lambda invoke --function-name hl-m07-sftp-batch-validator \
--cli-binary-format raw-in-base64-out --payload file://event.json response.json
# Metrics: invocations, errors, duration, recursive invocations dropped
aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Duration \
--dimensions Name=FunctionName,Value=hl-m07-sftp-batch-validator \
--start-time "$(date -u -d '-1 hour' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
--period 3600 --statistics Average MaximumNote that keys in S3 events are URL-encoded (= becomes %3D, spaces become +); the function decodes them.
Infrastructure as Code
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.Lambda.Destinations;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.S3;
using Amazon.CDK.AWS.S3.Notifications;
using Amazon.CDK.AWS.SQS;
using Constructs;
namespace Harbourline.DataLake.Infra;
public sealed class SftpBatchStack : Stack
{
public SftpBatchStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
var account = Aws.ACCOUNT_ID;
var raw = Bucket.FromBucketName(this, "Raw", $"hl-m07-raw-{account}");
var curated = Bucket.FromBucketName(this, "Curated", $"hl-m07-curated-{account}");
var quarantine = Bucket.FromBucketName(this, "Quarantine", $"hl-m07-quarantine-{account}");
var rawKey = Key.FromLookup(this, "RawKey", new KeyLookupOptions { AliasName = "alias/hl-m07-raw" });
var curatedKey = Key.FromLookup(this, "CuratedKey", new KeyLookupOptions { AliasName = "alias/hl-m07-curated" });
var dlq = new Queue(this, "Dlq", new QueueProps
{
QueueName = "hl-m07-sftp-batch-dlq",
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
RetentionPeriod = Duration.Days(4),
});
var logGroup = new LogGroup(this, "Logs", new LogGroupProps
{
LogGroupName = "/aws/lambda/hl-m07-sftp-batch-validator",
Retention = RetentionDays.TWO_WEEKS,
RemovalPolicy = RemovalPolicy.DESTROY,
});
var fn = new Function(this, "Validator", new FunctionProps
{
FunctionName = "hl-m07-sftp-batch-validator",
Runtime = new Runtime("dotnet10", RuntimeFamily.DOTNET_CORE),
Architecture = Architecture.ARM_64,
Handler = "SftpBatch::Harbourline.DataLake.SftpBatch.Function::HandleAsync",
Code = Code.FromAsset("../SftpBatch/bin/Release/net10.0/publish"),
MemorySize = 512,
Timeout = Duration.Seconds(60),
ReservedConcurrentExecutions = 10,
LogGroup = logGroup,
LoggingFormat = LoggingFormat.JSON,
RetryAttempts = 2,
MaxEventAge = Duration.Hours(1),
OnFailure = new SqsDestination(dlq),
Environment = new Dictionary<string, string>
{
["CURATED_BUCKET"] = curated.BucketName,
["QUARANTINE_BUCKET"] = quarantine.BucketName,
},
});
// Least privilege: read only the SFTP prefix, write only payment_batches/.
fn.AddToRolePolicy(new PolicyStatement(new PolicyStatementProps
{
Actions = new[] { "s3:GetObject", "s3:GetObjectVersion" },
Resources = new[] { raw.ArnForObjects("transactions/source=sftp/*") },
}));
fn.AddToRolePolicy(new PolicyStatement(new PolicyStatementProps
{
Actions = new[] { "s3:PutObject" },
Resources = new[] { curated.ArnForObjects("payment_batches/*"), quarantine.ArnForObjects("payment_batches/*") },
}));
rawKey.GrantEncryptDecrypt(fn); // raw + quarantine
curatedKey.GrantEncrypt(fn); // write curated
raw.AddEventNotification(EventType.OBJECT_CREATED, new LambdaDestination(fn),
new NotificationKeyFilter { Prefix = "transactions/source=sftp/", Suffix = ".csv" });
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m07-transaction-pipeline");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "your-name");
}
}Publish before cdk deploy: dotnet publish SftpBatch -c Release -r linux-arm64 --self-contained false -o SftpBatch/bin/Release/net10.0/publish. Adding a notification to an imported bucket deploys a small custom resource that calls PutBucketNotificationConfiguration; it merges with existing configuration such as the EventBridge setting ⚠ VERIFY on your CDK version, so check the bucket's notification configuration afterwards.
Complete code
C#: project file
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<RootNamespace>Harbourline.DataLake.SftpBatch</RootNamespace>
<AssemblyName>SftpBatch</AssemblyName>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
<CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Amazon.Lambda.Core" Version="3.4.0" />
<PackageReference Include="Amazon.Lambda.S3Events" Version="4.0.0" />
<PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="3.0.1" />
<PackageReference Include="AWSSDK.S3" Version="4.1.1" />
</ItemGroup>
</Project>C#: the validator (no AWS dependencies)
using System.Globalization;
using System.Text;
using System.Text.RegularExpressions;
namespace Harbourline.DataLake.SftpBatch;
public sealed record BatchHeader(string BatchId, string PartnerId, string DebtorAccountId,
string Currency, DateOnly RequestedExecutionDate);
public sealed record BatchLine(int LineNumber, string EndToEndId, string CreditorName,
string CreditorIbanMasked, decimal Amount, string Scheme, string Reference);
public sealed record Issue(int FileLine, string Code, string Message);
public sealed record BatchResult(BatchHeader? Header, IReadOnlyList<BatchLine> Lines, IReadOnlyList<Issue> Issues,
int? DeclaredCount, decimal? DeclaredSum, decimal ActualSum)
{
// All-or-nothing: a payroll batch is either entirely valid or returned to the partner.
public bool Accepted => Issues.Count == 0;
}
public static partial class BatchValidator
{
private static readonly HashSet<string> Schemes = ["FPS", "BACS", "CHAPS", "SEPA_CT", "SEPA_INST"];
private static readonly HashSet<string> Currencies = ["GBP", "EUR"];
[GeneratedRegex(@"^\d{1,16}\.\d{2}$")] private static partial Regex PositiveAmount();
[GeneratedRegex(@"^ACC-\d{10}$")] private static partial Regex AccountId();
public static BatchResult Validate(TextReader reader)
{
var issues = new List<Issue>();
var lines = new List<BatchLine>();
var seenEndToEnd = new HashSet<string>();
BatchHeader? header = null;
int? declaredCount = null;
decimal? declaredSum = null;
var actualSum = 0m;
var detailCount = 0;
var fileLine = 0;
var trailerSeen = false;
while (reader.ReadLine() is { } text)
{
fileLine++;
if (text.Length == 0) continue;
var f = Csv.Split(text);
if (trailerSeen)
{
issues.Add(new(fileLine, "AFTER_TRAILER", "content after the trailer record"));
continue;
}
switch (f[0])
{
case "H" when fileLine == 1:
header = ParseHeader(f, fileLine, issues);
break;
case "D" when header is not null || fileLine > 1:
detailCount++;
if (ParseDetail(f, fileLine, issues, seenEndToEnd) is { } line)
{
lines.Add(line);
actualSum += line.Amount;
}
break;
case "T":
trailerSeen = true;
if (f.Length != 3 || !int.TryParse(f[1], NumberStyles.None, CultureInfo.InvariantCulture, out var count)
|| !decimal.TryParse(f[2], NumberStyles.AllowDecimalPoint, CultureInfo.InvariantCulture, out var sum))
{
issues.Add(new(fileLine, "TRAILER_INVALID", "trailer must be T,record_count,control_sum"));
break;
}
declaredCount = count;
declaredSum = sum;
break;
default:
issues.Add(new(fileLine, "RECORD_TYPE", $"unexpected record type '{f[0]}' on line {fileLine}"));
break;
}
}
if (header is null) issues.Add(new(1, "HEADER_MISSING", "first record must be the header (H)"));
if (!trailerSeen) issues.Add(new(fileLine, "TRAILER_MISSING", "file has no trailer (T): possibly truncated"));
if (declaredCount is { } dc && dc != detailCount)
issues.Add(new(fileLine, "TRAILER_COUNT", $"trailer declares {dc} records, file has {detailCount}"));
if (declaredSum is { } ds && ds != actualSum)
issues.Add(new(fileLine, "TRAILER_SUM", $"trailer control sum {ds:0.00} differs from valid lines' sum {actualSum:0.00}"));
return new BatchResult(header, lines, issues, declaredCount, declaredSum, actualSum);
}
private static BatchHeader? ParseHeader(string[] f, int fileLine, List<Issue> issues)
{
// H,batch_id,partner_id,partner_name,created_at,debtor_account_id,debtor_iban,currency,requested_execution_date
if (f.Length != 9) { issues.Add(new(fileLine, "HEADER_INVALID", "header must have 9 fields")); return null; }
var ok = true;
if (!AccountId().IsMatch(f[5])) { issues.Add(new(fileLine, "INVALID_ID", "debtor_account_id")); ok = false; }
if (!Iban.IsValid(f[6])) { issues.Add(new(fileLine, "INVALID_IBAN", "debtor_iban fails mod-97")); ok = false; }
if (!Currencies.Contains(f[7])) { issues.Add(new(fileLine, "INVALID_CURRENCY", $"currency '{f[7]}'")); ok = false; }
if (!DateOnly.TryParseExact(f[8], "yyyy-MM-dd", CultureInfo.InvariantCulture, DateTimeStyles.None, out var execDate))
{
issues.Add(new(fileLine, "INVALID_DATE", $"requested_execution_date '{f[8]}' is not ISO yyyy-MM-dd"));
ok = false;
}
return ok ? new BatchHeader(f[1], f[2], f[5], f[7], execDate) : new BatchHeader(f[1], f[2], f[5], f[7], default);
}
private static BatchLine? ParseDetail(string[] f, int fileLine, List<Issue> issues, HashSet<string> seen)
{
// D,line_number,end_to_end_id,creditor_name,creditor_iban,amount,scheme,reference
if (f.Length != 8) { issues.Add(new(fileLine, "DETAIL_INVALID", "detail must have 8 fields")); return null; }
var before = issues.Count;
if (!seen.Add(f[2])) issues.Add(new(fileLine, "DUPLICATE", $"end_to_end_id {f[2]} repeated"));
if (!Iban.IsValid(f[4])) issues.Add(new(fileLine, "INVALID_IBAN", "creditor_iban fails mod-97"));
if (!PositiveAmount().IsMatch(f[5])) issues.Add(new(fileLine, "INVALID_AMOUNT", $"amount '{f[5]}' must be like 1234.56"));
if (!Schemes.Contains(f[6])) issues.Add(new(fileLine, "INVALID_CODE", $"scheme '{f[6]}'"));
if (issues.Count != before) return null;
return new BatchLine(int.Parse(f[1], CultureInfo.InvariantCulture), f[2], f[3], Iban.Mask(f[4]),
decimal.Parse(f[5], CultureInfo.InvariantCulture), f[6], f[7]);
}
}
public static class Iban
{
public static bool IsValid(string value)
{
var s = value.Replace(" ", "").ToUpperInvariant();
if (s.Length is < 15 or > 34 || !s.All(char.IsAsciiLetterOrDigit)) return false;
var remainder = 0;
foreach (var ch in s[4..] + s[..4])
{
var digits = char.IsAsciiLetter(ch) ? (ch - 'A' + 10).ToString(CultureInfo.InvariantCulture) : ch.ToString();
foreach (var d in digits) remainder = (remainder * 10 + (d - '0')) % 97;
}
return remainder == 1;
}
public static string Mask(string iban) => iban.Length <= 8 ? new string('*', iban.Length)
: string.Concat(iban.AsSpan(0, 4), new string('*', iban.Length - 8), iban.AsSpan(iban.Length - 4));
}
public static class Csv
{
public static string[] Split(string line)
{
var fields = new List<string>();
var sb = new StringBuilder();
var quoted = false;
for (var i = 0; i < line.Length; i++)
{
var c = line[i];
if (quoted && c == '"' && i + 1 < line.Length && line[i + 1] == '"') { sb.Append('"'); i++; }
else if (c == '"') quoted = !quoted;
else if (c == ',' && !quoted) { fields.Add(sb.ToString()); sb.Clear(); }
else sb.Append(c);
}
fields.Add(sb.ToString());
return fields.ToArray();
}
}C#: the Lambda handler
using System.Net;
using System.Text;
using System.Text.Json;
using Amazon.Lambda.Core;
using Amazon.Lambda.S3Events;
using Amazon.S3;
using Amazon.S3.Model;
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]
namespace Harbourline.DataLake.SftpBatch;
public sealed class Function
{
private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
private readonly IAmazonS3 _s3;
private readonly string _curatedBucket;
private readonly string _quarantineBucket;
// Used by Lambda: one client per execution environment, reused across invocations.
public Function() : this(new AmazonS3Client(),
Environment.GetEnvironmentVariable("CURATED_BUCKET") ?? throw new InvalidOperationException("CURATED_BUCKET not set"),
Environment.GetEnvironmentVariable("QUARANTINE_BUCKET") ?? throw new InvalidOperationException("QUARANTINE_BUCKET not set"))
{
}
// Used by tests: inject a fake IAmazonS3.
public Function(IAmazonS3 s3, string curatedBucket, string quarantineBucket)
{
_s3 = s3;
_curatedBucket = curatedBucket;
_quarantineBucket = quarantineBucket;
}
public async Task HandleAsync(S3Event s3Event, ILambdaContext context)
{
foreach (var record in s3Event.Records)
{
var bucket = record.S3.Bucket.Name;
var key = WebUtility.UrlDecode(record.S3.Object.Key); // keys arrive URL-encoded
var versionId = record.S3.Object.VersionId;
if (bucket == _curatedBucket || bucket == _quarantineBucket)
throw new InvalidOperationException("Refusing to process an output bucket (recursion guard).");
using var response = await _s3.GetObjectAsync(new GetObjectRequest
{
BucketName = bucket,
Key = key,
VersionId = string.IsNullOrEmpty(versionId) ? null : versionId, // exactly the bytes that triggered us
});
using var reader = new StreamReader(response.ResponseStream, new UTF8Encoding(false, throwOnInvalidBytes: true));
var result = BatchValidator.Validate(reader);
var fileName = Path.GetFileName(key);
var ingestDate = key.Split('/').FirstOrDefault(p => p.StartsWith("ingest_date=", StringComparison.Ordinal)) ?? "ingest_date=unknown";
string outputKey;
if (result.Accepted)
{
// Deterministic key from the input: re-processing the same file overwrites with identical content.
outputKey = $"payment_batches/partner={result.Header!.PartnerId}/execution_date={result.Header.RequestedExecutionDate:yyyy-MM-dd}/{result.Header.BatchId}.jsonl";
var body = string.Join('\n', result.Lines.Select(l => JsonSerializer.Serialize(new
{
batchId = result.Header.BatchId,
l.LineNumber, l.EndToEndId, l.CreditorName, l.CreditorIbanMasked, l.Amount, l.Scheme, l.Reference,
debtorAccountId = result.Header.DebtorAccountId,
currency = result.Header.Currency,
sourceKey = key,
sourceVersionId = versionId,
}, Json))) + "\n";
await PutAsync(_curatedBucket, outputKey, body, "application/x-ndjson");
}
else
{
outputKey = $"payment_batches/{ingestDate}/{fileName}.issues.json";
var report = JsonSerializer.Serialize(new
{
sourceBucket = bucket, sourceKey = key, sourceVersionId = versionId,
batchId = result.Header?.BatchId,
result.DeclaredCount, result.DeclaredSum, result.ActualSum,
issues = result.Issues,
}, Json);
await PutAsync(_quarantineBucket, outputKey, report, "application/json");
}
// One structured summary per file: counts and codes only, never names or IBANs.
context.Logger.LogInformation(JsonSerializer.Serialize(new
{
message = "batch validated",
sourceKey = key,
accepted = result.Accepted,
lines = result.Lines.Count,
declaredCount = result.DeclaredCount,
declaredSum = result.DeclaredSum,
actualSum = result.ActualSum,
issueCodes = result.Issues.Select(i => i.Code).Distinct(),
outputKey,
requestId = context.AwsRequestId,
}, Json));
}
}
private Task PutAsync(string bucket, string key, string body, string contentType) =>
_s3.PutObjectAsync(new PutObjectRequest
{
BucketName = bucket,
Key = key,
ContentBody = body,
ContentType = contentType,
ChecksumAlgorithm = ChecksumAlgorithm.SHA256,
});
}C#: tests (xUnit v3 on Microsoft.Testing.Platform)
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<OutputType>Exe</OutputType>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="xunit.v3" Version="4.0.3" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="../SftpBatch/SftpBatch.csproj" />
<None Include="../../datasets/samples/sftp/*.csv" Link="samples/%(Filename)%(Extension)" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>
</Project>using Harbourline.DataLake.SftpBatch;
namespace SftpBatch.Tests;
public class BatchValidatorTests
{
private static BatchResult Load(string file) =>
BatchValidator.Validate(new StreamReader(Path.Combine(AppContext.BaseDirectory, "samples", file)));
[Fact]
public void Clean_partner_file_is_accepted_and_matches_its_trailer()
{
var result = Load("HLPG_PTN0042_PAYMENTS_20261116_001.csv");
Assert.True(result.Accepted);
Assert.Equal(40, result.Lines.Count);
Assert.Equal(40, result.DeclaredCount);
Assert.Equal(89281.75m, result.ActualSum);
Assert.Equal(new DateOnly(2026, 11, 18), result.Header!.RequestedExecutionDate);
Assert.All(result.Lines, l => Assert.Contains("*", l.CreditorIbanMasked));
}
[Fact]
public void Messy_partner_file_is_rejected_with_every_injected_defect()
{
var result = Load("HLPG_PTN0042_PAYMENTS_20261116_001_messy.csv");
var codes = result.Issues.Select(i => i.Code).ToHashSet();
Assert.False(result.Accepted);
Assert.Contains("INVALID_DATE", codes); // header date 18/11/2026
Assert.Contains("INVALID_IBAN", codes); // line 3
Assert.Contains("INVALID_AMOUNT", codes); // line 4: "1,467.65"
Assert.Contains("DUPLICATE", codes); // line 6 repeats line 5
Assert.Contains("TRAILER_COUNT", codes); // 41 details vs trailer 40
}
[Fact]
public void Truncated_file_without_trailer_is_rejected()
{
var result = BatchValidator.Validate(new StringReader(
"H,HLPG-1,PTN0042,Saltmarsh,2026-11-16T21:30:00Z,ACC-1000000018,GB62HLBK99441212488220,GBP,2026-11-18\n" +
"D,1,E2E1,Test Person,GB09VMHY88522720202824,10.00,BACS,REF\n"));
Assert.Contains(result.Issues, i => i.Code == "TRAILER_MISSING");
}
}The tests reference the committed samples with a relative path; adjust ../../datasets/... to where your project sits in the repository.
Python: the same function
"""Python 3.13 Lambda: validate a Harbourline partner payment batch (header/detail/trailer)."""
import csv
import io
import json
import os
import re
import urllib.parse
from decimal import Decimal
import boto3
s3 = boto3.client("s3") # created once per execution environment
CURATED = os.environ["CURATED_BUCKET"]
QUARANTINE = os.environ["QUARANTINE_BUCKET"]
AMOUNT = re.compile(r"^\d{1,16}\.\d{2}$")
SCHEMES = {"FPS", "BACS", "CHAPS", "SEPA_CT", "SEPA_INST"}
def iban_ok(value: str) -> bool:
s = value.replace(" ", "").upper()
if not (15 <= len(s) <= 34 and s.isalnum()):
return False
digits = "".join(str(int(ch, 36)) for ch in s[4:] + s[:4]) # 'A' -> 10 ... 'Z' -> 35
return int(digits) % 97 == 1
def validate(text: str) -> dict:
issues, lines, seen, header, trailer, total = [], [], set(), None, None, Decimal("0")
rows = list(csv.reader(io.StringIO(text)))
for n, f in enumerate(rows, start=1):
kind = f[0] if f else ""
if kind == "H" and n == 1:
header = f
if not re.match(r"^\d{4}-\d{2}-\d{2}$", f[8]):
issues.append({"line": n, "code": "INVALID_DATE"})
elif kind == "D":
problems = []
if f[2] in seen: problems.append("DUPLICATE")
seen.add(f[2])
if not iban_ok(f[4]): problems.append("INVALID_IBAN")
if not AMOUNT.match(f[5]): problems.append("INVALID_AMOUNT")
if f[6] not in SCHEMES: problems.append("INVALID_CODE")
issues += [{"line": n, "code": c} for c in problems]
if not problems:
total += Decimal(f[5])
lines.append({"lineNumber": int(f[1]), "endToEndId": f[2], "amount": f[5], "scheme": f[6],
"creditorIbanMasked": f[4][:4] + "*" * (len(f[4]) - 8) + f[4][-4:]})
elif kind == "T":
trailer = f
details = sum(1 for f in rows if f and f[0] == "D")
if trailer is None:
issues.append({"line": len(rows), "code": "TRAILER_MISSING"})
else:
if int(trailer[1]) != details: issues.append({"line": len(rows), "code": "TRAILER_COUNT"})
if Decimal(trailer[2]) != total: issues.append({"line": len(rows), "code": "TRAILER_SUM"})
return {"header": header, "lines": lines, "issues": issues, "actual_sum": str(total)}
def handler(event, context):
for record in event["Records"]:
bucket = record["s3"]["bucket"]["name"]
key = urllib.parse.unquote_plus(record["s3"]["object"]["key"])
version = record["s3"]["object"].get("versionId")
args = {"Bucket": bucket, "Key": key} | ({"VersionId": version} if version else {})
text = s3.get_object(**args)["Body"].read().decode("utf-8") # strict UTF-8
result = validate(text)
name = key.rsplit("/", 1)[-1]
if not result["issues"]:
h = result["header"]
out_key = f"payment_batches/partner={h[2]}/execution_date={h[8]}/{h[1]}.jsonl"
body = "".join(json.dumps(line | {"batchId": h[1], "sourceKey": key}) + "\n" for line in result["lines"])
s3.put_object(Bucket=CURATED, Key=out_key, Body=body.encode(), ChecksumAlgorithm="SHA256")
else:
out_key = f"payment_batches/issues/{name}.issues.json"
report = {"sourceKey": key, "sourceVersionId": version, "issues": result["issues"]}
s3.put_object(Bucket=QUARANTINE, Key=out_key, Body=json.dumps(report).encode(), ChecksumAlgorithm="SHA256")
print(json.dumps({"message": "batch validated", "sourceKey": key, "accepted": not result["issues"],
"lines": len(result["lines"]), "outputKey": out_key}))Code explained line by line
Validator
BatchResult.Accepted => Issues.Count == 0— all-or-nothing. For payroll, paying 39 of 40 employees is worse than returning the file; the partner fixes and resends.[GeneratedRegex(...)]— compile-time regexes (.NET 7+): faster cold starts than runtime-compiled regexes, which matters in Lambda.switch (f[0])withcase "H" when fileLine == 1— the header must be the first record. Record-type routing is the core of header/detail/trailer formats used across banking (BACS, many SFTP feeds).trailerSeenandAFTER_TRAILER— anything after the trailer suggests two files concatenated or tampering.TRAILER_MISSING— the most important check for file transfers: a missing trailer usually means a truncated upload.TRAILER_COUNTandTRAILER_SUM— the control totals. The sum compares against valid lines only, so any rejected line also shows up as a sum break, which is what operations expects to see.NumberStyles.Nonefor the trailer count andAllowDecimalPointfor the sum — reject signs, spaces and thousands separators.seen.Add(f[2])— duplicate end-to-end IDs inside one file.Iban.IsValid— ISO 13616 mod-97, processing one digit at a time to avoid big integers (same algorithm asharbourline_etl/iban.pyand the lab's C#Ibanclass).Iban.Mask— curated output never contains the full creditor IBAN. The lab's pipeline adds a keyed token for joins.
Handler
[assembly: LambdaSerializer(...)]— tells the .NET runtime how to deserialise theS3EventJSON.- Two constructors — Lambda uses the parameterless one; tests inject a fake
IAmazonS3. Clients are created once per execution environment and reused across invocations. WebUtility.UrlDecode(record.S3.Object.Key)— S3 event keys are URL-encoded; forgetting this causesNoSuchKeyfor any key with=, spaces or special characters.- The bucket check — a second recursion guard: the function refuses to process its own output buckets even if someone misconfigures a trigger. Lambda's recursive loop detection (on by default) also stops S3–Lambda loops after about 16 invocations in a chain, but you should never rely on it.
VersionId = ...— read exactly the version that triggered the event, not whatever is current by the time the function runs.new UTF8Encoding(false, throwOnInvalidBytes: true)— strict decoding: a mis-encoded partner file fails loudly (and goes to the on-failure destination after retries) instead of silently producing�characters.- Deterministic
outputKey— from the batch ID and execution date. Reprocessing the same file writes the same object: idempotent. A second, different file with the same batch ID overwrites it, which is why the downstream payment step must also check batch IDs it has already submitted. - The quarantine report — contains codes, line numbers and totals, not the rejected content. Raw already holds the original file.
context.Logger.LogInformation(JsonSerializer.Serialize(...))— one JSON log line per file with counts and codes. With the function's log format set to JSON, CloudWatch Logs Insights can query fields directly.ChecksumAlgorithm.SHA256on output — integrity evidence for downstream consumers.
Python version
boto3.client("s3")at module level — created once per execution environment, like the C# field.int(ch, 36)— convertsA–Zto 10–35 in one call; Python's arbitrary-precision integers make the mod-97 check a single expression.urllib.parse.unquote_plus— the Python equivalent ofUrlDecode.csv.readerhandles the quoted"1,467.65"field in the messy file correctly; the amount regex then rejects it.
Security considerations
⚠ Learning shortcut — not for production
The function writes masked IBANs but no keyed token, has no VPC, and the lab uses a local SFTP container (labs/m07-transaction-pipeline/local/docker-compose.yml, password authentication) instead of AWS Transfer Family. In production: Transfer Family with SSH keys per partner and logical directories per partner prefix; a VPC-attached function with an S3 gateway endpoint if network policy requires it; tokenisation with a key from Secrets Manager; code signing for Lambda deployments; and alarms on quarantine counts.
- Least privilege by prefix. Read only
transactions/source=sftp/*; write onlypayment_batches/*. The function cannot read other partners' or other sources' files. - Recursion. Never write to the bucket and prefix that trigger the function. Keep recursive loop detection set to Terminate.
- Untrusted input. Partner files can be malformed or malicious (huge lines, wrong encodings, formula injection such as
=HYPERLINK(...)in names that later opens in Excel). Validate shape strictly; if files are exported to spreadsheets later, neutralise leading=,+,-,@. - Logs. Log counts and codes, not names, IBANs or amounts per person. Set log retention (14 days here).
- Reserved concurrency. Caps the blast radius of a flood of files (or a loop) on both your bill and downstream systems.
Banking use case
Harbourline's Partner Gateway flow for payroll bureaus:
- Partner uploads
HLPG_<partner>_PAYMENTS_<date>_<seq>.csvover SFTP; Transfer Family writes it toraw/transactions/source=sftp/partner=<id>/ingest_date=.../. - This Lambda validates it within a second. Accepted batches become JSON Lines in curated; the Payments Service picks them up (EventBridge rule on the curated prefix) and creates one payment per line with the end-to-end ID as idempotency key.
- Rejected batches produce an issues report; an EventBridge rule on the quarantine prefix notifies the partner-operations team, who contact the bureau. Nothing is paid from a rejected file.
- The nightly reconciliation (Glue/Athena) compares accepted batches, submitted payments and settled payments by batch ID and control sum.
Volumes are small (hundreds of files per day, each a few KB to a few MB), so Lambda costs pennies per month, well inside the free tier.
Common mistakes
- Writing output to the triggering bucket/prefix → infinite loop and a surprise bill → separate buckets or prefixes, plus a code guard.
- Not URL-decoding keys →
NoSuchKeyfor keys with=or spaces → decode first. - Reading the "latest" object instead of the event's version → validating different bytes from those that arrived → use
VersionId. - Loading large files into memory → out-of-memory or timeouts → stream line by line; switch to Glue for large files.
- Partial acceptance of batch files → some payments sent, some not → all-or-nothing for batches with control totals.
- No failure destination → failed async events vanish after retries → on-failure destination (SQS) and an alarm.
- Creating SDK clients inside the handler → slower invocations → create once per execution environment.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Function never runs after upload | Notification filter mismatch (prefix/suffix), or missing lambda:InvokeFunction permission for S3 | Check get-bucket-notification-configuration; add the permission |
NoSuchKey | Key not URL-decoded | WebUtility.UrlDecode / unquote_plus |
AccessDenied on GetObject | Missing s3:GetObjectVersion when using version IDs, or kms:Decrypt | Add both |
Task timed out after 60.00 seconds | File larger than expected, or slow S3 writes | Raise timeout modestly; stream; move big files to Glue |
DecoderFallbackException / UnicodeDecodeError | Partner sent Latin-1 | Expected: event goes to the on-failure destination; agree UTF-8 in the contract or repair with a warning |
RecursiveInvocationsDropped metric above 0 | Output written to the trigger location | Fix the output path; check the trigger filter |
| Throttles during month-end | Reserved concurrency or account concurrency reached | Put SQS between S3 and Lambda; raise reserved concurrency deliberately |
| EventBridge rules for raw stopped firing after deploying the trigger | Bucket notification configuration replaced without EventBridgeConfiguration | Include eventbridge = true (Terraform) or re-enable EventBridge on the bucket |
Cost and cleanup
Expected cost: free. A few invocations at 512 MB for under a second each are well inside the monthly free tier (1 million requests, 400,000 GB-seconds). Even outside it, 100 files cost about USD 0.0003. SQS queue with no messages: free. CloudWatch Logs for a few KB: effectively free. Unit tests run locally at no cost.
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
cdk destroy HlM07SftpBatch # or: terraform destroy
# CLI route:
aws lambda delete-function --function-name hl-m07-sftp-batch-validator
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m07-sftp-batch-dlq --query QueueUrl --output text)"
aws logs delete-log-group --log-group-name /aws/lambda/hl-m07-sftp-batch-validator
aws iam delete-role-policy --role-name hl-m07-sftp-batch-validator-role --policy-name sftp-batch
aws iam delete-role --role-name hl-m07-sftp-batch-validator-role
# Restore the raw bucket's notification configuration to EventBridge only
aws s3api put-bucket-notification-configuration --bucket "hl-m07-raw-$ACCOUNT" \
--notification-configuration '{"EventBridgeConfiguration":{}}'
# Outputs
aws s3 rm "s3://hl-m07-curated-$ACCOUNT/payment_batches/" --recursive
aws s3 rm "s3://hl-m07-quarantine-$ACCOUNT/payment_batches/" --recursive
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--query "ResourceTagMappingList[].ResourceARN"Exercises
- 🟢 Guided: Run the three xUnit tests. Then deploy, upload both sample files, and capture the JSON log line and the output object for each.
- 🟡 Challenge: Add a rule that the header's
debtor_account_idmust belong to a list of approved debtor accounts per partner (store the list in Parameter Store as aStringListand cache it for 5 minutes in the function). ReturnUNAUTHORISED_DEBTORwhen it does not match. - 🔴 Live problem: At 23:58 on the last working day of the month, the bureau uploads a 180 MB file with 1.2 million salary lines for a new large client. The function times out at 60 seconds, retries twice, and the event lands in the DLQ. Payroll must be submitted by 06:00. What do you do tonight, and what do you change for next month?
Assignment
Add the SftpBatch function and its tests to your portfolio repository with a README covering: the record layout, the rules and reason codes, the all-or-nothing decision, the recursion design, the Lambda-vs-Glue threshold you would set, and the exact monthly cost estimate at 300 files per day.
Solutions
Clean file: log line with "accepted":true,"lines":40,"declaredCount":40,"declaredSum":89281.75,"actualSum":89281.75 and an object at payment_batches/partner=PTN0042/execution_date=2026-11-18/HLPG-20261116-001.jsonl with 40 lines and masked IBANs. Messy file: "accepted":false with issue codes INVALID_DATE, INVALID_IBAN, INVALID_AMOUNT, DUPLICATE, TRAILER_COUNT and TRAILER_SUM, and an issues report in quarantine. Why: every defect in the answer key maps to at least one issue code, and the control totals catch anything a line rule missed.
Read /hl-m07/partners/PTN0042/approved-debtors with AWSSDK.SimpleSystemsManagement (GetParameterAsync), split the StringList on commas, and store it with a timestamp in a static field; refresh when older than 5 minutes. Grant ssm:GetParameter on that parameter ARN only. Pass the set into BatchValidator.Validate (keep the validator free of AWS calls so it stays unit-testable) and add UNAUTHORISED_DEBTOR in ParseHeader. Alternatively use the AWS Parameters and Secrets Lambda Extension for caching. Why: reference data changes without redeploying code, and the cache keeps cost and latency down.
Tonight: the file is in raw (unchanged, versioned), so nothing is lost. Run the same validation as a Glue job (or a one-off container task on ECS/Fargate, or Lambda with 10 GB memory and a 15-minute timeout if a quick test shows it fits) on that one object; the validator logic is the same code. Confirm control totals, write the curated output, and let the Payments Service proceed. Inform partner operations and record the incident. Next month: route by size: an EventBridge rule or a small Lambda checks ContentLength and starts Step Functions → Glue for files above, say, 50 MB; agree with the bureau that very large clients get split files with their own trailers; stream-parse instead of loading lists; and alarm on DLQ messages so this is seen at 00:00, not 05:30.
Knowledge check
Answer: 15 minutes. Files whose processing may approach that should go to Glue (or another long-running service), or be split.
Answer: The object may be overwritten between the event and the read; reading the event's version ID validates exactly the bytes that arrived.
Answer: Recursive loop detection stops a chain after about 16 invocations and notifies you. Some cost has already been incurred, and loops through unsupported services are not detected. Design outputs so they cannot trigger the function.
Answer: A total of the detail amounts declared by the sender in the trailer record. Comparing it with the computed total detects truncated, duplicated or altered files before any payment is made.
Answer: 0.5 GB × 0.3 s = 0.15 GB-seconds.
Interview questions
- "Lambda or Glue for ETL?" — Lambda for small, per-file, low-latency transforms and routing; Glue for large, multi-file, joined or long-running work. Measure size and duration and set a threshold; route large files to Glue automatically.
- "How do you make an S3-triggered function idempotent?" — Deterministic output keys from input identity, conditional writes where needed, process the event's version ID, and downstream de-duplication on business keys (batch ID, end-to-end ID).
- "How do you handle failures in asynchronous Lambda invocations?" — Retries (0–2), maximum event age, an on-failure destination (SQS, SNS, EventBridge or another Lambda), alarms on it, and a replay procedure.
- "What do you log from a data function?" — Structured JSON with counts, codes, keys and request IDs; never personal data.
Certification practice
A bank receives about 500 CSV files per day from partners, each under 2 MB. Each file must be validated within one minute of arrival and moved to a curated or rejected location. Which solution has the LOWEST cost and operational overhead?
A. An AWS Glue Spark job triggered for each file by an S3 event. B. An AWS Lambda function invoked by S3 event notifications for the landing prefix. C. An Amazon EMR cluster that polls the bucket every minute. D. An EC2 instance running a scheduled script every minute.
Answer: B. Lambda starts in milliseconds and bills per invocation; files are small. A pays at least the 1-minute minimum for 2 DPUs per file and has start-up latency. C and D run continuously and need management.
A developer's Lambda function is triggered by s3:ObjectCreated:* on a bucket and writes a processed copy of each object to a processed/ prefix in the same bucket. Costs spike shortly after deployment. What is the BEST fix?
A. Increase the function's timeout.
B. Write processed objects to a different bucket, or restrict the trigger with a prefix filter that excludes processed/.
C. Set reserved concurrency to 1000.
D. Disable recursive loop detection.
Answer: B. The function's own output triggers it again. A and C do not address the loop (C makes it worse). D removes a safety net.
At month end, thousands of files land in S3 within minutes and the processing Lambda function is throttled; some events are lost after retries. What change makes the design MORE resilient?
A. Send S3 event notifications to an Amazon SQS queue and configure the queue as the function's event source, with a dead-letter queue. B. Increase the function's memory to 10 GB. C. Use a larger instance type. D. Turn off S3 versioning.
Answer: A. SQS buffers the burst; Lambda polls at a controlled rate and failed messages go to a DLQ for replay. B does not change concurrency limits. C is not applicable to Lambda. D is unrelated and reduces data protection.
Completion checklist
- I can state Lambda's timeout, memory,
/tmpand payload limits and when they push work to Glue - My validator passes the clean, messy and truncated-file tests
- I deployed the function with least-privilege prefix permissions and an on-failure destination
- I proved the function cannot trigger itself and that re-processing is idempotent
- I deleted the function, queue, role and log group and restored the bucket notification
Key takeaways
- Lambda is the right tool for small, per-file, low-latency transforms; Glue for large, joined or long-running work.
- Put rules in a pure class or module, unit-test against the sample files, and keep the handler thin.
- Batch files with trailers are all-or-nothing: check record count and control sum before anything is paid.
- Design out recursion (separate output buckets/prefixes), read the event's version, and write deterministic outputs.
- Lambda at these volumes is effectively free; the real costs are mistakes such as loops and silent failures.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
PySpark in AWS Glue for beginners: DataFrames for C# developers
Learn just enough Python and Apache Spark to read, validate, transform and write Harbourline's transactions in an AWS Glue job, using LINQ as your mental model, and run the same job free on your laptop before paying for Glue.
Amazon Athena: partition pruning, CTAS, workgroups with cost limits and encrypted results
Query Harbourline's lake with Amazon Athena: prune partitions, compare bytes scanned on CSV and Parquet, build gold tables with CTAS, INSERT INTO and UNLOAD, cap spend with workgroup data-usage controls, encrypt query results, and run parameterised queries from C#.