Background processing: worker services, SQS consumers and scheduled jobs
Move slow and scheduled work out of the request path: .NET Worker Services on Fargate, SQS consumers with long polling, visibility timeouts, idempotency and dead-letter queues, Lambda SQS triggers with partial batch responses, and EventBridge Scheduler jobs.
Assumes: .NET 10 LTS · AWSSDK.SQS 4.1.0.2 · AWS.Messaging 1.6.0 · Amazon.Lambda.SQSEvents 3.0.1 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Decide which work belongs in the request, in a queue consumer, or in a scheduled job, and pick the AWS host for each (Fargate worker, Lambda, EventBridge Scheduler).
- Write a .NET Worker Service (
BackgroundService) that consumes Amazon SQS with long polling, processes messages concurrently with a bound, and shuts down gracefully onSIGTERM. - Explain SQS visibility timeout, at-least-once delivery, dead-letter queues and redrive, and make a consumer idempotent.
- Use Lambda with an SQS event source and partial batch responses so one bad message does not reprocess a whole batch.
- Schedule jobs with Amazon EventBridge Scheduler (cron in the
Europe/Londontime zone, retries, DLQ) instead of timers inside web processes. - Know when the AWS Message Processing Framework for .NET (
AWS.Messaging) saves you code.
Prerequisites
- Module 3 overview — hosting options; ECS and Lambda basics
- Scaling .NET apps — why timers in web processes run N times
- Logging and errors — correlation IDs, which must also flow through messages
- Tools: .NET 10 SDK or Docker, AWS CLI v2 with
learn-sandbox. The SQS parts fit in the free tier.
Why this matters in banking
Harbourline customers can request a PDF statement for any month. Generating it takes 3–20 seconds: read transactions, render, encrypt, store in S3, notify the customer. If the Accounts API did this inside the HTTP request:
- Requests would time out behind the load balancer and API Gateway (API Gateway's integration timeout is far shorter than 20 seconds for many setups).
- A spike of requests at month-end would exhaust the API's threads and database connections, slowing balance checks for everyone.
- A crash halfway through would lose the request; the customer would retry and might get two statements, or none.
Banks run on background work: statement generation, interest accrual at midnight, direct-debit files at 04:00, sanctions re-screening, notifications, reconciliation. Every one of these needs to survive restarts, avoid double execution, and leave an audit trail.
The idea in plain language
Background processing means doing work after responding to the user, or on a timetable, in a separate process.
A queue is a to-do list shared between a producer (the API: "please generate this statement") and consumers (workers that take items off the list). If workers are busy or down, items wait safely. If there is a lot of work, you add workers.
At-least-once delivery means the queue guarantees every item is delivered, but occasionally delivers one twice. Your worker must therefore be idempotent: doing the same item twice must have the same effect as doing it once.
A scheduler starts work at fixed times: "every weekday at 06:00 London time".
Analogy: the bank's back office in-tray. Branch staff (the API) drop request forms in the tray and tell the customer "you'll get it in the post". Clerks (workers) take forms out one at a time; while a clerk works on a form it is set aside, not thrown away (visibility timeout). If the clerk goes home sick, the form goes back in the tray. Forms that keep failing go to the supervisor's tray (dead-letter queue). And a wall calendar (the scheduler) says "run the overnight interest batch at 00:30".
What problem SQS and EventBridge Scheduler solve
Amazon SQS is a fully managed message queue. Two types:
| Standard queue | FIFO queue | |
|---|---|---|
| Ordering | Best effort | Strict per message group |
| Delivery | At least once (rare duplicates) | Exactly-once processing within a 5-minute deduplication window |
| Throughput | Nearly unlimited | Lower (high-throughput mode raises it) |
| Use for | Most background work | Per-account ordered processing, e.g. postings for one account |
Key behaviours and limits (verify current values in the SQS quotas page before designing around them):
- Visibility timeout: after a consumer receives a message, it is hidden from others for this long (default 30 seconds, maximum 12 hours). If the consumer does not delete it in time, it reappears.
- Long polling:
WaitTimeSecondsup to 20 seconds; the call returns as soon as messages arrive. It cuts empty responses, which are billed requests. - Batch: up to 10 messages per
ReceiveMessageand per batch send/delete. - Retention: default 4 days, maximum 14 days.
- Message size: the maximum payload is now 1 MiB per request; each 64 KB chunk is billed as one request. Keep messages small and put documents in S3.
- Dead-letter queue (DLQ): after
maxReceiveCountfailed receives, SQS moves the message to a DLQ for inspection. Redrive moves messages back after a fix.
Amazon EventBridge Scheduler is a serverless scheduler: one-off (at(...)), rate (rate(1 hour)) and cron schedules, with time zones (so 06:00 means 06:00 in London in both GMT and BST), flexible time windows, retries with a maximum event age, a DLQ, and targets in more than 270 services (SQS, Lambda, ECS RunTask, Step Functions and others).
Limitations: SQS does not do routing or fan-out (use SNS or EventBridge for that); a standard queue can deliver duplicates and out of order; a message being processed for longer than its visibility timeout is processed twice. Scheduler is at-least-once too, so scheduled jobs also need idempotency.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
BackgroundService / Worker Service | Same .NET code, hosted on ECS Fargate (a service with no load balancer) | Handle SIGTERM and ECS stopTimeout |
| Azure Service Bus queue | Amazon SQS (standard or FIFO) | No sessions/peek-lock API; visibility timeout plays the lock role. FIFO message groups ≈ sessions |
| Service Bus dead-letter sub-queue | SQS dead-letter queue (a separate queue) + redrive | DLQ is configured per source queue with maxReceiveCount |
| Azure Storage queues | Amazon SQS standard | Similar simplicity; SQS has DLQs built in |
| Azure Functions Service Bus / queue trigger | Lambda with an SQS event source mapping | Partial batch responses instead of per-message completion |
WebJobs TimerTrigger, Functions timer trigger | EventBridge Scheduler → Lambda / ECS task / SQS | Schedules live outside your code, with IAM roles, retries and DLQ |
| Hangfire / Quartz.NET inside the web app | EventBridge Scheduler + a worker | No shared database lock tables needed |
MassTransit / NServiceBus | AWS.Messaging (AWS Message Processing Framework for .NET) | Lighter: publish/subscribe, handlers, SQS poller, Lambda support |
Architecture
Message lifecycle:
Security boundary view. The API's role may only sqs:SendMessage to this queue; the worker's task role may only ReceiveMessage, DeleteMessage, ChangeMessageVisibility on it, plus PutObject on one bucket prefix and PutItem on the idempotency table. The queue is encrypted (SQS-managed SSE by default, or SSE-KMS with your key), and messages carry identifiers, never personal data or documents.
Idempotency: the rule that keeps money correct
Because every queue and scheduler here is at-least-once, design every handler so that a duplicate is harmless:
- A stable key per unit of work. For a statement:
{accountId}:{month}. For a payment: the client'sIdempotency-Key. - Record "done" atomically with a condition. A DynamoDB
PutItemwithConditionExpression = attribute_not_exists(pk)succeeds only the first time. A second attempt getsConditionalCheckFailedException, which you treat as "already done → delete the message". - Make side effects naturally idempotent where you can. Writing
s3://.../ACC-0000000001/2026-10.pdftwice produces the same object. Sending an email twice does not, so record "notified" before or with sending. - Expire idempotency records with DynamoDB TTL after the window in which duplicates can occur (days, not forever).
Graceful shutdown on ECS
When ECS stops a task (deployment, scale-in, Spot interruption), it sends SIGTERM, waits stopTimeout (default 30 seconds, configurable up to 120 seconds on Fargate), then sends SIGKILL. .NET's generic host translates SIGTERM into cancelling the stoppingToken of every BackgroundService and waits up to HostOptions.ShutdownTimeout (30 seconds by default).
Good worker behaviour on SIGTERM:
- Stop receiving new messages immediately.
- Finish in-flight messages if they can complete within the timeout; otherwise leave them undeleted so they reappear for another worker.
- Never delete a message whose work did not complete.
Set HostOptions.ShutdownTimeout a few seconds below the container stopTimeout so .NET finishes cleanly before the kill.
Step-by-step implementation
Create the queue and its DLQ
A DLQ first, then the main queue with a redrive policy (maxReceiveCount 5) and a visibility timeout longer than the slowest normal processing time (the code below uses 120 seconds plus heartbeats).
Make the API enqueue instead of doing the work
The API validates the request, sends a small message (accountId, month, requestedBy, correlationId), and returns 202 Accepted with a status URL.
Write the worker
A BackgroundService with long polling, a concurrency limit, per-message correlation scopes, idempotency, heartbeats for slow work, and delete-on-success only.
Run it on Fargate as a service without a load balancer
Desired count 1, auto scaling on the queue's ApproximateNumberOfMessagesVisible (or backlog per task), stopTimeout 60 seconds.
Add the schedule
EventBridge Scheduler sends a "generate all statements for last month" trigger message on the 1st at 06:00 London time, with retries and a DLQ.
Alarm on the DLQ
Any message in the DLQ is a failure a human must look at. Alarm on ApproximateNumberOfMessagesVisible > 0 for the DLQ, and on ApproximateAgeOfOldestMessage for the main queue.
AWS Console instructions
- Amazon SQS → Create queue → Standard, name
hl-m03-statement-requests-dlq, retention 14 days. Create. - Create queue → Standard, name
hl-m03-statement-requests; Visibility timeout 2 minutes; Receive message wait time 20 seconds; Encryption SSE-SQS (or SSE-KMS); Dead-letter queue enabled → the DLQ ARN, Maximum receives 5; tagsProject=cea-lab,Lab=m03-background. - Queue → Send and receive messages lets you send a test message and poll.
- DLQ → Start DLQ redrive moves messages back to the source queue after you fix a bug.
- Amazon EventBridge → Scheduler → Schedules → Create schedule: recurring, cron
0 6 1 * ? *, time zoneEurope/London, flexible window off; target Amazon SQS SendMessage → the queue; retry policy and DLQ under Settings.
AWS CLI
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
TAGS='Project=cea-lab,Lab=m03-background,Environment=learn,Owner=<your-name>'
# 1. DLQ, then the main queue with redrive to it
DLQ_URL=$(aws sqs create-queue --queue-name hl-m03-statement-requests-dlq \
--attributes MessageRetentionPeriod=1209600,SqsManagedSseEnabled=true \
--tags "$TAGS" --query QueueUrl --output text)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" --attribute-names QueueArn --query Attributes.QueueArn --output text)
cat > /tmp/queue-attrs.json <<EOF
{
"VisibilityTimeout": "120",
"ReceiveMessageWaitTimeSeconds": "20",
"SqsManagedSseEnabled": "true",
"RedrivePolicy": "{\"deadLetterTargetArn\":\"$DLQ_ARN\",\"maxReceiveCount\":\"5\"}"
}
EOF
QUEUE_URL=$(aws sqs create-queue --queue-name hl-m03-statement-requests \
--attributes file:///tmp/queue-attrs.json --tags "$TAGS" --query QueueUrl --output text)
# 2. Send a test request (no personal data: identifiers only)
aws sqs send-message --queue-url "$QUEUE_URL" \
--message-body '{"accountId":"ACC-0000000001","month":"2026-10","requestedBy":"cref_3f9a1c0d2b4e6f70"}' \
--message-attributes 'CorrelationId={DataType=String,StringValue=cli-test-0001}'
# 3. Inspect backlog and the DLQ
aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
--attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible
aws sqs get-queue-attributes --queue-url "$DLQ_URL" --attribute-names ApproximateNumberOfMessages
# 4. After fixing a bug: move DLQ messages back to the source queue
aws sqs start-message-move-task --source-arn "$DLQ_ARN"
# 5. EventBridge Scheduler: monthly trigger at 06:00 London time on the 1st
# (needs a role that scheduler.amazonaws.com can assume, allowed sqs:SendMessage on the queue)
QUEUE_ARN=$(aws sqs get-queue-attributes --queue-url "$QUEUE_URL" --attribute-names QueueArn --query Attributes.QueueArn --output text)
aws scheduler create-schedule --name hl-m03-monthly-statements \
--schedule-expression "cron(0 6 1 * ? *)" \
--schedule-expression-timezone "Europe/London" \
--flexible-time-window Mode=OFF \
--target "{\"Arn\":\"$QUEUE_ARN\",\"RoleArn\":\"arn:aws:iam::<account-id>:role/hl-m03-scheduler-sqs\",\"Input\":\"{\\\"batch\\\":\\\"monthly-statements\\\"}\",\"RetryPolicy\":{\"MaximumRetryAttempts\":3,\"MaximumEventAgeInSeconds\":3600},\"DeadLetterConfig\":{\"Arn\":\"$DLQ_ARN\"}}"Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.ApplicationAutoScaling;
using Amazon.CDK.AWS.ECS;
using Amazon.CDK.AWS.ECS.Patterns;
using Amazon.CDK.AWS.SQS;
var dlq = new Queue(this, "Dlq", new QueueProps
{
QueueName = "hl-m03-statement-requests-dlq",
RetentionPeriod = Duration.Days(14),
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
});
var queue = new Queue(this, "Requests", new QueueProps
{
QueueName = "hl-m03-statement-requests",
VisibilityTimeout = Duration.Minutes(2),
ReceiveMessageWaitTime = Duration.Seconds(20),
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 },
});
// Fargate service with no load balancer; scales on backlog. The pattern grants consume permissions.
var worker = new QueueProcessingFargateService(this, "Worker", new QueueProcessingFargateServiceProps
{
Cluster = cluster,
Queue = queue,
Image = ContainerImage.FromEcrRepository(repo, "worker-v1"),
Cpu = 256,
MemoryLimitMiB = 512,
MinScalingCapacity = 0, // scale to zero when the queue is empty
MaxScalingCapacity = 4,
ScalingSteps =
[
new ScalingInterval { Upper = 0, Change = -1 },
new ScalingInterval { Lower = 1, Change = +1 },
new ScalingInterval { Lower = 100, Change = +2 },
],
Environment = new Dictionary<string, string> { ["Worker__MaxConcurrency"] = "4" },
});
// Give in-flight messages time to finish on SIGTERM (Fargate allows up to 120 s).
((CfnTaskDefinition)worker.TaskDefinition.Node.DefaultChild!).AddPropertyOverride(
"ContainerDefinitions.0.StopTimeout", 60);Complete code
A Worker Service that consumes statement requests. It runs locally against real SQS with your SSO profile, or entirely offline in its unit test (the queue client is mocked).
<Project Sdk="Microsoft.NET.Sdk.Worker">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<RootNamespace>Harbourline.Statements.Worker</RootNamespace>
<InternalsVisibleTo>Harbourline.Statements.Worker.Tests</InternalsVisibleTo>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="4.1.0.2" />
<PackageReference Include="AWSSDK.SQS" Version="4.1.0.2" />
<PackageReference Include="AWSSDK.DynamoDBv2" Version="4.1.0.2" />
</ItemGroup>
</Project>using Amazon.DynamoDBv2;
using Amazon.SQS;
using Harbourline.Statements.Worker;
var builder = Host.CreateApplicationBuilder(args);
builder.Logging.ClearProviders();
builder.Logging.AddJsonConsole(o => o.IncludeScopes = true);
// Finish in-flight work before ECS sends SIGKILL (container stopTimeout is 60 s).
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(50));
builder.Services.Configure<WorkerOptions>(builder.Configuration.GetSection("Worker"));
builder.Services.AddDefaultAWSOptions(builder.Configuration.GetAWSOptions());
builder.Services.AddAWSService<IAmazonSQS>();
builder.Services.AddAWSService<IAmazonDynamoDB>();
builder.Services.AddSingleton<IIdempotencyStore, DynamoDbIdempotencyStore>();
builder.Services.AddSingleton<IStatementGenerator, FakeStatementGenerator>();
builder.Services.AddHostedService<StatementRequestConsumer>();
await builder.Build().RunAsync();using System.Text.Json;
using Amazon.SQS;
using Amazon.SQS.Model;
using Microsoft.Extensions.Options;
namespace Harbourline.Statements.Worker;
public sealed class WorkerOptions
{
public string QueueUrl { get; set; } = "";
public int MaxConcurrency { get; set; } = 4; // messages processed in parallel per task
public int VisibilityTimeoutSeconds { get; set; } = 120;
}
public sealed record StatementRequest(string AccountId, string Month, string RequestedBy);
public sealed class StatementRequestConsumer(
IAmazonSQS sqs,
IIdempotencyStore idempotency,
IStatementGenerator generator,
IOptions<WorkerOptions> options,
ILogger<StatementRequestConsumer> log) : BackgroundService
{
private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
private readonly WorkerOptions _o = options.Value;
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
using var slots = new SemaphoreSlim(_o.MaxConcurrency);
var inFlight = new List<Task>();
log.LogInformation("Consuming {QueueUrl} with concurrency {MaxConcurrency}", _o.QueueUrl, _o.MaxConcurrency);
while (!stoppingToken.IsCancellationRequested)
{
ReceiveMessageResponse response;
try
{
response = await sqs.ReceiveMessageAsync(new ReceiveMessageRequest
{
QueueUrl = _o.QueueUrl,
MaxNumberOfMessages = Math.Min(10, _o.MaxConcurrency),
WaitTimeSeconds = 20, // long polling
MessageAttributeNames = ["CorrelationId"],
MessageSystemAttributeNames = ["ApproximateReceiveCount"],
}, stoppingToken);
}
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
{
break; // SIGTERM while waiting: stop receiving
}
catch (AmazonSQSException ex)
{
log.LogWarning(ex, "ReceiveMessage failed; backing off");
await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
continue;
}
foreach (var message in response.Messages ?? []) // SDK v4: null when empty
{
await slots.WaitAsync(stoppingToken);
inFlight.Add(Task.Run(async () =>
{
try { await HandleAsync(message, stoppingToken); }
finally { slots.Release(); }
}, CancellationToken.None));
}
inFlight.RemoveAll(t => t.IsCompleted);
}
// Graceful shutdown: let started messages finish (or time out) before the host stops.
await Task.WhenAll(inFlight);
log.LogInformation("Consumer stopped");
}
internal async Task HandleAsync(Message message, CancellationToken stoppingToken)
{
var correlationId = message.MessageAttributes?.GetValueOrDefault("CorrelationId")?.StringValue ?? message.MessageId;
using var scope = log.BeginScope(new Dictionary<string, object> { ["CorrelationId"] = correlationId, ["MessageId"] = message.MessageId });
StatementRequest? request;
try
{
request = JsonSerializer.Deserialize<StatementRequest>(message.Body, Json);
}
catch (JsonException)
{
request = null;
}
if (request is null || !request.AccountId.StartsWith("ACC-", StringComparison.Ordinal))
{
// A poison message will never succeed: leave it for the DLQ, don't spin on it.
log.LogError("Malformed statement request; leaving it to reach the DLQ");
return;
}
var key = $"statement:{request.AccountId}:{request.Month}";
if (await idempotency.WasCompletedAsync(key, stoppingToken))
{
log.LogInformation("Duplicate delivery of {Key}; deleting", key);
await DeleteAsync(message);
return;
}
// Heartbeat: extend visibility while slow work runs, so no other worker picks it up.
using var work = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken);
var heartbeat = HeartbeatAsync(message, work.Token);
try
{
await generator.GenerateAsync(request, work.Token);
await idempotency.MarkCompletedAsync(key, stoppingToken);
await DeleteAsync(message);
log.LogInformation("Statement {Month} generated for {AccountId}", request.Month, request.AccountId);
}
catch (Exception ex) when (ex is not OperationCanceledException || !stoppingToken.IsCancellationRequested)
{
// Not deleted → visible again after the timeout → retried → DLQ after maxReceiveCount.
log.LogError(ex, "Statement generation failed for {AccountId}; it will be retried", request.AccountId);
}
finally
{
await work.CancelAsync();
await heartbeat.ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing);
}
}
private async Task HeartbeatAsync(Message message, CancellationToken ct)
{
var every = TimeSpan.FromSeconds(_o.VisibilityTimeoutSeconds / 2.0);
while (!ct.IsCancellationRequested)
{
await Task.Delay(every, ct);
await sqs.ChangeMessageVisibilityAsync(new ChangeMessageVisibilityRequest
{
QueueUrl = _o.QueueUrl,
ReceiptHandle = message.ReceiptHandle,
VisibilityTimeout = _o.VisibilityTimeoutSeconds,
}, ct);
}
}
// CancellationToken.None: once the work is done, the delete must happen even during shutdown.
private Task DeleteAsync(Message message) =>
sqs.DeleteMessageAsync(new DeleteMessageRequest { QueueUrl = _o.QueueUrl, ReceiptHandle = message.ReceiptHandle }, CancellationToken.None);
}using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
namespace Harbourline.Statements.Worker;
public interface IIdempotencyStore
{
Task<bool> WasCompletedAsync(string key, CancellationToken ct);
Task MarkCompletedAsync(string key, CancellationToken ct);
}
/// <summary>Table hl-m03-idempotency: partition key "pk" (string), TTL attribute "expires_at".</summary>
public sealed class DynamoDbIdempotencyStore(IAmazonDynamoDB dynamo) : IIdempotencyStore
{
private const string Table = "hl-m03-idempotency";
public async Task<bool> WasCompletedAsync(string key, CancellationToken ct)
{
var item = await dynamo.GetItemAsync(new GetItemRequest
{
TableName = Table,
Key = new() { ["pk"] = new AttributeValue { S = key } },
ConsistentRead = true,
}, ct);
return item.Item is { Count: > 0 };
}
public async Task MarkCompletedAsync(string key, CancellationToken ct)
{
try
{
await dynamo.PutItemAsync(new PutItemRequest
{
TableName = Table,
Item = new()
{
["pk"] = new AttributeValue { S = key },
["completed_at"] = new AttributeValue { S = DateTimeOffset.UtcNow.ToString("O") },
["expires_at"] = new AttributeValue { N = DateTimeOffset.UtcNow.AddDays(35).ToUnixTimeSeconds().ToString() },
},
ConditionExpression = "attribute_not_exists(pk)", // first writer wins
}, ct);
}
catch (ConditionalCheckFailedException)
{
// Another worker finished the same request concurrently: that's fine.
}
}
}
public interface IStatementGenerator
{
Task GenerateAsync(StatementRequest request, CancellationToken ct);
}
/// <summary>Stand-in for render + encrypt + S3 PutObject, so the sample runs without a renderer.</summary>
public sealed class FakeStatementGenerator : IStatementGenerator
{
public Task GenerateAsync(StatementRequest request, CancellationToken ct) => Task.Delay(TimeSpan.FromSeconds(2), ct);
}{
"AWS": { "Region": "us-east-1", "Profile": "learn-sandbox" },
"Worker": {
"QueueUrl": "https://sqs.us-east-1.amazonaws.com/<account-id>/hl-m03-statement-requests",
"MaxConcurrency": 4,
"VisibilityTimeoutSeconds": 120
}
}Remove "Profile" for deployed environments: on Fargate the task role is used automatically.
A unit test with xUnit v3 and NSubstitute that proves a duplicate delivery is deleted without regenerating:
using Amazon.SQS;
using Amazon.SQS.Model;
using Harbourline.Statements.Worker;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;
public class StatementRequestConsumerTests
{
[Fact]
public async Task Duplicate_delivery_is_deleted_without_generating_again()
{
var sqs = Substitute.For<IAmazonSQS>();
var store = Substitute.For<IIdempotencyStore>();
store.WasCompletedAsync("statement:ACC-0000000001:2026-10", Arg.Any<CancellationToken>()).Returns(true);
var generator = Substitute.For<IStatementGenerator>();
var consumer = new StatementRequestConsumer(sqs, store, generator,
Options.Create(new WorkerOptions { QueueUrl = "https://sqs.us-east-1.amazonaws.com/111122223333/q" }),
NullLogger<StatementRequestConsumer>.Instance);
await consumer.HandleAsync(new Message
{
MessageId = "m-1",
ReceiptHandle = "rh-1",
Body = """{"accountId":"ACC-0000000001","month":"2026-10","requestedBy":"cref_3f9a1c0d2b4e6f70"}""",
}, TestContext.Current.CancellationToken);
await generator.DidNotReceive().GenerateAsync(Arg.Any<StatementRequest>(), Arg.Any<CancellationToken>());
await sqs.Received(1).DeleteMessageAsync(Arg.Is<DeleteMessageRequest>(r => r.ReceiptHandle == "rh-1"), Arg.Any<CancellationToken>());
}
}The test project follows the lab pattern: xunit.v3 4.0.3 and NSubstitute 6.2.0, <OutputType>Exe</OutputType>, Microsoft.Testing.Platform enabled in global.json.
Code explained line by line
Microsoft.NET.Sdk.Worker— the Worker Service SDK: generic host, no web server. On ECS it is a service with no load balancer and no port.ShutdownTimeout = 50 s— the host waits this long forExecuteAsyncto finish afterSIGTERM. It is below the container's 60-secondstopTimeout, so .NET exits before ECS kills it.AddAWSService<IAmazonSQS>()— one thread-safe SQS client for the process.SemaphoreSlim(MaxConcurrency)— bounds parallel work per task. Total concurrency = tasks ×MaxConcurrency; keep that within what downstream systems tolerate.ReceiveMessageAsyncwithWaitTimeSeconds = 20— long polling: fewer empty, billed receives, faster pickup.MaxNumberOfMessages = Math.Min(10, MaxConcurrency)— SQS returns at most 10 per call; never receive more than you can start, or messages sit invisible while waiting.MessageAttributeNames = ["CorrelationId"]— the API puts the request's correlation ID on the message so logs connect the HTTP request and the background work.catch (OperationCanceledException) when stoppingToken...— onSIGTERMthe long poll is cancelled; leave the loop and stop receiving.response.Messages ?? []— in SDK v4, collection properties arenullwhen empty (v3 returned empty lists). A classic v3→v4 migration bug.Task.Run(..., CancellationToken.None)withfinally slots.Release()— starts each message independently and always frees its slot.log.BeginScope(...)— every log line for the message carriesCorrelationIdandMessageId.- Malformed message →
returnwithout delete — it will be received again up tomaxReceiveCount, then moved to the DLQ where someone can inspect it. (A refinement: send it to the DLQ immediately yourself.) WasCompletedAsync→ delete duplicate — the idempotency check that makes at-least-once delivery safe.HeartbeatAsync— every half visibility timeout,ChangeMessageVisibilityextends the lease. Without it, work longer than 120 seconds would be delivered to a second worker.MarkCompletedAsyncthenDeleteAsync— the order matters: record completion first, then delete. If the process dies in between, the redelivered message is recognised as a duplicate.catch ... when (...)— on failure, do not delete; SQS's redelivery is the retry mechanism. Shutdown cancellations are excluded so they are not logged as errors.DeleteAsync(..., CancellationToken.None)— once work is done, the delete must not be cancelled by shutdown.ConditionExpression = "attribute_not_exists(pk)"— DynamoDB's atomic "insert if absent";ConditionalCheckFailedExceptionmeans someone else already completed it.expires_atwith TTL — idempotency records expire after 35 days, longer than queue retention (14 days max) plus margin.
The same consumer with less code: AWS.Messaging and Lambda
AWS Message Processing Framework for .NET (AWS.Messaging 1.6.0) wraps polling, deserialisation, visibility extension and deletion. You write handlers:
builder.Services.AddAWSMessageBus(bus =>
{
bus.AddSQSPoller(builder.Configuration["Worker:QueueUrl"]!);
bus.AddMessageHandler<StatementRequestHandler, StatementRequest>();
});
public sealed class StatementRequestHandler(IStatementGenerator generator) : IMessageHandler<StatementRequest>
{
public async Task<MessageProcessStatus> HandleAsync(MessageEnvelope<StatementRequest> envelope, CancellationToken token = default)
{
await generator.GenerateAsync(envelope.Message, token);
return MessageProcessStatus.Success(); // the framework deletes the message
}
}The framework uses its own message envelope (CloudEvents-style) for messages it publishes, so producers should publish with IMessagePublisher from the same framework. Idempotency is still your job.
Lambda with an SQS event source mapping removes the worker host entirely. Lambda polls the queue, invokes your function with a batch, and deletes messages on success. With ReportBatchItemFailures enabled, you return only the failed message IDs, and only those return to the queue:
using Amazon.Lambda.Core;
using Amazon.Lambda.SQSEvents;
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]
namespace Harbourline.Statements.Lambda;
public sealed class Function
{
public async Task<SQSBatchResponse> HandleAsync(SQSEvent sqsEvent, ILambdaContext context)
{
var failures = new List<SQSBatchResponse.BatchItemFailure>();
foreach (var record in sqsEvent.Records)
{
try
{
await ProcessAsync(record.Body, context);
}
catch (Exception ex)
{
context.Logger.LogError($"Message {record.MessageId} failed: {ex.GetType().Name}");
failures.Add(new SQSBatchResponse.BatchItemFailure { ItemIdentifier = record.MessageId });
}
}
return new SQSBatchResponse(failures); // only these return to the queue
}
private static Task ProcessAsync(string body, ILambdaContext context) => Task.CompletedTask; // idempotent work here
}Rules for Lambda + SQS: set the queue's visibility timeout to at least six times the function timeout (AWS's guidance, so retries and throttling don't cause duplicate processing), cap concurrency with the event source mapping's maximum concurrency setting, and keep the DLQ on the queue (not the function) so failures land in one place.
| Fargate worker | Lambda SQS trigger | |
|---|---|---|
| Long jobs (minutes) | Yes, unlimited | Up to 15 minutes per invocation |
| Idle cost | Task-hours unless scaled to zero | None |
| Concurrency control | Tasks × MaxConcurrency | Event source mapping maximum concurrency |
| Heavy dependencies (PDF renderers, fonts) | Easy in a container | Possible with container images |
| Code you maintain | Polling loop (or AWS.Messaging) | Handler only |
Security considerations
⚠ Learning shortcut — not for production
The lab queues use SQS-managed encryption and the sample worker uses your SSO profile locally. In production use SSE-KMS with a customer managed key where your data classification requires it, a queue policy that denies non-TLS access (aws:SecureTransport) and restricts senders to the API's role, a dedicated task role for the worker with only receive/delete/change-visibility on this queue, and no Profile in deployed configuration.
- Messages are not a document store. Put identifiers in messages, documents in S3 (SSE-KMS, Block Public Access), and personal data nowhere it isn't needed. Messages appear in DLQs, in logs when something fails, and in redrive tools.
- Validate every message. A queue is an input boundary like an HTTP endpoint. Check formats and authorisation context (
requestedBy) before acting. - Least privilege for schedules. The Scheduler role may only
sqs:SendMessageon one queue (orlambda:InvokeFunctionon one function). - Audit trail. Log start, outcome and duration of each job with the correlation ID; operations and auditors ask "did the 1 October statement run complete for everyone?".
Banking use case
Harbourline's statement pipeline:
- The Accounts API accepts
POST /v1/statementswithIdempotency-Key, writes a "requested" record, enqueues{accountId, month, correlationId}and returns 202 withLocation: /v1/statements/{id}. - A Fargate worker service (0–4 tasks, scaling on backlog) renders PDFs, stores them encrypted in S3 under
statements/{accountId}/{yyyy-MM}.pdf, and records completion in DynamoDB. - On the 1st of each month at 06:00 London time, EventBridge Scheduler sends a trigger message; a small "fan-out" job enqueues one message per active account in batches of 10.
- Failures go to the DLQ; an alarm pages the on-call team; after a fix, a DLQ redrive reprocesses them; idempotency guarantees no customer gets two statements.
- Interest accrual, which must run exactly once per day per account, uses a FIFO queue with the account ID as message group and the business date in the deduplication ID, plus the same idempotency table.
Common mistakes
- Timers in the web API → the job runs once per task → EventBridge Scheduler or a single worker.
- Visibility timeout shorter than processing time → duplicates under load → longer timeout plus heartbeats; for Lambda, six times the function timeout.
- Deleting the message before the work is done → work lost on crash → delete last.
- No DLQ → poison messages loop until retention expires, costing money and hiding failures → DLQ with alarm.
- No idempotency → duplicates create double statements or double postings → conditional writes keyed by business identifiers.
- Short polling (
WaitTimeSeconds = 0) → thousands of empty billed receives → long polling. - Ignoring
SIGTERM→ half-done work killed at every deployment → honourstoppingToken, setstopTimeoutandShutdownTimeout. - SDK v4 null collections →
NullReferenceExceptiononresponse.Messageswhen the queue is empty →?? [].
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Same message processed twice | Processing longer than visibility timeout; or worker crashed after work, before delete | Heartbeats; idempotency |
Messages pile up, ApproximateAgeOfOldestMessage rises | Too few workers, or a stuck worker holding messages | Scale on backlog; check for exceptions; per-message timeouts |
| Messages go straight to the DLQ | Poison message, or maxReceiveCount too low with transient errors | Inspect the DLQ message; raise maxReceiveCount to 3–5 |
AccessDenied on ReceiveMessage | Task role lacks permission, or queue policy denies | Grant on the queue ARN; check KMS key policy for SSE-KMS |
KMS.AccessDeniedException sending/receiving | SSE-KMS key policy doesn't allow the role | Allow kms:GenerateDataKey (send) and kms:Decrypt (receive) |
| Worker killed at deploy, logs end abruptly | stopTimeout too short | Raise to 60–120 s; lower ShutdownTimeout below it |
| Lambda reprocesses whole batches | ReportBatchItemFailures not enabled, or function throws | Enable it; return SQSBatchResponse instead of throwing |
| Schedule fires an hour off in summer | Cron in UTC | Set schedule-expression-timezone Europe/London |
NullReferenceException on empty receive | SDK v4 returns null collections | Null-coalesce |
Cost and cleanup
- SQS: the first 1 million requests per month are free; beyond that standard queues cost cents per million requests ⚠ VERIFY current per-million price. Each 64 KB chunk of a payload is a request. Long polling keeps idle cost near zero.
- EventBridge Scheduler: a monthly free tier of invocations (14 million) and about USD 1.00 per million after ⚠ VERIFY.
- DynamoDB idempotency table on-demand: fractions of a cent for lab traffic.
- Fargate worker: billed per task-hour when running; scale to zero when the queue is empty, and destroy after the lab.
aws scheduler delete-schedule --name hl-m03-monthly-statements --profile learn-sandbox
aws sqs delete-queue --queue-url "$QUEUE_URL" --profile learn-sandbox
aws sqs delete-queue --queue-url "$DLQ_URL" --profile learn-sandbox
aws dynamodb delete-table --table-name hl-m03-idempotency --profile learn-sandbox # if you created it
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m03-background \
--query "ResourceTagMappingList[].ResourceARN" --profile learn-sandboxExercises
- 🟢 Guided: Create the two queues with the CLI, run the worker locally with your SSO profile, send three messages (one malformed), and observe: two processed and deleted, the malformed one received five times and then in the DLQ. Redrive it and watch it fail again.
- 🟡 Challenge: Replace the hand-written polling loop with
AWS.Messaging, keeping idempotency and correlation IDs. Compare lines of code and behaviour onSIGTERM(docker stopsends it). - 🔴 Live problem: "Some customers received two copies of their October statement email. Worker logs show both copies were processed by different tasks about 2.5 minutes apart. Visibility timeout is 120 s. Statement generation for business accounts takes up to 4 minutes." Find the cause, explain why idempotency didn't prevent it, and fix it.
Assignment
Add a docs/background-jobs.md to your portfolio: a table of every background job for a bank feature of your choice (trigger, host, schedule/time zone, idempotency key, retry policy, DLQ, alarm, owner), plus the sequence diagram of one job end to end.
Solutions
The valid messages log Statement 2026-10 generated and disappear from the queue (ApproximateNumberOfMessages 0). The malformed one is received, logged as malformed, not deleted, reappears after 120 s, and after the fifth receive SQS moves it to the DLQ (ApproximateNumberOfMessages 1 on the DLQ). It takes about 10 minutes because of the visibility timeout; lower it for the exercise if you are impatient. Redrive with start-message-move-task returns it to the source queue with a fresh receive count, and it fails again, which is the point: redrive only after fixing the cause.
Register AddAWSMessageBus with AddSQSPoller(queueUrl) and a handler implementing IMessageHandler<StatementRequest>; inside, run the idempotency check, generate, mark complete and return MessageProcessStatus.Success(); return MessageProcessStatus.Failed() to leave the message for retry. Use envelope.Id or a message attribute for correlation. The framework handles visibility extension and deletion, so the loop, semaphore and heartbeat disappear (roughly 80 lines fewer). On docker stop the generic host cancels the poller; in-flight handlers receive the cancellation token. Producers must publish with the framework's IMessagePublisher (or produce its envelope format), otherwise deserialisation fails, so migrate producer and consumer together.
Business statements take up to 4 minutes, longer than the 120-second visibility timeout. The heartbeat was probably missing or failing (check for ChangeMessageVisibility errors such as AccessDenied, because the task role must allow sqs:ChangeMessageVisibility). At 120 s the message became visible, a second task received it at about 2.5 minutes and, since the first task had not finished, the idempotency record did not exist yet: both passed the WasCompleted check. The conditional write only prevents a duplicate record, not duplicate side effects that happen before it. Fixes: grant and verify the heartbeat permission; claim the work at the start with a conditional "in progress" record (attribute_not_exists(pk), with an expiry so crashed claims can be retaken); send the email only after a successful conditional "notified" write; consider a longer visibility timeout for this queue.
Knowledge check
Answer: After the visibility timeout it becomes visible again and its receive count increases. After maxReceiveCount receives (if a redrive policy exists) it moves to the dead-letter queue; otherwise it keeps cycling until the retention period expires.
Answer: Standard queues deliver at least once, so duplicates can occur regardless of timeouts, and a consumer can crash after doing the work but before deleting. Idempotency makes duplicates harmless.
Answer: The function returns the IDs of failed messages; Lambda deletes the successful ones and returns only the failed ones to the queue. Without it, any error makes the whole batch visible again, reprocessing successful messages.
Answer: Stop receiving new messages, let in-flight work finish within the shutdown timeout, never delete unfinished messages, and exit before ECS's stopTimeout expires (set HostOptions.ShutdownTimeout slightly lower).
Answer: The UK switches between GMT and BST. A UTC cron would run at 06:00 in winter and 07:00 in summer. Scheduler evaluates the expression in Europe/London, so the job runs at 06:00 local time all year.
Interview questions
- "How do you process messages from SQS reliably in .NET?" — Long polling, bounded concurrency, idempotent handlers keyed by business IDs, delete only after success, visibility heartbeats for long work, DLQ with alarm, graceful
SIGTERMhandling; or Lambda with partial batch responses. - "Fargate worker or Lambda for queue processing?" — Lambda for short, spiky, simple handlers with no idle cost; Fargate for long jobs, heavy dependencies, steady throughput or tight control of concurrency and connections.
- "Where do scheduled jobs live?" — Outside the web app: EventBridge Scheduler targeting Lambda, an ECS task, Step Functions or a queue, with retries, DLQ and IAM roles, in the business time zone.
- "How do you avoid paying a customer twice when messages are duplicated?" — Idempotency keys and conditional writes; FIFO with deduplication IDs where ordering per account matters; reconciliation as a final control.
Certification practice
A Lambda function processes messages from an Amazon SQS standard queue. The function timeout is 60 seconds, and the queue's visibility timeout is 30 seconds. Some messages are processed twice. What should the developer do FIRST?
A. Switch to a FIFO queue. B. Increase the queue's visibility timeout to at least six times the function timeout. C. Decrease the function's batch size to 1. D. Enable long polling on the queue.
Answer: B. A visibility timeout shorter than processing lets the message reappear while still being processed. AWS recommends at least six times the function timeout. A changes semantics and throughput without fixing the cause. C reduces batch effects but not the timeout problem. D reduces empty receives only.
A Lambda function consumes batches of 10 messages from SQS. When one message fails, all 10 are processed again. How can the developer process only failed messages again?
A. Set the batch size to 1 and increase concurrency. B. Enable ReportBatchItemFailures on the event source mapping and return the failed message IDs. C. Catch all exceptions and return successfully. D. Delete each message manually at the start of processing.
Answer: B. Partial batch responses return only failed items. A works but loses batching efficiency and raises cost. C loses failed messages. D deletes before processing, so failures are lost.
An ASP.NET Core API generates PDF reports synchronously, causing timeouts and high latency during peak periods. Reports can be delivered within 10 minutes. Which design is MOST resilient?
A. Increase the load balancer idle timeout to 10 minutes. B. Enqueue report requests in Amazon SQS and process them with an auto scaled worker service, storing results in Amazon S3. C. Run report generation in a background thread inside the API process. D. Move the API to larger instances.
Answer: B. Queue-based decoupling absorbs spikes, survives restarts and scales workers independently. A keeps connections open and fails on restarts. C loses work when tasks stop and competes with requests. D only postpones the problem.
A company needs to start a containerised batch job every weekday at 05:00 UK local time, including during British Summer Time, with retries if the start fails. Which solution needs the LEAST custom code?
A. A cron daemon on an EC2 instance.
B. An Amazon EventBridge Scheduler schedule with the Europe/London time zone targeting ECS RunTask, with a retry policy and DLQ.
C. A System.Threading.Timer in the web application.
D. A Lambda function that sleeps until 05:00.
Answer: B. Scheduler supports time zones, ECS targets, retries and DLQs natively. A needs a server and manual DST handling. C runs once per web instance and dies with it. D cannot run longer than 15 minutes and wastes money.
Completion checklist
- I can explain visibility timeout, at-least-once delivery, DLQ and redrive
- I can describe the idempotency pattern with a conditional DynamoDB write
- I ran a worker locally against SQS and saw a message reach the DLQ
- I can explain graceful shutdown with
stopTimeoutandShutdownTimeout - I can choose between a Fargate worker and a Lambda SQS trigger
- I deleted my queues, schedule and idempotency table
Key takeaways
- Slow, bursty or scheduled work leaves the request path: queue + worker or schedule + job.
- Everything is at least once: idempotency with business keys and conditional writes is mandatory.
- Delete last, extend visibility for long work, and send poison messages to a DLQ with an alarm.
- Honour
SIGTERM: stop receiving, finish or abandon cleanly, exit beforestopTimeout. - Use EventBridge Scheduler with a time zone for jobs; never timers inside web processes.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Authentication and authorisation for .NET APIs on AWS
Protect an ASP.NET Core API with Amazon Cognito access tokens, validate JWTs correctly (issuer, token_use, client_id, scopes), use policy-based authorisation, and keep user identity separate from the IAM role your code runs as.
Porting .NET Framework apps to AWS
Decide whether to rehost, replatform or port a legacy .NET Framework 4.8 banking application, use AWS Transform for .NET and Microsoft's upgrade tooling, fix the classic blockers (System.Web, WCF, configuration, Windows-only APIs), and know when Windows on EC2 or Beanstalk is the right answer.