Skip to Content
Cloud Engineering Academy
3. .NET Application Development on AWS

Background processing: worker services, SQS consumers and scheduled jobs

Move slow and scheduled work out of the request path: .NET Worker Services on Fargate, SQS consumers with long polling, visibility timeouts, idempotency and dead-letter queues, Lambda SQS triggers with partial batch responses, and EventBridge Scheduler jobs.

intermediate⏱ 2 h 30 min🟢 FreeDVA-C03SAA-C03

Assumes: .NET 10 LTS · AWSSDK.SQS 4.1.0.2 · AWS.Messaging 1.6.0 · Amazon.Lambda.SQSEvents 3.0.1 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Decide which work belongs in the request, in a queue consumer, or in a scheduled job, and pick the AWS host for each (Fargate worker, Lambda, EventBridge Scheduler).
  • Write a .NET Worker Service (BackgroundService) that consumes Amazon SQS with long polling, processes messages concurrently with a bound, and shuts down gracefully on SIGTERM.
  • Explain SQS visibility timeout, at-least-once delivery, dead-letter queues and redrive, and make a consumer idempotent.
  • Use Lambda with an SQS event source and partial batch responses so one bad message does not reprocess a whole batch.
  • Schedule jobs with Amazon EventBridge Scheduler (cron in the Europe/London time zone, retries, DLQ) instead of timers inside web processes.
  • Know when the AWS Message Processing Framework for .NET (AWS.Messaging) saves you code.

Prerequisites

  • Module 3 overview — hosting options; ECS and Lambda basics
  • Scaling .NET apps — why timers in web processes run N times
  • Logging and errors — correlation IDs, which must also flow through messages
  • Tools: .NET 10 SDK or Docker, AWS CLI v2 with learn-sandbox. The SQS parts fit in the free tier.

Why this matters in banking

Harbourline customers can request a PDF statement for any month. Generating it takes 3–20 seconds: read transactions, render, encrypt, store in S3, notify the customer. If the Accounts API did this inside the HTTP request:

  • Requests would time out behind the load balancer and API Gateway (API Gateway's integration timeout is far shorter than 20 seconds for many setups).
  • A spike of requests at month-end would exhaust the API's threads and database connections, slowing balance checks for everyone.
  • A crash halfway through would lose the request; the customer would retry and might get two statements, or none.

Banks run on background work: statement generation, interest accrual at midnight, direct-debit files at 04:00, sanctions re-screening, notifications, reconciliation. Every one of these needs to survive restarts, avoid double execution, and leave an audit trail.

The idea in plain language

Background processing means doing work after responding to the user, or on a timetable, in a separate process.

A queue is a to-do list shared between a producer (the API: "please generate this statement") and consumers (workers that take items off the list). If workers are busy or down, items wait safely. If there is a lot of work, you add workers.

At-least-once delivery means the queue guarantees every item is delivered, but occasionally delivers one twice. Your worker must therefore be idempotent: doing the same item twice must have the same effect as doing it once.

A scheduler starts work at fixed times: "every weekday at 06:00 London time".

Analogy: the bank's back office in-tray. Branch staff (the API) drop request forms in the tray and tell the customer "you'll get it in the post". Clerks (workers) take forms out one at a time; while a clerk works on a form it is set aside, not thrown away (visibility timeout). If the clerk goes home sick, the form goes back in the tray. Forms that keep failing go to the supervisor's tray (dead-letter queue). And a wall calendar (the scheduler) says "run the overnight interest batch at 00:30".

What problem SQS and EventBridge Scheduler solve

Amazon SQS is a fully managed message queue. Two types:

Standard queueFIFO queue
OrderingBest effortStrict per message group
DeliveryAt least once (rare duplicates)Exactly-once processing within a 5-minute deduplication window
ThroughputNearly unlimitedLower (high-throughput mode raises it)
Use forMost background workPer-account ordered processing, e.g. postings for one account

Key behaviours and limits (verify current values in the SQS quotas page before designing around them):

  • Visibility timeout: after a consumer receives a message, it is hidden from others for this long (default 30 seconds, maximum 12 hours). If the consumer does not delete it in time, it reappears.
  • Long polling: WaitTimeSeconds up to 20 seconds; the call returns as soon as messages arrive. It cuts empty responses, which are billed requests.
  • Batch: up to 10 messages per ReceiveMessage and per batch send/delete.
  • Retention: default 4 days, maximum 14 days.
  • Message size: the maximum payload is now 1 MiB per request; each 64 KB chunk is billed as one request. Keep messages small and put documents in S3.
  • Dead-letter queue (DLQ): after maxReceiveCount failed receives, SQS moves the message to a DLQ for inspection. Redrive moves messages back after a fix.

Amazon EventBridge Scheduler is a serverless scheduler: one-off (at(...)), rate (rate(1 hour)) and cron schedules, with time zones (so 06:00 means 06:00 in London in both GMT and BST), flexible time windows, retries with a maximum event age, a DLQ, and targets in more than 270 services (SQS, Lambda, ECS RunTask, Step Functions and others).

Limitations: SQS does not do routing or fan-out (use SNS or EventBridge for that); a standard queue can deliver duplicates and out of order; a message being processed for longer than its visibility timeout is processed twice. Scheduler is at-least-once too, so scheduled jobs also need idempotency.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
BackgroundService / Worker ServiceSame .NET code, hosted on ECS Fargate (a service with no load balancer)Handle SIGTERM and ECS stopTimeout
Azure Service Bus queueAmazon SQS (standard or FIFO)No sessions/peek-lock API; visibility timeout plays the lock role. FIFO message groups ≈ sessions
Service Bus dead-letter sub-queueSQS dead-letter queue (a separate queue) + redriveDLQ is configured per source queue with maxReceiveCount
Azure Storage queuesAmazon SQS standardSimilar simplicity; SQS has DLQs built in
Azure Functions Service Bus / queue triggerLambda with an SQS event source mappingPartial batch responses instead of per-message completion
WebJobs TimerTrigger, Functions timer triggerEventBridge Scheduler → Lambda / ECS task / SQSSchedules live outside your code, with IAM roles, retries and DLQ
Hangfire / Quartz.NET inside the web appEventBridge Scheduler + a workerNo shared database lock tables needed
MassTransit / NServiceBusAWS.Messaging (AWS Message Processing Framework for .NET)Lighter: publish/subscribe, handlers, SQS poller, Lambda support

Architecture

Message lifecycle:

Security boundary view. The API's role may only sqs:SendMessage to this queue; the worker's task role may only ReceiveMessage, DeleteMessage, ChangeMessageVisibility on it, plus PutObject on one bucket prefix and PutItem on the idempotency table. The queue is encrypted (SQS-managed SSE by default, or SSE-KMS with your key), and messages carry identifiers, never personal data or documents.

Idempotency: the rule that keeps money correct

Because every queue and scheduler here is at-least-once, design every handler so that a duplicate is harmless:

  1. A stable key per unit of work. For a statement: {accountId}:{month}. For a payment: the client's Idempotency-Key.
  2. Record "done" atomically with a condition. A DynamoDB PutItem with ConditionExpression = attribute_not_exists(pk) succeeds only the first time. A second attempt gets ConditionalCheckFailedException, which you treat as "already done → delete the message".
  3. Make side effects naturally idempotent where you can. Writing s3://.../ACC-0000000001/2026-10.pdf twice produces the same object. Sending an email twice does not, so record "notified" before or with sending.
  4. Expire idempotency records with DynamoDB TTL after the window in which duplicates can occur (days, not forever).

Graceful shutdown on ECS

When ECS stops a task (deployment, scale-in, Spot interruption), it sends SIGTERM, waits stopTimeout (default 30 seconds, configurable up to 120 seconds on Fargate), then sends SIGKILL. .NET's generic host translates SIGTERM into cancelling the stoppingToken of every BackgroundService and waits up to HostOptions.ShutdownTimeout (30 seconds by default).

Good worker behaviour on SIGTERM:

  • Stop receiving new messages immediately.
  • Finish in-flight messages if they can complete within the timeout; otherwise leave them undeleted so they reappear for another worker.
  • Never delete a message whose work did not complete.

Set HostOptions.ShutdownTimeout a few seconds below the container stopTimeout so .NET finishes cleanly before the kill.

Step-by-step implementation

Create the queue and its DLQ

A DLQ first, then the main queue with a redrive policy (maxReceiveCount 5) and a visibility timeout longer than the slowest normal processing time (the code below uses 120 seconds plus heartbeats).

Make the API enqueue instead of doing the work

The API validates the request, sends a small message (accountId, month, requestedBy, correlationId), and returns 202 Accepted with a status URL.

Write the worker

A BackgroundService with long polling, a concurrency limit, per-message correlation scopes, idempotency, heartbeats for slow work, and delete-on-success only.

Run it on Fargate as a service without a load balancer

Desired count 1, auto scaling on the queue's ApproximateNumberOfMessagesVisible (or backlog per task), stopTimeout 60 seconds.

Add the schedule

EventBridge Scheduler sends a "generate all statements for last month" trigger message on the 1st at 06:00 London time, with retries and a DLQ.

Alarm on the DLQ

Any message in the DLQ is a failure a human must look at. Alarm on ApproximateNumberOfMessagesVisible > 0 for the DLQ, and on ApproximateAgeOfOldestMessage for the main queue.

AWS Console instructions

  • Amazon SQS → Create queue → Standard, name hl-m03-statement-requests-dlq, retention 14 days. Create.
  • Create queue → Standard, name hl-m03-statement-requests; Visibility timeout 2 minutes; Receive message wait time 20 seconds; Encryption SSE-SQS (or SSE-KMS); Dead-letter queue enabled → the DLQ ARN, Maximum receives 5; tags Project=cea-lab, Lab=m03-background.
  • Queue → Send and receive messages lets you send a test message and poll.
  • DLQ → Start DLQ redrive moves messages back to the source queue after you fix a bug.
  • Amazon EventBridge → Scheduler → Schedules → Create schedule: recurring, cron 0 6 1 * ? *, time zone Europe/London, flexible window off; target Amazon SQS SendMessage → the queue; retry policy and DLQ under Settings.

AWS CLI

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
TAGS='Project=cea-lab,Lab=m03-background,Environment=learn,Owner=<your-name>'

# 1. DLQ, then the main queue with redrive to it
DLQ_URL=$(aws sqs create-queue --queue-name hl-m03-statement-requests-dlq \
  --attributes MessageRetentionPeriod=1209600,SqsManagedSseEnabled=true \
  --tags "$TAGS" --query QueueUrl --output text)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" --attribute-names QueueArn --query Attributes.QueueArn --output text)

cat > /tmp/queue-attrs.json <<EOF
{
  "VisibilityTimeout": "120",
  "ReceiveMessageWaitTimeSeconds": "20",
  "SqsManagedSseEnabled": "true",
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"$DLQ_ARN\",\"maxReceiveCount\":\"5\"}"
}
EOF
QUEUE_URL=$(aws sqs create-queue --queue-name hl-m03-statement-requests \
  --attributes file:///tmp/queue-attrs.json --tags "$TAGS" --query QueueUrl --output text)

# 2. Send a test request (no personal data: identifiers only)
aws sqs send-message --queue-url "$QUEUE_URL" \
  --message-body '{"accountId":"ACC-0000000001","month":"2026-10","requestedBy":"cref_3f9a1c0d2b4e6f70"}' \
  --message-attributes 'CorrelationId={DataType=String,StringValue=cli-test-0001}'

# 3. Inspect backlog and the DLQ
aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
  --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible
aws sqs get-queue-attributes --queue-url "$DLQ_URL" --attribute-names ApproximateNumberOfMessages

# 4. After fixing a bug: move DLQ messages back to the source queue
aws sqs start-message-move-task --source-arn "$DLQ_ARN"

# 5. EventBridge Scheduler: monthly trigger at 06:00 London time on the 1st
#    (needs a role that scheduler.amazonaws.com can assume, allowed sqs:SendMessage on the queue)
QUEUE_ARN=$(aws sqs get-queue-attributes --queue-url "$QUEUE_URL" --attribute-names QueueArn --query Attributes.QueueArn --output text)
aws scheduler create-schedule --name hl-m03-monthly-statements \
  --schedule-expression "cron(0 6 1 * ? *)" \
  --schedule-expression-timezone "Europe/London" \
  --flexible-time-window Mode=OFF \
  --target "{\"Arn\":\"$QUEUE_ARN\",\"RoleArn\":\"arn:aws:iam::<account-id>:role/hl-m03-scheduler-sqs\",\"Input\":\"{\\\"batch\\\":\\\"monthly-statements\\\"}\",\"RetryPolicy\":{\"MaximumRetryAttempts\":3,\"MaximumEventAgeInSeconds\":3600},\"DeadLetterConfig\":{\"Arn\":\"$DLQ_ARN\"}}"

Infrastructure as Code

WorkerStack.cs (excerpt)
using Amazon.CDK;
using Amazon.CDK.AWS.ApplicationAutoScaling;
using Amazon.CDK.AWS.ECS;
using Amazon.CDK.AWS.ECS.Patterns;
using Amazon.CDK.AWS.SQS;

var dlq = new Queue(this, "Dlq", new QueueProps
{
    QueueName = "hl-m03-statement-requests-dlq",
    RetentionPeriod = Duration.Days(14),
    Encryption = QueueEncryption.SQS_MANAGED,
    EnforceSSL = true,
});

var queue = new Queue(this, "Requests", new QueueProps
{
    QueueName = "hl-m03-statement-requests",
    VisibilityTimeout = Duration.Minutes(2),
    ReceiveMessageWaitTime = Duration.Seconds(20),
    Encryption = QueueEncryption.SQS_MANAGED,
    EnforceSSL = true,
    DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 },
});

// Fargate service with no load balancer; scales on backlog. The pattern grants consume permissions.
var worker = new QueueProcessingFargateService(this, "Worker", new QueueProcessingFargateServiceProps
{
    Cluster = cluster,
    Queue = queue,
    Image = ContainerImage.FromEcrRepository(repo, "worker-v1"),
    Cpu = 256,
    MemoryLimitMiB = 512,
    MinScalingCapacity = 0,           // scale to zero when the queue is empty
    MaxScalingCapacity = 4,
    ScalingSteps =
    [
        new ScalingInterval { Upper = 0, Change = -1 },
        new ScalingInterval { Lower = 1, Change = +1 },
        new ScalingInterval { Lower = 100, Change = +2 },
    ],
    Environment = new Dictionary<string, string> { ["Worker__MaxConcurrency"] = "4" },
});
// Give in-flight messages time to finish on SIGTERM (Fargate allows up to 120 s).
((CfnTaskDefinition)worker.TaskDefinition.Node.DefaultChild!).AddPropertyOverride(
    "ContainerDefinitions.0.StopTimeout", 60);

Complete code

A Worker Service that consumes statement requests. It runs locally against real SQS with your SSO profile, or entirely offline in its unit test (the queue client is mocked).

Harbourline.Statements.Worker.csproj
<Project Sdk="Microsoft.NET.Sdk.Worker">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <RootNamespace>Harbourline.Statements.Worker</RootNamespace>
    <InternalsVisibleTo>Harbourline.Statements.Worker.Tests</InternalsVisibleTo>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="4.1.0.2" />
    <PackageReference Include="AWSSDK.SQS" Version="4.1.0.2" />
    <PackageReference Include="AWSSDK.DynamoDBv2" Version="4.1.0.2" />
  </ItemGroup>
</Project>
Program.cs
using Amazon.DynamoDBv2;
using Amazon.SQS;
using Harbourline.Statements.Worker;

var builder = Host.CreateApplicationBuilder(args);

builder.Logging.ClearProviders();
builder.Logging.AddJsonConsole(o => o.IncludeScopes = true);

// Finish in-flight work before ECS sends SIGKILL (container stopTimeout is 60 s).
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(50));

builder.Services.Configure<WorkerOptions>(builder.Configuration.GetSection("Worker"));
builder.Services.AddDefaultAWSOptions(builder.Configuration.GetAWSOptions());
builder.Services.AddAWSService<IAmazonSQS>();
builder.Services.AddAWSService<IAmazonDynamoDB>();
builder.Services.AddSingleton<IIdempotencyStore, DynamoDbIdempotencyStore>();
builder.Services.AddSingleton<IStatementGenerator, FakeStatementGenerator>();
builder.Services.AddHostedService<StatementRequestConsumer>();

await builder.Build().RunAsync();
StatementRequestConsumer.cs
using System.Text.Json;
using Amazon.SQS;
using Amazon.SQS.Model;
using Microsoft.Extensions.Options;

namespace Harbourline.Statements.Worker;

public sealed class WorkerOptions
{
    public string QueueUrl { get; set; } = "";
    public int MaxConcurrency { get; set; } = 4;           // messages processed in parallel per task
    public int VisibilityTimeoutSeconds { get; set; } = 120;
}

public sealed record StatementRequest(string AccountId, string Month, string RequestedBy);

public sealed class StatementRequestConsumer(
    IAmazonSQS sqs,
    IIdempotencyStore idempotency,
    IStatementGenerator generator,
    IOptions<WorkerOptions> options,
    ILogger<StatementRequestConsumer> log) : BackgroundService
{
    private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
    private readonly WorkerOptions _o = options.Value;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var slots = new SemaphoreSlim(_o.MaxConcurrency);
        var inFlight = new List<Task>();
        log.LogInformation("Consuming {QueueUrl} with concurrency {MaxConcurrency}", _o.QueueUrl, _o.MaxConcurrency);

        while (!stoppingToken.IsCancellationRequested)
        {
            ReceiveMessageResponse response;
            try
            {
                response = await sqs.ReceiveMessageAsync(new ReceiveMessageRequest
                {
                    QueueUrl = _o.QueueUrl,
                    MaxNumberOfMessages = Math.Min(10, _o.MaxConcurrency),
                    WaitTimeSeconds = 20,                       // long polling
                    MessageAttributeNames = ["CorrelationId"],
                    MessageSystemAttributeNames = ["ApproximateReceiveCount"],
                }, stoppingToken);
            }
            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
            {
                break; // SIGTERM while waiting: stop receiving
            }
            catch (AmazonSQSException ex)
            {
                log.LogWarning(ex, "ReceiveMessage failed; backing off");
                await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
                continue;
            }

            foreach (var message in response.Messages ?? [])   // SDK v4: null when empty
            {
                await slots.WaitAsync(stoppingToken);
                inFlight.Add(Task.Run(async () =>
                {
                    try { await HandleAsync(message, stoppingToken); }
                    finally { slots.Release(); }
                }, CancellationToken.None));
            }
            inFlight.RemoveAll(t => t.IsCompleted);
        }

        // Graceful shutdown: let started messages finish (or time out) before the host stops.
        await Task.WhenAll(inFlight);
        log.LogInformation("Consumer stopped");
    }

    internal async Task HandleAsync(Message message, CancellationToken stoppingToken)
    {
        var correlationId = message.MessageAttributes?.GetValueOrDefault("CorrelationId")?.StringValue ?? message.MessageId;
        using var scope = log.BeginScope(new Dictionary<string, object> { ["CorrelationId"] = correlationId, ["MessageId"] = message.MessageId });

        StatementRequest? request;
        try
        {
            request = JsonSerializer.Deserialize<StatementRequest>(message.Body, Json);
        }
        catch (JsonException)
        {
            request = null;
        }
        if (request is null || !request.AccountId.StartsWith("ACC-", StringComparison.Ordinal))
        {
            // A poison message will never succeed: leave it for the DLQ, don't spin on it.
            log.LogError("Malformed statement request; leaving it to reach the DLQ");
            return;
        }

        var key = $"statement:{request.AccountId}:{request.Month}";
        if (await idempotency.WasCompletedAsync(key, stoppingToken))
        {
            log.LogInformation("Duplicate delivery of {Key}; deleting", key);
            await DeleteAsync(message);
            return;
        }

        // Heartbeat: extend visibility while slow work runs, so no other worker picks it up.
        using var work = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken);
        var heartbeat = HeartbeatAsync(message, work.Token);
        try
        {
            await generator.GenerateAsync(request, work.Token);
            await idempotency.MarkCompletedAsync(key, stoppingToken);
            await DeleteAsync(message);
            log.LogInformation("Statement {Month} generated for {AccountId}", request.Month, request.AccountId);
        }
        catch (Exception ex) when (ex is not OperationCanceledException || !stoppingToken.IsCancellationRequested)
        {
            // Not deleted → visible again after the timeout → retried → DLQ after maxReceiveCount.
            log.LogError(ex, "Statement generation failed for {AccountId}; it will be retried", request.AccountId);
        }
        finally
        {
            await work.CancelAsync();
            await heartbeat.ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing);
        }
    }

    private async Task HeartbeatAsync(Message message, CancellationToken ct)
    {
        var every = TimeSpan.FromSeconds(_o.VisibilityTimeoutSeconds / 2.0);
        while (!ct.IsCancellationRequested)
        {
            await Task.Delay(every, ct);
            await sqs.ChangeMessageVisibilityAsync(new ChangeMessageVisibilityRequest
            {
                QueueUrl = _o.QueueUrl,
                ReceiptHandle = message.ReceiptHandle,
                VisibilityTimeout = _o.VisibilityTimeoutSeconds,
            }, ct);
        }
    }

    // CancellationToken.None: once the work is done, the delete must happen even during shutdown.
    private Task DeleteAsync(Message message) =>
        sqs.DeleteMessageAsync(new DeleteMessageRequest { QueueUrl = _o.QueueUrl, ReceiptHandle = message.ReceiptHandle }, CancellationToken.None);
}
Idempotency.cs
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;

namespace Harbourline.Statements.Worker;

public interface IIdempotencyStore
{
    Task<bool> WasCompletedAsync(string key, CancellationToken ct);
    Task MarkCompletedAsync(string key, CancellationToken ct);
}

/// <summary>Table hl-m03-idempotency: partition key "pk" (string), TTL attribute "expires_at".</summary>
public sealed class DynamoDbIdempotencyStore(IAmazonDynamoDB dynamo) : IIdempotencyStore
{
    private const string Table = "hl-m03-idempotency";

    public async Task<bool> WasCompletedAsync(string key, CancellationToken ct)
    {
        var item = await dynamo.GetItemAsync(new GetItemRequest
        {
            TableName = Table,
            Key = new() { ["pk"] = new AttributeValue { S = key } },
            ConsistentRead = true,
        }, ct);
        return item.Item is { Count: > 0 };
    }

    public async Task MarkCompletedAsync(string key, CancellationToken ct)
    {
        try
        {
            await dynamo.PutItemAsync(new PutItemRequest
            {
                TableName = Table,
                Item = new()
                {
                    ["pk"] = new AttributeValue { S = key },
                    ["completed_at"] = new AttributeValue { S = DateTimeOffset.UtcNow.ToString("O") },
                    ["expires_at"] = new AttributeValue { N = DateTimeOffset.UtcNow.AddDays(35).ToUnixTimeSeconds().ToString() },
                },
                ConditionExpression = "attribute_not_exists(pk)", // first writer wins
            }, ct);
        }
        catch (ConditionalCheckFailedException)
        {
            // Another worker finished the same request concurrently: that's fine.
        }
    }
}

public interface IStatementGenerator
{
    Task GenerateAsync(StatementRequest request, CancellationToken ct);
}

/// <summary>Stand-in for render + encrypt + S3 PutObject, so the sample runs without a renderer.</summary>
public sealed class FakeStatementGenerator : IStatementGenerator
{
    public Task GenerateAsync(StatementRequest request, CancellationToken ct) => Task.Delay(TimeSpan.FromSeconds(2), ct);
}
appsettings.json
{
  "AWS": { "Region": "us-east-1", "Profile": "learn-sandbox" },
  "Worker": {
    "QueueUrl": "https://sqs.us-east-1.amazonaws.com/<account-id>/hl-m03-statement-requests",
    "MaxConcurrency": 4,
    "VisibilityTimeoutSeconds": 120
  }
}

Remove "Profile" for deployed environments: on Fargate the task role is used automatically.

A unit test with xUnit v3 and NSubstitute that proves a duplicate delivery is deleted without regenerating:

StatementRequestConsumerTests.cs
using Amazon.SQS;
using Amazon.SQS.Model;
using Harbourline.Statements.Worker;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;

public class StatementRequestConsumerTests
{
    [Fact]
    public async Task Duplicate_delivery_is_deleted_without_generating_again()
    {
        var sqs = Substitute.For<IAmazonSQS>();
        var store = Substitute.For<IIdempotencyStore>();
        store.WasCompletedAsync("statement:ACC-0000000001:2026-10", Arg.Any<CancellationToken>()).Returns(true);
        var generator = Substitute.For<IStatementGenerator>();
        var consumer = new StatementRequestConsumer(sqs, store, generator,
            Options.Create(new WorkerOptions { QueueUrl = "https://sqs.us-east-1.amazonaws.com/111122223333/q" }),
            NullLogger<StatementRequestConsumer>.Instance);

        await consumer.HandleAsync(new Message
        {
            MessageId = "m-1",
            ReceiptHandle = "rh-1",
            Body = """{"accountId":"ACC-0000000001","month":"2026-10","requestedBy":"cref_3f9a1c0d2b4e6f70"}""",
        }, TestContext.Current.CancellationToken);

        await generator.DidNotReceive().GenerateAsync(Arg.Any<StatementRequest>(), Arg.Any<CancellationToken>());
        await sqs.Received(1).DeleteMessageAsync(Arg.Is<DeleteMessageRequest>(r => r.ReceiptHandle == "rh-1"), Arg.Any<CancellationToken>());
    }
}

The test project follows the lab pattern: xunit.v3 4.0.3 and NSubstitute 6.2.0, <OutputType>Exe</OutputType>, Microsoft.Testing.Platform enabled in global.json.

Code explained line by line

  1. Microsoft.NET.Sdk.Worker — the Worker Service SDK: generic host, no web server. On ECS it is a service with no load balancer and no port.
  2. ShutdownTimeout = 50 s — the host waits this long for ExecuteAsync to finish after SIGTERM. It is below the container's 60-second stopTimeout, so .NET exits before ECS kills it.
  3. AddAWSService<IAmazonSQS>() — one thread-safe SQS client for the process.
  4. SemaphoreSlim(MaxConcurrency) — bounds parallel work per task. Total concurrency = tasks × MaxConcurrency; keep that within what downstream systems tolerate.
  5. ReceiveMessageAsync with WaitTimeSeconds = 20 — long polling: fewer empty, billed receives, faster pickup.
  6. MaxNumberOfMessages = Math.Min(10, MaxConcurrency) — SQS returns at most 10 per call; never receive more than you can start, or messages sit invisible while waiting.
  7. MessageAttributeNames = ["CorrelationId"] — the API puts the request's correlation ID on the message so logs connect the HTTP request and the background work.
  8. catch (OperationCanceledException) when stoppingToken... — on SIGTERM the long poll is cancelled; leave the loop and stop receiving.
  9. response.Messages ?? [] — in SDK v4, collection properties are null when empty (v3 returned empty lists). A classic v3→v4 migration bug.
  10. Task.Run(..., CancellationToken.None) with finally slots.Release() — starts each message independently and always frees its slot.
  11. log.BeginScope(...) — every log line for the message carries CorrelationId and MessageId.
  12. Malformed message → return without delete — it will be received again up to maxReceiveCount, then moved to the DLQ where someone can inspect it. (A refinement: send it to the DLQ immediately yourself.)
  13. WasCompletedAsync → delete duplicate — the idempotency check that makes at-least-once delivery safe.
  14. HeartbeatAsync — every half visibility timeout, ChangeMessageVisibility extends the lease. Without it, work longer than 120 seconds would be delivered to a second worker.
  15. MarkCompletedAsync then DeleteAsync — the order matters: record completion first, then delete. If the process dies in between, the redelivered message is recognised as a duplicate.
  16. catch ... when (...) — on failure, do not delete; SQS's redelivery is the retry mechanism. Shutdown cancellations are excluded so they are not logged as errors.
  17. DeleteAsync(..., CancellationToken.None) — once work is done, the delete must not be cancelled by shutdown.
  18. ConditionExpression = "attribute_not_exists(pk)" — DynamoDB's atomic "insert if absent"; ConditionalCheckFailedException means someone else already completed it.
  19. expires_at with TTL — idempotency records expire after 35 days, longer than queue retention (14 days max) plus margin.

The same consumer with less code: AWS.Messaging and Lambda

AWS Message Processing Framework for .NET (AWS.Messaging 1.6.0) wraps polling, deserialisation, visibility extension and deletion. You write handlers:

With AWS.Messaging (sketch)
builder.Services.AddAWSMessageBus(bus =>
{
    bus.AddSQSPoller(builder.Configuration["Worker:QueueUrl"]!);
    bus.AddMessageHandler<StatementRequestHandler, StatementRequest>();
});

public sealed class StatementRequestHandler(IStatementGenerator generator) : IMessageHandler<StatementRequest>
{
    public async Task<MessageProcessStatus> HandleAsync(MessageEnvelope<StatementRequest> envelope, CancellationToken token = default)
    {
        await generator.GenerateAsync(envelope.Message, token);
        return MessageProcessStatus.Success(); // the framework deletes the message
    }
}

The framework uses its own message envelope (CloudEvents-style) for messages it publishes, so producers should publish with IMessagePublisher from the same framework. Idempotency is still your job.

Lambda with an SQS event source mapping removes the worker host entirely. Lambda polls the queue, invokes your function with a batch, and deletes messages on success. With ReportBatchItemFailures enabled, you return only the failed message IDs, and only those return to the queue:

Function.cs (Lambda SQS consumer)
using Amazon.Lambda.Core;
using Amazon.Lambda.SQSEvents;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace Harbourline.Statements.Lambda;

public sealed class Function
{
    public async Task<SQSBatchResponse> HandleAsync(SQSEvent sqsEvent, ILambdaContext context)
    {
        var failures = new List<SQSBatchResponse.BatchItemFailure>();
        foreach (var record in sqsEvent.Records)
        {
            try
            {
                await ProcessAsync(record.Body, context);
            }
            catch (Exception ex)
            {
                context.Logger.LogError($"Message {record.MessageId} failed: {ex.GetType().Name}");
                failures.Add(new SQSBatchResponse.BatchItemFailure { ItemIdentifier = record.MessageId });
            }
        }
        return new SQSBatchResponse(failures); // only these return to the queue
    }

    private static Task ProcessAsync(string body, ILambdaContext context) => Task.CompletedTask; // idempotent work here
}

Rules for Lambda + SQS: set the queue's visibility timeout to at least six times the function timeout (AWS's guidance, so retries and throttling don't cause duplicate processing), cap concurrency with the event source mapping's maximum concurrency setting, and keep the DLQ on the queue (not the function) so failures land in one place.

Fargate workerLambda SQS trigger
Long jobs (minutes)Yes, unlimitedUp to 15 minutes per invocation
Idle costTask-hours unless scaled to zeroNone
Concurrency controlTasks × MaxConcurrencyEvent source mapping maximum concurrency
Heavy dependencies (PDF renderers, fonts)Easy in a containerPossible with container images
Code you maintainPolling loop (or AWS.Messaging)Handler only

Security considerations

⚠ Learning shortcut — not for production

The lab queues use SQS-managed encryption and the sample worker uses your SSO profile locally. In production use SSE-KMS with a customer managed key where your data classification requires it, a queue policy that denies non-TLS access (aws:SecureTransport) and restricts senders to the API's role, a dedicated task role for the worker with only receive/delete/change-visibility on this queue, and no Profile in deployed configuration.

  • Messages are not a document store. Put identifiers in messages, documents in S3 (SSE-KMS, Block Public Access), and personal data nowhere it isn't needed. Messages appear in DLQs, in logs when something fails, and in redrive tools.
  • Validate every message. A queue is an input boundary like an HTTP endpoint. Check formats and authorisation context (requestedBy) before acting.
  • Least privilege for schedules. The Scheduler role may only sqs:SendMessage on one queue (or lambda:InvokeFunction on one function).
  • Audit trail. Log start, outcome and duration of each job with the correlation ID; operations and auditors ask "did the 1 October statement run complete for everyone?".

Banking use case

Harbourline's statement pipeline:

  • The Accounts API accepts POST /v1/statements with Idempotency-Key, writes a "requested" record, enqueues {accountId, month, correlationId} and returns 202 with Location: /v1/statements/{id}.
  • A Fargate worker service (0–4 tasks, scaling on backlog) renders PDFs, stores them encrypted in S3 under statements/{accountId}/{yyyy-MM}.pdf, and records completion in DynamoDB.
  • On the 1st of each month at 06:00 London time, EventBridge Scheduler sends a trigger message; a small "fan-out" job enqueues one message per active account in batches of 10.
  • Failures go to the DLQ; an alarm pages the on-call team; after a fix, a DLQ redrive reprocesses them; idempotency guarantees no customer gets two statements.
  • Interest accrual, which must run exactly once per day per account, uses a FIFO queue with the account ID as message group and the business date in the deduplication ID, plus the same idempotency table.

Common mistakes

  • Timers in the web API → the job runs once per task → EventBridge Scheduler or a single worker.
  • Visibility timeout shorter than processing time → duplicates under load → longer timeout plus heartbeats; for Lambda, six times the function timeout.
  • Deleting the message before the work is done → work lost on crash → delete last.
  • No DLQ → poison messages loop until retention expires, costing money and hiding failures → DLQ with alarm.
  • No idempotency → duplicates create double statements or double postings → conditional writes keyed by business identifiers.
  • Short polling (WaitTimeSeconds = 0) → thousands of empty billed receives → long polling.
  • Ignoring SIGTERM → half-done work killed at every deployment → honour stoppingToken, set stopTimeout and ShutdownTimeout.
  • SDK v4 null collections → NullReferenceException on response.Messages when the queue is empty → ?? [].

Troubleshooting

SymptomLikely causeFix
Same message processed twiceProcessing longer than visibility timeout; or worker crashed after work, before deleteHeartbeats; idempotency
Messages pile up, ApproximateAgeOfOldestMessage risesToo few workers, or a stuck worker holding messagesScale on backlog; check for exceptions; per-message timeouts
Messages go straight to the DLQPoison message, or maxReceiveCount too low with transient errorsInspect the DLQ message; raise maxReceiveCount to 3–5
AccessDenied on ReceiveMessageTask role lacks permission, or queue policy deniesGrant on the queue ARN; check KMS key policy for SSE-KMS
KMS.AccessDeniedException sending/receivingSSE-KMS key policy doesn't allow the roleAllow kms:GenerateDataKey (send) and kms:Decrypt (receive)
Worker killed at deploy, logs end abruptlystopTimeout too shortRaise to 60–120 s; lower ShutdownTimeout below it
Lambda reprocesses whole batchesReportBatchItemFailures not enabled, or function throwsEnable it; return SQSBatchResponse instead of throwing
Schedule fires an hour off in summerCron in UTCSet schedule-expression-timezone Europe/London
NullReferenceException on empty receiveSDK v4 returns null collectionsNull-coalesce

Cost and cleanup

  • SQS: the first 1 million requests per month are free; beyond that standard queues cost cents per million requests ⚠ VERIFY current per-million price. Each 64 KB chunk of a payload is a request. Long polling keeps idle cost near zero.
  • EventBridge Scheduler: a monthly free tier of invocations (14 million) and about USD 1.00 per million after ⚠ VERIFY.
  • DynamoDB idempotency table on-demand: fractions of a cent for lab traffic.
  • Fargate worker: billed per task-hour when running; scale to zero when the queue is empty, and destroy after the lab.
aws scheduler delete-schedule --name hl-m03-monthly-statements --profile learn-sandbox
aws sqs delete-queue --queue-url "$QUEUE_URL" --profile learn-sandbox
aws sqs delete-queue --queue-url "$DLQ_URL" --profile learn-sandbox
aws dynamodb delete-table --table-name hl-m03-idempotency --profile learn-sandbox   # if you created it

aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m03-background \
  --query "ResourceTagMappingList[].ResourceARN" --profile learn-sandbox

Exercises

  • 🟢 Guided: Create the two queues with the CLI, run the worker locally with your SSO profile, send three messages (one malformed), and observe: two processed and deleted, the malformed one received five times and then in the DLQ. Redrive it and watch it fail again.
  • 🟡 Challenge: Replace the hand-written polling loop with AWS.Messaging, keeping idempotency and correlation IDs. Compare lines of code and behaviour on SIGTERM (docker stop sends it).
  • 🔴 Live problem: "Some customers received two copies of their October statement email. Worker logs show both copies were processed by different tasks about 2.5 minutes apart. Visibility timeout is 120 s. Statement generation for business accounts takes up to 4 minutes." Find the cause, explain why idempotency didn't prevent it, and fix it.

Assignment

Add a docs/background-jobs.md to your portfolio: a table of every background job for a bank feature of your choice (trigger, host, schedule/time zone, idempotency key, retry policy, DLQ, alarm, owner), plus the sequence diagram of one job end to end.

Solutions

Knowledge check

Interview questions

  • "How do you process messages from SQS reliably in .NET?" — Long polling, bounded concurrency, idempotent handlers keyed by business IDs, delete only after success, visibility heartbeats for long work, DLQ with alarm, graceful SIGTERM handling; or Lambda with partial batch responses.
  • "Fargate worker or Lambda for queue processing?" — Lambda for short, spiky, simple handlers with no idle cost; Fargate for long jobs, heavy dependencies, steady throughput or tight control of concurrency and connections.
  • "Where do scheduled jobs live?" — Outside the web app: EventBridge Scheduler targeting Lambda, an ECS task, Step Functions or a queue, with retries, DLQ and IAM roles, in the business time zone.
  • "How do you avoid paying a customer twice when messages are duplicated?" — Idempotency keys and conditional writes; FIFO with deduplication IDs where ordering per account matters; reconciliation as a final control.

Certification practice

Completion checklist

  • I can explain visibility timeout, at-least-once delivery, DLQ and redrive
  • I can describe the idempotency pattern with a conditional DynamoDB write
  • I ran a worker locally against SQS and saw a message reach the DLQ
  • I can explain graceful shutdown with stopTimeout and ShutdownTimeout
  • I can choose between a Fargate worker and a Lambda SQS trigger
  • I deleted my queues, schedule and idempotency table

Key takeaways

  • Slow, bursty or scheduled work leaves the request path: queue + worker or schedule + job.
  • Everything is at least once: idempotency with business keys and conditional writes is mandatory.
  • Delete last, extend visibility for long work, and send poison messages to a DLQ with an alarm.
  • Honour SIGTERM: stop receiving, finish or abandon cleanly, exit before stopTimeout.
  • Use EventBridge Scheduler with a time zone for jobs; never timers inside web processes.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page