Amazon SQS in depth
Configure Amazon SQS standard and FIFO queues for banking workloads — visibility timeout, long polling, batching, message groups, deduplication and encryption — and build a production-shaped .NET 10 consumer loop with batch deletes and a visibility heartbeat.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SQS 4.1.0.2, AWSSDK.Extensions.NETCore.Setup 4.1.0.2) · Microsoft.Extensions.Hosting 10.0.15 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain the message lifecycle in SQS — send, receive, in flight, delete or reappear — and the role of the visibility timeout and receipt handle.
- Choose between standard and FIFO queues, and use message groups, deduplication IDs and high-throughput FIFO correctly.
- Configure long polling, batching, delay, retention and encryption for a banking workload, and say what each costs.
- Write a .NET 10 Worker Service consumer with the AWS SDK v4 that long-polls, processes in parallel, deletes in batches, extends visibility with a heartbeat while work is still running, and shuts down cleanly.
- Send ordered, deduplicated messages to a FIFO queue from .NET and explain what happens when a sender retries.
- Read the key CloudWatch metrics for a queue and know which one to alarm on.
Prerequisites
- Asynchronous communication and event-driven architecture — queue vs topic vs bus vs stream
- Module 3: .NET on AWS — Worker Services,
AddAWSService<T>(), Lambda event sources - Tools: .NET 10 SDK, Docker Desktop (for the free local option), AWS CLI v2 with the
learn-sandboxSSO profile
Why this matters in banking
Harbourline's Notifications Service reads an SQS queue and sends "your salary has arrived" texts. In its first week in production three incidents happened:
- Duplicate texts. Sending one SMS took up to 40 seconds when the provider was slow. The queue's visibility timeout was the default 30 seconds. The message became visible again while the first worker was still sending, a second worker picked it up, and customers got two texts. Nobody lost money — but the same bug in the Payments Service would post a debit twice.
- A cost spike. Workers polled with the default short polling in a tight loop. At night, with an empty queue, 40 workers made millions of empty
ReceiveMessagecalls a month. The free tier vanished by the 10th. - Out-of-order balances. A customer received "balance £120" after "balance £95" because the queue was a standard queue, which does not preserve order, and the texts embedded a running balance.
Each incident came from a default nobody questioned. SQS is simple to start with — SendMessage, ReceiveMessage, DeleteMessage — but its behaviour under load and failure depends entirely on a handful of settings. This lesson goes through every one of them with banking consequences, then gives you a consumer you can reuse.
The idea in plain language
An SQS queue is a managed, durable to-do list shared between programs. Producers add items (messages). Consumers take an item, work on it, and then cross it off (delete it). AWS stores messages redundantly across several Availability Zones and scales the queue for you; there is no broker to size or patch.
The twist that makes SQS different from a database table: taking an item does not remove it. When a consumer receives a message, SQS hides it from everyone else for a while — the visibility timeout. If the consumer deletes the message within that time, it is gone. If not (the consumer crashed, hung or was too slow), the message reappears and another consumer gets it. This is how SQS guarantees that work is never lost just because a worker died.
Analogy: A bank's cheque-processing tray. A clerk takes a cheque and puts a "being processed — back by 10:30" card on the tray slot. If the clerk finishes, the cheque goes in the shredder (delete). If 10:30 passes and the clerk has not returned — perhaps they went home ill — the supervisor puts the cheque back in the tray for someone else. A slow clerk who is still working at 10:31 must ring the supervisor and say "give me another half hour" (extend the visibility timeout), or a colleague will process the same cheque.
The message lifecycle
Key terms:
| Term | Meaning |
|---|---|
| Message ID | A unique ID that SQS assigns when the message is sent. Stable for the message's life |
| Receipt handle | A token returned with each receive. You need the most recent one to delete or change visibility. It is not the message ID, and it changes every time the message is received |
| In flight | Received but not yet deleted or timed out. Invisible to other consumers |
| Visibility timeout | How long a received message stays hidden. Default 30 seconds, range 0 seconds to 12 hours |
| Approximate receive count | How many times the message has been received. Drives the redrive policy into a dead-letter queue (next lesson) |
| Retention period | How long an undeleted message lives. Default 4 days, range 60 seconds to 14 days |
What problem Amazon SQS solves
SQS gives you a queue with no capacity planning: unlimited backlog, very high throughput on standard queues, pay per request, and IAM-based security. It exists so you never run MSMQ, RabbitMQ or a home-made "jobs" table again for simple work distribution.
Limits and defaults (checked 2026-11-18)
| Setting | Value |
|---|---|
| Maximum message size | 1 MiB (1,048,576 bytes). Raised from 256 KiB in 2025. Larger payloads: store in S3 and send a pointer |
| Billing unit | Each 64 KB chunk of a request counts as one request — a 1 MiB message costs 16 |
| Message retention | Default 4 days; 60 seconds to 14 days |
| Visibility timeout | Default 30 seconds; 0 seconds to 12 hours |
Long-poll wait (WaitTimeSeconds) | 0 to 20 seconds |
| Delay (queue default or per message) | 0 to 15 minutes |
| Messages per batch call | Up to 10 (send, receive, delete, change visibility) |
| Message attributes | Up to 10 per message |
| In-flight messages | About 120,000 per standard queue; 120,000 per FIFO queue |
| Backlog | Unlimited |
| Queue name | Up to 80 characters; letters, digits, -, _; FIFO names end in .fifo |
| FIFO deduplication interval | 5 minutes |
| FIFO throughput (normal mode) | 300 API calls per second per action per partition; with batches of 10, 3,000 messages per second |
| FIFO high-throughput mode, us-east-1 | Up to 70,000 transactions per second per action without batching; up to 700,000 messages per second with batching |
MessageGroupId length | Up to 128 characters |
Honest limitations
- No replay. Once deleted, a message is gone. If you need history, use a stream or an EventBridge archive.
- No message selection. You cannot "receive only messages where
currency = EUR". Filter upstream (SNS filter policies, EventBridge rules) and use separate queues. - No priority. Use two queues (
high,normal) and poll the high one first, or use separate consumers. - Approximate metrics. Queue depth figures are approximate and lag by up to a minute or so.
- Standard queues can deliver twice and out of order. That is the price of near-unlimited throughput.
Standard vs FIFO
| Standard | FIFO | |
|---|---|---|
| Delivery | At least once; occasionally more than once | Exactly-once send within a 5-minute deduplication interval; a message can still be received again if not deleted in time |
| Ordering | Best effort; no guarantee | Strict within a message group |
| Throughput | Nearly unlimited | 300 API calls/s per action (3,000 msgs/s with batching); far higher in high-throughput mode |
| Name | Any valid name | Must end in .fifo |
MessageGroupId | Optional; turns on fair queues (tenant fairness), not ordering | Required |
MessageDeduplicationId | Not used | Required unless content-based deduplication is on |
| Per-message delay | Supported | Not supported (queue-level delay only) |
| Price | Cheaper per million requests | Higher per million requests ⚠ VERIFY current prices on the SQS pricing page |
| Typical use | Independent jobs: notifications, PDF generation, cache refresh | Per-account ledger updates, per-card status changes, anything where order changes the outcome |
Message groups: ordering without a global bottleneck
A FIFO queue does not deliver one message at a time for the whole queue. It keeps messages in order within each message group, and different groups proceed in parallel. While a message from group ACC-0000123456 is in flight, no other message from that group is delivered — but messages from ACC-0000999999 flow freely.
Choosing the group ID is the most important FIFO design decision:
- Too coarse (one group for everything): you get global order and a throughput of one message at a time. A single slow message blocks the bank.
- Too fine (a new GUID per message): you get parallelism but no ordering at all — you have paid for FIFO and got a standard queue.
- Right: the entity whose state the messages change. For ledger postings that is the account ID. For card status changes, the card token.
A blocked group is a real operational risk: if the head message of a group keeps failing, everything behind it waits until it succeeds or moves to the dead-letter queue. Set a redrive policy on every FIFO queue (next lesson).
Deduplication: what FIFO does and does not protect you from
FIFO deduplication stops the same send from creating two messages within five minutes. If your producer times out talking to SQS and retries with the same MessageDeduplicationId, SQS accepts the retry and drops the duplicate. Two ways to supply the ID:
- Explicit ID (recommended in banking): use a business key such as
transactionId. Deterministic, survives producer restarts. - Content-based deduplication: SQS uses a SHA-256 hash of the body (not the attributes). Simple, but two genuinely different messages with identical bodies are treated as duplicates, and a body that differs by a timestamp is not deduplicated.
What FIFO does not protect against:
- A consumer that processes a message and then crashes before deleting it. The message reappears and is processed again. Consumers must still be idempotent.
- A producer that resends after more than five minutes.
- Duplicates created upstream (the same event published twice by different code paths with different IDs).
High-throughput FIFO
Turning on high throughput sets two attributes: DeduplicationScope=messageGroup and FifoThroughputLimit=perMessageGroupId. Throughput then scales with the number of active message groups because SQS spreads groups across partitions. Deduplication becomes per group: the same deduplication ID in two different groups is two messages. If you change either attribute back, the queue returns to normal throughput.
Fair queues on standard queues
Since 2025, setting MessageGroupId on a standard queue enables fair queues: when one tenant floods the queue, SQS favours messages from quieter tenants so their wait time stays low. It does not order anything and needs no consumer changes. For Harbourline, a shared "statement generation" queue used by both retail and business banking could set the group ID to the business line so a month-end business burst does not delay retail statements. Fair-queue calls are billed at an extra rate on top of standard requests — see the pricing page.
Polling, batching and timing settings
Short vs long polling
- Short polling (
WaitTimeSeconds = 0):ReceiveMessagequeries a subset of SQS servers and returns immediately, possibly empty even when messages exist. Every empty response is a billed request. - Long polling (
WaitTimeSeconds1 to 20): SQS waits until at least one message is available or the wait expires, and queries all servers. Fewer empty responses, lower cost, lower latency to first message.
Always long-poll with 20 seconds unless you have a specific reason not to. Set it on the queue (ReceiveMessageWaitTimeSeconds=20) and in the request; the request value wins. Make sure your HTTP client timeout is longer than the wait (the SDK's default is fine).
Cost check: one worker short-polling every 100 ms makes about 26 million requests a month. The same worker long-polling with a 20-second wait makes at most about 130,000.
Batching
Send, receive, delete and change-visibility all accept up to 10 messages per call. Batching cuts request count (cost) and round trips (latency). Two catches:
- Batch calls can partially fail.
SendMessageBatchandDeleteMessageBatchreturn HTTP 200 with aFailedlist. Check it every time. - The 1 MiB limit applies to the whole batch, not each message.
Visibility timeout: how to choose
Rule of thumb: visibility timeout ≥ the longest you expect to spend on one message (or one batch), with margin. For Lambda consumers, AWS recommends at least six times the function timeout (plus the batching window if you use one), so that retries after throttling do not cause duplicates.
When processing time is unpredictable (calling a slow provider, generating a large PDF), do not set a huge timeout. A crashed worker would then hide its message for hours. Instead use a heartbeat: start with a moderate timeout (say 60 seconds) and call ChangeMessageVisibility periodically while work continues. The consumer below does exactly this.
ChangeMessageVisibility sets a new timeout counted from now, not added to the remaining time. The total time a message can stay in flight is capped at 12 hours from the original receive.
Delay queues and message timers
DelaySeconds (0 to 15 minutes) hides a new message before its first delivery. Use it for "try again shortly" patterns or to wait for a related record to commit. For FIFO queues only the queue-level delay is allowed. For delays longer than 15 minutes, use EventBridge Scheduler or Step Functions.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Service Bus peek-lock | SQS visibility timeout | Same idea. SQS has no explicit "abandon"; set visibility to 0 to release early |
RenewMessageLockAsync / auto lock renewal in ServiceBusProcessor | ChangeMessageVisibility heartbeat | You write the renewal yourself (or use AWS.Messaging, lesson 15, which does it for you) |
CompleteMessageAsync | DeleteMessage with the receipt handle | Use batch delete for throughput |
| Lock token | Receipt handle | Changes on every receive; old handles may fail |
Service Bus sessions (SessionId) | FIFO message groups (MessageGroupId) | No session state store; no explicit session accept — SQS assigns groups automatically |
| Duplicate detection window (configurable) | FIFO deduplication interval (fixed 5 minutes) | Not configurable in SQS |
MaxDeliveryCount + dead-letter sub-queue | maxReceiveCount + a separate DLQ queue (redrive policy) | The DLQ is a normal queue you create and secure |
Scheduled messages (ScheduledEnqueueTime) | DelaySeconds (max 15 minutes) | Use EventBridge Scheduler for longer |
| Message size up to 256 KB (Standard tier) / 100 MB (Premium) | 1 MiB | Use S3 for larger payloads |
ServiceBusProcessor with MaxConcurrentCalls | Your own loop, AWS.Messaging, or a Lambda event source mapping with maximum concurrency | SQS has no client-side processor in the base SDK |
| Azure Storage Queues | SQS standard | Storage Queues have a similar visibility model, 64 KB messages and 7-day default TTL |
Architecture
Two ways to consume:
| Your own poller (Worker Service on ECS/EC2) | Lambda event source mapping | |
|---|---|---|
| Who polls | Your code | The Lambda service |
| Scaling | You scale tasks (for example on queue depth) | Lambda scales automatically: starts with five concurrent batches and adds up to 300 concurrent invocations a minute, up to 1,250 ⚠ VERIFY whether provisioned mode changes your numbers |
| Batch size | Up to 10 per receive | Up to 10,000 for standard queues (needs a batching window of at least 1 second above 10); up to 10 for FIFO |
| Visibility extension | Your heartbeat | Not available; set visibility timeout to at least 6 × function timeout |
| Partial failures | Delete only the successes | Return batchItemFailures (ReportBatchItemFailures) |
| Cost when idle | Tasks run (and long-poll) all the time | Polling is included; you pay only for invocations ⚠ VERIFY current note on provisioned-mode poller charges |
| Good for | Long, heavy or very steady work; strict control | Spiky, short work; the module project |
Security-boundary view. Producers need sqs:SendMessage on one queue ARN; consumers need sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility and sqs:GetQueueAttributes on their queue only. The queue policy denies non-TLS access. With SSE-KMS, both sides also need kms:GenerateDataKey (send) or kms:Decrypt (receive) on the key. Workers in private subnets reach SQS through an interface VPC endpoint (com.amazonaws.us-east-1.sqs) rather than a NAT gateway, and the queue policy can require aws:SourceVpce.
Step-by-step implementation
You will create a standard queue with a DLQ and a FIFO queue, then run the .NET Worker below in two modes: send and consume. Two ways to run it:
- Free and offline: ElasticMQ, an open-source SQS-compatible server, in Docker. Good for learning the API; not identical to SQS in every edge case ⚠ VERIFY your ElasticMQ version supports the SQS JSON protocol that SDK v4 uses (ElasticMQ 1.5 and later advertise it).
- Real SQS: the AWS CLI commands below; stays inside the free tier.
Option A — start ElasticMQ locally
docker run --rm -d --name hl-elasticmq -p 9324:9324 -p 9325:9325 softwaremill/elasticmq-nativePort 9324 is the SQS API; 9325 is a small web UI showing queue depths. Create the queues with the same CLI commands as in the AWS CLI section, adding --endpoint-url http://localhost:9324 to each, and set HL_SQS_SERVICE_URL=http://localhost:9324 for the app.
Option B — create the queues in AWS
Run the AWS CLI section. It creates hl-m08-l2-notify (standard, 60-second visibility, 20-second long polling, SSE-SQS, redrive to hl-m08-l2-notify-dlq after 5 receives) and hl-m08-l2-ledger.fifo (content-based deduplication off, so you must send IDs).
Create the project
dotnet new worker -n Harbourline.Queues.Worker
cd Harbourline.Queues.Worker
dotnet add package AWSSDK.SQS --version 4.1.0.2
dotnet add package AWSSDK.Extensions.NETCore.Setup --version 4.1.0.2Replace the generated files with those in Complete code.
Send to the FIFO queue twice
dotnet run -- send-fifo
dotnet run -- send-fifoThe app sends five postings for two accounts, using transactionId as the deduplication ID. The second run reuses the same IDs (the sample is seeded) within five minutes, so SQS accepts the calls but adds nothing. Check ApproximateNumberOfMessages: it stays at 5.
Consume and watch ordering
dotnet run -- consume-fifoMessages for each account arrive in the order sent. The two accounts interleave, because they are different groups.
Watch a slow message survive with the heartbeat
dotnet run -- send-standard
dotnet run -- consume-standardOne message is marked slow and takes 150 seconds to process — longer than the 60-second visibility timeout. Watch the log: the heartbeat extends visibility every 30 seconds, so no second worker receives it. Then set HL_DISABLE_HEARTBEAT=true and repeat with two consumers in two terminals: you will see the slow message processed twice. That is incident 1 from the banking scenario, reproduced.
Tear down
Run the cleanup commands, and stop ElasticMQ with docker stop hl-elasticmq.
AWS Console instructions
- Console → Amazon SQS → Create queue.
- Type: Standard. Name:
hl-m08-l2-notify-dlq. Retention: 14 days (DLQs should keep messages longer than their source). Encryption: SSE-SQS. Add tagsProject=cea-lab,Lab=m08-l2,Environment=learn,Owner=<you>. Create queue. - Create another standard queue
hl-m08-l2-notify:- Configuration: visibility timeout
60seconds, message retention4days, receive message wait time20seconds, delivery delay0. - Encryption: SSE-SQS enabled.
- Dead-letter queue: Enabled, choose
hl-m08-l2-notify-dlq, Maximum receives5. - Tags as above. Create queue.
- Configuration: visibility timeout
- Create a FIFO queue
hl-m08-l2-ledger.fifo: visibility timeout60, wait time20, Content-based deduplication off, High throughput FIFO off for now. Tags. Create. - Explore hl-m08-l2-notify → Send and receive messages. Send a message, then Poll for messages. Open the received message's Details: note the receive count and that the message disappears from other pollers until its visibility timeout ends.
- Monitoring tab: find
ApproximateNumberOfMessagesVisible,ApproximateNumberOfMessagesNotVisibleandApproximateAgeOfOldestMessage.
AWS CLI
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
TAGS=Project=cea-lab,Lab=m08-l2,Environment=learn,Owner=you
# 1. Dead-letter queue first, so the source queue can reference it
DLQ_URL=$(aws sqs create-queue --queue-name hl-m08-l2-notify-dlq \
--attributes SqsManagedSseEnabled=true,MessageRetentionPeriod=1209600 \
--tags $TAGS --query QueueUrl --output text)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
--attribute-names QueueArn --query Attributes.QueueArn --output text)
# 2. Standard queue: 60 s visibility, 20 s long polling, 4-day retention, redrive after 5 receives
cat > notify-attrs.json <<EOF
{
"VisibilityTimeout": "60",
"ReceiveMessageWaitTimeSeconds": "20",
"MessageRetentionPeriod": "345600",
"SqsManagedSseEnabled": "true",
"RedrivePolicy": "{\"deadLetterTargetArn\":\"$DLQ_ARN\",\"maxReceiveCount\":\"5\"}"
}
EOF
NOTIFY_URL=$(aws sqs create-queue --queue-name hl-m08-l2-notify \
--attributes file://notify-attrs.json --tags $TAGS --query QueueUrl --output text)
# 3. FIFO queue: explicit dedup IDs (content-based off), normal throughput
LEDGER_URL=$(aws sqs create-queue --queue-name hl-m08-l2-ledger.fifo \
--attributes FifoQueue=true,ContentBasedDeduplication=false,VisibilityTimeout=60,ReceiveMessageWaitTimeSeconds=20,SqsManagedSseEnabled=true \
--tags $TAGS --query QueueUrl --output text)
# 4. Send one message to each, by hand
aws sqs send-message --queue-url "$NOTIFY_URL" \
--message-body '{"eventId":"cli-1","accountId":"ACC-0000123456","amount":12.50,"currency":"GBP"}'
aws sqs send-message --queue-url "$LEDGER_URL" \
--message-body '{"transactionId":"tx-cli-1","accountId":"ACC-0000123456","amount":-20.00}' \
--message-group-id ACC-0000123456 --message-deduplication-id tx-cli-1
# 5. Long-poll receive, asking for the receive count and group ID
aws sqs receive-message --queue-url "$LEDGER_URL" --max-number-of-messages 10 \
--wait-time-seconds 20 --message-system-attribute-names ApproximateReceiveCount MessageGroupId SequenceNumber
# 6. Extend visibility of a received message (paste its receipt handle)
# aws sqs change-message-visibility --queue-url "$LEDGER_URL" --receipt-handle "<handle>" --visibility-timeout 120
# 7. Inspect depth (approximate)
aws sqs get-queue-attributes --queue-url "$NOTIFY_URL" \
--attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible
# 8. Turn on high-throughput FIFO later (both attributes are required)
# aws sqs set-queue-attributes --queue-url "$LEDGER_URL" \
# --attributes DeduplicationScope=messageGroup,FifoThroughputLimit=perMessageGroupIdNotes:
- Queue attributes are strings in the API, even numbers — hence the quotes in
notify-attrs.json. RedrivePolicyis a JSON string inside JSON. The file approach avoids shell-quoting pain on Windows.--message-system-attribute-namesreplaced the older--attribute-namesfor receives. If your CLI rejects it, update AWS CLI v2.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.SQS;
using Constructs;
namespace Harbourline.Queues.Infra;
public sealed class QueuesStack : Stack
{
public QueuesStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
var dlq = new Queue(this, "NotifyDlq", new QueueProps
{
QueueName = "hl-m08-l2-notify-dlq",
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
RetentionPeriod = Duration.Days(14),
});
var notify = new Queue(this, "Notify", new QueueProps
{
QueueName = "hl-m08-l2-notify",
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
VisibilityTimeout = Duration.Seconds(60),
ReceiveMessageWaitTime = Duration.Seconds(20),
RetentionPeriod = Duration.Days(4),
DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 },
});
var ledger = new Queue(this, "Ledger", new QueueProps
{
QueueName = "hl-m08-l2-ledger.fifo",
Fifo = true,
ContentBasedDeduplication = false,
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
VisibilityTimeout = Duration.Seconds(60),
ReceiveMessageWaitTime = Duration.Seconds(20),
// High-throughput FIFO: uncomment both
// DeduplicationScope = DeduplicationScope.MESSAGE_GROUP,
// FifoThroughputLimit = FifoThroughputLimit.PER_MESSAGE_GROUP_ID,
});
// Alarm when the oldest message is more than 5 minutes old: consumers are behind or down.
new Alarm(this, "NotifyAgeAlarm", new AlarmProps
{
Metric = notify.MetricApproximateAgeOfOldestMessage(new MetricOptions { Period = Duration.Minutes(1) }),
Threshold = 300,
EvaluationPeriods = 5,
ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING,
});
foreach (var (k, v) in new[] { ("Project", "cea-lab"), ("Lab", "m08-l2"), ("Environment", "learn"), ("Owner", "you") })
Amazon.CDK.Tags.Of(this).Add(k, v);
_ = ledger;
}
}Complete code
A .NET 10 Worker Service with four modes. It works against real SQS (using your SSO profile) or ElasticMQ (set HL_SQS_SERVICE_URL). The module project in labs/m08-notification-system/ contains a tested, in-memory SQS simulator if you want to explore the same behaviour in unit tests.
<Project Sdk="Microsoft.NET.Sdk.Worker">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<RootNamespace>Harbourline.Queues.Worker</RootNamespace>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AWSSDK.SQS" Version="4.1.0.2" />
<PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="4.1.0.2" />
<PackageReference Include="Microsoft.Extensions.Hosting" Version="10.0.15" />
</ItemGroup>
</Project>using Amazon;
using Amazon.SQS;
using Harbourline.Queues.Worker;
var mode = args.FirstOrDefault() ?? "consume-standard";
var builder = Host.CreateApplicationBuilder(args);
builder.Logging.AddJsonConsole(o => o.IncludeScopes = true);
// Queue URLs come from configuration (environment variables here).
builder.Services.Configure<QueueOptions>(o =>
{
o.StandardQueueUrl = Environment.GetEnvironmentVariable("HL_NOTIFY_QUEUE_URL") ?? "";
o.FifoQueueUrl = Environment.GetEnvironmentVariable("HL_LEDGER_QUEUE_URL") ?? "";
o.UseHeartbeat = Environment.GetEnvironmentVariable("HL_DISABLE_HEARTBEAT") != "true";
});
// Real SQS by default; ElasticMQ when HL_SQS_SERVICE_URL is set.
var serviceUrl = Environment.GetEnvironmentVariable("HL_SQS_SERVICE_URL");
if (string.IsNullOrEmpty(serviceUrl))
{
builder.Services.AddDefaultAWSOptions(new Amazon.Extensions.NETCore.Setup.AWSOptions
{
Region = RegionEndpoint.USEast1,
});
builder.Services.AddAWSService<IAmazonSQS>();
}
else
{
builder.Services.AddSingleton<IAmazonSQS>(_ => new AmazonSQSClient(
new Amazon.Runtime.BasicAWSCredentials("local", "local"), // ElasticMQ ignores credentials
new AmazonSQSConfig { ServiceURL = serviceUrl, AuthenticationRegion = "us-east-1" }));
}
builder.Services.AddSingleton<IMessageHandler, NotificationHandler>();
builder.Services.AddSingleton<Producer>();
switch (mode)
{
case "send-standard":
case "send-fifo":
using (var host = builder.Build())
{
var producer = host.Services.GetRequiredService<Producer>();
await (mode == "send-fifo" ? producer.SendFifoAsync() : producer.SendStandardAsync());
}
return;
case "consume-standard":
case "consume-fifo":
builder.Services.AddSingleton(new ConsumerSettings(Fifo: mode == "consume-fifo"));
builder.Services.AddHostedService<SqsConsumer>();
await builder.Build().RunAsync();
return;
default:
Console.Error.WriteLine("Modes: send-standard | send-fifo | consume-standard | consume-fifo");
return;
}namespace Harbourline.Queues.Worker;
public sealed class QueueOptions
{
public string StandardQueueUrl { get; set; } = "";
public string FifoQueueUrl { get; set; } = "";
public bool UseHeartbeat { get; set; } = true;
// Must match the queue's VisibilityTimeout. The heartbeat extends by this much.
public int VisibilityTimeoutSeconds { get; set; } = 60;
// Stop extending after this long; let the message go back for another attempt.
public TimeSpan MaxProcessingTime { get; set; } = TimeSpan.FromMinutes(15);
}
public sealed record ConsumerSettings(bool Fifo);
public sealed record PostingMessage(
Guid TransactionId,
string AccountId,
decimal Amount,
string Currency,
string Kind,
DateTimeOffset OccurredAt);using System.Text.Json;
using Amazon.SQS;
using Amazon.SQS.Model;
using Microsoft.Extensions.Options;
namespace Harbourline.Queues.Worker;
public sealed class Producer(IAmazonSQS sqs, IOptions<QueueOptions> options, ILogger<Producer> log)
{
private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
// Deterministic sample data: same IDs on every run, so a re-run inside 5 minutes is deduplicated.
private static IReadOnlyList<PostingMessage> SamplePostings()
{
var baseTime = new DateTimeOffset(2026, 11, 27, 1, 0, 0, TimeSpan.Zero);
return
[
new(Guid.Parse("01937a4e-0000-7000-8000-000000000001"), "ACC-0000123456", 2450.00m, "GBP", "normal", baseTime),
new(Guid.Parse("01937a4e-0000-7000-8000-000000000002"), "ACC-0000123456", -35.20m, "GBP", "normal", baseTime.AddSeconds(5)),
new(Guid.Parse("01937a4e-0000-7000-8000-000000000003"), "ACC-0000999999", -120.00m, "EUR", "normal", baseTime.AddSeconds(6)),
new(Guid.Parse("01937a4e-0000-7000-8000-000000000004"), "ACC-0000123456", -9.99m, "GBP", "slow", baseTime.AddSeconds(9)),
new(Guid.Parse("01937a4e-0000-7000-8000-000000000005"), "ACC-0000999999", 60.00m, "EUR", "normal", baseTime.AddSeconds(12)),
];
}
public async Task SendFifoAsync(CancellationToken ct = default)
{
var entries = SamplePostings().Select((p, i) => new SendMessageBatchRequestEntry
{
Id = $"m{i}", // unique within this batch only
MessageBody = JsonSerializer.Serialize(p, Json),
MessageGroupId = p.AccountId, // order per account
MessageDeduplicationId = p.TransactionId.ToString(), // business key: retries are safe
}).ToList();
await SendBatchAsync(options.Value.FifoQueueUrl, entries, ct);
}
public async Task SendStandardAsync(CancellationToken ct = default)
{
var entries = SamplePostings().Select((p, i) => new SendMessageBatchRequestEntry
{
Id = $"m{i}",
MessageBody = JsonSerializer.Serialize(p, Json),
MessageAttributes = new Dictionary<string, MessageAttributeValue>
{
["eventType"] = new() { DataType = "String", StringValue = "TransactionPosted" },
},
}).ToList();
await SendBatchAsync(options.Value.StandardQueueUrl, entries, ct);
}
private async Task SendBatchAsync(string queueUrl, List<SendMessageBatchRequestEntry> entries, CancellationToken ct)
{
// SendMessageBatch accepts at most 10 entries; chunk larger lists.
foreach (var chunk in entries.Chunk(10))
{
var response = await sqs.SendMessageBatchAsync(new SendMessageBatchRequest
{
QueueUrl = queueUrl,
Entries = [.. chunk],
}, ct);
// SDK v4: collections are null when empty.
foreach (var ok in response.Successful ?? [])
log.LogInformation("Sent {EntryId} as {MessageId} seq {Sequence}", ok.Id, ok.MessageId, ok.SequenceNumber);
var failed = response.Failed ?? [];
foreach (var f in failed)
log.LogError("Send failed {EntryId}: {Code} {Message} (sender fault: {SenderFault})", f.Id, f.Code, f.Message, f.SenderFault);
if (failed.Count > 0)
throw new InvalidOperationException($"{failed.Count} message(s) were not sent; retry those entries with the same deduplication IDs.");
}
}
}namespace Harbourline.Queues.Worker;
public interface IMessageHandler
{
Task HandleAsync(PostingMessage message, CancellationToken ct);
}
// Stand-in for real work (calling an SMS provider). "slow" messages take 150 s,
// longer than the 60 s visibility timeout, to demonstrate the heartbeat.
public sealed class NotificationHandler(ILogger<NotificationHandler> log) : IMessageHandler
{
public async Task HandleAsync(PostingMessage message, CancellationToken ct)
{
var duration = message.Kind == "slow" ? TimeSpan.FromSeconds(150) : TimeSpan.FromMilliseconds(200);
log.LogInformation("Processing {TransactionId} for {AccountId} ({Amount} {Currency}), expected {Duration}",
message.TransactionId, message.AccountId, message.Amount, message.Currency, duration);
await Task.Delay(duration, ct);
}
}using System.Text.Json;
using Amazon.SQS;
using Amazon.SQS.Model;
using Microsoft.Extensions.Options;
namespace Harbourline.Queues.Worker;
public sealed class SqsConsumer(
IAmazonSQS sqs,
IMessageHandler handler,
IOptions<QueueOptions> options,
ConsumerSettings settings,
ILogger<SqsConsumer> log) : BackgroundService
{
private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
private readonly QueueOptions _opt = options.Value;
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
var queueUrl = settings.Fifo ? _opt.FifoQueueUrl : _opt.StandardQueueUrl;
log.LogInformation("Consuming {QueueUrl} (heartbeat {Heartbeat})", queueUrl, _opt.UseHeartbeat);
while (!stoppingToken.IsCancellationRequested)
{
ReceiveMessageResponse response;
try
{
response = await sqs.ReceiveMessageAsync(new ReceiveMessageRequest
{
QueueUrl = queueUrl,
MaxNumberOfMessages = 10, // batch receive
WaitTimeSeconds = 20, // long polling
VisibilityTimeout = _opt.VisibilityTimeoutSeconds,
MessageSystemAttributeNames = ["ApproximateReceiveCount", "MessageGroupId", "SequenceNumber"],
MessageAttributeNames = ["All"],
}, stoppingToken);
}
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
{
break; // shutting down
}
catch (AmazonSQSException ex)
{
// Throttling or a transient service error the SDK's own retries did not absorb.
log.LogWarning(ex, "Receive failed; backing off");
await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
continue;
}
var messages = response.Messages ?? [];
if (messages.Count == 0) continue; // long poll ended empty: just poll again
var succeeded = settings.Fifo
? await ProcessInOrderAsync(queueUrl, messages, stoppingToken)
: await ProcessInParallelAsync(queueUrl, messages, stoppingToken);
await DeleteBatchAsync(queueUrl, succeeded, stoppingToken);
}
}
// Standard queue: order does not matter, so process the batch concurrently.
private async Task<List<Message>> ProcessInParallelAsync(string queueUrl, List<Message> messages, CancellationToken ct)
{
var results = await Task.WhenAll(messages.Select(m => ProcessOneAsync(queueUrl, m, ct)));
return messages.Where((_, i) => results[i]).ToList();
}
// FIFO queue: messages in one group must be handled in sequence. Different groups may run
// in parallel. If one message in a group fails, skip the rest of that group in this batch so
// that later messages are not processed ahead of it.
private async Task<List<Message>> ProcessInOrderAsync(string queueUrl, List<Message> messages, CancellationToken ct)
{
var groups = messages.GroupBy(m => m.Attributes?.GetValueOrDefault("MessageGroupId") ?? "");
var perGroup = await Task.WhenAll(groups.Select(async g =>
{
var done = new List<Message>();
foreach (var m in g)
{
if (!await ProcessOneAsync(queueUrl, m, ct)) break;
done.Add(m);
}
return done;
}));
return perGroup.SelectMany(x => x).ToList();
}
private async Task<bool> ProcessOneAsync(string queueUrl, Message message, CancellationToken ct)
{
var receiveCount = message.Attributes?.GetValueOrDefault("ApproximateReceiveCount") ?? "?";
using var scope = log.BeginScope(new Dictionary<string, object>
{
["MessageId"] = message.MessageId,
["ReceiveCount"] = receiveCount,
});
using var workCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
workCts.CancelAfter(_opt.MaxProcessingTime); // never hold a message forever
var heartbeat = _opt.UseHeartbeat
? RunHeartbeatAsync(queueUrl, message.ReceiptHandle, workCts.Token)
: Task.CompletedTask;
try
{
var posting = JsonSerializer.Deserialize<PostingMessage>(message.Body, Json)
?? throw new JsonException("Empty body");
await handler.HandleAsync(posting, workCts.Token);
return true;
}
catch (JsonException ex)
{
// A malformed message will never succeed. Leave it; after maxReceiveCount
// the redrive policy moves it to the DLQ (next lesson).
log.LogError(ex, "Poison message (receive count {ReceiveCount})", receiveCount);
return false;
}
catch (Exception ex) when (ex is not OperationCanceledException || !ct.IsCancellationRequested)
{
log.LogError(ex, "Processing failed; message will reappear after the visibility timeout");
return false;
}
finally
{
await workCts.CancelAsync(); // stop the heartbeat
await heartbeat;
}
}
// Extends visibility at half the timeout while work is running.
private async Task RunHeartbeatAsync(string queueUrl, string receiptHandle, CancellationToken ct)
{
var interval = TimeSpan.FromSeconds(_opt.VisibilityTimeoutSeconds / 2.0);
using var timer = new PeriodicTimer(interval);
try
{
while (await timer.WaitForNextTickAsync(ct))
{
await sqs.ChangeMessageVisibilityAsync(new ChangeMessageVisibilityRequest
{
QueueUrl = queueUrl,
ReceiptHandle = receiptHandle,
VisibilityTimeout = _opt.VisibilityTimeoutSeconds, // counted from now
}, ct);
log.LogInformation("Heartbeat: visibility extended by {Seconds}s", _opt.VisibilityTimeoutSeconds);
}
}
catch (OperationCanceledException)
{
// Normal: processing finished or shutdown requested.
}
catch (AmazonSQSException ex)
{
// Receipt handle expired or message already deleted. Another consumer may now own it.
log.LogWarning(ex, "Heartbeat failed; another consumer may process this message");
}
}
private async Task DeleteBatchAsync(string queueUrl, List<Message> succeeded, CancellationToken ct)
{
if (succeeded.Count == 0) return;
var response = await sqs.DeleteMessageBatchAsync(new DeleteMessageBatchRequest
{
QueueUrl = queueUrl,
Entries = succeeded.Select((m, i) => new DeleteMessageBatchRequestEntry
{
Id = i.ToString(),
ReceiptHandle = m.ReceiptHandle,
}).ToList(),
}, CancellationToken.None); // finish deletes even during shutdown
foreach (var f in response.Failed ?? [])
log.LogWarning("Delete failed for entry {Id}: {Code} {Message}. The message may be redelivered.", f.Id, f.Code, f.Message);
log.LogInformation("Deleted {Count} message(s)", (response.Successful ?? []).Count);
}
}Run against real SQS:
aws sso login --profile learn-sandbox
export AWS_PROFILE=learn-sandbox
export HL_NOTIFY_QUEUE_URL=$(aws sqs get-queue-url --queue-name hl-m08-l2-notify --query QueueUrl --output text)
export HL_LEDGER_QUEUE_URL=$(aws sqs get-queue-url --queue-name hl-m08-l2-ledger.fifo --query QueueUrl --output text)
dotnet run -- send-fifo && dotnet run -- consume-fifoExpected output (trimmed, IDs differ): postings for ACC-0000123456 print in the order 2450.00, -35.20, -9.99; postings for ACC-0000999999 print -120.00 then 60.00; the two accounts interleave. During the slow posting you see Heartbeat: visibility extended by 60s twice or more before Deleted 1 message(s).
Code explained line by line
Program.cs
Host.CreateApplicationBuilder(args)— the .NET generic host: dependency injection, configuration, logging and graceful shutdown on Ctrl+C or SIGTERM (which ECS sends when stopping a task).AddJsonConsole— structured JSON logs, as the conventions require; scopes carryMessageIdandReceiveCounton every line.AddDefaultAWSOptions+AddAWSService<IAmazonSQS>()— registers a singleton SQS client using the default credential chain (your SSO profile locally, the task role in ECS). SDK clients are thread-safe; create one and reuse it.- The
elsebranch builds a client for ElasticMQ withServiceURLand dummy credentials.AuthenticationRegionkeeps request signing happy. - The
switcheither runs a one-shot producer or registers the consumer as a hosted service and runs until stopped.
Producer.cs
SamplePostings()uses fixed GUIDs so that a second run produces identical deduplication IDs.Id = $"m{i}"— the batch entry ID only correlates results within one call; it is not stored.MessageGroupId = p.AccountId— ordering per account, parallelism across accounts.MessageDeduplicationId = p.TransactionId.ToString()— a business key. If this call times out and the code retries, SQS drops the duplicate. A random GUID here would defeat deduplication.entries.Chunk(10)— the batch maximum is 10 entries.response.Failed ?? []— partial failure is normal; SDK v4 returnsnullinstead of an empty list. Throwing makes the failure visible; a production producer retries just those entries with backoff, reusing their deduplication IDs.MessageAttributeson the standard send — metadata outside the body (hereeventType) that consumers can read without parsing JSON, and that SNS filter policies can match.
SqsConsumer.cs
BackgroundService.ExecuteAsync— the polling loop, withstoppingTokensignalled on shutdown.MaxNumberOfMessages = 10,WaitTimeSeconds = 20— batch receive plus long polling: the cheapest and lowest-latency combination.VisibilityTimeout = ...on the request — overrides the queue default for these messages only. Keeping it equal to the queue value avoids surprises.MessageSystemAttributeNames— asks SQS to returnApproximateReceiveCount,MessageGroupIdandSequenceNumber. Without this,message.Attributesisnull.catch (AmazonSQSException)— the SDK already retries throttling and 5xx errors (standard retry mode is the v4 default); anything that escapes gets a short pause so the loop does not spin.ProcessInParallelAsync— standard queues give no ordering, so the batch runs concurrently.Task.WhenAllkeeps results in input order, soresults[i]lines up withmessages[i].ProcessInOrderAsync— for FIFO, group byMessageGroupId, process each group's messages sequentially, and stop at the first failure in a group so a later message never overtakes an earlier failed one. The failed message and those after it in the batch are not deleted, become visible again after the timeout, and come back in order.CancelAfter(_opt.MaxProcessingTime)— a hard ceiling. Without it, a hung handler plus a heartbeat would hide a message until the 12-hour cap.RunHeartbeatAsync— aPeriodicTimerat half the visibility timeout callsChangeMessageVisibilitywith the full timeout. Because the new value counts from now, the message always has at least half a timeout of headroom.- Heartbeat
catch (AmazonSQSException)— if the receipt handle is no longer valid, someone else may now be processing the message. Logging this is important: it tells you idempotency is about to be tested. catch (JsonException)— a poison message. The consumer deliberately does not delete it; the queue's redrive policy will move it to the DLQ after five receives. The dead-letter queues lesson explores this.finally { await workCts.CancelAsync(); await heartbeat; }— always stop the heartbeat, even on failure, so a failed message is released at its normal time.DeleteBatchAsync— deletes only successfully processed messages, in one call, using the receipt handles from this receive.CancellationToken.Noneso a shutdown signal cannot abandon deletes for work that has already been done (which would cause redelivery).response.Failedon delete — a failed delete means the message will be redelivered. This is a second reason consumers must be idempotent.
What this code intentionally leaves out: idempotency (lesson 11 and the module project add a DynamoDB idempotency table), retries with jitter inside the handler (lesson 9), and DLQ redrive (next lesson). AWS.Messaging (lesson 15) packages this loop, the heartbeat and the deletes into a framework.
Security considerations
⚠ Learning shortcut — not for production
The lab queues use SSE-SQS (keys owned and managed by SQS). For queues that carry anything linked to a customer, use SSE-KMS with a customer managed key: key use is logged in CloudTrail, the key policy is a second authorisation layer, and disabling the key makes the messages unreadable. Remember that SSE encrypts only the body — message attributes, queue names and metrics are not encrypted, so never put personal data in attributes.
⚠ Learning shortcut — not for production
You run the worker with your SSO identity, which can read every queue in the sandbox. In ECS the task role should allow only sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility and sqs:GetQueueAttributes on arn:aws:sqs:us-east-1:<account>:hl-m08-l2-notify. The producer's role gets only sqs:SendMessage on the same ARN.
- Queue policies. Deny
aws:SecureTransport = false. Allow cross-service senders (SNS, EventBridge) only withaws:SourceArnconditions. Avoid"Principal": "*"without conditions — a public queue is a real and recurring finding. - Private connectivity. Use an interface VPC endpoint for SQS and a queue policy condition on
aws:SourceVpcefor queues that only internal workloads should reach. - Retention is a data-protection setting. Messages live up to 14 days; DLQs often keep them longest. Agree retention with the data-protection team and document it.
- Logging bodies. Do not log whole message bodies in production; log IDs and correlation IDs. The sample logs amounts only because the data is fictional.
- Purge is destructive.
PurgeQueuedeletes everything with no undo. Restrictsqs:PurgeQueueandsqs:DeleteQueueto break-glass roles.
Banking use case
Harbourline's ledger-update pipeline uses one FIFO queue per region, hl-ledger-postings.fifo:
- Group ID = account ID. Postings to one account are applied in order, so the running balance in each notification is correct (incident 3 solved). Thousands of accounts proceed in parallel.
- Deduplication ID = transaction ID. The core banking adapter retries sends freely after network errors; SQS drops duplicate sends within five minutes. The consumer still checks a DynamoDB idempotency table because a crash between "apply posting" and "delete message" would redeliver.
- High-throughput FIFO is on, because salary night peaks at several thousand postings a second.
- Visibility timeout 120 seconds with a heartbeat,
maxReceiveCount5, DLQ with 14-day retention and a CloudWatch alarm on any message in the DLQ. - The Notifications queue, by contrast, is standard: texts are independent, ordering is not needed once the balance is removed from the text, and throughput is effectively unlimited. Incident 1 was fixed with the heartbeat plus an idempotency check, and incident 2 with 20-second long polling.
Common mistakes
- Visibility timeout shorter than processing time. Causes duplicates under load. → Measure p99 processing time; use a heartbeat for variable work; for Lambda use at least 6 × function timeout.
- Short polling in a loop. Expensive and slower. →
WaitTimeSeconds = 20on the queue and the request. - Deleting before processing. "Receive, delete, then process" loses the message if the process crashes. → Delete only after successful processing.
- Using a random GUID as the FIFO deduplication ID. Retries then create duplicates. → Use a business key.
- One message group for the whole FIFO queue. Throughput collapses and one poison message blocks everything. → Group by the entity whose order matters.
- Ignoring
Failedin batch responses. Messages silently not sent or not deleted. → Check and retry or alert. - Treating
ApproximateNumberOfMessagesas exact. → Use it for trends and scaling, not reconciliation. - Expecting SQS to filter by content. → Filter upstream and use separate queues.
- Forgetting SDK v4 nulls.
response.Messages.Countthrows when the queue is empty. →response.Messages ?? [].
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Same message processed twice | Visibility timeout shorter than processing; failed delete; standard-queue duplicate | Heartbeat or longer timeout; check DeleteMessageBatch failures; idempotent consumer |
ReceiptHandleIsInvalid on delete | Using a handle from an earlier receive, or after the message was received again | Always use the handle from the latest receive; finish within the visibility timeout |
| Message deleted but processed again later | It had been received again (new handle) before your delete with the old handle | Same as above; extend visibility while working |
| FIFO queue appears stuck; messages visible but not delivered | Head-of-line message in each group is in flight or keeps failing | Find it via the DLQ or logs; ensure a redrive policy; check consumers are deleting |
MissingParameter: MessageGroupId | Sending to a FIFO queue without a group ID | Set MessageGroupId on every FIFO send |
InvalidParameterValue about deduplication | FIFO queue without content-based dedup and no MessageDeduplicationId | Send an explicit ID or enable content-based deduplication |
OverLimit on receive | About 120,000 messages in flight (short polling) | Delete faster; shorter visibility; more consumers or queues |
High bill, NumberOfEmptyReceives very high | Short polling | Enable long polling |
KMS.AccessDeniedException | Producer/consumer role lacks kms:GenerateDataKey / kms:Decrypt on the queue key | Update the role and the key policy |
NullReferenceException on response.Messages | SDK v4 null collections | Null-coalesce |
| Lambda keeps retrying a whole batch | One bad message, no partial batch response | Enable ReportBatchItemFailures and return failed IDs |
Cost and cleanup
Cost: free at lab volumes. SQS includes 1 million requests per month free, permanently, across standard and FIFO. Beyond that, standard queues cost about $0.40 per million requests and FIFO about $0.50 per million in us-east-1 ⚠ VERIFY on the pricing page. Each 64 KB of payload is one request; a batch of 10 small messages is one request. Data transfer within the Region is free. KMS adds a per-request charge when you use SSE-KMS (mitigated by the data key reuse period, default 5 minutes). This lesson's exercises use a few hundred requests.
The free alternative is ElasticMQ in Docker, at no cost.
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
for q in hl-m08-l2-notify hl-m08-l2-notify-dlq hl-m08-l2-ledger.fifo; do
url=$(aws sqs get-queue-url --queue-name "$q" --query QueueUrl --output text 2>/dev/null) \
&& aws sqs delete-queue --queue-url "$url" && echo "deleted $q"
done
rm -f notify-attrs.json
# If you deployed the CDK or Terraform version, destroy it instead:
# cdk destroy | terraform destroy
# Confirm nothing tagged for this lesson remains (may lag a few minutes)
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m08-l2 \
--query 'ResourceTagMappingList[].ResourceARN'
# Local option
docker stop hl-elasticmqAfter deleting a queue, wait 60 seconds before creating one with the same name.
Exercises
- 🟢 Guided: Run
send-fifotwice within five minutes and confirm the queue holds 5 messages, not 10. Then wait more than five minutes and run it a third time. Record what happened and explain why. - 🟡 Challenge: Add a
prioritymode: createhl-m08-l2-notify-highand change the consumer so that it drains the high-priority queue first (short wait) and only long-polls the normal queue when the high queue is empty. Keep the empty-receive count low. - 🔴 Live problem: Ticket from operations at 02:10 on salary night: "
hl-ledger-postings.fifo—ApproximateNumberOfMessagesVisibleis 180,000 and climbing, consumers are at 5% CPU,ApproximateAgeOfOldestMessageis 40 minutes, no errors in the logs. Customers complain balances aren't updating." Diagnose the likely causes and propose an immediate mitigation and a lasting fix.
Assignment
Add an sqs-design.md to your portfolio repository describing one queue you would build for Harbourline (your choice of flow): standard or FIFO and why, group and deduplication keys, visibility timeout with the arithmetic behind it, polling and batch settings, DLQ and maxReceiveCount, encryption choice, IAM policies for producer and consumer, the two CloudWatch alarms you would create, and a monthly cost estimate at 10 million messages.
Solutions
First and second runs: the second run's SendMessageBatch calls succeed (you even get MessageIds back), but SQS recognises each MessageDeduplicationId within the 5-minute deduplication interval and does not enqueue new copies. ApproximateNumberOfMessages stays at 5 (if you have not consumed them).
Third run after five minutes: the interval has passed, so the same IDs are accepted as new messages and the queue holds 10. Why it matters: FIFO deduplication protects against short-term producer retries, not against replays hours later. A batch job that is re-run the next morning will create duplicates; only an idempotent consumer catches those.
Change the loop to try the high queue first with a short wait, then the normal queue with a long wait:
var high = await sqs.ReceiveMessageAsync(new ReceiveMessageRequest
{
QueueUrl = _highQueueUrl,
MaxNumberOfMessages = 10,
WaitTimeSeconds = 1, // short: just check
}, stoppingToken);
var batch = high.Messages ?? [];
var sourceUrl = _highQueueUrl;
if (batch.Count == 0)
{
var normal = await sqs.ReceiveMessageAsync(new ReceiveMessageRequest
{
QueueUrl = _normalQueueUrl,
MaxNumberOfMessages = 10,
WaitTimeSeconds = 10, // long, but not 20, so high-priority work is noticed quickly
}, stoppingToken);
batch = normal.Messages ?? [];
sourceUrl = _normalQueueUrl;
}
// process batch, then delete from sourceUrlWhy: SQS has no priority within a queue, so priority comes from separate queues. The trade-off is latency for high-priority messages (up to the normal queue's wait time) against empty receives on the high queue. A cleaner alternative at scale is separate consumer pools per queue, so a flood of normal work cannot slow high-priority work at all. Remember to delete with the receipt handle from the queue the message came from.
Read the signals. Huge visible backlog, old messages, idle consumers and no errors means consumers are not receiving work they could do. On a FIFO queue the usual cause is message-group blocking:
- Too few groups. If the adapter recently changed the group ID to something coarse (for example the sort code, or a constant), almost all messages share a handful of groups, and FIFO delivers only one in-flight message per group. Check by sampling
MessageGroupIdvalues. - Head-of-line in-flight messages. A message per hot group is in flight with a long visibility timeout and a hung handler (heartbeat keeps extending it). Check
ApproximateNumberOfMessagesNotVisibleand look for heartbeats in the logs without completion. - Throughput limit. If high-throughput mode is off, 300 API calls per second per action caps the queue. Check for throttling in SDK metrics.
Immediate mitigation. Fix whichever cause applies: restart hung consumers (their in-flight messages return after the timeout) and lower MaxProcessingTime; enable high-throughput FIFO (DeduplicationScope=messageGroup, FifoThroughputLimit=perMessageGroupId); if the group ID regressed, deploy the fix — but new messages only. Communicate balance-update delays to customer services.
Lasting fix. Group by account ID, keep high-throughput mode on, cap processing time, alarm on ApproximateAgeOfOldestMessage (here it would have fired at 5 minutes, not 40), and add a dashboard of in-flight vs visible counts.
Knowledge check
Answer: Nothing is lost. When the visibility timeout expires, the message becomes visible again with its receive count increased, and another consumer receives it. After maxReceiveCount receives, the redrive policy moves it to the DLQ.
Answer: It sets the remaining visibility to 60 seconds from now — it does not add 60 to the 10 remaining. (Setting it to 0 releases the message immediately.)
Answer: No. The deduplication interval is 5 minutes. After that, the same ID is accepted as a new message.
Answer: All messages fall into one group, so only one can be in flight at a time. Throughput collapses to one message per processing time, and a single failing message blocks the whole queue until it is deleted or moved to the DLQ.
Answer: 1 MiB (1,048,576 bytes). Each 64 KB chunk counts as one request, so a 1 MiB message costs 16 requests.
Answer: At least 6 × 30 + 10 = 190 seconds, following AWS's guidance, so that throttled or retried invocations do not let messages reappear while still being processed.
Interview questions
- "How does SQS guarantee that a message is not lost if a consumer dies?" — Receiving does not delete. The message is hidden for the visibility timeout; if it is not deleted with the receipt handle in that time, it reappears for another consumer. Redrive to a DLQ stops infinite retries.
- "Standard or FIFO for ledger postings?" — FIFO with the account ID as message group, so postings to one account apply in order while accounts run in parallel; transaction ID as deduplication ID; high-throughput mode for peaks. And still an idempotent consumer, because FIFO dedup only covers sends within five minutes.
- "How do you handle processing that sometimes takes longer than the visibility timeout?" — A heartbeat that calls
ChangeMessageVisibilityperiodically while work runs, with an overall processing cap. For Lambda, where you cannot heartbeat, set visibility to at least six times the function timeout. - "How would you reduce an SQS bill?" — Long polling (20 seconds), batching sends and deletes, smaller payloads (each 64 KB is a request), fewer idle pollers, and Lambda event source mappings for spiky workloads.
- "What changed in SDK for .NET v4 that bites SQS code?" — Collections in responses default to
nulland value types are nullable, soresponse.Messagesmust be null-checked; and standard retry mode is the default.
Certification practice
An application on Amazon ECS consumes an Amazon SQS standard queue. Processing usually takes 10 seconds but sometimes takes up to 90 seconds. The queue's visibility timeout is 30 seconds. Customers occasionally receive two notifications. What is the MOST effective fix that does not delay redelivery of messages from crashed consumers by more than about a minute?
A. Increase the visibility timeout to 12 hours.
B. Convert the queue to FIFO.
C. Keep a 60-second visibility timeout and have the consumer call ChangeMessageVisibility periodically while processing.
D. Reduce MaxNumberOfMessages to 1.
Answer: C. A heartbeat keeps long-running messages hidden while a crashed consumer's messages reappear within about a minute. A hides messages from crashed consumers for hours. B does not stop redelivery after visibility expiry. D reduces batch size but not processing time.
A payment producer sends messages to an Amazon SQS FIFO queue. When the network times out, the producer retries the send within seconds. Content-based deduplication is disabled. How should the developer prevent duplicate messages in the queue?
A. Generate a new MessageDeduplicationId for each retry.
B. Use the payment ID as the MessageDeduplicationId for the original send and all retries.
C. Use a single MessageGroupId for all messages.
D. Set DelaySeconds to 300.
Answer: B. SQS drops sends with a deduplication ID it has seen in the last 5 minutes. A defeats deduplication. C affects ordering, not deduplication, and hurts throughput. D delays delivery and is not supported per message on FIFO queues.
A company's SQS consumers run continuously. The monthly bill shows millions of requests, and CloudWatch shows NumberOfEmptyReceives far exceeding NumberOfMessagesReceived. Which change reduces cost MOST without increasing message latency significantly?
A. Enable long polling by setting the receive wait time to 20 seconds. B. Increase the message retention period. C. Switch to a FIFO queue. D. Enable server-side encryption.
Answer: A. Long polling waits for messages instead of returning empty responses immediately, cutting billed empty receives while returning messages as soon as they arrive. B, C and D do not reduce empty receives (C costs more per request).
A bank must process account postings in order per account. Volume peaks at 20,000 messages per second across 2 million accounts. Which design meets the requirement?
A. SQS standard queue with a sequence number in each message. B. SQS FIFO queue with high-throughput mode enabled and the account ID as the message group ID. C. SQS FIFO queue with a single message group ID. D. Amazon SNS standard topic with SQS subscribers.
Answer: B. Per-account groups preserve order where it matters and spread load across partitions; high-throughput mode lifts the per-partition limits. A pushes ordering into consumers and still allows concurrent out-of-order delivery. C serialises everything. D gives no ordering.
Completion checklist
- I can draw the message lifecycle and explain the receipt handle
- I can choose standard or FIFO and pick a message group and deduplication ID
- I can calculate a visibility timeout, including for Lambda
- I ran the consumer and saw the heartbeat keep a slow message hidden
- I saw FIFO deduplication drop a repeated send
- I deleted the queues (or stopped ElasticMQ) and the tag check returned
[]
Key takeaways
- Receiving hides a message; only
DeleteMessagewith the latest receipt handle removes it. The visibility timeout is the safety net and the main source of duplicates. - FIFO gives ordering per message group and send-side deduplication for 5 minutes; choose the group key carefully and still make consumers idempotent.
- Always long-poll (20 seconds), batch up to 10, check
Failedin batch responses, and alarm onApproximateAgeOfOldestMessage. - Use a heartbeat (
ChangeMessageVisibility) for variable-length work in your own consumers; for Lambda, set visibility to at least 6 × the function timeout. - Messages can now be 1 MiB, billed per 64 KB; SDK v4 returns
nullcollections — code for both.
Further reading
- Amazon SQS message quotas
- Amazon SQS FIFO queue quotas
- Amazon SQS visibility timeout
- Amazon SQS short and long polling
- Exactly-once processing in Amazon SQS
- Using the message group ID with Amazon SQS FIFO queues
- Enabling high throughput for FIFO queues
- Amazon SQS fair queues
- Encryption at rest in Amazon SQS
- Creating and configuring an Amazon SQS event source mapping (Lambda)
- Handling errors for an SQS event source in Lambda
- Amazon SQS pricing
- Next: Dead-letter queues and redrive
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Asynchronous communication and event-driven architecture
Tell queues, topics, event buses, streams and workflows apart, decide which AWS service fits a Harbourline integration with a written decision tree, and publish your first well-formed domain event from .NET 10.
Dead-letter queues and redrive
Stop one bad message from blocking a payment run: configure SQS dead-letter queues with a sensible maxReceiveCount, use partial batch responses in .NET Lambda handlers, alarm on DLQ depth, and redrive messages safely after a fix.