Playbook: SQS messages stuck or redelivered
Diagnose SQS messages that never finish, keep reappearing or end up in a dead-letter queue by reading visibility timeout, receive count, consumer errors and FIFO group blocking, then inspect and redrive safely with idempotent handlers.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SQS) · AWS CLI v2 · Amazon SQS standard and FIFO · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this playbook you can:
- Explain, step by step, what happens to a message from
SendMessagetoDeleteMessage, including what the visibility timeout and receive count do. - Tell apart the four symptoms: messages not arriving, stuck invisible, redelivered repeatedly and landing in the dead-letter queue (DLQ).
- Read the queue's CloudWatch metrics and attributes to identify the symptom in under five minutes.
- Fix the classic causes: a visibility timeout shorter than the work, a Lambda concurrency or batch mismatch, a poison message, and a blocked FIFO message group.
- Inspect a DLQ without destroying evidence and redrive messages at a controlled rate.
- Run a simulation that reproduces duplicate processing and the three fixes, with no AWS account.
Prerequisites
- Amazon SQS in depth
- Dead-letter queues and redrive and Delivery guarantees and ordering
- Idempotency in .NET and Retry strategies
- Duplicate payment on client retry, the sibling playbook for the HTTP side of the same problem
- Tools: AWS CLI v2 with the
learn-sandboxprofile; .NET 10 SDK for the simulation
Why this matters in banking
A queue named hl-payment-instructions feeds the worker that submits payments to the clearing partner. At 15:20 the operations dashboard shows the queue growing and a handful of customers report that a payment was taken twice. Two opposite-looking symptoms, one root cause: the worker's partner call sometimes takes 45 seconds, the queue's visibility timeout is 30 seconds, so the message reappears while the first worker is still busy and a second worker picks it up. Meanwhile one malformed instruction fails every time, is retried until it lands in the DLQ, and nobody is watching the DLQ.
SQS standard queues promise at least once delivery. Duplicates are not a bug in SQS, they are the contract. A bank-grade consumer must therefore be idempotent (safe to run twice) and must treat the queue's retry settings as part of the design, not as defaults.
The idea in plain language
SQS does not hand a message to a consumer and forget it. It lends the message:
- A consumer calls
ReceiveMessage. SQS makes the message invisible to other consumers for the visibility timeout and increments its receive count. - If the consumer finishes, it calls
DeleteMessageand the message is gone. - If the consumer fails, crashes or is too slow, the timeout expires and the message becomes visible again and is delivered to somebody (maybe the same worker).
- If a message has been received more times than the queue's
maxReceiveCount, SQS moves it to the dead-letter queue instead.
Every strange SQS symptom is one of these four moves going differently from what you expected.
Analogy: A bank's paper work-tray. A clerk takes a form and puts a "being worked on" sign on the tray slot for 30 minutes. If the clerk finishes, the form is filed (deleted). If the clerk goes to lunch, after 30 minutes the sign expires and another clerk takes it. If the same form has gone round five clerks and none could process it, the supervisor moves it to the exceptions tray (the DLQ). Two clerks may both process a form when the first one was only slow, not gone.
Delivery vocabulary
| Term | Meaning |
|---|---|
| Visible | Available to be received. Counted by ApproximateNumberOfMessagesVisible |
| In flight / not visible | Received but not yet deleted, within its visibility timeout. Counted by ApproximateNumberOfMessagesNotVisible |
| Receive count | How many times the message has been received. Available as the ApproximateReceiveCount attribute |
| maxReceiveCount | The redrive policy threshold, after which the message moves to the DLQ |
| Retention period | How long an unprocessed message is kept (default 4 days, up to 14) |
| Delay | Seconds a new message stays invisible before first delivery |
| Message group (FIFO) | Messages with the same group ID are delivered one at a time, in order |
What problem the AWS service solves
SQS decouples the producer from the consumer so a slow or failing consumer does not break callers, and it absorbs bursts. The cost of that convenience is that you own consumer behaviour: how long processing may take, what happens on failure, what to do with poison messages and how to avoid duplicate effects.
Limits and sharp edges (⚠ VERIFY current values before quoting): the visibility timeout defaults to 30 seconds with a maximum of 12 hours; long polling waits up to 20 seconds; the number of in-flight messages is capped (for standard queues in the order of 120,000; for FIFO queues 20,000); a FIFO queue deduplicates by deduplication ID within a five-minute window; a DLQ for a standard queue must be a standard queue and for FIFO must be FIFO; and messages moved to a DLQ keep their original enqueue time, so the DLQ's retention period counts from then.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Service Bus lock duration (peek-lock) | SQS visibility timeout | Similar, but SQS has no explicit "abandon" or lock renewal: you call ChangeMessageVisibility to extend |
| Service Bus MaxDeliveryCount → dead-letter sub-queue | SQS maxReceiveCount → separate DLQ queue | The DLQ is an ordinary queue you create and monitor |
| Service Bus sessions | SQS FIFO message groups | A failing message blocks its group, not the whole queue |
| Service Bus duplicate detection | FIFO deduplication ID (5 minutes) | Standard queues have no dedup at all |
Azure Storage Queue DequeueCount | ApproximateReceiveCount | Same idea. The word "approximate" is literal |
| Azure Functions queue trigger poison queue | Lambda event source mapping with a DLQ or an on-failure destination | You configure the redrive and partial batch failure yourself |
| Resubmit from the Service Bus dead-letter in the portal | SQS DLQ redrive (console and API) | Redrive can move messages back at a limited rate |
Architecture
The life of one message
Symptom to cause map
Step-by-step implementation
Minutes 0 to 5: identify the symptom from numbers
Read these queue attributes and metrics together:
| Metric or attribute | What it tells you |
|---|---|
ApproximateNumberOfMessagesVisible | Waiting to be picked up. Rising means consumers are too few, too slow or stopped |
ApproximateNumberOfMessagesNotVisible | In flight. High and stuck means consumers are not deleting |
ApproximateAgeOfOldestMessage | How long the oldest message has waited. This is your user-facing delay |
NumberOfMessagesReceived versus NumberOfMessagesDeleted | Received much greater than deleted means redelivery or failures |
DLQ ApproximateNumberOfMessagesVisible | Anything above 0 is an incident for a payments queue |
Queue attributes VisibilityTimeout, RedrivePolicy, ReceiveMessageWaitTimeSeconds | The configuration you are diagnosing |
Compare received to deleted. If they are close, the queue is healthy but behind. If received greatly exceeds deleted, messages are coming back.
Minutes 5 to 20: check the consumer side
- Are consumers running and polling? Check the ECS service desired and running counts, or the Lambda event source mapping state.
- Do they have
sqs:ReceiveMessage,sqs:DeleteMessage,sqs:ChangeMessageVisibilityandsqs:GetQueueAttributeson the queue, andkms:Decryptif the queue uses a customer managed key? A missing KMS permission makes every receive fail. - Look at the consumer logs for the message ID and its receive count. Repeating errors with the same ID mean a poison message. Many different IDs, one delivery each but long processing times, mean the visibility timeout is the issue.
- For Lambda consumers: check throttles, errors and the function timeout. AWS guidance is to set the queue visibility timeout to at least six times the function timeout (⚠ VERIFY current guidance), and to keep
maxReceiveCountcomfortably above one so brief throttling does not push messages to the DLQ.
Minutes 20 to 40: pick the cause
| Observation | Cause | Fix |
|---|---|---|
| Processing time is longer than the visibility timeout | Message reappears while still processed | Raise the timeout above worst-case processing, or extend it with a heartbeat while working |
| Same message ID with rising receive count, handler throws | Poison message | Fix the code or data, move aside, never loop forever |
| Lambda receive counts jump during traffic spikes without handler errors | Throttling returns messages to the queue | Set reserved or maximum concurrency sensibly, raise maxReceiveCount |
| Batch of 10: one failing record makes all 10 retry | Whole batch fails on one error | Enable partial batch responses (ReportBatchItemFailures) |
| NotVisible high, nothing completing | Consumers fetch but never delete (code path forgets DeleteMessage) | Delete on success in every path, add a test |
| FIFO queue stalled, one group stuck | A failing message blocks its group while others flow | Fix or move the message, use more groups to limit blast radius |
| Messages vanish after four days | Retention period expired | Increase retention, consume faster, DLQ retention longer than source |
| Consumers poll but receive nothing | Long polling not used with many empty receives, or wrong queue URL, or a delay on the queue | Check ReceiveMessageWaitTimeSeconds, queue URL and delay |
Minutes 40 to 60: inspect the DLQ safely, then fix
- Look, do not drain. Receive a few messages from the DLQ for inspection with a long visibility timeout so nobody else sees them, record the message ID, body shape (not full personal data in chat), attributes and
ApproximateReceiveCount, and let them return. Receiving does not delete. - Find the failure reason in the consumer logs by message ID.
- Fix the cause first. Redriving unchanged messages into a consumer that still fails just fills the DLQ again.
Minutes 60 to 75: redrive deliberately and prevent
Redrive from the DLQ back to the source queue (console Start DLQ redrive, or the start-message-move-task command) at a controlled rate, with the idempotent consumer ready so any message that was already processed has no second effect. Then add the preventive controls: an alarm on DLQ depth above zero, an alarm on oldest message age, a visibility timeout derived from measured processing time, partial batch responses, and an idempotency guard in every handler. Record the message IDs redriven and the outcome for audit.
AWS Console instructions
- Console → SQS → Queues → (queue) → Monitoring: visible, not visible, oldest message age, received and deleted.
- Console → SQS → (queue) → Details: visibility timeout, retention, delivery delay, receive message wait time, encryption and the Dead-letter queue section.
- Console → SQS → (queue) → Send and receive messages → Poll for messages: view message bodies and attributes (including Receive count) without deleting them. Keep the poll duration short.
- Console → SQS → (DLQ) → Start DLQ redrive: choose the destination and a velocity setting (system optimised or custom messages per second), then watch the redrive status.
- Console → Lambda → (function) → Configuration → Triggers → SQS trigger: batch size, batch window, maximum concurrency and Report batch item failures.
AWS CLI
# 1. Queue configuration and current depth in one call.
aws sqs get-queue-attributes \
--queue-url https://sqs.us-east-1.amazonaws.com/111122223333/hl-payment-instructions \
--attribute-names VisibilityTimeout MessageRetentionPeriod RedrivePolicy ReceiveMessageWaitTimeSeconds \
ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible ApproximateNumberOfMessagesDelayed \
--profile learn-sandbox
# 2. Age of the oldest message (seconds), last hour.
aws cloudwatch get-metric-statistics --namespace AWS/SQS --metric-name ApproximateAgeOfOldestMessage \
--dimensions Name=QueueName,Value=hl-payment-instructions \
--start-time $(date -u -d '1 hour ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
--period 300 --statistics Maximum --profile learn-sandbox
# 3. Peek at DLQ messages WITHOUT deleting: long visibility timeout so nobody else grabs them.
aws sqs receive-message \
--queue-url https://sqs.us-east-1.amazonaws.com/111122223333/hl-payment-instructions-dlq \
--max-number-of-messages 3 --visibility-timeout 300 --wait-time-seconds 5 \
--attribute-names All --message-attribute-names All --profile learn-sandbox
# 4. Extend visibility of a message you are inspecting (the receipt handle comes from step 3).
aws sqs change-message-visibility \
--queue-url https://sqs.us-east-1.amazonaws.com/111122223333/hl-payment-instructions-dlq \
--receipt-handle "<receipt-handle>" --visibility-timeout 900 --profile learn-sandbox
# 5. Redrive from the DLQ back to the source queue at a limited rate.
aws sqs start-message-move-task \
--source-arn arn:aws:sqs:us-east-1:111122223333:hl-payment-instructions-dlq \
--destination-arn arn:aws:sqs:us-east-1:111122223333:hl-payment-instructions \
--max-number-of-messages-per-second 5 --profile learn-sandbox
# 6. Watch the redrive task, and cancel it if the consumer starts failing again.
aws sqs list-message-move-tasks \
--source-arn arn:aws:sqs:us-east-1:111122223333:hl-payment-instructions-dlq --profile learn-sandbox
# aws sqs cancel-message-move-task --task-handle "<task-handle>" --profile learn-sandboxNever use purge-queue on a payments queue during an investigation: it deletes every message, cannot be undone and destroys evidence.
Infrastructure as Code
The settings that prevent this incident: a visibility timeout based on the work, a DLQ with longer retention, a sensible maxReceiveCount, encryption, and alarms that watch the DLQ and the age of the oldest message.
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.SQS;
using Constructs;
public class PaymentQueueStack : Stack
{
public PaymentQueueStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
{
var dlq = new Queue(this, "InstructionsDlq", new QueueProps
{
QueueName = "hl-payment-instructions-dlq",
RetentionPeriod = Duration.Days(14), // longer than the source queue: the clock starts at the ORIGINAL enqueue time
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true
});
var queue = new Queue(this, "Instructions", new QueueProps
{
QueueName = "hl-payment-instructions",
VisibilityTimeout = Duration.Seconds(180), // comfortably above the slowest observed processing time
RetentionPeriod = Duration.Days(4),
ReceiveMessageWaitTime = Duration.Seconds(20), // long polling
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 }
});
// Any message in the DLQ is an incident for a payments queue.
new Alarm(this, "DlqNotEmpty", new AlarmProps
{
AlarmName = "hl-payment-instructions-dlq-not-empty",
AlarmDescription = "Messages in the payment instructions DLQ. See playbook sqs-messages-stuck-or-redelivered.",
Metric = dlq.MetricApproximateNumberOfMessagesVisible(new MetricOptions { Period = Duration.Minutes(1) }),
Threshold = 0,
EvaluationPeriods = 1,
ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING
});
// Delay seen by customers: oldest message older than 5 minutes for 3 periods.
new Alarm(this, "OldestMessageAge", new AlarmProps
{
AlarmName = "hl-payment-instructions-age",
Metric = queue.MetricApproximateAgeOfOldestMessage(new MetricOptions { Period = Duration.Minutes(1), Statistic = "Maximum" }),
Threshold = 300,
EvaluationPeriods = 3,
ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-sqs");
}
}Complete code
A simulation that reproduces the duplicate and the fixes
This program needs no AWS account. It models an SQS queue's lending rules in virtual time (one tick is one second) with two consumers, and runs four scenarios: the duplicate-processing bug, then the fix by longer visibility timeout, the fix by an idempotency guard, and a poison message that ends in the DLQ. The model is deliberately simplified (it ignores network delay, receipt handles and approximate counts) so the cause and effect are clear.
var scenarios = new[]
{
new Scenario("1 BUG: visibility 30s, work 40s, no idempotency", VisibilitySeconds: 30, WorkSeconds: 40, Idempotent: false, AlwaysFails: false, MaxReceiveCount: 5),
new Scenario("2 FIX: visibility 60s, work 40s", VisibilitySeconds: 60, WorkSeconds: 40, Idempotent: false, AlwaysFails: false, MaxReceiveCount: 5),
new Scenario("3 FIX: visibility 30s, work 40s, idempotent", VisibilitySeconds: 30, WorkSeconds: 40, Idempotent: true, AlwaysFails: false, MaxReceiveCount: 5),
new Scenario("4 POISON: handler always fails, maxReceive 3", VisibilitySeconds: 30, WorkSeconds: 5, Idempotent: true, AlwaysFails: true, MaxReceiveCount: 3),
};
foreach (var s in scenarios)
{
var r = Simulate(s);
Console.WriteLine($"== {s.Name}");
foreach (var line in r.Log) Console.WriteLine(" " + line);
Console.WriteLine($" RESULT: side effects executed = {r.SideEffects}, receives = {r.Receives}, in DLQ = {r.InDlq}");
Console.WriteLine();
}
return 0;
static Result Simulate(Scenario s)
{
const string id = "pay-1001";
var visibleAt = 0; // virtual second at which the message is next visible
var receiveCount = 0;
var exists = true; // still in the source queue
var inDlq = false;
var sideEffects = 0;
var done = new HashSet<string>(); // the idempotency store
var log = new List<string>();
var consumers = new[] { new Consumer("A"), new Consumer("B") };
for (var t = 0; t <= 200; t++)
{
foreach (var c in consumers)
{
// 1. A consumer that finishes its work now.
if (c.Busy && t >= c.FinishAt)
{
c.Busy = false;
if (s.AlwaysFails)
{
log.Add($"t={t,3}s {c.Name} FAILED, does not delete (message returns after its timeout)");
}
else if (s.Idempotent && done.Contains(id))
{
log.Add($"t={t,3}s {c.Name} finished: duplicate detected, side effect SKIPPED");
exists = false;
}
else
{
sideEffects++;
done.Add(id);
log.Add($"t={t,3}s {c.Name} finished: payment submitted (side effects so far: {sideEffects}), deletes message");
exists = false;
}
}
// 2. An idle consumer polls the queue.
if (!c.Busy && exists && t >= visibleAt)
{
receiveCount++;
if (receiveCount > s.MaxReceiveCount)
{
exists = false;
inDlq = true;
log.Add($"t={t,3}s {c.Name} polled: receive count would be {receiveCount} (more than {s.MaxReceiveCount}), SQS moves the message to the DLQ");
continue;
}
visibleAt = t + s.VisibilitySeconds;
c.Busy = true;
c.FinishAt = t + s.WorkSeconds;
log.Add($"t={t,3}s {c.Name} received (receive count {receiveCount}), invisible until t={visibleAt}s, working {s.WorkSeconds}s");
}
}
}
return new Result(log, sideEffects, receiveCount, inDlq);
}
record Scenario(string Name, int VisibilitySeconds, int WorkSeconds, bool Idempotent, bool AlwaysFails, int MaxReceiveCount);
record Result(List<string> Log, int SideEffects, int Receives, bool InDlq);
class Consumer(string name)
{
public string Name { get; } = name;
public bool Busy { get; set; }
public int FinishAt { get; set; }
}Create a console project targeting net10.0, save the code as Program.cs and run dotnet run. In scenario 1 you should see two "payment submitted" lines (side effects = 2). Scenario 2 shows one, scenario 3 shows one plus a "duplicate detected" line, and scenario 4 ends with the message in the DLQ after three attempts.
A real consumer with a visibility heartbeat and an idempotency guard
This sketch shows the production shape with AWSSDK.SQS v4. It extends the visibility timeout while work is running and skips work already done. Replace the in-memory idempotency store with a conditional write to DynamoDB, as shown in Idempotency in .NET.
using Amazon.SQS;
using Amazon.SQS.Model;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
public interface IIdempotencyStore
{
// Returns true if this caller is the first to claim the key, false if it was already claimed.
Task<bool> TryClaimAsync(string key, CancellationToken ct);
}
public sealed class PaymentWorker(IAmazonSQS sqs, IIdempotencyStore store, ILogger<PaymentWorker> log, string queueUrl)
: BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
var response = await sqs.ReceiveMessageAsync(new ReceiveMessageRequest
{
QueueUrl = queueUrl,
MaxNumberOfMessages = 10,
WaitTimeSeconds = 20, // long polling: fewer empty receives, lower cost
MessageSystemAttributeNames = new List<string> { "ApproximateReceiveCount" }
}, stoppingToken);
foreach (var message in response.Messages ?? new List<Message>())
await HandleAsync(message, stoppingToken);
}
}
private async Task HandleAsync(Message message, CancellationToken ct)
{
message.Attributes.TryGetValue("ApproximateReceiveCount", out var receives);
// Heartbeat: keep the message invisible while the work is genuinely running.
using var heartbeatCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
var heartbeat = Task.Run(async () =>
{
while (!heartbeatCts.IsCancellationRequested)
{
await Task.Delay(TimeSpan.FromSeconds(20), heartbeatCts.Token);
await sqs.ChangeMessageVisibilityAsync(queueUrl, message.ReceiptHandle, 60, heartbeatCts.Token);
}
}, heartbeatCts.Token);
try
{
// The SQS MessageId is stable across redeliveries of one message. If the PRODUCER may also send
// duplicates (a retried HTTP call), key on a business identifier carried in the body instead.
if (!await store.TryClaimAsync($"payment-instruction:{message.MessageId}", ct))
{
log.LogInformation("Duplicate delivery skipped, message {MessageId}, receive count {Receives}", message.MessageId, receives);
}
else
{
await SubmitToClearingAsync(message.Body, ct); // the real side effect
}
await sqs.DeleteMessageAsync(queueUrl, message.ReceiptHandle, ct); // delete ONLY after success
}
catch (Exception ex)
{
// Do not delete: the message returns after the visibility timeout and counts towards maxReceiveCount.
log.LogError(ex, "Handler failed for message {MessageId}, receive count {Receives}", message.MessageId, receives);
}
finally
{
await heartbeatCts.CancelAsync();
try { await heartbeat; } catch (OperationCanceledException) { }
}
}
private static Task SubmitToClearingAsync(string body, CancellationToken ct) => Task.CompletedTask; // placeholder
}For Lambda consumers, enable Report batch item failures on the event source mapping and return the failed IDs so one bad record does not retry the whole batch:
using Amazon.Lambda.Core;
using Amazon.Lambda.SQSEvents;
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]
public class Function
{
public async Task<SQSBatchResponse> Handle(SQSEvent evnt, ILambdaContext context)
{
var failures = new List<SQSBatchResponse.BatchItemFailure>();
foreach (var record in evnt.Records)
{
try { await ProcessAsync(record.Body); }
catch (Exception ex)
{
context.Logger.LogError($"Failed {record.MessageId}: {ex.Message}");
failures.Add(new SQSBatchResponse.BatchItemFailure { ItemIdentifier = record.MessageId });
}
}
return new SQSBatchResponse(failures); // only these return to the queue
}
private static Task ProcessAsync(string body) => Task.CompletedTask; // placeholder
}Code explained line by line
- The simulation's
visibleAtis the second at which the message can next be received. Receiving sets it to now plus the visibility timeout. This one variable is the entire lending mechanism. - A consumer that finishes calls "delete" by setting
exists = false. In scenario 1 consumer A finishes at second 40 and deletes, but consumer B had already received the message at second 30, so B also performs the side effect at second 70. That is the double payment. - In scenario 3 the
doneset plays the idempotency store. B finds the key already claimed and skips the side effect, so the duplicate delivery is harmless. - Scenario 4 never deletes. After each failure the timeout expires, the message returns, and when the receive count would exceed
MaxReceiveCount, the message goes to the DLQ instead of looping forever. - In the real worker,
WaitTimeSeconds = 20turns on long polling. MessageSystemAttributeNameswithApproximateReceiveCountasks SQS to include the receive count so logs can show how many times a message has been delivered.- The heartbeat task calls
ChangeMessageVisibilityAsyncevery 20 seconds, resetting the timeout to 60 seconds from now. This keeps a legitimately slow message invisible without a huge static timeout. TryClaimAsyncis the idempotency guard. In production it is a DynamoDB conditional put, so only one caller can claim the key.DeleteMessageAsyncruns only after success. If anything throws, the code does not delete, so SQS redelivers it, which is the retry mechanism.- In the Lambda handler, returning
SQSBatchResponsewith only the failed message IDs tells the service to delete the successes and retry just the failures.
Security considerations
⚠ Learning shortcut — not for production
Inspecting messages by polling the DLQ in the console or CLI shows message bodies, which can contain personal and payment data. In production, restrict who can receive from the DLQ, do not paste bodies into tickets or chat, and log identifiers rather than payloads. The simulation also stores nothing sensitive; a real idempotency store must be durable and access controlled.
- Least privilege on the queue. Give consumers receive, delete and change-visibility on their queue only, producers send-only, and responders a separate read-only role for the DLQ. Use a queue policy to restrict which principals and source ARNs may send.
- Encrypt at rest and in transit. Use SQS-managed or KMS encryption and enforce TLS. With a customer managed key, consumers need
kms:Decryptand producerskms:GenerateDataKey, which is a common cause of mysterious receive failures (see KMS envelope encryption). - Never
purge-queueduring an incident, and protect the permission with an explicit deny for everyone but a break-glass role. - Redrive is a change. It reintroduces old instructions into the live system. For payments, confirm with the business owner that instructions are still valid (not stale, not already compensated) before redriving.
- Poison messages can be attacks. Validate and bound message size and shape, and never execute content from a message.
- Audit. Record message IDs moved, who redrove, when and with what outcome.
Banking use case
Harbourline's payment-instruction queue has a visibility timeout of 180 seconds derived from the 99th percentile of the partner call (measured, not guessed), a heartbeat for exceptional cases, maxReceiveCount of 5, a 14-day DLQ and an alarm that pages on any DLQ message. Each instruction carries a business identifier, and the worker claims it with a conditional write before calling the clearing partner, so a duplicate delivery cannot cause a second payment. During the 15:20 incident, the team compared received and deleted counts, found receive counts of 2 on slow messages, raised the visibility timeout, added the idempotency guard, and redrove the three poison instructions from the DLQ at one message per second after correcting a malformed currency field. The audit record lists the three message IDs, the correction approved by the payments owner, and the redrive task handle.
Common mistakes
- Visibility timeout left at the 30-second default while the work takes longer.
- No idempotency, assuming SQS delivers once.
- No DLQ, or a DLQ nobody watches.
- DLQ retention equal to or shorter than the source queue, so messages expire right after moving.
- Deleting before the work finishes, which loses messages on crash.
- Retrying a whole Lambda batch because of one bad record.
maxReceiveCountof 1 with Lambda, so a throttle sends good messages to the DLQ.- Using one FIFO message group for everything, so one poison message blocks all traffic.
- Purging a queue to "fix" it, deleting evidence and good messages.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Visible messages growing, consumers idle | Consumers cannot receive (permission, KMS, wrong URL, VPC endpoint missing) | Check consumer logs for AccessDenied or timeouts, the queue policy and the endpoint |
| NotVisible stays high, never drops | Consumers receive but never delete | Fix the code path, delete on success |
| Same message repeats with rising receive count | Handler failing or processing slower than the visibility timeout | Check logs by message ID, raise the timeout or add a heartbeat |
| Many messages in the DLQ after a short outage | Downstream was down and maxReceiveCount was low | Fix the dependency, redrive at a low rate, raise maxReceiveCount |
| Lambda errors "throttled" and messages in the DLQ | Concurrency limit reached | Adjust reserved and maximum concurrency, raise maxReceiveCount |
| FIFO queue only delivers part of the messages | A failing message blocks its group | Fix, delete or redrive the blocking message, split group IDs |
| Redrive fails or does nothing | DLQ and source types differ, or missing permission | Match standard to standard, FIFO to FIFO, check redrive permissions |
| DLQ message expired quickly | Retention counted from original enqueue time | Set DLQ retention longer than the source queue |
| Duplicate effects despite a DLQ | Consumer not idempotent | Add a durable idempotency guard |
Cost and cleanup
Cost label: free. The simulation needs no AWS resources. A sandbox queue and a handful of messages cost far less than a cent (⚠ VERIFY current pricing, which has a monthly free tier), and the alarms in this lesson can incur a small charge, so delete them afterwards.
# Delete the sandbox queues and alarms
aws sqs delete-queue --queue-url https://sqs.us-east-1.amazonaws.com/111122223333/hl-payment-instructions --profile learn-sandbox
aws sqs delete-queue --queue-url https://sqs.us-east-1.amazonaws.com/111122223333/hl-payment-instructions-dlq --profile learn-sandbox
aws cloudwatch delete-alarms --alarm-names hl-payment-instructions-dlq-not-empty hl-payment-instructions-age --profile learn-sandbox
# Confirm nothing tagged for this lesson remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-sqs --profile learn-sandboxExercises
- 🟢 Guided: Run the simulation. For scenario 1, write the exact second each consumer received the message and each side effect happened. Then change
VisibilitySecondsto 45 and 41 and explain the result. - 🟡 Challenge: Add a scenario with a heartbeat: the consumer extends visibility by 30 seconds every 20 seconds while working. Show that with a 30-second base timeout and 90 seconds of work there is still only one receive.
- 🔴 Live problem: "A FIFO queue of customer notifications stopped delivering for one customer's messages two hours ago; others are fine. DLQ is empty. The consumer logs show the same message failing every 30 seconds." Explain what is happening, what you check, and what you do about it.
Assignment
Write an SQS runbook page for your portfolio: the message lifecycle diagram, the metric table, the symptom-to-cause table, the safe DLQ inspection and redrive commands, the alarm definitions and the idempotency approach for one capstone queue. Commit as runbooks/sqs-stuck-or-redelivered.md with fictional identifiers.
Solutions
In scenario 1, A receives at t=0 (receive count 1, invisible until 30). At t=30 the message is visible again and B receives it (count 2, invisible until 60). A finishes at t=40: first side effect, deletes. B finishes at t=70: second side effect, because B already held the message. With visibility 45, A still finishes at 40, which is inside the 45 seconds, so the message never reappears and there is one side effect. With 41 it is also one: the message would only become visible at 41, after A has finished and deleted at 40. Why: a duplicate happens only when work time exceeds the visibility timeout, so the boundary is exactly where the timeout is shorter than the work.
Add to Scenario a HeartbeatSeconds value. In the busy consumer's per-tick step, if t - c.StartedAt is a positive multiple of HeartbeatSeconds, set visibleAt = t + 30 and log "extended visibility". With a 30-second base timeout, 90 seconds of work and a 20-second heartbeat, the visibility is pushed out at t=20, 40, 60 and 80 (to 50, 70, 90, 110), so it is always in the future while the consumer works, and the message never reappears. When the work ends at t=90 the consumer deletes it. Why: a heartbeat keeps the message hidden only as long as the consumer is alive: if it crashes, the extensions stop and the message returns quickly.
FIFO queues deliver messages of one group in order, one at a time. A message that keeps failing blocks every later message in the same group until it succeeds, is deleted or moves to the DLQ. The customer's ID is probably the group ID, so only that customer is blocked, and the empty DLQ means maxReceiveCount has not yet been exceeded (or is set very high). Check: the queue's redrive policy and the receive count of the blocking message, the consumer log for the error, whether the message body is malformed. Actions: fix the cause if it is code or data, or move the poison message aside (let it reach the DLQ by lowering maxReceiveCount, or inspect and delete it with approval), then confirm the group drains. Prevent: sensible maxReceiveCount on FIFO queues, an alarm on oldest message age, and group IDs fine-grained enough to limit the blast radius. Why: ordering guarantees mean one stuck item stops all items behind it in that group.
Knowledge check
Answer: After 30 seconds the message becomes visible again and another consumer receives it, so the work may run twice. Raise the timeout above worst-case time or extend it with a heartbeat, and make the handler idempotent.
Answer: A message keeps its original enqueue time when it moves, so the DLQ's retention clock counts from then. A short retention makes it expire soon after arriving.
Answer: The function returns only the failed message IDs, so successful messages are deleted and only the failed ones return to the queue instead of the whole batch.
Answer: It deletes all messages irreversibly, including good ones and the evidence you need. Inspect, fix and redrive instead.
Answer: A failing message blocks its message group. Other groups continue, so only that customer's messages stall.
Interview questions
- Messages in an SQS queue are processed twice. Why, and what do you do? At-least-once delivery, plus a visibility timeout shorter than processing or consumer retries. Make handlers idempotent, align the timeout and use a heartbeat if needed.
- How do you handle poison messages? A redrive policy with a sensible
maxReceiveCount, a DLQ with an alarm, inspection without deleting, fixing the cause and controlled redrive. - How do you set the visibility timeout? From measured processing time at a high percentile with headroom, six times the function timeout for Lambda (⚠ VERIFY), with a heartbeat for exceptional cases.
- What is different about FIFO queues when things fail? Ordering within a group means one failing message blocks its group, and deduplication is limited to a five-minute window.
- How would you redrive 10,000 messages safely? Fix the cause first, ensure idempotency, redrive at a controlled rate, watch the consumer's error rate and be ready to cancel the task.
Certification practice
A consumer processes SQS messages in about 50 seconds. The queue's visibility timeout is the default 30 seconds. Some orders are processed twice. What is the best fix?
A. Increase the message retention period. B. Increase the visibility timeout to exceed the processing time (and make the processing idempotent). C. Reduce the batch size to 1 only. D. Switch to a smaller instance type.
Answer: B. The message reappears after 30 seconds while the first consumer is still working. Retention, batch size and instance size do not stop redelivery.
Operations wants an alert when any message is moved to the dead-letter queue of a critical queue. Which is the most appropriate solution?
A. A CloudWatch alarm on ApproximateNumberOfMessagesVisible of the DLQ greater than 0.
B. A weekly review of the DLQ.
C. An alarm on the source queue's retention period.
D. CloudTrail data events for the DLQ.
Answer: A. The DLQ depth metric directly signals failed messages. Weekly review is too slow, and the other options do not measure dead-lettered messages.
A payment processing system uses SQS with Lambda. Occasional downstream throttling sends valid messages to the DLQ after one failed attempt. Which change best reduces that?
A. Set maxReceiveCount to 1.
B. Increase maxReceiveCount, set the visibility timeout well above the function timeout, and use partial batch responses.
C. Disable the DLQ.
D. Use a larger message size.
Answer: B. More receive attempts and a suitable timeout tolerate transient failures while the DLQ still catches real poison messages. A makes it worse, C loses failed messages, D is unrelated.
Completion checklist
- I can describe the receive, invisible, delete and redeliver cycle
- I can tell a slow consumer from a poison message using receive count
- I ran the simulation and explained scenario 1 second by second
- I know how to inspect a DLQ without deleting and how to redrive at a controlled rate
- I can explain why handlers must be idempotent
- I ran the cleanup steps and confirmed nothing tagged
m17-playbook-sqsremains
Key takeaways
- SQS lends messages: visibility timeout, receive count and
maxReceiveCountexplain nearly every symptom. - At-least-once delivery means idempotent handlers are mandatory.
- Visibility timeout should exceed worst-case processing time, with a heartbeat for exceptions.
- Watch the DLQ and the age of the oldest message, and inspect before redriving.
- Never purge during an incident.
Further reading
- Amazon SQS visibility timeout
- Amazon SQS dead-letter queues
- Previous playbook: Slow SQL after migration to RDS
- Next playbook: Key rotation without downtime
- Related: Module 17 overview
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Playbook: slow SQL after migration to RDS
Find out why a query that was fast on-premises is slow on Amazon RDS by separating network round trips, plan and statistics changes, storage limits, parameter differences and connection-pool behaviour, using database insights, wait statistics and Query Store.
Playbook: key rotation without downtime
Rotate KMS keys, database secrets, API keys and signing keys without an outage by using overlap periods: KMS automatic rotation versus manual key swap, Secrets Manager staging labels and alternating users, and clients that refresh credentials on failure.