Delivery guarantees and ordering
Know exactly what each AWS messaging service promises — at-most-once, at-least-once or exactly-once, ordered or not — and build .NET consumers that stay correct when messages arrive twice, late or out of order, using FIFO message groups, sequence numbers and version checks.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SQS / AWSSDK.DynamoDBv2 4.1.0.2) · Amazon.Lambda.SQSEvents 3.0.1 · Amazon.Lambda.Serialization.SystemTextJson 3.0.1 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Define at-most-once, at-least-once and exactly-once, and explain why "exactly-once processing" in practice means at-least-once delivery plus idempotent handling.
- State the delivery and ordering guarantees of SQS standard and FIFO, SNS standard and FIFO, EventBridge (Classic and the newer Custom Event Bus), Kinesis Data Streams, DynamoDB Streams, Lambda async invocation and Step Functions.
- Choose an ordering key (FIFO message group, partition key, event group) that gives you the order you need without killing throughput.
- Handle out-of-order and duplicate events in .NET with per-aggregate sequence numbers and DynamoDB conditional writes.
- Configure an SQS FIFO → Lambda consumer that keeps per-account order and reports partial batch failures correctly.
Prerequisites
- The SQS, SNS, EventBridge and Kinesis lessons earlier in this module (see the module overview)
- Retry strategies — retries are the main source of duplicates
- Module 6: Databases — DynamoDB conditional writes
- Tools: .NET 10 SDK, AWS CLI v2 with the
learn-sandboxprofile
Why this matters in banking
Harbourline's mobile app shows a live balance fed by BalanceChanged events. Two incidents in one month:
- The phantom balance. A customer's salary of £2,400.00 arrived, then a £60.00 direct debit went out. The events arrived at the app's projection service in the wrong order. For eleven seconds the app showed a balance £60 higher than it had ever really been, and the customer moved money based on it. The direct debit then failed for insufficient funds.
- The double notification. A standard SQS queue delivered one
CardPaymentAuthorisedevent twice. The notification service had no duplicate check, so the customer received two "you paid £18.40 at Example Coffee" messages. They called to report fraud.
Neither incident was a bug in AWS. Both services behaved exactly as documented. The design simply assumed guarantees the services never gave.
For ledgers, balances, statements and regulatory reporting, you must know which order and which uniqueness you can rely on, and design for the rest.
The idea in plain language
A messaging system can promise three levels of delivery:
| Guarantee | Plain meaning | What you must handle |
|---|---|---|
| At-most-once | Each message is delivered once or not at all | Lost messages |
| At-least-once | Each message is delivered one or more times | Duplicates |
| Exactly-once | Each message is delivered once | In distributed systems, only within a stated scope and time window |
Ordering also comes in levels:
- No ordering (best effort): messages usually arrive roughly in order, but nothing is promised.
- Per-key ordering (partial order): messages with the same key, such as one account ID, arrive in the order sent. Messages with different keys are independent.
- Total ordering: every message in the system in one global sequence. This is expensive and rarely needed. Banks usually need per-account order, not global order.
Analogy: Think of the post. Ordinary post is at-least-once with no ordering: letters sometimes overtake each other, and occasionally a returned letter is re-sent. Recorded delivery with a sequence number on each envelope lets the recipient say "I already have letter 41, and 43 can't be filed until 42 arrives". The postal service hasn't become perfect. The recipient has become careful. That is what your consumers must do.
The most important sentence in this lesson:
Exactly-once processing is something you build. It equals at-least-once delivery (nothing is lost) plus idempotent processing (duplicates have no extra effect), and, where order matters, per-key ordering or version checks.
Even when a service offers "exactly-once", that promise covers only a specific scope. SQS FIFO deduplicates sends within a five-minute window. Your Lambda can still crash after updating DynamoDB and before the message is deleted, so it sees the message again.
What each AWS service guarantees
Checked against AWS documentation on 2026-11-18.
| Service | Delivery | Ordering | Deduplication | Notes |
|---|---|---|---|---|
| SQS standard | At-least-once | Best effort | None | Nearly unlimited throughput. A message can occasionally be delivered more than once |
| SQS FIFO | Exactly-once processing within the deduplication interval | Strict, per MessageGroupId | MessageDeduplicationId or content-based (SHA-256 of body), 5-minute interval | 300 API calls/s per partition without high-throughput mode (3,000 messages/s with batches of 10); high-throughput mode raises this greatly |
| SNS standard | At-least-once | None | None | Fan-out to many protocols |
| SNS FIFO | Ordered, deduplicated delivery to SQS FIFO subscribers | Per message group | Same 5-minute model as SQS FIFO | Delivers only to SQS (FIFO or standard) queues; standard queue subscribers get best-effort ordering and at-least-once |
| EventBridge — Custom Event Bus - Classic (rules and targets) | At-least-once | None | None | Retries up to 24 h / 185 attempts by default |
| EventBridge — Custom Event Bus (subscribers, launched 2026) | At-least-once to targets; publish-time dedup | FIFO per EventGroupId per FIFO subscriber | DeduplicationId or content-based, 5-minute window | Retention and replay by subscriber starting position. Newer service — check availability and SDK support in your Region ⚠ VERIFY |
| Kinesis Data Streams | At-least-once (producer and consumer retries both create duplicates) | Strict per shard, by sequence number; records with the same partition key go to the same shard | None | Replay by position within the retention period |
| DynamoDB Streams | Each change record appears exactly once in the stream | Per item (primary key) | Not needed in the stream itself | 24-hour retention. Your Lambda consumer can still be invoked more than once for a batch |
| Lambda async invocation | At-least-once (duplicates possible even without errors) | None | None | Event age and retry limits, destinations |
| Step Functions Standard | Exactly-once workflow execution (tasks re-run only if you declare Retry) | Within an execution | Starting an execution with the same name as a running one is idempotent | Good for non-idempotent steps such as payments |
| Step Functions Express | Async: at-least-once. Sync: at-most-once | Within an execution | None automatic | Make every task idempotent |
What "exactly-once processing" in SQS FIFO really covers: if a producer retries SendMessage with the same deduplication ID within five minutes, SQS doesn't create a second message. While one consumer holds a message, others can't receive messages from the same group. It does not cover:
- a producer that resends after more than five minutes;
- a producer that generates a new deduplication ID on retry;
- a consumer that processes a message and then fails to delete it before the visibility timeout ends.
That last one is the classic duplicate, and SQS FIFO will deliver that message again by design.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
Service Bus sessions (SessionId) for ordered processing | SQS FIFO MessageGroupId; SNS FIFO message groups; EventBridge Custom Event Bus EventGroupId | Same idea: order per key. SQS doesn't need an explicit session lock — receiving from a group blocks that group for other consumers |
| Service Bus duplicate detection (configurable window, up to days) | SQS FIFO / SNS FIFO deduplication | AWS window is fixed at 5 minutes — longer-term dedup is your job (DynamoDB idempotency table) |
Service Bus PeekLock + Complete | SQS receive + DeleteMessage (visibility timeout) | Same at-least-once outcome if you crash before completing |
| Event Hubs partitions + partition key | Kinesis shards + partition key; MSK partitions | Kinesis sequence numbers are per shard; order is per partition key |
| Event Grid (at-least-once, no ordering) | EventBridge Classic | Both need idempotent, order-tolerant handlers |
| Cosmos DB change feed (ordered per logical partition key) | DynamoDB Streams (ordered per item) | DynamoDB orders per item, not per partition key |
| Durable Functions (replay-based, activities at-least-once) | Step Functions Standard (exactly-once execution) / Express (at-least-once) | Choose the workflow type by whether steps are idempotent |
Architecture
The design below keeps Harbourline's balance projection correct, even though messages may arrive twice or out of order.
Three layers of protection work together:
- Ordering at the transport. All events for one account share a message group, so SQS FIFO delivers them in order and never hands two messages from the same group to different consumers at once.
- Deduplication at the transport. The transaction ID as
MessageDeduplicationIdabsorbs producer retries within five minutes. - Version checks at the consumer. Every event carries
accountSequence, assigned by the system of record. The projection applies an event only if its sequence is newer than the stored one. This catches everything the transport doesn't: duplicates after five minutes, replays from a DLQ or archive, and events that arrive by another path.
Layer 3 alone would keep the data correct, even with a standard queue. Layers 1 and 2 reduce wasted work and make the intermediate states correct. Your requirements decide whether you need them.
Ordering in depth
Choosing the ordering key
The key decides both correctness and throughput:
| Key | Order guaranteed for | Parallelism | Harbourline verdict |
|---|---|---|---|
One constant ("all") | Everything, globally | One message at a time | Never. A single slow message stops the bank |
customerId | All of a customer's accounts | One stream per customer | Only if cross-account order matters (it rarely does) |
accountId | Each account's postings | One stream per account | Usual choice for balances and statements |
paymentId | Each payment's life cycle (received → screened → settled) | One stream per payment | Good for payment state machines |
transactionId | Nothing useful (each group has one message) | Maximum | Equivalent to no ordering |
SQS FIFO in high-throughput mode spreads message groups across partitions. Throughput therefore scales with the number of active groups, and a single hot group is limited to what one consumer can process in sequence.
Head-of-line blocking
Ordering has a price. If message 42 for account A fails, message 43 for account A must wait, because delivering 43 first would break the order. With SQS FIFO, while 42 is in flight or invisible, the group is blocked. Other accounts' groups keep flowing.
Implications:
- A poison message blocks its group until it reaches the DLQ. Keep
maxReceiveCountand the visibility timeout modest on FIFO queues. - Once a message from a group goes to the DLQ, the next message in the group is delivered. Your consumer must then deal with the gap: it sees sequence 43 while the projection is at 41.
- On the new EventBridge Custom Event Bus, a failed delivery on a FIFO subscriber holds back all groups on that subscriber until it succeeds or retries run out. The blast radius is larger than SQS FIFO's per-group blocking. Plan retry limits accordingly.
Lambda with SQS FIFO
With an SQS FIFO event source mapping, Lambda delivers batches in order within each group, and doesn't process the same group concurrently. With partial batch responses, the rule differs from standard queues: when one message fails, stop processing that group and report the failed message and every later message from the same group as failures. Otherwise Lambda deletes the later messages that "succeeded", and they are processed out of order or lost from the sequence. The Complete code section shows how.
Kinesis and DynamoDB Streams
- Kinesis: order is per shard. Records with the same partition key go to the same shard, so choose
accountIdas the partition key for per-account order. With Lambda,ParallelizationFactorabove 1 processes a shard with several concurrent invocations while still keeping order per partition key. - DynamoDB Streams: order is guaranteed per item (primary key), not per partition key and not across items. If one account's postings are separate items, the stream does not order them relative to each other. Put the account sequence in the data and check it.
Handling out-of-order and duplicate events
When the transport can't guarantee what you need, the consumer must. Four techniques, from most to least common:
- Version or sequence check (last-writer-wins by sequence). Each event carries a monotonically increasing number per aggregate (account, payment). Store the last applied number. Apply only newer events; ignore equal or older ones. This works perfectly when events carry the resulting state (
balanceAfter = 2,340.00), because the newest event alone is enough. - Gap detection and buffering. When events carry deltas (
amount = -60.00), you can't skip one. If you receive 43 while at 41, park 43 (retry later, or store it) until 42 arrives. Set a timeout and an alert, because 42 may be in a DLQ. - Commutative operations. Some updates give the same result in any order, for example adding a transaction to a set keyed by
transactionId, or summing deltas into a counter. Then order doesn't matter, only uniqueness. - Re-read the source of truth. On any doubt (gap, stale event), fetch the current state from the system of record instead of trusting the event. This is simple and robust, but it adds load and coupling.
Event time vs processing time
occurredAt (when the posting happened in the ledger) and the time your consumer sees the event are different. Order by the sequence assigned by the system of record, not by timestamps. Clocks on different hosts disagree, and two postings can share a millisecond. Use timestamps only for display and for late-data windows in analytics.
Step-by-step implementation
Create the FIFO queue, DLQ and table
Run the AWS CLI commands: a FIFO queue with content-based deduplication off (you supply IDs), a FIFO DLQ, and a DynamoDB table hl-m08-balances keyed by accountId.
Write the projector and its tests first
BalanceProjector is pure C# with no AWS dependency. It decides Apply, Duplicate/Stale or Gap for each event. Write the xUnit tests from Complete code and make them pass. That logic is the heart of correctness.
Build the producer
A small console app sends ten BalanceChanged events for two accounts, deliberately including a duplicate send. Note how SendMessageBatch returns the same MessageId for the duplicate within five minutes.
Build the Lambda consumer
Deploy the function with an SQS FIFO event source mapping (batch size 10, ReportBatchItemFailures on). The function applies events through a DynamoDB conditional update and stops a group at its first failure.
Prove it
Replay an old event by hand (send sequence 3 again with a new deduplication ID). Confirm the projection ignores it and logs Stale. Then send sequence 12 while the table is at 10, and watch the function report a gap.
AWS Console instructions
- Console → Amazon SQS → Create queue → FIFO. Name
hl-m08-ledger-events-dlq.fifo; keep encryption on; add the lab tags. Create. - Create queue → FIFO
hl-m08-ledger-events.fifo. Leave Content-based deduplication off. Set Visibility timeout to 60 seconds. Under Dead-letter queue choose the FIFO DLQ with Maximum receives 5. (A FIFO queue's DLQ must also be FIFO.) Optionally set Deduplication scope to Message group and FIFO throughput limit to Per message group ID, which together enable high-throughput mode. - Console → DynamoDB → Create table
hl-m08-balances, partition keyaccountId(String), on-demand capacity, encryption with an AWS owned key (default) or a customer managed key for production. - Console → Lambda → (function) → Add trigger → SQS, choose the FIFO queue, batch size 10, and tick Report batch item failures.
- To observe order: CloudWatch → Logs → Log groups → /aws/lambda/hl-m08-balance-projector, filter on an account ID.
AWS CLI
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
TAGS='Project=cea-lab,Lab=m08-ordering,Environment=learn,Owner=alex'
# FIFO DLQ (must be FIFO when the source queue is FIFO)
DLQ_URL=$(aws sqs create-queue --queue-name hl-m08-ledger-events-dlq.fifo \
--attributes FifoQueue=true,SqsManagedSseEnabled=true,MessageRetentionPeriod=1209600 \
--tags "$TAGS" --query QueueUrl --output text)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" --attribute-names QueueArn \
--query Attributes.QueueArn --output text)
# Source FIFO queue: explicit dedup IDs, high-throughput settings, redrive after 5 receives
cat > fifo.json <<EOF
{
"FifoQueue": "true",
"ContentBasedDeduplication": "false",
"DeduplicationScope": "messageGroup",
"FifoThroughputLimit": "perMessageGroupId",
"VisibilityTimeout": "60",
"SqsManagedSseEnabled": "true",
"RedrivePolicy": "{\"deadLetterTargetArn\":\"$DLQ_ARN\",\"maxReceiveCount\":\"5\"}"
}
EOF
Q_URL=$(aws sqs create-queue --queue-name hl-m08-ledger-events.fifo \
--attributes file://fifo.json --tags "$TAGS" --query QueueUrl --output text)
# Projection table
aws dynamodb create-table --table-name hl-m08-balances \
--attribute-definitions AttributeName=accountId,AttributeType=S \
--key-schema AttributeName=accountId,KeyType=HASH \
--billing-mode PAY_PER_REQUEST \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m08-ordering Key=Environment,Value=learn Key=Owner,Value=alex
# Send the same event twice with the same dedup ID: the second is accepted but not enqueued again
BODY='{"accountId":"ACC-0000012345","accountSequence":1,"transactionId":"0192d1c2-7a00-7000-8000-000000000001","amount":2400.00,"balanceAfter":2400.00,"currency":"GBP","occurredAt":"2026-11-18T09:00:00Z"}'
for i in 1 2; do
aws sqs send-message --queue-url "$Q_URL" --message-body "$BODY" \
--message-group-id ACC-0000012345 \
--message-deduplication-id 0192d1c2-7a00-7000-8000-000000000001 \
--query '[MessageId,SequenceNumber]' --output text
done
# Both calls print the same MessageId — the duplicate send was absorbed.
# Event source mapping with partial batch responses
aws lambda create-event-source-mapping --function-name hl-m08-balance-projector \
--event-source-arn "$(aws sqs get-queue-attributes --queue-url "$Q_URL" --attribute-names QueueArn --query Attributes.QueueArn --output text)" \
--batch-size 10 --function-response-types ReportBatchItemFailuresSQS FIFO returns a SequenceNumber for each message. It is unique per message group and increases with each send, but it is not your business sequence: it is assigned at send time, so a resend after the window gets a new one. Use your own accountSequence from the ledger.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.Lambda.EventSources;
using Amazon.CDK.AWS.SQS;
using Constructs;
namespace Harbourline.Infra;
public sealed class OrderingStack : Stack
{
public OrderingStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
var dlq = new Queue(this, "LedgerDlq", new QueueProps
{
QueueName = "hl-m08-ledger-events-dlq.fifo",
Fifo = true,
Encryption = QueueEncryption.SQS_MANAGED,
RetentionPeriod = Duration.Days(14),
EnforceSSL = true
});
var queue = new Queue(this, "LedgerEvents", new QueueProps
{
QueueName = "hl-m08-ledger-events.fifo",
Fifo = true,
ContentBasedDeduplication = false,
DeduplicationScope = DeduplicationScope.MESSAGE_GROUP,
FifoThroughputLimit = FifoThroughputLimit.PER_MESSAGE_GROUP_ID,
VisibilityTimeout = Duration.Seconds(60),
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 }
});
var table = new TableV2(this, "Balances", new TablePropsV2
{
TableName = "hl-m08-balances",
PartitionKey = new Attribute { Name = "accountId", Type = AttributeType.STRING },
Billing = Billing.OnDemand(),
RemovalPolicy = RemovalPolicy.DESTROY // lab only
});
var fn = new Function(this, "Projector", new FunctionProps
{
FunctionName = "hl-m08-balance-projector",
Runtime = Runtime.DOTNET_10,
Handler = "Harbourline.Accounts.Projection::Harbourline.Accounts.Projection.Function::Handle",
Code = Code.FromAsset("../src/Harbourline.Accounts.Projection/publish"),
MemorySize = 512,
Timeout = Duration.Seconds(10),
Environment = new Dictionary<string, string> { ["BALANCES_TABLE"] = table.TableName }
});
table.GrantReadWriteData(fn);
fn.AddEventSource(new SqsEventSource(queue, new SqsEventSourceProps
{
BatchSize = 10,
ReportBatchItemFailures = true
}));
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m08-ordering");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "alex");
}
}Complete code
The event and the pure projector
namespace Harbourline.Accounts.Projection;
/// <summary>State-carrying event: includes the balance after the posting.</summary>
public sealed record BalanceChanged(
string AccountId,
long AccountSequence, // assigned by Core Banking, +1 per posting, per account
string TransactionId, // UUID v7
decimal Amount, // positive = credit, negative = debit
decimal BalanceAfter,
string Currency,
DateTimeOffset OccurredAt);namespace Harbourline.Accounts.Projection;
public enum Decision { Apply, Stale, Gap }
public sealed record ProjectionState(long LastSequence, decimal Balance);
public static class BalanceProjector
{
/// <param name="requireContiguous">
/// true for delta-style consumers that must not skip a posting;
/// false for state-carrying events where the newest event is enough.
/// </param>
public static Decision Decide(ProjectionState? current, BalanceChanged e, bool requireContiguous)
{
var last = current?.LastSequence ?? 0;
if (e.AccountSequence <= last) return Decision.Stale; // duplicate or out-of-date
if (requireContiguous && e.AccountSequence != last + 1) return Decision.Gap;
return Decision.Apply;
}
}The Lambda consumer
using System.Globalization;
using System.Text.Json;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
using Amazon.Lambda.Core;
using Amazon.Lambda.SQSEvents;
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]
namespace Harbourline.Accounts.Projection;
public sealed class GapDetectedException(string message) : Exception(message);
public sealed class Function
{
private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
private readonly IAmazonDynamoDB _ddb;
private readonly string _table;
public Function() : this(new AmazonDynamoDBClient(),
Environment.GetEnvironmentVariable("BALANCES_TABLE") ?? "hl-m08-balances") { }
public Function(IAmazonDynamoDB ddb, string table) => (_ddb, _table) = (ddb, table);
public async Task<SQSBatchResponse> Handle(SQSEvent sqsEvent, ILambdaContext context)
{
var failures = new List<SQSBatchResponse.BatchItemFailure>();
var blockedGroups = new HashSet<string>();
foreach (var record in sqsEvent.Records)
{
var group = record.Attributes.GetValueOrDefault("MessageGroupId") ?? "";
// FIFO rule: once a message in a group fails, every later message of that group
// in this batch must also be reported as failed, to keep the order.
if (blockedGroups.Contains(group))
{
failures.Add(new SQSBatchResponse.BatchItemFailure { ItemIdentifier = record.MessageId });
continue;
}
try
{
var e = JsonSerializer.Deserialize<BalanceChanged>(record.Body, Json)
?? throw new JsonException("empty body");
var outcome = await ApplyAsync(e);
context.Logger.LogInformation(
$"account={e.AccountId} seq={e.AccountSequence} outcome={outcome}");
}
catch (Exception ex)
{
context.Logger.LogError($"message={record.MessageId} group={group} error={ex.GetType().Name}");
blockedGroups.Add(group);
failures.Add(new SQSBatchResponse.BatchItemFailure { ItemIdentifier = record.MessageId });
}
}
return new SQSBatchResponse { BatchItemFailures = failures };
}
private async Task<Decision> ApplyAsync(BalanceChanged e)
{
// State-carrying events: write if newer. The condition makes the check atomic,
// so two concurrent writers can never move the balance backwards.
try
{
await _ddb.UpdateItemAsync(new UpdateItemRequest
{
TableName = _table,
Key = new() { ["accountId"] = new AttributeValue { S = e.AccountId } },
UpdateExpression =
"SET balance = :bal, currency = :cur, lastSequence = :seq, lastTransactionId = :tx, asOf = :at",
ConditionExpression = "attribute_not_exists(lastSequence) OR lastSequence < :seq",
ExpressionAttributeValues = new()
{
[":bal"] = new AttributeValue { N = e.BalanceAfter.ToString(CultureInfo.InvariantCulture) },
[":cur"] = new AttributeValue { S = e.Currency },
[":seq"] = new AttributeValue { N = e.AccountSequence.ToString(CultureInfo.InvariantCulture) },
[":tx"] = new AttributeValue { S = e.TransactionId },
[":at"] = new AttributeValue { S = e.OccurredAt.ToString("O") }
}
});
return Decision.Apply;
}
catch (ConditionalCheckFailedException)
{
// Already at this sequence or newer: a duplicate or a late event. Success, not failure.
return Decision.Stale;
}
}
}The producer
using System.Text.Json;
using Amazon.SQS;
using Amazon.SQS.Model;
using Harbourline.Accounts.Projection;
var queueUrl = args.Length > 0 ? args[0] : throw new ArgumentException("Pass the FIFO queue URL");
var sqs = new AmazonSQSClient(Amazon.RegionEndpoint.USEast1);
var json = new JsonSerializerOptions(JsonSerializerDefaults.Web);
var events = new List<BalanceChanged>();
decimal a = 0m, b = 0m;
for (var seq = 1; seq <= 5; seq++)
{
a += seq == 1 ? 2400.00m : -60.00m;
b += 125.50m;
events.Add(new("ACC-0000012345", seq, Guid.CreateVersion7().ToString(), seq == 1 ? 2400.00m : -60.00m, a, "GBP", DateTimeOffset.UtcNow));
events.Add(new("ACC-0000067890", seq, Guid.CreateVersion7().ToString(), 125.50m, b, "GBP", DateTimeOffset.UtcNow));
}
events.Add(events[2]); // deliberate duplicate send (same transactionId → same dedup ID)
foreach (var chunk in events.Chunk(10)) // SQS batch limit: 10 entries
{
var response = await sqs.SendMessageBatchAsync(new SendMessageBatchRequest
{
QueueUrl = queueUrl,
Entries = chunk.Select((e, i) => new SendMessageBatchRequestEntry
{
Id = i.ToString(),
MessageBody = JsonSerializer.Serialize(e, json),
MessageGroupId = e.AccountId, // order per account
MessageDeduplicationId = e.TransactionId // absorb producer retries for 5 minutes
}).ToList()
});
foreach (var f in response.Failed ?? [])
Console.Error.WriteLine($"FAILED entry {f.Id}: {f.Code} {f.Message} (sender fault: {f.SenderFault})");
foreach (var s in response.Successful ?? [])
Console.WriteLine($"sent {s.Id} messageId={s.MessageId} seq={s.SequenceNumber}");
}Tests (no AWS needed)
using Harbourline.Accounts.Projection;
using Xunit;
public class BalanceProjectorTests
{
private static BalanceChanged Evt(long seq) =>
new("ACC-0000012345", seq, Guid.CreateVersion7().ToString(), -60.00m, 100m, "GBP", DateTimeOffset.UtcNow);
[Fact]
public void First_event_is_applied() =>
Assert.Equal(Decision.Apply, BalanceProjector.Decide(null, Evt(1), requireContiguous: true));
[Theory]
[InlineData(5, 5)] // duplicate
[InlineData(5, 3)] // late, out of order
public void Older_or_equal_sequence_is_stale(long last, long incoming) =>
Assert.Equal(Decision.Stale,
BalanceProjector.Decide(new ProjectionState(last, 0m), Evt(incoming), requireContiguous: false));
[Fact]
public void Gap_is_detected_only_for_delta_consumers()
{
var state = new ProjectionState(10, 0m);
Assert.Equal(Decision.Gap, BalanceProjector.Decide(state, Evt(12), requireContiguous: true));
Assert.Equal(Decision.Apply, BalanceProjector.Decide(state, Evt(12), requireContiguous: false));
}
[Fact]
public void Shuffled_stream_converges_to_latest_state()
{
var shuffled = new[] { 3L, 1, 5, 2, 5, 4, 1 };
ProjectionState? state = null;
foreach (var seq in shuffled)
if (BalanceProjector.Decide(state, Evt(seq), requireContiguous: false) == Decision.Apply)
state = new ProjectionState(seq, 0m);
Assert.Equal(5, state!.LastSequence);
}
}The test project uses xunit.v3 4.0.3 on Microsoft.Testing.Platform, as in every lab in this course.
Code explained line by line
AccountSequence— the ordering key's version number, assigned by the system of record (Core Banking) at posting time. It is the only reliable order. Timestamps and SQS sequence numbers are not.BalanceAfter— makes the event state-carrying. A consumer that misses event 4 but receives 5 still has the right balance. Delta-only events would force gap handling.BalanceProjector.Decide— pure function, tested without AWS.<= lastmeans duplicate or late, so ignore it.requireContiguousswitches between the two consumer styles.record.Attributes.GetValueOrDefault("MessageGroupId")— Lambda passes SQS system attributes, includingMessageGroupIdfor FIFO queues, in each record.blockedGroups— the FIFO partial-batch rule. After the first failure in a group, every later record of that group is reported as failed too, so Lambda returns them to the queue in order.ConditionExpression = "attribute_not_exists(lastSequence) OR lastSequence < :seq"— the version check happens inside DynamoDB, atomically. Even if two invocations race (for example during a replay), the balance can never go backwards.catch (ConditionalCheckFailedException)→Stale— a failed condition is a normal outcome here, not an error. Treating it as an error would send harmless duplicates to the DLQ.ToString(CultureInfo.InvariantCulture)— DynamoDB numbers are strings. Never let a server's locale write2400,00.MessageGroupId = e.AccountId— per-account ordering and per-account parallelism.MessageDeduplicationId = e.TransactionId— a business ID, so a retried send after a timeout is recognised as the same message. A random GUID per attempt would defeat deduplication.response.Failed ?? []—SendMessageBatchcan partly succeed. Always checkFailed. SDK v4 returnsnullrather than empty lists.events.Chunk(10)— SQS batches hold at most 10 messages.
Security considerations
- Ordering keys can leak data.
MessageGroupIdvalues appear in logs, metrics and DLQ tooling. An account ID is acceptable inside the bank; never use names, emails or card numbers as group IDs or deduplication IDs. - Events carry balances, which are personal data. Encrypt queues and tables (SSE-SQS or SSE-KMS; DynamoDB encryption is always on). Restrict who can read the DLQ, because people browse DLQs during incidents.
- Sequence numbers must come from a trusted source. If a producer could set
accountSequencearbitrarily high, it could "freeze" a projection by making every real event look stale. Only the ledger service may publish to this queue: use a queue policy that allowssqs:SendMessageonly from its role. - Least privilege for the projector:
dynamodb:UpdateItemon the one table, plus the SQS permissions the event source mapping needs (sqs:ReceiveMessage,sqs:DeleteMessage,sqs:GetQueueAttributes).
⚠ Learning shortcut — not for production
The projector trusts BalanceAfter from the event and uses last-writer-wins by sequence. That's fine for a display balance. For anything that decides (overdraft checks, payment authorisation), read the authoritative balance from the ledger. Also alert when the projection's lastSequence lags the ledger's by more than a threshold, because a parked or dead-lettered event means the displayed balance is stale. Add asOfSequence to the API response so the app can show "updated a moment ago" honestly.
Banking use case
Harbourline's statements service builds monthly PDF statements from postings. Requirements:
- Every posting exactly once, in ledger order. A statement with a missing or doubled line is a regulatory problem.
- Late postings (for example a card settlement two days after authorisation) must land in the correct statement period by booking date, not arrival time.
The design:
- Core Banking writes postings to DynamoDB; DynamoDB Streams gives each change exactly once in the stream, ordered per item.
- Because postings are separate items, the stream does not order them across an account. The statement builder therefore sorts by
accountSequencewhen it builds a period, and checks the sequence is contiguous (no gaps) before it issues the statement. - A gap holds the statement and raises a ticket instead of issuing an incomplete document.
- The builder stores postings keyed by
transactionId, so a redelivered batch (Lambda can invoke a stream batch more than once) is a harmless overwrite.
Whether the statement process meets specific regulatory obligations is for Harbourline's compliance team to confirm.
Common mistakes
- Believing "FIFO means no duplicates, ever" → consumers skip idempotency → a crash between processing and delete reprocesses the message → make handlers idempotent anyway.
- Using a random deduplication ID per send attempt → retries create duplicates → derive it from a business ID (
transactionId,paymentId). - One message group for the whole queue → throughput of a single consumer, and one poison message stops everything → group by the narrowest key that gives the order you need.
- Reporting only the failed message on a FIFO partial batch → later messages of the group are deleted as "successful" and processed out of order → report the failure and all later messages of that group.
- Ordering by timestamp → clock skew and identical timestamps → order by a sequence from the system of record.
- Assuming DynamoDB Streams orders across items → it orders per item only → carry a sequence and check it.
- Treating
ConditionalCheckFailedExceptionas an error → duplicates fill the DLQ → treat it as "already applied". - Expecting EventBridge Classic rules to keep order → it never did → use SQS FIFO, Kinesis, or a FIFO subscriber on the newer Custom Event Bus.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
MissingParameter for MessageGroupId | Sending to a FIFO queue without a group ID | Always set MessageGroupId on FIFO sends |
InvalidParameterValue about deduplication ID | Content-based dedup is off and no MessageDeduplicationId was given | Supply one, or enable content-based deduplication |
| One account's events stall while others flow | Head-of-line blocking: a failing message in that group | Check logs for the group; fix or let it reach the DLQ; consider a lower maxReceiveCount |
| Projection shows an old balance after a replay | Version check missing; replay overwrote newer state | Add the conditional lastSequence < :seq update |
| Duplicates still processed with a FIFO queue | Consumer failed after processing but before delete, or resend after 5 minutes | Idempotent handler and version check |
| FIFO throughput lower than expected | Few message groups, or high-throughput mode off | Use more groups (narrower key); set DeduplicationScope=messageGroup and FifoThroughputLimit=perMessageGroupId |
| Messages appear processed out of order with Lambda + FIFO | Partial batch response reports only the failed message | Report all later messages in the same group as failed |
| Can't set a FIFO queue as an EventBridge target DLQ | EventBridge DLQs must be standard queues | Use a standard queue for the target DLQ |
Cost and cleanup
| Item | Cost at lab volume |
|---|---|
| SQS FIFO requests | Priced slightly higher than standard per million ⚠ VERIFY current rates; a few hundred requests here are within the free allowance |
| DynamoDB on-demand | A few hundred writes — fractions of a cent, or within the always-free tier |
| Lambda | Within the always-free tier |
The free alternative is to run the producer logic and BalanceProjectorTests locally. Shuffle events in a unit test instead of using a real queue.
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
UUID=$(aws lambda list-event-source-mappings --function-name hl-m08-balance-projector \
--query 'EventSourceMappings[0].UUID' --output text)
[ "$UUID" != "None" ] && aws lambda delete-event-source-mapping --uuid "$UUID"
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m08-ledger-events.fifo --query QueueUrl --output text)"
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m08-ledger-events-dlq.fifo --query QueueUrl --output text)"
aws dynamodb delete-table --table-name hl-m08-balances
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m08-ordering \
--query "ResourceTagMappingList[].ResourceARN"If you deployed the Lambda function, delete it too (aws lambda delete-function --function-name hl-m08-balance-projector) or destroy the CDK/Terraform stack.
Exercises
- 🟢 Guided: Run the producer twice within five minutes and once more after ten minutes. Count how many messages the queue received each time (
ApproximateNumberOfMessagesor Lambda logs) and explain the numbers. - 🟡 Challenge: Change the events to delta-only (remove
BalanceAfter) and the projector torequireContiguous: true. Implement gap handling: onGap, throw aGapDetectedExceptionso the message returns to the queue. Then design what happens if the missing event is in the DLQ. - 🔴 Live problem: Product wants the "recent transactions" list in the app to update in real time from EventBridge (Classic bus). The architect says "EventBridge doesn't do ordering, so we can't". The product owner says "just sort by timestamp in the app". Write a short design note with your recommendation.
Assignment
Produce a guarantees matrix for Harbourline's event flows in your portfolio repository. For each flow, record:
- the transport;
- the delivery guarantee;
- the ordering scope and key;
- the deduplication mechanism and window;
- the consumer-side protection (idempotency and version check).
Cover payments, notifications, balances and statements. Add the BalanceProjector and its tests.
Solutions
- First run: 11 sends, 10 messages enqueued. The duplicate of
events[2]shares aMessageDeduplicationIdwith an earlier entry, so SQS accepted it but didn't enqueue it again. - Second run within five minutes: the producer generates new transaction IDs (
Guid.CreateVersion7()), so all 10 are new messages; only its own internal duplicate is absorbed. The projector applies some and marks othersStale, because sequences 1–5 are already stored. - Third run after ten minutes: same result as the second.
Why: Transport deduplication only works when the ID represents the business event and the resend is within five minutes. The projector's version check is what protects the balance in the second and third runs.
var decision = BalanceProjector.Decide(current, e, requireContiguous: true);
switch (decision)
{
case Decision.Stale: return decision;
case Decision.Gap:
// Return the message to the queue; FIFO keeps the rest of this account's events behind it.
throw new GapDetectedException($"{e.AccountId}: have {current?.LastSequence}, got {e.AccountSequence}");
case Decision.Apply:
// UpdateItem: SET balance = balance + :amt, lastSequence = :seq
// CONDITION lastSequence = :prev (strict +1, atomic)
break;
}Use the condition lastSequence = :prev (where :prev is seq - 1) rather than <, because delta events must apply in exact sequence.
If the missing event is in the DLQ, the group is stuck: every later event loops until it too reaches the DLQ. Mitigations:
- alarm on DLQ depth with a short evaluation period;
- a runbook that fixes and redrives the missing event first;
- a "resync from ledger" operation that rebuilds the projection from the source of truth and sets
lastSequenceto the ledger's current value.
Why: Delta streams are unforgiving. This is why state-carrying events are usually preferable for projections.
Recommendation: don't sort by timestamp, and don't move everything to FIFO either.
- The transactions list is a set keyed by
transactionId, displayed sorted by the ledger'saccountSequence(or booking date plus sequence). Adding an item is commutative: order of arrival doesn't matter, only the sort key carried in the event. - Handle duplicates by upserting by
transactionId. Handle late events by inserting them at their sequence position. Handle reversals by updating the item's status (keyed by the same ID), guarded by a per-transaction version. - Timestamps from different producers are unreliable for ordering, so the sort key must come from the ledger.
- If a strictly ordered stream is genuinely needed later, use a FIFO subscriber with
EventGroupId = accountIdon the newer EventBridge Custom Event Bus (verify Region availability), or route through SQS FIFO.
Why: Most "we need ordering" requirements are really "we need a correct final view". Commutative, keyed updates give that without the throughput and head-of-line costs of strict ordering.
Knowledge check
Answer: It prevents a producer's retried send (same deduplication ID, within 5 minutes) from creating a second message. It also stops two consumers from processing the same message group at the same time. It does not protect against a consumer that processes a message but fails to delete it (redelivery), resends after 5 minutes, or new deduplication IDs per retry. Consumers still need idempotency.
Answer: It gives the order that matters (postings per account) with the most parallelism. A constant serialises everything, so one poison message stops all processing. customerId adds ordering across a customer's accounts that is rarely needed and reduces parallelism.
Answer: Each change record appears exactly once, and changes to the same item (primary key) appear in the order they happened. There is no ordering across different items, even with the same partition key.
Answer: Standard. It has exactly-once workflow execution: steps are not re-run unless you declare Retry. Asynchronous Express is at-least-once, so an execution can run more than once. Synchronous Express is at-most-once.
Answer: Messages 3, 4, 5 and 6 of that group. Returning only 3 would let Lambda delete 4–6, which then would not be redelivered after 3, breaking the order.
Interview questions
- "Is exactly-once delivery possible?" — Not end to end across independent systems without coordination. In practice, you build exactly-once effects from at-least-once delivery plus idempotent processing, using a deduplication key and atomic conditional writes. Services that advertise exactly-once do so within a scope (SQS FIFO: sends within 5 minutes).
- "How would you keep per-account order at scale?" — Partition by account: SQS FIFO message groups, Kinesis partition key or EventBridge
EventGroupId. Throughput scales with the number of accounts. Accept head-of-line blocking per account and keep DLQ handling fast. - "Events arrive out of order. What do you do?" — Carry a per-aggregate sequence from the source of truth. Apply only newer versions using a conditional write. Use state-carrying events where possible. If you use deltas, detect gaps and park or resync.
- "Why not just sort by timestamp?" — Clock skew, equal timestamps, late arrivals, and the event time vs processing time confusion. A sequence number from the system of record is unambiguous.
Certification practice
An application must process transaction messages for each account in the order they were sent, but messages for different accounts may be processed in parallel. Duplicate sends caused by producer retries within a few minutes must be ignored. Which solution meets these requirements?
A. SQS standard queue with long polling and a Lambda consumer.
B. SQS FIFO queue, MessageGroupId set to the account ID, MessageDeduplicationId set to the transaction ID.
C. SQS FIFO queue with a single MessageGroupId for all messages.
D. SNS standard topic with an SQS standard queue subscriber.
Answer: B. Message groups give per-account order with parallelism across accounts; the deduplication ID absorbs retries within the 5-minute interval. A and D don't guarantee order or deduplication. C orders everything globally and removes parallelism.
A bank orchestrates a payment process with several steps, including a call to an external scheme that must not be repeated. The process can take up to two days because of manual approvals. Which service configuration is MOST appropriate?
A. Step Functions Express workflow, asynchronous. B. Step Functions Standard workflow. C. Lambda function with a 15-minute timeout and SQS retries. D. Step Functions Express workflow, synchronous.
Answer: B. Standard workflows run up to one year with exactly-once execution, and support the callback pattern for approvals. Express workflows run for at most 5 minutes, and async Express is at-least-once (A, D). C can't wait two days, and its retries could repeat the scheme call.
A data engineer ingests card transactions into Kinesis Data Streams. Downstream analytics require transactions for each card to be processed in order, while maximising throughput across cards. What should the engineer do?
A. Use a random partition key for each record to spread load evenly. B. Use the card token as the partition key. C. Use a single shard for the stream. D. Use the transaction timestamp as the partition key.
Answer: B. Records with the same partition key go to the same shard, where order is preserved by sequence number; different cards spread across shards. A loses per-card order. C limits throughput to one shard. D scatters a card's transactions across shards.
Completion checklist
- I can define at-most-once, at-least-once and exactly-once, and explain "exactly-once processing"
- I can state each AWS messaging service's delivery and ordering guarantees from memory
- I chose an ordering key and can explain its throughput consequences
- I ran the producer and saw a duplicate send absorbed
- My consumer applies events with a conditional version check and handles FIFO partial failures correctly
- I deleted the queues, table and event source mapping
Key takeaways
- Exactly-once effects = at-least-once delivery + idempotent processing (+ ordering where needed). Build it; don't assume it.
- SQS/SNS FIFO and the newer EventBridge Custom Event Bus deduplicate within a 5-minute window. Anything longer is your consumer's job.
- Order per key, not globally.
accountIdis the usual key; throughput scales with the number of keys. - Carry a sequence from the system of record and apply events with an atomic conditional write. Prefer state-carrying events for projections.
- On FIFO partial batch failures, fail the message and everything after it in its group.
Further reading
- Amazon SQS FIFO queues
- Handling errors for an SQS event source in Lambda
- Exactly-once processing in Amazon SQS
- Amazon SQS message quotas
- Amazon SNS message delivery for FIFO topics
- Ordering and deduplicating events on a Custom Event Bus
- What is the EventBridge Custom Event Bus?
- Change data capture for DynamoDB Streams
- Choosing workflow type in Step Functions
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Retry strategies: backoff, jitter and avoiding retry storms
Design retries that help rather than hurt: classify errors, use exponential backoff with full jitter, configure AWS SDK for .NET retry modes, line up retries across SQS, Lambda, SNS, EventBridge and Step Functions, and stop layered retries from turning an outage into a retry storm.
Idempotency in .NET with DynamoDB and Powertools
Make every money-moving API call and every message handler safe to repeat: choose idempotency keys, build a DynamoDB idempotency store with conditional writes and TTL, combine business writes atomically with TransactWriteItems, and use the Powertools for AWS Lambda (.NET) idempotency utility.