Skip to Content
Cloud Engineering Academy
8. Messaging, Events & Microservices

Delivery guarantees and ordering

Know exactly what each AWS messaging service promises — at-most-once, at-least-once or exactly-once, ordered or not — and build .NET consumers that stay correct when messages arrive twice, late or out of order, using FIFO message groups, sequence numbers and version checks.

advanced⏱ 2 h 🟢 FreeDVA-C03SAA-C03DEA-C01

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SQS / AWSSDK.DynamoDBv2 4.1.0.2) · Amazon.Lambda.SQSEvents 3.0.1 · Amazon.Lambda.Serialization.SystemTextJson 3.0.1 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Define at-most-once, at-least-once and exactly-once, and explain why "exactly-once processing" in practice means at-least-once delivery plus idempotent handling.
  • State the delivery and ordering guarantees of SQS standard and FIFO, SNS standard and FIFO, EventBridge (Classic and the newer Custom Event Bus), Kinesis Data Streams, DynamoDB Streams, Lambda async invocation and Step Functions.
  • Choose an ordering key (FIFO message group, partition key, event group) that gives you the order you need without killing throughput.
  • Handle out-of-order and duplicate events in .NET with per-aggregate sequence numbers and DynamoDB conditional writes.
  • Configure an SQS FIFO → Lambda consumer that keeps per-account order and reports partial batch failures correctly.

Prerequisites

  • The SQS, SNS, EventBridge and Kinesis lessons earlier in this module (see the module overview)
  • Retry strategies — retries are the main source of duplicates
  • Module 6: Databases — DynamoDB conditional writes
  • Tools: .NET 10 SDK, AWS CLI v2 with the learn-sandbox profile

Why this matters in banking

Harbourline's mobile app shows a live balance fed by BalanceChanged events. Two incidents in one month:

  1. The phantom balance. A customer's salary of £2,400.00 arrived, then a £60.00 direct debit went out. The events arrived at the app's projection service in the wrong order. For eleven seconds the app showed a balance £60 higher than it had ever really been, and the customer moved money based on it. The direct debit then failed for insufficient funds.
  2. The double notification. A standard SQS queue delivered one CardPaymentAuthorised event twice. The notification service had no duplicate check, so the customer received two "you paid £18.40 at Example Coffee" messages. They called to report fraud.

Neither incident was a bug in AWS. Both services behaved exactly as documented. The design simply assumed guarantees the services never gave.

For ledgers, balances, statements and regulatory reporting, you must know which order and which uniqueness you can rely on, and design for the rest.

The idea in plain language

A messaging system can promise three levels of delivery:

GuaranteePlain meaningWhat you must handle
At-most-onceEach message is delivered once or not at allLost messages
At-least-onceEach message is delivered one or more timesDuplicates
Exactly-onceEach message is delivered onceIn distributed systems, only within a stated scope and time window

Ordering also comes in levels:

  • No ordering (best effort): messages usually arrive roughly in order, but nothing is promised.
  • Per-key ordering (partial order): messages with the same key, such as one account ID, arrive in the order sent. Messages with different keys are independent.
  • Total ordering: every message in the system in one global sequence. This is expensive and rarely needed. Banks usually need per-account order, not global order.

Analogy: Think of the post. Ordinary post is at-least-once with no ordering: letters sometimes overtake each other, and occasionally a returned letter is re-sent. Recorded delivery with a sequence number on each envelope lets the recipient say "I already have letter 41, and 43 can't be filed until 42 arrives". The postal service hasn't become perfect. The recipient has become careful. That is what your consumers must do.

The most important sentence in this lesson:

Exactly-once processing is something you build. It equals at-least-once delivery (nothing is lost) plus idempotent processing (duplicates have no extra effect), and, where order matters, per-key ordering or version checks.

Even when a service offers "exactly-once", that promise covers only a specific scope. SQS FIFO deduplicates sends within a five-minute window. Your Lambda can still crash after updating DynamoDB and before the message is deleted, so it sees the message again.

What each AWS service guarantees

Checked against AWS documentation on 2026-11-18.

ServiceDeliveryOrderingDeduplicationNotes
SQS standardAt-least-onceBest effortNoneNearly unlimited throughput. A message can occasionally be delivered more than once
SQS FIFOExactly-once processing within the deduplication intervalStrict, per MessageGroupIdMessageDeduplicationId or content-based (SHA-256 of body), 5-minute interval300 API calls/s per partition without high-throughput mode (3,000 messages/s with batches of 10); high-throughput mode raises this greatly
SNS standardAt-least-onceNoneNoneFan-out to many protocols
SNS FIFOOrdered, deduplicated delivery to SQS FIFO subscribersPer message groupSame 5-minute model as SQS FIFODelivers only to SQS (FIFO or standard) queues; standard queue subscribers get best-effort ordering and at-least-once
EventBridge — Custom Event Bus - Classic (rules and targets)At-least-onceNoneNoneRetries up to 24 h / 185 attempts by default
EventBridge — Custom Event Bus (subscribers, launched 2026)At-least-once to targets; publish-time dedupFIFO per EventGroupId per FIFO subscriberDeduplicationId or content-based, 5-minute windowRetention and replay by subscriber starting position. Newer service — check availability and SDK support in your Region ⚠ VERIFY
Kinesis Data StreamsAt-least-once (producer and consumer retries both create duplicates)Strict per shard, by sequence number; records with the same partition key go to the same shardNoneReplay by position within the retention period
DynamoDB StreamsEach change record appears exactly once in the streamPer item (primary key)Not needed in the stream itself24-hour retention. Your Lambda consumer can still be invoked more than once for a batch
Lambda async invocationAt-least-once (duplicates possible even without errors)NoneNoneEvent age and retry limits, destinations
Step Functions StandardExactly-once workflow execution (tasks re-run only if you declare Retry)Within an executionStarting an execution with the same name as a running one is idempotentGood for non-idempotent steps such as payments
Step Functions ExpressAsync: at-least-once. Sync: at-most-onceWithin an executionNone automaticMake every task idempotent

What "exactly-once processing" in SQS FIFO really covers: if a producer retries SendMessage with the same deduplication ID within five minutes, SQS doesn't create a second message. While one consumer holds a message, others can't receive messages from the same group. It does not cover:

  • a producer that resends after more than five minutes;
  • a producer that generates a new deduplication ID on retry;
  • a consumer that processes a message and then fails to delete it before the visibility timeout ends.

That last one is the classic duplicate, and SQS FIFO will deliver that message again by design.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Service Bus sessions (SessionId) for ordered processingSQS FIFO MessageGroupId; SNS FIFO message groups; EventBridge Custom Event Bus EventGroupIdSame idea: order per key. SQS doesn't need an explicit session lock — receiving from a group blocks that group for other consumers
Service Bus duplicate detection (configurable window, up to days)SQS FIFO / SNS FIFO deduplicationAWS window is fixed at 5 minutes — longer-term dedup is your job (DynamoDB idempotency table)
Service Bus PeekLock + CompleteSQS receive + DeleteMessage (visibility timeout)Same at-least-once outcome if you crash before completing
Event Hubs partitions + partition keyKinesis shards + partition key; MSK partitionsKinesis sequence numbers are per shard; order is per partition key
Event Grid (at-least-once, no ordering)EventBridge ClassicBoth need idempotent, order-tolerant handlers
Cosmos DB change feed (ordered per logical partition key)DynamoDB Streams (ordered per item)DynamoDB orders per item, not per partition key
Durable Functions (replay-based, activities at-least-once)Step Functions Standard (exactly-once execution) / Express (at-least-once)Choose the workflow type by whether steps are idempotent

Architecture

The design below keeps Harbourline's balance projection correct, even though messages may arrive twice or out of order.

Three layers of protection work together:

  1. Ordering at the transport. All events for one account share a message group, so SQS FIFO delivers them in order and never hands two messages from the same group to different consumers at once.
  2. Deduplication at the transport. The transaction ID as MessageDeduplicationId absorbs producer retries within five minutes.
  3. Version checks at the consumer. Every event carries accountSequence, assigned by the system of record. The projection applies an event only if its sequence is newer than the stored one. This catches everything the transport doesn't: duplicates after five minutes, replays from a DLQ or archive, and events that arrive by another path.

Layer 3 alone would keep the data correct, even with a standard queue. Layers 1 and 2 reduce wasted work and make the intermediate states correct. Your requirements decide whether you need them.

Ordering in depth

Choosing the ordering key

The key decides both correctness and throughput:

KeyOrder guaranteed forParallelismHarbourline verdict
One constant ("all")Everything, globallyOne message at a timeNever. A single slow message stops the bank
customerIdAll of a customer's accountsOne stream per customerOnly if cross-account order matters (it rarely does)
accountIdEach account's postingsOne stream per accountUsual choice for balances and statements
paymentIdEach payment's life cycle (received → screened → settled)One stream per paymentGood for payment state machines
transactionIdNothing useful (each group has one message)MaximumEquivalent to no ordering

SQS FIFO in high-throughput mode spreads message groups across partitions. Throughput therefore scales with the number of active groups, and a single hot group is limited to what one consumer can process in sequence.

Head-of-line blocking

Ordering has a price. If message 42 for account A fails, message 43 for account A must wait, because delivering 43 first would break the order. With SQS FIFO, while 42 is in flight or invisible, the group is blocked. Other accounts' groups keep flowing.

Implications:

  • A poison message blocks its group until it reaches the DLQ. Keep maxReceiveCount and the visibility timeout modest on FIFO queues.
  • Once a message from a group goes to the DLQ, the next message in the group is delivered. Your consumer must then deal with the gap: it sees sequence 43 while the projection is at 41.
  • On the new EventBridge Custom Event Bus, a failed delivery on a FIFO subscriber holds back all groups on that subscriber until it succeeds or retries run out. The blast radius is larger than SQS FIFO's per-group blocking. Plan retry limits accordingly.

Lambda with SQS FIFO

With an SQS FIFO event source mapping, Lambda delivers batches in order within each group, and doesn't process the same group concurrently. With partial batch responses, the rule differs from standard queues: when one message fails, stop processing that group and report the failed message and every later message from the same group as failures. Otherwise Lambda deletes the later messages that "succeeded", and they are processed out of order or lost from the sequence. The Complete code section shows how.

Kinesis and DynamoDB Streams

  • Kinesis: order is per shard. Records with the same partition key go to the same shard, so choose accountId as the partition key for per-account order. With Lambda, ParallelizationFactor above 1 processes a shard with several concurrent invocations while still keeping order per partition key.
  • DynamoDB Streams: order is guaranteed per item (primary key), not per partition key and not across items. If one account's postings are separate items, the stream does not order them relative to each other. Put the account sequence in the data and check it.

Handling out-of-order and duplicate events

When the transport can't guarantee what you need, the consumer must. Four techniques, from most to least common:

  1. Version or sequence check (last-writer-wins by sequence). Each event carries a monotonically increasing number per aggregate (account, payment). Store the last applied number. Apply only newer events; ignore equal or older ones. This works perfectly when events carry the resulting state (balanceAfter = 2,340.00), because the newest event alone is enough.
  2. Gap detection and buffering. When events carry deltas (amount = -60.00), you can't skip one. If you receive 43 while at 41, park 43 (retry later, or store it) until 42 arrives. Set a timeout and an alert, because 42 may be in a DLQ.
  3. Commutative operations. Some updates give the same result in any order, for example adding a transaction to a set keyed by transactionId, or summing deltas into a counter. Then order doesn't matter, only uniqueness.
  4. Re-read the source of truth. On any doubt (gap, stale event), fetch the current state from the system of record instead of trusting the event. This is simple and robust, but it adds load and coupling.

Event time vs processing time

occurredAt (when the posting happened in the ledger) and the time your consumer sees the event are different. Order by the sequence assigned by the system of record, not by timestamps. Clocks on different hosts disagree, and two postings can share a millisecond. Use timestamps only for display and for late-data windows in analytics.

Step-by-step implementation

Create the FIFO queue, DLQ and table

Run the AWS CLI commands: a FIFO queue with content-based deduplication off (you supply IDs), a FIFO DLQ, and a DynamoDB table hl-m08-balances keyed by accountId.

Write the projector and its tests first

BalanceProjector is pure C# with no AWS dependency. It decides Apply, Duplicate/Stale or Gap for each event. Write the xUnit tests from Complete code and make them pass. That logic is the heart of correctness.

Build the producer

A small console app sends ten BalanceChanged events for two accounts, deliberately including a duplicate send. Note how SendMessageBatch returns the same MessageId for the duplicate within five minutes.

Build the Lambda consumer

Deploy the function with an SQS FIFO event source mapping (batch size 10, ReportBatchItemFailures on). The function applies events through a DynamoDB conditional update and stops a group at its first failure.

Prove it

Replay an old event by hand (send sequence 3 again with a new deduplication ID). Confirm the projection ignores it and logs Stale. Then send sequence 12 while the table is at 10, and watch the function report a gap.

AWS Console instructions

  1. Console → Amazon SQS → Create queue → FIFO. Name hl-m08-ledger-events-dlq.fifo; keep encryption on; add the lab tags. Create.
  2. Create queue → FIFO hl-m08-ledger-events.fifo. Leave Content-based deduplication off. Set Visibility timeout to 60 seconds. Under Dead-letter queue choose the FIFO DLQ with Maximum receives 5. (A FIFO queue's DLQ must also be FIFO.) Optionally set Deduplication scope to Message group and FIFO throughput limit to Per message group ID, which together enable high-throughput mode.
  3. Console → DynamoDB → Create table hl-m08-balances, partition key accountId (String), on-demand capacity, encryption with an AWS owned key (default) or a customer managed key for production.
  4. Console → Lambda → (function) → Add trigger → SQS, choose the FIFO queue, batch size 10, and tick Report batch item failures.
  5. To observe order: CloudWatch → Logs → Log groups → /aws/lambda/hl-m08-balance-projector, filter on an account ID.

AWS CLI

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
TAGS='Project=cea-lab,Lab=m08-ordering,Environment=learn,Owner=alex'

# FIFO DLQ (must be FIFO when the source queue is FIFO)
DLQ_URL=$(aws sqs create-queue --queue-name hl-m08-ledger-events-dlq.fifo \
  --attributes FifoQueue=true,SqsManagedSseEnabled=true,MessageRetentionPeriod=1209600 \
  --tags "$TAGS" --query QueueUrl --output text)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" --attribute-names QueueArn \
  --query Attributes.QueueArn --output text)

# Source FIFO queue: explicit dedup IDs, high-throughput settings, redrive after 5 receives
cat > fifo.json <<EOF
{
  "FifoQueue": "true",
  "ContentBasedDeduplication": "false",
  "DeduplicationScope": "messageGroup",
  "FifoThroughputLimit": "perMessageGroupId",
  "VisibilityTimeout": "60",
  "SqsManagedSseEnabled": "true",
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"$DLQ_ARN\",\"maxReceiveCount\":\"5\"}"
}
EOF
Q_URL=$(aws sqs create-queue --queue-name hl-m08-ledger-events.fifo \
  --attributes file://fifo.json --tags "$TAGS" --query QueueUrl --output text)

# Projection table
aws dynamodb create-table --table-name hl-m08-balances \
  --attribute-definitions AttributeName=accountId,AttributeType=S \
  --key-schema AttributeName=accountId,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m08-ordering Key=Environment,Value=learn Key=Owner,Value=alex

# Send the same event twice with the same dedup ID: the second is accepted but not enqueued again
BODY='{"accountId":"ACC-0000012345","accountSequence":1,"transactionId":"0192d1c2-7a00-7000-8000-000000000001","amount":2400.00,"balanceAfter":2400.00,"currency":"GBP","occurredAt":"2026-11-18T09:00:00Z"}'
for i in 1 2; do
  aws sqs send-message --queue-url "$Q_URL" --message-body "$BODY" \
    --message-group-id ACC-0000012345 \
    --message-deduplication-id 0192d1c2-7a00-7000-8000-000000000001 \
    --query '[MessageId,SequenceNumber]' --output text
done
# Both calls print the same MessageId — the duplicate send was absorbed.

# Event source mapping with partial batch responses
aws lambda create-event-source-mapping --function-name hl-m08-balance-projector \
  --event-source-arn "$(aws sqs get-queue-attributes --queue-url "$Q_URL" --attribute-names QueueArn --query Attributes.QueueArn --output text)" \
  --batch-size 10 --function-response-types ReportBatchItemFailures

SQS FIFO returns a SequenceNumber for each message. It is unique per message group and increases with each send, but it is not your business sequence: it is assigned at send time, so a resend after the window gets a new one. Use your own accountSequence from the ledger.

Infrastructure as Code

OrderingStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.Lambda.EventSources;
using Amazon.CDK.AWS.SQS;
using Constructs;

namespace Harbourline.Infra;

public sealed class OrderingStack : Stack
{
    public OrderingStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        var dlq = new Queue(this, "LedgerDlq", new QueueProps
        {
            QueueName = "hl-m08-ledger-events-dlq.fifo",
            Fifo = true,
            Encryption = QueueEncryption.SQS_MANAGED,
            RetentionPeriod = Duration.Days(14),
            EnforceSSL = true
        });

        var queue = new Queue(this, "LedgerEvents", new QueueProps
        {
            QueueName = "hl-m08-ledger-events.fifo",
            Fifo = true,
            ContentBasedDeduplication = false,
            DeduplicationScope = DeduplicationScope.MESSAGE_GROUP,
            FifoThroughputLimit = FifoThroughputLimit.PER_MESSAGE_GROUP_ID,
            VisibilityTimeout = Duration.Seconds(60),
            Encryption = QueueEncryption.SQS_MANAGED,
            EnforceSSL = true,
            DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 }
        });

        var table = new TableV2(this, "Balances", new TablePropsV2
        {
            TableName = "hl-m08-balances",
            PartitionKey = new Attribute { Name = "accountId", Type = AttributeType.STRING },
            Billing = Billing.OnDemand(),
            RemovalPolicy = RemovalPolicy.DESTROY // lab only
        });

        var fn = new Function(this, "Projector", new FunctionProps
        {
            FunctionName = "hl-m08-balance-projector",
            Runtime = Runtime.DOTNET_10,
            Handler = "Harbourline.Accounts.Projection::Harbourline.Accounts.Projection.Function::Handle",
            Code = Code.FromAsset("../src/Harbourline.Accounts.Projection/publish"),
            MemorySize = 512,
            Timeout = Duration.Seconds(10),
            Environment = new Dictionary<string, string> { ["BALANCES_TABLE"] = table.TableName }
        });
        table.GrantReadWriteData(fn);

        fn.AddEventSource(new SqsEventSource(queue, new SqsEventSourceProps
        {
            BatchSize = 10,
            ReportBatchItemFailures = true
        }));

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m08-ordering");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        Amazon.CDK.Tags.Of(this).Add("Owner", "alex");
    }
}

Complete code

The event and the pure projector

BalanceChanged.cs
namespace Harbourline.Accounts.Projection;

/// <summary>State-carrying event: includes the balance after the posting.</summary>
public sealed record BalanceChanged(
    string AccountId,
    long AccountSequence,      // assigned by Core Banking, +1 per posting, per account
    string TransactionId,      // UUID v7
    decimal Amount,            // positive = credit, negative = debit
    decimal BalanceAfter,
    string Currency,
    DateTimeOffset OccurredAt);
BalanceProjector.cs
namespace Harbourline.Accounts.Projection;

public enum Decision { Apply, Stale, Gap }

public sealed record ProjectionState(long LastSequence, decimal Balance);

public static class BalanceProjector
{
    /// <param name="requireContiguous">
    /// true for delta-style consumers that must not skip a posting;
    /// false for state-carrying events where the newest event is enough.
    /// </param>
    public static Decision Decide(ProjectionState? current, BalanceChanged e, bool requireContiguous)
    {
        var last = current?.LastSequence ?? 0;
        if (e.AccountSequence <= last) return Decision.Stale;          // duplicate or out-of-date
        if (requireContiguous && e.AccountSequence != last + 1) return Decision.Gap;
        return Decision.Apply;
    }
}

The Lambda consumer

Function.cs
using System.Globalization;
using System.Text.Json;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
using Amazon.Lambda.Core;
using Amazon.Lambda.SQSEvents;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace Harbourline.Accounts.Projection;

public sealed class GapDetectedException(string message) : Exception(message);

public sealed class Function
{
    private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
    private readonly IAmazonDynamoDB _ddb;
    private readonly string _table;

    public Function() : this(new AmazonDynamoDBClient(),
        Environment.GetEnvironmentVariable("BALANCES_TABLE") ?? "hl-m08-balances") { }

    public Function(IAmazonDynamoDB ddb, string table) => (_ddb, _table) = (ddb, table);

    public async Task<SQSBatchResponse> Handle(SQSEvent sqsEvent, ILambdaContext context)
    {
        var failures = new List<SQSBatchResponse.BatchItemFailure>();
        var blockedGroups = new HashSet<string>();

        foreach (var record in sqsEvent.Records)
        {
            var group = record.Attributes.GetValueOrDefault("MessageGroupId") ?? "";

            // FIFO rule: once a message in a group fails, every later message of that group
            // in this batch must also be reported as failed, to keep the order.
            if (blockedGroups.Contains(group))
            {
                failures.Add(new SQSBatchResponse.BatchItemFailure { ItemIdentifier = record.MessageId });
                continue;
            }

            try
            {
                var e = JsonSerializer.Deserialize<BalanceChanged>(record.Body, Json)
                        ?? throw new JsonException("empty body");
                var outcome = await ApplyAsync(e);
                context.Logger.LogInformation(
                    $"account={e.AccountId} seq={e.AccountSequence} outcome={outcome}");
            }
            catch (Exception ex)
            {
                context.Logger.LogError($"message={record.MessageId} group={group} error={ex.GetType().Name}");
                blockedGroups.Add(group);
                failures.Add(new SQSBatchResponse.BatchItemFailure { ItemIdentifier = record.MessageId });
            }
        }

        return new SQSBatchResponse { BatchItemFailures = failures };
    }

    private async Task<Decision> ApplyAsync(BalanceChanged e)
    {
        // State-carrying events: write if newer. The condition makes the check atomic,
        // so two concurrent writers can never move the balance backwards.
        try
        {
            await _ddb.UpdateItemAsync(new UpdateItemRequest
            {
                TableName = _table,
                Key = new() { ["accountId"] = new AttributeValue { S = e.AccountId } },
                UpdateExpression =
                    "SET balance = :bal, currency = :cur, lastSequence = :seq, lastTransactionId = :tx, asOf = :at",
                ConditionExpression = "attribute_not_exists(lastSequence) OR lastSequence < :seq",
                ExpressionAttributeValues = new()
                {
                    [":bal"] = new AttributeValue { N = e.BalanceAfter.ToString(CultureInfo.InvariantCulture) },
                    [":cur"] = new AttributeValue { S = e.Currency },
                    [":seq"] = new AttributeValue { N = e.AccountSequence.ToString(CultureInfo.InvariantCulture) },
                    [":tx"] = new AttributeValue { S = e.TransactionId },
                    [":at"] = new AttributeValue { S = e.OccurredAt.ToString("O") }
                }
            });
            return Decision.Apply;
        }
        catch (ConditionalCheckFailedException)
        {
            // Already at this sequence or newer: a duplicate or a late event. Success, not failure.
            return Decision.Stale;
        }
    }
}

The producer

Producer/Program.cs
using System.Text.Json;
using Amazon.SQS;
using Amazon.SQS.Model;
using Harbourline.Accounts.Projection;

var queueUrl = args.Length > 0 ? args[0] : throw new ArgumentException("Pass the FIFO queue URL");
var sqs = new AmazonSQSClient(Amazon.RegionEndpoint.USEast1);
var json = new JsonSerializerOptions(JsonSerializerDefaults.Web);

var events = new List<BalanceChanged>();
decimal a = 0m, b = 0m;
for (var seq = 1; seq <= 5; seq++)
{
    a += seq == 1 ? 2400.00m : -60.00m;
    b += 125.50m;
    events.Add(new("ACC-0000012345", seq, Guid.CreateVersion7().ToString(), seq == 1 ? 2400.00m : -60.00m, a, "GBP", DateTimeOffset.UtcNow));
    events.Add(new("ACC-0000067890", seq, Guid.CreateVersion7().ToString(), 125.50m, b, "GBP", DateTimeOffset.UtcNow));
}
events.Add(events[2]); // deliberate duplicate send (same transactionId → same dedup ID)

foreach (var chunk in events.Chunk(10)) // SQS batch limit: 10 entries
{
    var response = await sqs.SendMessageBatchAsync(new SendMessageBatchRequest
    {
        QueueUrl = queueUrl,
        Entries = chunk.Select((e, i) => new SendMessageBatchRequestEntry
        {
            Id = i.ToString(),
            MessageBody = JsonSerializer.Serialize(e, json),
            MessageGroupId = e.AccountId,              // order per account
            MessageDeduplicationId = e.TransactionId   // absorb producer retries for 5 minutes
        }).ToList()
    });

    foreach (var f in response.Failed ?? [])
        Console.Error.WriteLine($"FAILED entry {f.Id}: {f.Code} {f.Message} (sender fault: {f.SenderFault})");
    foreach (var s in response.Successful ?? [])
        Console.WriteLine($"sent {s.Id} messageId={s.MessageId} seq={s.SequenceNumber}");
}

Tests (no AWS needed)

tests/BalanceProjectorTests.cs
using Harbourline.Accounts.Projection;
using Xunit;

public class BalanceProjectorTests
{
    private static BalanceChanged Evt(long seq) =>
        new("ACC-0000012345", seq, Guid.CreateVersion7().ToString(), -60.00m, 100m, "GBP", DateTimeOffset.UtcNow);

    [Fact]
    public void First_event_is_applied() =>
        Assert.Equal(Decision.Apply, BalanceProjector.Decide(null, Evt(1), requireContiguous: true));

    [Theory]
    [InlineData(5, 5)] // duplicate
    [InlineData(5, 3)] // late, out of order
    public void Older_or_equal_sequence_is_stale(long last, long incoming) =>
        Assert.Equal(Decision.Stale,
            BalanceProjector.Decide(new ProjectionState(last, 0m), Evt(incoming), requireContiguous: false));

    [Fact]
    public void Gap_is_detected_only_for_delta_consumers()
    {
        var state = new ProjectionState(10, 0m);
        Assert.Equal(Decision.Gap, BalanceProjector.Decide(state, Evt(12), requireContiguous: true));
        Assert.Equal(Decision.Apply, BalanceProjector.Decide(state, Evt(12), requireContiguous: false));
    }

    [Fact]
    public void Shuffled_stream_converges_to_latest_state()
    {
        var shuffled = new[] { 3L, 1, 5, 2, 5, 4, 1 };
        ProjectionState? state = null;
        foreach (var seq in shuffled)
            if (BalanceProjector.Decide(state, Evt(seq), requireContiguous: false) == Decision.Apply)
                state = new ProjectionState(seq, 0m);
        Assert.Equal(5, state!.LastSequence);
    }
}

The test project uses xunit.v3 4.0.3 on Microsoft.Testing.Platform, as in every lab in this course.

Code explained line by line

  1. AccountSequence — the ordering key's version number, assigned by the system of record (Core Banking) at posting time. It is the only reliable order. Timestamps and SQS sequence numbers are not.
  2. BalanceAfter — makes the event state-carrying. A consumer that misses event 4 but receives 5 still has the right balance. Delta-only events would force gap handling.
  3. BalanceProjector.Decide — pure function, tested without AWS. <= last means duplicate or late, so ignore it. requireContiguous switches between the two consumer styles.
  4. record.Attributes.GetValueOrDefault("MessageGroupId") — Lambda passes SQS system attributes, including MessageGroupId for FIFO queues, in each record.
  5. blockedGroups — the FIFO partial-batch rule. After the first failure in a group, every later record of that group is reported as failed too, so Lambda returns them to the queue in order.
  6. ConditionExpression = "attribute_not_exists(lastSequence) OR lastSequence < :seq" — the version check happens inside DynamoDB, atomically. Even if two invocations race (for example during a replay), the balance can never go backwards.
  7. catch (ConditionalCheckFailedException) → Stale — a failed condition is a normal outcome here, not an error. Treating it as an error would send harmless duplicates to the DLQ.
  8. ToString(CultureInfo.InvariantCulture) — DynamoDB numbers are strings. Never let a server's locale write 2400,00.
  9. MessageGroupId = e.AccountId — per-account ordering and per-account parallelism.
  10. MessageDeduplicationId = e.TransactionId — a business ID, so a retried send after a timeout is recognised as the same message. A random GUID per attempt would defeat deduplication.
  11. response.Failed ?? [] — SendMessageBatch can partly succeed. Always check Failed. SDK v4 returns null rather than empty lists.
  12. events.Chunk(10) — SQS batches hold at most 10 messages.

Security considerations

  • Ordering keys can leak data. MessageGroupId values appear in logs, metrics and DLQ tooling. An account ID is acceptable inside the bank; never use names, emails or card numbers as group IDs or deduplication IDs.
  • Events carry balances, which are personal data. Encrypt queues and tables (SSE-SQS or SSE-KMS; DynamoDB encryption is always on). Restrict who can read the DLQ, because people browse DLQs during incidents.
  • Sequence numbers must come from a trusted source. If a producer could set accountSequence arbitrarily high, it could "freeze" a projection by making every real event look stale. Only the ledger service may publish to this queue: use a queue policy that allows sqs:SendMessage only from its role.
  • Least privilege for the projector: dynamodb:UpdateItem on the one table, plus the SQS permissions the event source mapping needs (sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueAttributes).

⚠ Learning shortcut — not for production

The projector trusts BalanceAfter from the event and uses last-writer-wins by sequence. That's fine for a display balance. For anything that decides (overdraft checks, payment authorisation), read the authoritative balance from the ledger. Also alert when the projection's lastSequence lags the ledger's by more than a threshold, because a parked or dead-lettered event means the displayed balance is stale. Add asOfSequence to the API response so the app can show "updated a moment ago" honestly.

Banking use case

Harbourline's statements service builds monthly PDF statements from postings. Requirements:

  • Every posting exactly once, in ledger order. A statement with a missing or doubled line is a regulatory problem.
  • Late postings (for example a card settlement two days after authorisation) must land in the correct statement period by booking date, not arrival time.

The design:

  1. Core Banking writes postings to DynamoDB; DynamoDB Streams gives each change exactly once in the stream, ordered per item.
  2. Because postings are separate items, the stream does not order them across an account. The statement builder therefore sorts by accountSequence when it builds a period, and checks the sequence is contiguous (no gaps) before it issues the statement.
  3. A gap holds the statement and raises a ticket instead of issuing an incomplete document.
  4. The builder stores postings keyed by transactionId, so a redelivered batch (Lambda can invoke a stream batch more than once) is a harmless overwrite.

Whether the statement process meets specific regulatory obligations is for Harbourline's compliance team to confirm.

Common mistakes

  • Believing "FIFO means no duplicates, ever" → consumers skip idempotency → a crash between processing and delete reprocesses the message → make handlers idempotent anyway.
  • Using a random deduplication ID per send attempt → retries create duplicates → derive it from a business ID (transactionId, paymentId).
  • One message group for the whole queue → throughput of a single consumer, and one poison message stops everything → group by the narrowest key that gives the order you need.
  • Reporting only the failed message on a FIFO partial batch → later messages of the group are deleted as "successful" and processed out of order → report the failure and all later messages of that group.
  • Ordering by timestamp → clock skew and identical timestamps → order by a sequence from the system of record.
  • Assuming DynamoDB Streams orders across items → it orders per item only → carry a sequence and check it.
  • Treating ConditionalCheckFailedException as an error → duplicates fill the DLQ → treat it as "already applied".
  • Expecting EventBridge Classic rules to keep order → it never did → use SQS FIFO, Kinesis, or a FIFO subscriber on the newer Custom Event Bus.

Troubleshooting

SymptomLikely causeFix
MissingParameter for MessageGroupIdSending to a FIFO queue without a group IDAlways set MessageGroupId on FIFO sends
InvalidParameterValue about deduplication IDContent-based dedup is off and no MessageDeduplicationId was givenSupply one, or enable content-based deduplication
One account's events stall while others flowHead-of-line blocking: a failing message in that groupCheck logs for the group; fix or let it reach the DLQ; consider a lower maxReceiveCount
Projection shows an old balance after a replayVersion check missing; replay overwrote newer stateAdd the conditional lastSequence < :seq update
Duplicates still processed with a FIFO queueConsumer failed after processing but before delete, or resend after 5 minutesIdempotent handler and version check
FIFO throughput lower than expectedFew message groups, or high-throughput mode offUse more groups (narrower key); set DeduplicationScope=messageGroup and FifoThroughputLimit=perMessageGroupId
Messages appear processed out of order with Lambda + FIFOPartial batch response reports only the failed messageReport all later messages in the same group as failed
Can't set a FIFO queue as an EventBridge target DLQEventBridge DLQs must be standard queuesUse a standard queue for the target DLQ

Cost and cleanup

ItemCost at lab volume
SQS FIFO requestsPriced slightly higher than standard per million ⚠ VERIFY current rates; a few hundred requests here are within the free allowance
DynamoDB on-demandA few hundred writes — fractions of a cent, or within the always-free tier
LambdaWithin the always-free tier

The free alternative is to run the producer logic and BalanceProjectorTests locally. Shuffle events in a unit test instead of using a real queue.

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
UUID=$(aws lambda list-event-source-mappings --function-name hl-m08-balance-projector \
  --query 'EventSourceMappings[0].UUID' --output text)
[ "$UUID" != "None" ] && aws lambda delete-event-source-mapping --uuid "$UUID"
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m08-ledger-events.fifo --query QueueUrl --output text)"
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m08-ledger-events-dlq.fifo --query QueueUrl --output text)"
aws dynamodb delete-table --table-name hl-m08-balances

aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m08-ordering \
  --query "ResourceTagMappingList[].ResourceARN"

If you deployed the Lambda function, delete it too (aws lambda delete-function --function-name hl-m08-balance-projector) or destroy the CDK/Terraform stack.

Exercises

  • 🟢 Guided: Run the producer twice within five minutes and once more after ten minutes. Count how many messages the queue received each time (ApproximateNumberOfMessages or Lambda logs) and explain the numbers.
  • 🟡 Challenge: Change the events to delta-only (remove BalanceAfter) and the projector to requireContiguous: true. Implement gap handling: on Gap, throw a GapDetectedException so the message returns to the queue. Then design what happens if the missing event is in the DLQ.
  • 🔴 Live problem: Product wants the "recent transactions" list in the app to update in real time from EventBridge (Classic bus). The architect says "EventBridge doesn't do ordering, so we can't". The product owner says "just sort by timestamp in the app". Write a short design note with your recommendation.

Assignment

Produce a guarantees matrix for Harbourline's event flows in your portfolio repository. For each flow, record:

  • the transport;
  • the delivery guarantee;
  • the ordering scope and key;
  • the deduplication mechanism and window;
  • the consumer-side protection (idempotency and version check).

Cover payments, notifications, balances and statements. Add the BalanceProjector and its tests.

Solutions

Knowledge check

Interview questions

  • "Is exactly-once delivery possible?" — Not end to end across independent systems without coordination. In practice, you build exactly-once effects from at-least-once delivery plus idempotent processing, using a deduplication key and atomic conditional writes. Services that advertise exactly-once do so within a scope (SQS FIFO: sends within 5 minutes).
  • "How would you keep per-account order at scale?" — Partition by account: SQS FIFO message groups, Kinesis partition key or EventBridge EventGroupId. Throughput scales with the number of accounts. Accept head-of-line blocking per account and keep DLQ handling fast.
  • "Events arrive out of order. What do you do?" — Carry a per-aggregate sequence from the source of truth. Apply only newer versions using a conditional write. Use state-carrying events where possible. If you use deltas, detect gaps and park or resync.
  • "Why not just sort by timestamp?" — Clock skew, equal timestamps, late arrivals, and the event time vs processing time confusion. A sequence number from the system of record is unambiguous.

Certification practice

Completion checklist

  • I can define at-most-once, at-least-once and exactly-once, and explain "exactly-once processing"
  • I can state each AWS messaging service's delivery and ordering guarantees from memory
  • I chose an ordering key and can explain its throughput consequences
  • I ran the producer and saw a duplicate send absorbed
  • My consumer applies events with a conditional version check and handles FIFO partial failures correctly
  • I deleted the queues, table and event source mapping

Key takeaways

  • Exactly-once effects = at-least-once delivery + idempotent processing (+ ordering where needed). Build it; don't assume it.
  • SQS/SNS FIFO and the newer EventBridge Custom Event Bus deduplicate within a 5-minute window. Anything longer is your consumer's job.
  • Order per key, not globally. accountId is the usual key; throughput scales with the number of keys.
  • Carry a sequence from the system of record and apply events with an atomic conditional write. Prefer state-carrying events for projections.
  • On FIFO partial batch failures, fail the message and everything after it in its group.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page