Skip to Content
Cloud Engineering Academy
8. Messaging, Events & Microservices

Module project: banking transaction notification system

Build, break and repair a notification system: EventBridge to SNS to SQS to Lambda in .NET, with idempotent handlers, retries with backoff, a poison message in a DLQ, redrive after a fix and recovery from a downstream outage, using the m08-notification-system lab.

advanced⏱ 4 h 🟢 FreeDVA-C03SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.EventBridge 4.1.0.2, AWSSDK.DynamoDBv2 4.1.0.2) · Amazon.Lambda.Core 3.4.0 · Amazon.Lambda.SQSEvents 3.0.1 · xUnit v3 4.0.3 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this project you can:

  • Run and read a complete EventBridge → SNS → SQS → Lambda pipeline for a banking notification, and explain what each hop is for.
  • Prove duplicate prevention: the same transaction delivered three times produces exactly one SMS and one e-mail, using a DynamoDB idempotency table.
  • Demonstrate retries with exponential backoff and full jitter inside a handler, and explain why long outages are left to SQS rather than to in-process retries.
  • Watch a poison message travel to a dead-letter queue after maxReceiveCount receives without blocking good messages, then redrive it after a fix.
  • Recover from a downstream outage with no lost and no repeated notifications.
  • Write and follow a short runbook for a DLQ alarm, and tear everything down.

Prerequisites

This project uses the whole module. Read the lessons first if you have not:

The code lives in labs/m08-notification-system/ (see its README.md and RUNBOOK.md). This page is the guided walkthrough.

Why this matters in banking

A customer pays £42.50 by card. Within seconds they expect an SMS: "debit of GBP 42.50 on account ending 0001". If it never arrives, they ring the contact centre. If it arrives three times, they ring the contact centre and suspect fraud. If it carries their full account number, a security reviewer will be called. If the SMS provider is down for an hour on salary day, and the system then sends the backlog 40,000 messages at once, the provider throttles you and the second wave fails too.

Notifications are a good first project because the stakes are real but bounded: no money moves, so a mistake costs trust, not funds. Yet every hard problem in this module is present in miniature: at-least-once delivery, duplicates, malformed input, partial outages, backlog recovery and the need to prove afterwards what happened.

Design note for banks

Notification text (amounts, partial account numbers) is still personal data. Which channels may carry which data, how long queues and archives may retain it, and what customers have agreed to receive are decisions for your data-protection, security and customer-communications teams. The project sends identifiers and minimal text, never full account numbers, and never contacts a real SMS or e-mail provider.

The idea in plain language

When a transaction posts, the core banking system announces it on an event bus. It does not know that notifications exist. A rule on the bus forwards each TransactionPosted event to a topic, and the topic fans out to one queue per channel (SMS, e-mail) based on which channels the customer opted into. A function reads each queue, checks "have I already told this customer about this transaction?" in a small table, and sends the message. Anything that can never succeed is parked in a dead-letter queue, an alarm rings, and a person reads a short runbook.

Analogy: Think of the bank's post room again. The cashier shouts "transaction posted" (event). The post room copies the slip into the SMS tray and the e-mail tray depending on the customer's preferences (fan-out). Each clerk keeps a ledger of slips already handled, so a slip that arrives twice is stamped "already done" and binned (idempotency). A slip that is torn or has an impossible amount goes into a locked drawer (DLQ) with a bell on it. When the printer breaks, clerks put slips back for later instead of throwing them away (retry and visibility timeout), and after the printer is fixed a supervisor feeds the locked-drawer slips back in at a sensible pace (redrive).

What problem the AWS services solve

ConcernService and settingWhy it is there
Announce a fact to unknown listenersEventBridge bus hl-m08-eventsPublisher does not know consumers; a rule matches on source and detail-type
Replay events after a fixEventBridge archive hl-m08-events-archive, 7 daysRe-run a day's events through a repaired consumer
Fan out by customer preferenceSNS topic hl-m08-transaction-posted with one subscription per channel and a payload filter policy on detail.channelsEach queue only receives what it must handle
Absorb spikes, isolate failuresSQS queues hl-m08-notify-sms, hl-m08-notify-email, visibility timeout 180 sA slow provider backs up one queue, not the whole system
Park poison messagesDLQs with maxReceiveCount 5, 14 days retentionOne malformed message cannot block the rest
Process and sendLambda (.NET 10, ARM64) per queue with ReportBatchItemFailures and maximum concurrency 5Only failed messages return; concurrency caps pressure on the provider
Never send twiceDynamoDB table hl-m08-idempotency, conditional PutItem, TTL on expiresAtTurns at-least-once delivery into effectively-once sending
Know when something is wrongCloudWatch alarms on every DLQAny message in a DLQ means a customer was not notified

Limits and honest caveats of the design:

  • The idempotency pattern narrows the duplicate window; it does not remove it. If a function dies after the provider accepted the message but before the table row says COMPLETED, the claim expires and the customer can be notified twice. A real provider that accepts an idempotency key closes that gap; the lab's fake provider does not.
  • The lab's provider is a fake that logs "would send" (no SMS sandbox, no cost, no real phone numbers). The failure-injection switch is an environment variable on the function.
  • SNS and EventBridge are both at-least-once. The project does not rely on ordering.
  • EventBridge archive and replay, SNS filter policy scope and the SQS redrive API are fast-moving features. Check current behaviour and limits in the AWS documentation before you rely on them in production ⚠ VERIFY.
  • The notification SNS topic uses SQS-managed encryption on queues but the topic itself is not encrypted with a customer managed key in the lab. Payloads hold identifiers only, with no names, IBANs or balances. Production would decide this with your security team.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalent in this projectWhat's different
Event Grid topic + subscription filtersEventBridge bus + rule with an event patternPattern matching on content; archive and replay built in
Service Bus topic with subscription filter rulesSNS topic with subscription filter policiesSNS pushes to SQS; the filter policy can match on the message body
Service Bus queue + dead-letter sub-queue + MaxDeliveryCountSQS queue + separate DLQ + maxReceiveCountThe DLQ is another queue you create, alarm and drain
Azure Functions Service Bus triggerLambda SQS event source mappingBatch size, partial batch responses and maximum concurrency are mapping settings
Cosmos DB / Table Storage for dedup recordsDynamoDB conditional PutItem with TTLOne atomic write claims the key
Azure Monitor alert on dead-letter message countCloudWatch alarm on ApproximateNumberOfMessagesVisibleSame idea; you pick the metric and threshold
Polly retry with jitterBackoffRetry class in the lab (full jitter)Small and testable; a real service might use Microsoft.Extensions.Http.Resilience

Architecture

What the handler decides for each message. One class, NotificationProcessor, runs in the Lambda function, in the offline simulator and in the tests:

Security-boundary view. Each function role may PutItem and DeleteItem on one table and read one queue. The topic policy lets only this rule publish, and denies non-TLS publishes. Each queue policy lets only this topic send. Logs record identifiers and outcomes, never message text. No secret exists in the lab; a real provider key would live in AWS Secrets Manager.

Step-by-step implementation

Run the tests (no AWS account)

From the lab folder, run the unit tests. They cover the processor, the DynamoDB adapter (mocked), backoff and jitter, the filter-policy matcher, the in-memory queue and the whole-system scenarios.

Run the failure drills offline

hl-notify simulate replays four scenarios on a simulated clock: duplicates, poison messages, a downstream outage and recovery. You read the log and see the same decisions the deployed system makes.

Deploy

Publish the Lambda package for ARM64, then deploy the CDK stack (or the Terraform equivalent below). Without the package the stack still creates the bus, topic, queues, table and alarms, so you can watch messages pile up.

Happy path and duplicates

Publish three transactions three times. You expect three notifications per channel the customer opted into, not nine.

Poison message

Publish an event with an invalid currency and plant a malformed message on the queue. Watch both reach the DLQ after five receives, while good messages keep flowing. The alarm fires.

Downstream outage and recovery

Switch the "provider" off, publish, watch the backlog build, switch it back on, and watch it drain with no duplicates.

Fix and redrive

Treat the poison message as a bug in a consumer: fix it, deploy, redrive the DLQ at a low rate and verify the DLQ is empty and nobody was notified twice.

Replay from the archive (optional)

Start an archive replay for a short window and confirm that the idempotency table absorbs the replayed events.

Tear down and verify

Destroy the stack and check that no tagged resources remain.

AWS Console instructions

Use the console to observe; the scripts and the stack do the work.

  1. Console → Amazon EventBridge → Event buses → hl-m08-events → Rules. Open hl-m08-transaction-posted-to-sns: the event pattern and the SNS target, with its retry policy and dead-letter queue. Archives shows hl-m08-events-archive.
  2. Console → Amazon SNS → Topics → hl-m08-transaction-posted → Subscriptions. Open each SQS subscription: Raw message delivery enabled, and a Subscription filter policy scoped to Message body.
  3. Console → Amazon SQS → Queues. For hl-m08-notify-sms look at Monitoring: messages visible, messages not visible (in flight), age of oldest message. Open the Dead-letter queue tab to see the redrive policy (maxReceiveCount 5). Use Start DLQ redrive on a DLQ to redrive from the console (the script uses the same API).
  4. Console → AWS Lambda → hl-m08-notify-sms-handler → Configuration → Triggers: the SQS trigger with batch size 10, Report batch item failures on and a maximum concurrency of 5. Configuration → Environment variables shows DOWNSTREAM_OUTAGE.
  5. Console → CloudWatch → Logs → Log groups → /hl-m08/notify-sms, then Logs Insights with the query in the runbook section below.
  6. Console → CloudWatch → Alarms to see the three -dlq-not-empty alarms.
  7. Console → DynamoDB → Tables → hl-m08-idempotency → Explore items to see notify#sms#<transactionId> rows with a status of COMPLETED and an expiresAt timestamp.

AWS CLI

# What the three DLQs hold right now.
for q in hl-m08-notify-sms-dlq hl-m08-notify-email-dlq hl-m08-events-target-dlq; do
  url=$(aws sqs get-queue-url --queue-name "$q" --query QueueUrl --output text --profile learn-sandbox)
  echo -n "$q: "
  aws sqs get-queue-attributes --queue-url "$url" --attribute-names ApproximateNumberOfMessages \
    --query Attributes.ApproximateNumberOfMessages --output text --profile learn-sandbox
done

# Test an event pattern against a sample event, without publishing anything.
aws events test-event-pattern --profile learn-sandbox \
  --event-pattern '{"source":["harbourline.core-banking"],"detail-type":["TransactionPosted"]}' \
  --event '{"id":"1","account":"111122223333","time":"2026-11-18T09:00:00Z","region":"us-east-1","resources":[],"source":"harbourline.core-banking","detail-type":"TransactionPosted","detail":{}}'

# Check the SNS subscription filter policy and its scope.
aws sns list-subscriptions-by-topic --profile learn-sandbox \
  --topic-arn "$(aws sns list-topics --profile learn-sandbox --query "Topics[?contains(TopicArn,'hl-m08-transaction-posted')].TopicArn" --output text)" \
  --query "Subscriptions[].SubscriptionArn" --output text

# Check that the event source mapping reports batch item failures.
aws lambda get-event-source-mapping --profile learn-sandbox \
  --uuid "$(aws lambda list-event-source-mappings --function-name hl-m08-notify-sms-handler --profile learn-sandbox --query 'EventSourceMappings[0].UUID' --output text)" \
  --query "{types:FunctionResponseTypes,concurrency:ScalingConfig}"

# Move DLQ messages back to the source queue at 10 per second (after the cause is fixed).
DLQ_ARN=$(aws sqs get-queue-attributes --profile learn-sandbox --attribute-names QueueArn --query Attributes.QueueArn --output text \
  --queue-url "$(aws sqs get-queue-url --queue-name hl-m08-notify-sms-dlq --query QueueUrl --output text --profile learn-sandbox)")
aws sqs start-message-move-task --source-arn "$DLQ_ARN" --max-number-of-messages-per-second 10 --profile learn-sandbox
aws sqs list-message-move-tasks --source-arn "$DLQ_ARN" --max-results 1 --profile learn-sandbox

Infrastructure as Code

The lab ships a CDK (C#) stack (infra/cdk, stack HlM08-Notifications, checked with cdk-nag). The tab below shows the heart of it; the full file is in the lab. The Terraform tab is a complete equivalent of the same resources. Both create the same names so the scripts work with either.

infra/cdk/Harbourline.Notifications.Infra/NotificationStack.cs (excerpt)
// ---- Idempotency table: on-demand, TTL cleans old keys, encrypted at rest by default.
var table = new Table(this, "IdempotencyTable", new TableProps
{
    TableName = "hl-m08-idempotency",
    PartitionKey = new Amazon.CDK.AWS.DynamoDB.Attribute { Name = "id", Type = AttributeType.STRING },
    BillingMode = BillingMode.PAY_PER_REQUEST,
    TimeToLiveAttribute = "expiresAt",
    PointInTimeRecoverySpecification = new PointInTimeRecoverySpecification { PointInTimeRecoveryEnabled = true },
    RemovalPolicy = RemovalPolicy.DESTROY, // lab only: production keeps the table (RETAIN)
});

// ---- Event bus + archive (replay lets you re-run a day's events after fixing a consumer).
var bus = new EventBus(this, "Bus", new EventBusProps { EventBusName = "hl-m08-events" });
bus.Archive("Archive", new BaseArchiveProps
{
    ArchiveName = "hl-m08-events-archive",
    Retention = Duration.Days(7),
    EventPattern = new EventPattern { Source = ["harbourline.core-banking"] },
});

// ---- Topic. EnforceSSL adds a topic policy that denies non-TLS publishes.
var topic = new Topic(this, "TransactionPostedTopic", new TopicProps
{
    TopicName = "hl-m08-transaction-posted",
    EnforceSSL = true,
});

// If EventBridge cannot deliver to SNS after its retries, the event lands in a DLQ instead of vanishing.
rule.AddTarget(new EventTargets.SnsTopic(topic, new EventTargets.SnsTopicProps
{
    DeadLetterQueue = ruleTargetDlq,
    RetryAttempts = 20,
    MaxEventAge = Duration.Hours(2),
}));

// ---- One queue + DLQ + Lambda per channel.
var queue = new Queue(this, $"{Pascal(channel)}Queue", new QueueProps
{
    QueueName = $"hl-m08-notify-{channel}",
    VisibilityTimeout = Duration.Seconds(6 * FunctionTimeoutSeconds), // at least 6 x the function timeout
    Encryption = QueueEncryption.SQS_MANAGED,
    EnforceSSL = true,
    DeadLetterQueue = new Amazon.CDK.AWS.SQS.DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 },
});

// Raw delivery: the SQS body is the EventBridge event itself. Body-scoped filter: EventBridge sets no SNS attributes.
topic.AddSubscription(new SqsSubscription(queue, new SqsSubscriptionProps
{
    RawMessageDelivery = true,
    FilterPolicyWithMessageBody = new Dictionary<string, FilterOrPolicy>
    {
        ["detail-type"] = FilterOrPolicy.Filter(Amazon.CDK.AWS.SNS.SubscriptionFilter.StringFilter(new StringConditions { Allowlist = ["TransactionPosted"] })),
        ["detail"] = FilterOrPolicy.Policy(new Dictionary<string, FilterOrPolicy>
        {
            ["channels"] = FilterOrPolicy.Filter(Amazon.CDK.AWS.SNS.SubscriptionFilter.StringFilter(new StringConditions { Allowlist = [channel] })),
        }),
    },
}));

// The event source mapping polls the queue. ReportBatchItemFailures = partial batch responses.
function.AddEventSource(new SqsEventSource(queue, new SqsEventSourceProps
{
    BatchSize = 10,
    ReportBatchItemFailures = true,
    MaxConcurrency = 5, // caps how hard a backlog can hit the provider after an outage
}));

The stack compiles with dotnet build infra/cdk/Harbourline.Notifications.Infra (checked 2026-11-18, after disambiguating SubscriptionFilter, which exists in both the SNS and CloudWatch Logs namespaces). Deploy with cdk deploy HlM08-Notifications --profile learn-sandbox -c owner=<your-name> from infra/cdk.

Complete code

The code is in labs/m08-notification-system/. The five pieces that carry the lesson are shown here with explanations; read the rest in the repository.

1. The event contract (Events/TransactionPosted.cs): minimal on purpose.

Events/TransactionPosted.cs (excerpt)
public sealed record TransactionPosted
{
    public required string TransactionId { get; init; }
    public required string AccountId { get; init; }
    public required string CustomerId { get; init; }
    public required decimal Amount { get; init; }
    public required string Currency { get; init; }
    public required string Type { get; init; }
    public required DateTimeOffset PostedAt { get; init; }

    /// <summary>Channels the customer opted into, e.g. ["sms","email"]. SNS filter policies route on this.</summary>
    public IReadOnlyList<string> Channels { get; init; } = [];

    /// <summary>Last four digits of the account number, for the message text. Never the full number.</summary>
    public string? AccountLast4 { get; init; }
}

2. The idempotency claim (Idempotency/DynamoDbIdempotencyStore.cs): one atomic conditional write.

Idempotency/DynamoDbIdempotencyStore.cs (claim)
public async Task<ClaimResult> TryClaimAsync(string key, TimeSpan inProgressTimeout, CancellationToken ct = default)
{
    var now = time.GetUtcNow();
    try
    {
        await dynamoDb.PutItemAsync(new PutItemRequest
        {
            TableName = tableName,
            Item = new Dictionary<string, AttributeValue>
            {
                ["id"] = new() { S = key },
                ["status"] = new() { S = StatusInProgress },
                ["expiresAt"] = new() { N = Epoch(now + inProgressTimeout) },
            },
            // Free if nobody has the key, or if the old record has expired. TTL deletion runs in the
            // background and can lag, so the expiry is checked here rather than trusted to TTL.
            ConditionExpression = "attribute_not_exists(id) OR expiresAt < :now",
            ExpressionAttributeValues = new Dictionary<string, AttributeValue> { [":now"] = new() { N = Epoch(now) } },
            ReturnValuesOnConditionCheckFailure = ReturnValuesOnConditionCheckFailure.ALL_OLD,
        }, ct);
        return ClaimResult.Acquired;
    }
    catch (ConditionalCheckFailedException ex)
    {
        var status = ex.Item is not null && ex.Item.TryGetValue("status", out var s) ? s.S : null;
        return status == StatusCompleted ? ClaimResult.AlreadyCompleted : ClaimResult.InProgress;
    }
}

3. Backoff with full jitter (Resilience/BackoffRetry.cs): wait a random time between zero and min(cap, base × 2^attempt).

Resilience/BackoffRetry.cs (core)
public TimeSpan Ceiling(int attempt)
{
    var exponential = _baseDelay.TotalMilliseconds * Math.Pow(2, attempt);
    return TimeSpan.FromMilliseconds(Math.Min(_maxDelay.TotalMilliseconds, exponential));
}

public async Task<T> ExecuteAsync<T>(Func<CancellationToken, Task<T>> action, CancellationToken ct = default)
{
    for (var attempt = 0; ; attempt++)
    {
        try
        {
            return await action(ct);
        }
        catch (TransientDownstreamException) when (attempt + 1 < _maxAttempts)
        {
            var wait = TimeSpan.FromMilliseconds(Ceiling(attempt).TotalMilliseconds * _random());
            History.Add(wait);
            await _delay(wait, ct);
        }
    }
}

4. The decision (Processing/NotificationProcessor.cs): parse and validate, claim, send with retries, complete.

Processing/NotificationProcessor.cs (ProcessAsync)
public async Task<ProcessingResult> ProcessAsync(string body, CancellationToken ct = default)
{
    // 1. Parse and validate. Anything that fails here will fail on every retry: it is poison.
    if (!TryParse(body, out var tx, out var parseError)) return new ProcessingResult(Outcome.Poison, parseError);
    if (Validate(tx!) is { } validationError) return new ProcessingResult(Outcome.Poison, validationError);

    // 2. Idempotency key from the BUSINESS identity (transaction + channel), not the SQS MessageId.
    var key = $"notify#{options.Channel}#{tx!.TransactionId}";
    var claim = await store.TryClaimAsync(key, options.InProgressTimeout, ct);
    switch (claim)
    {
        case ClaimResult.AlreadyCompleted: return new ProcessingResult(Outcome.Duplicate, "already notified", key);
        case ClaimResult.InProgress:       return new ProcessingResult(Outcome.RetryLater, "another worker holds the claim", key);
    }

    // 3. Side effect, with in-process retries for short blips.
    var notification = new Notification(options.Channel, tx.CustomerId, NotificationComposer.Compose(options.Channel, tx), key);
    try
    {
        await retry.ExecuteAsync(async token => { await sender.SendAsync(notification, token); return true; }, ct);
    }
    catch (TransientDownstreamException ex)
    {
        await store.ReleaseAsync(key, ct);   // longer outage: give the claim back, let SQS redeliver
        return new ProcessingResult(Outcome.RetryLater, ex.Message, key);
    }

    // 4. Record success.
    await store.CompleteAsync(key, options.CompletedRetention, ct);
    return new ProcessingResult(Outcome.Sent, null, key);
}

5. The Lambda adapter (Functions/SqsNotificationFunction.cs): partial batch response and a log line with identifiers only.

Functions/SqsNotificationFunction.cs (HandleAsync)
public async Task<SQSBatchResponse> HandleAsync(SQSEvent sqsEvent, ILambdaContext context)
{
    var failures = new List<SQSBatchResponse.BatchItemFailure>();

    foreach (var record in sqsEvent.Records ?? [])
    {
        ProcessingResult result;
        try
        {
            result = await _processor.ProcessAsync(record.Body);
        }
        catch (Exception ex)
        {
            // Unknown failure (for example DynamoDB throttling): retry the message, never lose it.
            result = new ProcessingResult(Outcome.RetryLater, ex.GetType().Name);
        }

        var receiveCount = record.Attributes is not null && record.Attributes.TryGetValue("ApproximateReceiveCount", out var rc) ? rc : "?";
        // Structured log line: identifiers and outcome only, never the message text (it is personal data).
        context.Logger.LogInformation(JsonSerializer.Serialize(new
        {
            messageId = record.MessageId,
            receiveCount,
            outcome = result.Outcome.ToString(),
            reason = result.Reason,
            idempotencyKey = result.IdempotencyKey,
            awsRequestId = context.AwsRequestId,
        }));

        if (result.ShouldRetainMessage)
        {
            failures.Add(new SQSBatchResponse.BatchItemFailure { ItemIdentifier = record.MessageId });
        }
    }

    return new SQSBatchResponse { BatchItemFailures = failures };
}

Run it

# 1. Tests, no AWS account. From labs/m08-notification-system on Windows with Git Bash:
MSYS_NO_PATHCONV=1 docker run --rm -v "$PWD:/src" -v cea-nuget-m08:/root/.nuget/packages -w /src \
  mcr.microsoft.com/dotnet/sdk:10.0 sh -c "dotnet test --solution NotificationSystem.slnx"
# Expected: total: 41, failed: 0, succeeded: 40, skipped: 1  (the skipped one needs a real table)

# 2. Offline failure drills
dotnet run --project src/Harbourline.Notifications.Cli -- simulate

# 3. Deploy (optional)
dotnet publish src/Harbourline.Notifications.Functions -c Release -r linux-arm64 --self-contained false -o publish/functions
cd infra/cdk && cdk deploy HlM08-Notifications --profile learn-sandbox -c owner=<your-name>

Expected simulate output (abridged):

== 1-2. Happy path and duplicate deliveries
   publish 0193a5c0-0000-7000-8000-000000000000 -> 2 queue(s)
   ...
   hl-m08-notify-sms receive#1 -> Duplicate (already notified)
Notifications sent: 4 (first pass 4; the duplicate pass added none)

== 3. Poison messages
   hl-m08-notify-sms receive#3 -> Poison (currency is not a 3-letter ISO 4217 code)
SMS DLQ depth: 2 (alarm would fire on > 0)

== 4. Downstream outage and recovery
   hl-m08-notify-sms receive#2 -> RetryLater (Notification provider unavailable (simulated outage).)
   hl-m08-notify-sms receive#3 -> Sent
Sent after recovery: 3, DLQ depth: 0

The simulator uses maxReceiveCount = 3 and a 30-second visibility timeout on a clock it moves by hand, so the drill finishes in a moment. The deployed stack uses 5 receives and 180 seconds, so a poison message takes about 15 minutes to reach the DLQ.

Failure injection against AWS

pwsh ./scripts/inject-failures.ps1 -Scenario duplicates   # 3 events x 3 publishes
pwsh ./scripts/inject-failures.ps1 -Scenario status       # queue and DLQ depths
pwsh ./scripts/inject-failures.ps1 -Scenario poison       # lands in hl-m08-notify-sms-dlq after 5 receives (about 15 min)
pwsh ./scripts/inject-failures.ps1 -Scenario outage-on    # provider down (DOWNSTREAM_OUTAGE=true)
dotnet run --project src/Harbourline.Notifications.Cli -- publish --count 3
pwsh ./scripts/inject-failures.ps1 -Scenario outage-off   # provider back: the backlog drains
pwsh ./scripts/redrive.ps1 -Channel sms                   # after a fix: DLQ -> source queue at 10 messages per second

The RUN_AWS_TESTS=1 integration test then runs against the deployed table:

RUN_AWS_TESTS=1 AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1 dotnet test --solution NotificationSystem.slnx

Code explained line by line

  1. TransactionPosted has no names, balances or full account numbers. Notifications need to know that something happened and to whom; they look up nothing else. AccountLast4 exists only for the message text. This is data minimisation in the contract, not a filter applied later.
  2. Channels is part of the event. The SNS filter policy detail.channels contains sms (or email) routes without the publisher knowing that queues exist. A customer who opted into both gets one event and two queue deliveries.
  3. attribute_not_exists(id) OR expiresAt < :now claims a key only if nobody holds it, or the previous holder's claim has expired. DynamoDB evaluates the condition and writes in one step, so two Lambda instances racing on the same duplicate cannot both win. The expiry is compared explicitly because TTL deletion is background work and can lag.
  4. ReturnValuesOnConditionCheckFailure = ALL_OLD returns the blocking item with the exception, saving a second GetItem. The code reads its status: COMPLETED means skip (duplicate), anything else means someone is working on it (retry later).
  5. The key notify#{channel}#{transactionId} is the business identity. A publisher retry creates a new SQS message ID for the same transaction, so keying on MessageId would let the duplicate through. Including the channel means the SMS and the e-mail for one transaction are two separate notifications.
  6. Outcome.Poison for invalid data (bad JSON, wrong detail type, currency not three letters). These can never succeed, so retrying wastes receives. Returning them as failures lets SQS count receives and move them to the DLQ.
  7. Outcome.RetryLater for expected, temporary problems (provider down, claim held elsewhere). The message stays on the queue and returns after the visibility timeout.
  8. Three quick in-process retries with jitter (200 ms base, 1 s cap) absorb a blip inside one invocation. Anything longer is left to SQS: sleeping inside Lambda costs money and risks the function timeout. Ceiling(attempt) is the cap before jitter; the random fraction spreads concurrent workers apart.
  9. ReleaseAsync before returning RetryLater hands the claim back, so the redelivery can take it immediately instead of waiting for the claim to expire. The delete is conditional on IN_PROGRESS, so it can never remove a COMPLETED record.
  10. CompleteAsync after the send. If the function dies in between, the claim expires and the notification may go out twice. That is the documented, accepted window for a side effect that is not transactional; a provider that accepts an idempotency key closes it.
  11. failures.Add(new BatchItemFailure { ItemIdentifier = record.MessageId }) reports only the failed messages. With ReportBatchItemFailures on the mapping, the nine good messages in a batch of ten are deleted and the one bad message returns alone. Without it, all ten would be reprocessed.
  12. The log line carries messageId, receiveCount, outcome, reason and idempotencyKey, never the text. In an incident you can follow one transaction from publish to DLQ without putting customer data in logs.
  13. MaxConcurrency = 5 on the event source mapping limits how many function instances poll one queue at once. After an outage, a large backlog cannot hit the provider with hundreds of parallel calls.
  14. visibility timeout = 6 × function timeout is the AWS guidance for Lambda consumers: a slow batch should not become visible again while it is still running.

Security considerations

⚠ Learning shortcut — not for production

The provider is a fake that logs "would send". The failure switch is an environment variable that anyone with lambda:UpdateFunctionConfiguration can flip. The SNS topic and event bus use AWS-owned encryption rather than customer managed KMS keys, and the Lambda functions are not in a VPC. Production: a real provider adapter with its credential in AWS Secrets Manager and an idempotency key sent to the provider; customer managed KMS keys where your policy requires them (key policies must allow events.amazonaws.com and sns.amazonaws.com as needed); the failure switch removed from production builds; alarms routed to an on-call channel; and a tested DLQ retention and redrive procedure.

  • Least privilege. Each function role can PutItem/DeleteItem on one table, read one queue and write one log group. The topic policy allows publish only from this rule's ARN and denies plain HTTP. Each queue policy allows SendMessage only from this topic.
  • No personal data in logs. Logs hold identifiers and outcomes. The SMS text is composed and sent but not logged. Keep it that way.
  • Validation at the consumer. Every message is untrusted input; the processor validates identifiers and currency format before doing anything.
  • Retention is a data-protection decision. DLQs keep messages up to 14 days; the archive keeps events 7 days. Whether that is acceptable, and how deletion requests are honoured, needs review by your data-protection team.
  • Runbook hygiene. Inspecting a DLQ message means seeing personal data. The runbook says look at the minimum, do not paste bodies into tickets or chat, and note who looked.
  • Compliance. Nothing here claims compliance with GDPR, UK GDPR or any financial regulation. Legal, security and compliance teams decide channels, content, consent and retention.

Banking use case

This system is a starter for a real Harbourline "transaction alerts" capability: card payments, transfers in and out, direct debits, ATM withdrawals and fees all publish TransactionPosted. Marketing wants push notifications later; with this design they add a third queue and subscription with detail.channels containing push, and no existing code changes. Risk wants unusual transactions to notify the fraud team in real time; they add another subscription with a numeric filter on detail.amount. Operations wants a weekly report of undelivered notifications; the DLQ alarms and the CloudWatch Logs Insights query below already provide the data.

The runbook is part of the deliverable. At 3 a.m. the engineer who sees hl-m08-notify-sms-dlq-not-empty should be able to classify the cause in five minutes and know whether to redrive, fix first or escalate.

Runbook summary (full text in RUNBOOK.md)

  1. Triage: which DLQ; how many messages; is it growing (inject-failures.ps1 -Scenario status).

  2. Classify using Logs Insights:

    fields @timestamp, messageId, receiveCount, outcome, reason
    | filter outcome != "Sent" and outcome != "Duplicate"
    | stats count() by outcome, reason
    What you seeClassAction
    RetryLater + "provider unavailable" on many messagesDownstream outageWait for recovery, then redrive at a low rate
    Poison with the same reason on many messagesBug or contract changeFix consumer or publisher, deploy, then redrive
    Poison on one or two messagesBad dataInspect minimally; usually do not redrive; tell the publishing team
    Messages in hl-m08-events-target-dlqSNS rejected EventBridge (policy, KMS, throttling)Fix the permission, then replay from the archive or the DLQ
  3. Inspect safely: aws sqs receive-message --visibility-timeout 60 ... hides the message for a minute and does not remove it.

  4. Redrive only after the cause is fixed: pwsh ./scripts/redrive.ps1 -Channel sms -MaxPerSecond 10. Redrive is safe to repeat because the table stops a customer being notified twice. Keep the rate low after an outage.

  5. Close: DLQ at zero, alarm OK, record cause, number affected, time to redrive and whether any customer needs a manual message.

Common mistakes

  • Keying idempotency on the SQS MessageId. A publisher retry gets a new ID and walks past the check. Key on transaction plus channel.
  • Forgetting ReportBatchItemFailures. One bad message makes the whole batch return, and the nine good ones are processed again.
  • Treating poison like an outage. Retrying malformed data forever wastes receives; failing fast reaches the DLQ. Classify errors: validation failures are poison, transport failures are retryable.
  • Redriving before the fix. The messages come straight back and the DLQ refills. Fix, deploy, verify, then redrive.
  • Redriving at full speed after an outage. The recovering provider is flooded again. Use a low MaxPerSecond and the mapping's maximum concurrency.
  • DLQ retention shorter than the source queue's. For a standard queue the retention clock keeps running from the original enqueue time, so a message can expire in the DLQ almost immediately. The stack gives DLQs 14 days.
  • Filtering on message attributes when EventBridge is the publisher. EventBridge does not set SNS message attributes, so the subscription's filter policy must use the message body scope.
  • Putting a full IBAN or the SMS text in a log line. Log identifiers and outcomes only.

Troubleshooting

SymptomLikely causeFix
Messages never reach a queueFilter policy does not match, or its scope is not MessageBodyaws sns get-subscription-attributes; check FilterPolicyScope; run hl-notify detail to see the payload shape
Events never reach SNSRule pattern or bus name wrong, or topic policy blocks the ruleaws events test-event-pattern; read hl-m08-events-target-dlq
Every message is retried as a whole batchReportBatchItemFailures missingaws lambda get-event-source-mapping, check FunctionResponseTypes
AccessDeniedException on PutItemWrong table name in IDEMPOTENCY_TABLE or role policyCheck the variable equals hl-m08-idempotency; check the role
Redrive fails with an access or not-allowed errorSource queue's redrive-allow policy or missing permissionsRun as an admin role in the sandbox; see Dead-letter queues and redrive
Poison message takes far longer than expected to reach the DLQFive receives at 180 seconds each is about 15 minutesExpected; use the offline simulator for the quick view
cdk deploy stops on a cdk-nag findingA new findingRead the finding; fix it or acknowledge it with a written reason
Everything "sends" but nothing is loggedWrong log group, or the function never ranCheck /hl-m08/notify-sms; check the event source mapping state is Enabled

Cost and cleanup

Cost label: 🟢 free. The tests and simulator run offline. At lab volumes (dozens of messages) SQS, SNS, EventBridge custom events, Lambda, DynamoDB on-demand and CloudWatch alarms stay within always-free allowances or cost cents ⚠ VERIFY current allowances; CloudWatch alarm and log-storage pricing differ from the free usage of the other services, so check them on the pricing pages. Archive storage for a handful of events is negligible. Destroy the stack the same day.

pwsh ./scripts/teardown.ps1

The script runs cdk destroy HlM08-Notifications --force and lists any resources still tagged Project=cea-lab and Lab=m08-notification-system. Expect none (tag data can lag a few minutes). With Terraform, run terraform destroy and the same tag query:

aws resourcegroupstaggingapi get-resources --profile learn-sandbox --region us-east-1 \
  --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m08-notification-system \
  --query "ResourceTagMappingList[].ResourceARN" --output text

Delete publish/ locally if you like. Run docker volume rm cea-nuget-m08 if you used that NuGet cache volume.

Exercises

  • 🟢 Guided: Run the tests and simulate. In the output of scenario 4, count how many times each message was received before it was sent, and explain from the code which line decided "try again later" rather than "give up".
  • 🟡 Challenge: Add a third channel, push. Change the sample data, the filter policy JSON in the simulator and the stack, add a test that proves a customer who opted into only sms never reaches the push queue, and update the runbook table.
  • 🔴 Live problem: The contact centre reports that about 600 customers got the same SMS twice after a deploy yesterday. The DLQ is empty and there were no alarms. You have the function logs and the DynamoDB table. Work out how duplicates passed the idempotency check, whether the cause is in the code, the table or the infrastructure, and what you change so it cannot recur.

Assignment

Create a portfolio repository harbourline-notifications from the lab. Add: the CDK or Terraform stack you deployed; a docs/ folder with the two Mermaid diagrams above; a one-page failure-injection report with a table of scenario, what you did, what you observed (queue depths, log lines, alarm times) and what it proves; and your edited RUNBOOK.md. Do not commit any AWS account IDs, access keys or logs containing message content.

Solutions

Knowledge check

Interview questions

  • "Walk me through what happens when a transaction posts." — The core system puts TransactionPosted on an EventBridge bus; a rule forwards it to an SNS topic; per-channel subscriptions with body filters fan out to SQS queues; a Lambda per queue validates, claims an idempotency key in DynamoDB, sends with short jittered retries, marks completion and returns only failed messages for retry. Failures that never succeed land in a DLQ with an alarm.
  • "How do you stop duplicate notifications?" — A conditional write on a durable key built from the business identity, with in-progress and completed states, and an expiry for crashed workers. Accept that a tiny window remains unless the provider also supports idempotency keys.
  • "What is the difference between a retry and a redrive?" — A retry is automatic: the message returns after the visibility timeout. A redrive is a deliberate act: after the cause is fixed, moving DLQ messages back to the source queue at a controlled rate.
  • "The provider is down for an hour. What do you want to happen?" — Messages stay on the queue (or return after failed receives), no customer is notified twice, the backlog drains at a controlled rate when the provider recovers, and the DLQ stays empty because the outage is shorter than the retention and receive count allow. If it fills, redrive after recovery.
  • "What would you change before production?" — A real provider adapter with an idempotency key and secrets in Secrets Manager; customer managed KMS keys if required; private networking; the failure switch removed; alarms to on-call; a tested redrive procedure; load tests; and a data-protection review of content and retention.

Certification practice

Completion checklist

  • I ran the 41 tests (40 pass, 1 gated) and read the simulate output
  • I can explain each hop of the pipeline and what would break without it
  • I demonstrated duplicate prevention, a poison message in the DLQ, an outage and a redrive
  • I can say what Outcome.Poison, RetryLater, Duplicate and Sent mean and what the Lambda returns for each
  • I wrote or edited the runbook and followed it for one scenario
  • I ran the teardown and confirmed nothing tagged Project=cea-lab / Lab=m08-notification-system remains
  • I did not commit account IDs, keys or message content

Key takeaways

  • Every hop is at least once, so the consumer must be idempotent on a business key, not a message ID.
  • Classify failures. Poison goes to the DLQ quickly; temporary problems come back later; short blips are absorbed by a few jittered in-process retries.
  • Report only failed messages with partial batch responses, and cap concurrency so a recovered dependency is not flooded.
  • Redrive after the fix, at a low rate; the idempotency table makes it safe.
  • Alarm every DLQ and write the runbook; reliability is a practice, not a configuration.
  • Keep personal data out of events and logs; send identifiers and minimal text.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page