Skip to Content
Cloud Engineering Academy
8. Messaging, Events & Microservices

Idempotency in .NET with DynamoDB and Powertools

Make every money-moving API call and every message handler safe to repeat: choose idempotency keys, build a DynamoDB idempotency store with conditional writes and TTL, combine business writes atomically with TransactWriteItems, and use the Powertools for AWS Lambda (.NET) idempotency utility.

advanced⏱ 2 h 30 min🟢 FreeDVA-C03SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.DynamoDBv2 4.1.0.2) · AWS.Lambda.Powertools.Idempotency 3.2.2 · Amazon.Lambda.SQSEvents 3.0.1 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain idempotency and the difference between naturally idempotent and made-idempotent operations.
  • Choose a correct idempotency key for an HTTP API (Idempotency-Key header) and for a message consumer (a business ID, not a transport ID).
  • Build a DynamoDB idempotency store in C#:
    • claim the key with a conditional write;
    • store the result;
    • recover from crashes with an in-progress expiry;
    • reject a reused key that comes with a different payload.
  • Make the business write and the idempotency record atomic with TransactWriteItems, and pass keys downstream for side effects you can't make atomic.
  • Use Powertools for AWS Lambda (.NET) [Idempotent] with Idempotency.Configure, and know its table layout, defaults, IAM needs and failure behaviour.
  • Pick an idempotency window that covers every upstream retry and redelivery horizon.

Prerequisites

Why this matters in banking

A Harbourline business customer submits a £48,250.00 supplier payment from the web app. The request reaches the Payments API, which writes the payment and calls the scheme connector. The response is slow, the browser times out after 30 seconds, and the user clicks Submit again.

Without idempotency, the second request is a new payment. The supplier receives £96,500.00, and Harbourline's operations team spends a week recalling the duplicate through the scheme.

The same pattern hides behind every arrow in an event-driven system:

  • An SQS consumer crashes after posting a fee but before deleting the message.
  • SNS retries a delivery that actually succeeded.
  • An EventBridge archive replay re-sends a month of PaymentSettled events.

Each of these delivers the same instruction twice, and every one is documented behaviour.

Idempotency is how a bank makes "deliver at least once" safe. Regulators and auditors don't ask whether you use SQS; they ask how you prevent duplicate payments. This lesson is the answer.

The idea in plain language

An operation is idempotent if doing it twice has the same effect as doing it once.

  • Naturally idempotent: "set the account status to frozen" or "store document STMT-2026-11 in S3 at this key". Repeating them changes nothing.
  • Not naturally idempotent: "debit £60", "send an SMS" or "create a payment". Each repeat adds another effect.

You make the second kind idempotent by giving each intended operation a unique idempotency key, and remembering which keys you have already handled:

  1. Claim the key: "I'm processing request K". If it's already claimed, don't do the work again.
  2. Do the work.
  3. Record the result against the key, so a repeat can get the same answer back, not an error.
  4. Expire the record after a window, when repeats can no longer happen.

Analogy: A cheque has a serial number. If the same cheque is presented twice, the clearing system recognises the number and rejects the second presentation. It doesn't matter how the duplicate arose (a scanner hiccup or a branch resubmitting). The serial number identifies the intended payment, not the attempt to process it. An idempotency key is the cheque number for your API calls and messages.

The two hardest parts:

  • Choosing the key, so that one intention maps to exactly one key across every retry path.
  • Making "do the work" and "record it" atomic, so a crash between them can't cause a repeat.

What problem the AWS tools solve

AWS doesn't offer a single "idempotency service". It gives you building blocks.

Building blockWhat it gives youLimits and sharp edges
DynamoDB conditional writes (attribute_not_exists)An atomic "claim this key only if nobody has"One item per condition; design keys carefully
DynamoDB TransactWriteItemsUp to 100 writes across tables, all-or-nothing — business row and idempotency record together4 MB per transaction; costs twice the write capacity of normal writes; DynamoDB only
ClientRequestToken on TransactWriteItemsDynamoDB itself deduplicates repeated identical transaction calls for 10 minutesCovers SDK and network retries of the same call, not your business intent over days
DynamoDB TTLAutomatic, free deletion of expired idempotency recordsDeletion is not immediate (typically within days) — check expiry in your own logic
Powertools for AWS Lambda (.NET) Idempotency[Idempotent] attribute on handlers or methods, with in-progress locking, stored results, payload validation and timeout handlingLambda-oriented; DynamoDB is the only persistence store; responses must fit in a 400 KB item
SQS FIFO / SNS FIFO / EventBridge Custom Event Bus deduplicationTransport-level dedup of sends5-minute window only (see Delivery guarantees and ordering)
Step Functions Standard execution nameStarting an execution with the name of a running one doesn't start a secondOnly while the first is running; names reusable after history expires

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Hand-rolled IdempotencyKeys table in SQL Server with a unique indexDynamoDB table with attribute_not_exists conditional putSame idea; DynamoDB's condition is the unique constraint
SQL transaction wrapping business row + idempotency rowTransactWriteItems (DynamoDB), or one EF Core transaction (RDS/Aurora)DynamoDB transactions are limited to 100 items
NServiceBus / MassTransit outbox + inbox deduplicationYour idempotency store, or Powertools Idempotency in LambdaNo framework does it automatically for every handler in AWS.Messaging; you add it
Service Bus duplicate detectionSQS FIFO deduplicationAWS window is fixed at 5 minutes
Stripe-style Idempotency-Key header in ASP.NET Core middlewareSame header; store in DynamoDB or the API's databaseNothing AWS-specific; API Gateway does not deduplicate for you
Durable Functions orchestration instance ID as dedup keyStep Functions execution nameReusable only after the execution history expires

Architecture

Two places, one rule. The API path uses the client-supplied key and writes the payment and the claim in one transaction. The message path uses the business ID inside the message (notificationId) and passes it on to the SMS provider as its idempotency key, because sending an SMS can't be part of a DynamoDB transaction.

The idempotency record's life cycle

Choosing the idempotency key

The key must be identical for every retry of one intention and different for different intentions.

SituationGood keyBad keyWhy
Client calls POST /paymentsClient-generated Idempotency-Key header (UUID), stored with the request hashServer-generated payment IDThe server only generates the ID after it has decided it's new; a client retry gets a new one
SQS consumer of PaymentSettledpaymentId + event type (PAY-…#settled)SQS MessageIdA producer that resends creates a new MessageId for the same business event
Notification for a posted transactionnotificationId assigned when the notification was decided, or transactionId#channelHash of the full message bodyBodies can differ trivially (timestamps, trace IDs) for the same intention
EventBridge eventdetail.paymentId (business ID)EventBridge idReplays from an archive get new event IDs ⚠ VERIFY for your bus type
Scheduled job "charge monthly fee"accountId#fee#2026-11Run ID of the jobA re-run of the job must not charge again

Also store a hash of the meaningful payload with the key. If the same key arrives with a different amount, it's a client bug or an attack. Reject it (HTTP 422 or 409) rather than returning the first result.

How long to keep keys

The window must outlast every way a duplicate can arrive:

UpstreamLongest duplicate horizon
Client retriesYour API's documented retry window (Harbourline: 24 hours)
SQS redeliveryUp to the queue's retention period (max 14 days), plus DLQ redrive later
SNS → SQS/Lambda retriesUp to 23 days for AWS-managed endpoints
EventBridge Classic target retriesUp to 24 hours by default
EventBridge archive replayAs far back as you choose to replay

Practical policy for Harbourline:

  • Payments: keep keys for 35 days (beyond a monthly cycle and the SNS horizon).
  • Notifications: 7 days.
  • Deliberate replays: an archive replay is an operational event that must pass a "replay-safe" check (see Banking use case). Never rely on short TTLs for it.

The Powertools default of one hour is fine for API retries, and too short for queue consumers.

Step-by-step implementation

Create the tables

Create hl-m08-idempotency (partition key id, TTL attribute expiration). This layout matches the Powertools default, so both the hand-built store and Powertools can share it. Also create hl-m08-payments (partition key paymentId). Use the CLI or IaC below.

Build the store and test it locally

Add DynamoDbIdempotencyStore from Complete code. Run DynamoDB Local in Docker (docker run -p 8000:8000 amazon/dynamodb-local), point the client's ServiceURL at http://localhost:8000, and run the tests. Use the in-memory fake for fast unit tests.

Add the API endpoint

Add POST /payments with the Idempotency-Key header. A repeat with the same key and body returns the original 201 response. The same key with a different body returns 422. Concurrent duplicates return 409 with Retry-After.

Add the Lambda consumer with Powertools

Create the notification Lambda with AWS.Lambda.Powertools.Idempotency 3.2.2. Configure it in the constructor and decorate the per-message send method with [Idempotent].

Prove it

Send the same notification message three times to the queue (different MessageId, same notificationId). Confirm one SMS "send" in the logs, and one COMPLETED record in the table.

AWS Console instructions

  1. Console → DynamoDB → Tables → Create table. Name hl-m08-idempotency, partition key id (String), no sort key. Table settings → Customize → capacity mode On-demand. Add the lab tags. Create.
  2. Open the table → Additional settings → Time to Live → Turn on, attribute name expiration. (In the console, TTL can only be set after the table exists.)
  3. Create hl-m08-payments with partition key paymentId (String), on-demand.
  4. To inspect records: open hl-m08-idempotency → Explore table items. Look at status (INPROGRESS / COMPLETED), expiration (epoch seconds) and data (stored result).
  5. Console → Lambda → (function) → Configuration → Environment variables: add IDEMPOTENCY_TABLE=hl-m08-idempotency. In Permissions, check the execution role has dynamodb:GetItem, PutItem, UpdateItem and DeleteItem on the table ARN only.

AWS CLI

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
TAGS="Key=Project,Value=cea-lab Key=Lab,Value=m08-idempotency Key=Environment,Value=learn Key=Owner,Value=alex"

aws dynamodb create-table --table-name hl-m08-idempotency \
  --attribute-definitions AttributeName=id,AttributeType=S \
  --key-schema AttributeName=id,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST --tags $TAGS
aws dynamodb wait table-exists --table-name hl-m08-idempotency
aws dynamodb update-time-to-live --table-name hl-m08-idempotency \
  --time-to-live-specification "Enabled=true,AttributeName=expiration"

aws dynamodb create-table --table-name hl-m08-payments \
  --attribute-definitions AttributeName=paymentId,AttributeType=S \
  --key-schema AttributeName=paymentId,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST --tags $TAGS

# Try the core primitive by hand: claim a key only if it is absent.
NOW=$(date +%s); EXP=$((NOW + 86400))
aws dynamodb put-item --table-name hl-m08-idempotency \
  --item "{\"id\":{\"S\":\"api#7c9e6679-7425-40de-944b-e07fc1f90ae7\"},\"status\":{\"S\":\"INPROGRESS\"},\"expiration\":{\"N\":\"$EXP\"}}" \
  --condition-expression "attribute_not_exists(id)"
# Run it again: ConditionalCheckFailedException — the key is already claimed.

# Send the same notification three times (new MessageId each time, same notificationId).
Q_URL=$(aws sqs get-queue-url --queue-name hl-m08-notifications --query QueueUrl --output text)
for i in 1 2 3; do
  aws sqs send-message --queue-url "$Q_URL" --message-body \
    '{"notificationId":"NTF-0192d1c2-0000-7000-8000-0000000000aa","customerId":"CUS-00012345","channel":"sms","templateId":"payment-received","amount":2400.00,"currency":"GBP"}'
done

Infrastructure as Code

IdempotencyStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.Lambda.EventSources;
using Amazon.CDK.AWS.SQS;
using Constructs;

namespace Harbourline.Infra;

public sealed class IdempotencyStack : Stack
{
    public IdempotencyStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        var idempotency = new TableV2(this, "Idempotency", new TablePropsV2
        {
            TableName = "hl-m08-idempotency",
            PartitionKey = new Attribute { Name = "id", Type = AttributeType.STRING },
            TimeToLiveAttribute = "expiration",
            Billing = Billing.OnDemand(),
            PointInTimeRecoverySpecification = new PointInTimeRecoverySpecification { PointInTimeRecoveryEnabled = true },
            RemovalPolicy = RemovalPolicy.DESTROY // lab only
        });

        var queue = new Queue(this, "Notifications", new QueueProps
        {
            QueueName = "hl-m08-notifications",
            VisibilityTimeout = Duration.Seconds(90),
            Encryption = QueueEncryption.SQS_MANAGED,
            EnforceSSL = true,
            DeadLetterQueue = new DeadLetterQueue
            {
                MaxReceiveCount = 5,
                Queue = new Queue(this, "NotificationsDlq", new QueueProps
                {
                    QueueName = "hl-m08-notifications-dlq",
                    Encryption = QueueEncryption.SQS_MANAGED,
                    RetentionPeriod = Duration.Days(14),
                    EnforceSSL = true
                })
            }
        });

        var fn = new Function(this, "Notify", new FunctionProps
        {
            FunctionName = "hl-m08-notify-idempotent",
            Runtime = Runtime.DOTNET_10,
            Handler = "Harbourline.Notifications::Harbourline.Notifications.Function::Handle",
            Code = Code.FromAsset("../src/Harbourline.Notifications/publish"),
            MemorySize = 512,
            Timeout = Duration.Seconds(15),
            Environment = new Dictionary<string, string>
            {
                ["IDEMPOTENCY_TABLE"] = idempotency.TableName,
                ["POWERTOOLS_SERVICE_NAME"] = "notifications"
            }
        });

        // Exactly the four actions Powertools documents — on this table only.
        idempotency.Grant(fn, "dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:UpdateItem", "dynamodb:DeleteItem");
        fn.AddEventSource(new SqsEventSource(queue, new SqsEventSourceProps
        {
            BatchSize = 10,
            ReportBatchItemFailures = true
        }));

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m08-idempotency");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        Amazon.CDK.Tags.Of(this).Add("Owner", "alex");
    }
}

Complete code

1. A hand-built idempotency store (works anywhere: API, Worker Service, Lambda)

IIdempotencyStore.cs
namespace Harbourline.Idempotency;

public enum ClaimOutcome { Claimed, AlreadyCompleted, InProgress, PayloadMismatch }

public sealed record ClaimResult(ClaimOutcome Outcome, string? StoredResult = null);

public interface IIdempotencyStore
{
    /// <summary>Claims <paramref name="key"/> for processing, or reports why it can't.</summary>
    Task<ClaimResult> TryClaimAsync(string key, string payloadHash, TimeSpan lockFor, TimeSpan keepFor, CancellationToken ct);
    Task CompleteAsync(string key, string result, TimeSpan keepFor, CancellationToken ct);
    /// <summary>Releases a claim after a failure so a retry can run the work again.</summary>
    Task ReleaseAsync(string key, CancellationToken ct);
}
DynamoDbIdempotencyStore.cs
using System.Globalization;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;

namespace Harbourline.Idempotency;

public sealed class DynamoDbIdempotencyStore(IAmazonDynamoDB ddb, string tableName, TimeProvider clock)
    : IIdempotencyStore
{
    private const string InProgress = "INPROGRESS";
    private const string Completed = "COMPLETED";

    public async Task<ClaimResult> TryClaimAsync(string key, string payloadHash, TimeSpan lockFor,
        TimeSpan keepFor, CancellationToken ct)
    {
        var now = clock.GetUtcNow().ToUnixTimeSeconds();
        try
        {
            await ddb.PutItemAsync(new PutItemRequest
            {
                TableName = tableName,
                Item = new()
                {
                    ["id"] = S(key),
                    ["status"] = S(InProgress),
                    ["payloadHash"] = S(payloadHash),
                    ["inProgressExpiry"] = N(now + (long)lockFor.TotalSeconds),
                    ["expiration"] = N(now + (long)keepFor.TotalSeconds)
                },
                // New key, OR an expired record (TTL hasn't deleted it yet), OR a stale lock from a crash.
                ConditionExpression =
                    "attribute_not_exists(id) OR expiration < :now OR (#s = :inprogress AND inProgressExpiry < :now)",
                ExpressionAttributeNames = new() { ["#s"] = "status" },
                ExpressionAttributeValues = new() { [":now"] = N(now), [":inprogress"] = S(InProgress) },
                ReturnValuesOnConditionCheckFailure = ReturnValuesOnConditionCheckFailure.ALL_OLD
            }, ct);
            return new ClaimResult(ClaimOutcome.Claimed);
        }
        catch (ConditionalCheckFailedException ex)
        {
            // ALL_OLD returns the existing item without a second read.
            var existing = ex.Item ?? [];
            var storedHash = existing.TryGetValue("payloadHash", out var h) ? h.S : null;
            if (storedHash is not null && storedHash != payloadHash)
                return new ClaimResult(ClaimOutcome.PayloadMismatch);

            var status = existing.TryGetValue("status", out var s) ? s.S : null;
            return status == Completed
                ? new ClaimResult(ClaimOutcome.AlreadyCompleted, existing.TryGetValue("data", out var d) ? d.S : null)
                : new ClaimResult(ClaimOutcome.InProgress);
        }
    }

    public Task CompleteAsync(string key, string result, TimeSpan keepFor, CancellationToken ct) =>
        ddb.UpdateItemAsync(new UpdateItemRequest
        {
            TableName = tableName,
            Key = new() { ["id"] = S(key) },
            UpdateExpression = "SET #s = :completed, #d = :data, expiration = :exp REMOVE inProgressExpiry",
            ConditionExpression = "#s = :inprogress", // only the claimant completes
            ExpressionAttributeNames = new() { ["#s"] = "status", ["#d"] = "data" },
            ExpressionAttributeValues = new()
            {
                [":completed"] = S(Completed),
                [":inprogress"] = S(InProgress),
                [":data"] = S(result),
                [":exp"] = N(clock.GetUtcNow().ToUnixTimeSeconds() + (long)keepFor.TotalSeconds)
            }
        }, ct);

    public Task ReleaseAsync(string key, CancellationToken ct) =>
        ddb.DeleteItemAsync(new DeleteItemRequest
        {
            TableName = tableName,
            Key = new() { ["id"] = S(key) },
            ConditionExpression = "#s = :inprogress", // never delete a completed record
            ExpressionAttributeNames = new() { ["#s"] = "status" },
            ExpressionAttributeValues = new() { [":inprogress"] = S(InProgress) }
        }, ct);

    private static AttributeValue S(string v) => new() { S = v };
    private static AttributeValue N(long v) => new() { N = v.ToString(CultureInfo.InvariantCulture) };
}

2. The Payments API: claim and business write in one transaction

Program.cs (Payments API, excerpt)
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddSingleton<IAmazonDynamoDB>(_ => new AmazonDynamoDBClient(new AmazonDynamoDBConfig
{
    RegionEndpoint = Amazon.RegionEndpoint.USEast1,
    RetryMode = Amazon.Runtime.RequestRetryMode.Standard
}));
builder.Services.AddProblemDetails();
var app = builder.Build();

const string IdemTable = "hl-m08-idempotency";
const string PaymentsTable = "hl-m08-payments";
var keep = TimeSpan.FromDays(35);
var json = new JsonSerializerOptions(JsonSerializerDefaults.Web);

app.MapPost("/payments", async (HttpRequest http, CreatePayment body, IAmazonDynamoDB ddb, CancellationToken ct) =>
{
    if (!http.Headers.TryGetValue("Idempotency-Key", out var keyValues) ||
        !Guid.TryParse(keyValues.ToString(), out var clientKey))
        return Results.Problem(statusCode: 400, title: "Idempotency-Key header (UUID) is required");

    var key = $"api#payments#{clientKey}";
    var hash = Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(JsonSerializer.Serialize(body, json))));
    var paymentId = $"PAY-{Guid.CreateVersion7()}";
    var response = new PaymentCreated(paymentId, "received", body.Amount, body.Currency);
    var responseJson = JsonSerializer.Serialize(response, json);
    var now = DateTimeOffset.UtcNow.ToUnixTimeSeconds();

    try
    {
        // One atomic transaction: the payment row AND the completed idempotency record.
        // Either both exist or neither does — no window where a crash causes a duplicate.
        await ddb.TransactWriteItemsAsync(new TransactWriteItemsRequest
        {
            ClientRequestToken = clientKey.ToString(), // DynamoDB-level dedup of identical calls (10 min)
            TransactItems =
            [
                new TransactWriteItem { Put = new Put
                {
                    TableName = IdemTable,
                    Item = new()
                    {
                        ["id"] = new() { S = key },
                        ["status"] = new() { S = "COMPLETED" },
                        ["payloadHash"] = new() { S = hash },
                        ["data"] = new() { S = responseJson },
                        ["expiration"] = new() { N = (now + (long)keep.TotalSeconds).ToString() }
                    },
                    ConditionExpression = "attribute_not_exists(id) OR expiration < :now",
                    ExpressionAttributeValues = new() { [":now"] = new() { N = now.ToString() } }
                }},
                new TransactWriteItem { Put = new Put
                {
                    TableName = PaymentsTable,
                    Item = new()
                    {
                        ["paymentId"] = new() { S = paymentId },
                        ["debtorAccountId"] = new() { S = body.DebtorAccountId },
                        ["creditorName"] = new() { S = body.CreditorName },
                        ["creditorIban"] = new() { S = body.CreditorIban },
                        ["amount"] = new() { N = body.Amount.ToString(System.Globalization.CultureInfo.InvariantCulture) },
                        ["currency"] = new() { S = body.Currency },
                        ["status"] = new() { S = "received" },
                        ["idempotencyKey"] = new() { S = clientKey.ToString() }
                    },
                    ConditionExpression = "attribute_not_exists(paymentId)"
                }}
            ]
        }, ct);

        return Results.Created($"/payments/{paymentId}", response);
    }
    catch (TransactionCanceledException ex)
        when (ex.CancellationReasons?.FirstOrDefault()?.Code == "ConditionalCheckFailed")
    {
        // Key already used: return the ORIGINAL response, or reject a different payload.
        var existing = await ddb.GetItemAsync(new GetItemRequest
        {
            TableName = IdemTable,
            Key = new() { ["id"] = new() { S = key } },
            ConsistentRead = true
        }, ct);
        var item = existing.Item ?? [];
        if (item.TryGetValue("payloadHash", out var h) && h.S != hash)
            return Results.Problem(statusCode: 422, title: "Idempotency-Key reused with a different request body");

        var original = JsonSerializer.Deserialize<PaymentCreated>(item["data"].S, json)!;
        return Results.Created($"/payments/{original.PaymentId}", original);
    }
});

app.Run();

public sealed record CreatePayment(string DebtorAccountId, string CreditorName, string CreditorIban, decimal Amount, string Currency);
public sealed record PaymentCreated(string PaymentId, string Status, decimal Amount, string Currency);

3. The notification consumer with Powertools Idempotency

Function.cs (Harbourline.Notifications)
using System.Text.Json;
using Amazon.Lambda.Core;
using Amazon.Lambda.SQSEvents;
using AWS.Lambda.Powertools.Idempotency;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace Harbourline.Notifications;

public sealed record NotificationRequest(
    string NotificationId, string CustomerId, string Channel, string TemplateId, decimal Amount, string Currency);

public sealed class Function
{
    private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
    private readonly ISmsSender _sms;

    public Function() : this(new FakeSmsSender()) { }

    public Function(ISmsSender sms)
    {
        _sms = sms;
        // Configure once, outside the handler (Powertools requirement).
        Idempotency.Configure(builder => builder
            .WithOptions(o => o.WithExpiration(TimeSpan.FromDays(7)))
            .UseDynamoDb(Environment.GetEnvironmentVariable("IDEMPOTENCY_TABLE") ?? "hl-m08-idempotency"));
    }

    public async Task<SQSBatchResponse> Handle(SQSEvent evt, ILambdaContext context)
    {
        // Lets Powertools expire the in-progress lock if this invocation times out.
        Idempotency.RegisterLambdaContext(context);
        var failures = new List<SQSBatchResponse.BatchItemFailure>();

        foreach (var record in evt.Records)
        {
            try
            {
                var request = JsonSerializer.Deserialize<NotificationRequest>(record.Body, Json)
                              ?? throw new JsonException("empty body");
                var providerRef = await SendOnceAsync(request.NotificationId, request);
                context.Logger.LogInformation($"notification={request.NotificationId} providerRef={providerRef}");
            }
            catch (IdempotencyAlreadyInProgressException)
            {
                // Another invocation is sending this notification right now: retry later.
                failures.Add(new SQSBatchResponse.BatchItemFailure { ItemIdentifier = record.MessageId });
            }
            catch (Exception ex)
            {
                context.Logger.LogError($"message={record.MessageId} error={ex.GetType().Name}");
                failures.Add(new SQSBatchResponse.BatchItemFailure { ItemIdentifier = record.MessageId });
            }
        }
        return new SQSBatchResponse { BatchItemFailures = failures };
    }

    /// <summary>Runs at most once per notificationId within the expiry window; repeats return the stored result.</summary>
    [Idempotent]
    private async Task<string> SendOnceAsync([IdempotencyKey] string notificationId, NotificationRequest request)
    {
        // Pass the same key to the provider, so even a crash *after* the provider call but *before*
        // the record is completed doesn't produce a second SMS (if the provider honours the key).
        return await _sms.SendAsync(request.CustomerId, request.TemplateId, idempotencyKey: notificationId);
    }
}

public interface ISmsSender
{
    Task<string> SendAsync(string customerId, string templateId, string idempotencyKey);
}

public sealed class FakeSmsSender : ISmsSender
{
    public Task<string> SendAsync(string customerId, string templateId, string idempotencyKey) =>
        Task.FromResult($"sms-{idempotencyKey[^8..]}");
}

4. Tests without AWS

tests/InMemoryIdempotencyStore.cs
using System.Collections.Concurrent;
using Harbourline.Idempotency;

public sealed class InMemoryIdempotencyStore(TimeProvider clock) : IIdempotencyStore
{
    private sealed record Entry(string Status, string Hash, DateTimeOffset LockUntil, DateTimeOffset Expires, string? Data);
    private readonly ConcurrentDictionary<string, Entry> _items = new();

    public Task<ClaimResult> TryClaimAsync(string key, string hash, TimeSpan lockFor, TimeSpan keepFor, CancellationToken ct)
    {
        var now = clock.GetUtcNow();
        var fresh = new Entry("INPROGRESS", hash, now + lockFor, now + keepFor, null);
        while (true)
        {
            if (_items.TryAdd(key, fresh)) return Task.FromResult(new ClaimResult(ClaimOutcome.Claimed));
            var cur = _items[key];
            var reclaimable = cur.Expires < now || (cur.Status == "INPROGRESS" && cur.LockUntil < now);
            if (reclaimable)
            {
                if (_items.TryUpdate(key, fresh, cur)) return Task.FromResult(new ClaimResult(ClaimOutcome.Claimed));
                continue; // lost a race; re-evaluate
            }
            if (cur.Hash != hash) return Task.FromResult(new ClaimResult(ClaimOutcome.PayloadMismatch));
            return Task.FromResult(cur.Status == "COMPLETED"
                ? new ClaimResult(ClaimOutcome.AlreadyCompleted, cur.Data)
                : new ClaimResult(ClaimOutcome.InProgress));
        }
    }

    public Task CompleteAsync(string key, string result, TimeSpan keepFor, CancellationToken ct)
    {
        _items.AddOrUpdate(key, _ => throw new InvalidOperationException("not claimed"),
            (_, e) => e with { Status = "COMPLETED", Data = result, Expires = clock.GetUtcNow() + keepFor });
        return Task.CompletedTask;
    }

    public Task ReleaseAsync(string key, CancellationToken ct)
    {
        if (_items.TryGetValue(key, out var e) && e.Status == "INPROGRESS") _items.TryRemove(key, out _);
        return Task.CompletedTask;
    }
}
tests/IdempotencyTests.cs
using Harbourline.Idempotency;
using Microsoft.Extensions.Time.Testing;
using Xunit;

public class IdempotencyTests
{
    private static readonly TimeSpan Lock = TimeSpan.FromSeconds(30), Keep = TimeSpan.FromDays(7);

    [Fact]
    public async Task Second_claim_returns_stored_result()
    {
        var store = new InMemoryIdempotencyStore(new FakeTimeProvider());
        Assert.Equal(ClaimOutcome.Claimed, (await store.TryClaimAsync("k", "h", Lock, Keep, default)).Outcome);
        await store.CompleteAsync("k", "sms-123", Keep, default);

        var repeat = await store.TryClaimAsync("k", "h", Lock, Keep, default);
        Assert.Equal(ClaimOutcome.AlreadyCompleted, repeat.Outcome);
        Assert.Equal("sms-123", repeat.StoredResult);
    }

    [Fact]
    public async Task Same_key_different_payload_is_rejected()
    {
        var store = new InMemoryIdempotencyStore(new FakeTimeProvider());
        await store.TryClaimAsync("k", "hash-A", Lock, Keep, default);
        Assert.Equal(ClaimOutcome.PayloadMismatch, (await store.TryClaimAsync("k", "hash-B", Lock, Keep, default)).Outcome);
    }

    [Fact]
    public async Task Crashed_claim_can_be_reclaimed_after_lock_expires()
    {
        var time = new FakeTimeProvider();
        var store = new InMemoryIdempotencyStore(time);
        await store.TryClaimAsync("k", "h", Lock, Keep, default);           // worker crashes here
        Assert.Equal(ClaimOutcome.InProgress, (await store.TryClaimAsync("k", "h", Lock, Keep, default)).Outcome);

        time.Advance(TimeSpan.FromSeconds(31));
        Assert.Equal(ClaimOutcome.Claimed, (await store.TryClaimAsync("k", "h", Lock, Keep, default)).Outcome);
    }

    [Fact]
    public async Task Concurrent_duplicates_only_one_wins()
    {
        var store = new InMemoryIdempotencyStore(new FakeTimeProvider());
        var results = await Task.WhenAll(Enumerable.Range(0, 50)
            .Select(_ => Task.Run(() => store.TryClaimAsync("k", "h", Lock, Keep, default))));
        Assert.Single(results, r => r.Outcome == ClaimOutcome.Claimed);
    }
}

FakeTimeProvider comes from the Microsoft.Extensions.TimeProvider.Testing package (10.11.0). For Powertools-decorated code, set POWERTOOLS_IDEMPOTENCY_DISABLED=true in unit tests to test the business logic alone. Test the idempotent behaviour itself against DynamoDB Local or a Testcontainers DynamoDB container.

Code explained line by line

  1. Table layout id + expiration — the Powertools defaults (partition key id, TTL attribute expiration). Using them lets the hand-built store and Powertools share one table. Prefix keys (api#payments#…) so they don't collide. Powertools adds its own prefix (class and method name) automatically.
  2. ConditionExpression in TryClaimAsync — the whole idempotency guarantee rests on this single atomic condition. It claims the key if:
    • it doesn't exist; or
    • the record has expired (TTL deletion lags, so expiration < :now treats it as gone); or
    • a previous claimant crashed and its in-progress lock has run out.
  3. ReturnValuesOnConditionCheckFailure = ALL_OLD — on a failed condition, DynamoDB returns the existing item in the exception, saving a second read and a race. In SDK v4 it's available on ConditionalCheckFailedException.Item ⚠ VERIFY against your package version.
  4. payloadHash check — the same key with a different body is never "the same request". Returning the stored result would hide a client bug. Reject it.
  5. CompleteAsync condition #s = :inprogress — only a claimant can complete, and completion never overwrites a different claimant's completed record. status and data are DynamoDB reserved words, hence #s and #d.
  6. ReleaseAsync — after a failure, delete the in-progress record so the retry can run. Never delete a completed one. Powertools does the same: an unhandled exception inside an [Idempotent] method deletes the record.
  7. TransactWriteItemsAsync in the API — the idempotency record (already COMPLETED, with the response) and the payment row are written together. Neither a crash nor a timeout can leave a payment without its idempotency record, so there is no in-progress state for this path.
  8. ClientRequestToken = clientKey — if the SDK retries the identical transaction call after a network timeout, DynamoDB recognises it for 10 minutes and doesn't report a false conflict. This works alongside your business key, not instead of it.
  9. TransactionCanceledException with reason ConditionalCheckFailed — the first item (the idempotency record) failed its condition, so the key is used. A consistent read fetches the original response to return.
  10. Idempotency.Configure(...) in the constructor — Powertools must be configured once, outside the handler. WithExpiration(TimeSpan.FromDays(7)) overrides the one-hour default, which is too short for queue consumers.
  11. Idempotency.RegisterLambdaContext(context) — needed when [Idempotent] is on a method other than the handler. It lets Powertools store the remaining invocation time as the in-progress expiry, so a timed-out invocation's lock expires quickly instead of blocking retries.
  12. [Idempotent] with [IdempotencyKey] string notificationId — with two parameters, Powertools needs to be told which one is the key. The method's return value (providerRef) is stored and returned on repeats.
  13. IdempotencyAlreadyInProgressException — a concurrent duplicate is being processed. Reporting it as a batch item failure means SQS redelivers it later, by which time the record is COMPLETED and the repeat returns the stored result.
  14. idempotencyKey: notificationId to the provider — the claim/complete pattern still has a gap: the provider call succeeds, then the function dies before COMPLETED is written. Passing your key to a provider that deduplicates closes that gap. If the provider can't deduplicate, document the residual risk.

Security considerations

  • Idempotency records may contain personal data. The stored response (data) can include names, IBANs and amounts. Store only what you need to replay the response, encrypt the table, and set retention by the data-protection policy, not just by convenience. Don't store full request bodies; store a hash.
  • Scope keys per caller. An attacker who guesses another client's idempotency key could receive their stored response. Prefix keys with the authenticated client or customer ID (api#payments#{clientId}#{key}), and only return a stored response to the same caller.
  • Validate the key format. Accept only UUIDs (as above) to stop oversized or crafted keys from becoming partition keys.
  • Least privilege. Powertools needs dynamodb:GetItem, PutItem, UpdateItem and DeleteItem on the idempotency table only, not dynamodb:*.
  • Replay is a privileged operation. Replaying archived events can re-trigger side effects outside the idempotency window. Restrict events:StartReplay and require a change record.

⚠ Learning shortcut — not for production

The API example returns 201 for a repeat and stores the full response body. In production, return the original status code and headers, follow your API standard (some teams add an Idempotent-Replayed: true header), and keep the stored response minimal. The example also has no handling for "claimed but still in progress" on the API path, because the transaction makes that state impossible there. If your API calls an external system before it writes, you need the claim → call → complete flow with an in-progress lock, and 409 Conflict with Retry-After for concurrent duplicates. Treat the IETF Idempotency-Key HTTP header specification as a draft unless your API standard says otherwise ⚠ VERIFY current status.

Banking use case

Harbourline's platform team made idempotency a non-negotiable control for money movement:

  1. Every POST that moves money requires Idempotency-Key. The API gateway layer rejects requests without one (400). Keys are stored for 35 days, scoped by client ID.
  2. Every consumer declares its idempotency key in its design document (for example, paymentId#settled). Code review checks that the key is a business ID, not a transport ID.
  3. Side effects outside the database carry the key downstream. Scheme submissions use the ISO 20022 EndToEndId, which the scheme itself checks. The SMS provider receives notificationId.
  4. Replays are change-controlled. Before an EventBridge archive replay, the operator checks that every target consumer's idempotency window covers the replay period. If not, consumers run in a "dry-run" mode that compares and reports instead of acting.
  5. Evidence: a monthly report counts AlreadyCompleted and PayloadMismatch outcomes per service. A spike in mismatches has twice revealed client bugs before they caused harm.

Whether these controls meet a specific regulatory requirement is for Harbourline's risk and compliance functions to confirm.

Common mistakes

  • Using the SQS MessageId as the key → a producer's resend has a new MessageId → duplicate processing → use a business ID from the message body.
  • Check-then-act with two calls (GetItem, then PutItem if absent) → two concurrent duplicates both see "absent" → both process → claim with one conditional write.
  • Writing the idempotency record after the side effect without a lock → a crash in between causes a repeat → claim first (in-progress), then act, then complete, or make it one transaction.
  • Deleting the record on every exception, including after the side effect happened → the retry repeats the side effect → separate "failed before effect" from "failed after effect", and pass keys downstream.
  • Trusting TTL to delete on time → expired records linger for days → include expiration < :now in the claim condition.
  • One-hour expiry on a queue consumer → SNS or SQS redelivers after the window → size the window to the longest upstream horizon.
  • Ignoring payload mismatch → a reused key with a new amount silently returns the old result → hash and compare.
  • Configuring Powertools inside the handler → re-configures on every invocation → configure in the constructor, as the docs require.

Troubleshooting

SymptomLikely causeFix
IdempotencyPersistenceLayerException from PowertoolsTable missing, wrong name, or the role lacks dynamodb:PutItem/GetItem/UpdateItem/DeleteItemCheck IDEMPOTENCY_TABLE, the role policy and the Region
IdempotencyAlreadyInProgressException keeps happeningA previous invocation crashed and its lock hasn't expired; RegisterLambdaContext not calledCall RegisterLambdaContext; wait for the lock to expire; check function timeout
IdempotencyValidationExceptionPayload validation is configured and the same key came with a different validated fieldExpected for client bugs; investigate the caller
Duplicates still processedKey derived from something that differs per attempt (timestamp, MessageId, full body)Use a stable business ID
ValidationException: The provided key element does not match the schemaTable's partition key isn't id, or it has a sort key Powertools doesn't know aboutUse the default layout or configure WithKeyAttr/WithSortKeyAttr
Item size errors when completingStored result over the 400 KB item limitStore a reference (payment ID), not the full response
TransactionCanceledException with TransactionConflictAnother transaction touched the same items concurrentlyRetry with backoff; usually a concurrent duplicate request
Records never disappearTTL not enabled, or expiration stored in milliseconds/ISO formatEnable TTL on expiration; store epoch seconds as a Number

Cost and cleanup

ItemCost at lab volume
DynamoDB on-demand writes and readsA claim plus a complete is about 2 writes per message; a repeat is about 1 read. Lab volumes are fractions of a cent, or within the always-free tier ⚠ VERIFY current allowance
TransactWriteItemsTransactional writes cost twice a standard write
TTL deletesFree
Lambda, SQSWithin the always-free tier for this lesson

The free alternative is to run everything against DynamoDB Local in Docker and use the in-memory store for unit tests. No AWS account is needed.

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
aws dynamodb delete-table --table-name hl-m08-idempotency
aws dynamodb delete-table --table-name hl-m08-payments
# If you deployed the CDK stack: cdk destroy IdempotencyStack

aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m08-idempotency \
  --query "ResourceTagMappingList[].ResourceARN"

Exercises

  • 🟢 Guided: Run the four IdempotencyTests, then add one for "expired completed record can be claimed again" using FakeTimeProvider.Advance.
  • 🟡 Challenge: Turn the API's idempotency logic into an ASP.NET Core endpoint filter (IEndpointFilter) that any money-moving endpoint can opt into with .AddEndpointFilter<IdempotencyFilter>(). It must cover the claim → execute → complete flow with IIdempotencyStore, return the stored response on repeats, 422 on payload mismatch, and 409 with Retry-After: 2 while in progress.
  • 🔴 Live problem: After a deployment, the fees service charged 312 customers a £5.00 overdraft fee twice. The fee job runs nightly, and a deploy restarted it mid-run. The job's idempotency key is jobRunId#accountId. Find the flaw, fix it, and describe the remediation for affected customers.

Assignment

Add a small, reusable Harbourline.Idempotency library to your portfolio repository. Include:

  • IIdempotencyStore;
  • the DynamoDB implementation;
  • the in-memory fake;
  • the endpoint filter from the challenge;
  • tests against DynamoDB Local run with Testcontainers (gated so CI still passes without Docker if needed).

Write a README section titled "Choosing your key" with three examples from Harbourline.

Solutions

Knowledge check

Interview questions

  • "How do you prevent duplicate payments in an event-driven system?" — Require an idempotency key at the API. Claim it atomically (conditional write or one transaction with the business write). Store and replay the original response. Make every consumer idempotent on a business key. Pass keys downstream (EndToEndId). Size windows to redelivery horizons.
  • "What's a good idempotency key for a message consumer?" — A business identifier that's the same for every redelivery and resend of one intention, for example paymentId#settled. Not the transport's message ID, and not a hash of the full body.
  • "What's the hardest failure case?" — A crash after an external side effect but before recording completion. Mitigate with downstream idempotency keys, ordering the steps so the risky effect comes last, and reconciliation that detects and reverses duplicates.
  • "Have you used Powertools idempotency? What are its limits?" — [Idempotent] on a handler or method with a DynamoDB store, in-progress locking, payload validation and timeout-aware locks. Limits: DynamoDB only, 400 KB result size, a one-hour default expiry (raise it for queues), and the record is deleted on exceptions.

Certification practice

Completion checklist

  • I can explain idempotency and give a naturally idempotent and a non-idempotent banking example
  • I can choose a correct idempotency key for an API and for a message consumer
  • My store claims keys with one conditional write, handles crashed locks and expired records, and rejects payload mismatches
  • I wrote a payment and its idempotency record in one TransactWriteItems call
  • I configured Powertools Idempotency with a window that fits the queue's redelivery horizon
  • I deleted the tables and confirmed nothing tagged Lab=m08-idempotency remains

Key takeaways

  • At-least-once delivery is safe only with idempotent processing. Every money-moving API and every consumer needs it.
  • The key identifies the intention (paymentId#settled, client Idempotency-Key), never the attempt (MessageId, run ID, timestamp).
  • Claim with one atomic conditional write. Better still, write the business change and the idempotency record in one transaction.
  • Size the window to the longest redelivery horizon. Powertools' one-hour default is for API retries, not queues.
  • For side effects outside your database, pass the key downstream and reconcile, because no local trick closes that gap completely.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page