Skip to Content
Cloud Engineering Academy
4. API Development & Integration

Idempotency keys: making POST safe to retry

Implement the Idempotency-Key header in ASP.NET Core 10 with an endpoint filter and a DynamoDB store that uses conditional writes and TTL, so retries never create duplicate statements or payments.

advanced⏱ 1 h 30 min🟢 FreeDVA-C03SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.DynamoDBv2 4.x) · AWS.Lambda.Powertools.Idempotency 3.2.x · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-17.

Learning objectives

By the end of this lesson you can:

  • Explain why retries happen between a mobile app, API Gateway, Lambda and a core banking system, and why "exactly once" delivery is a myth.
  • Implement the Idempotency-Key request header following the IETF httpapi draft: 400 when missing, replay on a repeat, 422 for a reused key with a different payload, 409 while the first request is still running.
  • Build a request fingerprint and scope keys per customer with an expiry.
  • Store idempotency records in Amazon DynamoDB using a conditional PutItem with attribute_not_exists, a lock timeout and TTL.
  • Decide when to use your own filter and when to use Powertools for AWS Lambda (.NET) Idempotency.

Prerequisites

  • REST API design for banking — safe vs idempotent methods, status codes, problem details
  • Module 3: .NET on AWS — dependency injection and the AWS SDK for .NET v4
  • Tools: .NET 10 SDK, Docker (for DynamoDB Local), AWS CLI v2 with the learn-sandbox profile (optional AWS part)

Why this matters in banking

A customer on a train taps Request statement. The request reaches Harbourline's API, the statement job is queued, and the 202 response is lost in a tunnel. The app's HTTP client retries. Without protection, the bank generates two statements — annoying. Replace "statement" with "send £2,000 to a new payee" and it becomes a complaint, a refund, a fraud-team investigation and possibly a regulatory report.

Retries are not a bug to eliminate; they are how distributed systems survive failure. The fix is to make the operation safe to repeat. That is what an idempotency key does.

The idea in plain language

The client generates a unique value — usually a UUID — for each intended operation and sends it in an Idempotency-Key header. The server remembers the key, together with a fingerprint of the request and the response it produced. If the same key arrives again:

  • same fingerprint and the first request finished → send back the stored response (replay), do not run the operation again;
  • same fingerprint but the first request is still running → 409 Conflict, "try again shortly";
  • different fingerprint → 422 Unprocessable Content, because the client reused a key for a different operation.

Analogy: A cheque has a serial number. If the same cheque is presented twice, the bank does not pay twice — it recognises the number and returns the original outcome. If someone presents a cheque with the same number but a different amount, that is not a retry; it is an error to investigate.

Why retries happen

SourceWhat happensDuplicate risk
Mobile networksResponse lost after the server committedClient retries a successful request
API Gateway integration timeoutREST API gives up after 29 s by default and returns 504, but the Lambda function keeps running and may finishClient retries an operation that is still in progress or already done
Client librariesHttpClient with Microsoft.Extensions.Http.Resilience retries on 5xx, 408, 429 and network errorsRetries POST unless you disable it or add a key
Load balancers and proxiesConnection reset mid-responseSame as above
Asynchronous AWS servicesLambda asynchronous invocations retry on error; SQS and EventBridge deliver at least onceConsumers see the same message more than once
HumansDouble-click, refresh, "it didn't work"Two intentional-looking requests

The "exactly once" myth

Networks can give you at most once (never retry, sometimes lose work) or at least once (retry, sometimes duplicate). "Exactly once" is not a delivery guarantee you can buy; it is an effect you build: at-least-once delivery plus idempotent processing. An idempotency key covers the HTTP edge. Downstream, every step that moves money needs its own natural duplicate check — for payments, a unique constraint on the ISO 20022 EndToEndId (end_to_end_id in Harbourline's schema) per debtor account, so that even if two different keys slip through, the second payment is rejected by the payments service. See ISO 20022 messages.

What problem the AWS service solves

You need a store that many API instances (Lambda execution environments or Fargate tasks) share, that can atomically say "this key did not exist and now it does", and that forgets old keys automatically.

Amazon DynamoDB fits well:

  • A conditional write (PutItem with ConditionExpression = "attribute_not_exists(pk)") succeeds for exactly one concurrent writer; the others get ConditionalCheckFailedException. No locks, no transactions.
  • Time to Live (TTL) deletes expired items automatically without consuming write throughput — but "within a few days" of expiry, not instantly. Your condition must therefore treat items whose expiresAt has passed as absent.
  • ReturnValuesOnConditionCheckFailure = ALL_OLD returns the existing item inside the exception, saving a second read.

Limits and sharp edges:

  • An item is at most 400 KB, so store compact responses (Powertools documents the same limit). The statement-request response is a few hundred bytes.
  • Cost: the DynamoDB free tier gives 25 GB of storage per month per Region, but its free 25 WCU / 25 RCU apply to provisioned capacity only. On-demand requests are billed from the first one: about $0.625 per million write request units and $0.125 per million read request units (us-east-1, checked 2026-11-17). A lab of a few hundred requests costs well under one cent; use provisioned capacity of 1 WCU/1 RCU if you want to stay strictly inside the free tier.
  • An in-memory store is fine for a single process or tests, but wrong for Lambda: each execution environment has its own memory.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Cosmos DB conditional create (unique id, 409 Conflict on duplicate)DynamoDB PutItem + attribute_not_existsDynamoDB raises ConditionalCheckFailedException; you map it to HTTP codes
Cosmos DB TTL (ttl property, seconds)DynamoDB TTL on a Number attribute holding an epoch-seconds expiryDynamoDB stores the absolute expiry time, not a duration; deletion can lag by days
Azure Cache for Redis SET key NX EXDynamoDB conditional put (or ElastiCache/Valkey SET NX)A cache can evict keys under memory pressure — dangerous for money movement
Service Bus duplicate detection (MessageId window)SQS FIFO deduplication (5-minute window)Covers messaging, not your HTTP API
Polly retry in HttpClientFactorySame: Microsoft.Extensions.Http.ResilienceRetrying POST is only safe with a key
ASP.NET Core action filterEndpoint filter (IEndpointFilter) on a minimal API routeSame idea; runs after model binding

Architecture

Security-boundary view: the key is client-controlled input. The server scopes it as customerId#key, where customerId comes from the validated token, so customer A can never replay or block customer B's operation by guessing a key. The table holds response bodies, which may contain personal data: it is encrypted at rest by default, access is limited to the API's IAM role, and TTL bounds retention (24 hours here).

Step-by-step implementation

Read the standard and decide your contract

The IETF httpapi working group's "The Idempotency-Key HTTP Header Field" is still an Internet-Draft (revision 08, 22 October 2026, not yet an RFC). Its key points:

  • The header is a Structured Field String, so the canonical form is quoted: Idempotency-Key: "8c1e5a5e-4f6a-4a51-9d55-1f7a4c2b3e90". Many APIs also accept the unquoted form; Harbourline accepts both.
  • Keys should be unique; a UUID (RFC 9562) is recommended.
  • The resource defines whether a fingerprint is used and documents the expiry policy.
  • Errors: 400 if the key is missing for an operation that requires it, 422 if a key is reused with a different payload, 409 if a retry arrives while the original is still being processed — ideally as RFC 9457 problem details.

The draft does not define a header that signals a replay. Harbourline adds Idempotent-Replayed: true, a convention some payment APIs use, so clients and logs can tell replays apart.

Harbourline's documented policy: required on POST .../statement-requests; keys are 8–128 characters of letters, digits, - and _; scoped per customer; kept for 24 hours; responses with status codes below 500 are stored and replayed; 5xx responses and exceptions release the key so the client can retry.

Define the fingerprint

The fingerprint answers "is this really the same request?". Harbourline uses SHA-256 over the method, the path and the raw body bytes. Same key + same fingerprint = retry; same key + different fingerprint = client bug (422).

Choices to make consciously: excluding headers such as User-Agent (they can change on retry), and hashing raw bytes (simple; a retry from the same client sends identical bytes) rather than canonicalised JSON (more tolerant, more code). Because the endpoint filter runs after model binding has read the body, the app calls Request.EnableBuffering() early in the pipeline so the filter can rewind and hash it.

Design the storage record

AttributeTypePurpose
pkSCUS-00012345#8c1e5a5e-... — partition key, customer-scoped
statusSIN_PROGRESS or COMPLETED (a DynamoDB reserved word, so expressions use #s)
fingerprintSHex SHA-256 of method, path and body
lockExpiresAtNEpoch seconds after which an IN_PROGRESS record is considered abandoned (crash recovery)
expiresAtNEpoch seconds; the TTL attribute (24 h)
statusCode, contentType, responseLocation, bodyN / SStored response for replay

The first write uses this condition:

attribute_not_exists(pk) OR expiresAt < :now OR (#s = :inProgress AND lockExpiresAt < :now)

It succeeds if the key is new, if an old record has expired but TTL has not deleted it yet, or if a previous attempt crashed while holding the lock.

Handle the race

Two retries can arrive within milliseconds on two different Lambda execution environments. Both call PutItem; DynamoDB serialises writes to the same item, so exactly one succeeds. The loser receives ConditionalCheckFailedException with the winner's item and returns 409 (in progress) or a replay. No distributed lock service is needed.

Protect the business operation too

There is still a gap: if the handler succeeds but the final UpdateItem to COMPLETED fails (throttling, a crash), the record stays IN_PROGRESS until lockExpiresAt, after which a retry runs the handler again. Close it in the domain: the statement-request handler stores the idempotency key on the statement request it creates, with a unique constraint, and a payment carries a unique end_to_end_id. Defence in depth — the HTTP layer prevents almost all duplicates, the domain prevents the rest.

AWS Console instructions

Optional, if you prefer clicking to the CLI: Console → DynamoDB → Tables → Create table. Table name hl-m04-idempotency, partition key pk (String), no sort key. Under Table settings choose Customize settings → Capacity mode: On-demand. Add tags Project=cea-lab, Lab=m04-idempotency, Environment=learn, Owner=<your-name>. Create, then open the table → Additional settings → Time to Live (TTL) → Turn on, attribute name expiresAt. To inspect records later: Explore items on the table.

AWS CLI

# 1. Create the table: on-demand, one String partition key, mandatory tags.
aws dynamodb create-table \
  --table-name hl-m04-idempotency \
  --attribute-definitions AttributeName=pk,AttributeType=S \
  --key-schema AttributeName=pk,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m04-idempotency Key=Environment,Value=learn Key=Owner,Value=your-name \
  --profile learn-sandbox --region us-east-1

# 2. Wait until the table is ACTIVE.
aws dynamodb wait table-exists --table-name hl-m04-idempotency --profile learn-sandbox --region us-east-1

# 3. Turn on TTL using the expiresAt attribute (epoch seconds).
aws dynamodb update-time-to-live \
  --table-name hl-m04-idempotency \
  --time-to-live-specification "Enabled=true,AttributeName=expiresAt" \
  --profile learn-sandbox --region us-east-1

# 4. Prove the conditional write: the first put succeeds, the second fails with ConditionalCheckFailedException.
aws dynamodb put-item --table-name hl-m04-idempotency \
  --item '{"pk":{"S":"CUS-00012345#demo-key-001"},"status":{"S":"IN_PROGRESS"}}' \
  --condition-expression "attribute_not_exists(pk)" \
  --profile learn-sandbox --region us-east-1
aws dynamodb put-item --table-name hl-m04-idempotency \
  --item '{"pk":{"S":"CUS-00012345#demo-key-001"},"status":{"S":"IN_PROGRESS"}}' \
  --condition-expression "attribute_not_exists(pk)" \
  --profile learn-sandbox --region us-east-1

# 5. Inspect the item.
aws dynamodb get-item --table-name hl-m04-idempotency \
  --key '{"pk":{"S":"CUS-00012345#demo-key-001"}}' --consistent-read \
  --profile learn-sandbox --region us-east-1

In PowerShell, wrap the JSON arguments in single quotes as shown, or put them in files and use file://item.json to avoid quoting problems.

Free local alternative: run DynamoDB Local in Docker and point the CLI and SDK at it.

docker run --rm -p 8000:8000 amazon/dynamodb-local
aws dynamodb create-table --table-name hl-m04-idempotency \
  --attribute-definitions AttributeName=pk,AttributeType=S \
  --key-schema AttributeName=pk,KeyType=HASH --billing-mode PAY_PER_REQUEST \
  --endpoint-url http://localhost:8000 --region us-east-1

Infrastructure as Code

using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.Lambda;
using Constructs;

namespace Harbourline.AccountInformation.Infra;

public sealed class IdempotencyTable : Construct
{
    public TableV2 Table { get; }

    public IdempotencyTable(Construct scope, string id, IFunction apiFunction) : base(scope, id)
    {
        Table = new TableV2(this, "Table", new TablePropsV2
        {
            TableName = "hl-m04-idempotency",
            PartitionKey = new Amazon.CDK.AWS.DynamoDB.Attribute { Name = "pk", Type = AttributeType.STRING },
            Billing = Billing.OnDemand(),
            TimeToLiveAttribute = "expiresAt",
            RemovalPolicy = RemovalPolicy.DESTROY // ⚠ lab only: production keeps RETAIN
        });

        // Least privilege: data-plane read/write on this table only.
        Table.GrantReadWriteData(apiFunction);

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m04-idempotency");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        Amazon.CDK.Tags.Of(this).Add("Owner", "your-name");
    }
}

Complete code

These files are simplified excerpts of the module project, labs/m04-account-information-api/src/Harbourline.AccountInformation.Api/Idempotency/. They compile on their own (with AWSSDK.DynamoDBv2 4.x); member names in the lab may differ slightly, and the lab is the source of truth.

Idempotency/IIdempotencyStore.cs
namespace Harbourline.AccountInformation.Api.Idempotency;

public sealed record StoredResponse(int StatusCode, string? ContentType, string? Location, string Body);

public enum BeginOutcome { Started, Replay, InProgress, FingerprintMismatch }

public sealed record BeginResult(BeginOutcome Outcome, StoredResponse? Response = null);

public interface IIdempotencyStore
{
    /// <summary>Atomically claims the key, or reports what already exists for it.</summary>
    Task<BeginResult> TryBeginAsync(string scopedKey, string fingerprint, CancellationToken ct);

    /// <summary>Marks the key COMPLETED and stores the response for replay.</summary>
    Task CompleteAsync(string scopedKey, StoredResponse response, CancellationToken ct);

    /// <summary>Deletes an IN_PROGRESS record so the client may retry (used after 5xx or exceptions).</summary>
    Task ReleaseAsync(string scopedKey, CancellationToken ct);
}
Idempotency/DynamoDbIdempotencyStore.cs
using System.Globalization;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
using Microsoft.Extensions.Options;

namespace Harbourline.AccountInformation.Api.Idempotency;

public sealed class IdempotencyOptions
{
    public string TableName { get; set; } = "hl-m04-idempotency";
    public TimeSpan RecordTtl { get; set; } = TimeSpan.FromHours(24);
    public TimeSpan LockTimeout { get; set; } = TimeSpan.FromSeconds(60);
}

public sealed class DynamoDbIdempotencyStore(
    IAmazonDynamoDB dynamo, IOptions<IdempotencyOptions> options, TimeProvider time) : IIdempotencyStore
{
    private const string InProgress = "IN_PROGRESS";
    private const string Completed = "COMPLETED";
    private readonly IdempotencyOptions _opt = options.Value;

    public async Task<BeginResult> TryBeginAsync(string scopedKey, string fingerprint, CancellationToken ct)
    {
        var now = time.GetUtcNow().ToUnixTimeSeconds();
        var request = new PutItemRequest
        {
            TableName = _opt.TableName,
            Item = new Dictionary<string, AttributeValue>
            {
                ["pk"] = S(scopedKey),
                ["status"] = S(InProgress),
                ["fingerprint"] = S(fingerprint),
                ["lockExpiresAt"] = N(now + (long)_opt.LockTimeout.TotalSeconds),
                ["expiresAt"] = N(now + (long)_opt.RecordTtl.TotalSeconds),
            },
            ConditionExpression =
                "attribute_not_exists(pk) OR expiresAt < :now OR (#s = :inProgress AND lockExpiresAt < :now)",
            ExpressionAttributeNames = new Dictionary<string, string> { ["#s"] = "status" },
            ExpressionAttributeValues = new Dictionary<string, AttributeValue>
            {
                [":now"] = N(now),
                [":inProgress"] = S(InProgress),
            },
            ReturnValuesOnConditionCheckFailure = ReturnValuesOnConditionCheckFailure.ALL_OLD,
        };

        try
        {
            await dynamo.PutItemAsync(request, ct);
            return new BeginResult(BeginOutcome.Started);
        }
        catch (ConditionalCheckFailedException ex)
        {
            var existing = ex.Item is { Count: > 0 } ? ex.Item : await ReadAsync(scopedKey, ct);
            if (existing is null)
            {
                return new BeginResult(BeginOutcome.InProgress); // deleted in between: ask the client to retry
            }

            if (Get(existing, "fingerprint") != fingerprint)
            {
                return new BeginResult(BeginOutcome.FingerprintMismatch);
            }

            if (Get(existing, "status") != Completed)
            {
                return new BeginResult(BeginOutcome.InProgress);
            }

            var stored = new StoredResponse(
                int.Parse(Get(existing, "statusCode") ?? "200", CultureInfo.InvariantCulture),
                Get(existing, "contentType"),
                Get(existing, "responseLocation") is { Length: > 0 } location ? location : null,
                Get(existing, "body") ?? string.Empty);
            return new BeginResult(BeginOutcome.Replay, stored);
        }
    }

    public Task CompleteAsync(string scopedKey, StoredResponse response, CancellationToken ct) =>
        dynamo.UpdateItemAsync(new UpdateItemRequest
        {
            TableName = _opt.TableName,
            Key = new Dictionary<string, AttributeValue> { ["pk"] = S(scopedKey) },
            UpdateExpression = "SET #s = :completed, statusCode = :code, contentType = :ct, responseLocation = :loc, #b = :body",
            ConditionExpression = "attribute_exists(pk)",
            ExpressionAttributeNames = new Dictionary<string, string> { ["#s"] = "status", ["#b"] = "body" },
            ExpressionAttributeValues = new Dictionary<string, AttributeValue>
            {
                [":completed"] = S(Completed),
                [":code"] = N(response.StatusCode),
                [":ct"] = S(response.ContentType ?? "application/json"),
                [":loc"] = S(response.Location ?? string.Empty),
                [":body"] = S(response.Body),
            },
        }, ct);

    public Task ReleaseAsync(string scopedKey, CancellationToken ct) =>
        dynamo.DeleteItemAsync(new DeleteItemRequest
        {
            TableName = _opt.TableName,
            Key = new Dictionary<string, AttributeValue> { ["pk"] = S(scopedKey) },
            ConditionExpression = "#s = :inProgress",
            ExpressionAttributeNames = new Dictionary<string, string> { ["#s"] = "status" },
            ExpressionAttributeValues = new Dictionary<string, AttributeValue> { [":inProgress"] = S(InProgress) },
        }, ct);

    private async Task<Dictionary<string, AttributeValue>?> ReadAsync(string scopedKey, CancellationToken ct)
    {
        var response = await dynamo.GetItemAsync(new GetItemRequest
        {
            TableName = _opt.TableName,
            Key = new Dictionary<string, AttributeValue> { ["pk"] = S(scopedKey) },
            ConsistentRead = true,
        }, ct);
        return response.Item is { Count: > 0 } ? response.Item : null;
    }

    private static AttributeValue S(string value) => new() { S = value };
    private static AttributeValue N(long value) => new() { N = value.ToString(CultureInfo.InvariantCulture) };

    private static string? Get(Dictionary<string, AttributeValue> item, string name) =>
        item.TryGetValue(name, out var v) ? v.S ?? v.N : null;
}
Idempotency/IdempotencyFilter.cs
using System.Security.Claims;
using System.Security.Cryptography;
using System.Text;
using System.Text.RegularExpressions;

namespace Harbourline.AccountInformation.Api.Idempotency;

public sealed partial class IdempotencyFilter(IIdempotencyStore store, ILogger<IdempotencyFilter> logger) : IEndpointFilter
{
    public const string HeaderName = "Idempotency-Key";
    private const string ProblemBase = "https://api.harbourline.example/problems/";

    public async ValueTask<object?> InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
    {
        var http = context.HttpContext;
        var ct = http.RequestAborted;

        var key = ParseKey(http.Request.Headers[HeaderName].ToString());
        if (key is null)
        {
            return Results.Problem(statusCode: 400, type: ProblemBase + "idempotency-key-missing",
                title: "A valid Idempotency-Key header is required",
                detail: "Send a UUID in the Idempotency-Key header (8-128 characters: letters, digits, '-' or '_').");
        }

        // Scope by the customer resolved from the validated token, never from the request body or URL.
        var customerId = http.User.FindFirstValue("customer_id");
        if (customerId is null)
        {
            return Results.Unauthorized();
        }

        var scopedKey = $"{customerId}#{key}";
        var fingerprint = await FingerprintAsync(http.Request, ct);
        var begin = await store.TryBeginAsync(scopedKey, fingerprint, ct);

        switch (begin.Outcome)
        {
            case BeginOutcome.FingerprintMismatch:
                return Results.Problem(statusCode: 422, type: ProblemBase + "idempotency-key-reused",
                    title: "Idempotency-Key was already used with a different request");
            case BeginOutcome.InProgress:
                http.Response.Headers.RetryAfter = "2";
                return Results.Problem(statusCode: 409, type: ProblemBase + "idempotency-request-in-progress",
                    title: "A request with this Idempotency-Key is still being processed");
            case BeginOutcome.Replay when begin.Response is not null:
                logger.LogInformation("Replaying stored response for idempotency key {ScopedKey}", scopedKey);
                await WriteStoredAsync(http, begin.Response, ct);
                return Results.Empty;
        }

        // Started: run the endpoint, capture its response, persist it, then send it.
        var originalBody = http.Response.Body;
        using var buffer = new MemoryStream();
        http.Response.Body = buffer;
        try
        {
            var result = await next(context);
            var toExecute = result as IResult ?? (result is null ? Results.Empty : Results.Json(result));
            await toExecute.ExecuteAsync(http);
        }
        catch
        {
            http.Response.Body = originalBody;
            await store.ReleaseAsync(scopedKey, CancellationToken.None);
            throw;
        }
        finally
        {
            http.Response.Body = originalBody;
        }

        var body = Encoding.UTF8.GetString(buffer.ToArray());
        if (http.Response.StatusCode >= 500)
        {
            await store.ReleaseAsync(scopedKey, CancellationToken.None);
        }
        else
        {
            var stored = new StoredResponse(http.Response.StatusCode, http.Response.ContentType,
                http.Response.Headers.Location.ToString() is { Length: > 0 } loc ? loc : null, body);
            await store.CompleteAsync(scopedKey, stored, CancellationToken.None);
        }

        buffer.Position = 0;
        await buffer.CopyToAsync(originalBody, ct);
        return Results.Empty;
    }

    private static string? ParseKey(string raw)
    {
        var value = raw.Trim();
        if (value.Length >= 2 && value[0] == '"' && value[^1] == '"')
        {
            value = value[1..^1]; // Structured Field String form: "..."
        }
        return KeyPattern().IsMatch(value) ? value : null;
    }

    private static async Task<string> FingerprintAsync(HttpRequest request, CancellationToken ct)
    {
        using var hash = IncrementalHash.CreateHash(HashAlgorithmName.SHA256);
        hash.AppendData(Encoding.UTF8.GetBytes($"{request.Method}\n{request.Path}\n"));

        request.Body.Position = 0; // requires Request.EnableBuffering() earlier in the pipeline
        var chunk = new byte[8192];
        int read;
        while ((read = await request.Body.ReadAsync(chunk, ct)) > 0)
        {
            hash.AppendData(chunk, 0, read);
        }
        request.Body.Position = 0;
        return Convert.ToHexString(hash.GetHashAndReset());
    }

    private static async Task WriteStoredAsync(HttpContext http, StoredResponse stored, CancellationToken ct)
    {
        http.Response.StatusCode = stored.StatusCode;
        http.Response.ContentType = stored.ContentType;
        if (stored.Location is not null)
        {
            http.Response.Headers.Location = stored.Location;
        }
        http.Response.Headers["Idempotent-Replayed"] = "true";
        await http.Response.WriteAsync(stored.Body, ct);
    }

    [GeneratedRegex("^[A-Za-z0-9_-]{8,128}$")]
    private static partial Regex KeyPattern();
}
Program.cs (excerpt)
using Amazon.DynamoDBv2;
using Harbourline.AccountInformation.Api.Idempotency;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddProblemDetails();
builder.Services.AddSingleton(TimeProvider.System);
builder.Services.Configure<IdempotencyOptions>(builder.Configuration.GetSection("Idempotency"));
builder.Services.AddAWSService<IAmazonDynamoDB>();                       // AWSSDK.Extensions.NETCore.Setup
builder.Services.AddSingleton<IIdempotencyStore, DynamoDbIdempotencyStore>();
// Authentication and authorization setup omitted; see the Cognito and JWT lessons.

var app = builder.Build();

app.UseExceptionHandler();
app.Use((http, next) =>                                                   // let the filter re-read the body
{
    if (HttpMethods.IsPost(http.Request.Method)) http.Request.EnableBuffering();
    return next(http);
});
app.UseAuthentication();
app.UseAuthorization();

app.MapPost("/v1/accounts/{accountId}/statement-requests",
        (string accountId, StatementRequestBody body) =>
        {
            var id = $"STR-{Guid.CreateVersion7():N}";
            return Results.Accepted($"/v1/statement-requests/{id}", new { statementRequestId = id, status = "queued" });
        })
    .RequireAuthorization("statements:write")
    .AddEndpointFilter<IdempotencyFilter>();

app.Run();

public sealed record StatementRequestBody(DateOnly FromDate, DateOnly ToDate, string Format);

Code explained line by line

  1. IIdempotencyStore hides the storage technology; tests use InMemoryIdempotencyStore, production uses DynamoDB.
  2. TryBeginAsync builds the item with status = IN_PROGRESS, the fingerprint, a short lockExpiresAt (60 s) and a long expiresAt (24 h, the TTL attribute).
  3. The ConditionExpression lets the write succeed only for a new key, an expired-but-not-yet-deleted record, or an abandoned lock. status is a reserved word, hence the #s placeholder.
  4. ReturnValuesOnConditionCheckFailure.ALL_OLD makes DynamoDB return the existing item in ConditionalCheckFailedException.Item; with SDK v4 collections can be null, so the code checks Count and falls back to a consistent GetItem.
  5. The comparison order matters: a different fingerprint is always 422, even if the original is still running; only matching requests become 409 or a replay.
  6. CompleteAsync stores status code, content type, Location and body. Keep these small (item limit 400 KB).
  7. ReleaseAsync deletes the record only if it is still IN_PROGRESS, so a completed record can never be deleted by a late failure path.
  8. In the filter, ParseKey accepts the quoted Structured Field form and the bare form, and validates length and characters before the key touches storage or logs.
  9. http.User.FindFirstValue("customer_id") assumes the authentication step has added the resolved customer as a claim (the module project resolves sub to a customer through ICustomerDirectory). The key is scoped as customerId#key.
  10. FingerprintAsync hashes method, path and raw body with SHA-256 and rewinds the stream afterwards; EnableBuffering() in Program.cs makes the rewind possible after model binding.
  11. The response is captured by swapping Response.Body for a MemoryStream, executing the endpoint's IResult into it, persisting, and then copying it to the real stream. Results.Empty tells the framework there is nothing more to write.
  12. Exceptions and 5xx release the key; 2xx and 4xx are stored, so a retry gets the same answer.
  13. AddEndpointFilter<IdempotencyFilter>() creates the filter once per endpoint with constructor injection, so its dependencies must be singletons — IAmazonDynamoDB and the store are.

Where Powertools fits

Powertools for AWS Lambda (.NET) ships an Idempotency utility as the NuGet package AWS.Lambda.Powertools.Idempotency (3.2.x, on AWS SDK for .NET v4). You call Idempotency.Configure(b => b.UseDynamoDb("table")) and decorate a handler or any method with [Idempotent]. It hashes the event (or a JMESPath subset) into the key, stores INPROGRESS/COMPLETED records with an expiration TTL attribute (default 1 hour), throws IdempotencyAlreadyInProgressException for concurrent duplicates and can validate a payload subset (IdempotencyValidationException). It is the right tool for event-driven Lambda handlers (SQS, EventBridge) and simple API handlers. For an ASP.NET Core API on Lambda, the custom filter above gives you the HTTP contract the draft describes — header-based keys, 409/422 problem details, per-customer scoping and replay headers — which you would otherwise have to rebuild around the utility.

Security considerations

  • Scope keys per customer from the token. A global key space lets one customer block or probe another's operations.
  • Validate the key format before using it in storage or logs (length, characters) to prevent log injection and oversized items.
  • Stored responses are personal data: encrypt at rest (default for DynamoDB), restrict IAM to the API role (GetItem, PutItem, UpdateItem, DeleteItem on this table only), and keep TTL short. Document the retention in your records of processing — a UK GDPR question for the data protection officer, not a technical default.
  • Do not replay across identities. If the key matches but the caller differs, the scoped key differs, so this cannot happen by design.
  • An idempotency key is not an authorisation token and not a replay-attack defence for signed messages (see webhooks for that).

⚠ Learning shortcut — not for production

The demo table uses RemovalPolicy.DESTROY and no point-in-time recovery, and the filter stores whole response bodies. In production keep RETAIN, enable point-in-time recovery if the table supports dispute investigations, store only what replay needs, consider a customer-managed AWS KMS key if your data classification requires it, and alarm on ConditionalCheckFailedRequests and throttling metrics so abuse or bugs are visible.

Banking use case

Harbourline's payments API (Module 7 builds it) requires Idempotency-Key on POST /v1/payments. The mobile app generates a UUID v4 when the customer presses Confirm and reuses it for every retry of that confirmation, including after the app restarts (the key is stored with the draft payment). The API scopes the key per customer and keeps it for 24 hours. Behind the API, the payments service rejects a second payment with the same end_to_end_id for the same debtor account. In a quarterly review, operations sees 0.4% of payment POSTs replayed and zero duplicate payments — and can prove it from the Idempotent-Replayed field in the access logs.

Common mistakes

  • Generating a new key on every retry → the key is created inside the retry loop → create it once per user intent and reuse it.
  • Check-then-insert (GetItem, then PutItem if absent) → looks natural → two concurrent requests both see "absent"; use a single conditional write.
  • Relying on TTL for exact expiry → TTL deletion can lag by days → compare expiresAt in the condition and in reads.
  • Storing 5xx responses → "replay everything" → the client can never recover from a transient failure; release the key instead.
  • Global key scope → simpler code → cross-customer interference; prefix with the customer id.
  • Caching keys in memory on Lambda → works in local tests → each execution environment has its own memory; use a shared store.
  • Assuming the key makes downstream processing exactly-once → it only covers the HTTP edge; add natural keys such as end_to_end_id downstream.

Troubleshooting

SymptomLikely causeFix
Every retry returns 409 foreverProcess crashed after claiming the key; no lock timeoutUse lockExpiresAt in the condition (as above)
Retries return 422 although the client resent "the same" requestBody bytes differ (timestamp field, different JSON serializer settings, null vs missing)Remove volatile fields from the body or fingerprint a canonical form
ValidationException: Invalid ConditionExpression: Attribute name is a reserved keyword; reserved keyword: statusUsed status directly in an expressionUse ExpressionAttributeNames (#s)
Unable to read beyond the end of the stream or empty fingerprintBody not buffered before model bindingCall Request.EnableBuffering() in middleware before endpoints
AccessDeniedException on PutItemLambda role lacks DynamoDB permissionsGrant dynamodb:PutItem, GetItem, UpdateItem, DeleteItem on the table ARN
Old keys still visible in the table after 24 hTTL deletes "within a few days"Expected; your condition already treats them as expired
ResourceNotFoundExceptionWrong Region or table nameCheck --region us-east-1 and the Idempotency:TableName setting

Cost and cleanup

Cost: effectively $0. Local runs use DynamoDB Local (free). In AWS, storage is inside the 25 GB free tier; on-demand requests cost about $0.625 per million writes and $0.125 per million reads in us-east-1 (checked 2026-11-17), so a few hundred lab requests cost a small fraction of a cent. TTL deletions do not consume write capacity.

Teardown:

# Delete the table (irreversible; the lab has no point-in-time recovery).
aws dynamodb delete-table --table-name hl-m04-idempotency --profile learn-sandbox --region us-east-1
aws dynamodb wait table-not-exists --table-name hl-m04-idempotency --profile learn-sandbox --region us-east-1

# Confirm nothing tagged for the course remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab --profile learn-sandbox --region us-east-1

Exercises

  • 🟢 Guided: Run DynamoDB Local, create the table, and write an xUnit test against DynamoDbIdempotencyStore (pointing AmazonDynamoDBConfig.ServiceURL at http://localhost:8000) that proves: first call Started, second call with the same fingerprint InProgress, after CompleteAsync a third call Replay.
  • 🟡 Challenge: Fire 20 concurrent TryBeginAsync calls with the same key using Task.WhenAll and assert that exactly one returns Started. Then do the same with an in-memory store built on ConcurrentDictionary.TryAdd and explain why both pass but only one is correct on Lambda.
  • 🔴 Live problem: Support reports two statement PDFs for the same request. Logs show the first POST returned 504 from API Gateway after 29 seconds, and the retry 31 seconds later returned 202 without Idempotent-Replayed. The lock timeout is 30 seconds. Explain the sequence and propose two fixes.

Assignment

Add an Idempotency folder to your portfolio copy of the Account Information API with the store interface, an in-memory store, the DynamoDB store and the filter; add tests for 400 (missing key), replay with Idempotent-Replayed: true, 422 (same key, different body) and 409 (concurrent request). Write a short docs/idempotency.md that documents your policy exactly as a client developer would need it.

Solutions

Knowledge check

Interview questions

  • "How would you prevent duplicate payments when a mobile client retries?" — Client-generated Idempotency-Key per user intent, server-side store with an atomic conditional write, fingerprint check (422 on mismatch), 409 while in progress, replay of the stored response, per-customer scope, TTL; plus a unique end_to_end_id constraint in the payments service.
  • "Why not just make the client stop retrying POST?" — Because then lost responses become lost operations or confused customers; retries are necessary, so the server must make them safe.
  • "What do you store, and for how long?" — Status, fingerprint, response status/headers/body needed for replay, lock and record expiry; 24 hours is common; shorter if the data is sensitive and clients retry quickly.
  • "How do you handle a crash between claiming the key and finishing?" — A lock expiry so the key becomes claimable again, plus domain-level uniqueness so a re-run cannot duplicate the effect.
  • "When would you use Powertools Idempotency?" — For Lambda event handlers (SQS, EventBridge, Kinesis) where the event payload or a JMESPath subset is the natural key.

Certification practice

Completion checklist

  • I can explain why retries happen and why exactly-once is an effect, not a guarantee
  • I know the draft's 400 / 422 / 409 semantics and that it is not yet an RFC
  • I implemented and tested the filter and store against DynamoDB Local
  • I created, tested and deleted the hl-m04-idempotency table (or chose the local-only path)
  • The tag check shows nothing with Project=cea-lab remaining

Key takeaways

  • Retries are inevitable; idempotency keys make POST safe to repeat.
  • Same key + same fingerprint → replay; same key + different fingerprint → 422; still running → 409; missing → 400.
  • One atomic conditional write in DynamoDB is the arbiter; handle expired-but-undeleted TTL items and abandoned locks in the condition.
  • Scope keys per customer, keep records for a documented window, and treat stored responses as personal data.
  • Back the HTTP layer with domain uniqueness (end_to_end_id) — that is how "exactly once" is actually achieved.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page