Idempotency keys: making POST safe to retry
Implement the Idempotency-Key header in ASP.NET Core 10 with an endpoint filter and a DynamoDB store that uses conditional writes and TTL, so retries never create duplicate statements or payments.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.DynamoDBv2 4.x) · AWS.Lambda.Powertools.Idempotency 3.2.x · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-17.
Learning objectives
By the end of this lesson you can:
- Explain why retries happen between a mobile app, API Gateway, Lambda and a core banking system, and why "exactly once" delivery is a myth.
- Implement the
Idempotency-Keyrequest header following the IETF httpapi draft: 400 when missing, replay on a repeat, 422 for a reused key with a different payload, 409 while the first request is still running. - Build a request fingerprint and scope keys per customer with an expiry.
- Store idempotency records in Amazon DynamoDB using a conditional
PutItemwithattribute_not_exists, a lock timeout and TTL. - Decide when to use your own filter and when to use Powertools for AWS Lambda (.NET) Idempotency.
Prerequisites
- REST API design for banking — safe vs idempotent methods, status codes, problem details
- Module 3: .NET on AWS — dependency injection and the AWS SDK for .NET v4
- Tools: .NET 10 SDK, Docker (for DynamoDB Local), AWS CLI v2 with the
learn-sandboxprofile (optional AWS part)
Why this matters in banking
A customer on a train taps Request statement. The request reaches Harbourline's API, the statement job is queued, and the 202 response is lost in a tunnel. The app's HTTP client retries. Without protection, the bank generates two statements — annoying. Replace "statement" with "send £2,000 to a new payee" and it becomes a complaint, a refund, a fraud-team investigation and possibly a regulatory report.
Retries are not a bug to eliminate; they are how distributed systems survive failure. The fix is to make the operation safe to repeat. That is what an idempotency key does.
The idea in plain language
The client generates a unique value — usually a UUID — for each intended operation and sends it in an Idempotency-Key header. The server remembers the key, together with a fingerprint of the request and the response it produced. If the same key arrives again:
- same fingerprint and the first request finished → send back the stored response (replay), do not run the operation again;
- same fingerprint but the first request is still running → 409 Conflict, "try again shortly";
- different fingerprint → 422 Unprocessable Content, because the client reused a key for a different operation.
Analogy: A cheque has a serial number. If the same cheque is presented twice, the bank does not pay twice — it recognises the number and returns the original outcome. If someone presents a cheque with the same number but a different amount, that is not a retry; it is an error to investigate.
Why retries happen
| Source | What happens | Duplicate risk |
|---|---|---|
| Mobile networks | Response lost after the server committed | Client retries a successful request |
| API Gateway integration timeout | REST API gives up after 29 s by default and returns 504, but the Lambda function keeps running and may finish | Client retries an operation that is still in progress or already done |
| Client libraries | HttpClient with Microsoft.Extensions.Http.Resilience retries on 5xx, 408, 429 and network errors | Retries POST unless you disable it or add a key |
| Load balancers and proxies | Connection reset mid-response | Same as above |
| Asynchronous AWS services | Lambda asynchronous invocations retry on error; SQS and EventBridge deliver at least once | Consumers see the same message more than once |
| Humans | Double-click, refresh, "it didn't work" | Two intentional-looking requests |
The "exactly once" myth
Networks can give you at most once (never retry, sometimes lose work) or at least once (retry, sometimes duplicate). "Exactly once" is not a delivery guarantee you can buy; it is an effect you build: at-least-once delivery plus idempotent processing. An idempotency key covers the HTTP edge. Downstream, every step that moves money needs its own natural duplicate check — for payments, a unique constraint on the ISO 20022 EndToEndId (end_to_end_id in Harbourline's schema) per debtor account, so that even if two different keys slip through, the second payment is rejected by the payments service. See ISO 20022 messages.
What problem the AWS service solves
You need a store that many API instances (Lambda execution environments or Fargate tasks) share, that can atomically say "this key did not exist and now it does", and that forgets old keys automatically.
Amazon DynamoDB fits well:
- A conditional write (
PutItemwithConditionExpression = "attribute_not_exists(pk)") succeeds for exactly one concurrent writer; the others getConditionalCheckFailedException. No locks, no transactions. - Time to Live (TTL) deletes expired items automatically without consuming write throughput — but "within a few days" of expiry, not instantly. Your condition must therefore treat items whose
expiresAthas passed as absent. ReturnValuesOnConditionCheckFailure = ALL_OLDreturns the existing item inside the exception, saving a second read.
Limits and sharp edges:
- An item is at most 400 KB, so store compact responses (Powertools documents the same limit). The statement-request response is a few hundred bytes.
- Cost: the DynamoDB free tier gives 25 GB of storage per month per Region, but its free 25 WCU / 25 RCU apply to provisioned capacity only. On-demand requests are billed from the first one: about $0.625 per million write request units and $0.125 per million read request units (us-east-1, checked 2026-11-17). A lab of a few hundred requests costs well under one cent; use provisioned capacity of 1 WCU/1 RCU if you want to stay strictly inside the free tier.
- An in-memory store is fine for a single process or tests, but wrong for Lambda: each execution environment has its own memory.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
Cosmos DB conditional create (unique id, 409 Conflict on duplicate) | DynamoDB PutItem + attribute_not_exists | DynamoDB raises ConditionalCheckFailedException; you map it to HTTP codes |
Cosmos DB TTL (ttl property, seconds) | DynamoDB TTL on a Number attribute holding an epoch-seconds expiry | DynamoDB stores the absolute expiry time, not a duration; deletion can lag by days |
Azure Cache for Redis SET key NX EX | DynamoDB conditional put (or ElastiCache/Valkey SET NX) | A cache can evict keys under memory pressure — dangerous for money movement |
Service Bus duplicate detection (MessageId window) | SQS FIFO deduplication (5-minute window) | Covers messaging, not your HTTP API |
Polly retry in HttpClientFactory | Same: Microsoft.Extensions.Http.Resilience | Retrying POST is only safe with a key |
| ASP.NET Core action filter | Endpoint filter (IEndpointFilter) on a minimal API route | Same idea; runs after model binding |
Architecture
Security-boundary view: the key is client-controlled input. The server scopes it as customerId#key, where customerId comes from the validated token, so customer A can never replay or block customer B's operation by guessing a key. The table holds response bodies, which may contain personal data: it is encrypted at rest by default, access is limited to the API's IAM role, and TTL bounds retention (24 hours here).
Step-by-step implementation
Read the standard and decide your contract
The IETF httpapi working group's "The Idempotency-Key HTTP Header Field" is still an Internet-Draft (revision 08, 22 October 2026, not yet an RFC). Its key points:
- The header is a Structured Field String, so the canonical form is quoted:
Idempotency-Key: "8c1e5a5e-4f6a-4a51-9d55-1f7a4c2b3e90". Many APIs also accept the unquoted form; Harbourline accepts both. - Keys should be unique; a UUID (RFC 9562) is recommended.
- The resource defines whether a fingerprint is used and documents the expiry policy.
- Errors: 400 if the key is missing for an operation that requires it, 422 if a key is reused with a different payload, 409 if a retry arrives while the original is still being processed — ideally as RFC 9457 problem details.
The draft does not define a header that signals a replay. Harbourline adds Idempotent-Replayed: true, a convention some payment APIs use, so clients and logs can tell replays apart.
Harbourline's documented policy: required on POST .../statement-requests; keys are 8–128 characters of letters, digits, - and _; scoped per customer; kept for 24 hours; responses with status codes below 500 are stored and replayed; 5xx responses and exceptions release the key so the client can retry.
Define the fingerprint
The fingerprint answers "is this really the same request?". Harbourline uses SHA-256 over the method, the path and the raw body bytes. Same key + same fingerprint = retry; same key + different fingerprint = client bug (422).
Choices to make consciously: excluding headers such as User-Agent (they can change on retry), and hashing raw bytes (simple; a retry from the same client sends identical bytes) rather than canonicalised JSON (more tolerant, more code). Because the endpoint filter runs after model binding has read the body, the app calls Request.EnableBuffering() early in the pipeline so the filter can rewind and hash it.
Design the storage record
| Attribute | Type | Purpose |
|---|---|---|
pk | S | CUS-00012345#8c1e5a5e-... — partition key, customer-scoped |
status | S | IN_PROGRESS or COMPLETED (a DynamoDB reserved word, so expressions use #s) |
fingerprint | S | Hex SHA-256 of method, path and body |
lockExpiresAt | N | Epoch seconds after which an IN_PROGRESS record is considered abandoned (crash recovery) |
expiresAt | N | Epoch seconds; the TTL attribute (24 h) |
statusCode, contentType, responseLocation, body | N / S | Stored response for replay |
The first write uses this condition:
attribute_not_exists(pk) OR expiresAt < :now OR (#s = :inProgress AND lockExpiresAt < :now)It succeeds if the key is new, if an old record has expired but TTL has not deleted it yet, or if a previous attempt crashed while holding the lock.
Handle the race
Two retries can arrive within milliseconds on two different Lambda execution environments. Both call PutItem; DynamoDB serialises writes to the same item, so exactly one succeeds. The loser receives ConditionalCheckFailedException with the winner's item and returns 409 (in progress) or a replay. No distributed lock service is needed.
Protect the business operation too
There is still a gap: if the handler succeeds but the final UpdateItem to COMPLETED fails (throttling, a crash), the record stays IN_PROGRESS until lockExpiresAt, after which a retry runs the handler again. Close it in the domain: the statement-request handler stores the idempotency key on the statement request it creates, with a unique constraint, and a payment carries a unique end_to_end_id. Defence in depth — the HTTP layer prevents almost all duplicates, the domain prevents the rest.
AWS Console instructions
Optional, if you prefer clicking to the CLI: Console → DynamoDB → Tables → Create table. Table name hl-m04-idempotency, partition key pk (String), no sort key. Under Table settings choose Customize settings → Capacity mode: On-demand. Add tags Project=cea-lab, Lab=m04-idempotency, Environment=learn, Owner=<your-name>. Create, then open the table → Additional settings → Time to Live (TTL) → Turn on, attribute name expiresAt. To inspect records later: Explore items on the table.
AWS CLI
# 1. Create the table: on-demand, one String partition key, mandatory tags.
aws dynamodb create-table \
--table-name hl-m04-idempotency \
--attribute-definitions AttributeName=pk,AttributeType=S \
--key-schema AttributeName=pk,KeyType=HASH \
--billing-mode PAY_PER_REQUEST \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m04-idempotency Key=Environment,Value=learn Key=Owner,Value=your-name \
--profile learn-sandbox --region us-east-1
# 2. Wait until the table is ACTIVE.
aws dynamodb wait table-exists --table-name hl-m04-idempotency --profile learn-sandbox --region us-east-1
# 3. Turn on TTL using the expiresAt attribute (epoch seconds).
aws dynamodb update-time-to-live \
--table-name hl-m04-idempotency \
--time-to-live-specification "Enabled=true,AttributeName=expiresAt" \
--profile learn-sandbox --region us-east-1
# 4. Prove the conditional write: the first put succeeds, the second fails with ConditionalCheckFailedException.
aws dynamodb put-item --table-name hl-m04-idempotency \
--item '{"pk":{"S":"CUS-00012345#demo-key-001"},"status":{"S":"IN_PROGRESS"}}' \
--condition-expression "attribute_not_exists(pk)" \
--profile learn-sandbox --region us-east-1
aws dynamodb put-item --table-name hl-m04-idempotency \
--item '{"pk":{"S":"CUS-00012345#demo-key-001"},"status":{"S":"IN_PROGRESS"}}' \
--condition-expression "attribute_not_exists(pk)" \
--profile learn-sandbox --region us-east-1
# 5. Inspect the item.
aws dynamodb get-item --table-name hl-m04-idempotency \
--key '{"pk":{"S":"CUS-00012345#demo-key-001"}}' --consistent-read \
--profile learn-sandbox --region us-east-1In PowerShell, wrap the JSON arguments in single quotes as shown, or put them in files and use file://item.json to avoid quoting problems.
Free local alternative: run DynamoDB Local in Docker and point the CLI and SDK at it.
docker run --rm -p 8000:8000 amazon/dynamodb-local
aws dynamodb create-table --table-name hl-m04-idempotency \
--attribute-definitions AttributeName=pk,AttributeType=S \
--key-schema AttributeName=pk,KeyType=HASH --billing-mode PAY_PER_REQUEST \
--endpoint-url http://localhost:8000 --region us-east-1Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.Lambda;
using Constructs;
namespace Harbourline.AccountInformation.Infra;
public sealed class IdempotencyTable : Construct
{
public TableV2 Table { get; }
public IdempotencyTable(Construct scope, string id, IFunction apiFunction) : base(scope, id)
{
Table = new TableV2(this, "Table", new TablePropsV2
{
TableName = "hl-m04-idempotency",
PartitionKey = new Amazon.CDK.AWS.DynamoDB.Attribute { Name = "pk", Type = AttributeType.STRING },
Billing = Billing.OnDemand(),
TimeToLiveAttribute = "expiresAt",
RemovalPolicy = RemovalPolicy.DESTROY // ⚠ lab only: production keeps RETAIN
});
// Least privilege: data-plane read/write on this table only.
Table.GrantReadWriteData(apiFunction);
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m04-idempotency");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "your-name");
}
}Complete code
These files are simplified excerpts of the module project, labs/m04-account-information-api/src/Harbourline.AccountInformation.Api/Idempotency/. They compile on their own (with AWSSDK.DynamoDBv2 4.x); member names in the lab may differ slightly, and the lab is the source of truth.
namespace Harbourline.AccountInformation.Api.Idempotency;
public sealed record StoredResponse(int StatusCode, string? ContentType, string? Location, string Body);
public enum BeginOutcome { Started, Replay, InProgress, FingerprintMismatch }
public sealed record BeginResult(BeginOutcome Outcome, StoredResponse? Response = null);
public interface IIdempotencyStore
{
/// <summary>Atomically claims the key, or reports what already exists for it.</summary>
Task<BeginResult> TryBeginAsync(string scopedKey, string fingerprint, CancellationToken ct);
/// <summary>Marks the key COMPLETED and stores the response for replay.</summary>
Task CompleteAsync(string scopedKey, StoredResponse response, CancellationToken ct);
/// <summary>Deletes an IN_PROGRESS record so the client may retry (used after 5xx or exceptions).</summary>
Task ReleaseAsync(string scopedKey, CancellationToken ct);
}using System.Globalization;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
using Microsoft.Extensions.Options;
namespace Harbourline.AccountInformation.Api.Idempotency;
public sealed class IdempotencyOptions
{
public string TableName { get; set; } = "hl-m04-idempotency";
public TimeSpan RecordTtl { get; set; } = TimeSpan.FromHours(24);
public TimeSpan LockTimeout { get; set; } = TimeSpan.FromSeconds(60);
}
public sealed class DynamoDbIdempotencyStore(
IAmazonDynamoDB dynamo, IOptions<IdempotencyOptions> options, TimeProvider time) : IIdempotencyStore
{
private const string InProgress = "IN_PROGRESS";
private const string Completed = "COMPLETED";
private readonly IdempotencyOptions _opt = options.Value;
public async Task<BeginResult> TryBeginAsync(string scopedKey, string fingerprint, CancellationToken ct)
{
var now = time.GetUtcNow().ToUnixTimeSeconds();
var request = new PutItemRequest
{
TableName = _opt.TableName,
Item = new Dictionary<string, AttributeValue>
{
["pk"] = S(scopedKey),
["status"] = S(InProgress),
["fingerprint"] = S(fingerprint),
["lockExpiresAt"] = N(now + (long)_opt.LockTimeout.TotalSeconds),
["expiresAt"] = N(now + (long)_opt.RecordTtl.TotalSeconds),
},
ConditionExpression =
"attribute_not_exists(pk) OR expiresAt < :now OR (#s = :inProgress AND lockExpiresAt < :now)",
ExpressionAttributeNames = new Dictionary<string, string> { ["#s"] = "status" },
ExpressionAttributeValues = new Dictionary<string, AttributeValue>
{
[":now"] = N(now),
[":inProgress"] = S(InProgress),
},
ReturnValuesOnConditionCheckFailure = ReturnValuesOnConditionCheckFailure.ALL_OLD,
};
try
{
await dynamo.PutItemAsync(request, ct);
return new BeginResult(BeginOutcome.Started);
}
catch (ConditionalCheckFailedException ex)
{
var existing = ex.Item is { Count: > 0 } ? ex.Item : await ReadAsync(scopedKey, ct);
if (existing is null)
{
return new BeginResult(BeginOutcome.InProgress); // deleted in between: ask the client to retry
}
if (Get(existing, "fingerprint") != fingerprint)
{
return new BeginResult(BeginOutcome.FingerprintMismatch);
}
if (Get(existing, "status") != Completed)
{
return new BeginResult(BeginOutcome.InProgress);
}
var stored = new StoredResponse(
int.Parse(Get(existing, "statusCode") ?? "200", CultureInfo.InvariantCulture),
Get(existing, "contentType"),
Get(existing, "responseLocation") is { Length: > 0 } location ? location : null,
Get(existing, "body") ?? string.Empty);
return new BeginResult(BeginOutcome.Replay, stored);
}
}
public Task CompleteAsync(string scopedKey, StoredResponse response, CancellationToken ct) =>
dynamo.UpdateItemAsync(new UpdateItemRequest
{
TableName = _opt.TableName,
Key = new Dictionary<string, AttributeValue> { ["pk"] = S(scopedKey) },
UpdateExpression = "SET #s = :completed, statusCode = :code, contentType = :ct, responseLocation = :loc, #b = :body",
ConditionExpression = "attribute_exists(pk)",
ExpressionAttributeNames = new Dictionary<string, string> { ["#s"] = "status", ["#b"] = "body" },
ExpressionAttributeValues = new Dictionary<string, AttributeValue>
{
[":completed"] = S(Completed),
[":code"] = N(response.StatusCode),
[":ct"] = S(response.ContentType ?? "application/json"),
[":loc"] = S(response.Location ?? string.Empty),
[":body"] = S(response.Body),
},
}, ct);
public Task ReleaseAsync(string scopedKey, CancellationToken ct) =>
dynamo.DeleteItemAsync(new DeleteItemRequest
{
TableName = _opt.TableName,
Key = new Dictionary<string, AttributeValue> { ["pk"] = S(scopedKey) },
ConditionExpression = "#s = :inProgress",
ExpressionAttributeNames = new Dictionary<string, string> { ["#s"] = "status" },
ExpressionAttributeValues = new Dictionary<string, AttributeValue> { [":inProgress"] = S(InProgress) },
}, ct);
private async Task<Dictionary<string, AttributeValue>?> ReadAsync(string scopedKey, CancellationToken ct)
{
var response = await dynamo.GetItemAsync(new GetItemRequest
{
TableName = _opt.TableName,
Key = new Dictionary<string, AttributeValue> { ["pk"] = S(scopedKey) },
ConsistentRead = true,
}, ct);
return response.Item is { Count: > 0 } ? response.Item : null;
}
private static AttributeValue S(string value) => new() { S = value };
private static AttributeValue N(long value) => new() { N = value.ToString(CultureInfo.InvariantCulture) };
private static string? Get(Dictionary<string, AttributeValue> item, string name) =>
item.TryGetValue(name, out var v) ? v.S ?? v.N : null;
}using System.Security.Claims;
using System.Security.Cryptography;
using System.Text;
using System.Text.RegularExpressions;
namespace Harbourline.AccountInformation.Api.Idempotency;
public sealed partial class IdempotencyFilter(IIdempotencyStore store, ILogger<IdempotencyFilter> logger) : IEndpointFilter
{
public const string HeaderName = "Idempotency-Key";
private const string ProblemBase = "https://api.harbourline.example/problems/";
public async ValueTask<object?> InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
{
var http = context.HttpContext;
var ct = http.RequestAborted;
var key = ParseKey(http.Request.Headers[HeaderName].ToString());
if (key is null)
{
return Results.Problem(statusCode: 400, type: ProblemBase + "idempotency-key-missing",
title: "A valid Idempotency-Key header is required",
detail: "Send a UUID in the Idempotency-Key header (8-128 characters: letters, digits, '-' or '_').");
}
// Scope by the customer resolved from the validated token, never from the request body or URL.
var customerId = http.User.FindFirstValue("customer_id");
if (customerId is null)
{
return Results.Unauthorized();
}
var scopedKey = $"{customerId}#{key}";
var fingerprint = await FingerprintAsync(http.Request, ct);
var begin = await store.TryBeginAsync(scopedKey, fingerprint, ct);
switch (begin.Outcome)
{
case BeginOutcome.FingerprintMismatch:
return Results.Problem(statusCode: 422, type: ProblemBase + "idempotency-key-reused",
title: "Idempotency-Key was already used with a different request");
case BeginOutcome.InProgress:
http.Response.Headers.RetryAfter = "2";
return Results.Problem(statusCode: 409, type: ProblemBase + "idempotency-request-in-progress",
title: "A request with this Idempotency-Key is still being processed");
case BeginOutcome.Replay when begin.Response is not null:
logger.LogInformation("Replaying stored response for idempotency key {ScopedKey}", scopedKey);
await WriteStoredAsync(http, begin.Response, ct);
return Results.Empty;
}
// Started: run the endpoint, capture its response, persist it, then send it.
var originalBody = http.Response.Body;
using var buffer = new MemoryStream();
http.Response.Body = buffer;
try
{
var result = await next(context);
var toExecute = result as IResult ?? (result is null ? Results.Empty : Results.Json(result));
await toExecute.ExecuteAsync(http);
}
catch
{
http.Response.Body = originalBody;
await store.ReleaseAsync(scopedKey, CancellationToken.None);
throw;
}
finally
{
http.Response.Body = originalBody;
}
var body = Encoding.UTF8.GetString(buffer.ToArray());
if (http.Response.StatusCode >= 500)
{
await store.ReleaseAsync(scopedKey, CancellationToken.None);
}
else
{
var stored = new StoredResponse(http.Response.StatusCode, http.Response.ContentType,
http.Response.Headers.Location.ToString() is { Length: > 0 } loc ? loc : null, body);
await store.CompleteAsync(scopedKey, stored, CancellationToken.None);
}
buffer.Position = 0;
await buffer.CopyToAsync(originalBody, ct);
return Results.Empty;
}
private static string? ParseKey(string raw)
{
var value = raw.Trim();
if (value.Length >= 2 && value[0] == '"' && value[^1] == '"')
{
value = value[1..^1]; // Structured Field String form: "..."
}
return KeyPattern().IsMatch(value) ? value : null;
}
private static async Task<string> FingerprintAsync(HttpRequest request, CancellationToken ct)
{
using var hash = IncrementalHash.CreateHash(HashAlgorithmName.SHA256);
hash.AppendData(Encoding.UTF8.GetBytes($"{request.Method}\n{request.Path}\n"));
request.Body.Position = 0; // requires Request.EnableBuffering() earlier in the pipeline
var chunk = new byte[8192];
int read;
while ((read = await request.Body.ReadAsync(chunk, ct)) > 0)
{
hash.AppendData(chunk, 0, read);
}
request.Body.Position = 0;
return Convert.ToHexString(hash.GetHashAndReset());
}
private static async Task WriteStoredAsync(HttpContext http, StoredResponse stored, CancellationToken ct)
{
http.Response.StatusCode = stored.StatusCode;
http.Response.ContentType = stored.ContentType;
if (stored.Location is not null)
{
http.Response.Headers.Location = stored.Location;
}
http.Response.Headers["Idempotent-Replayed"] = "true";
await http.Response.WriteAsync(stored.Body, ct);
}
[GeneratedRegex("^[A-Za-z0-9_-]{8,128}$")]
private static partial Regex KeyPattern();
}using Amazon.DynamoDBv2;
using Harbourline.AccountInformation.Api.Idempotency;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddProblemDetails();
builder.Services.AddSingleton(TimeProvider.System);
builder.Services.Configure<IdempotencyOptions>(builder.Configuration.GetSection("Idempotency"));
builder.Services.AddAWSService<IAmazonDynamoDB>(); // AWSSDK.Extensions.NETCore.Setup
builder.Services.AddSingleton<IIdempotencyStore, DynamoDbIdempotencyStore>();
// Authentication and authorization setup omitted; see the Cognito and JWT lessons.
var app = builder.Build();
app.UseExceptionHandler();
app.Use((http, next) => // let the filter re-read the body
{
if (HttpMethods.IsPost(http.Request.Method)) http.Request.EnableBuffering();
return next(http);
});
app.UseAuthentication();
app.UseAuthorization();
app.MapPost("/v1/accounts/{accountId}/statement-requests",
(string accountId, StatementRequestBody body) =>
{
var id = $"STR-{Guid.CreateVersion7():N}";
return Results.Accepted($"/v1/statement-requests/{id}", new { statementRequestId = id, status = "queued" });
})
.RequireAuthorization("statements:write")
.AddEndpointFilter<IdempotencyFilter>();
app.Run();
public sealed record StatementRequestBody(DateOnly FromDate, DateOnly ToDate, string Format);Code explained line by line
IIdempotencyStorehides the storage technology; tests useInMemoryIdempotencyStore, production uses DynamoDB.TryBeginAsyncbuilds the item withstatus = IN_PROGRESS, the fingerprint, a shortlockExpiresAt(60 s) and a longexpiresAt(24 h, the TTL attribute).- The
ConditionExpressionlets the write succeed only for a new key, an expired-but-not-yet-deleted record, or an abandoned lock.statusis a reserved word, hence the#splaceholder. ReturnValuesOnConditionCheckFailure.ALL_OLDmakes DynamoDB return the existing item inConditionalCheckFailedException.Item; with SDK v4 collections can benull, so the code checksCountand falls back to a consistentGetItem.- The comparison order matters: a different fingerprint is always 422, even if the original is still running; only matching requests become 409 or a replay.
CompleteAsyncstores status code, content type,Locationand body. Keep these small (item limit 400 KB).ReleaseAsyncdeletes the record only if it is stillIN_PROGRESS, so a completed record can never be deleted by a late failure path.- In the filter,
ParseKeyaccepts the quoted Structured Field form and the bare form, and validates length and characters before the key touches storage or logs. http.User.FindFirstValue("customer_id")assumes the authentication step has added the resolved customer as a claim (the module project resolvessubto a customer throughICustomerDirectory). The key is scoped ascustomerId#key.FingerprintAsynchashes method, path and raw body with SHA-256 and rewinds the stream afterwards;EnableBuffering()inProgram.csmakes the rewind possible after model binding.- The response is captured by swapping
Response.Bodyfor aMemoryStream, executing the endpoint'sIResultinto it, persisting, and then copying it to the real stream.Results.Emptytells the framework there is nothing more to write. - Exceptions and 5xx release the key; 2xx and 4xx are stored, so a retry gets the same answer.
AddEndpointFilter<IdempotencyFilter>()creates the filter once per endpoint with constructor injection, so its dependencies must be singletons —IAmazonDynamoDBand the store are.
Where Powertools fits
Powertools for AWS Lambda (.NET) ships an Idempotency utility as the NuGet package AWS.Lambda.Powertools.Idempotency (3.2.x, on AWS SDK for .NET v4). You call Idempotency.Configure(b => b.UseDynamoDb("table")) and decorate a handler or any method with [Idempotent]. It hashes the event (or a JMESPath subset) into the key, stores INPROGRESS/COMPLETED records with an expiration TTL attribute (default 1 hour), throws IdempotencyAlreadyInProgressException for concurrent duplicates and can validate a payload subset (IdempotencyValidationException). It is the right tool for event-driven Lambda handlers (SQS, EventBridge) and simple API handlers. For an ASP.NET Core API on Lambda, the custom filter above gives you the HTTP contract the draft describes — header-based keys, 409/422 problem details, per-customer scoping and replay headers — which you would otherwise have to rebuild around the utility.
Security considerations
- Scope keys per customer from the token. A global key space lets one customer block or probe another's operations.
- Validate the key format before using it in storage or logs (length, characters) to prevent log injection and oversized items.
- Stored responses are personal data: encrypt at rest (default for DynamoDB), restrict IAM to the API role (
GetItem,PutItem,UpdateItem,DeleteItemon this table only), and keep TTL short. Document the retention in your records of processing — a UK GDPR question for the data protection officer, not a technical default. - Do not replay across identities. If the key matches but the caller differs, the scoped key differs, so this cannot happen by design.
- An idempotency key is not an authorisation token and not a replay-attack defence for signed messages (see webhooks for that).
⚠ Learning shortcut — not for production
The demo table uses RemovalPolicy.DESTROY and no point-in-time recovery, and the filter stores whole response bodies. In production keep RETAIN, enable point-in-time recovery if the table supports dispute investigations, store only what replay needs, consider a customer-managed AWS KMS key if your data classification requires it, and alarm on ConditionalCheckFailedRequests and throttling metrics so abuse or bugs are visible.
Banking use case
Harbourline's payments API (Module 7 builds it) requires Idempotency-Key on POST /v1/payments. The mobile app generates a UUID v4 when the customer presses Confirm and reuses it for every retry of that confirmation, including after the app restarts (the key is stored with the draft payment). The API scopes the key per customer and keeps it for 24 hours. Behind the API, the payments service rejects a second payment with the same end_to_end_id for the same debtor account. In a quarterly review, operations sees 0.4% of payment POSTs replayed and zero duplicate payments — and can prove it from the Idempotent-Replayed field in the access logs.
Common mistakes
- Generating a new key on every retry → the key is created inside the retry loop → create it once per user intent and reuse it.
- Check-then-insert (
GetItem, thenPutItemif absent) → looks natural → two concurrent requests both see "absent"; use a single conditional write. - Relying on TTL for exact expiry → TTL deletion can lag by days → compare
expiresAtin the condition and in reads. - Storing 5xx responses → "replay everything" → the client can never recover from a transient failure; release the key instead.
- Global key scope → simpler code → cross-customer interference; prefix with the customer id.
- Caching keys in memory on Lambda → works in local tests → each execution environment has its own memory; use a shared store.
- Assuming the key makes downstream processing exactly-once → it only covers the HTTP edge; add natural keys such as
end_to_end_iddownstream.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Every retry returns 409 forever | Process crashed after claiming the key; no lock timeout | Use lockExpiresAt in the condition (as above) |
| Retries return 422 although the client resent "the same" request | Body bytes differ (timestamp field, different JSON serializer settings, null vs missing) | Remove volatile fields from the body or fingerprint a canonical form |
ValidationException: Invalid ConditionExpression: Attribute name is a reserved keyword; reserved keyword: status | Used status directly in an expression | Use ExpressionAttributeNames (#s) |
Unable to read beyond the end of the stream or empty fingerprint | Body not buffered before model binding | Call Request.EnableBuffering() in middleware before endpoints |
AccessDeniedException on PutItem | Lambda role lacks DynamoDB permissions | Grant dynamodb:PutItem, GetItem, UpdateItem, DeleteItem on the table ARN |
| Old keys still visible in the table after 24 h | TTL deletes "within a few days" | Expected; your condition already treats them as expired |
ResourceNotFoundException | Wrong Region or table name | Check --region us-east-1 and the Idempotency:TableName setting |
Cost and cleanup
Cost: effectively $0. Local runs use DynamoDB Local (free). In AWS, storage is inside the 25 GB free tier; on-demand requests cost about $0.625 per million writes and $0.125 per million reads in us-east-1 (checked 2026-11-17), so a few hundred lab requests cost a small fraction of a cent. TTL deletions do not consume write capacity.
Teardown:
# Delete the table (irreversible; the lab has no point-in-time recovery).
aws dynamodb delete-table --table-name hl-m04-idempotency --profile learn-sandbox --region us-east-1
aws dynamodb wait table-not-exists --table-name hl-m04-idempotency --profile learn-sandbox --region us-east-1
# Confirm nothing tagged for the course remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab --profile learn-sandbox --region us-east-1Exercises
- 🟢 Guided: Run DynamoDB Local, create the table, and write an xUnit test against
DynamoDbIdempotencyStore(pointingAmazonDynamoDBConfig.ServiceURLathttp://localhost:8000) that proves: first callStarted, second call with the same fingerprintInProgress, afterCompleteAsynca third callReplay. - 🟡 Challenge: Fire 20 concurrent
TryBeginAsynccalls with the same key usingTask.WhenAlland assert that exactly one returnsStarted. Then do the same with an in-memory store built onConcurrentDictionary.TryAddand explain why both pass but only one is correct on Lambda. - 🔴 Live problem: Support reports two statement PDFs for the same request. Logs show the first POST returned 504 from API Gateway after 29 seconds, and the retry 31 seconds later returned 202 without
Idempotent-Replayed. The lock timeout is 30 seconds. Explain the sequence and propose two fixes.
Assignment
Add an Idempotency folder to your portfolio copy of the Account Information API with the store interface, an in-memory store, the DynamoDB store and the filter; add tests for 400 (missing key), replay with Idempotent-Replayed: true, 422 (same key, different body) and 409 (concurrent request). Write a short docs/idempotency.md that documents your policy exactly as a client developer would need it.
Solutions
Create the client with new AmazonDynamoDBClient(new BasicAWSCredentials("local", "local"), new AmazonDynamoDBConfig { ServiceURL = "http://localhost:8000" }) — DynamoDB Local accepts any credentials. Use FakeTimeProvider (package Microsoft.Extensions.TimeProvider.Testing) so you can also test expiry by advancing time past LockTimeout. Why: the conditional write and the reserved-word handling are only proven against a real DynamoDB engine, not a mock.
Task.WhenAll(Enumerable.Range(0, 20).Select(_ => store.TryBeginAsync(key, fp, ct))) returns one Started and nineteen InProgress, because DynamoDB serialises conditional writes on the same item. The ConcurrentDictionary store also passes inside one process, but on Lambda each execution environment has its own dictionary, so two environments can both "start". Why: correctness requires one shared, atomic arbiter.
The first request claimed the key and ran longer than 30 seconds; API Gateway returned 504 at 29 seconds while Lambda kept working. The retry arrived after lockExpiresAt had passed, so the condition treated the lock as abandoned and ran the handler again. Fixes: (1) set LockTimeout longer than the maximum handler duration (at least the Lambda timeout), and (2) make the handler itself idempotent — persist the statement request with the idempotency key under a unique constraint so a second insert fails and returns the existing request. Also consider making the POST fast (enqueue and return 202) so it never approaches the gateway timeout. Why: the HTTP-level lock is only as good as its timeout; the domain guard covers the rest.
Knowledge check
Answer: 400 Bad Request, 422 Unprocessable Content and 409 Conflict respectively, ideally with RFC 9457 problem details. The draft is still an Internet-Draft (revision 08, October 2026), not an RFC.
Answer: It is atomic. Two concurrent requests can both read "not found", but only one conditional write can succeed; the other receives ConditionalCheckFailedException.
Answer: No. Expired items are usually deleted within a few days. Treat items whose expiresAt is in the past as absent in your conditions and reads.
Answer: Keys are client-chosen. Without scoping, one customer could replay or block another customer's operation by reusing their key, and accidental collisions between customers would cause wrong replays.
Answer: No. It deduplicates at the HTTP edge. Downstream steps still receive at-least-once delivery and need natural keys (for example a unique end_to_end_id for payments).
Interview questions
- "How would you prevent duplicate payments when a mobile client retries?" — Client-generated
Idempotency-Keyper user intent, server-side store with an atomic conditional write, fingerprint check (422 on mismatch), 409 while in progress, replay of the stored response, per-customer scope, TTL; plus a uniqueend_to_end_idconstraint in the payments service. - "Why not just make the client stop retrying POST?" — Because then lost responses become lost operations or confused customers; retries are necessary, so the server must make them safe.
- "What do you store, and for how long?" — Status, fingerprint, response status/headers/body needed for replay, lock and record expiry; 24 hours is common; shorter if the data is sensitive and clients retry quickly.
- "How do you handle a crash between claiming the key and finishing?" — A lock expiry so the key becomes claimable again, plus domain-level uniqueness so a re-run cannot duplicate the effect.
- "When would you use Powertools Idempotency?" — For Lambda event handlers (SQS, EventBridge, Kinesis) where the event payload or a JMESPath subset is the natural key.
Certification practice
A Lambda function behind Amazon API Gateway creates orders in Amazon DynamoDB. Clients sometimes retry after timeouts, creating duplicate orders. Each request includes a client-generated request ID. What is the MOST reliable way to prevent duplicates?
A. Configure API Gateway caching for the POST method
B. Write the order with PutItem and a condition expression attribute_not_exists on a key derived from the request ID
C. Reduce the Lambda function timeout so clients receive errors faster
D. Use a Scan before each write to check whether the request ID exists
Answer: B. A conditional write is atomic and works across concurrent invocations. A does not apply caching semantics to POST and would not stop the write. C makes timeouts more likely. D is slow, expensive and still racy between the scan and the write.
A developer enables DynamoDB TTL on an expiresAt attribute so idempotency records disappear after 24 hours. Testers report that records older than 24 hours are still returned by GetItem. What is the cause and the correct fix?
A. TTL only works on provisioned tables; switch capacity mode
B. TTL deletes expired items asynchronously, typically within a few days; the application must ignore items whose expiresAt is in the past
C. The TTL attribute must be a String in ISO 8601 format
D. TTL requires DynamoDB Streams to be enabled
Answer: B. TTL deletion is a background process. A is false — TTL works in both capacity modes. C is wrong: the attribute must be a Number holding epoch seconds. D is wrong: Streams are optional and only record the deletions.
An order service consumes messages from an Amazon SQS standard queue and charges customers' cards through a partner API. Occasionally a customer is charged twice. What should the solutions architect recommend?
A. Switch to a larger instance type for the consumer B. Make the consumer idempotent by recording each processed message's business key in a store with a conditional write before calling the partner, and pass an idempotency key to the partner API C. Increase the queue's visibility timeout to 12 hours D. Enable long polling on the queue
Answer: B. SQS standard queues deliver at least once, so consumers must tolerate duplicates; recording a business key atomically and forwarding an idempotency key protects the side effect. A, C and D change performance or timing but cannot rule out duplicate delivery.
Completion checklist
- I can explain why retries happen and why exactly-once is an effect, not a guarantee
- I know the draft's 400 / 422 / 409 semantics and that it is not yet an RFC
- I implemented and tested the filter and store against DynamoDB Local
- I created, tested and deleted the
hl-m04-idempotencytable (or chose the local-only path) - The tag check shows nothing with
Project=cea-labremaining
Key takeaways
- Retries are inevitable; idempotency keys make POST safe to repeat.
- Same key + same fingerprint → replay; same key + different fingerprint → 422; still running → 409; missing → 400.
- One atomic conditional write in DynamoDB is the arbiter; handle expired-but-undeleted TTL items and abandoned locks in the condition.
- Scope keys per customer, keep records for a documented window, and treat stored responses as personal data.
- Back the HTTP layer with domain uniqueness (
end_to_end_id) — that is how "exactly once" is actually achieved.
Further reading
- IETF draft: The Idempotency-Key HTTP Header Field
- Amazon Builders' Library: Making retries safe with idempotent APIs
- DynamoDB condition expression examples (conditional put)
- Using time to live (TTL) in DynamoDB
- DynamoDB on-demand pricing
- Powertools for AWS Lambda (.NET) — Idempotency
- RFC 9457 — Problem Details for HTTP APIs
- Module project: Account Information API
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
REST API design for banking
Design resource URIs, status codes, pagination, money fields, versioning and RFC 9457 problem details for a banking API, and implement them in ASP.NET Core 10.
OpenAPI contracts and contract testing
Design the Account Information API in OpenAPI, generate it from ASP.NET Core 10, import it into API Gateway, generate a typed C# client with Kiota, and stop breaking changes with contract tests, Pact and oasdiff.