Playbook: duplicate payment on client retry
Trace how one customer instruction became two payments, prove it from logs and the transactions table, reverse the duplicate safely with an auditable correcting entry, and close the gap with idempotency keys and conditional writes.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · Amazon DynamoDB · CloudWatch Logs Insights · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this playbook you can:
- Explain the five common ways one customer instruction turns into two payments: client timeout retry, double tap, at-least-once message delivery, an asynchronous Lambda retry and a re-run batch.
- Prove a duplicate from evidence: a CloudWatch Logs Insights query grouped by idempotency key, correlation ID and a transactions-table query.
- Contain the incident in the first five minutes without making it worse (no deletes, no blind replays).
- Reverse a duplicate with a correcting ledger entry that carries its own audit trail, instead of editing or deleting the original.
- Fix the root cause with a DynamoDB conditional write that stores the key and a request fingerprint, and prove the fix with a test that retries concurrently.
- Write the incident record an auditor and the payments operations team will accept.
Prerequisites
- Idempotency keys: making POST safe to retry: the design of the
Idempotency-Keyheader and the store - Idempotency in .NET with DynamoDB and Powertools: the reusable store and the Lambda utility
- Retry strategies: why retries exist and how they multiply
- Troubleshooting production issues methodically: the symptom, hypothesis, evidence and fix loop used here
- Tools: AWS CLI v2 with the
learn-sandboxSSO profile. The local reproduction needs only .NET 10 and Docker (DynamoDB Local)
Why this matters in banking
On Monday morning a Harbourline Bank customer rings the contact centre: "I paid my landlord GBP 1,250.00 on Friday evening and it left my account twice." The statement agrees. The mobile app showed a spinner, the customer tapped Pay again, and nobody noticed. A second possibility is worse: the customer tapped once, the app timed out after 10 seconds, the network library retried automatically, and both requests reached the Payments Service.
A duplicated payment is not just a bug. It is a customer-harm event: money has left the account, an overdraft fee may have been triggered, and under complaint-handling rules the bank must respond within set time limits (check the exact limits with your compliance team). It also reaches the payment scheme, so a plain "delete the row" is impossible. The fix is a correcting transaction that is itself approved, logged and explainable. The method in this playbook is what keeps a calm senior engineer from making the incident bigger.
The idea in plain language
A payment API is a promise: "do this once". The network cannot keep that promise on its own, because a client that sends a request and hears nothing cannot tell "the server never got it" from "the server did it but the reply was lost". A well-behaved client therefore retries, and a well-behaved server must make a retry harmless. That property is called idempotency: doing the same operation twice has the same effect as doing it once.
Analogy: The cashier at a Harbourline branch hands you a numbered slip when you pay in a cheque. If you come back five minutes later and say "did you process my cheque?", the cashier looks the slip number up instead of processing it again. The slip number is the idempotency key. Without it, the cashier has to guess whether you are a new customer or the same one.
The five ways one instruction becomes two payments
| # | Cause | Where it happens | Tell-tale evidence |
|---|---|---|---|
| 1 | Client timeout, then automatic retry | Mobile app, HTTP library, API client | Two requests seconds apart, same customer, same amount. The first one has a long latency or a 504 recorded by the gateway |
| 2 | Double tap or double submit | Browser or app UI | Two requests under one second apart, often with different request IDs but the same session |
| 3 | At-least-once delivery | SQS standard queue, EventBridge, SNS to SQS | Same message ID processed twice. A visibility timeout shorter than processing time is the classic trigger |
| 4 | Asynchronous Lambda retry | Lambda invoked asynchronously (for example from S3 or EventBridge) | Same awsRequestId family retried. The default for asynchronous invocation is up to two retries after the first failure |
| 5 | Re-run batch or replayed file | EOD batch restarted halfway, a partner file loaded twice | Many duplicates with the same batch or file identifier and timestamps minutes apart |
Only causes 1 and 2 involve the customer's device. Causes 3 to 5 are inside the bank, and the customer never did anything wrong.
What problem the AWS services solve
No AWS service makes your payment logic idempotent for you. The services give you the building blocks:
- Amazon DynamoDB gives you a conditional write (
attribute_not_exists) that is atomic for a single item. It is the cheapest correct "claim this key" operation.TransactWriteItemsaccepts aClientRequestTokenthat makes the whole transaction idempotent for a 10-minute window (⚠ VERIFY the window against the current API reference before you rely on the number). - Amazon SQS FIFO queues de-duplicate messages that share a
MessageDeduplicationIdwithin a 5-minute window. That is a delivery-layer guard, not a business-level one: it does not stop two different messages that both mean "pay this invoice". - AWS Lambda Powertools (.NET) idempotency utility wraps a handler and uses a DynamoDB table so a repeated event returns the stored result.
- CloudWatch Logs Insights, X-Ray and CloudTrail are your evidence tools.
Honest limits:
- A DynamoDB TTL deletes expired items eventually (often within hours, but AWS documents up to about 48 hours, ⚠ VERIFY), so your code must treat an item whose TTL has passed as expired, not wait for the delete.
- An idempotency store protects only the code paths that use it. A second, older code path that writes the same table (an admin tool, a batch job) bypasses it.
- Retry-safe design does not remove the need for reconciliation. See Reconciliation architecture.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
Service Bus duplicate detection (RequiresDuplicateDetection, history window) | SQS FIFO MessageDeduplicationId (5-minute window) | Both are delivery-layer only. Neither knows two different messages mean the same business instruction |
| Cosmos DB unique key constraint or ETag precondition | DynamoDB ConditionExpression (attribute_not_exists, version checks) | The condition is per request and per item; there is no multi-item unique constraint |
SQL Server unique index on IdempotencyKey plus INSERT catching error 2627 | DynamoDB conditional PutItem catching ConditionalCheckFailedException | No locking or deadlocks, but no cross-item constraint either. Use TransactWriteItems for payment plus key |
| Azure Functions durable orchestration with instance IDs | Step Functions execution name (unique per state machine for standard workflows for 90 days) | A second StartExecution with the same name and the same input returns the existing execution. ⚠ VERIFY the exact behaviour for the workflow type you use |
Application Insights operation_Id | A correlation ID in structured logs plus the X-Ray trace ID | You must put it in the logs yourself. Nothing adds it by default |
| Polly retry policy | AddStandardResilienceHandler and SDK retry settings | A retry policy without idempotency on the server is the root cause of this playbook |
Architecture
Where a duplicate can enter
There are three places the same instruction can arrive twice: the client retry at the front (key K1), the queue redelivery in the middle (use the payment ID as the key) and the call to the scheme at the end (use end_to_end_id as the key). A fix at only one layer leaves the other two open.
The investigation loop
Step-by-step implementation
This is the playbook itself. Work through it top to bottom the first time; after that use it as a checklist.
Minutes 0 to 5: scope and contain
Answer three questions before touching anything:
- How many customers? One duplicate is a support case. A hundred is a live incident. Query the transactions table for pairs (see the next step) over the last 24 hours before you decide.
- Is it still happening? If new duplicates appear every minute, stopping the bleeding comes before diagnosis. The safest brakes, from least to most disruptive:
- raise a feature flag or configuration value that makes the payments endpoint return
503withRetry-Afterfor new payments only; - set the queue's consumer concurrency to zero (Lambda event source mapping disabled, or ECS desired count 0 for the worker) so nothing new is submitted to the scheme while the backlog stays safe in the queue;
- never purge the queue and never delete table rows. Both destroy evidence and can lose genuine payments.
- raise a feature flag or configuration value that makes the payments endpoint return
- Who must know? Open an incident in your bank's own tooling and notify payments operations (they can recall or return funds with the scheme) and the data protection or complaints owner if customer-visible harm is confirmed. This playbook does not depend on any AWS incident tool.
Write the first timeline entry now: time, who, what you saw, what you did. The log you start here becomes the audit record.
Minutes 5 to 20: collect evidence
Capture before you fix. Evidence you need, in order:
- the two (or more) transaction rows, with
transaction_id,created_at,channel,idempotency_keyandstatus; - the API request log lines for each, with correlation ID, latency and status code;
- the gateway view: did API Gateway or the load balancer record a
504or a client disconnect for the first request? - the queue view:
ApproximateAgeOfOldestMessage,NumberOfMessagesReceivedversusNumberOfMessagesSent, and the DLQ depth; - the Lambda view, if a function is involved: invocation type, the
Errorsmetric and anyTask timed outlines.
The queries are in AWS CLI below.
Minutes 20 to 40: rank the causes
Use the key pattern to decide:
| Pattern in the evidence | Most likely cause | Next check |
|---|---|---|
Same idempotency_key, two rows | The key check is missing on this path, or is a read-then-write race | Search the code for a GetItem followed by PutItem. Look for a second writer to the table |
| Different keys, same customer, same amount and creditor, seconds apart | The client creates a new key per attempt (for example a new GUID inside the retry loop) | Read the client code. The key must be generated once per user intent, before the first attempt |
No key at all (null or empty) | A caller that never sends the header (older app version, partner, batch) | Count requests by userAgent and app version in the logs |
| One API request, one payment row, two scheme submissions | Redelivery or async retry downstream | Compare queue NumberOfMessagesReceived with sent, check visibility timeout versus processing time, and check the worker's idempotency on payment_id |
| Many duplicates sharing a batch or file ID | Batch restart or file loaded twice | Check the batch run history and the file's checksum register |
Minutes 40 to 60: reverse the duplicate safely
Rules for the correction:
- Never delete or edit the original rows. The ledger is append-only. Two payments happened; one of them must be corrected by a new, linked entry.
- Check the scheme status first. If the second payment is still
receivedorvalidated, it can usually be cancelled before submission. If it issubmittedorsettled, payments operations decides between a recall or return request through the scheme and a refund. Ask them; do not guess scheme rules (⚠ VERIFY the current process with your payments operations team). - Use the bank's normal four-eyes approval for any reversal. An engineer running a script on their own is a segregation-of-duties failure even if the maths is right.
- Make the reversal itself idempotent. Derive its key from the duplicate's ID (for example
reversal-<transaction_id>), so running the script twice cannot refund twice. This is the same lesson, applied to the fix. - Notify the customer through the normal channel and link the case ID, per your complaints process.
Minutes 60 onwards: fix the root cause and prove it
Add the conditional write from Complete code, add a test that fires the same request concurrently, and deploy through the normal pipeline. Then add the detection: an alarm on duplicate-key hits (they are normal in small numbers, because they mean the guard worked) and a daily reconciliation query for pairs that slipped through. Details are in the Exercises.
AWS Console instructions
- Console → CloudWatch → Logs → Logs Insights. Select the Payments Service log group (for example
/hl/payments-service), set the time range to cover the report, paste a query from the CLI section and choose Run query. - Console → DynamoDB → Tables →
hl-payments→ Explore table items. Use Query on theGSI1index to list one customer's payments, sorted bycreated_at. - Console → SQS → Queues → payment-submission → Monitoring. Compare Number of messages received with Number of messages sent, and look at Approximate age of oldest message.
- Console → Lambda → Functions → your worker → Monitor. Check Errors, Throttles and Duration, then View CloudWatch logs.
AWS CLI
# 1. Find payments for one customer account in a time window (fictional IDs).
# GSI1 is keyed on debtor_account_id with created_at as the sort key.
aws dynamodb query \
--table-name hl-payments \
--index-name GSI1 \
--key-condition-expression "debtor_account_id = :a AND created_at BETWEEN :from AND :to" \
--expression-attribute-values '{":a":{"S":"ACC-0000123456"},":from":{"S":"2026-11-13T18:00:00Z"},":to":{"S":"2026-11-13T20:00:00Z"}}' \
--projection-expression "payment_id, amount, creditor_iban, status, created_at, idempotency_key" \
--profile learn-sandbox
# 2. Start a Logs Insights query: which idempotency keys appear on more than one accepted payment?
aws logs start-query \
--log-group-name /hl/payments-service \
--start-time $(date -d '2 days ago' +%s) --end-time $(date +%s) \
--query-string 'fields @timestamp, idempotencyKey, correlationId, paymentId
| filter event = "PaymentAccepted"
| stats count(*) as accepted, min(@timestamp) as first, max(@timestamp) as last by idempotencyKey
| filter accepted > 1
| sort accepted desc' \
--profile learn-sandbox
# The command returns a queryId. Fetch results (re-run until status is Complete):
aws logs get-query-results --query-id <queryId> --profile learn-sandbox
# 3. Queue health: messages received versus sent in the last 24 hours (5-minute sums).
aws cloudwatch get-metric-statistics \
--namespace AWS/SQS --metric-name NumberOfMessagesReceived \
--dimensions Name=QueueName,Value=hl-payment-submission \
--start-time $(date -u -d '24 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
--period 300 --statistics Sum --profile learn-sandbox
# 4. Brake: stop the worker consuming without deleting anything.
# List the event source mapping, then disable it. Messages stay safely in the queue.
aws lambda list-event-source-mappings --function-name hl-payment-submit --profile learn-sandbox
aws lambda update-event-source-mapping --uuid <uuid-from-list> --no-enabled --profile learn-sandboxTwo more Logs Insights queries worth keeping in your runbook:
# Requests that arrived with no idempotency key at all, by app version
fields @timestamp, appVersion, userAgent
| filter path = "/payments" and method = "POST" and ispresent(idempotencyKey) = 0
| stats count(*) as n by appVersion
| sort n desc
# First attempts that the gateway recorded as slow or failed (retries follow these)
fields @timestamp, correlationId, status, latencyMs
| filter path = "/payments" and (status >= 500 or latencyMs > 8000)
| sort @timestamp desc
| limit 50Infrastructure as Code
The idempotency table is the heart of the fix: one partition key, TTL on an expiry attribute, point-in-time recovery on and encryption with a customer managed key.
using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.KMS;
using Constructs;
public class IdempotencyStack : Stack
{
public IdempotencyStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
{
var key = new Key(this, "IdemKey", new KeyProps { EnableKeyRotation = true });
var table = new Table(this, "IdempotencyTable", new TableProps
{
TableName = "hl-payments-idempotency",
PartitionKey = new Amazon.CDK.AWS.DynamoDB.Attribute { Name = "pk", Type = AttributeType.STRING },
BillingMode = BillingMode.PAY_PER_REQUEST,
TimeToLiveAttribute = "expires_at", // epoch seconds
PointInTimeRecoverySpecification = new PointInTimeRecoverySpecification
{
PointInTimeRecoveryEnabled = true
},
Encryption = TableEncryption.CUSTOMER_MANAGED,
EncryptionKey = key,
RemovalPolicy = RemovalPolicy.DESTROY // learning only
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-duplicate");
}
}Complete code
This is the fix: a store that claims a key atomically, records a fingerprint of the request body (so the same key with a different amount is rejected rather than replayed) and replays the stored result on a true retry.
using System.Security.Cryptography;
using System.Text;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
namespace Harbourline.Payments;
public enum ClaimOutcome { Claimed, ReplayCompleted, InProgress, KeyReusedWithDifferentRequest }
public sealed record ClaimResult(ClaimOutcome Outcome, string? StoredResponseJson);
public sealed class IdempotencyStore(IAmazonDynamoDB ddb, string tableName, TimeProvider clock)
{
private static readonly TimeSpan Retention = TimeSpan.FromHours(24);
private static readonly TimeSpan InProgressLease = TimeSpan.FromSeconds(60);
public static string Fingerprint(string customerId, string requestBody) =>
Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes($"{customerId}\n{requestBody}")));
public async Task<ClaimResult> TryClaimAsync(
string customerId, string idempotencyKey, string requestBody, CancellationToken ct)
{
var pk = $"{customerId}#{idempotencyKey}"; // scope the key per customer
var fingerprint = Fingerprint(customerId, requestBody);
var now = clock.GetUtcNow();
var item = new Dictionary<string, AttributeValue>
{
["pk"] = new() { S = pk },
["fingerprint"] = new() { S = fingerprint },
["status"] = new() { S = "IN_PROGRESS" },
["lease_until"] = new() { N = now.Add(InProgressLease).ToUnixTimeSeconds().ToString() },
["expires_at"] = new() { N = now.Add(Retention).ToUnixTimeSeconds().ToString() }
};
try
{
await ddb.PutItemAsync(new PutItemRequest
{
TableName = tableName,
Item = item,
// Succeeds only if nobody has the key, or the old record has expired
// (TTL deletion can lag, so we also test expires_at ourselves).
ConditionExpression = "attribute_not_exists(pk) OR expires_at < :now",
ExpressionAttributeValues = new() { [":now"] = new() { N = now.ToUnixTimeSeconds().ToString() } },
ReturnValuesOnConditionCheckFailure = ReturnValuesOnConditionCheckFailure.ALL_OLD
}, ct);
return new ClaimResult(ClaimOutcome.Claimed, null);
}
catch (ConditionalCheckFailedException ex) when (ex.Item is { Count: > 0 })
{
var existing = ex.Item;
if (existing["fingerprint"].S != fingerprint)
return new ClaimResult(ClaimOutcome.KeyReusedWithDifferentRequest, null);
if (existing["status"].S == "COMPLETED")
return new ClaimResult(ClaimOutcome.ReplayCompleted, existing["response"].S);
return new ClaimResult(ClaimOutcome.InProgress, null);
}
}
public Task CompleteAsync(string customerId, string idempotencyKey, string responseJson, CancellationToken ct) =>
ddb.UpdateItemAsync(new UpdateItemRequest
{
TableName = tableName,
Key = new() { ["pk"] = new() { S = $"{customerId}#{idempotencyKey}" } },
UpdateExpression = "SET #s = :done, response = :r",
ExpressionAttributeNames = new() { ["#s"] = "status" },
ExpressionAttributeValues = new()
{
[":done"] = new() { S = "COMPLETED" },
[":r"] = new() { S = responseJson }
}
}, ct);
}And the endpoint that uses it:
using System.Text.Json;
using Harbourline.Payments;
public static class PaymentEndpoint
{
public static void Map(WebApplication app) =>
app.MapPost("/payments", async (
HttpRequest http, PaymentRequest body, IdempotencyStore store,
PaymentService payments, ILogger<Program> log, CancellationToken ct) =>
{
if (!http.Headers.TryGetValue("Idempotency-Key", out var keyValues) || string.IsNullOrWhiteSpace(keyValues))
return Results.Problem("Idempotency-Key header is required.", statusCode: 400);
var key = keyValues.ToString();
var customerId = http.HttpContext.User.FindFirst("sub")!.Value;
var raw = JsonSerializer.Serialize(body);
var claim = await store.TryClaimAsync(customerId, key, raw, ct);
log.LogInformation("Idempotency {Outcome} for key {IdempotencyKey}", claim.Outcome, key);
switch (claim.Outcome)
{
case ClaimOutcome.ReplayCompleted:
return Results.Content(claim.StoredResponseJson!, "application/json", statusCode: 200);
case ClaimOutcome.InProgress:
http.HttpContext.Response.Headers.RetryAfter = "2";
return Results.Problem("The original request is still being processed.", statusCode: 409);
case ClaimOutcome.KeyReusedWithDifferentRequest:
return Results.Problem("This Idempotency-Key was used with a different request.", statusCode: 422);
}
var payment = await payments.CreateAsync(body, key, ct); // writes payment + outbox atomically
var response = JsonSerializer.Serialize(payment);
await store.CompleteAsync(customerId, key, response, ct);
log.LogInformation("PaymentAccepted {PaymentId} {IdempotencyKey}", payment.PaymentId, key);
return Results.Content(response, "application/json", statusCode: 201);
});
}
public sealed record PaymentRequest(string DebtorAccountId, string CreditorIban, decimal Amount, string Currency);Code explained line by line
Fingerprinthashes the customer ID and the body, so a client that reuses a key for a different amount is detected rather than silently replayed. Hashing keeps the stored record small and avoids storing full request bodies unnecessarily (data minimisation).var pk = $"{customerId}#{idempotencyKey}"scopes keys per customer. Without this, two customers who both sendkey-1would collide, and one could read the other's stored response.- The
PutItemwithConditionExpressionis the atomic claim. Two concurrent requests with the same key both try it; DynamoDB lets exactly one succeed. A read followed by a write would have a race window between the two calls. expires_at < :nowhandles the lag between a record expiring and DynamoDB's TTL process actually deleting it.ReturnValuesOnConditionCheckFailure.ALL_OLDreturns the existing item inside the exception, saving a second read.InProgressreturns409withRetry-After. The client can safely retry after a short wait and will then receive the stored response.CompleteAsyncstores the response, so later retries replay the original answer, including the original payment ID.- The endpoint calls
payments.CreateAsync(...)only when the claim succeeded. If the process crashes between claim andCompleteAsync, the record staysIN_PROGRESSuntil its lease passes. Reconciliation must handle that rare case (see Security considerations).
Security considerations
⚠ Learning shortcut — not for production
The code above claims the key and then writes the payment in a separate step. A crash between the two leaves a claimed key with no payment, and a crash after the payment but before CompleteAsync leaves a payment with an IN_PROGRESS key. For production, write the payment, its outbox row and the completed idempotency record in one TransactWriteItems call (or one SQL transaction), as shown in the idempotency lesson and the outbox lesson.
- Scope keys per authenticated principal. A key must never allow one customer to learn another's response. Take the customer ID from the validated token, never from the request body.
- Do not log full request bodies or account numbers while investigating. Log the key, the payment ID and a correlation ID. The fingerprint is enough to prove two requests were identical.
- The reversal is a privileged operation. Limit it to a dedicated role, require approval, and log the approver and the ticket. See CloudTrail for banks for the evidence your reversal role's activity leaves behind.
- Retention is a policy choice. The 24-hour window here is a learning default. A key that expires too early re-opens the duplicate window; one kept forever grows the table and holds personal data longer than needed. Agree the window with your data protection team (this is a UK GDPR storage-limitation question, not only a technical one).
- This playbook does not make a system compliant with anything; it describes engineering practice to bring to your own compliance review.
Banking use case
Harbourline's Payments Service accepts Faster Payments style instructions. The app generates a UUID once per user intent (when the user taps Pay, before the first attempt) and sends it as Idempotency-Key. API Gateway passes the header through; the service claims it in DynamoDB; the payment and an outbox row are written; a worker submits to the scheme gateway using end_to_end_id as its own de-duplication key. Reconciliation (Module 14) runs a nightly job that lists any two booked payments with the same debtor, creditor, amount and a gap of under 10 minutes, and flags them for review. That nightly job is the detective control; the idempotency store is the preventive one. A bank needs both.
Common mistakes
- Generating the key inside the retry loop. Every attempt gets a fresh GUID, so the server sees distinct requests. Generate once, outside the loop.
- Read-then-write.
GetItem, "not found",PutItem. Two parallel requests both see "not found". Always use a conditional write. - Treating the DLQ or purge button as a fix. Purging a queue during an incident removes genuine payments and the evidence.
- Deleting the duplicate row. The scheme already has the instruction. Append a correcting entry instead.
- Forgetting downstream layers. A perfect API guard does nothing if the SQS worker resubmits on redelivery. Give every hop its own key.
- Silently replaying on key reuse with a different body. That hides client bugs and can move the wrong amount. Reject with
422.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Duplicates continue after deploying the guard | An older service version, an admin tool or a batch job writes payments without the guard | List all writers to the table with CloudTrail data events or code search. Route every writer through the same service |
409 responses never clear | A crash left the record IN_PROGRESS and the lease has no recovery path | Treat lease_until in the past as claimable (extend the condition), or add a recovery sweeper |
| Legitimate second payment is rejected as a duplicate | Key reused by the client for two different intents, but the body matched | Ensure the client creates a new key per user action. Include the intended execution date in the fingerprint |
ConditionalCheckFailedException with no Item | ReturnValuesOnConditionCheckFailure not set, or an old SDK | Set ALL_OLD, or fall back to a GetItem after the failure |
| Replay returns a different payment ID | The stored response was not saved, or a retry took a different code path | Always store the response before returning. Test with a retry in your suite |
| Idempotency table throttled during a peak | On-demand table warming, or one hot partition key | Keys are per customer so spread is usually fine; check ThrottledRequests and the key design |
Cost and cleanup
Cost label: free. The local reproduction uses Docker and DynamoDB Local. If you deploy the table to the sandbox, on-demand DynamoDB for a few hundred test writes costs a few cents at most (⚠ VERIFY current pricing), and the KMS key costs about one US dollar per month if left for the whole month (⚠ VERIFY). Destroy it the same day.
# Local reproduction cleanup
docker rm -f ddb-local
# If you deployed the table in the sandbox
aws dynamodb delete-table --table-name hl-payments-idempotency --profile learn-sandbox
# Schedule the KMS key for deletion (minimum waiting period is 7 days)
aws kms schedule-key-deletion --key-id <key-id> --pending-window-in-days 7 --profile learn-sandbox
# Confirm nothing tagged for this lesson is left
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-duplicate --profile learn-sandboxExercises
- 🟢 Guided: Start DynamoDB Local (
docker run -d --name ddb-local -p 8000:8000 amazon/dynamodb-local). Create the idempotency table, callTryClaimAsynctwice with the same key and body, and confirm the second result isInProgress. CallCompleteAsync, call again, and confirmReplayCompleted. - 🟡 Challenge: Write an xUnit test that fires 50 concurrent
TryClaimAsynccalls with one key and asserts that exactly one returnsClaimed. Then change the body for one call and assertKeyReusedWithDifferentRequest. - 🔴 Live problem: Tuesday 09:10: "Eleven customers report duplicate card top-ups since the weekend release. The app team says the app always sends a key. The payments team says the keys are all different." You have read access to the logs and the table. Write the three Logs Insights queries you run first, say what each would prove, and draft the first three lines of the incident timeline.
Assignment
Write a blameless incident report (one page) for a simulated duplicate-payment incident, and commit it to your portfolio repository as incident-reports/duplicate-payment.md. Include: summary, customer impact, timeline, root cause, the fix with a link to your test, what went well, what you would change, and the audit evidence list. Use fictional IDs only. This feeds the Module 17 project.
Solutions
The first TryClaimAsync returns Claimed and writes an IN_PROGRESS record. The second call, with the same key and the same body, hits the condition (pk exists and has not expired), receives the old item in the exception, sees the fingerprint matches and the status is not COMPLETED, so it returns InProgress. After CompleteAsync, the status is COMPLETED and the stored response is present, so the third call returns ReplayCompleted with the original response. Why: the claim is atomic inside DynamoDB, so the outcome does not depend on timing in your code.
Use Task.WhenAll(Enumerable.Range(0, 50).Select(_ => store.TryClaimAsync(c, k, body, ct))). Group results by Outcome: you should see one Claimed and 49 InProgress. Do the same-key different-body call after the first claim; it must return KeyReusedWithDifferentRequest. Why: the test proves atomicity under contention, which is the property a read-then-write implementation fails. If the test is flaky with a naive version, that flakiness is your proof of the race.
Query 1: payments in the window grouped by customer_id, amount, creditor_iban with count above one, to confirm the pairs and the scale. Query 2: for those pairs, show idempotencyKey per row. If keys differ, the client generates a key per attempt. Query 3: gateway latency and status for the first request of each pair, to confirm timeouts triggered the retry. First timeline lines: 09:10 report received (eleven customers, top-ups); 09:15 scope query run (count, time range, channel); 09:20 hypothesis recorded: client generates a new key per attempt after the release, to be confirmed against the app version in the logs. Then raise it with the app team with evidence, containing by either rolling back the app release or enabling a server-side fingerprint guard (same customer, amount, creditor and a short window) as a temporary measure agreed with product. Why: the two teams disagree about the same facts; the logs by key settle it without blame.
Knowledge check
Answer: Return the stored response of the first request (replay), without creating a second payment. The key plus the request fingerprint tell the server it is a true retry. Creating a new payment, or returning an error without the stored result, would either duplicate the payment or leave the client unsure.
Answer: Two concurrent requests can both read "not found" before either writes, so both proceed. A single conditional write (attribute_not_exists) makes the check and the claim one atomic operation.
Answer: Reject (for example HTTP 422). Replaying the first response would hide a client bug and the customer might think the second amount was accepted. Replay is only correct when the request is identical.
Answer: The payment was already part of the ledger and may have reached the scheme. Deleting it falsifies history, breaks reconciliation and removes the audit trail. Post a linked, approved correcting entry instead.
Answer: It should be, but only if your code checks expires_at itself. DynamoDB TTL deletion is eventual, so a conditional write that only tests attribute_not_exists would wrongly reject the new request.
Interview questions
- A customer says they were charged twice. How do you start? Scope and contain first (how many, still happening, who must be told), then collect evidence by idempotency key, correlation ID and transaction rows before changing anything, then rank hypotheses by the key pattern.
- How do you make a payment API safe to retry? A client-generated key per user intent, a server-side atomic claim with a request fingerprint, replay of the stored response, and a distinct key at each downstream hop.
- Why is SQS FIFO deduplication not enough? It only covers identical
MessageDeduplicationIdvalues inside a 5-minute window. It cannot tell that two different messages mean one business instruction. - How do you reverse a duplicate? A correcting ledger entry through four-eyes approval, an idempotent reversal key, checking scheme status with payments operations, customer notification and a full audit record.
- What is the difference between preventive and detective controls here? The idempotency store prevents; the nightly reconciliation for near-identical payments detects what slipped through. Banks run both.
Certification practice
A payment service on AWS Lambda behind API Gateway is occasionally invoked twice for the same customer instruction after client timeouts. Payments are stored in Amazon DynamoDB. Which change most reliably prevents the second invocation from creating a second payment?
A. Increase the Lambda function timeout to 15 minutes.
B. Store a client-supplied idempotency key with a conditional write (attribute_not_exists) before processing, and return the stored result on repeats.
C. Enable DynamoDB Streams and delete the later item when two items match.
D. Put the Lambda function in a VPC so retries cannot reach it.
Answer: B. The conditional write is atomic, so exactly one invocation claims the key. A hides the symptom only for slow processing, not for lost responses. C acts after the damage (and deleting a payment is wrong anyway). D has no effect on client retries.
An operations engineer must find every request in the last two days that reached the application more than once with the same idempotencyKey, using structured JSON logs in CloudWatch Logs. Which approach is best?
A. Create a metric filter and wait for new data.
B. Run a CloudWatch Logs Insights query with stats count(*) by idempotencyKey and filter counts above one.
C. Enable VPC Flow Logs and search for repeated source IPs.
D. Use CloudTrail event history to search for the key.
Answer: B. Logs Insights queries existing log data and aggregates by field. A only counts future events. C shows network metadata, not application fields. D records AWS API calls, not application-level request data.
A standard SQS queue feeds a payment submission worker. Occasionally the same message is delivered twice. The company needs each payment to be submitted once. Which design is most appropriate?
A. Switch to a larger instance type for the worker. B. Make the worker idempotent using the payment ID as a key in a conditional write before submitting, and set the visibility timeout above the processing time. C. Reduce the queue's retention period to one minute. D. Disable the dead-letter queue.
Answer: B. Standard queues deliver at least once, so consumers must be idempotent, and the visibility timeout must exceed processing time to avoid premature redelivery. A, C and D do not address redelivery (C and D can lose messages).
Completion checklist
- I can list the five ways one instruction becomes two payments
- I can run the Logs Insights query that groups accepted payments by idempotency key
- I know the three brakes I can pull in the first five minutes, and that purging a queue is not one of them
- I can explain why the reversal is a new linked entry, not a delete
- I completed the guided exercise and the concurrency test
- I ran the cleanup steps and confirmed nothing tagged
m17-playbook-duplicateremains
Key takeaways
- A retry is only safe if the server can recognise it: client key per intent, atomic claim, fingerprint, replay.
- Contain first, then collect evidence, then fix. Never purge or delete during an incident.
- Reverse with an approved, idempotent, linked correcting entry, never an edit.
- Guard every hop: API key, queue consumer (payment ID), scheme submission (end-to-end ID).
- Pair the preventive guard with a detective reconciliation check.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Module 17: Daily-Work Playbooks & Career
Step-by-step diagnosis guides for the problems you will actually face in a bank running .NET on AWS, from duplicate payments to AccessDenied errors and surprise bills. The module closes with interview preparation, whiteboard practice and a guide to presenting your capstone projects as a portfolio.
Playbook: end-of-day batch overrunning its window
Find the slow stage of a nightly batch, tell data skew from saturation from waiting, buy time without skipping controls, and add progress-based alarms so the next overrun is caught at 01:00 instead of 05:00.