Skip to Content
Cloud Engineering Academy
17. Daily-Work Playbooks & Career

Playbook: duplicate payment on client retry

Trace how one customer instruction became two payments, prove it from logs and the transactions table, reverse the duplicate safely with an auditable correcting entry, and close the gap with idempotency keys and conditional writes.

intermediate⏱ 1 h 15 min🟢 FreeDVA-C03SAA-C03SOA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · Amazon DynamoDB · CloudWatch Logs Insights · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this playbook you can:

  • Explain the five common ways one customer instruction turns into two payments: client timeout retry, double tap, at-least-once message delivery, an asynchronous Lambda retry and a re-run batch.
  • Prove a duplicate from evidence: a CloudWatch Logs Insights query grouped by idempotency key, correlation ID and a transactions-table query.
  • Contain the incident in the first five minutes without making it worse (no deletes, no blind replays).
  • Reverse a duplicate with a correcting ledger entry that carries its own audit trail, instead of editing or deleting the original.
  • Fix the root cause with a DynamoDB conditional write that stores the key and a request fingerprint, and prove the fix with a test that retries concurrently.
  • Write the incident record an auditor and the payments operations team will accept.

Prerequisites

Why this matters in banking

On Monday morning a Harbourline Bank customer rings the contact centre: "I paid my landlord GBP 1,250.00 on Friday evening and it left my account twice." The statement agrees. The mobile app showed a spinner, the customer tapped Pay again, and nobody noticed. A second possibility is worse: the customer tapped once, the app timed out after 10 seconds, the network library retried automatically, and both requests reached the Payments Service.

A duplicated payment is not just a bug. It is a customer-harm event: money has left the account, an overdraft fee may have been triggered, and under complaint-handling rules the bank must respond within set time limits (check the exact limits with your compliance team). It also reaches the payment scheme, so a plain "delete the row" is impossible. The fix is a correcting transaction that is itself approved, logged and explainable. The method in this playbook is what keeps a calm senior engineer from making the incident bigger.

The idea in plain language

A payment API is a promise: "do this once". The network cannot keep that promise on its own, because a client that sends a request and hears nothing cannot tell "the server never got it" from "the server did it but the reply was lost". A well-behaved client therefore retries, and a well-behaved server must make a retry harmless. That property is called idempotency: doing the same operation twice has the same effect as doing it once.

Analogy: The cashier at a Harbourline branch hands you a numbered slip when you pay in a cheque. If you come back five minutes later and say "did you process my cheque?", the cashier looks the slip number up instead of processing it again. The slip number is the idempotency key. Without it, the cashier has to guess whether you are a new customer or the same one.

The five ways one instruction becomes two payments

#CauseWhere it happensTell-tale evidence
1Client timeout, then automatic retryMobile app, HTTP library, API clientTwo requests seconds apart, same customer, same amount. The first one has a long latency or a 504 recorded by the gateway
2Double tap or double submitBrowser or app UITwo requests under one second apart, often with different request IDs but the same session
3At-least-once deliverySQS standard queue, EventBridge, SNS to SQSSame message ID processed twice. A visibility timeout shorter than processing time is the classic trigger
4Asynchronous Lambda retryLambda invoked asynchronously (for example from S3 or EventBridge)Same awsRequestId family retried. The default for asynchronous invocation is up to two retries after the first failure
5Re-run batch or replayed fileEOD batch restarted halfway, a partner file loaded twiceMany duplicates with the same batch or file identifier and timestamps minutes apart

Only causes 1 and 2 involve the customer's device. Causes 3 to 5 are inside the bank, and the customer never did anything wrong.

What problem the AWS services solve

No AWS service makes your payment logic idempotent for you. The services give you the building blocks:

  • Amazon DynamoDB gives you a conditional write (attribute_not_exists) that is atomic for a single item. It is the cheapest correct "claim this key" operation. TransactWriteItems accepts a ClientRequestToken that makes the whole transaction idempotent for a 10-minute window (⚠ VERIFY the window against the current API reference before you rely on the number).
  • Amazon SQS FIFO queues de-duplicate messages that share a MessageDeduplicationId within a 5-minute window. That is a delivery-layer guard, not a business-level one: it does not stop two different messages that both mean "pay this invoice".
  • AWS Lambda Powertools (.NET) idempotency utility wraps a handler and uses a DynamoDB table so a repeated event returns the stored result.
  • CloudWatch Logs Insights, X-Ray and CloudTrail are your evidence tools.

Honest limits:

  • A DynamoDB TTL deletes expired items eventually (often within hours, but AWS documents up to about 48 hours, ⚠ VERIFY), so your code must treat an item whose TTL has passed as expired, not wait for the delete.
  • An idempotency store protects only the code paths that use it. A second, older code path that writes the same table (an admin tool, a batch job) bypasses it.
  • Retry-safe design does not remove the need for reconciliation. See Reconciliation architecture.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Service Bus duplicate detection (RequiresDuplicateDetection, history window)SQS FIFO MessageDeduplicationId (5-minute window)Both are delivery-layer only. Neither knows two different messages mean the same business instruction
Cosmos DB unique key constraint or ETag preconditionDynamoDB ConditionExpression (attribute_not_exists, version checks)The condition is per request and per item; there is no multi-item unique constraint
SQL Server unique index on IdempotencyKey plus INSERT catching error 2627DynamoDB conditional PutItem catching ConditionalCheckFailedExceptionNo locking or deadlocks, but no cross-item constraint either. Use TransactWriteItems for payment plus key
Azure Functions durable orchestration with instance IDsStep Functions execution name (unique per state machine for standard workflows for 90 days)A second StartExecution with the same name and the same input returns the existing execution. ⚠ VERIFY the exact behaviour for the workflow type you use
Application Insights operation_IdA correlation ID in structured logs plus the X-Ray trace IDYou must put it in the logs yourself. Nothing adds it by default
Polly retry policyAddStandardResilienceHandler and SDK retry settingsA retry policy without idempotency on the server is the root cause of this playbook

Architecture

Where a duplicate can enter

There are three places the same instruction can arrive twice: the client retry at the front (key K1), the queue redelivery in the middle (use the payment ID as the key) and the call to the scheme at the end (use end_to_end_id as the key). A fix at only one layer leaves the other two open.

The investigation loop

Step-by-step implementation

This is the playbook itself. Work through it top to bottom the first time; after that use it as a checklist.

Minutes 0 to 5: scope and contain

Answer three questions before touching anything:

  1. How many customers? One duplicate is a support case. A hundred is a live incident. Query the transactions table for pairs (see the next step) over the last 24 hours before you decide.
  2. Is it still happening? If new duplicates appear every minute, stopping the bleeding comes before diagnosis. The safest brakes, from least to most disruptive:
    • raise a feature flag or configuration value that makes the payments endpoint return 503 with Retry-After for new payments only;
    • set the queue's consumer concurrency to zero (Lambda event source mapping disabled, or ECS desired count 0 for the worker) so nothing new is submitted to the scheme while the backlog stays safe in the queue;
    • never purge the queue and never delete table rows. Both destroy evidence and can lose genuine payments.
  3. Who must know? Open an incident in your bank's own tooling and notify payments operations (they can recall or return funds with the scheme) and the data protection or complaints owner if customer-visible harm is confirmed. This playbook does not depend on any AWS incident tool.

Write the first timeline entry now: time, who, what you saw, what you did. The log you start here becomes the audit record.

Minutes 5 to 20: collect evidence

Capture before you fix. Evidence you need, in order:

  • the two (or more) transaction rows, with transaction_id, created_at, channel, idempotency_key and status;
  • the API request log lines for each, with correlation ID, latency and status code;
  • the gateway view: did API Gateway or the load balancer record a 504 or a client disconnect for the first request?
  • the queue view: ApproximateAgeOfOldestMessage, NumberOfMessagesReceived versus NumberOfMessagesSent, and the DLQ depth;
  • the Lambda view, if a function is involved: invocation type, the Errors metric and any Task timed out lines.

The queries are in AWS CLI below.

Minutes 20 to 40: rank the causes

Use the key pattern to decide:

Pattern in the evidenceMost likely causeNext check
Same idempotency_key, two rowsThe key check is missing on this path, or is a read-then-write raceSearch the code for a GetItem followed by PutItem. Look for a second writer to the table
Different keys, same customer, same amount and creditor, seconds apartThe client creates a new key per attempt (for example a new GUID inside the retry loop)Read the client code. The key must be generated once per user intent, before the first attempt
No key at all (null or empty)A caller that never sends the header (older app version, partner, batch)Count requests by userAgent and app version in the logs
One API request, one payment row, two scheme submissionsRedelivery or async retry downstreamCompare queue NumberOfMessagesReceived with sent, check visibility timeout versus processing time, and check the worker's idempotency on payment_id
Many duplicates sharing a batch or file IDBatch restart or file loaded twiceCheck the batch run history and the file's checksum register

Minutes 40 to 60: reverse the duplicate safely

Rules for the correction:

  1. Never delete or edit the original rows. The ledger is append-only. Two payments happened; one of them must be corrected by a new, linked entry.
  2. Check the scheme status first. If the second payment is still received or validated, it can usually be cancelled before submission. If it is submitted or settled, payments operations decides between a recall or return request through the scheme and a refund. Ask them; do not guess scheme rules (⚠ VERIFY the current process with your payments operations team).
  3. Use the bank's normal four-eyes approval for any reversal. An engineer running a script on their own is a segregation-of-duties failure even if the maths is right.
  4. Make the reversal itself idempotent. Derive its key from the duplicate's ID (for example reversal-<transaction_id>), so running the script twice cannot refund twice. This is the same lesson, applied to the fix.
  5. Notify the customer through the normal channel and link the case ID, per your complaints process.

Minutes 60 onwards: fix the root cause and prove it

Add the conditional write from Complete code, add a test that fires the same request concurrently, and deploy through the normal pipeline. Then add the detection: an alarm on duplicate-key hits (they are normal in small numbers, because they mean the guard worked) and a daily reconciliation query for pairs that slipped through. Details are in the Exercises.

AWS Console instructions

  1. Console → CloudWatch → Logs → Logs Insights. Select the Payments Service log group (for example /hl/payments-service), set the time range to cover the report, paste a query from the CLI section and choose Run query.
  2. Console → DynamoDB → Tables → hl-payments → Explore table items. Use Query on the GSI1 index to list one customer's payments, sorted by created_at.
  3. Console → SQS → Queues → payment-submission → Monitoring. Compare Number of messages received with Number of messages sent, and look at Approximate age of oldest message.
  4. Console → Lambda → Functions → your worker → Monitor. Check Errors, Throttles and Duration, then View CloudWatch logs.

AWS CLI

# 1. Find payments for one customer account in a time window (fictional IDs).
#    GSI1 is keyed on debtor_account_id with created_at as the sort key.
aws dynamodb query \
  --table-name hl-payments \
  --index-name GSI1 \
  --key-condition-expression "debtor_account_id = :a AND created_at BETWEEN :from AND :to" \
  --expression-attribute-values '{":a":{"S":"ACC-0000123456"},":from":{"S":"2026-11-13T18:00:00Z"},":to":{"S":"2026-11-13T20:00:00Z"}}' \
  --projection-expression "payment_id, amount, creditor_iban, status, created_at, idempotency_key" \
  --profile learn-sandbox

# 2. Start a Logs Insights query: which idempotency keys appear on more than one accepted payment?
aws logs start-query \
  --log-group-name /hl/payments-service \
  --start-time $(date -d '2 days ago' +%s) --end-time $(date +%s) \
  --query-string 'fields @timestamp, idempotencyKey, correlationId, paymentId
    | filter event = "PaymentAccepted"
    | stats count(*) as accepted, min(@timestamp) as first, max(@timestamp) as last by idempotencyKey
    | filter accepted > 1
    | sort accepted desc' \
  --profile learn-sandbox
# The command returns a queryId. Fetch results (re-run until status is Complete):
aws logs get-query-results --query-id <queryId> --profile learn-sandbox

# 3. Queue health: messages received versus sent in the last 24 hours (5-minute sums).
aws cloudwatch get-metric-statistics \
  --namespace AWS/SQS --metric-name NumberOfMessagesReceived \
  --dimensions Name=QueueName,Value=hl-payment-submission \
  --start-time $(date -u -d '24 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --statistics Sum --profile learn-sandbox

# 4. Brake: stop the worker consuming without deleting anything.
#    List the event source mapping, then disable it. Messages stay safely in the queue.
aws lambda list-event-source-mappings --function-name hl-payment-submit --profile learn-sandbox
aws lambda update-event-source-mapping --uuid <uuid-from-list> --no-enabled --profile learn-sandbox

Two more Logs Insights queries worth keeping in your runbook:

# Requests that arrived with no idempotency key at all, by app version
fields @timestamp, appVersion, userAgent
| filter path = "/payments" and method = "POST" and ispresent(idempotencyKey) = 0
| stats count(*) as n by appVersion
| sort n desc

# First attempts that the gateway recorded as slow or failed (retries follow these)
fields @timestamp, correlationId, status, latencyMs
| filter path = "/payments" and (status >= 500 or latencyMs > 8000)
| sort @timestamp desc
| limit 50

Infrastructure as Code

The idempotency table is the heart of the fix: one partition key, TTL on an expiry attribute, point-in-time recovery on and encryption with a customer managed key.

using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.KMS;
using Constructs;

public class IdempotencyStack : Stack
{
    public IdempotencyStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
    {
        var key = new Key(this, "IdemKey", new KeyProps { EnableKeyRotation = true });

        var table = new Table(this, "IdempotencyTable", new TableProps
        {
            TableName = "hl-payments-idempotency",
            PartitionKey = new Amazon.CDK.AWS.DynamoDB.Attribute { Name = "pk", Type = AttributeType.STRING },
            BillingMode = BillingMode.PAY_PER_REQUEST,
            TimeToLiveAttribute = "expires_at",            // epoch seconds
            PointInTimeRecoverySpecification = new PointInTimeRecoverySpecification
            {
                PointInTimeRecoveryEnabled = true
            },
            Encryption = TableEncryption.CUSTOMER_MANAGED,
            EncryptionKey = key,
            RemovalPolicy = RemovalPolicy.DESTROY          // learning only
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-duplicate");
    }
}

Complete code

This is the fix: a store that claims a key atomically, records a fingerprint of the request body (so the same key with a different amount is rejected rather than replayed) and replays the stored result on a true retry.

IdempotencyStore.cs
using System.Security.Cryptography;
using System.Text;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;

namespace Harbourline.Payments;

public enum ClaimOutcome { Claimed, ReplayCompleted, InProgress, KeyReusedWithDifferentRequest }

public sealed record ClaimResult(ClaimOutcome Outcome, string? StoredResponseJson);

public sealed class IdempotencyStore(IAmazonDynamoDB ddb, string tableName, TimeProvider clock)
{
    private static readonly TimeSpan Retention = TimeSpan.FromHours(24);
    private static readonly TimeSpan InProgressLease = TimeSpan.FromSeconds(60);

    public static string Fingerprint(string customerId, string requestBody) =>
        Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes($"{customerId}\n{requestBody}")));

    public async Task<ClaimResult> TryClaimAsync(
        string customerId, string idempotencyKey, string requestBody, CancellationToken ct)
    {
        var pk = $"{customerId}#{idempotencyKey}";          // scope the key per customer
        var fingerprint = Fingerprint(customerId, requestBody);
        var now = clock.GetUtcNow();

        var item = new Dictionary<string, AttributeValue>
        {
            ["pk"] = new() { S = pk },
            ["fingerprint"] = new() { S = fingerprint },
            ["status"] = new() { S = "IN_PROGRESS" },
            ["lease_until"] = new() { N = now.Add(InProgressLease).ToUnixTimeSeconds().ToString() },
            ["expires_at"] = new() { N = now.Add(Retention).ToUnixTimeSeconds().ToString() }
        };

        try
        {
            await ddb.PutItemAsync(new PutItemRequest
            {
                TableName = tableName,
                Item = item,
                // Succeeds only if nobody has the key, or the old record has expired
                // (TTL deletion can lag, so we also test expires_at ourselves).
                ConditionExpression = "attribute_not_exists(pk) OR expires_at < :now",
                ExpressionAttributeValues = new() { [":now"] = new() { N = now.ToUnixTimeSeconds().ToString() } },
                ReturnValuesOnConditionCheckFailure = ReturnValuesOnConditionCheckFailure.ALL_OLD
            }, ct);
            return new ClaimResult(ClaimOutcome.Claimed, null);
        }
        catch (ConditionalCheckFailedException ex) when (ex.Item is { Count: > 0 })
        {
            var existing = ex.Item;
            if (existing["fingerprint"].S != fingerprint)
                return new ClaimResult(ClaimOutcome.KeyReusedWithDifferentRequest, null);

            if (existing["status"].S == "COMPLETED")
                return new ClaimResult(ClaimOutcome.ReplayCompleted, existing["response"].S);

            return new ClaimResult(ClaimOutcome.InProgress, null);
        }
    }

    public Task CompleteAsync(string customerId, string idempotencyKey, string responseJson, CancellationToken ct) =>
        ddb.UpdateItemAsync(new UpdateItemRequest
        {
            TableName = tableName,
            Key = new() { ["pk"] = new() { S = $"{customerId}#{idempotencyKey}" } },
            UpdateExpression = "SET #s = :done, response = :r",
            ExpressionAttributeNames = new() { ["#s"] = "status" },
            ExpressionAttributeValues = new()
            {
                [":done"] = new() { S = "COMPLETED" },
                [":r"] = new() { S = responseJson }
            }
        }, ct);
}

And the endpoint that uses it:

PaymentEndpoint.cs
using System.Text.Json;
using Harbourline.Payments;

public static class PaymentEndpoint
{
    public static void Map(WebApplication app) =>
        app.MapPost("/payments", async (
            HttpRequest http, PaymentRequest body, IdempotencyStore store,
            PaymentService payments, ILogger<Program> log, CancellationToken ct) =>
        {
            if (!http.Headers.TryGetValue("Idempotency-Key", out var keyValues) || string.IsNullOrWhiteSpace(keyValues))
                return Results.Problem("Idempotency-Key header is required.", statusCode: 400);

            var key = keyValues.ToString();
            var customerId = http.HttpContext.User.FindFirst("sub")!.Value;
            var raw = JsonSerializer.Serialize(body);

            var claim = await store.TryClaimAsync(customerId, key, raw, ct);
            log.LogInformation("Idempotency {Outcome} for key {IdempotencyKey}", claim.Outcome, key);

            switch (claim.Outcome)
            {
                case ClaimOutcome.ReplayCompleted:
                    return Results.Content(claim.StoredResponseJson!, "application/json", statusCode: 200);
                case ClaimOutcome.InProgress:
                    http.HttpContext.Response.Headers.RetryAfter = "2";
                    return Results.Problem("The original request is still being processed.", statusCode: 409);
                case ClaimOutcome.KeyReusedWithDifferentRequest:
                    return Results.Problem("This Idempotency-Key was used with a different request.", statusCode: 422);
            }

            var payment = await payments.CreateAsync(body, key, ct);   // writes payment + outbox atomically
            var response = JsonSerializer.Serialize(payment);
            await store.CompleteAsync(customerId, key, response, ct);
            log.LogInformation("PaymentAccepted {PaymentId} {IdempotencyKey}", payment.PaymentId, key);
            return Results.Content(response, "application/json", statusCode: 201);
        });
}

public sealed record PaymentRequest(string DebtorAccountId, string CreditorIban, decimal Amount, string Currency);

Code explained line by line

  1. Fingerprint hashes the customer ID and the body, so a client that reuses a key for a different amount is detected rather than silently replayed. Hashing keeps the stored record small and avoids storing full request bodies unnecessarily (data minimisation).
  2. var pk = $"{customerId}#{idempotencyKey}" scopes keys per customer. Without this, two customers who both send key-1 would collide, and one could read the other's stored response.
  3. The PutItem with ConditionExpression is the atomic claim. Two concurrent requests with the same key both try it; DynamoDB lets exactly one succeed. A read followed by a write would have a race window between the two calls.
  4. expires_at < :now handles the lag between a record expiring and DynamoDB's TTL process actually deleting it.
  5. ReturnValuesOnConditionCheckFailure.ALL_OLD returns the existing item inside the exception, saving a second read.
  6. InProgress returns 409 with Retry-After. The client can safely retry after a short wait and will then receive the stored response.
  7. CompleteAsync stores the response, so later retries replay the original answer, including the original payment ID.
  8. The endpoint calls payments.CreateAsync(...) only when the claim succeeded. If the process crashes between claim and CompleteAsync, the record stays IN_PROGRESS until its lease passes. Reconciliation must handle that rare case (see Security considerations).

Security considerations

⚠ Learning shortcut — not for production

The code above claims the key and then writes the payment in a separate step. A crash between the two leaves a claimed key with no payment, and a crash after the payment but before CompleteAsync leaves a payment with an IN_PROGRESS key. For production, write the payment, its outbox row and the completed idempotency record in one TransactWriteItems call (or one SQL transaction), as shown in the idempotency lesson and the outbox lesson.

  • Scope keys per authenticated principal. A key must never allow one customer to learn another's response. Take the customer ID from the validated token, never from the request body.
  • Do not log full request bodies or account numbers while investigating. Log the key, the payment ID and a correlation ID. The fingerprint is enough to prove two requests were identical.
  • The reversal is a privileged operation. Limit it to a dedicated role, require approval, and log the approver and the ticket. See CloudTrail for banks for the evidence your reversal role's activity leaves behind.
  • Retention is a policy choice. The 24-hour window here is a learning default. A key that expires too early re-opens the duplicate window; one kept forever grows the table and holds personal data longer than needed. Agree the window with your data protection team (this is a UK GDPR storage-limitation question, not only a technical one).
  • This playbook does not make a system compliant with anything; it describes engineering practice to bring to your own compliance review.

Banking use case

Harbourline's Payments Service accepts Faster Payments style instructions. The app generates a UUID once per user intent (when the user taps Pay, before the first attempt) and sends it as Idempotency-Key. API Gateway passes the header through; the service claims it in DynamoDB; the payment and an outbox row are written; a worker submits to the scheme gateway using end_to_end_id as its own de-duplication key. Reconciliation (Module 14) runs a nightly job that lists any two booked payments with the same debtor, creditor, amount and a gap of under 10 minutes, and flags them for review. That nightly job is the detective control; the idempotency store is the preventive one. A bank needs both.

Common mistakes

  • Generating the key inside the retry loop. Every attempt gets a fresh GUID, so the server sees distinct requests. Generate once, outside the loop.
  • Read-then-write. GetItem, "not found", PutItem. Two parallel requests both see "not found". Always use a conditional write.
  • Treating the DLQ or purge button as a fix. Purging a queue during an incident removes genuine payments and the evidence.
  • Deleting the duplicate row. The scheme already has the instruction. Append a correcting entry instead.
  • Forgetting downstream layers. A perfect API guard does nothing if the SQS worker resubmits on redelivery. Give every hop its own key.
  • Silently replaying on key reuse with a different body. That hides client bugs and can move the wrong amount. Reject with 422.

Troubleshooting

SymptomLikely causeFix
Duplicates continue after deploying the guardAn older service version, an admin tool or a batch job writes payments without the guardList all writers to the table with CloudTrail data events or code search. Route every writer through the same service
409 responses never clearA crash left the record IN_PROGRESS and the lease has no recovery pathTreat lease_until in the past as claimable (extend the condition), or add a recovery sweeper
Legitimate second payment is rejected as a duplicateKey reused by the client for two different intents, but the body matchedEnsure the client creates a new key per user action. Include the intended execution date in the fingerprint
ConditionalCheckFailedException with no ItemReturnValuesOnConditionCheckFailure not set, or an old SDKSet ALL_OLD, or fall back to a GetItem after the failure
Replay returns a different payment IDThe stored response was not saved, or a retry took a different code pathAlways store the response before returning. Test with a retry in your suite
Idempotency table throttled during a peakOn-demand table warming, or one hot partition keyKeys are per customer so spread is usually fine; check ThrottledRequests and the key design

Cost and cleanup

Cost label: free. The local reproduction uses Docker and DynamoDB Local. If you deploy the table to the sandbox, on-demand DynamoDB for a few hundred test writes costs a few cents at most (⚠ VERIFY current pricing), and the KMS key costs about one US dollar per month if left for the whole month (⚠ VERIFY). Destroy it the same day.

# Local reproduction cleanup
docker rm -f ddb-local

# If you deployed the table in the sandbox
aws dynamodb delete-table --table-name hl-payments-idempotency --profile learn-sandbox
# Schedule the KMS key for deletion (minimum waiting period is 7 days)
aws kms schedule-key-deletion --key-id <key-id> --pending-window-in-days 7 --profile learn-sandbox
# Confirm nothing tagged for this lesson is left
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-duplicate --profile learn-sandbox

Exercises

  • 🟢 Guided: Start DynamoDB Local (docker run -d --name ddb-local -p 8000:8000 amazon/dynamodb-local). Create the idempotency table, call TryClaimAsync twice with the same key and body, and confirm the second result is InProgress. Call CompleteAsync, call again, and confirm ReplayCompleted.
  • 🟡 Challenge: Write an xUnit test that fires 50 concurrent TryClaimAsync calls with one key and asserts that exactly one returns Claimed. Then change the body for one call and assert KeyReusedWithDifferentRequest.
  • 🔴 Live problem: Tuesday 09:10: "Eleven customers report duplicate card top-ups since the weekend release. The app team says the app always sends a key. The payments team says the keys are all different." You have read access to the logs and the table. Write the three Logs Insights queries you run first, say what each would prove, and draft the first three lines of the incident timeline.

Assignment

Write a blameless incident report (one page) for a simulated duplicate-payment incident, and commit it to your portfolio repository as incident-reports/duplicate-payment.md. Include: summary, customer impact, timeline, root cause, the fix with a link to your test, what went well, what you would change, and the audit evidence list. Use fictional IDs only. This feeds the Module 17 project.

Solutions

Knowledge check

Interview questions

  • A customer says they were charged twice. How do you start? Scope and contain first (how many, still happening, who must be told), then collect evidence by idempotency key, correlation ID and transaction rows before changing anything, then rank hypotheses by the key pattern.
  • How do you make a payment API safe to retry? A client-generated key per user intent, a server-side atomic claim with a request fingerprint, replay of the stored response, and a distinct key at each downstream hop.
  • Why is SQS FIFO deduplication not enough? It only covers identical MessageDeduplicationId values inside a 5-minute window. It cannot tell that two different messages mean one business instruction.
  • How do you reverse a duplicate? A correcting ledger entry through four-eyes approval, an idempotent reversal key, checking scheme status with payments operations, customer notification and a full audit record.
  • What is the difference between preventive and detective controls here? The idempotency store prevents; the nightly reconciliation for near-identical payments detects what slipped through. Banks run both.

Certification practice

Completion checklist

  • I can list the five ways one instruction becomes two payments
  • I can run the Logs Insights query that groups accepted payments by idempotency key
  • I know the three brakes I can pull in the first five minutes, and that purging a queue is not one of them
  • I can explain why the reversal is a new linked entry, not a delete
  • I completed the guided exercise and the concurrency test
  • I ran the cleanup steps and confirmed nothing tagged m17-playbook-duplicate remains

Key takeaways

  • A retry is only safe if the server can recognise it: client key per intent, atomic claim, fingerprint, replay.
  • Contain first, then collect evidence, then fix. Never purge or delete during an incident.
  • Reverse with an approved, idempotent, linked correcting entry, never an edit.
  • Guard every hop: API key, queue consumer (payment ID), scheme submission (end-to-end ID).
  • Pair the preventive guard with a detective reconciliation check.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page