Payment processing: initiation, screening, settlement and idempotency
Design an end-to-end payment architecture on AWS: idempotent initiation, validation, a fail-closed sanctions-screening hook, a hold-submit-settle saga with a pivot point, a double-entry ledger, ISO 20022 messages, and the failure scenarios a design authority will probe.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.DynamoDBv2 4.x) · AWS Step Functions · CDK v2 (Amazon.CDK.Lib 2.277.0) · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Trace a payment from customer initiation to settlement and name each status it passes through.
- Design idempotent initiation: the same request retried ten times moves money once, and a reused key with a different body is rejected.
- Place a sanctions-screening hook in the flow and explain why the design must fail closed.
- Use a hold, submit, settle sequence with compensation before the scheme accepts and forward-only recovery after it (the pivot).
- Explain what a timeout from the payment scheme really means, and why you query before you compensate.
- Choose between Step Functions orchestration and in-service orchestration, and justify it with cost and failure arithmetic.
- Map the flow to ISO 20022 messages and write the 15-section architecture for payments.
Prerequisites
- How to read and write a banking architecture and account management
- Idempotency keys and ISO 20022 messages
- Saga pattern and idempotency in .NET
- Transaction integrity and non-repudiation
- The lab for this lesson: Lab: payment saga failure scenarios
Why this matters in banking
Payments are the one place where a bug moves real money and cannot be quietly rolled back. Four questions decide whether a payments design survives review:
- If the customer's phone retries, do we pay twice?
- If the sanctions provider is down, do we pay anyway?
- If the scheme does not answer, did the money leave?
- If we crash half-way, can we prove where every pound is?
Each question has a wrong, tempting answer. "Retry on failure" without idempotency pays twice. "Skip screening if the provider is slow" is a regulatory incident. "Refund the customer on timeout" can pay the beneficiary and refund the sender, so the bank loses the money. "Fix it in reconciliation tomorrow" means customers see wrong balances today.
Educational content, not legal or scheme advice
Payment-scheme rules (cut-off times, return windows, liability), sanctions obligations and operational-resilience requirements are set by the schemes, regulators and your bank's policies. This lesson shows where they plug into the architecture. Specifics are marked ⚠ VERIFY, and every design here needs review by payments operations, financial-crime compliance, risk and security before use.
The idea in plain language
A payment is a promise to move value from the customer's account to someone else's, through a scheme the bank does not control. Three principles organise the whole design.
- Say it once, safely. Every request carries a client-chosen key. The first time we see it we do the work; later times we return the same answer.
- Hold before you send. Reserve the money in the customer's account first (debit the customer, credit a suspense account), then send the instruction. If the scheme refuses, release the hold. If the scheme accepts, settle the hold.
- Know your pivot. Before the scheme accepts, you can still undo. After it accepts, you only move forward (settle, or later refund through a return). Between "sent" and "answered" lies a gray zone: unknown. The design must have a named state for it.
Analogy: posting a cheque with a courier. You put the money aside first (hold). The courier takes the cheque (submit). If the courier says "I could not deliver it", you release the money back. If the courier loses contact after taking the cheque, you cannot assume it was lost: you ring the recipient's bank to ask. Only when you know does the hold become real payment or a refund.
The lifecycle
The first seven statuses (received, validated, screening, submitted, settled, rejected, returned) are the canonical payments.status values in Harbourline's data model. This lesson adds PendingReview, Held (an internal sub-state) and OutcomeUnknown, because the failure paths need names.
ISO 20022 in one table
| Message | Direction | Meaning | Where it appears |
|---|---|---|---|
pain.001 | Customer (corporate) to bank | Customer credit transfer initiation | Corporate file or API channel |
pacs.008 | Bank to bank | Financial-institution credit transfer | Submitting to the scheme or correspondent |
pacs.002 | Bank to bank | Payment status report (accepted, rejected, reason) | The answer to a submit |
pacs.004 | Bank to bank | Payment return | A settled payment sent back |
camt.053 | Bank to customer | Account statement | Reconciliation (see lesson on reconciliation) |
Retail mobile payments usually arrive as JSON from your API, and your service builds the outbound message. Which scheme uses which message version and which usage rules apply varies by scheme and over time ⚠ VERIFY. The field to remember is EndToEndId: it travels unchanged from the originator to the beneficiary and is the natural duplicate-detection key. See ISO 20022 messages.
What problem the AWS services solve
| Service | Role in payments | Limits and sharp edges |
|---|---|---|
| Amazon API Gateway | Authenticated entry, throttling, request validation | Synchronous timeout (about 29 s default for REST APIs ⚠ VERIFY); do not hold a request open for scheme settlement |
| AWS Step Functions Standard | Durable orchestration with history, retries, catches, waits for callbacks | Billed per state transition ⚠ VERIFY price; execution history retained for a limited period, so export what audit needs |
| AWS Step Functions Express | High-volume, short workflows (up to 5 minutes) | At-least-once for asynchronous Express, at-most-once for synchronous; no long waits; history lives in CloudWatch Logs |
| Amazon DynamoDB | Idempotency store, payment state, conditional writes | Idempotency item is not in the same transaction as the ledger unless you design for it |
| Amazon Aurora PostgreSQL | Double-entry ledger with constraints and transactions | You operate schemas, migrations and capacity; strong integrity features |
| Amazon EventBridge / SQS | Events (PaymentSettled) and decoupled work queues | At-least-once; use FIFO queues only where ordering per account matters ⚠ VERIFY throughput |
| AWS KMS | Signing payment instructions, encrypting data | Asymmetric signing has per-request cost and size limits (integrity lesson) |
| AWS PrivateLink / VPN / Direct Connect | Private path to partners, scheme gateways and core banking | Lead times for Direct Connect are weeks; plan ahead |
| Amazon QLDB | Closed: support ended 2025-07-31 | Do not design new ledgers on it; use Aurora PostgreSQL with append-only design |
Honest limitation: there is no AWS "payment service". You assemble a flow from general-purpose services, and the scheme connection is usually a partner gateway or a bank-operated component outside AWS.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Durable Functions orchestrator with compensation | Step Functions Standard with Catch states | Declarative JSON; compensation states are explicit nodes you can see and audit |
| Azure Service Bus sessions and duplicate detection | SQS FIFO message group and deduplication (5-minute window) | The broker window is short; your idempotency key store covers days |
| Azure SQL ledger tables | Aurora PostgreSQL with append-only table, constraint trigger and signed batch hashes | No built-in verifiable ledger table; you design integrity |
| Azure API Management idempotency policy (custom) | Idempotency in your service plus DynamoDB conditional put | No managed gateway feature; the guarantee lives in your code |
| Azure Logic Apps for B2B (AS2, X12) | Transfer Family, Step Functions, partner gateways | SFTP and file flows are separate services (SWIFT and file integration) |
Architecture
Component view
Sequence of the happy path and the timeout path
Security-boundary view
| Zone | Contains | Controls |
|---|---|---|
| Customer zone | App, BFF | OIDC, step-up above a threshold, rate limits |
| Payments workload account | API, saga, idempotency store | Private subnets, task roles per component, KMS keys per store, no internet route |
| Ledger account | Aurora cluster | Only the saga's role can post; no human write access; read replicas for reporting |
| Partner/egress zone | Gateway to screening provider and scheme | Allow-listed egress, mTLS, certificates in ACM or Secrets Manager, request signing |
| Audit zone | Log archive account | Write-only from workloads; Object Lock for payment audit evidence |
Step-by-step implementation
Fix the requirements (sections 1 to 3)
Fictional targets:
| Quality | Target |
|---|---|
| Initiation availability | 99.95% monthly |
| API latency | p99 under 800 ms to acknowledge (202), excluding scheme time |
| Duplicates | Zero duplicate settlements under client retry or message redelivery |
| Screening | 100% of payments screened before submission; unscreened release is not allowed |
| Ledger | Every posting balanced per currency; no negative customer balance unless an overdraft is agreed |
| Throughput | 200 payments per second sustained; 800 at peak |
| Recovery | RPO 0 for the ledger, RTO 15 minutes for initiation |
| Retention | Payment records and audit trail retention per policy ⚠ VERIFY |
Make initiation idempotent (section 8)
The API requires Idempotency-Key. It computes a hash of the request body and tries to claim the key with a conditional put. Outcomes:
| Outcome | Response |
|---|---|
| Key is new | Process, return 202 with the payment id |
| Key exists, same body, finished | Return the stored result (same payment id and current status) |
| Key exists, same body, still running | Return 202 with status Received and the same id if known, or 409 with Retry-After |
| Key exists, different body | 422 ProblemDetails: key reuse with a different request |
A second layer protects against clients that invent new keys for the same payment: a unique constraint on EndToEndId per debtor account in the payment table.
Validate cheaply first, then screen (section 10)
Validation (positive amount, two decimals, supported currency, IBAN checksum, EndToEndId at most 35 characters, execution date) is deterministic and costs nothing. Run it before claiming screening capacity. Screening sends the minimum personal data the provider needs.
Put the sanctions hook in a place that cannot be skipped (section 7 and 11)
Make "screened clear" a precondition that the submit step checks itself, not just a step earlier in the workflow. A bug that routes around the screening state must still be unable to submit. In the lab this is guaranteed by control flow; in production, add a check at the scheme gateway that a screening result id is present and recent.
Failure behaviour: bounded retries with backoff, then park in PendingReview and tell the customer "pending". Never treat a timeout as "no match".
Hold, submit, then settle (section 10)
- Hold: one atomic ledger posting: debit customer, credit suspense. Rejects if funds are insufficient.
- Submit: build the outbound message; call the gateway. The gateway call is idempotent on the
EndToEndId/MsgIdso a retry cannot create a second scheme instruction. - Settle: after acceptance, post suspense to the scheme settlement account. The posting id is derived from the payment id (
settle:<paymentId>), so a retry after a crash posts once.
Handle the timeout correctly (section 11)
On a timeout, query the scheme's status for the EndToEndId before deciding. If it says accepted, move forward. If it says rejected, release the hold. If it still does not know, keep the hold, set OutcomeUnknown and let the reconciliation job resolve it from the scheme's status report or statement. Compensating blindly risks paying twice. The lab's tests demonstrate each branch.
Emit events and keep the audit trail (section 10 and 15)
Publish PaymentSettled, PaymentRejected, PaymentPendingReview through a transactional outbox (outbox pattern) so events are not lost between database commit and publish. Write an application audit record at each state change.
Estimate cost (section 13)
Step Functions Standard charges per state transition. If a payment takes 12 transitions and the price is USD 0.025 per 1,000 transitions ⚠ VERIFY, one payment costs about USD 0.0003 and 10 million payments a month cost about USD 3,000 in orchestration. Express workflows are cheaper per execution but cap duration and change delivery semantics. Decide with numbers, and record the choice as an ADR.
Orchestration choice
| Option | Strength | Weakness | Choose when |
|---|---|---|---|
| Step Functions Standard | Visible state, history, retries, long waits and callbacks | Per-transition cost; JSON definitions; history retention is limited | Long or human-involved flows, audit-heavy flows |
| Step Functions Express (sync) | Cheap at volume, low latency | 5-minute limit, at-most-once semantics, harder audit | Short, high-volume steps with idempotent actions |
| In-service orchestration (code + DynamoDB state) | Full control, testable in unit tests, no per-step cost | You build retries, timers and visibility | Small team, strong testing culture, simple flow |
| Choreography (events only) | Loose coupling | Hard to see "where is payment X?" | Side effects like notifications, not the money flow |
The lab implements the in-service form so every failure path is unit tested. The deployed proof of concept uses an Express state machine for cheapness, and a production version would typically use Standard for the money path.
AWS Console instructions
- Console → Step Functions → State machines → Create state machine → Blank. Choose Standard for the money path. Open Workflow Studio and add Choice, Task and Catch states to see the saga visually.
- Console → DynamoDB → Create table
hl-m14-idempotency, partition keypk, on-demand, TTL attributeexpiresAt. - Console → CloudWatch → Log groups: open the state machine's log group to see execution events (keep execution data out of logs for real payments).
- After running executions, open Step Functions → Executions → one execution → Graph view and read how the failed branch routed.
AWS CLI
# Start a synchronous Express execution of the lab's proof-of-concept state machine.
# The "simulate" block replaces real partners so no provider is needed.
aws stepfunctions start-sync-execution \
--state-machine-arn arn:aws:states:us-east-1:111122223333:stateMachine:hl-m14-payment-saga \
--input '{"idempotencyKey":"k-0001","debtorAccountId":"ACC-0000000001","amount":"250.00","currency":"GBP","simulate":{"sanctions":"clear","scheme":"accept"}}' \
--profile learn-sandbox --region us-east-1
# Send the SAME key again: the conditional put fails and the state machine returns DUPLICATE.
aws stepfunctions start-sync-execution \
--state-machine-arn arn:aws:states:us-east-1:111122223333:stateMachine:hl-m14-payment-saga \
--input '{"idempotencyKey":"k-0001","debtorAccountId":"ACC-0000000001","amount":"250.00","currency":"GBP","simulate":{"sanctions":"clear","scheme":"accept"}}' \
--profile learn-sandbox --region us-east-1
# Read what the saga recorded for the key.
aws dynamodb get-item --table-name hl-m14-idempotency \
--key '{"pk":{"S":"k-0001"}}' --profile learn-sandbox --region us-east-1111122223333 is the AWS documentation placeholder account id: use the ARN printed by cdk deploy. The CLI sends fictional data only.
Infrastructure as Code
The lab's CDK app (labs/m14-payment-saga-failures/infra/cdk) deploys an IAM-authorised API Gateway in front of an Express state machine with two DynamoDB tables. Key parts:
var stateMachine = new StateMachine(this, "PaymentSaga", new StateMachineProps
{
StateMachineName = "hl-m14-payment-saga",
StateMachineType = StateMachineType.EXPRESS,
DefinitionBody = DefinitionBody.FromString(SagaDefinition.Asl),
DefinitionSubstitutions = new Dictionary<string, string>
{
["IdempotencyTable"] = idempotency.TableName,
["LedgerTable"] = ledger.TableName,
},
Logs = new LogOptions
{
Destination = logs,
Level = LogLevel.ERROR,
IncludeExecutionData = false, // payment bodies hold personal data
},
TracingEnabled = true,
Timeout = Duration.Minutes(1),
});
idempotency.GrantReadWriteData(stateMachine); // least privilege: only these two tables
ledger.GrantWriteData(stateMachine);Why this SQL: the primary key on posting_id makes a replayed posting fail instead of double-posting; the deferred trigger enforces the double-entry invariant at commit; revoking update and delete makes history append-only. A database role used by the saga should hold only INSERT and SELECT. Test the trigger against your PostgreSQL version before relying on it ⚠ VERIFY.
Complete code
The full saga with every failure path lives in labs/m14-payment-saga-failures/src/Harbourline.Payments.Saga and is covered by 22 tests. This lesson adds the one piece the lab deliberately fakes: an idempotency store on DynamoDB that implements the same IIdempotencyStore interface.
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
using Harbourline.Payments.Saga;
namespace Harbourline.Payments.Aws;
public sealed class DynamoDbIdempotencyStore(IAmazonDynamoDB dynamo, string table, TimeProvider clock) : IIdempotencyStore
{
private static readonly TimeSpan Retention = TimeSpan.FromHours(24);
public async Task<BeginResult> TryBeginAsync(string key, string requestHash, CancellationToken ct)
{
try
{
await dynamo.PutItemAsync(new PutItemRequest
{
TableName = table,
Item = new Dictionary<string, AttributeValue>
{
["pk"] = new() { S = key },
["requestHash"] = new() { S = requestHash },
["status"] = new() { S = "IN_PROGRESS" },
["expiresAt"] = new() { N = clock.GetUtcNow().Add(Retention).ToUnixTimeSeconds().ToString() },
},
ConditionExpression = "attribute_not_exists(pk)",
}, ct);
return new BeginResult(BeginOutcome.Started);
}
catch (ConditionalCheckFailedException)
{
var existing = await dynamo.GetItemAsync(new GetItemRequest
{
TableName = table,
Key = new Dictionary<string, AttributeValue> { ["pk"] = new() { S = key } },
ConsistentRead = true,
}, ct);
if (existing.Item is null || existing.Item.Count == 0)
{
return new BeginResult(BeginOutcome.InProgress); // expired between the two calls: let the caller retry
}
if (existing.Item["requestHash"].S != requestHash)
{
return new BeginResult(BeginOutcome.Conflict);
}
if (existing.Item["status"].S != "COMPLETED")
{
return new BeginResult(BeginOutcome.InProgress);
}
var stored = new PaymentResult(
existing.Item["paymentId"].S,
Enum.Parse<PaymentStatus>(existing.Item["resultStatus"].S),
existing.Item.TryGetValue("reason", out var reason) ? reason.S : null);
return new BeginResult(BeginOutcome.Replay, stored);
}
}
public async Task CompleteAsync(string key, PaymentResult result, CancellationToken ct)
{
var values = new Dictionary<string, AttributeValue>
{
[":done"] = new() { S = "COMPLETED" },
[":pid"] = new() { S = result.PaymentId },
[":rs"] = new() { S = result.Status.ToString() },
};
var expression = "SET #s = :done, paymentId = :pid, resultStatus = :rs";
if (result.Reason is not null)
{
expression += ", reason = :reason";
values[":reason"] = new() { S = result.Reason };
}
await dynamo.UpdateItemAsync(new UpdateItemRequest
{
TableName = table,
Key = new Dictionary<string, AttributeValue> { ["pk"] = new() { S = key } },
UpdateExpression = expression,
ExpressionAttributeNames = new Dictionary<string, string> { ["#s"] = "status" },
ExpressionAttributeValues = values,
}, ct);
}
}Code explained line by line
- The class implements the same
IIdempotencyStorethe lab's saga uses, so the saga code does not change when you swap the in-memory store for DynamoDB. That is the point of the interface. PutItemAsyncwithConditionExpression = "attribute_not_exists(pk)"is the atomic claim. Two concurrent requests with the same key: exactly one put succeeds.expiresAtis a Unix time in seconds. If the table has TTL enabled on that attribute, DynamoDB deletes expired items later; deletion can lag by hours, so the code must not assume an item vanishes exactly at expiry ⚠ VERIFY current TTL behaviour.- On
ConditionalCheckFailedException, the code reads the existing item withConsistentRead = true, because an eventually consistent read could miss the item just written by the other request. - The body-hash comparison separates a retry (same hash) from a key reuse (different hash,
Conflict). - A status other than
COMPLETEDmeans another request is working on it:InProgress. CompleteAsyncstores the result fields needed to answer a replay.statusis a DynamoDB reserved word, so the update expression aliases it as#s.- The replay returns the stored payment id. The saga's own replay logic then reads the current status from the payment record, so an
OutcomeUnknownpayment that has since been resolved shows its new state.
Security considerations
⚠ Learning shortcut — not for production
The lab fakes sanctions screening, the scheme gateway and the ledger with in-memory classes, and the proof-of-concept state machine stubs screening with a simulate field in the request. A client that can set simulate.sanctions = "clear" could bypass screening. In production there is no such field: screening results come only from the provider call made by the saga, the submit step verifies a recent screening result id, and the API rejects unknown properties. Also add: message signing with KMS, mTLS to the scheme, ledger in Aurora with the schema above, and WAF and throttling in front.
- Fail closed: no screening answer means no payment.
- Tipping-off: the customer-facing reason for a screening hold or rejection is generic ("declined"); the detailed reason stays internal. What may be said is a legal question ⚠ VERIFY with compliance.
- Segregation of duties: no engineer can post to the ledger by hand. Corrections are new postings made through an approved tool with four-eyes.
- Non-repudiation: sign instructions where the scheme requires it and log the key use (integrity lesson).
- Data minimisation: the screening provider gets only the fields it needs; payment references can carry personal data, so treat them as sensitive in logs.
- Rate limits and velocity controls: limits per customer and per time window belong in the payments service as well as at the edge.
- Replay and tampering: a message signature that covers amount, creditor and
EndToEndIdstops an attacker changing the creditor in transit.
Banking use case
Harbourline sends faster payments for retail customers (fictional numbers). At 09:00 on payday:
- 4,000 requests arrive in a minute. About 3% are client retries of the same payment; idempotency returns the original result without a second saga.
- At 09:12 the sanctions provider starts returning errors. The saga retries three times, then parks payments in
PendingReview; customers see "pending". Operations is paged because the review queue exceeds its threshold. At 09:47 the provider recovers and compliance clears the queue. - At 10:05 the scheme gateway times out on 14 submissions. Twelve are resolved by status query within a minute (accepted); two remain
OutcomeUnknownand are resolved by the reconciliation job when the scheme's status report arrives at 10:30. Customers with those two see their balance reduced (hold) and the status "processing". - At 17:00 end-of-day reconciliation finds every hold either settled or explained.
Common mistakes
- Retrying without idempotency → double payments → key claim plus
EndToEndIdconstraint. - Treating a timeout as failure and refunding → paid twice → query first,
OutcomeUnknownotherwise. - Failing open on screening → regulatory breach → fail closed and park.
- Posting debit and credit in separate transactions → money vanishes on a crash → one atomic posting.
- Using
doublefor money → rounding drift →decimal,numeric(18,2), strings in JSON. - Idempotency record and payment write not coordinated → "key claimed, payment never created" → let a retry resume from stored state, and expire stale
IN_PROGRESSentries with an alarm. - Orchestrating a 20-minute settlement inside one HTTP request → timeouts → return
202and notify. - Putting everything in one Step Functions payload → size limit and personal data in history → pass references.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Second request with same key returns 422 | Body differs (for example amount formatting 10 vs 10.00) | Canonicalise the body before hashing; fix the client |
Payments stuck IN_PROGRESS | Process crashed after claiming the key | Resume from stored saga state; alarm on items older than N minutes |
ConditionalCheckFailedException shown as a 500 | Not handled as business outcome | Catch and map to replay/conflict |
Step Functions States.DataLimitExceeded | Payload over the size limit | Store large data in S3 or DynamoDB and pass keys |
| Duplicate events downstream | At-least-once delivery from EventBridge or SQS | Idempotent consumers keyed by paymentId plus event type |
| Hold never released | Compensation state failed | Dead-letter the failure, alarm, and let reconciliation post the release (idempotent posting id) |
OutcomeUnknown count rising | Scheme gateway degraded | Check gateway health, widen query intervals, escalate to payments operations |
start-sync-execution fails with StateMachineTypeNotSupported | State machine is Standard | Sync execution requires Express; use start-execution for Standard |
Cost and cleanup
Label: 🟡 credits. The C# lab is free offline. The deployed proof of concept uses on-demand DynamoDB, an Express state machine and API Gateway; at lab volumes the cost is cents ⚠ VERIFY, but destroy it the same day. There is no Aurora in the lab, so there is no hourly database cost. Aurora would be the main cost in a real deployment.
# From labs/m14-payment-saga-failures/infra/cdk
cdk destroy --all --profile learn-sandbox
# Confirm no state machines, tables or APIs remain
aws stepfunctions list-state-machines --profile learn-sandbox --region us-east-1
aws dynamodb list-tables --profile learn-sandbox --region us-east-1
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1Exercises
- 🟢 Guided: run the lab's tests and read
PaymentSaga.RunAsync. Draw the state diagram it implements and mark the pivot. - 🟡 Challenge: add a fifth failure scenario to the lab: the scheme accepts the payment twice (duplicate acknowledgement). Make the saga ignore the second acknowledgement and prove with a test that settlement posts once.
- 🔴 Live problem: operations reports 40 payments in
OutcomeUnknownfor over two hours; customers are calling. Write the triage steps, the query that finds them, what you tell customers, how reconciliation resolves them safely (never by guessing), and what you change in the design so a long unknown period alerts earlier.
Assignment
Produce the 15-section payments architecture (portfolio project part 1): context and container diagrams, the lifecycle state diagram, the sequence diagram with the timeout path, ADRs for orchestration choice and for fail-closed screening, the ledger schema, a failure table with at least ten scenarios, a cost estimate with assumptions and a compliance-touchpoints table (payments operations, financial-crime compliance, risk, security, data protection, third-party risk).
Solutions
RunAsync validates, screens (with retries), holds funds, submits, then FinishAsync chooses the direction. The pivot is the scheme's acceptance: before it, a rejection releases the hold (compensation); after it, SettleAsync only moves forward and, even if posting keeps failing, leaves the payment Submitted with settlement-posting-pending rather than refunding. The null outcome (no answer after queries) is the gray zone and keeps the hold. The test Persistent_settlement_failure_after_scheme_acceptance_moves_forward_not_back pins this behaviour.
Settlement is already keyed by "settle:" + paymentId in the ledger, so a second FinishAsync(Accepted) posts nothing new. Add a test that calls FinishAsync path twice, for instance by calling ResolveAsync on an already Settled payment: it returns the existing status because ResolveAsync only acts on OutcomeUnknown. Assert the suspense account is zero and the scheme account holds the amount once. Why: idempotency belongs at every step that moves money, not only at the edge.
Triage: (1) confirm scope with a query for status OutcomeUnknown older than 15 minutes; (2) check gateway and scheme health; (3) run the status query for each EndToEndId; (4) resolve with the answers (accepted: settle; rejected: release hold); (5) for those still unknown, escalate to the scheme operator with the message ids; (6) message customers: "your payment is being processed; funds are reserved; we will confirm shortly", without speculation. Design change: an alarm on the age of the oldest OutcomeUnknown, an escalation timer in the saga, and a runbook linked from the alarm. Never resolve by assuming the payment failed.
Knowledge check
Answer: The stored result: the same payment id and the current status. It must not start a second saga.
Answer: PendingReview (parked), with no funds moved and no submission. Timeouts are not clears. Compliance or an automatic retry later resolves it.
Answer: The scheme may have received and accepted the payment. Releasing the hold would return the money to the customer while the beneficiary is also paid. Query status first; if still unknown keep the hold and reconcile.
Answer: So a retry after a crash or an ambiguous failure posts the same posting id, which the ledger treats as already done. The retry cannot post twice.
Answer: For short, high-volume, idempotent steps where the 5-minute limit and Express delivery semantics are acceptable and CloudWatch Logs provide enough audit. For the money path with waits, callbacks and long audit history, Standard is usually the better fit.
Interview questions
- How do you prevent duplicate payments? Idempotency key claimed with a conditional write plus a body hash, a unique
EndToEndIdper debtor account, idempotent ledger postings and idempotent consumers. No single layer is enough. - What happens when the scheme times out? Query the status for the same identifier; if accepted continue, if rejected compensate, if unknown keep the hold and reconcile. Never guess.
- Why a hold account? It makes the intermediate state visible and balanced: money is neither in the customer's account nor lost, and every pound can be explained.
- How do you design for sanctions-provider outages? Bounded retries, fail closed, park in a review state, alert on queue age, consider a second provider with its own third-party risk review.
- Where does eventual consistency hurt in payments? Reading a balance from a projection to approve a payment. The ledger or core is the authority for funds checks.
Certification practice
A payments API must not process the same request twice when mobile clients retry after network timeouts. Which approach is best?
A. Increase the API Gateway timeout. B. Require a client-generated idempotency key and claim it with a DynamoDB conditional write before processing. C. Use a larger Fargate task. D. Deduplicate only in the mobile app.
Answer: B. The server must enforce uniqueness atomically. A and C do not change duplicate behaviour. D cannot protect against a second device or a modified client.
A Step Functions workflow calls a partner API that sometimes returns transient errors. Which feature should the developer use to retry with increasing delays?
A. A Retry block with IntervalSeconds, BackoffRate and MaxAttempts
B. A Parallel state
C. A Pass state
D. A Lambda environment variable
Answer: A. Retry blocks implement declarative exponential backoff. B runs branches concurrently, C passes data through, and D does not control retry behaviour.
A payment workflow calls a screening provider that is occasionally unavailable. Policy states that no payment may leave the bank without a screening result. What should the design do when retries are exhausted?
A. Submit the payment and screen afterwards. B. Mark the payment as pending review without moving funds and alert operations. C. Reject the payment silently. D. Retry forever in the API request.
Answer: B. Failing closed honours the policy and preserves the customer's request for later resolution. A breaks the policy. C loses a legitimate request without a record. D exhausts resources and hides the problem.
Which measure best protects the integrity of a payment instruction between the bank and a scheme gateway?
A. A plain checksum in a header B. A digital signature created with an AWS KMS asymmetric key, plus mTLS on the connection C. Compressing the message D. Obfuscating field names
Answer: B. A signature detects tampering and supports non-repudiation; mTLS authenticates the endpoints. A checksum can be recomputed by an attacker, and C and D add no security.
Completion checklist
- I can state the three principles: say it once, hold before send, know the pivot
- I can explain why a timeout is not a failure
- I ran the lab tests and traced each scenario
- I can justify Standard versus Express with numbers
- I wrote the 15-section payments architecture with no compliance claims
- I destroyed any deployed resources and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- Idempotency has layers: edge key,
EndToEndIdconstraint, ledger posting id, consumer deduplication. - Hold, submit, settle: compensate before the scheme accepts, move forward after.
- A timeout is an unknown, so query before you decide and give the unknown a named state.
- Screening is a precondition that fails closed, with a generic message to the customer.
- Money data uses
decimal, balanced postings and append-only history.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Account management: lifecycle APIs, data ownership and customer-only authorisation
Design the account management service: a lifecycle state machine, account APIs that return 404 for other people's accounts, an access-grant model for joint and delegated access, DynamoDB tables with optimistic concurrency, and the 15-section write-up.
Lab: payment saga failure scenarios (design studies 14.3 and 14.4)
Run and extend a unit-tested payment saga in C# that survives a sanctions provider outage, duplicate client retries, a scheme timeout and a partial settlement, then optionally deploy a minimal Step Functions version and destroy it the same day.