Lab: payment saga failure scenarios (design studies 14.3 and 14.4)
Run and extend a unit-tested payment saga in C# that survives a sanctions provider outage, duplicate client retries, a scheme timeout and a partial settlement, then optionally deploy a minimal Step Functions version and destroy it the same day.
Assumes: .NET 10 LTS · xUnit v3 4.0.3 (Microsoft.Testing.Platform) · CDK v2 (Amazon.CDK.Lib 2.277.0) · AWS Step Functions Express · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lab you can:
- Run 22 unit tests that specify how a payment behaves under four kinds of failure, with no AWS account.
- Trace each failure through
PaymentSagaand point at the line that decides the outcome. - Show that a duplicate client retry, ten concurrent duplicates and a replayed ledger posting each move money once.
- Explain why a scheme timeout keeps the hold (
OutcomeUnknown) rather than refunding the customer. - Extend the saga with a fifth scenario and prove it with a test.
- Optionally deploy the Step Functions proof of concept (cents), exercise the same scenarios through it, and tear it down.
Prerequisites
- Payment processing: the design this lab implements
- Saga pattern and idempotency in .NET
- The lab code:
labs/m14-payment-saga-failures/in the course repository (itsREADME.mdrepeats the commands) - Tools: Docker (or the .NET 10 SDK). AWS CLI v2 and the
learn-sandboxprofile only for the optional deployment
Why this matters in banking
A design review does not ask "does it work?". It asks "show me what happens when the sanctions provider is down, when the client retries, when the scheme goes quiet and when settlement fails halfway". This lab turns those four questions into tests that a reviewer can read as specifications, which is more convincing than a paragraph in a document.
The idea in plain language
The saga is a short sequence of steps with a pivot in the middle:
Before the scheme accepts, a failure is compensated by releasing the hold. After it accepts, the saga moves forward only. In between, "no answer" is a named state, not a failure.
Analogy: sending a parcel by courier. You set the money aside, hand over the parcel and if the courier loses contact you ring the recipient before refunding yourself.
Setup and run
Run the tests
From the repository root:
MSYS_NO_PATHCONV=1 docker run --rm --memory 3g \
-v "$PWD/labs/m14-payment-saga-failures:/src" \
-v cea-nuget-m14:/root/.nuget/packages \
-w /src mcr.microsoft.com/dotnet/sdk:10.0 \
sh -c "dotnet test --solution PaymentSagaFailures.slnx; find . -type d \( -name bin -o -name obj \) -prune -exec rm -rf {} +"The first run downloads the CDK packages (a few minutes). Expected output ends with Test run summary: Passed! and total: 22. dotnet test does not compile the CDK project; dotnet build PaymentSagaFailures.slnx does.
Read the saga
Open src/Harbourline.Payments.Saga/PaymentSaga.cs. Find ExecuteAsync (validation, idempotency, then RunAsync), RunAsync (screen, hold, submit), FinishAsync (the decision after the scheme's answer), SettleAsync (idempotent posting with retries) and ResolveAsync (the reconciliation entry point).
Map each test to a scenario
Open tests/Harbourline.Payments.Saga.Tests/FailureScenarioTests.cs. Each test builds a Harness (ledger with GBP 1,000 for ACC-0000000001) and scripts a fake: ScriptedScreener ("down", "hit", "clear"), ScriptedGateway ("accept", "reject", "timeout-received", "timeout-lost") and FlakySettlementLedger.
Architecture
Every dependency is an interface, so the same saga code runs against DynamoDB, Aurora, a real screening provider and a real scheme gateway in production. The doubles are the only difference in the tests.
Code walkthrough
The idempotent entry point
var begin = await idempotency.TryBeginAsync(request.IdempotencyKey, request.ComputeBodyHash(), ct);
switch (begin.Outcome)
{
case BeginOutcome.Conflict:
throw new IdempotencyConflictException(request.IdempotencyKey);
case BeginOutcome.InProgress:
return new PaymentResult(string.Empty, PaymentStatus.Received, "in-progress");
case BeginOutcome.Replay:
var stored = begin.Stored!;
var current = store.Get(stored.PaymentId);
return current is null ? stored : new PaymentResult(current.PaymentId, current.Status, current.Detail);
}ComputeBodyHashhashes everything except the key, so a retry (same key, same body) is recognised and a key reuse (same key, different body) is aConflict.InProgresscovers the concurrent duplicate: only the caller that gotStarteddoes the work.Replayreturns the current state of the stored payment, not the first response. AnOutcomeUnknownpayment that reconciliation has since settled therefore showsSettled.
Fail closed on screening
var screening = await ScreenWithRetryAsync(paymentId, request, ct);
if (screening is null)
{
log.Record(paymentId, "screening", "unavailable after retries; parked for review");
return Save(paymentId, request, PaymentStatus.PendingReview, "screening-unavailable");
}null means every attempt threw ScreeningUnavailableException. The payment is parked, no money moves and no submission happens. A hit returns the generic reason declined, and the detailed reason goes only to the internal log.
The pivot
case null:
log.Record(paymentId, "submit", "outcome unknown; hold kept; reconciliation will resolve");
return Save(paymentId, request, PaymentStatus.OutcomeUnknown, "awaiting-confirmation");
case SubmissionOutcome.Rejected:
await ledger.PostAsync("release:" + paymentId, [...], ct);
return Save(paymentId, request, PaymentStatus.Rejected, "scheme-rejected");FinishAsync receives the scheme's answer (or null for "still unknown after the status queries"). Only a definite rejection releases the hold. The unknown case keeps it.
Idempotent postings
await ledger.PostAsync(
"settle:" + paymentId,
[ new LedgerEntry(_options.SuspenseAccountId, -request.Amount, request.Currency),
new LedgerEntry(_options.SchemeAccountId, request.Amount, request.Currency) ],
ct);The posting id is derived from the payment id. InMemoryLedger.PostAsync ignores a second posting with the same id, so retrying after a transient failure (or after a crash that left the outcome ambiguous) cannot post twice. It also rejects any posting that does not sum to zero per currency, and any customer account that would go negative.
Exercises
- 🟢 Guided: add a test named
Hold_is_visible_in_suspense_while_outcome_is_unknownasserting the customer balance is 750.00 and the suspense balance is 250.00 after atimeout-lostrun, then 1,000.00 and 0 afterResolveAsyncwith a scheme rejection. - 🟡 Challenge: add the fifth scenario from the lesson: the scheme acknowledges twice. Prove that calling
ResolveAsyncandExecuteAsyncagain on aSettledpayment changes nothing, and that the scheme account holds the amount exactly once. - 🔴 Live problem: payments operations says that for payments over GBP 10,000 a screening outage must not park the payment silently: the customer's relationship manager must be told within 15 minutes. Without changing the fail-closed behaviour, design where this notification hooks in (saga, event, notification platform), what the test looks like and what you must ask compliance before releasing it.
Optional deployment (credits, cents)
The CDK app in infra/cdk deploys: two DynamoDB tables (idempotency with a TTL attribute, ledger), an Express state machine with the saga as Amazon States Language, and an IAM-authorised API Gateway in front of it. Costs at lab volumes are cents ⚠ VERIFY; destroy it the same day.
cd labs/m14-payment-saga-failures/infra/cdk
cdk deploy -c owner=<your-name> --profile learn-sandbox
aws stepfunctions start-sync-execution \
--state-machine-arn <StateMachineArn> \
--input '{"idempotencyKey":"k-0001","debtorAccountId":"ACC-0000000001","amount":"250.00","currency":"GBP","simulate":{"sanctions":"clear","scheme":"accept"}}' \
--profile learn-sandbox --region us-east-1Then try: the same key again (DUPLICATE), "sanctions":"hit" (status REJECTED), "sanctions":"down" (PENDING_REVIEW) and "scheme":"reject" (a RELEASE# ledger item then REJECTED). Check aws dynamodb scan on both tables to see what was recorded.
Security considerations
⚠ Learning shortcut — not for production
The simulate field in the deployed state machine lets a caller choose the screening outcome. It exists only so the lab runs without partners. In production there is no such field; screening comes from a provider call, the submit step verifies a recent screening result, and the API rejects unknown properties. The deployed ledger writes items without balance checks; production uses transactions or Aurora, WAF, a custom authoriser and customer-managed keys.
- The stack's table encryption uses AWS managed keys, and the state machine logs errors only (
IncludeExecutionData = false) so request bodies do not reach CloudWatch. - The API uses IAM authorisation, so call it with SigV4 only; there is no anonymous path.
- The saga returns the reason
declinedfor a screening hit. The wording customers may receive is a compliance decision ⚠ VERIFY.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Tests hang or run slowly in Docker | Low memory | Keep --memory 3g; close other containers |
error CS0176 in your CDK additions | Amazon.CDK.Tags.Of(this) inside a Stack | Write Amazon.CDK.Tags.Of(this) |
| Concurrent-duplicate test is flaky after your change | Shared mutable state without a lock | Use the Lock pattern shown in InMemoryIdempotencyStore |
Test fails with UnbalancedPostingException | A posting's entries do not sum to zero | Check signs: debit negative, credit positive |
State machine returns DUPLICATE unexpectedly | Idempotency key reused | Use a new key, or delete the item from hl-m14-idempotency |
cdk deploy fails on table name conflict | Tables from an earlier run remain | cdk destroy, or delete the tables, then redeploy |
Cost and cleanup
The tests are free. The optional deployment costs cents for a few executions ⚠ VERIFY. Destroy the stack the same day:
cd labs/m14-payment-saga-failures/infra/cdk
cdk destroy -c owner=<your-name> --profile learn-sandbox
# Check nothing is left behind.
aws stepfunctions list-state-machines --profile learn-sandbox --region us-east-1
aws dynamodb list-tables --profile learn-sandbox --region us-east-1
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1Also delete bin/ and obj/ folders (the Docker command above does it) before committing.
Assignment
Add a short FAILURE-WORKSHOP.md to your portfolio copy of this lab: for each of the four scenarios list the failure, how the saga detects it, the automatic response, the customer impact, the manual step and the test that proves it. Add your fifth scenario and its test. Link it from the payments architecture you wrote in payment processing.
Solutions
[Fact]
public async Task Hold_is_visible_in_suspense_while_outcome_is_unknown()
{
var h = new Harness();
var gateway = new ScriptedGateway("timeout-lost") { QueriesUnavailable = true };
var saga = h.Build(new ScriptedScreener(), gateway);
var first = await saga.ExecuteAsync(Harness.Request());
Assert.Equal(750.00m, await h.Ledger.BalanceAsync(Harness.Customer, default));
Assert.Equal(250.00m, await h.Ledger.BalanceAsync("SYS-SUSPENSE", default));
gateway.QueriesUnavailable = false;
gateway.SchemeNowKnows(first.PaymentId, SubmissionOutcome.Rejected);
await saga.ResolveAsync(first.PaymentId);
Assert.Equal(1000.00m, await h.Ledger.BalanceAsync(Harness.Customer, default));
Assert.Equal(0m, await h.Ledger.BalanceAsync("SYS-SUSPENSE", default));
}Why: the money is never lost or doubled: it sits in suspense while the outcome is unknown, and the same ledger invariant (everything sums to zero) holds at every step.
[Fact]
public async Task Second_resolution_attempt_changes_nothing()
{
var h = new Harness();
var saga = h.Build(new ScriptedScreener(), new ScriptedGateway("accept"));
var first = await saga.ExecuteAsync(Harness.Request());
var again = await saga.ResolveAsync(first.PaymentId); // not OutcomeUnknown: returns current state
var replay = await saga.ExecuteAsync(Harness.Request()); // idempotent replay
Assert.Equal(PaymentStatus.Settled, again.Status);
Assert.Equal(PaymentStatus.Settled, replay.Status);
Assert.Equal(250.00m, await h.Ledger.BalanceAsync("SYS-SCHEME-FPS", default));
Assert.Equal(0m, await h.Ledger.BalanceAsync("SYS-SUSPENSE", default));
}Why: two independent protections exist: ResolveAsync acts only on OutcomeUnknown, and the ledger's settle:<paymentId> posting id makes a second settlement a no-op. Keeping both means a bug in one cannot double-pay.
Keep the saga's behaviour unchanged (fail closed, park in PendingReview). Emit a domain event PaymentPendingReview with the amount band and relationship manager reference (not the payment details) from the same code path, through the outbox so it is not lost. The notification platform plans an internal operations notification with a 15-minute SLA timer. Test: with ScriptedScreener("down","down","down") and an amount above 10,000, an IPaymentLog or event sink fake receives exactly one PaymentPendingReview event. Questions for compliance: what may be disclosed to the relationship manager (tipping-off rules), who may contact the customer, and what wording is permitted ⚠ VERIFY.
Knowledge check
Answer: Duplicate_retry_returns_the_same_payment_and_posts_once (and Ten_concurrent_duplicates_move_money_once for the concurrent case): they assert one scheme submission and one ledger movement.
Answer: The scheme may have accepted the payment. Refunding would pay the beneficiary and return the money to the customer. The hold is kept until a definite answer arrives.
Answer: Acceptance is the pivot: the payment is on its way. The only correct direction is forward, so the status stays Submitted with settlement-posting-pending and reconciliation finishes the posting.
Answer: The posting id derived from the payment id: a second posting with the same id is ignored.
Interview questions
- Walk me through what happens when the scheme times out. Query the status; accepted means continue, rejected means release the hold, unknown means keep the hold and let reconciliation resolve it from the scheme's report. Never guess.
- How would you test a saga? Put every dependency behind an interface, script failures with fakes (down, slow, hit, ambiguous timeout, transient posting error) and assert on both status and ledger balances.
- Why check balances, not just statuses, in tests? Statuses can be right while money is wrong. Balances prove the double-entry invariant.
Certification practice
A payment workflow's final step might be retried after a transient database error. Which design ensures a retry posts the ledger entry only once?
A. A random posting id each attempt B. A posting id derived from the payment id, with the ledger ignoring a repeated id C. Disabling retries D. A longer timeout
Answer: B. A deterministic id turns a retry into a no-op. A creates duplicates, C loses work on transient errors and D does not prevent duplicates.
A bank's payment service calls a scheme that sometimes times out after receiving the message. Which behaviour is correct?
A. Immediately refund the customer and resubmit with a new id B. Query the scheme for the status of the same identifier before deciding, keeping funds on hold if the answer is unknown C. Mark the payment as settled D. Delete the payment record
Answer: B. The outcome is unknown, so the safe action is to ask and hold. A risks paying twice, C records an unproven outcome, and D loses the audit trail.
Completion checklist
- I ran the tests and saw 22 pass
- I can point to the line that decides each of the four scenarios
- I wrote the guided test and the challenge test
- I can explain the pivot and why a timeout is an unknown
- If I deployed, I ran
cdk destroyand confirmed nothing taggedProject=cea-labremains - I deleted
bin/andobj/before committing
Key takeaways
- Specify failure behaviour as tests with scripted fakes; assert on balances as well as statuses.
- Idempotency exists at several levels: request key, scheme identifier, ledger posting id.
- Before the pivot compensate; after it move forward; in between, keep a named unknown state.
- Fail closed on screening and keep customer messages generic.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Payment processing: initiation, screening, settlement and idempotency
Design an end-to-end payment architecture on AWS: idempotent initiation, validation, a fail-closed sanctions-screening hook, a hold-submit-settle saga with a pivot point, a double-entry ledger, ISO 20022 messages, and the failure scenarios a design authority will probe.
Real-time transaction processing: latency budgets, back-pressure and ordering
Design a real-time authorisation path that decides in milliseconds on a small hot data set, then streams events to everything else: latency budgets with deadline propagation, DynamoDB balance reservations, Kinesis ordering per account, back-pressure and load shedding, and graceful degradation.