Skip to Content
Cloud Engineering Academy
14. Advanced Banking Architecture

Lab: payment saga failure scenarios (design studies 14.3 and 14.4)

Run and extend a unit-tested payment saga in C# that survives a sanctions provider outage, duplicate client retries, a scheme timeout and a partial settlement, then optionally deploy a minimal Step Functions version and destroy it the same day.

advanced⏱ 2 h 30 min🟢 FreeSAP-C03SAA-C03DVA-C03

Assumes: .NET 10 LTS · xUnit v3 4.0.3 (Microsoft.Testing.Platform) · CDK v2 (Amazon.CDK.Lib 2.277.0) · AWS Step Functions Express · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lab you can:

  • Run 22 unit tests that specify how a payment behaves under four kinds of failure, with no AWS account.
  • Trace each failure through PaymentSaga and point at the line that decides the outcome.
  • Show that a duplicate client retry, ten concurrent duplicates and a replayed ledger posting each move money once.
  • Explain why a scheme timeout keeps the hold (OutcomeUnknown) rather than refunding the customer.
  • Extend the saga with a fifth scenario and prove it with a test.
  • Optionally deploy the Step Functions proof of concept (cents), exercise the same scenarios through it, and tear it down.

Prerequisites

  • Payment processing: the design this lab implements
  • Saga pattern and idempotency in .NET
  • The lab code: labs/m14-payment-saga-failures/ in the course repository (its README.md repeats the commands)
  • Tools: Docker (or the .NET 10 SDK). AWS CLI v2 and the learn-sandbox profile only for the optional deployment

Why this matters in banking

A design review does not ask "does it work?". It asks "show me what happens when the sanctions provider is down, when the client retries, when the scheme goes quiet and when settlement fails halfway". This lab turns those four questions into tests that a reviewer can read as specifications, which is more convincing than a paragraph in a document.

The idea in plain language

The saga is a short sequence of steps with a pivot in the middle:

Before the scheme accepts, a failure is compensated by releasing the hold. After it accepts, the saga moves forward only. In between, "no answer" is a named state, not a failure.

Analogy: sending a parcel by courier. You set the money aside, hand over the parcel and if the courier loses contact you ring the recipient before refunding yourself.

Setup and run

Run the tests

From the repository root:

MSYS_NO_PATHCONV=1 docker run --rm --memory 3g \
  -v "$PWD/labs/m14-payment-saga-failures:/src" \
  -v cea-nuget-m14:/root/.nuget/packages \
  -w /src mcr.microsoft.com/dotnet/sdk:10.0 \
  sh -c "dotnet test --solution PaymentSagaFailures.slnx; find . -type d \( -name bin -o -name obj \) -prune -exec rm -rf {} +"

The first run downloads the CDK packages (a few minutes). Expected output ends with Test run summary: Passed! and total: 22. dotnet test does not compile the CDK project; dotnet build PaymentSagaFailures.slnx does.

Read the saga

Open src/Harbourline.Payments.Saga/PaymentSaga.cs. Find ExecuteAsync (validation, idempotency, then RunAsync), RunAsync (screen, hold, submit), FinishAsync (the decision after the scheme's answer), SettleAsync (idempotent posting with retries) and ResolveAsync (the reconciliation entry point).

Map each test to a scenario

Open tests/Harbourline.Payments.Saga.Tests/FailureScenarioTests.cs. Each test builds a Harness (ledger with GBP 1,000 for ACC-0000000001) and scripts a fake: ScriptedScreener ("down", "hit", "clear"), ScriptedGateway ("accept", "reject", "timeout-received", "timeout-lost") and FlakySettlementLedger.

Architecture

Every dependency is an interface, so the same saga code runs against DynamoDB, Aurora, a real screening provider and a real scheme gateway in production. The doubles are the only difference in the tests.

Code walkthrough

The idempotent entry point

var begin = await idempotency.TryBeginAsync(request.IdempotencyKey, request.ComputeBodyHash(), ct);
switch (begin.Outcome)
{
    case BeginOutcome.Conflict:
        throw new IdempotencyConflictException(request.IdempotencyKey);
    case BeginOutcome.InProgress:
        return new PaymentResult(string.Empty, PaymentStatus.Received, "in-progress");
    case BeginOutcome.Replay:
        var stored = begin.Stored!;
        var current = store.Get(stored.PaymentId);
        return current is null ? stored : new PaymentResult(current.PaymentId, current.Status, current.Detail);
}
  1. ComputeBodyHash hashes everything except the key, so a retry (same key, same body) is recognised and a key reuse (same key, different body) is a Conflict.
  2. InProgress covers the concurrent duplicate: only the caller that got Started does the work.
  3. Replay returns the current state of the stored payment, not the first response. An OutcomeUnknown payment that reconciliation has since settled therefore shows Settled.

Fail closed on screening

var screening = await ScreenWithRetryAsync(paymentId, request, ct);
if (screening is null)
{
    log.Record(paymentId, "screening", "unavailable after retries; parked for review");
    return Save(paymentId, request, PaymentStatus.PendingReview, "screening-unavailable");
}

null means every attempt threw ScreeningUnavailableException. The payment is parked, no money moves and no submission happens. A hit returns the generic reason declined, and the detailed reason goes only to the internal log.

The pivot

case null:
    log.Record(paymentId, "submit", "outcome unknown; hold kept; reconciliation will resolve");
    return Save(paymentId, request, PaymentStatus.OutcomeUnknown, "awaiting-confirmation");

case SubmissionOutcome.Rejected:
    await ledger.PostAsync("release:" + paymentId, [...], ct);
    return Save(paymentId, request, PaymentStatus.Rejected, "scheme-rejected");

FinishAsync receives the scheme's answer (or null for "still unknown after the status queries"). Only a definite rejection releases the hold. The unknown case keeps it.

Idempotent postings

await ledger.PostAsync(
    "settle:" + paymentId,
    [ new LedgerEntry(_options.SuspenseAccountId, -request.Amount, request.Currency),
      new LedgerEntry(_options.SchemeAccountId, request.Amount, request.Currency) ],
    ct);

The posting id is derived from the payment id. InMemoryLedger.PostAsync ignores a second posting with the same id, so retrying after a transient failure (or after a crash that left the outcome ambiguous) cannot post twice. It also rejects any posting that does not sum to zero per currency, and any customer account that would go negative.

Exercises

  • 🟢 Guided: add a test named Hold_is_visible_in_suspense_while_outcome_is_unknown asserting the customer balance is 750.00 and the suspense balance is 250.00 after a timeout-lost run, then 1,000.00 and 0 after ResolveAsync with a scheme rejection.
  • 🟡 Challenge: add the fifth scenario from the lesson: the scheme acknowledges twice. Prove that calling ResolveAsync and ExecuteAsync again on a Settled payment changes nothing, and that the scheme account holds the amount exactly once.
  • 🔴 Live problem: payments operations says that for payments over GBP 10,000 a screening outage must not park the payment silently: the customer's relationship manager must be told within 15 minutes. Without changing the fail-closed behaviour, design where this notification hooks in (saga, event, notification platform), what the test looks like and what you must ask compliance before releasing it.

Optional deployment (credits, cents)

The CDK app in infra/cdk deploys: two DynamoDB tables (idempotency with a TTL attribute, ledger), an Express state machine with the saga as Amazon States Language, and an IAM-authorised API Gateway in front of it. Costs at lab volumes are cents ⚠ VERIFY; destroy it the same day.

cd labs/m14-payment-saga-failures/infra/cdk
cdk deploy -c owner=<your-name> --profile learn-sandbox

aws stepfunctions start-sync-execution \
  --state-machine-arn <StateMachineArn> \
  --input '{"idempotencyKey":"k-0001","debtorAccountId":"ACC-0000000001","amount":"250.00","currency":"GBP","simulate":{"sanctions":"clear","scheme":"accept"}}' \
  --profile learn-sandbox --region us-east-1

Then try: the same key again (DUPLICATE), "sanctions":"hit" (status REJECTED), "sanctions":"down" (PENDING_REVIEW) and "scheme":"reject" (a RELEASE# ledger item then REJECTED). Check aws dynamodb scan on both tables to see what was recorded.

Security considerations

⚠ Learning shortcut — not for production

The simulate field in the deployed state machine lets a caller choose the screening outcome. It exists only so the lab runs without partners. In production there is no such field; screening comes from a provider call, the submit step verifies a recent screening result, and the API rejects unknown properties. The deployed ledger writes items without balance checks; production uses transactions or Aurora, WAF, a custom authoriser and customer-managed keys.

  • The stack's table encryption uses AWS managed keys, and the state machine logs errors only (IncludeExecutionData = false) so request bodies do not reach CloudWatch.
  • The API uses IAM authorisation, so call it with SigV4 only; there is no anonymous path.
  • The saga returns the reason declined for a screening hit. The wording customers may receive is a compliance decision ⚠ VERIFY.

Troubleshooting

SymptomLikely causeFix
Tests hang or run slowly in DockerLow memoryKeep --memory 3g; close other containers
error CS0176 in your CDK additionsAmazon.CDK.Tags.Of(this) inside a StackWrite Amazon.CDK.Tags.Of(this)
Concurrent-duplicate test is flaky after your changeShared mutable state without a lockUse the Lock pattern shown in InMemoryIdempotencyStore
Test fails with UnbalancedPostingExceptionA posting's entries do not sum to zeroCheck signs: debit negative, credit positive
State machine returns DUPLICATE unexpectedlyIdempotency key reusedUse a new key, or delete the item from hl-m14-idempotency
cdk deploy fails on table name conflictTables from an earlier run remaincdk destroy, or delete the tables, then redeploy

Cost and cleanup

The tests are free. The optional deployment costs cents for a few executions ⚠ VERIFY. Destroy the stack the same day:

cd labs/m14-payment-saga-failures/infra/cdk
cdk destroy -c owner=<your-name> --profile learn-sandbox

# Check nothing is left behind.
aws stepfunctions list-state-machines --profile learn-sandbox --region us-east-1
aws dynamodb list-tables --profile learn-sandbox --region us-east-1
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --profile learn-sandbox --region us-east-1

Also delete bin/ and obj/ folders (the Docker command above does it) before committing.

Assignment

Add a short FAILURE-WORKSHOP.md to your portfolio copy of this lab: for each of the four scenarios list the failure, how the saga detects it, the automatic response, the customer impact, the manual step and the test that proves it. Add your fifth scenario and its test. Link it from the payments architecture you wrote in payment processing.

Solutions

Knowledge check

Interview questions

  • Walk me through what happens when the scheme times out. Query the status; accepted means continue, rejected means release the hold, unknown means keep the hold and let reconciliation resolve it from the scheme's report. Never guess.
  • How would you test a saga? Put every dependency behind an interface, script failures with fakes (down, slow, hit, ambiguous timeout, transient posting error) and assert on both status and ledger balances.
  • Why check balances, not just statuses, in tests? Statuses can be right while money is wrong. Balances prove the double-entry invariant.

Certification practice

Completion checklist

  • I ran the tests and saw 22 pass
  • I can point to the line that decides each of the four scenarios
  • I wrote the guided test and the challenge test
  • I can explain the pivot and why a timeout is an unknown
  • If I deployed, I ran cdk destroy and confirmed nothing tagged Project=cea-lab remains
  • I deleted bin/ and obj/ before committing

Key takeaways

  • Specify failure behaviour as tests with scripted fakes; assert on balances as well as statuses.
  • Idempotency exists at several levels: request key, scheme identifier, ledger posting id.
  • Before the pivot compensate; after it move forward; in between, keep a named unknown state.
  • Fail closed on screening and keep customer messages generic.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page