Distributed transactions and the saga pattern
Replace the distributed transaction you can't have with a saga: compare orchestration and choreography, design compensating actions and pivot steps, and build a Harbourline payment saga in AWS Step Functions with retries, catches, DynamoDB integrations and idempotent .NET Lambda steps.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · Amazon.Lambda.Core 3.4.0 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain why two-phase commit (MSDTC-style distributed transactions) is not available across AWS services, and what a saga replaces it with.
- Compare orchestration (Step Functions) and choreography (events on EventBridge), and choose between them for a given banking flow.
- Classify saga steps as compensatable, pivot or retriable, and design compensating actions that are idempotent and always eventually succeed.
- Handle the isolation problem (other requests seeing half-finished sagas) with semantic locks such as a "reserved" status.
- Build a payment saga in AWS Step Functions (Standard) with
Retry,Catch, optimised DynamoDB integrations and .NET 10 Lambda steps, in CDK (C#) and Terraform.
Prerequisites
- The Step Functions and EventBridge lessons earlier in this module (see the module overview)
- Retry strategies, Delivery guarantees and ordering and Idempotency in .NET
- Module 6: Databases — DynamoDB conditional updates
- Tools: .NET 10 SDK, AWS CLI v2 with
learn-sandbox; optionally the AWS CDK CLI or Terraform 1.14
Why this matters in banking
A Harbourline customer sends £1,250.00 to a new payee by Faster Payments. Behind one button, four systems must agree:
- Accounts must reserve the money, so it can't be spent twice.
- Sanctions screening must clear the payee.
- The scheme connector must submit the payment and get an acceptance.
- Accounts must turn the reservation into a posting, and Notifications must tell the customer.
In the on-premises world, the team wrapped steps 1 and 4 in a SQL Server transaction and hoped the rest wouldn't fail. In AWS, these are separate services with separate databases (DynamoDB, Aurora, a partner API). There is no transaction coordinator that spans them.
If screening finds a match after the money is reserved, someone must release the reservation. If the scheme rejects the payment, the reservation must be released and the customer told. If the release itself fails, the customer's money stays locked until someone notices, and the customer calls.
A saga is the pattern that makes these "undo" steps explicit, reliable and auditable. Regulators care about the outcome: no money created or lost, a clear record of every decision, customers informed. A saga gives you both the outcome and the record.
The idea in plain language
A distributed transaction (two-phase commit, 2PC) asks every participant "can you commit?", then tells them all "commit" together. It needs every participant to support the protocol and to hold locks while waiting. SQL Server and MSMQ could do this through MSDTC. DynamoDB, SQS, Lambda and a partner's REST API cannot.
A saga splits the business transaction into a sequence of local transactions. Each one commits on its own. If a later step fails, the saga runs compensating transactions for the steps that already committed, in reverse order. The system is never "rolled back" in the database sense. It moves forward to a correct state: a reservation followed by a release is a correct, auditable history.
Analogy: Booking a holiday through a travel agent. The agent books the flight, then the hotel, then the car. If the car hire fails, the agent doesn't pretend the flight never happened. They cancel the flight and the hotel, and you may see the booking and the cancellation on your statement. Cancellation is a new action with its own rules (maybe a fee, maybe a delay), not an undo button. Sagas work the same way.
Three kinds of step:
| Step type | Meaning | Harbourline example |
|---|---|---|
| Compensatable | Can be undone by a later compensating action | Reserve funds (compensation: release funds) |
| Pivot | The point of no return; once it succeeds, the saga must complete | Submit to the Faster Payments scheme; you can't "un-send" money |
| Retriable | Comes after the pivot; must eventually succeed, so it is retried rather than compensated | Post the debit, notify the customer |
Put the compensatable steps first, the pivot in the middle, and the retriable steps last. A failure before the pivot means compensate. A failure after the pivot means retry until done, or escalate to a person.
Two ways to coordinate a saga:
- Orchestration: a central coordinator (a Step Functions state machine) tells each service what to do and decides what happens on failure. Like a conductor.
- Choreography: no coordinator. Each service listens for events and publishes its own (
FundsReserved→ screening listens →PaymentScreened→ connector listens…). Like dancers who each know the routine.
What problem the AWS service solves
AWS Step Functions (Standard workflows) is an orchestrator built for sagas:
- Durable state: the execution's state is persisted between steps. A crashed Lambda doesn't lose the saga.
- Exactly-once workflow execution for Standard workflows. A step runs again only if you declare
Retry, which is what you want around a pivot like a payment submission. - Declarative error handling:
Retrywith backoff and jitter for transient errors, andCatchto route business failures to compensation states. - Direct service integrations: DynamoDB, SQS, SNS, EventBridge and more are called without a Lambda function in between, which means less code to maintain.
- Long waits: Standard workflows can run for up to one year. Callbacks (
.waitForTaskToken) wait for a scheme acknowledgement or a human approval. - Audit trail: every state transition is in the execution history (kept for 90 days) and can be sent to CloudWatch Logs for longer retention.
Limitations to know:
- Not a database transaction. Other requests can see intermediate states (the "isolation" problem below).
- Payload size limits for state input and output ⚠ VERIFY current limit. Pass IDs, not documents.
- Standard workflows are billed per state transition, including retries. Express workflows are cheaper for high volume but are at-least-once (async) and run for at most five minutes, so they are unsuitable around a non-idempotent pivot.
- Execution history is retained for 90 days. Export to CloudWatch Logs or S3 for longer audit retention.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
TransactionScope + MSDTC across SQL Server and MSMQ | No equivalent across AWS services; saga instead | You design compensation explicitly |
| Durable Functions orchestrator (C# code, replay-based) | Step Functions Standard workflow (Amazon States Language JSON, or the visual Workflow Studio) | Declarative definition instead of replayed C#; steps are Lambda functions or direct service calls |
| NServiceBus / MassTransit sagas (state stored in a saga table) | Step Functions (state in the execution) or choreography over EventBridge with your own saga state table | Step Functions hosts the state for you |
| Logic Apps with scopes and "run after failed" | Step Functions Catch | Catch routes by error name to any state |
| Service Bus + handlers reacting to events (choreography) | EventBridge rules + SQS + Lambda | Same pattern, same trade-offs |
Orchestration vs choreography
| Concern | Orchestration (Step Functions) | Choreography (EventBridge events) |
|---|---|---|
| Where the flow is defined | One state machine you can read top to bottom | Spread across every service's subscriptions |
| Visibility | Execution history shows each step, input, output and error | Needs distributed tracing and correlation IDs to reconstruct |
| Coupling | Orchestrator knows every participant's API | Services know only event contracts |
| Adding a step | Change the state machine | Add a subscriber; but "who compensates what" gets harder |
| Failure handling | Centralised Catch and compensation path | Each service must listen for failure events and compensate itself |
| Cyclic dependencies | Impossible by construction | Easy to create by accident |
| Best for | Money movement with strict, auditable sequencing | Loosely related reactions (notifications, analytics, loyalty points) |
Harbourline's rule of thumb: orchestrate the money, choreograph the side effects. The payment saga is a Step Functions workflow. When it finishes, it publishes PaymentSettled to EventBridge, and notifications, statements and analytics react independently.
Architecture
Read the failure paths carefully:
- Before the pivot (reserve, screen), a failure releases the funds and marks the payment rejected.
- At the pivot, a clear rejection from the scheme compensates. An unknown outcome (timeout) must not compensate blindly: the payment may have gone. The saga queries the scheme status by
EndToEndIdand continues down the right path. - After the pivot, failures are retried. Nothing is compensated, because the money has left.
- Compensations themselves retry with backoff. If they still fail, the saga hands the case to people through a manual queue rather than leaving funds locked silently.
The isolation problem
Sagas lack the "I" in ACID. While a saga runs, other requests can see its intermediate states. If the reservation simply subtracted £1,250.00 from the balance, a statement generated mid-saga would show a debit that may yet be released.
The fix is a semantic lock: the reservation is recorded as a hold (holds.PAY-… = 1250.00, available = available - 1250.00) without changing the booked balance. Other features read available for spending decisions and balance for statements. Only the final posting step changes balance.
Step-by-step implementation
Create the tables
hl-m08-accounts (partition key accountId) holds balance, available and a holds map. hl-m08-payments (partition key paymentId) holds the payment status. Seed one test account with £5,000.00.
Write the .NET Lambda steps
Two functions: ScreenPayee and SubmitToScheme. Each throws a named business exception for business outcomes (SanctionsHitException, SchemeRejectedException), and lets transient failures surface as other exceptions. The state machine matches on these names.
Write the state machine
Use the ASL definition in Complete code. Reserve and release use the optimised DynamoDB integration (no Lambda). Each Lambda task has a Retry for Lambda service errors with JitterStrategy: FULL, and a Catch that routes to compensation.
Deploy with CDK or Terraform
Both are shown below. Each grants the state machine role only the actions it needs.
Run the four scenarios
Start executions with name = paymentId and inputs that trigger:
- the happy path;
- a sanctions hit (creditor name contains
BLOCKED); - a scheme rejection (amount over £25,000.00);
- insufficient funds.
Open each execution in the console's graph view and follow the path.
AWS Console instructions
- Console → Step Functions → State machines → Create state machine → Create from blank. Choose Code view, paste the definition from Complete code, and replace the placeholders with your function ARNs and table names.
- Config tab: type Standard (this can't be changed later). Name
hl-m08-payment-saga. Logging: level ALL, include execution data off (payment data is personal). Choose or create a log group with a retention period. - Permissions: let the console create a role, then review it under IAM → Roles. It should allow
lambda:InvokeFunctionon the two functions,dynamodb:UpdateItemon the two tables,sqs:SendMessageon the manual queue andevents:PutEventson the bus. - Add the lab tags under Tags. Create.
- Start execution: set Name to the payment ID (for example
PAY-0192d1c2-5c4e-7000-8000-000000000001) and paste a test input. - Open the execution. Graph view colours each state. Table view shows retries and errors. Events shows the full history with inputs and outputs.
AWS CLI
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
SM_ARN=arn:aws:states:us-east-1:111122223333:stateMachine:hl-m08-payment-saga # from your deployment
# Seed a test account: balance and available both £5,000.00, no holds.
aws dynamodb put-item --table-name hl-m08-accounts --item '{
"accountId":{"S":"ACC-0000012345"},"currency":{"S":"GBP"},
"balance":{"N":"5000.00"},"available":{"N":"5000.00"},"holds":{"M":{}}}'
# Happy path. The execution name = paymentId makes a duplicate start (while running) idempotent.
aws stepfunctions start-execution --state-machine-arn "$SM_ARN" \
--name PAY-0192d1c2-5c4e-7000-8000-000000000001 \
--input '{"paymentId":"PAY-0192d1c2-5c4e-7000-8000-000000000001","debtorAccountId":"ACC-0000012345",
"creditorName":"Example Supplies Ltd","creditorIban":"GB33BUKB20201555555555",
"amount":"1250.00","currency":"GBP","scheme":"FPS",
"endToEndId":"E2E-0192d1c2-5c4e-7000-8000-000000000001"}'
# Sanctions hit → compensation path
aws stepfunctions start-execution --state-machine-arn "$SM_ARN" \
--name PAY-0192d1c2-5c4e-7000-8000-000000000002 \
--input '{"paymentId":"PAY-0192d1c2-5c4e-7000-8000-000000000002","debtorAccountId":"ACC-0000012345",
"creditorName":"BLOCKED Trading Co","creditorIban":"GB94BARC10201530093459",
"amount":"300.00","currency":"GBP","scheme":"FPS",
"endToEndId":"E2E-0192d1c2-5c4e-7000-8000-000000000002"}'
# Inspect: status, and the path taken
aws stepfunctions describe-execution --execution-arn "${SM_ARN/stateMachine/execution}:PAY-0192d1c2-5c4e-7000-8000-000000000002" \
--query '{status:status,output:output}'
aws stepfunctions get-execution-history --execution-arn "${SM_ARN/stateMachine/execution}:PAY-0192d1c2-5c4e-7000-8000-000000000002" \
--query 'events[?type==`TaskStateEntered`].stateEnteredEventDetails.name'
# Account should be back to available 5000.00 with no hold for payment 2.
aws dynamodb get-item --table-name hl-m08-accounts --key '{"accountId":{"S":"ACC-0000012345"}}'Amounts are passed as strings ("1250.00") and written to DynamoDB as numbers, so they never pass through a binary floating-point value. The IBANs above are commonly published example IBANs used only as test data.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.SQS;
using Amazon.CDK.AWS.StepFunctions;
using Constructs;
namespace Harbourline.Infra;
public sealed class PaymentSagaStack : Stack
{
public PaymentSagaStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
var accounts = new TableV2(this, "Accounts", new TablePropsV2
{
TableName = "hl-m08-accounts",
PartitionKey = new Attribute { Name = "accountId", Type = AttributeType.STRING },
Billing = Billing.OnDemand(),
RemovalPolicy = RemovalPolicy.DESTROY // lab only
});
var payments = new TableV2(this, "Payments", new TablePropsV2
{
TableName = "hl-m08-payments",
PartitionKey = new Attribute { Name = "paymentId", Type = AttributeType.STRING },
Billing = Billing.OnDemand(),
RemovalPolicy = RemovalPolicy.DESTROY
});
var manual = new Queue(this, "SagaManual", new QueueProps
{
QueueName = "hl-m08-saga-manual",
Encryption = QueueEncryption.SQS_MANAGED,
RetentionPeriod = Duration.Days(14),
EnforceSSL = true
});
var bus = new EventBus(this, "PaymentsBus", new EventBusProps { EventBusName = "hl-m08-payments" });
Function Step(string name, string handler) => new(this, name, new FunctionProps
{
FunctionName = $"hl-m08-{name.ToLowerInvariant()}",
Runtime = Runtime.DOTNET_10,
Handler = handler,
Code = Code.FromAsset("../src/Harbourline.Payments.Saga/publish"),
MemorySize = 512,
Timeout = Duration.Seconds(10)
});
var screen = Step("ScreenPayee", "Harbourline.Payments.Saga::Harbourline.Payments.Saga.Steps::ScreenPayee");
var submit = Step("SubmitToScheme", "Harbourline.Payments.Saga::Harbourline.Payments.Saga.Steps::SubmitToScheme");
var logs = new LogGroup(this, "SagaLogs", new LogGroupProps
{
LogGroupName = "/aws/vendedlogs/states/hl-m08-payment-saga",
Retention = RetentionDays.ONE_MONTH,
RemovalPolicy = RemovalPolicy.DESTROY
});
var sm = new StateMachine(this, "PaymentSaga", new StateMachineProps
{
StateMachineName = "hl-m08-payment-saga",
StateMachineType = StateMachineType.STANDARD,
DefinitionBody = DefinitionBody.FromFile("../statemachine/payment-saga.asl.json"),
DefinitionSubstitutions = new Dictionary<string, string>
{
["AccountsTable"] = accounts.TableName,
["PaymentsTable"] = payments.TableName,
["ScreenFunctionArn"] = screen.FunctionArn,
["SubmitFunctionArn"] = submit.FunctionArn,
["ManualQueueUrl"] = manual.QueueUrl,
["EventBusName"] = bus.EventBusName
},
Logs = new LogOptions { Destination = logs, Level = LogLevel.ALL, IncludeExecutionData = false },
TracingEnabled = true
});
// Least privilege: exactly what the definition uses.
screen.GrantInvoke(sm);
submit.GrantInvoke(sm);
accounts.Grant(sm, "dynamodb:UpdateItem");
payments.Grant(sm, "dynamodb:UpdateItem");
manual.GrantSendMessages(sm);
bus.GrantPutEventsTo(sm);
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m08-saga");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "alex");
}
}DefinitionSubstitutions replaces ${AccountsTable}-style placeholders in the ASL file at deploy time, so the definition file stays readable and environment-independent.
Complete code
The state machine (Amazon States Language, JSONPath)
{
"Comment": "Harbourline payment saga: reserve -> screen -> submit (pivot) -> post -> publish",
"StartAt": "ReserveFunds",
"States": {
"ReserveFunds": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:updateItem",
"Parameters": {
"TableName": "${AccountsTable}",
"Key": { "accountId": { "S.$": "$.debtorAccountId" } },
"UpdateExpression": "SET available = available - :amt, holds.#p = :amt",
"ConditionExpression": "available >= :amt AND attribute_not_exists(holds.#p)",
"ExpressionAttributeNames": { "#p.$": "$.paymentId" },
"ExpressionAttributeValues": { ":amt": { "N.$": "$.amount" } }
},
"ResultPath": null,
"Retry": [
{ "ErrorEquals": ["DynamoDB.ProvisionedThroughputExceededException", "DynamoDB.InternalServerErrorException"],
"IntervalSeconds": 1, "MaxAttempts": 4, "BackoffRate": 2.0, "JitterStrategy": "FULL" }
],
"Catch": [
{ "ErrorEquals": ["DynamoDB.ConditionalCheckFailedException"], "ResultPath": "$.error", "Next": "RejectInsufficientFunds" },
{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "MarkRejected" }
],
"Next": "ScreenPayee"
},
"ScreenPayee": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": { "FunctionName": "${ScreenFunctionArn}", "Payload.$": "$" },
"ResultSelector": { "screeningRef.$": "$.Payload.screeningRef" },
"ResultPath": "$.screening",
"Retry": [
{ "ErrorEquals": ["Lambda.ServiceException", "Lambda.SdkClientException", "Lambda.TooManyRequestsException"],
"IntervalSeconds": 1, "MaxAttempts": 4, "BackoffRate": 2.0, "MaxDelaySeconds": 10, "JitterStrategy": "FULL" },
{ "ErrorEquals": ["TransientProviderException"],
"IntervalSeconds": 5, "MaxAttempts": 3, "BackoffRate": 2.0, "JitterStrategy": "FULL" }
],
"Catch": [
{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "ReleaseFunds" }
],
"Next": "SubmitToScheme"
},
"SubmitToScheme": {
"Comment": "PIVOT. Never retried on ambiguous errors; the Lambda itself is idempotent on endToEndId.",
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": { "FunctionName": "${SubmitFunctionArn}", "Payload.$": "$" },
"ResultSelector": { "schemeRef.$": "$.Payload.schemeRef", "outcome.$": "$.Payload.outcome" },
"ResultPath": "$.submission",
"Retry": [
{ "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException"],
"IntervalSeconds": 1, "MaxAttempts": 3, "BackoffRate": 2.0, "JitterStrategy": "FULL" }
],
"Catch": [
{ "ErrorEquals": ["SchemeRejectedException"], "ResultPath": "$.error", "Next": "ReleaseFunds" },
{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "OutcomeUnknown" }
],
"Next": "PostDebit"
},
"OutcomeUnknown": {
"Comment": "Timeout or unexpected error at the pivot: do NOT compensate blindly. Hand to operations to query the scheme by EndToEndId.",
"Type": "Task",
"Resource": "arn:aws:states:::sqs:sendMessage",
"Parameters": {
"QueueUrl": "${ManualQueueUrl}",
"MessageBody": {
"reason": "PIVOT_OUTCOME_UNKNOWN",
"paymentId.$": "$.paymentId",
"endToEndId.$": "$.endToEndId",
"executionArn.$": "$$.Execution.Id"
}
},
"ResultPath": null,
"Next": "AwaitingInvestigation"
},
"AwaitingInvestigation": { "Type": "Fail", "Error": "Harbourline.PivotOutcomeUnknown", "Cause": "Sent to hl-m08-saga-manual" },
"PostDebit": {
"Comment": "RETRIABLE. After the pivot we retry, never compensate.",
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:updateItem",
"Parameters": {
"TableName": "${AccountsTable}",
"Key": { "accountId": { "S.$": "$.debtorAccountId" } },
"UpdateExpression": "SET balance = balance - :amt REMOVE holds.#p",
"ConditionExpression": "attribute_exists(holds.#p)",
"ExpressionAttributeNames": { "#p.$": "$.paymentId" },
"ExpressionAttributeValues": { ":amt": { "N.$": "$.amount" } }
},
"ResultPath": null,
"Retry": [
{ "ErrorEquals": ["DynamoDB.ConditionalCheckFailedException"], "MaxAttempts": 0 },
{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 10, "BackoffRate": 2.0,
"MaxDelaySeconds": 300, "JitterStrategy": "FULL" }
],
"Catch": [
{ "ErrorEquals": ["DynamoDB.ConditionalCheckFailedException"], "ResultPath": null, "Next": "MarkSettled" },
{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "EscalatePostFailure" }
],
"Next": "MarkSettled"
},
"MarkSettled": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:updateItem",
"Parameters": {
"TableName": "${PaymentsTable}",
"Key": { "paymentId": { "S.$": "$.paymentId" } },
"UpdateExpression": "SET #s = :s, schemeRef = :r",
"ExpressionAttributeNames": { "#s": "status" },
"ExpressionAttributeValues": { ":s": { "S": "settled" }, ":r": { "S.$": "$.submission.schemeRef" } }
},
"ResultPath": null,
"Retry": [ { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 10, "BackoffRate": 2.0, "MaxDelaySeconds": 300, "JitterStrategy": "FULL" } ],
"Next": "PublishSettled"
},
"PublishSettled": {
"Type": "Task",
"Resource": "arn:aws:states:::events:putEvents",
"Parameters": {
"Entries": [ {
"EventBusName": "${EventBusName}",
"Source": "harbourline.payments",
"DetailType": "PaymentSettled",
"Detail": { "paymentId.$": "$.paymentId", "debtorAccountId.$": "$.debtorAccountId",
"amount.$": "$.amount", "currency.$": "$.currency" }
} ]
},
"ResultPath": null,
"Retry": [ { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 10, "BackoffRate": 2.0, "MaxDelaySeconds": 300, "JitterStrategy": "FULL" } ],
"End": true
},
"ReleaseFunds": {
"Comment": "COMPENSATION for ReserveFunds. Idempotent: only acts if the hold still exists.",
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:updateItem",
"Parameters": {
"TableName": "${AccountsTable}",
"Key": { "accountId": { "S.$": "$.debtorAccountId" } },
"UpdateExpression": "SET available = available + :amt REMOVE holds.#p",
"ConditionExpression": "attribute_exists(holds.#p)",
"ExpressionAttributeNames": { "#p.$": "$.paymentId" },
"ExpressionAttributeValues": { ":amt": { "N.$": "$.amount" } }
},
"ResultPath": null,
"Retry": [
{ "ErrorEquals": ["DynamoDB.ConditionalCheckFailedException"], "MaxAttempts": 0 },
{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 10, "BackoffRate": 2.0,
"MaxDelaySeconds": 300, "JitterStrategy": "FULL" }
],
"Catch": [
{ "ErrorEquals": ["DynamoDB.ConditionalCheckFailedException"], "ResultPath": null, "Next": "MarkRejected" },
{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.compensationError", "Next": "EscalateCompensationFailure" }
],
"Next": "MarkRejected"
},
"RejectInsufficientFunds": { "Type": "Pass", "Result": "INSUFFICIENT_FUNDS", "ResultPath": "$.rejectionReason", "Next": "MarkRejected" },
"MarkRejected": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:updateItem",
"Parameters": {
"TableName": "${PaymentsTable}",
"Key": { "paymentId": { "S.$": "$.paymentId" } },
"UpdateExpression": "SET #s = :s",
"ExpressionAttributeNames": { "#s": "status" },
"ExpressionAttributeValues": { ":s": { "S": "rejected" } }
},
"ResultPath": null,
"Retry": [ { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 10, "BackoffRate": 2.0, "MaxDelaySeconds": 300, "JitterStrategy": "FULL" } ],
"Next": "PublishRejected"
},
"PublishRejected": {
"Type": "Task",
"Resource": "arn:aws:states:::events:putEvents",
"Parameters": {
"Entries": [ {
"EventBusName": "${EventBusName}",
"Source": "harbourline.payments",
"DetailType": "PaymentRejected",
"Detail": { "paymentId.$": "$.paymentId", "debtorAccountId.$": "$.debtorAccountId" }
} ]
},
"ResultPath": null,
"Retry": [ { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 10, "BackoffRate": 2.0, "MaxDelaySeconds": 300, "JitterStrategy": "FULL" } ],
"Next": "PaymentRejected"
},
"PaymentRejected": { "Type": "Fail", "Error": "Harbourline.PaymentRejected", "Cause": "See execution history for the reason" },
"EscalateCompensationFailure": {
"Type": "Task",
"Resource": "arn:aws:states:::sqs:sendMessage",
"Parameters": {
"QueueUrl": "${ManualQueueUrl}",
"MessageBody": { "reason": "COMPENSATION_FAILED", "paymentId.$": "$.paymentId", "executionArn.$": "$$.Execution.Id" }
},
"ResultPath": null,
"Next": "CompensationFailed"
},
"CompensationFailed": { "Type": "Fail", "Error": "Harbourline.CompensationFailed", "Cause": "Funds may still be held; sent to operations" },
"EscalatePostFailure": {
"Type": "Task",
"Resource": "arn:aws:states:::sqs:sendMessage",
"Parameters": {
"QueueUrl": "${ManualQueueUrl}",
"MessageBody": { "reason": "POST_AFTER_PIVOT_FAILED", "paymentId.$": "$.paymentId", "executionArn.$": "$$.Execution.Id" }
},
"ResultPath": null,
"Next": "PostFailed"
},
"PostFailed": { "Type": "Fail", "Error": "Harbourline.PostAfterPivotFailed", "Cause": "Money sent but not posted; sent to operations" }
}
}The .NET Lambda steps
using Amazon.Lambda.Core;
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]
namespace Harbourline.Payments.Saga;
/// <summary>Saga input. Amount is a string so it never passes through binary floating point.</summary>
public sealed record PaymentCommand(
string PaymentId, string DebtorAccountId, string CreditorName, string CreditorIban,
string Amount, string Currency, string Scheme, string EndToEndId);
public sealed record ScreeningResult(string ScreeningRef);
public sealed record SubmissionResult(string SchemeRef, string Outcome);
// Exception class names become the Step Functions error names matched in ErrorEquals.
public sealed class SanctionsHitException(string message) : Exception(message);
public sealed class TransientProviderException(string message) : Exception(message);
public sealed class SchemeRejectedException(string message) : Exception(message);
public interface ISchemeConnector
{
/// <summary>Must be idempotent on endToEndId: a repeat returns the original outcome.</summary>
Task<SubmissionResult> SubmitAsync(PaymentCommand payment, CancellationToken ct);
}
public sealed class Steps
{
private readonly ISchemeConnector _scheme;
public Steps() : this(new FakeSchemeConnector()) { }
public Steps(ISchemeConnector scheme) => _scheme = scheme;
public Task<ScreeningResult> ScreenPayee(PaymentCommand payment, ILambdaContext context)
{
context.Logger.LogInformation($"screening payment={payment.PaymentId}");
// Placeholder rule for the lab; a real adapter calls the screening provider.
if (payment.CreditorName.Contains("BLOCKED", StringComparison.OrdinalIgnoreCase))
throw new SanctionsHitException($"Potential match for payment {payment.PaymentId}");
return Task.FromResult(new ScreeningResult($"SCR-{payment.PaymentId[4..]}"));
}
public async Task<SubmissionResult> SubmitToScheme(PaymentCommand payment, ILambdaContext context)
{
// Leave a margin so we fail cleanly before Lambda kills us mid-call.
using var cts = new CancellationTokenSource(context.RemainingTime - TimeSpan.FromSeconds(2));
var result = await _scheme.SubmitAsync(payment, cts.Token);
if (result.Outcome == "rejected")
throw new SchemeRejectedException($"Scheme rejected {payment.EndToEndId}");
context.Logger.LogInformation($"submitted payment={payment.PaymentId} schemeRef={result.SchemeRef}");
return result;
}
}
/// <summary>Lab stand-in for the Faster Payments connector. Rejects amounts over £25,000.00.</summary>
public sealed class FakeSchemeConnector : ISchemeConnector
{
public Task<SubmissionResult> SubmitAsync(PaymentCommand payment, CancellationToken ct)
{
var amount = decimal.Parse(payment.Amount, System.Globalization.CultureInfo.InvariantCulture);
return Task.FromResult(amount > 25_000.00m
? new SubmissionResult($"FPS-{payment.EndToEndId[4..]}", "rejected")
: new SubmissionResult($"FPS-{payment.EndToEndId[4..]}", "accepted"));
}
}Unit tests (no AWS needed)
using Amazon.Lambda.TestUtilities;
using Harbourline.Payments.Saga;
using Xunit;
public class StepsTests
{
private static PaymentCommand Payment(string creditor = "Example Supplies Ltd", string amount = "1250.00") =>
new("PAY-0192d1c2-5c4e-7000-8000-000000000001", "ACC-0000012345", creditor,
"GB33BUKB20201555555555", amount, "GBP", "FPS", "E2E-0192d1c2-5c4e-7000-8000-000000000001");
private static TestLambdaContext Ctx() => new() { RemainingTime = TimeSpan.FromSeconds(10) };
[Fact]
public async Task Clean_payee_is_screened()
{
var r = await new Steps().ScreenPayee(Payment(), Ctx());
Assert.StartsWith("SCR-", r.ScreeningRef);
}
[Fact]
public async Task Sanctions_match_throws_named_business_error() =>
await Assert.ThrowsAsync<SanctionsHitException>(() => new Steps().ScreenPayee(Payment("BLOCKED Trading Co"), Ctx()));
[Fact]
public async Task Scheme_rejection_is_a_named_error_not_a_generic_failure() =>
await Assert.ThrowsAsync<SchemeRejectedException>(() => new Steps().SubmitToScheme(Payment(amount: "30000.00"), Ctx()));
[Fact]
public async Task Resubmission_with_same_end_to_end_id_returns_same_reference()
{
var steps = new Steps();
var a = await steps.SubmitToScheme(Payment(), Ctx());
var b = await steps.SubmitToScheme(Payment(), Ctx());
Assert.Equal(a.SchemeRef, b.SchemeRef);
}
}The tests use Amazon.Lambda.TestUtilities for TestLambdaContext, and xunit.v3 4.0.3 on Microsoft.Testing.Platform. To test the state machine without deploying it, use the Step Functions TestState API (aws stepfunctions test-state), which runs a single state with a given input and shows the output, retries and next state ⚠ VERIFY pricing and quotas. Or use the Step Functions Local tooling.
Code explained line by line
ReserveFundswitharn:aws:states:::dynamodb:updateItem— a direct service integration: no Lambda, no code. The update reducesavailableand records a hold keyed by payment ID in theholdsmap. This is the semantic lock.ConditionExpression: available >= :amt AND attribute_not_exists(holds.#p)— atomic funds check and idempotency. If the state is re-run, the existing hold makes the condition fail instead of reserving twice."#p.$": "$.paymentId"— the.$suffix tells Step Functions to take the value from the input. That's how a map key becomes the payment ID."N.$": "$.amount"— DynamoDB numbers are strings in the API, so the string amount flows straight through without floating-point conversion.CatchonDynamoDB.ConditionalCheckFailedException— for the reservation, a failed condition means insufficient funds (or a duplicate). It routes to a clean rejection, not to a retry.ResultPath: null— discards the task result and passes the original input on unchanged. This keeps the saga's payload small and predictable.ResultSelectoron Lambda tasks — picks only the fields you need from the Lambda response ($.Payload.…), so step outputs don't bloat the execution state.- Two retriers on
ScreenPayee— Lambda service errors (retry quickly) andTransientProviderException(retry slower).JitterStrategy: FULLis opt-in.SanctionsHitExceptionisn't in any retrier, so it goes straight toCatch. SubmitToScheme— the pivot. It retries only Lambda service errors, where the function never ran. A function timeout or unknown error goes toOutcomeUnknown, not to compensation. The money may have left, and releasing the hold could let the customer spend it twice.OutcomeUnknown→ SQS →Fail— a person (or an automated reconciler) queries the scheme byEndToEndIdand completes the saga manually. In a fuller design, use.waitForTaskTokento pause the saga until the reconciler callsSendTaskSuccessorSendTaskFailure.PostDebit— retriable after the pivot. Up to 10 attempts with capped backoff. AConditionalCheckFailedExceptionmeans the hold is already gone (a previous attempt succeeded), so it is treated as done and routes toMarkSettled.ReleaseFunds— the compensation. It is idempotent by the same trick: only acts if the hold exists. It is retried generously. If it still fails,EscalateCompensationFailuremakes sure a person knows money is stuck.$$.Execution.Id— the$$prefix reads the context object. The execution ARN goes into escalation messages so operations can open the exact execution.- Exception class names as error names — when a .NET Lambda function throws, the error type reported to Step Functions is the exception's type name (for example
SchemeRejectedException), which is whatErrorEqualsmatches ⚠ VERIFY the exact format (simple vs fully qualified name) for your runtime by checking an execution's error output. context.RemainingTime - 2 s— the scheme call is cancelled before Lambda's own timeout. The step then fails with a clear error instead of a hard timeout mid-request.ISchemeConnector.SubmitAsyncmust be idempotent onendToEndId— even with a careful state machine, an invocation might be repeated, for example on a Lambda service error after the request actually reached the function. The ISO 20022EndToEndIdis the natural idempotency key that schemes and connectors recognise.
Security considerations
- Don't log execution data.
IncludeExecutionData: falsekeeps names, IBANs and amounts out of CloudWatch Logs. The execution history in Step Functions still contains them, so restrictstates:DescribeExecutionandstates:GetExecutionHistoryto the operations role. - Least privilege for the state machine role: invoke only the two functions,
dynamodb:UpdateItemonly on the two tables, send only to the manual queue, and put events only on the payments bus. Avoid console-generated roles with wildcards in production. - Who can start executions? Only the Payments API role should have
states:StartExecutionon this state machine. Anyone who can start it can move money. - Segregation of duties for manual resolution: the people who resolve
PIVOT_OUTCOME_UNKNOWNcases should not be the people who can change the state machine definition. - Encrypt the tables, the manual queue (it holds payment IDs) and the logs. Use customer managed KMS keys where your policy requires it.
⚠ Learning shortcut — not for production
The sanctions step is a string match and the scheme connector is a fake. A real screening step calls a provider and may return "potential match — needs review", which is a wait (callback pattern with .waitForTaskToken), not a rejection. Real scheme submissions are asynchronous: you submit, then wait for an acknowledgement message. Model that as a callback task with HeartbeatSeconds and a timeout, and send the timeout to OutcomeUnknown. Also add a reconciliation job that compares holds older than N minutes against saga states, because a saga is a process, and processes need independent checks.
Banking use case
The Harbourline design review for this saga produced a compensation table, now required for every saga:
| Step | Type | Compensation | Compensation idempotency | If compensation fails |
|---|---|---|---|---|
| Reserve funds | Compensatable | Release hold | Conditional on hold existing | Retry 10× with backoff, then operations queue (funds stuck) |
| Screen payee | Compensatable (no state to undo) | None | — | — |
| Submit to scheme | Pivot | None. A clear rejection triggers compensation of earlier steps; an unknown outcome triggers investigation | Scheme deduplicates on EndToEndId | — |
| Post debit | Retriable | — | Conditional on hold existing | Retry 10×, then operations queue (money sent, not posted) |
| Mark settled, publish event | Retriable | — | Upsert / event consumers idempotent | Retry 10× |
Two further rules came from the risk team:
- Each escalation reason maps to a runbook and a service-level target. Funds held for a rejected payment must be released within 4 business hours, so customers aren't locked out of their money.
- Reconciliation runs every 15 minutes. Holds older than 30 minutes without a running execution raise an alert.
These are Harbourline's internal targets. Any regulatory or contractual timelines need confirming with the compliance team.
Common mistakes
- Compensating after the pivot → releasing funds for a payment that actually went → double spend → classify steps; send ambiguous pivot outcomes to investigation, not compensation.
- Non-idempotent compensations → a retried release adds the money back twice → condition every compensation on the existence of what it undoes.
- Compensations that can fail permanently with no escalation → money silently stuck → generous retries, then a manual queue with an alarm and a runbook.
- Retrying business errors →
SanctionsHitExceptionretried three times, delaying the customer's rejection → only transient errors inRetry; business errors inCatch. - Using Express workflows for the payment saga → at-least-once async execution can submit twice; 5-minute limit → Standard for money movement.
- Passing whole documents through the state → payload limits and personal data in execution history → pass IDs; read details inside the steps.
- Choreographing the money → compensation logic spread across five services, no single place that shows the payment's state → orchestrate money, choreograph side effects.
- Mutating the booked balance at reservation time → statements and interest calculations see uncommitted debits → semantic lock (
availablevsbalance).
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
States.Runtime error: "JSONPath ... could not be found" | A field referenced with .$ isn't in the state input (often lost because ResultPath overwrote the input) | Use ResultPath to put results under a sub-path, or null to discard |
| Catch never fires for a business error | ErrorEquals doesn't match the error name the Lambda reported | Open the failed state's output in the console; copy the exact Error value |
ExecutionAlreadyExists on start | An execution with this name exists with different input, or a finished one is still within history retention | For retries of the same payment, treat as "already started"; never reuse a payment ID for a new payment |
AccessDeniedException on dynamodb:UpdateItem | State machine role missing the action or table ARN | Add exactly that action on that table |
DynamoDB.ValidationException about the update expression | Map attribute holds doesn't exist on the account item | Seed accounts with "holds":{"M":{}}, or use if_not_exists |
Saga stuck in SubmitToScheme for minutes | Lambda timeout longer than expected; no state TimeoutSeconds | Set Lambda and state timeouts; route timeouts to OutcomeUnknown |
| Costs higher than expected | Large MaxAttempts on frequently failing states; each retry is a billed transition | Fix the failing dependency; cap attempts; add a circuit breaker upstream |
Cost and cleanup
| Item | Cost at lab volume |
|---|---|
| Step Functions Standard | Billed per state transition; the free tier includes a monthly allowance of transitions ⚠ VERIFY current allowance and price. Each saga run here is about 6–10 transitions |
| Lambda | Within the always-free tier |
| DynamoDB on-demand | Fractions of a cent |
| CloudWatch Logs | Small ingestion and storage; one-month retention set |
| EventBridge custom events | Negligible at lab volume |
Free alternative: run the unit tests, and use aws stepfunctions test-state against individual states, or a local Step Functions emulator. The design work (the compensation table) is the most valuable part, and it's free.
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
# CDK: cdk destroy PaymentSagaStack Terraform: terraform destroy
# Manual cleanup if you created resources by hand:
aws stepfunctions delete-state-machine --state-machine-arn "$SM_ARN"
aws dynamodb delete-table --table-name hl-m08-accounts
aws dynamodb delete-table --table-name hl-m08-payments
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m08-saga-manual --query QueueUrl --output text)"
aws events delete-event-bus --name hl-m08-payments
aws logs delete-log-group --log-group-name /aws/vendedlogs/states/hl-m08-payment-saga
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m08-saga \
--query "ResourceTagMappingList[].ResourceARN"Exercises
- 🟢 Guided: Run the four scenarios (happy path, sanctions hit, scheme rejection over £25,000.00, insufficient funds with £9,000.00). For each, list the states entered and the final values of
available,balanceandholdson the account. - 🟡 Challenge: Replace
OutcomeUnknownwith a callback: a Task with the resourcearn:aws:states:::sqs:sendMessage.waitForTaskTokenthat sends the task token to a reconciler queue, waits up to 4 hours (TimeoutSeconds), and continues toPostDebitorReleaseFundsdepending on the reconciler's answer. Write the small .NET reconciler that callsSendTaskSuccess. - 🔴 Live problem: The card team wants to build "buy now, pay later" as a choreographed saga across Cards, Credit Decisioning, Merchant Settlement and Notifications, all reacting to EventBridge events. They say "Step Functions is a single point of failure". Prepare your review comments.
Assignment
Add to your portfolio repository:
- the ASL definition;
- the compensation table for the payment saga;
- the unit-tested step functions;
- a Mermaid diagram of the happy and failure paths.
Then write a one-page saga design template that your team could use for any new multi-service flow. It should cover steps, classification, compensations, idempotency keys, escalation and reconciliation.
Solutions
Starting from balance = available = 5000.00:
| Scenario | States entered | After |
|---|---|---|
| Happy path, £1,250.00 | ReserveFunds → ScreenPayee → SubmitToScheme → PostDebit → MarkSettled → PublishSettled | balance 3750.00, available 3750.00, no hold |
| Sanctions hit, £300.00 | ReserveFunds → ScreenPayee → ReleaseFunds → MarkRejected → PublishRejected → PaymentRejected | unchanged from before the run; hold added then removed |
| Scheme rejection, £30,000.00 | ReserveFunds → RejectInsufficientFunds → MarkRejected → PublishRejected → PaymentRejected (with only £5,000.00 available, the reservation fails first) | unchanged |
| Insufficient funds, £9,000.00 | ReserveFunds → RejectInsufficientFunds → MarkRejected → PublishRejected → PaymentRejected | unchanged |
To see a genuine scheme rejection, first raise the test account to £50,000.00. Then the £30,000.00 payment reaches SubmitToScheme, is rejected, and follows the ReleaseFunds path.
Why: Checking holds after each run proves the semantic lock and the compensation are both correct and idempotent.
"AwaitReconciliation": {
"Type": "Task",
"Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
"Parameters": {
"QueueUrl": "${ReconcileQueueUrl}",
"MessageBody": { "taskToken.$": "$$.Task.Token", "endToEndId.$": "$.endToEndId", "paymentId.$": "$.paymentId" }
},
"TimeoutSeconds": 14400,
"ResultPath": "$.reconciliation",
"Catch": [ { "ErrorEquals": ["States.Timeout"], "ResultPath": "$.error", "Next": "EscalateUnknown" } ],
"Next": "ReconciledOutcome"
},
"ReconciledOutcome": {
"Type": "Choice",
"Choices": [ { "Variable": "$.reconciliation.outcome", "StringEquals": "accepted", "Next": "PostDebit" } ],
"Default": "ReleaseFunds"
}// Reconciler: reads the queue, queries the scheme by EndToEndId, reports back.
var sfn = new Amazon.StepFunctions.AmazonStepFunctionsClient();
var status = await scheme.QueryStatusAsync(msg.EndToEndId, ct); // "accepted" | "rejected" | "unknown"
if (status == "unknown") return; // leave message; retry later
await sfn.SendTaskSuccessAsync(new Amazon.StepFunctions.Model.SendTaskSuccessRequest
{
TaskToken = msg.TaskToken,
Output = System.Text.Json.JsonSerializer.Serialize(new { outcome = status })
}, ct);Why: The saga waits durably (free while waiting, apart from the transition) instead of failing. The reconciler holds the domain knowledge of how to ask the scheme. Grant the reconciler only states:SendTaskSuccess and states:SendTaskFailure.
Review comments:
- "Single point of failure" is a misunderstanding. Step Functions is a managed, Regional, multi-AZ service, and executions are durable. The real single point of failure risk in choreography is the knowledge of the flow, which ends up spread across four teams.
- Money movement needs a single, auditable view. For BNPL, the credit decision, the merchant settlement and the repayment schedule are tightly linked. Auditors and complaints handlers need one place that shows the state of each agreement and why. With choreography you must build that view yourself (a saga state table fed by every event).
- Compensation complexity. List every failure combination. If Merchant Settlement fails after Credit Decisioning approved, who cancels the credit agreement, and how do they know? In choreography each service must subscribe to every relevant failure event, which risks cycles and missed cases.
- Proposal: orchestrate the agreement life cycle (decision → agreement → settlement) in a Standard workflow. Publish domain events (
AgreementOpened,MerchantSettled) for Notifications, Analytics and Collections to choreograph around it. - Ask for: the compensation table, idempotency keys per step, the escalation path for "settled but agreement failed", and a reconciliation design. Note that consumer credit carries regulatory requirements that compliance must review.
Why: Choosing orchestration or choreography isn't about fashion. It depends on where you need a single source of truth and a clear failure story. For money, that is usually orchestration.
Knowledge check
Answer: Those participants don't support a two-phase commit protocol, and 2PC requires every participant to hold locks while a coordinator decides. That's unavailable and undesirable at cloud scale. A saga uses local transactions plus compensations instead.
Answer: The point of no return, such as sending money to a scheme. Steps before it are compensated on failure; steps after it are retried until they succeed. Compensating after a pivot (or when its outcome is unknown) can create a double spend.
Answer: Make it conditional on the thing it undoes, here attribute_exists(holds.<paymentId>), and remove the hold in the same update. A repeat finds no hold, fails the condition, and is treated as "already done".
Answer: Standard workflows provide exactly-once workflow execution, durable state, run for up to a year and support callbacks. Asynchronous Express workflows are at-least-once and limited to five minutes, so a non-idempotent step could run twice and long waits are impossible.
Answer: Other transactions can see intermediate saga states, because there's no global isolation. A semantic lock, such as recording a hold and reducing available while leaving balance unchanged until posting, lets other features make correct decisions while the saga runs.
Interview questions
- "Orchestration or choreography?" — Orchestrate flows that need a single source of truth and strict sequencing, such as money movement. Choreograph independent reactions. A common hybrid is an orchestrated core that publishes events for choreographed side effects.
- "How do you handle a timeout when calling a payment scheme?" — Treat it as an unknown outcome, not a failure. Don't compensate. Query the status by the idempotency key (
EndToEndId), or wait for the asynchronous acknowledgement with a callback task. Escalate if still unknown, and reconcile independently. - "What makes a good compensating action?" — It is semantically correct (a release or a reversal, not a delete), idempotent, retried until it succeeds, logged for audit, and escalated if it can't complete.
- "How would you test a saga?" — Unit-test each step's business decisions. Use TestState for individual states, including retry and catch routing. Run integration tests for every failure path (one per row of the compensation table). Inject faults in a test environment. Reconcile in production.
Certification practice
A Step Functions Standard workflow invokes a Lambda function that validates a payment. When validation fails for a business reason, the function throws ValidationFailedException. The developer wants the workflow to route these failures to a compensation state immediately, while still retrying transient Lambda service errors. What should the developer configure?
A. A Retry with ErrorEquals: ["States.ALL"] and MaxAttempts: 3.
B. A Retry for Lambda.ServiceException and Lambda.TooManyRequestsException, and a Catch for ValidationFailedException with Next set to the compensation state.
C. A Catch for States.ALL only.
D. Set the function's asynchronous retry attempts to 0.
Answer: B. Retriers handle transient errors; the catcher routes the named business error to compensation without retrying it. A retries business errors pointlessly. C skips retries for transient errors. D doesn't apply: Step Functions invokes Lambda synchronously in this pattern.
A bank's transfer process updates an account in DynamoDB, calls an external payment network, and records the result in another DynamoDB table. If the external call fails, the account update must be reversed. The process must be auditable and survive component failures. Which design is MOST appropriate?
A. A single Lambda function that performs all steps and uses try/catch to reverse the account update. B. An AWS Step Functions Standard workflow implementing a saga with compensating steps. C. A DynamoDB transaction that includes the external API call. D. Amazon SQS FIFO queues between each step with no compensation logic.
Answer: B. Step Functions persists state between steps, retries transient failures, routes errors to compensation, and keeps an execution history. A loses state if the function crashes or times out mid-process. C: DynamoDB transactions can't include external calls. D gives ordering but no reversal.
A developer wants to check that a single Task state's Catch block routes a specific error to the correct next state, without deploying the full workflow or invoking downstream services. Which approach is MOST efficient?
A. Deploy the state machine and run end-to-end tests in production. B. Use the Step Functions TestState API with a mocked or provided input for that state. C. Add logging to every Lambda function. D. Convert the workflow to Express to make it cheaper to test.
Answer: B. TestState runs one state definition and reports its output, error handling and next state. A is slow and risky. C doesn't test routing. D changes semantics and still needs a deployment.
Completion checklist
- I can explain why 2PC isn't available and what a saga does instead
- I can classify steps as compensatable, pivot or retriable for a given flow
- I deployed or read through the payment saga and can follow every failure path
- My compensations are idempotent and escalate when they can't complete
- I filled in a compensation table for the payment saga
- I deleted the state machine, tables, queue, bus and log group
Key takeaways
- A saga = local transactions + compensating transactions, coordinated by orchestration or choreography. It moves forward to a correct state; it doesn't roll back.
- Order steps compensatable → pivot → retriable. Never compensate after the pivot. Treat an unknown pivot outcome as "investigate", not "failed".
- Compensations must be idempotent and must eventually succeed. Escalate to people when they can't.
- Use Step Functions Standard for money movement: durable state, exactly-once execution,
Retrywith jitter,Catchto compensation, and a full history. - Handle missing isolation with semantic locks (holds and
available), and back every saga with reconciliation.
Further reading
- Saga pattern (AWS Prescriptive Guidance)
- Handling errors in Step Functions workflows
- Choosing workflow type in Step Functions
- Amazon DynamoDB Transactions: how it works
aws-samples/saga-orchestration-netcore-blogon GitHub (MIT-0) — a .NET saga with Step Functions and CDK in C#, the basis for Lab 8.4 in this module
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Idempotency in .NET with DynamoDB and Powertools
Make every money-moving API call and every message handler safe to repeat: choose idempotency keys, build a DynamoDB idempotency store with conditional writes and TTL, combine business writes atomically with TransactWriteItems, and use the Powertools for AWS Lambda (.NET) idempotency utility.
AWS Step Functions: workflows, retries and callbacks
Orchestrate multi-step banking processes with AWS Step Functions: Standard versus Express workflows, the state types, JSONata, service integrations, Retry and Catch, human-approval callbacks with task tokens, Map and Distributed Map, redrive, testing with TestState, and a .NET 10 client, CDK and Terraform.