Skip to Content
Cloud Engineering Academy
8. Messaging, Events & Microservices

Distributed transactions and the saga pattern

Replace the distributed transaction you can't have with a saga: compare orchestration and choreography, design compensating actions and pivot steps, and build a Harbourline payment saga in AWS Step Functions with retries, catches, DynamoDB integrations and idempotent .NET Lambda steps.

advanced⏱ 2 h 30 min🟢 FreeDVA-C03SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 · Amazon.Lambda.Core 3.4.0 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain why two-phase commit (MSDTC-style distributed transactions) is not available across AWS services, and what a saga replaces it with.
  • Compare orchestration (Step Functions) and choreography (events on EventBridge), and choose between them for a given banking flow.
  • Classify saga steps as compensatable, pivot or retriable, and design compensating actions that are idempotent and always eventually succeed.
  • Handle the isolation problem (other requests seeing half-finished sagas) with semantic locks such as a "reserved" status.
  • Build a payment saga in AWS Step Functions (Standard) with Retry, Catch, optimised DynamoDB integrations and .NET 10 Lambda steps, in CDK (C#) and Terraform.

Prerequisites

Why this matters in banking

A Harbourline customer sends £1,250.00 to a new payee by Faster Payments. Behind one button, four systems must agree:

  1. Accounts must reserve the money, so it can't be spent twice.
  2. Sanctions screening must clear the payee.
  3. The scheme connector must submit the payment and get an acceptance.
  4. Accounts must turn the reservation into a posting, and Notifications must tell the customer.

In the on-premises world, the team wrapped steps 1 and 4 in a SQL Server transaction and hoped the rest wouldn't fail. In AWS, these are separate services with separate databases (DynamoDB, Aurora, a partner API). There is no transaction coordinator that spans them.

If screening finds a match after the money is reserved, someone must release the reservation. If the scheme rejects the payment, the reservation must be released and the customer told. If the release itself fails, the customer's money stays locked until someone notices, and the customer calls.

A saga is the pattern that makes these "undo" steps explicit, reliable and auditable. Regulators care about the outcome: no money created or lost, a clear record of every decision, customers informed. A saga gives you both the outcome and the record.

The idea in plain language

A distributed transaction (two-phase commit, 2PC) asks every participant "can you commit?", then tells them all "commit" together. It needs every participant to support the protocol and to hold locks while waiting. SQL Server and MSMQ could do this through MSDTC. DynamoDB, SQS, Lambda and a partner's REST API cannot.

A saga splits the business transaction into a sequence of local transactions. Each one commits on its own. If a later step fails, the saga runs compensating transactions for the steps that already committed, in reverse order. The system is never "rolled back" in the database sense. It moves forward to a correct state: a reservation followed by a release is a correct, auditable history.

Analogy: Booking a holiday through a travel agent. The agent books the flight, then the hotel, then the car. If the car hire fails, the agent doesn't pretend the flight never happened. They cancel the flight and the hotel, and you may see the booking and the cancellation on your statement. Cancellation is a new action with its own rules (maybe a fee, maybe a delay), not an undo button. Sagas work the same way.

Three kinds of step:

Step typeMeaningHarbourline example
CompensatableCan be undone by a later compensating actionReserve funds (compensation: release funds)
PivotThe point of no return; once it succeeds, the saga must completeSubmit to the Faster Payments scheme; you can't "un-send" money
RetriableComes after the pivot; must eventually succeed, so it is retried rather than compensatedPost the debit, notify the customer

Put the compensatable steps first, the pivot in the middle, and the retriable steps last. A failure before the pivot means compensate. A failure after the pivot means retry until done, or escalate to a person.

Two ways to coordinate a saga:

  • Orchestration: a central coordinator (a Step Functions state machine) tells each service what to do and decides what happens on failure. Like a conductor.
  • Choreography: no coordinator. Each service listens for events and publishes its own (FundsReserved → screening listens → PaymentScreened → connector listens…). Like dancers who each know the routine.

What problem the AWS service solves

AWS Step Functions (Standard workflows) is an orchestrator built for sagas:

  • Durable state: the execution's state is persisted between steps. A crashed Lambda doesn't lose the saga.
  • Exactly-once workflow execution for Standard workflows. A step runs again only if you declare Retry, which is what you want around a pivot like a payment submission.
  • Declarative error handling: Retry with backoff and jitter for transient errors, and Catch to route business failures to compensation states.
  • Direct service integrations: DynamoDB, SQS, SNS, EventBridge and more are called without a Lambda function in between, which means less code to maintain.
  • Long waits: Standard workflows can run for up to one year. Callbacks (.waitForTaskToken) wait for a scheme acknowledgement or a human approval.
  • Audit trail: every state transition is in the execution history (kept for 90 days) and can be sent to CloudWatch Logs for longer retention.

Limitations to know:

  • Not a database transaction. Other requests can see intermediate states (the "isolation" problem below).
  • Payload size limits for state input and output ⚠ VERIFY current limit. Pass IDs, not documents.
  • Standard workflows are billed per state transition, including retries. Express workflows are cheaper for high volume but are at-least-once (async) and run for at most five minutes, so they are unsuitable around a non-idempotent pivot.
  • Execution history is retained for 90 days. Export to CloudWatch Logs or S3 for longer audit retention.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
TransactionScope + MSDTC across SQL Server and MSMQNo equivalent across AWS services; saga insteadYou design compensation explicitly
Durable Functions orchestrator (C# code, replay-based)Step Functions Standard workflow (Amazon States Language JSON, or the visual Workflow Studio)Declarative definition instead of replayed C#; steps are Lambda functions or direct service calls
NServiceBus / MassTransit sagas (state stored in a saga table)Step Functions (state in the execution) or choreography over EventBridge with your own saga state tableStep Functions hosts the state for you
Logic Apps with scopes and "run after failed"Step Functions CatchCatch routes by error name to any state
Service Bus + handlers reacting to events (choreography)EventBridge rules + SQS + LambdaSame pattern, same trade-offs

Orchestration vs choreography

ConcernOrchestration (Step Functions)Choreography (EventBridge events)
Where the flow is definedOne state machine you can read top to bottomSpread across every service's subscriptions
VisibilityExecution history shows each step, input, output and errorNeeds distributed tracing and correlation IDs to reconstruct
CouplingOrchestrator knows every participant's APIServices know only event contracts
Adding a stepChange the state machineAdd a subscriber; but "who compensates what" gets harder
Failure handlingCentralised Catch and compensation pathEach service must listen for failure events and compensate itself
Cyclic dependenciesImpossible by constructionEasy to create by accident
Best forMoney movement with strict, auditable sequencingLoosely related reactions (notifications, analytics, loyalty points)

Harbourline's rule of thumb: orchestrate the money, choreograph the side effects. The payment saga is a Step Functions workflow. When it finishes, it publishes PaymentSettled to EventBridge, and notifications, statements and analytics react independently.

Architecture

Read the failure paths carefully:

  • Before the pivot (reserve, screen), a failure releases the funds and marks the payment rejected.
  • At the pivot, a clear rejection from the scheme compensates. An unknown outcome (timeout) must not compensate blindly: the payment may have gone. The saga queries the scheme status by EndToEndId and continues down the right path.
  • After the pivot, failures are retried. Nothing is compensated, because the money has left.
  • Compensations themselves retry with backoff. If they still fail, the saga hands the case to people through a manual queue rather than leaving funds locked silently.

The isolation problem

Sagas lack the "I" in ACID. While a saga runs, other requests can see its intermediate states. If the reservation simply subtracted £1,250.00 from the balance, a statement generated mid-saga would show a debit that may yet be released.

The fix is a semantic lock: the reservation is recorded as a hold (holds.PAY-… = 1250.00, available = available - 1250.00) without changing the booked balance. Other features read available for spending decisions and balance for statements. Only the final posting step changes balance.

Step-by-step implementation

Create the tables

hl-m08-accounts (partition key accountId) holds balance, available and a holds map. hl-m08-payments (partition key paymentId) holds the payment status. Seed one test account with £5,000.00.

Write the .NET Lambda steps

Two functions: ScreenPayee and SubmitToScheme. Each throws a named business exception for business outcomes (SanctionsHitException, SchemeRejectedException), and lets transient failures surface as other exceptions. The state machine matches on these names.

Write the state machine

Use the ASL definition in Complete code. Reserve and release use the optimised DynamoDB integration (no Lambda). Each Lambda task has a Retry for Lambda service errors with JitterStrategy: FULL, and a Catch that routes to compensation.

Deploy with CDK or Terraform

Both are shown below. Each grants the state machine role only the actions it needs.

Run the four scenarios

Start executions with name = paymentId and inputs that trigger:

  • the happy path;
  • a sanctions hit (creditor name contains BLOCKED);
  • a scheme rejection (amount over £25,000.00);
  • insufficient funds.

Open each execution in the console's graph view and follow the path.

AWS Console instructions

  1. Console → Step Functions → State machines → Create state machine → Create from blank. Choose Code view, paste the definition from Complete code, and replace the placeholders with your function ARNs and table names.
  2. Config tab: type Standard (this can't be changed later). Name hl-m08-payment-saga. Logging: level ALL, include execution data off (payment data is personal). Choose or create a log group with a retention period.
  3. Permissions: let the console create a role, then review it under IAM → Roles. It should allow lambda:InvokeFunction on the two functions, dynamodb:UpdateItem on the two tables, sqs:SendMessage on the manual queue and events:PutEvents on the bus.
  4. Add the lab tags under Tags. Create.
  5. Start execution: set Name to the payment ID (for example PAY-0192d1c2-5c4e-7000-8000-000000000001) and paste a test input.
  6. Open the execution. Graph view colours each state. Table view shows retries and errors. Events shows the full history with inputs and outputs.

AWS CLI

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
SM_ARN=arn:aws:states:us-east-1:111122223333:stateMachine:hl-m08-payment-saga   # from your deployment

# Seed a test account: balance and available both £5,000.00, no holds.
aws dynamodb put-item --table-name hl-m08-accounts --item '{
  "accountId":{"S":"ACC-0000012345"},"currency":{"S":"GBP"},
  "balance":{"N":"5000.00"},"available":{"N":"5000.00"},"holds":{"M":{}}}'

# Happy path. The execution name = paymentId makes a duplicate start (while running) idempotent.
aws stepfunctions start-execution --state-machine-arn "$SM_ARN" \
  --name PAY-0192d1c2-5c4e-7000-8000-000000000001 \
  --input '{"paymentId":"PAY-0192d1c2-5c4e-7000-8000-000000000001","debtorAccountId":"ACC-0000012345",
            "creditorName":"Example Supplies Ltd","creditorIban":"GB33BUKB20201555555555",
            "amount":"1250.00","currency":"GBP","scheme":"FPS",
            "endToEndId":"E2E-0192d1c2-5c4e-7000-8000-000000000001"}'

# Sanctions hit → compensation path
aws stepfunctions start-execution --state-machine-arn "$SM_ARN" \
  --name PAY-0192d1c2-5c4e-7000-8000-000000000002 \
  --input '{"paymentId":"PAY-0192d1c2-5c4e-7000-8000-000000000002","debtorAccountId":"ACC-0000012345",
            "creditorName":"BLOCKED Trading Co","creditorIban":"GB94BARC10201530093459",
            "amount":"300.00","currency":"GBP","scheme":"FPS",
            "endToEndId":"E2E-0192d1c2-5c4e-7000-8000-000000000002"}'

# Inspect: status, and the path taken
aws stepfunctions describe-execution --execution-arn "${SM_ARN/stateMachine/execution}:PAY-0192d1c2-5c4e-7000-8000-000000000002" \
  --query '{status:status,output:output}'
aws stepfunctions get-execution-history --execution-arn "${SM_ARN/stateMachine/execution}:PAY-0192d1c2-5c4e-7000-8000-000000000002" \
  --query 'events[?type==`TaskStateEntered`].stateEnteredEventDetails.name'

# Account should be back to available 5000.00 with no hold for payment 2.
aws dynamodb get-item --table-name hl-m08-accounts --key '{"accountId":{"S":"ACC-0000012345"}}'

Amounts are passed as strings ("1250.00") and written to DynamoDB as numbers, so they never pass through a binary floating-point value. The IBANs above are commonly published example IBANs used only as test data.

Infrastructure as Code

PaymentSagaStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.SQS;
using Amazon.CDK.AWS.StepFunctions;
using Constructs;

namespace Harbourline.Infra;

public sealed class PaymentSagaStack : Stack
{
    public PaymentSagaStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        var accounts = new TableV2(this, "Accounts", new TablePropsV2
        {
            TableName = "hl-m08-accounts",
            PartitionKey = new Attribute { Name = "accountId", Type = AttributeType.STRING },
            Billing = Billing.OnDemand(),
            RemovalPolicy = RemovalPolicy.DESTROY // lab only
        });
        var payments = new TableV2(this, "Payments", new TablePropsV2
        {
            TableName = "hl-m08-payments",
            PartitionKey = new Attribute { Name = "paymentId", Type = AttributeType.STRING },
            Billing = Billing.OnDemand(),
            RemovalPolicy = RemovalPolicy.DESTROY
        });
        var manual = new Queue(this, "SagaManual", new QueueProps
        {
            QueueName = "hl-m08-saga-manual",
            Encryption = QueueEncryption.SQS_MANAGED,
            RetentionPeriod = Duration.Days(14),
            EnforceSSL = true
        });
        var bus = new EventBus(this, "PaymentsBus", new EventBusProps { EventBusName = "hl-m08-payments" });

        Function Step(string name, string handler) => new(this, name, new FunctionProps
        {
            FunctionName = $"hl-m08-{name.ToLowerInvariant()}",
            Runtime = Runtime.DOTNET_10,
            Handler = handler,
            Code = Code.FromAsset("../src/Harbourline.Payments.Saga/publish"),
            MemorySize = 512,
            Timeout = Duration.Seconds(10)
        });
        var screen = Step("ScreenPayee", "Harbourline.Payments.Saga::Harbourline.Payments.Saga.Steps::ScreenPayee");
        var submit = Step("SubmitToScheme", "Harbourline.Payments.Saga::Harbourline.Payments.Saga.Steps::SubmitToScheme");

        var logs = new LogGroup(this, "SagaLogs", new LogGroupProps
        {
            LogGroupName = "/aws/vendedlogs/states/hl-m08-payment-saga",
            Retention = RetentionDays.ONE_MONTH,
            RemovalPolicy = RemovalPolicy.DESTROY
        });

        var sm = new StateMachine(this, "PaymentSaga", new StateMachineProps
        {
            StateMachineName = "hl-m08-payment-saga",
            StateMachineType = StateMachineType.STANDARD,
            DefinitionBody = DefinitionBody.FromFile("../statemachine/payment-saga.asl.json"),
            DefinitionSubstitutions = new Dictionary<string, string>
            {
                ["AccountsTable"] = accounts.TableName,
                ["PaymentsTable"] = payments.TableName,
                ["ScreenFunctionArn"] = screen.FunctionArn,
                ["SubmitFunctionArn"] = submit.FunctionArn,
                ["ManualQueueUrl"] = manual.QueueUrl,
                ["EventBusName"] = bus.EventBusName
            },
            Logs = new LogOptions { Destination = logs, Level = LogLevel.ALL, IncludeExecutionData = false },
            TracingEnabled = true
        });

        // Least privilege: exactly what the definition uses.
        screen.GrantInvoke(sm);
        submit.GrantInvoke(sm);
        accounts.Grant(sm, "dynamodb:UpdateItem");
        payments.Grant(sm, "dynamodb:UpdateItem");
        manual.GrantSendMessages(sm);
        bus.GrantPutEventsTo(sm);

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m08-saga");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        Amazon.CDK.Tags.Of(this).Add("Owner", "alex");
    }
}

DefinitionSubstitutions replaces ${AccountsTable}-style placeholders in the ASL file at deploy time, so the definition file stays readable and environment-independent.

Complete code

The state machine (Amazon States Language, JSONPath)

statemachine/payment-saga.asl.json
{
  "Comment": "Harbourline payment saga: reserve -> screen -> submit (pivot) -> post -> publish",
  "StartAt": "ReserveFunds",
  "States": {
    "ReserveFunds": {
      "Type": "Task",
      "Resource": "arn:aws:states:::dynamodb:updateItem",
      "Parameters": {
        "TableName": "${AccountsTable}",
        "Key": { "accountId": { "S.$": "$.debtorAccountId" } },
        "UpdateExpression": "SET available = available - :amt, holds.#p = :amt",
        "ConditionExpression": "available >= :amt AND attribute_not_exists(holds.#p)",
        "ExpressionAttributeNames": { "#p.$": "$.paymentId" },
        "ExpressionAttributeValues": { ":amt": { "N.$": "$.amount" } }
      },
      "ResultPath": null,
      "Retry": [
        { "ErrorEquals": ["DynamoDB.ProvisionedThroughputExceededException", "DynamoDB.InternalServerErrorException"],
          "IntervalSeconds": 1, "MaxAttempts": 4, "BackoffRate": 2.0, "JitterStrategy": "FULL" }
      ],
      "Catch": [
        { "ErrorEquals": ["DynamoDB.ConditionalCheckFailedException"], "ResultPath": "$.error", "Next": "RejectInsufficientFunds" },
        { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "MarkRejected" }
      ],
      "Next": "ScreenPayee"
    },

    "ScreenPayee": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "${ScreenFunctionArn}", "Payload.$": "$" },
      "ResultSelector": { "screeningRef.$": "$.Payload.screeningRef" },
      "ResultPath": "$.screening",
      "Retry": [
        { "ErrorEquals": ["Lambda.ServiceException", "Lambda.SdkClientException", "Lambda.TooManyRequestsException"],
          "IntervalSeconds": 1, "MaxAttempts": 4, "BackoffRate": 2.0, "MaxDelaySeconds": 10, "JitterStrategy": "FULL" },
        { "ErrorEquals": ["TransientProviderException"],
          "IntervalSeconds": 5, "MaxAttempts": 3, "BackoffRate": 2.0, "JitterStrategy": "FULL" }
      ],
      "Catch": [
        { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "ReleaseFunds" }
      ],
      "Next": "SubmitToScheme"
    },

    "SubmitToScheme": {
      "Comment": "PIVOT. Never retried on ambiguous errors; the Lambda itself is idempotent on endToEndId.",
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "${SubmitFunctionArn}", "Payload.$": "$" },
      "ResultSelector": { "schemeRef.$": "$.Payload.schemeRef", "outcome.$": "$.Payload.outcome" },
      "ResultPath": "$.submission",
      "Retry": [
        { "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException"],
          "IntervalSeconds": 1, "MaxAttempts": 3, "BackoffRate": 2.0, "JitterStrategy": "FULL" }
      ],
      "Catch": [
        { "ErrorEquals": ["SchemeRejectedException"], "ResultPath": "$.error", "Next": "ReleaseFunds" },
        { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "OutcomeUnknown" }
      ],
      "Next": "PostDebit"
    },

    "OutcomeUnknown": {
      "Comment": "Timeout or unexpected error at the pivot: do NOT compensate blindly. Hand to operations to query the scheme by EndToEndId.",
      "Type": "Task",
      "Resource": "arn:aws:states:::sqs:sendMessage",
      "Parameters": {
        "QueueUrl": "${ManualQueueUrl}",
        "MessageBody": {
          "reason": "PIVOT_OUTCOME_UNKNOWN",
          "paymentId.$": "$.paymentId",
          "endToEndId.$": "$.endToEndId",
          "executionArn.$": "$$.Execution.Id"
        }
      },
      "ResultPath": null,
      "Next": "AwaitingInvestigation"
    },
    "AwaitingInvestigation": { "Type": "Fail", "Error": "Harbourline.PivotOutcomeUnknown", "Cause": "Sent to hl-m08-saga-manual" },

    "PostDebit": {
      "Comment": "RETRIABLE. After the pivot we retry, never compensate.",
      "Type": "Task",
      "Resource": "arn:aws:states:::dynamodb:updateItem",
      "Parameters": {
        "TableName": "${AccountsTable}",
        "Key": { "accountId": { "S.$": "$.debtorAccountId" } },
        "UpdateExpression": "SET balance = balance - :amt REMOVE holds.#p",
        "ConditionExpression": "attribute_exists(holds.#p)",
        "ExpressionAttributeNames": { "#p.$": "$.paymentId" },
        "ExpressionAttributeValues": { ":amt": { "N.$": "$.amount" } }
      },
      "ResultPath": null,
      "Retry": [
        { "ErrorEquals": ["DynamoDB.ConditionalCheckFailedException"], "MaxAttempts": 0 },
        { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 10, "BackoffRate": 2.0,
          "MaxDelaySeconds": 300, "JitterStrategy": "FULL" }
      ],
      "Catch": [
        { "ErrorEquals": ["DynamoDB.ConditionalCheckFailedException"], "ResultPath": null, "Next": "MarkSettled" },
        { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "EscalatePostFailure" }
      ],
      "Next": "MarkSettled"
    },

    "MarkSettled": {
      "Type": "Task",
      "Resource": "arn:aws:states:::dynamodb:updateItem",
      "Parameters": {
        "TableName": "${PaymentsTable}",
        "Key": { "paymentId": { "S.$": "$.paymentId" } },
        "UpdateExpression": "SET #s = :s, schemeRef = :r",
        "ExpressionAttributeNames": { "#s": "status" },
        "ExpressionAttributeValues": { ":s": { "S": "settled" }, ":r": { "S.$": "$.submission.schemeRef" } }
      },
      "ResultPath": null,
      "Retry": [ { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 10, "BackoffRate": 2.0, "MaxDelaySeconds": 300, "JitterStrategy": "FULL" } ],
      "Next": "PublishSettled"
    },

    "PublishSettled": {
      "Type": "Task",
      "Resource": "arn:aws:states:::events:putEvents",
      "Parameters": {
        "Entries": [ {
          "EventBusName": "${EventBusName}",
          "Source": "harbourline.payments",
          "DetailType": "PaymentSettled",
          "Detail": { "paymentId.$": "$.paymentId", "debtorAccountId.$": "$.debtorAccountId",
                      "amount.$": "$.amount", "currency.$": "$.currency" }
        } ]
      },
      "ResultPath": null,
      "Retry": [ { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 10, "BackoffRate": 2.0, "MaxDelaySeconds": 300, "JitterStrategy": "FULL" } ],
      "End": true
    },

    "ReleaseFunds": {
      "Comment": "COMPENSATION for ReserveFunds. Idempotent: only acts if the hold still exists.",
      "Type": "Task",
      "Resource": "arn:aws:states:::dynamodb:updateItem",
      "Parameters": {
        "TableName": "${AccountsTable}",
        "Key": { "accountId": { "S.$": "$.debtorAccountId" } },
        "UpdateExpression": "SET available = available + :amt REMOVE holds.#p",
        "ConditionExpression": "attribute_exists(holds.#p)",
        "ExpressionAttributeNames": { "#p.$": "$.paymentId" },
        "ExpressionAttributeValues": { ":amt": { "N.$": "$.amount" } }
      },
      "ResultPath": null,
      "Retry": [
        { "ErrorEquals": ["DynamoDB.ConditionalCheckFailedException"], "MaxAttempts": 0 },
        { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 10, "BackoffRate": 2.0,
          "MaxDelaySeconds": 300, "JitterStrategy": "FULL" }
      ],
      "Catch": [
        { "ErrorEquals": ["DynamoDB.ConditionalCheckFailedException"], "ResultPath": null, "Next": "MarkRejected" },
        { "ErrorEquals": ["States.ALL"], "ResultPath": "$.compensationError", "Next": "EscalateCompensationFailure" }
      ],
      "Next": "MarkRejected"
    },

    "RejectInsufficientFunds": { "Type": "Pass", "Result": "INSUFFICIENT_FUNDS", "ResultPath": "$.rejectionReason", "Next": "MarkRejected" },

    "MarkRejected": {
      "Type": "Task",
      "Resource": "arn:aws:states:::dynamodb:updateItem",
      "Parameters": {
        "TableName": "${PaymentsTable}",
        "Key": { "paymentId": { "S.$": "$.paymentId" } },
        "UpdateExpression": "SET #s = :s",
        "ExpressionAttributeNames": { "#s": "status" },
        "ExpressionAttributeValues": { ":s": { "S": "rejected" } }
      },
      "ResultPath": null,
      "Retry": [ { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 10, "BackoffRate": 2.0, "MaxDelaySeconds": 300, "JitterStrategy": "FULL" } ],
      "Next": "PublishRejected"
    },

    "PublishRejected": {
      "Type": "Task",
      "Resource": "arn:aws:states:::events:putEvents",
      "Parameters": {
        "Entries": [ {
          "EventBusName": "${EventBusName}",
          "Source": "harbourline.payments",
          "DetailType": "PaymentRejected",
          "Detail": { "paymentId.$": "$.paymentId", "debtorAccountId.$": "$.debtorAccountId" }
        } ]
      },
      "ResultPath": null,
      "Retry": [ { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 10, "BackoffRate": 2.0, "MaxDelaySeconds": 300, "JitterStrategy": "FULL" } ],
      "Next": "PaymentRejected"
    },
    "PaymentRejected": { "Type": "Fail", "Error": "Harbourline.PaymentRejected", "Cause": "See execution history for the reason" },

    "EscalateCompensationFailure": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sqs:sendMessage",
      "Parameters": {
        "QueueUrl": "${ManualQueueUrl}",
        "MessageBody": { "reason": "COMPENSATION_FAILED", "paymentId.$": "$.paymentId", "executionArn.$": "$$.Execution.Id" }
      },
      "ResultPath": null,
      "Next": "CompensationFailed"
    },
    "CompensationFailed": { "Type": "Fail", "Error": "Harbourline.CompensationFailed", "Cause": "Funds may still be held; sent to operations" },

    "EscalatePostFailure": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sqs:sendMessage",
      "Parameters": {
        "QueueUrl": "${ManualQueueUrl}",
        "MessageBody": { "reason": "POST_AFTER_PIVOT_FAILED", "paymentId.$": "$.paymentId", "executionArn.$": "$$.Execution.Id" }
      },
      "ResultPath": null,
      "Next": "PostFailed"
    },
    "PostFailed": { "Type": "Fail", "Error": "Harbourline.PostAfterPivotFailed", "Cause": "Money sent but not posted; sent to operations" }
  }
}

The .NET Lambda steps

Steps.cs
using Amazon.Lambda.Core;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace Harbourline.Payments.Saga;

/// <summary>Saga input. Amount is a string so it never passes through binary floating point.</summary>
public sealed record PaymentCommand(
    string PaymentId, string DebtorAccountId, string CreditorName, string CreditorIban,
    string Amount, string Currency, string Scheme, string EndToEndId);

public sealed record ScreeningResult(string ScreeningRef);
public sealed record SubmissionResult(string SchemeRef, string Outcome);

// Exception class names become the Step Functions error names matched in ErrorEquals.
public sealed class SanctionsHitException(string message) : Exception(message);
public sealed class TransientProviderException(string message) : Exception(message);
public sealed class SchemeRejectedException(string message) : Exception(message);

public interface ISchemeConnector
{
    /// <summary>Must be idempotent on endToEndId: a repeat returns the original outcome.</summary>
    Task<SubmissionResult> SubmitAsync(PaymentCommand payment, CancellationToken ct);
}

public sealed class Steps
{
    private readonly ISchemeConnector _scheme;

    public Steps() : this(new FakeSchemeConnector()) { }
    public Steps(ISchemeConnector scheme) => _scheme = scheme;

    public Task<ScreeningResult> ScreenPayee(PaymentCommand payment, ILambdaContext context)
    {
        context.Logger.LogInformation($"screening payment={payment.PaymentId}");
        // Placeholder rule for the lab; a real adapter calls the screening provider.
        if (payment.CreditorName.Contains("BLOCKED", StringComparison.OrdinalIgnoreCase))
            throw new SanctionsHitException($"Potential match for payment {payment.PaymentId}");
        return Task.FromResult(new ScreeningResult($"SCR-{payment.PaymentId[4..]}"));
    }

    public async Task<SubmissionResult> SubmitToScheme(PaymentCommand payment, ILambdaContext context)
    {
        // Leave a margin so we fail cleanly before Lambda kills us mid-call.
        using var cts = new CancellationTokenSource(context.RemainingTime - TimeSpan.FromSeconds(2));
        var result = await _scheme.SubmitAsync(payment, cts.Token);
        if (result.Outcome == "rejected")
            throw new SchemeRejectedException($"Scheme rejected {payment.EndToEndId}");
        context.Logger.LogInformation($"submitted payment={payment.PaymentId} schemeRef={result.SchemeRef}");
        return result;
    }
}

/// <summary>Lab stand-in for the Faster Payments connector. Rejects amounts over £25,000.00.</summary>
public sealed class FakeSchemeConnector : ISchemeConnector
{
    public Task<SubmissionResult> SubmitAsync(PaymentCommand payment, CancellationToken ct)
    {
        var amount = decimal.Parse(payment.Amount, System.Globalization.CultureInfo.InvariantCulture);
        return Task.FromResult(amount > 25_000.00m
            ? new SubmissionResult($"FPS-{payment.EndToEndId[4..]}", "rejected")
            : new SubmissionResult($"FPS-{payment.EndToEndId[4..]}", "accepted"));
    }
}

Unit tests (no AWS needed)

tests/StepsTests.cs
using Amazon.Lambda.TestUtilities;
using Harbourline.Payments.Saga;
using Xunit;

public class StepsTests
{
    private static PaymentCommand Payment(string creditor = "Example Supplies Ltd", string amount = "1250.00") =>
        new("PAY-0192d1c2-5c4e-7000-8000-000000000001", "ACC-0000012345", creditor,
            "GB33BUKB20201555555555", amount, "GBP", "FPS", "E2E-0192d1c2-5c4e-7000-8000-000000000001");

    private static TestLambdaContext Ctx() => new() { RemainingTime = TimeSpan.FromSeconds(10) };

    [Fact]
    public async Task Clean_payee_is_screened()
    {
        var r = await new Steps().ScreenPayee(Payment(), Ctx());
        Assert.StartsWith("SCR-", r.ScreeningRef);
    }

    [Fact]
    public async Task Sanctions_match_throws_named_business_error() =>
        await Assert.ThrowsAsync<SanctionsHitException>(() => new Steps().ScreenPayee(Payment("BLOCKED Trading Co"), Ctx()));

    [Fact]
    public async Task Scheme_rejection_is_a_named_error_not_a_generic_failure() =>
        await Assert.ThrowsAsync<SchemeRejectedException>(() => new Steps().SubmitToScheme(Payment(amount: "30000.00"), Ctx()));

    [Fact]
    public async Task Resubmission_with_same_end_to_end_id_returns_same_reference()
    {
        var steps = new Steps();
        var a = await steps.SubmitToScheme(Payment(), Ctx());
        var b = await steps.SubmitToScheme(Payment(), Ctx());
        Assert.Equal(a.SchemeRef, b.SchemeRef);
    }
}

The tests use Amazon.Lambda.TestUtilities for TestLambdaContext, and xunit.v3 4.0.3 on Microsoft.Testing.Platform. To test the state machine without deploying it, use the Step Functions TestState API (aws stepfunctions test-state), which runs a single state with a given input and shows the output, retries and next state ⚠ VERIFY pricing and quotas. Or use the Step Functions Local tooling.

Code explained line by line

  1. ReserveFunds with arn:aws:states:::dynamodb:updateItem — a direct service integration: no Lambda, no code. The update reduces available and records a hold keyed by payment ID in the holds map. This is the semantic lock.
  2. ConditionExpression: available >= :amt AND attribute_not_exists(holds.#p) — atomic funds check and idempotency. If the state is re-run, the existing hold makes the condition fail instead of reserving twice.
  3. "#p.$": "$.paymentId" — the .$ suffix tells Step Functions to take the value from the input. That's how a map key becomes the payment ID.
  4. "N.$": "$.amount" — DynamoDB numbers are strings in the API, so the string amount flows straight through without floating-point conversion.
  5. Catch on DynamoDB.ConditionalCheckFailedException — for the reservation, a failed condition means insufficient funds (or a duplicate). It routes to a clean rejection, not to a retry.
  6. ResultPath: null — discards the task result and passes the original input on unchanged. This keeps the saga's payload small and predictable.
  7. ResultSelector on Lambda tasks — picks only the fields you need from the Lambda response ($.Payload.…), so step outputs don't bloat the execution state.
  8. Two retriers on ScreenPayee — Lambda service errors (retry quickly) and TransientProviderException (retry slower). JitterStrategy: FULL is opt-in. SanctionsHitException isn't in any retrier, so it goes straight to Catch.
  9. SubmitToScheme — the pivot. It retries only Lambda service errors, where the function never ran. A function timeout or unknown error goes to OutcomeUnknown, not to compensation. The money may have left, and releasing the hold could let the customer spend it twice.
  10. OutcomeUnknown → SQS → Fail — a person (or an automated reconciler) queries the scheme by EndToEndId and completes the saga manually. In a fuller design, use .waitForTaskToken to pause the saga until the reconciler calls SendTaskSuccess or SendTaskFailure.
  11. PostDebit — retriable after the pivot. Up to 10 attempts with capped backoff. A ConditionalCheckFailedException means the hold is already gone (a previous attempt succeeded), so it is treated as done and routes to MarkSettled.
  12. ReleaseFunds — the compensation. It is idempotent by the same trick: only acts if the hold exists. It is retried generously. If it still fails, EscalateCompensationFailure makes sure a person knows money is stuck.
  13. $$.Execution.Id — the $$ prefix reads the context object. The execution ARN goes into escalation messages so operations can open the exact execution.
  14. Exception class names as error names — when a .NET Lambda function throws, the error type reported to Step Functions is the exception's type name (for example SchemeRejectedException), which is what ErrorEquals matches ⚠ VERIFY the exact format (simple vs fully qualified name) for your runtime by checking an execution's error output.
  15. context.RemainingTime - 2 s — the scheme call is cancelled before Lambda's own timeout. The step then fails with a clear error instead of a hard timeout mid-request.
  16. ISchemeConnector.SubmitAsync must be idempotent on endToEndId — even with a careful state machine, an invocation might be repeated, for example on a Lambda service error after the request actually reached the function. The ISO 20022 EndToEndId is the natural idempotency key that schemes and connectors recognise.

Security considerations

  • Don't log execution data. IncludeExecutionData: false keeps names, IBANs and amounts out of CloudWatch Logs. The execution history in Step Functions still contains them, so restrict states:DescribeExecution and states:GetExecutionHistory to the operations role.
  • Least privilege for the state machine role: invoke only the two functions, dynamodb:UpdateItem only on the two tables, send only to the manual queue, and put events only on the payments bus. Avoid console-generated roles with wildcards in production.
  • Who can start executions? Only the Payments API role should have states:StartExecution on this state machine. Anyone who can start it can move money.
  • Segregation of duties for manual resolution: the people who resolve PIVOT_OUTCOME_UNKNOWN cases should not be the people who can change the state machine definition.
  • Encrypt the tables, the manual queue (it holds payment IDs) and the logs. Use customer managed KMS keys where your policy requires it.

⚠ Learning shortcut — not for production

The sanctions step is a string match and the scheme connector is a fake. A real screening step calls a provider and may return "potential match — needs review", which is a wait (callback pattern with .waitForTaskToken), not a rejection. Real scheme submissions are asynchronous: you submit, then wait for an acknowledgement message. Model that as a callback task with HeartbeatSeconds and a timeout, and send the timeout to OutcomeUnknown. Also add a reconciliation job that compares holds older than N minutes against saga states, because a saga is a process, and processes need independent checks.

Banking use case

The Harbourline design review for this saga produced a compensation table, now required for every saga:

StepTypeCompensationCompensation idempotencyIf compensation fails
Reserve fundsCompensatableRelease holdConditional on hold existingRetry 10× with backoff, then operations queue (funds stuck)
Screen payeeCompensatable (no state to undo)None——
Submit to schemePivotNone. A clear rejection triggers compensation of earlier steps; an unknown outcome triggers investigationScheme deduplicates on EndToEndId—
Post debitRetriable—Conditional on hold existingRetry 10×, then operations queue (money sent, not posted)
Mark settled, publish eventRetriable—Upsert / event consumers idempotentRetry 10×

Two further rules came from the risk team:

  • Each escalation reason maps to a runbook and a service-level target. Funds held for a rejected payment must be released within 4 business hours, so customers aren't locked out of their money.
  • Reconciliation runs every 15 minutes. Holds older than 30 minutes without a running execution raise an alert.

These are Harbourline's internal targets. Any regulatory or contractual timelines need confirming with the compliance team.

Common mistakes

  • Compensating after the pivot → releasing funds for a payment that actually went → double spend → classify steps; send ambiguous pivot outcomes to investigation, not compensation.
  • Non-idempotent compensations → a retried release adds the money back twice → condition every compensation on the existence of what it undoes.
  • Compensations that can fail permanently with no escalation → money silently stuck → generous retries, then a manual queue with an alarm and a runbook.
  • Retrying business errors → SanctionsHitException retried three times, delaying the customer's rejection → only transient errors in Retry; business errors in Catch.
  • Using Express workflows for the payment saga → at-least-once async execution can submit twice; 5-minute limit → Standard for money movement.
  • Passing whole documents through the state → payload limits and personal data in execution history → pass IDs; read details inside the steps.
  • Choreographing the money → compensation logic spread across five services, no single place that shows the payment's state → orchestrate money, choreograph side effects.
  • Mutating the booked balance at reservation time → statements and interest calculations see uncommitted debits → semantic lock (available vs balance).

Troubleshooting

SymptomLikely causeFix
States.Runtime error: "JSONPath ... could not be found"A field referenced with .$ isn't in the state input (often lost because ResultPath overwrote the input)Use ResultPath to put results under a sub-path, or null to discard
Catch never fires for a business errorErrorEquals doesn't match the error name the Lambda reportedOpen the failed state's output in the console; copy the exact Error value
ExecutionAlreadyExists on startAn execution with this name exists with different input, or a finished one is still within history retentionFor retries of the same payment, treat as "already started"; never reuse a payment ID for a new payment
AccessDeniedException on dynamodb:UpdateItemState machine role missing the action or table ARNAdd exactly that action on that table
DynamoDB.ValidationException about the update expressionMap attribute holds doesn't exist on the account itemSeed accounts with "holds":{"M":{}}, or use if_not_exists
Saga stuck in SubmitToScheme for minutesLambda timeout longer than expected; no state TimeoutSecondsSet Lambda and state timeouts; route timeouts to OutcomeUnknown
Costs higher than expectedLarge MaxAttempts on frequently failing states; each retry is a billed transitionFix the failing dependency; cap attempts; add a circuit breaker upstream

Cost and cleanup

ItemCost at lab volume
Step Functions StandardBilled per state transition; the free tier includes a monthly allowance of transitions ⚠ VERIFY current allowance and price. Each saga run here is about 6–10 transitions
LambdaWithin the always-free tier
DynamoDB on-demandFractions of a cent
CloudWatch LogsSmall ingestion and storage; one-month retention set
EventBridge custom eventsNegligible at lab volume

Free alternative: run the unit tests, and use aws stepfunctions test-state against individual states, or a local Step Functions emulator. The design work (the compensation table) is the most valuable part, and it's free.

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
# CDK: cdk destroy PaymentSagaStack     Terraform: terraform destroy
# Manual cleanup if you created resources by hand:
aws stepfunctions delete-state-machine --state-machine-arn "$SM_ARN"
aws dynamodb delete-table --table-name hl-m08-accounts
aws dynamodb delete-table --table-name hl-m08-payments
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m08-saga-manual --query QueueUrl --output text)"
aws events delete-event-bus --name hl-m08-payments
aws logs delete-log-group --log-group-name /aws/vendedlogs/states/hl-m08-payment-saga

aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m08-saga \
  --query "ResourceTagMappingList[].ResourceARN"

Exercises

  • 🟢 Guided: Run the four scenarios (happy path, sanctions hit, scheme rejection over £25,000.00, insufficient funds with £9,000.00). For each, list the states entered and the final values of available, balance and holds on the account.
  • 🟡 Challenge: Replace OutcomeUnknown with a callback: a Task with the resource arn:aws:states:::sqs:sendMessage.waitForTaskToken that sends the task token to a reconciler queue, waits up to 4 hours (TimeoutSeconds), and continues to PostDebit or ReleaseFunds depending on the reconciler's answer. Write the small .NET reconciler that calls SendTaskSuccess.
  • 🔴 Live problem: The card team wants to build "buy now, pay later" as a choreographed saga across Cards, Credit Decisioning, Merchant Settlement and Notifications, all reacting to EventBridge events. They say "Step Functions is a single point of failure". Prepare your review comments.

Assignment

Add to your portfolio repository:

  • the ASL definition;
  • the compensation table for the payment saga;
  • the unit-tested step functions;
  • a Mermaid diagram of the happy and failure paths.

Then write a one-page saga design template that your team could use for any new multi-service flow. It should cover steps, classification, compensations, idempotency keys, escalation and reconciliation.

Solutions

Knowledge check

Interview questions

  • "Orchestration or choreography?" — Orchestrate flows that need a single source of truth and strict sequencing, such as money movement. Choreograph independent reactions. A common hybrid is an orchestrated core that publishes events for choreographed side effects.
  • "How do you handle a timeout when calling a payment scheme?" — Treat it as an unknown outcome, not a failure. Don't compensate. Query the status by the idempotency key (EndToEndId), or wait for the asynchronous acknowledgement with a callback task. Escalate if still unknown, and reconcile independently.
  • "What makes a good compensating action?" — It is semantically correct (a release or a reversal, not a delete), idempotent, retried until it succeeds, logged for audit, and escalated if it can't complete.
  • "How would you test a saga?" — Unit-test each step's business decisions. Use TestState for individual states, including retry and catch routing. Run integration tests for every failure path (one per row of the compensation table). Inject faults in a test environment. Reconcile in production.

Certification practice

Completion checklist

  • I can explain why 2PC isn't available and what a saga does instead
  • I can classify steps as compensatable, pivot or retriable for a given flow
  • I deployed or read through the payment saga and can follow every failure path
  • My compensations are idempotent and escalate when they can't complete
  • I filled in a compensation table for the payment saga
  • I deleted the state machine, tables, queue, bus and log group

Key takeaways

  • A saga = local transactions + compensating transactions, coordinated by orchestration or choreography. It moves forward to a correct state; it doesn't roll back.
  • Order steps compensatable → pivot → retriable. Never compensate after the pivot. Treat an unknown pivot outcome as "investigate", not "failed".
  • Compensations must be idempotent and must eventually succeed. Escalate to people when they can't.
  • Use Step Functions Standard for money movement: durable state, exactly-once execution, Retry with jitter, Catch to compensation, and a full history.
  • Handle missing isolation with semantic locks (holds and available), and back every saga with reconciliation.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page