Skip to Content
Cloud Engineering Academy
14. Advanced Banking Architecture

API orchestration: Step Functions or in-service

Decide when a multi-step banking operation should be orchestrated by AWS Step Functions and when by code inside the service, design timeouts, retries and compensation, and run a C# saga runner and an Amazon States Language definition.

advanced⏱ 2 h 30 min🟢 FreeSAP-C03SAA-C03DVA-C03

Assumes: .NET 10 LTS (file-based apps) · AWS SDK for .NET v4 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Compare orchestration (a coordinator tells each step what to do) with choreography (services react to each other's events) and choose per use case.
  • Decide between Step Functions Standard, Step Functions Express and in-service orchestration, using latency, duration, audit and cost as criteria.
  • Design per-step timeouts, bounded retries and compensation for a payment-style flow.
  • Write an Amazon States Language (ASL) definition with Retry, Catch and TimeoutSeconds.
  • Run a C# saga runner that times out a step, retries with backoff and compensates in reverse order.
  • Know the Step Functions limits that change designs: payload size, history size, execution duration and start rates.

Prerequisites

Why this matters in banking

A transfer is not one action. It validates, reserves funds, screens, submits to a scheme, books and notifies. Each step can fail, time out or succeed twice. Someone must own the question "what happens next?" in every situation, and an auditor must be able to see what did happen.

If that logic is scattered across services with ad hoc retries, you get stuck payments, double postings and a support team reading logs to find out where a payment is. If it is centralised without care, you get a single orchestrator that becomes the bank's most fragile component. Choosing where orchestration lives is a first-order architecture decision, and the design authority will ask you to defend it.

Educational content, not regulatory advice

Timeouts, hold policies and retry limits for payments are risk decisions as well as technical ones. Sanctions-screening behaviour, scheme rules and customer-communication obligations need review by payments operations, financial-crime compliance and risk ⚠ VERIFY. Figures here are illustrative.

The idea in plain language

Orchestration: one coordinator holds the process definition. It calls step A, waits, calls step B, and handles errors. You can read the process in one place.

Choreography: no coordinator. Service A emits "funds reserved"; service B reacts and emits "screening passed"; and so on. Loosely coupled, but the process exists only implicitly, spread across subscriptions (event-driven architecture).

AspectOrchestrationChoreography
Visibility of the processOne definition; one execution historyReconstructed from events
CouplingOrchestrator knows all participantsParticipants know only events
Changing the flowEdit one definitionTouch several services
Failure handlingCentral Retry/Catch/compensationEach service handles its own, plus timeouts for "nothing happened"
Best forMoney movement, regulated approvals, long waitsNotifications, analytics, independent reactions

Most banks use both: orchestrate the critical money path, choreograph the reactions (notifications, reporting, fraud analytics).

Analogy: an orchestra and a jazz session. A symphony needs a conductor so that a payment's steps happen in a defined order. Jazz works when musicians listen and react, which suits the "tell the customer, update the dashboard" side.

Three places to orchestrate

OptionWhat it isStrengthsWeaknesses
Step Functions StandardManaged workflow, up to one year, execution history kept for 90 days by defaultDurable, visual, built-in retry and wait, task tokens for approval, many AWS integrationsPer-transition cost; start-rate and payload limits; latency per transition
Step Functions ExpressHigh-volume, short (up to 5 minutes) workflows, sync or asyncCheap at scale, low latency, synchronous startNo execution history unless logging is configured; at-least-once for async
In-service codeA C# class in your service (as in the runner below)Fast, testable with xUnit, full language powerYou must persist state for crash recovery, and build your own visibility

What problem the AWS services solve

Step Functions provides durable state, retries with backoff, timeouts, parallelism, waiting for human approval, and direct integrations with AWS services. It does not decide your business rules, make your steps idempotent or choose what compensation means.

Limits that change designs (checked against the Step Functions quota page on 2026-11-18; re-verify before relying on them):

LimitValueDesign consequence
Max execution timeStandard 1 year; Express 5 minutesLong approvals need Standard
Max input or output per state256 KiBPass references to data in S3 or a table, never the data
Execution history events25,000 for StandardLong loops must start child executions
Execution history retention90 days after close (Standard)Copy audit-relevant facts to your own store
StartExecution rate, Standard, us-east-1Bucket 1,300, refill 300 per second (soft quota)Burst control and quota requests for peak payment starts
RedriveStandard only, within 14 daysFailed executions can resume from the failed state

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Durable Functions (orchestrator code)Step Functions (declarative JSON) or in-service C#Step Functions is declarative; Durable Functions replays your C# code
Logic AppsStep FunctionsBoth visual; Step Functions state is explicit JSON between states
Azure Service Bus sessions for orderingSQS FIFO groups, Step Functions per paymentOrchestrator per payment instance instead of per session
Polly retry and circuit breakerStep Functions Retry, plus Polly in codeRetry config is data, not code
Saga libraries (MassTransit, NServiceBus sagas)Step Functions or a library in-serviceSame saga idea; Step Functions persists for you

Architecture

Security-boundary view

The API role may only start the payment state machine. The state machine's own role may only invoke the named step functions. Nothing has broad permissions.

The 15-section design

Sections 1 to 3: requirements

SectionContent
1 BusinessMake every payment's progress visible and every failure recoverable without manual database edits; cut stuck-payment tickets by 80 percent (fictional)
2 FunctionalF1 Start a saga per payment, once. F2 Run steps in order. F3 Time out and retry each step. F4 Hold on screening uncertainty. F5 Compensate on scheme rejection. F6 Emit status events. F7 Allow redrive or manual resolution
3 Non-functionalMedian end-to-end orchestration overhead under 300 ms. 99.95% of started payments reach a terminal or held state within 5 minutes. 300 payments per second at peak. Audit: who, what, when for each step for 7 years ⚠ VERIFY. No double reservation or double submission

Section 5: service selection and the decision

ChoiceWhyRejected alternativeTrade-off
Step Functions Standard for the payment sagaDurable, visual, redrive, history for auditExpressHigher per-step cost; the saga includes a possible human hold, which exceeds Express limits
API invokes Step Functions asynchronously and returns 202Keeps API latency stableSync Express under the APIClient polls or receives an event for the final result
Lambda or ECS tasks per stepIsolated, individually scalableOne monolithic stepMore units to deploy
In-service orchestration for the hot validation pathSub-50 ms, pure CPUStep FunctionsNot durable; acceptable because it is stateless and repeatable
Choreography for notificationsLoose couplingOrchestrated notification stepsProcess is implicit; keep subscriptions documented

Decision rules. Use Step Functions when the flow lasts longer than a request, needs human approval, needs an audit-visible execution history, or has several AWS integrations. Use in-service code when the flow is short, pure and latency-critical, or logic is easier to test in C#. Use Express for high-volume short workflows where logging is enough. Use choreography for independent reactions.

Sections 6 and 7: network and security

  • Network. Step Functions is a regional service reached over the AWS network. Task functions can run in VPC private subnets when they must reach private resources; use VPC endpoints for AWS APIs.
  • Identity. One execution role per state machine; task-level least privilege; the caller can start only the intended state machine.
  • Data in executions. Execution history stores state input and output. Never put card numbers, full names or identifiers beyond what is needed into state; pass paymentId and look up data in a protected store. Encrypt with a customer-managed key where supported ⚠ VERIFY current options.
  • Logging. Enable execution logging to CloudWatch Logs at an appropriate level, and exclude execution data from logs if it could contain personal data.
  • Tracing. Turn on X-Ray or OpenTelemetry-based tracing consistent with your observability standard (Module 11).

Sections 8 to 10: API, data and flow

  • API. POST /payments with Idempotency-Key; returns 202 and Location: /payments/{id}. GET /payments/{id} returns status (RECEIVED, RESERVED, SCREENING, SUBMITTED, BOOKED, HELD, FAILED).
  • Idempotent start. Use the payment ID as the execution name. Starting a Standard execution with the same name and the same input while it is open does not start a second one; verify the exact semantics for closed executions in the current documentation ⚠ VERIFY.
  • Database. A payments table stores the authoritative status; the execution history is a diagnostic record, not the system of record.
  • Data flow. API stores payment, starts execution, returns 202. Each step reads and writes the payments table by ID. The final state publishes an EventBridge event.

Section 11: failure scenarios

FailureDetectionAutomatic responseImpactManual step
Screening step hangsTimeoutSeconds (5 s)Retry twice with backoff; then Catch to HoldPayment heldOperations review
Transient Lambda errorLambda.ServiceException etc.Retry with exponential backoffNoneNone
Scheme rejectionError from SubmitCompensate: release reservationPayment failedCustomer notified
Compensation failsRelease step error after retriesAlarm; execution fails visiblyFunds stuck reservedOperations repair, then redrive
Execution start throttledThrottlingException or ExecutionLimitExceededClient retries with jitter; queue in front of start if burstySlower acceptanceRequest quota increase
Payload exceeds 256 KiBStates.DataLimitExceededExecution failsFailed paymentFix design to pass references
Deployment during in-flight executionsNone neededRunning executions keep their definition versionNoneUse versions and aliases for staged roll-out
Approver never respondsTask token timeoutTimeout error to catchHeld longerEscalation

Sections 12 to 15

  • Scalability. At 300 payments per second, check StartExecution throttling (Standard bucket and refill in us-east-1 as above ⚠ VERIFY), the state-transition throttle (5,000 per second bucket in us-east-1), and downstream Lambda or ECS concurrency. Smooth bursts with an SQS queue in front of StartExecution.
  • Cost. Standard pricing is per state transition ⚠ VERIFY the current price. Worked example with an assumed price of 0.025 USD per 1,000 transitions: a saga with 10 transitions at 10 million payments a month is 100 million transitions, about 2,500 USD. The same flow on Express is billed by requests and duration and may be cheaper at that volume ⚠ VERIFY. Always compute for your own transition count.
  • Implementation. Phase 1 happy path with stubs. Phase 2 idempotent steps and compensation. Phase 3 failure injection (timeouts, errors). Phase 4 quotas and load test. Phase 5 game day with operations.
  • Compliance touchpoints.
AreaWhere it touchesReviewing teamEvidence
Payment screening hold policyCatch to Hold behaviourFinancial-crime complianceSigned policy, test results
Auditability of money movementExecution history, payments tableInternal auditRetention approach, sample trace
Data protectionData in state and logsData protection officerLog-data exclusion, key policy
Operational resilienceImpact tolerance and timeoutsOperational resilience leadScenario tests
Change controlVersioned definitionsChange advisory boardRelease records

Step-by-step implementation

Draw the state machine on paper first

List steps, which can fail, what each failure means, and what compensates what. A compensation is a business operation (release a reservation), not a rollback.

Make every step idempotent

Each step receives paymentId and uses it as the idempotency key. Running it twice must have the same effect as once (idempotency keys).

Write the ASL definition with explicit timeouts

Every Task gets TimeoutSeconds. A task without a timeout can wait for a very long time, because the default is effectively the execution limit.

Add Retry for transient errors and Catch for business failures

Retry network and service errors with backoff; never retry a business rejection. Route States.Timeout and States.TaskFailed to the correct compensation or hold state.

Deploy with IaC and invoke with an idempotent name

Use the payment ID as the execution name and test that a second start does not create a second execution.

Inject failures and watch the execution graph

Make Screen sleep longer than its timeout, make Submit return an error, and confirm the console shows the path you designed.

AWS Console instructions

  • Console → Step Functions → State machines → Create state machine. Choose Standard, use Workflow Studio to draw or paste the definition, set the execution role, enable logging and tracing.
  • Start execution with input {"paymentId":"PAY-0001","amount":"25.00","currency":"GBP"} and a name such as PAY-0001.
  • Executions → select one → Graph view to see which state failed, with each state's input and output. Redrive appears on failed Standard executions.

AWS CLI

# Start one execution; the name makes the start idempotent for the same payment.
aws stepfunctions start-execution \
  --state-machine-arn arn:aws:states:us-east-1:111122223333:stateMachine:hl-m14-payment-saga \
  --name PAY-0001 \
  --input '{"paymentId":"PAY-0001","amount":"25.00","currency":"GBP"}' \
  --profile learn-sandbox --region us-east-1

# See its status and output.
aws stepfunctions describe-execution \
  --execution-arn arn:aws:states:us-east-1:111122223333:execution:hl-m14-payment-saga:PAY-0001 \
  --profile learn-sandbox --region us-east-1

# Read the step-by-step history (what auditors and operators look at).
aws stepfunctions get-execution-history \
  --execution-arn arn:aws:states:us-east-1:111122223333:execution:hl-m14-payment-saga:PAY-0001 \
  --profile learn-sandbox --region us-east-1

# Resume a failed Standard execution from the failed state (within the redrive period).
aws stepfunctions redrive-execution \
  --execution-arn arn:aws:states:us-east-1:111122223333:execution:hl-m14-payment-saga:PAY-0001 \
  --profile learn-sandbox --region us-east-1

Account IDs shown are fictional placeholders.

Infrastructure as Code

The same ASL file serves both tools. It uses ${...} placeholders that CDK fills through DefinitionSubstitutions and Terraform fills through templatefile.

payment-saga.asl.json
{
  "Comment": "Learning payment saga: validate, reserve, screen, submit, with compensation",
  "StartAt": "Validate",
  "TimeoutSeconds": 600,
  "States": {
    "Validate": {
      "Type": "Task",
      "Resource": "${ValidateFn}",
      "TimeoutSeconds": 10,
      "Retry": [
        { "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException", "States.Timeout"],
          "IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2.0 }
      ],
      "Catch": [ { "ErrorEquals": ["States.ALL"], "Next": "Failed", "ResultPath": "$.error" } ],
      "Next": "ReserveFunds"
    },
    "ReserveFunds": {
      "Type": "Task",
      "Resource": "${ReserveFn}",
      "TimeoutSeconds": 10,
      "Retry": [
        { "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException", "States.Timeout"],
          "IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2.0 }
      ],
      "Catch": [ { "ErrorEquals": ["States.ALL"], "Next": "Failed", "ResultPath": "$.error" } ],
      "Next": "ScreenSanctions"
    },
    "ScreenSanctions": {
      "Type": "Task",
      "Resource": "${ScreenFn}",
      "TimeoutSeconds": 5,
      "Retry": [
        { "ErrorEquals": ["States.Timeout", "Lambda.ServiceException"],
          "IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2.0 }
      ],
      "Catch": [ { "ErrorEquals": ["States.ALL"], "Next": "HoldForReview", "ResultPath": "$.error" } ],
      "Next": "SubmitToScheme"
    },
    "SubmitToScheme": {
      "Type": "Task",
      "Resource": "${SubmitFn}",
      "TimeoutSeconds": 15,
      "Retry": [
        { "ErrorEquals": ["Lambda.ServiceException", "States.Timeout"],
          "IntervalSeconds": 2, "MaxAttempts": 2, "BackoffRate": 2.0 }
      ],
      "Catch": [ { "ErrorEquals": ["States.ALL"], "Next": "ReleaseFunds", "ResultPath": "$.error" } ],
      "Next": "Booked"
    },
    "ReleaseFunds": {
      "Type": "Task",
      "Resource": "${ReleaseFn}",
      "TimeoutSeconds": 10,
      "Retry": [
        { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 5, "BackoffRate": 2.0 }
      ],
      "Next": "Failed"
    },
    "HoldForReview": {
      "Type": "Task",
      "Resource": "${HoldFn}",
      "TimeoutSeconds": 10,
      "Retry": [
        { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 5, "BackoffRate": 2.0 }
      ],
      "Next": "Held"
    },
    "Held": { "Type": "Succeed", "Comment": "Payment parked with status HELD for operations review" },
    "Booked": { "Type": "Succeed" },
    "Failed": { "Type": "Fail", "Error": "PaymentFailed", "Cause": "See $.error and the payments table" }
  }
}
PaymentSagaStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.StepFunctions;
using Constructs;

namespace Harbourline.Architecture.Orchestration;

public class PaymentSagaStack : Stack
{
    public PaymentSagaStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        // Stub steps. In a real design these are .NET functions or ECS tasks.
        Function Step(string name) => new(this, name, new FunctionProps
        {
            Runtime = Runtime.PYTHON_3_13,
            Handler = "index.handler",
            Code = Code.FromInline("def handler(event, context):\n    return event\n"),
            Timeout = Duration.Seconds(10),
        });

        var validate = Step("Validate");
        var reserve = Step("Reserve");
        var screen = Step("Screen");
        var submit = Step("Submit");
        var release = Step("Release");
        var hold = Step("Hold");

        var logs = new LogGroup(this, "SagaLogs", new LogGroupProps
        {
            Retention = RetentionDays.ONE_MONTH,
            RemovalPolicy = RemovalPolicy.DESTROY,
        });

        var machine = new StateMachine(this, "PaymentSaga", new StateMachineProps
        {
            StateMachineName = "hl-m14-payment-saga",
            StateMachineType = StateMachineType.STANDARD,
            DefinitionBody = DefinitionBody.FromString(File.ReadAllText("payment-saga.asl.json")),
            DefinitionSubstitutions = new Dictionary<string, string>
            {
                ["ValidateFn"] = validate.FunctionArn,
                ["ReserveFn"] = reserve.FunctionArn,
                ["ScreenFn"] = screen.FunctionArn,
                ["SubmitFn"] = submit.FunctionArn,
                ["ReleaseFn"] = release.FunctionArn,
                ["HoldFn"] = hold.FunctionArn,
            },
            Timeout = Duration.Minutes(10),
            TracingEnabled = true,
            Logs = new LogOptions { Destination = logs, Level = LogLevel.ERROR, IncludeExecutionData = false },
        });

        foreach (var fn in new[] { validate, reserve, screen, submit, release, hold })
            fn.GrantInvoke(machine);

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m14-l13");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        Amazon.CDK.Tags.Of(this).Add("Owner", "learner");
    }
}

IncludeExecutionData = false keeps state input and output out of CloudWatch Logs, which matters when a payload could hold personal data. LogLevel here is the Step Functions enum, not the .NET logging one.

Complete code

When the flow is short and you want the logic in C#, a small in-service runner gives you the same ideas: per-step timeout, bounded retries with backoff, compensation in reverse order, and an explicit "needs a human" outcome when compensation itself fails.

SagaRunner.cs
// Run with: dotnet run SagaRunner.cs
// A tiny in-service orchestrator: per-step timeout, bounded retries with backoff,
// compensation in reverse order, and an explicit "needs a human" outcome.
var log = new List<string>();
var attemptsAtSubmit = 0;

SagaStep[] happyButFlakyPath =
[
    new("ReserveFunds", _ => Note("reserve"), _ => Note("release reservation"), TimeSpan.FromSeconds(1), 1),
    new("Screen", _ => Note("screen ok"), null, TimeSpan.FromSeconds(1), 2),
    new("Submit", _ => ++attemptsAtSubmit < 2 ? throw new HttpRequestException("scheme 503") : Note("submitted"),
        _ => Note("recall from scheme"), TimeSpan.FromSeconds(1), 3),
];
Console.WriteLine("Scenario A: " + await SagaRunner.RunAsync(happyButFlakyPath, log));

log.Clear();
SagaStep[] screeningHangs =
[
    new("ReserveFunds", _ => Note("reserve"), _ => Note("release reservation"), TimeSpan.FromSeconds(1), 1),
    new("Screen", ct => Task.Delay(Timeout.Infinite, ct), null, TimeSpan.FromMilliseconds(100), 2),
    new("Submit", _ => Note("submitted"), _ => Note("recall from scheme"), TimeSpan.FromSeconds(1), 3),
];
Console.WriteLine("Scenario B: " + await SagaRunner.RunAsync(screeningHangs, log));
foreach (var line in log) Console.WriteLine("   " + line);
return 0;

Task Note(string text) { log.Add(text); return Task.CompletedTask; }

public enum SagaOutcome { Completed, Compensated, NeedsManualIntervention }

public sealed record SagaStep(string Name, Func<CancellationToken, Task> Do, Func<CancellationToken, Task>? Undo,
    TimeSpan Timeout, int MaxAttempts);

public static class SagaRunner
{
    public static async Task<SagaOutcome> RunAsync(IReadOnlyList<SagaStep> steps, List<string> log)
    {
        var done = new Stack<SagaStep>();
        foreach (var step in steps)
        {
            if (await TryAsync(step.Name, step.Do, step.Timeout, step.MaxAttempts, log)) { done.Push(step); continue; }

            log.Add($"{step.Name} failed permanently: compensating {done.Count} completed step(s)");
            while (done.Count > 0)
            {
                var finished = done.Pop();
                if (finished.Undo is null) continue;
                if (!await TryAsync(finished.Name + ".undo", finished.Undo, finished.Timeout, 3, log))
                {
                    log.Add($"compensation of {finished.Name} failed: raise an alarm for operations");
                    return SagaOutcome.NeedsManualIntervention;
                }
            }
            return SagaOutcome.Compensated;
        }
        return SagaOutcome.Completed;
    }

    private static async Task<bool> TryAsync(string name, Func<CancellationToken, Task> action, TimeSpan timeout,
        int maxAttempts, List<string> log)
    {
        for (var attempt = 1; attempt <= maxAttempts; attempt++)
        {
            try
            {
                using var cts = new CancellationTokenSource(timeout);
                await action(cts.Token);
                return true;
            }
            catch (Exception e) when (e is OperationCanceledException or HttpRequestException)
            {
                log.Add($"{name} attempt {attempt}/{maxAttempts}: {e.GetType().Name}");
                if (attempt < maxAttempts) await Task.Delay(TimeSpan.FromMilliseconds(10 * Math.Pow(2, attempt)));
            }
        }
        return false;
    }
}

Expected output:

Scenario A: Completed
Scenario B: Compensated
   reserve
   Screen attempt 1/2: TaskCanceledException
   Screen attempt 2/2: TaskCanceledException
   Screen failed permanently: compensating 1 completed step(s)
   release reservation

This runner holds no durable state. If the process crashes between steps, nothing remembers where it was. That is the main reason a long-running or money-moving saga normally belongs in Step Functions, or in code that persists each transition.

Code explained line by line

  1. SagaStep bundles what a step does (Do), how to undo it (Undo, optional), its timeout and its attempt limit. Putting these in data makes them reviewable, like ASL.
  2. RunAsync pushes each completed step onto a stack. A stack pops in reverse order, which is exactly the order compensations must run.
  3. When a step fails permanently, the loop pops and runs each Undo. A step with no Undo (such as Screen) is skipped: a read-only check has nothing to compensate.
  4. If a compensation itself fails after three attempts, the runner returns NeedsManualIntervention instead of pretending success. Compensation failure is a first-class outcome.
  5. TryAsync wraps each attempt in a CancellationTokenSource(timeout). A step that ignores its token cannot be cancelled, so real steps must pass the token to I/O calls.
  6. The catch ... when clause retries only transport errors and timeouts. A business rejection would be a different exception type and would not be retried.
  7. The backoff 10 * 2^attempt milliseconds doubles each time. Production code adds jitter and uses seconds, not milliseconds.
  8. In Scenario A the Submit step fails once with a 503, then succeeds on attempt 2. attemptsAtSubmit is captured by the lambda, which is how the demo makes failure deterministic.

Security considerations

⚠ Learning shortcut — not for production

The lab uses stub Lambda functions with inline code, a simple role setup and logs at ERROR level. Production steps are separately deployed, signed and scanned, each with a least-privilege role; the state machine uses versions and aliases for controlled roll-out; execution data is excluded from logs or encrypted and access-controlled; and callers are restricted to starting only the state machine they own.

  • Least privilege on both sides. Caller can states:StartExecution on one ARN. State machine role can invoke only its listed functions.
  • No sensitive data in state. State is stored in history. Pass identifiers.
  • Validate input at the start. The first state should reject malformed input before any side effect.
  • Idempotent steps are a security control. Replays and redrives must not move money twice.
  • Protect the task token. If you use callbacks for approval, treat tokens as secrets and bind them to the approver identity.

Banking use case

Harbourline's payments API starts one execution per payment, named with the payment ID. When the sanctions provider is slow, the screening step times out after 5 seconds, retries twice, then drops to HoldForReview; operations see the held payment in a queue with the execution link. When the scheme rejects a payment, the saga releases the reserved funds, marks the payment failed and the customer receives a clear message. During a game day the team deploys a new version of the saga definition while 200 payments are in flight; running executions keep the old definition, new ones use the new alias target, and nothing is lost.

Common mistakes

  • Orchestrating everything. Notifications and analytics do not need an orchestrator.
  • No timeouts. A task without TimeoutSeconds waits very long.
  • Retrying business rejections. Retrying "insufficient funds" three times does not help.
  • Using execution history as the system of record. It expires; keep status in your own table.
  • Passing big payloads. The 256 KiB limit will bite in production. Pass references.
  • Compensation that is not itself idempotent. Releasing a reservation twice must be safe.

Troubleshooting

SymptomLikely causeFix
States.DataLimitExceededPayload over 256 KiBStore data in S3 or a table and pass the key
States.Runtime on a TaskBad ResultPath or InputPath JSONPathTest with the TestState API or Workflow Studio
AccessDeniedException invoking a stepExecution role lacks lambda:InvokeFunctionGrant on the exact function ARN
Execution stuck at "Running"Waiting on a task token with no timeoutAdd TimeoutSeconds or HeartbeatSeconds
Throttled starts at peakStartExecution rate bucket exhaustedQueue in front, add jitter, request a quota increase
Second start created a duplicateDifferent execution namesUse payment ID as the name
History missing after three months90-day retentionExport audit facts to your own store

Cost and cleanup

The C# runner is free. The deploy creates five tiny Lambda functions and one Standard state machine; at lab volumes the cost is effectively zero (Step Functions and Lambda free allowances apply ⚠ VERIFY). Run the deploy briefly and destroy it.

cdk destroy --profile learn-sandbox
terraform destroy

# Confirm nothing tagged for the course remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --profile learn-sandbox --region us-east-1

Exercises

  • 🟢 Guided: run SagaRunner.cs. Add a third scenario where the Submit step always fails; predict the log lines, including which compensations run.
  • 🟡 Challenge: make the Undo of ReserveFunds fail permanently and confirm the outcome is NeedsManualIntervention. Then change the ASL so ReleaseFunds failure routes to a state that notifies operations before failing.
  • 🔴 Live problem: the design authority says "Step Functions is too expensive for 10 million payments a month". Build the cost comparison for Standard, Express and in-service orchestration with your own transition counts, and state what you would give up with each.

Assignment

Add architecture/payment-orchestration/ to your portfolio: the 15-section document, an ADR "orchestrate payments with Step Functions Standard, choreograph notifications", the ASL file deployed with CDK or Terraform, and the saga runner with unit tests covering timeout, retry and compensation failure.

Solutions

Knowledge check

Interview questions

  • Step Functions or code for a payment saga? Step Functions when you need durability, long waits, audit-visible history and redrive; code when the flow is short and latency-critical, persisting transitions if state loss is unacceptable.
  • How do you prevent double processing on retry? Idempotent steps keyed by business ID, and idempotent execution names.
  • How do you handle failure of a compensating action? Retry with backoff, then alarm and a visible failed state for operations; never swallow it.
  • What are the main Step Functions limits you design around? 256 KiB payloads, 25,000 history events, start-rate and transition throttles, 90-day history retention.
  • How do you roll out a new workflow version safely? Publish versions, shift an alias gradually, and let in-flight executions finish on their original definition.

Certification practice

Completion checklist

  • I can state when I would orchestrate and when I would choreograph
  • I ran the saga runner and explained both scenarios
  • My ASL gives every Task a timeout and separates retry from catch
  • I compared Standard, Express and in-service cost for my own transition count
  • I ran cleanup and confirmed nothing tagged Project=cea-lab remains

Key takeaways

  • Orchestrate the money path; choreograph reactions.
  • Every step is idempotent, timed out and either retried (transient) or compensated (business failure).
  • Know the limits: 256 KiB payload, 25,000 history events, 90-day history, start-rate throttles.
  • Compensation failure is a real outcome that needs an alarm and a human.
  • Hold policies and timeouts are risk decisions that need compliance and operations review.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page