API orchestration: Step Functions or in-service
Decide when a multi-step banking operation should be orchestrated by AWS Step Functions and when by code inside the service, design timeouts, retries and compensation, and run a C# saga runner and an Amazon States Language definition.
Assumes: .NET 10 LTS (file-based apps) · AWS SDK for .NET v4 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Compare orchestration (a coordinator tells each step what to do) with choreography (services react to each other's events) and choose per use case.
- Decide between Step Functions Standard, Step Functions Express and in-service orchestration, using latency, duration, audit and cost as criteria.
- Design per-step timeouts, bounded retries and compensation for a payment-style flow.
- Write an Amazon States Language (ASL) definition with
Retry,CatchandTimeoutSeconds. - Run a C# saga runner that times out a step, retries with backoff and compensates in reverse order.
- Know the Step Functions limits that change designs: payload size, history size, execution duration and start rates.
Prerequisites
- Saga pattern and Microservices communication
- Retry strategies and Idempotency in .NET
- Container APIs with VPC Link
- Core banking integration for the unknown-outcome problem
- Tools: .NET 10 SDK or Docker for the saga runner
Why this matters in banking
A transfer is not one action. It validates, reserves funds, screens, submits to a scheme, books and notifies. Each step can fail, time out or succeed twice. Someone must own the question "what happens next?" in every situation, and an auditor must be able to see what did happen.
If that logic is scattered across services with ad hoc retries, you get stuck payments, double postings and a support team reading logs to find out where a payment is. If it is centralised without care, you get a single orchestrator that becomes the bank's most fragile component. Choosing where orchestration lives is a first-order architecture decision, and the design authority will ask you to defend it.
Educational content, not regulatory advice
Timeouts, hold policies and retry limits for payments are risk decisions as well as technical ones. Sanctions-screening behaviour, scheme rules and customer-communication obligations need review by payments operations, financial-crime compliance and risk ⚠ VERIFY. Figures here are illustrative.
The idea in plain language
Orchestration: one coordinator holds the process definition. It calls step A, waits, calls step B, and handles errors. You can read the process in one place.
Choreography: no coordinator. Service A emits "funds reserved"; service B reacts and emits "screening passed"; and so on. Loosely coupled, but the process exists only implicitly, spread across subscriptions (event-driven architecture).
| Aspect | Orchestration | Choreography |
|---|---|---|
| Visibility of the process | One definition; one execution history | Reconstructed from events |
| Coupling | Orchestrator knows all participants | Participants know only events |
| Changing the flow | Edit one definition | Touch several services |
| Failure handling | Central Retry/Catch/compensation | Each service handles its own, plus timeouts for "nothing happened" |
| Best for | Money movement, regulated approvals, long waits | Notifications, analytics, independent reactions |
Most banks use both: orchestrate the critical money path, choreograph the reactions (notifications, reporting, fraud analytics).
Analogy: an orchestra and a jazz session. A symphony needs a conductor so that a payment's steps happen in a defined order. Jazz works when musicians listen and react, which suits the "tell the customer, update the dashboard" side.
Three places to orchestrate
| Option | What it is | Strengths | Weaknesses |
|---|---|---|---|
| Step Functions Standard | Managed workflow, up to one year, execution history kept for 90 days by default | Durable, visual, built-in retry and wait, task tokens for approval, many AWS integrations | Per-transition cost; start-rate and payload limits; latency per transition |
| Step Functions Express | High-volume, short (up to 5 minutes) workflows, sync or async | Cheap at scale, low latency, synchronous start | No execution history unless logging is configured; at-least-once for async |
| In-service code | A C# class in your service (as in the runner below) | Fast, testable with xUnit, full language power | You must persist state for crash recovery, and build your own visibility |
What problem the AWS services solve
Step Functions provides durable state, retries with backoff, timeouts, parallelism, waiting for human approval, and direct integrations with AWS services. It does not decide your business rules, make your steps idempotent or choose what compensation means.
Limits that change designs (checked against the Step Functions quota page on 2026-11-18; re-verify before relying on them):
| Limit | Value | Design consequence |
|---|---|---|
| Max execution time | Standard 1 year; Express 5 minutes | Long approvals need Standard |
| Max input or output per state | 256 KiB | Pass references to data in S3 or a table, never the data |
| Execution history events | 25,000 for Standard | Long loops must start child executions |
| Execution history retention | 90 days after close (Standard) | Copy audit-relevant facts to your own store |
StartExecution rate, Standard, us-east-1 | Bucket 1,300, refill 300 per second (soft quota) | Burst control and quota requests for peak payment starts |
| Redrive | Standard only, within 14 days | Failed executions can resume from the failed state |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Durable Functions (orchestrator code) | Step Functions (declarative JSON) or in-service C# | Step Functions is declarative; Durable Functions replays your C# code |
| Logic Apps | Step Functions | Both visual; Step Functions state is explicit JSON between states |
| Azure Service Bus sessions for ordering | SQS FIFO groups, Step Functions per payment | Orchestrator per payment instance instead of per session |
| Polly retry and circuit breaker | Step Functions Retry, plus Polly in code | Retry config is data, not code |
| Saga libraries (MassTransit, NServiceBus sagas) | Step Functions or a library in-service | Same saga idea; Step Functions persists for you |
Architecture
Security-boundary view
The API role may only start the payment state machine. The state machine's own role may only invoke the named step functions. Nothing has broad permissions.
The 15-section design
Sections 1 to 3: requirements
| Section | Content |
|---|---|
| 1 Business | Make every payment's progress visible and every failure recoverable without manual database edits; cut stuck-payment tickets by 80 percent (fictional) |
| 2 Functional | F1 Start a saga per payment, once. F2 Run steps in order. F3 Time out and retry each step. F4 Hold on screening uncertainty. F5 Compensate on scheme rejection. F6 Emit status events. F7 Allow redrive or manual resolution |
| 3 Non-functional | Median end-to-end orchestration overhead under 300 ms. 99.95% of started payments reach a terminal or held state within 5 minutes. 300 payments per second at peak. Audit: who, what, when for each step for 7 years ⚠ VERIFY. No double reservation or double submission |
Section 5: service selection and the decision
| Choice | Why | Rejected alternative | Trade-off |
|---|---|---|---|
| Step Functions Standard for the payment saga | Durable, visual, redrive, history for audit | Express | Higher per-step cost; the saga includes a possible human hold, which exceeds Express limits |
API invokes Step Functions asynchronously and returns 202 | Keeps API latency stable | Sync Express under the API | Client polls or receives an event for the final result |
| Lambda or ECS tasks per step | Isolated, individually scalable | One monolithic step | More units to deploy |
| In-service orchestration for the hot validation path | Sub-50 ms, pure CPU | Step Functions | Not durable; acceptable because it is stateless and repeatable |
| Choreography for notifications | Loose coupling | Orchestrated notification steps | Process is implicit; keep subscriptions documented |
Decision rules. Use Step Functions when the flow lasts longer than a request, needs human approval, needs an audit-visible execution history, or has several AWS integrations. Use in-service code when the flow is short, pure and latency-critical, or logic is easier to test in C#. Use Express for high-volume short workflows where logging is enough. Use choreography for independent reactions.
Sections 6 and 7: network and security
- Network. Step Functions is a regional service reached over the AWS network. Task functions can run in VPC private subnets when they must reach private resources; use VPC endpoints for AWS APIs.
- Identity. One execution role per state machine; task-level least privilege; the caller can start only the intended state machine.
- Data in executions. Execution history stores state input and output. Never put card numbers, full names or identifiers beyond what is needed into state; pass
paymentIdand look up data in a protected store. Encrypt with a customer-managed key where supported ⚠ VERIFY current options. - Logging. Enable execution logging to CloudWatch Logs at an appropriate level, and exclude execution data from logs if it could contain personal data.
- Tracing. Turn on X-Ray or OpenTelemetry-based tracing consistent with your observability standard (Module 11).
Sections 8 to 10: API, data and flow
- API.
POST /paymentswithIdempotency-Key; returns202andLocation: /payments/{id}.GET /payments/{id}returnsstatus(RECEIVED,RESERVED,SCREENING,SUBMITTED,BOOKED,HELD,FAILED). - Idempotent start. Use the payment ID as the execution name. Starting a Standard execution with the same name and the same input while it is open does not start a second one; verify the exact semantics for closed executions in the current documentation ⚠ VERIFY.
- Database. A
paymentstable stores the authoritative status; the execution history is a diagnostic record, not the system of record. - Data flow. API stores payment, starts execution, returns 202. Each step reads and writes the payments table by ID. The final state publishes an EventBridge event.
Section 11: failure scenarios
| Failure | Detection | Automatic response | Impact | Manual step |
|---|---|---|---|---|
| Screening step hangs | TimeoutSeconds (5 s) | Retry twice with backoff; then Catch to Hold | Payment held | Operations review |
| Transient Lambda error | Lambda.ServiceException etc. | Retry with exponential backoff | None | None |
| Scheme rejection | Error from Submit | Compensate: release reservation | Payment failed | Customer notified |
| Compensation fails | Release step error after retries | Alarm; execution fails visibly | Funds stuck reserved | Operations repair, then redrive |
| Execution start throttled | ThrottlingException or ExecutionLimitExceeded | Client retries with jitter; queue in front of start if bursty | Slower acceptance | Request quota increase |
| Payload exceeds 256 KiB | States.DataLimitExceeded | Execution fails | Failed payment | Fix design to pass references |
| Deployment during in-flight executions | None needed | Running executions keep their definition version | None | Use versions and aliases for staged roll-out |
| Approver never responds | Task token timeout | Timeout error to catch | Held longer | Escalation |
Sections 12 to 15
- Scalability. At 300 payments per second, check
StartExecutionthrottling (Standard bucket and refill in us-east-1 as above ⚠ VERIFY), the state-transition throttle (5,000 per second bucket in us-east-1), and downstream Lambda or ECS concurrency. Smooth bursts with an SQS queue in front ofStartExecution. - Cost. Standard pricing is per state transition ⚠ VERIFY the current price. Worked example with an assumed price of 0.025 USD per 1,000 transitions: a saga with 10 transitions at 10 million payments a month is 100 million transitions, about 2,500 USD. The same flow on Express is billed by requests and duration and may be cheaper at that volume ⚠ VERIFY. Always compute for your own transition count.
- Implementation. Phase 1 happy path with stubs. Phase 2 idempotent steps and compensation. Phase 3 failure injection (timeouts, errors). Phase 4 quotas and load test. Phase 5 game day with operations.
- Compliance touchpoints.
| Area | Where it touches | Reviewing team | Evidence |
|---|---|---|---|
| Payment screening hold policy | Catch to Hold behaviour | Financial-crime compliance | Signed policy, test results |
| Auditability of money movement | Execution history, payments table | Internal audit | Retention approach, sample trace |
| Data protection | Data in state and logs | Data protection officer | Log-data exclusion, key policy |
| Operational resilience | Impact tolerance and timeouts | Operational resilience lead | Scenario tests |
| Change control | Versioned definitions | Change advisory board | Release records |
Step-by-step implementation
Draw the state machine on paper first
List steps, which can fail, what each failure means, and what compensates what. A compensation is a business operation (release a reservation), not a rollback.
Make every step idempotent
Each step receives paymentId and uses it as the idempotency key. Running it twice must have the same effect as once (idempotency keys).
Write the ASL definition with explicit timeouts
Every Task gets TimeoutSeconds. A task without a timeout can wait for a very long time, because the default is effectively the execution limit.
Add Retry for transient errors and Catch for business failures
Retry network and service errors with backoff; never retry a business rejection. Route States.Timeout and States.TaskFailed to the correct compensation or hold state.
Deploy with IaC and invoke with an idempotent name
Use the payment ID as the execution name and test that a second start does not create a second execution.
Inject failures and watch the execution graph
Make Screen sleep longer than its timeout, make Submit return an error, and confirm the console shows the path you designed.
AWS Console instructions
- Console → Step Functions → State machines → Create state machine. Choose Standard, use Workflow Studio to draw or paste the definition, set the execution role, enable logging and tracing.
- Start execution with input
{"paymentId":"PAY-0001","amount":"25.00","currency":"GBP"}and a name such asPAY-0001. - Executions → select one → Graph view to see which state failed, with each state's input and output. Redrive appears on failed Standard executions.
AWS CLI
# Start one execution; the name makes the start idempotent for the same payment.
aws stepfunctions start-execution \
--state-machine-arn arn:aws:states:us-east-1:111122223333:stateMachine:hl-m14-payment-saga \
--name PAY-0001 \
--input '{"paymentId":"PAY-0001","amount":"25.00","currency":"GBP"}' \
--profile learn-sandbox --region us-east-1
# See its status and output.
aws stepfunctions describe-execution \
--execution-arn arn:aws:states:us-east-1:111122223333:execution:hl-m14-payment-saga:PAY-0001 \
--profile learn-sandbox --region us-east-1
# Read the step-by-step history (what auditors and operators look at).
aws stepfunctions get-execution-history \
--execution-arn arn:aws:states:us-east-1:111122223333:execution:hl-m14-payment-saga:PAY-0001 \
--profile learn-sandbox --region us-east-1
# Resume a failed Standard execution from the failed state (within the redrive period).
aws stepfunctions redrive-execution \
--execution-arn arn:aws:states:us-east-1:111122223333:execution:hl-m14-payment-saga:PAY-0001 \
--profile learn-sandbox --region us-east-1Account IDs shown are fictional placeholders.
Infrastructure as Code
The same ASL file serves both tools. It uses ${...} placeholders that CDK fills through DefinitionSubstitutions and Terraform fills through templatefile.
{
"Comment": "Learning payment saga: validate, reserve, screen, submit, with compensation",
"StartAt": "Validate",
"TimeoutSeconds": 600,
"States": {
"Validate": {
"Type": "Task",
"Resource": "${ValidateFn}",
"TimeoutSeconds": 10,
"Retry": [
{ "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException", "States.Timeout"],
"IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2.0 }
],
"Catch": [ { "ErrorEquals": ["States.ALL"], "Next": "Failed", "ResultPath": "$.error" } ],
"Next": "ReserveFunds"
},
"ReserveFunds": {
"Type": "Task",
"Resource": "${ReserveFn}",
"TimeoutSeconds": 10,
"Retry": [
{ "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException", "States.Timeout"],
"IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2.0 }
],
"Catch": [ { "ErrorEquals": ["States.ALL"], "Next": "Failed", "ResultPath": "$.error" } ],
"Next": "ScreenSanctions"
},
"ScreenSanctions": {
"Type": "Task",
"Resource": "${ScreenFn}",
"TimeoutSeconds": 5,
"Retry": [
{ "ErrorEquals": ["States.Timeout", "Lambda.ServiceException"],
"IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2.0 }
],
"Catch": [ { "ErrorEquals": ["States.ALL"], "Next": "HoldForReview", "ResultPath": "$.error" } ],
"Next": "SubmitToScheme"
},
"SubmitToScheme": {
"Type": "Task",
"Resource": "${SubmitFn}",
"TimeoutSeconds": 15,
"Retry": [
{ "ErrorEquals": ["Lambda.ServiceException", "States.Timeout"],
"IntervalSeconds": 2, "MaxAttempts": 2, "BackoffRate": 2.0 }
],
"Catch": [ { "ErrorEquals": ["States.ALL"], "Next": "ReleaseFunds", "ResultPath": "$.error" } ],
"Next": "Booked"
},
"ReleaseFunds": {
"Type": "Task",
"Resource": "${ReleaseFn}",
"TimeoutSeconds": 10,
"Retry": [
{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 5, "BackoffRate": 2.0 }
],
"Next": "Failed"
},
"HoldForReview": {
"Type": "Task",
"Resource": "${HoldFn}",
"TimeoutSeconds": 10,
"Retry": [
{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 5, "BackoffRate": 2.0 }
],
"Next": "Held"
},
"Held": { "Type": "Succeed", "Comment": "Payment parked with status HELD for operations review" },
"Booked": { "Type": "Succeed" },
"Failed": { "Type": "Fail", "Error": "PaymentFailed", "Cause": "See $.error and the payments table" }
}
}using Amazon.CDK;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.Logs;
using Amazon.CDK.AWS.StepFunctions;
using Constructs;
namespace Harbourline.Architecture.Orchestration;
public class PaymentSagaStack : Stack
{
public PaymentSagaStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
// Stub steps. In a real design these are .NET functions or ECS tasks.
Function Step(string name) => new(this, name, new FunctionProps
{
Runtime = Runtime.PYTHON_3_13,
Handler = "index.handler",
Code = Code.FromInline("def handler(event, context):\n return event\n"),
Timeout = Duration.Seconds(10),
});
var validate = Step("Validate");
var reserve = Step("Reserve");
var screen = Step("Screen");
var submit = Step("Submit");
var release = Step("Release");
var hold = Step("Hold");
var logs = new LogGroup(this, "SagaLogs", new LogGroupProps
{
Retention = RetentionDays.ONE_MONTH,
RemovalPolicy = RemovalPolicy.DESTROY,
});
var machine = new StateMachine(this, "PaymentSaga", new StateMachineProps
{
StateMachineName = "hl-m14-payment-saga",
StateMachineType = StateMachineType.STANDARD,
DefinitionBody = DefinitionBody.FromString(File.ReadAllText("payment-saga.asl.json")),
DefinitionSubstitutions = new Dictionary<string, string>
{
["ValidateFn"] = validate.FunctionArn,
["ReserveFn"] = reserve.FunctionArn,
["ScreenFn"] = screen.FunctionArn,
["SubmitFn"] = submit.FunctionArn,
["ReleaseFn"] = release.FunctionArn,
["HoldFn"] = hold.FunctionArn,
},
Timeout = Duration.Minutes(10),
TracingEnabled = true,
Logs = new LogOptions { Destination = logs, Level = LogLevel.ERROR, IncludeExecutionData = false },
});
foreach (var fn in new[] { validate, reserve, screen, submit, release, hold })
fn.GrantInvoke(machine);
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m14-l13");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "learner");
}
}IncludeExecutionData = false keeps state input and output out of CloudWatch Logs, which matters when a payload could hold personal data. LogLevel here is the Step Functions enum, not the .NET logging one.
Complete code
When the flow is short and you want the logic in C#, a small in-service runner gives you the same ideas: per-step timeout, bounded retries with backoff, compensation in reverse order, and an explicit "needs a human" outcome when compensation itself fails.
// Run with: dotnet run SagaRunner.cs
// A tiny in-service orchestrator: per-step timeout, bounded retries with backoff,
// compensation in reverse order, and an explicit "needs a human" outcome.
var log = new List<string>();
var attemptsAtSubmit = 0;
SagaStep[] happyButFlakyPath =
[
new("ReserveFunds", _ => Note("reserve"), _ => Note("release reservation"), TimeSpan.FromSeconds(1), 1),
new("Screen", _ => Note("screen ok"), null, TimeSpan.FromSeconds(1), 2),
new("Submit", _ => ++attemptsAtSubmit < 2 ? throw new HttpRequestException("scheme 503") : Note("submitted"),
_ => Note("recall from scheme"), TimeSpan.FromSeconds(1), 3),
];
Console.WriteLine("Scenario A: " + await SagaRunner.RunAsync(happyButFlakyPath, log));
log.Clear();
SagaStep[] screeningHangs =
[
new("ReserveFunds", _ => Note("reserve"), _ => Note("release reservation"), TimeSpan.FromSeconds(1), 1),
new("Screen", ct => Task.Delay(Timeout.Infinite, ct), null, TimeSpan.FromMilliseconds(100), 2),
new("Submit", _ => Note("submitted"), _ => Note("recall from scheme"), TimeSpan.FromSeconds(1), 3),
];
Console.WriteLine("Scenario B: " + await SagaRunner.RunAsync(screeningHangs, log));
foreach (var line in log) Console.WriteLine(" " + line);
return 0;
Task Note(string text) { log.Add(text); return Task.CompletedTask; }
public enum SagaOutcome { Completed, Compensated, NeedsManualIntervention }
public sealed record SagaStep(string Name, Func<CancellationToken, Task> Do, Func<CancellationToken, Task>? Undo,
TimeSpan Timeout, int MaxAttempts);
public static class SagaRunner
{
public static async Task<SagaOutcome> RunAsync(IReadOnlyList<SagaStep> steps, List<string> log)
{
var done = new Stack<SagaStep>();
foreach (var step in steps)
{
if (await TryAsync(step.Name, step.Do, step.Timeout, step.MaxAttempts, log)) { done.Push(step); continue; }
log.Add($"{step.Name} failed permanently: compensating {done.Count} completed step(s)");
while (done.Count > 0)
{
var finished = done.Pop();
if (finished.Undo is null) continue;
if (!await TryAsync(finished.Name + ".undo", finished.Undo, finished.Timeout, 3, log))
{
log.Add($"compensation of {finished.Name} failed: raise an alarm for operations");
return SagaOutcome.NeedsManualIntervention;
}
}
return SagaOutcome.Compensated;
}
return SagaOutcome.Completed;
}
private static async Task<bool> TryAsync(string name, Func<CancellationToken, Task> action, TimeSpan timeout,
int maxAttempts, List<string> log)
{
for (var attempt = 1; attempt <= maxAttempts; attempt++)
{
try
{
using var cts = new CancellationTokenSource(timeout);
await action(cts.Token);
return true;
}
catch (Exception e) when (e is OperationCanceledException or HttpRequestException)
{
log.Add($"{name} attempt {attempt}/{maxAttempts}: {e.GetType().Name}");
if (attempt < maxAttempts) await Task.Delay(TimeSpan.FromMilliseconds(10 * Math.Pow(2, attempt)));
}
}
return false;
}
}Expected output:
Scenario A: Completed
Scenario B: Compensated
reserve
Screen attempt 1/2: TaskCanceledException
Screen attempt 2/2: TaskCanceledException
Screen failed permanently: compensating 1 completed step(s)
release reservationThis runner holds no durable state. If the process crashes between steps, nothing remembers where it was. That is the main reason a long-running or money-moving saga normally belongs in Step Functions, or in code that persists each transition.
Code explained line by line
SagaStepbundles what a step does (Do), how to undo it (Undo, optional), its timeout and its attempt limit. Putting these in data makes them reviewable, like ASL.RunAsyncpushes each completed step onto a stack. A stack pops in reverse order, which is exactly the order compensations must run.- When a step fails permanently, the loop pops and runs each
Undo. A step with noUndo(such as Screen) is skipped: a read-only check has nothing to compensate. - If a compensation itself fails after three attempts, the runner returns
NeedsManualInterventioninstead of pretending success. Compensation failure is a first-class outcome. TryAsyncwraps each attempt in aCancellationTokenSource(timeout). A step that ignores its token cannot be cancelled, so real steps must pass the token to I/O calls.- The
catch ... whenclause retries only transport errors and timeouts. A business rejection would be a different exception type and would not be retried. - The backoff
10 * 2^attemptmilliseconds doubles each time. Production code adds jitter and uses seconds, not milliseconds. - In Scenario A the Submit step fails once with a 503, then succeeds on attempt 2.
attemptsAtSubmitis captured by the lambda, which is how the demo makes failure deterministic.
Security considerations
⚠ Learning shortcut — not for production
The lab uses stub Lambda functions with inline code, a simple role setup and logs at ERROR level. Production steps are separately deployed, signed and scanned, each with a least-privilege role; the state machine uses versions and aliases for controlled roll-out; execution data is excluded from logs or encrypted and access-controlled; and callers are restricted to starting only the state machine they own.
- Least privilege on both sides. Caller can
states:StartExecutionon one ARN. State machine role can invoke only its listed functions. - No sensitive data in state. State is stored in history. Pass identifiers.
- Validate input at the start. The first state should reject malformed input before any side effect.
- Idempotent steps are a security control. Replays and redrives must not move money twice.
- Protect the task token. If you use callbacks for approval, treat tokens as secrets and bind them to the approver identity.
Banking use case
Harbourline's payments API starts one execution per payment, named with the payment ID. When the sanctions provider is slow, the screening step times out after 5 seconds, retries twice, then drops to HoldForReview; operations see the held payment in a queue with the execution link. When the scheme rejects a payment, the saga releases the reserved funds, marks the payment failed and the customer receives a clear message. During a game day the team deploys a new version of the saga definition while 200 payments are in flight; running executions keep the old definition, new ones use the new alias target, and nothing is lost.
Common mistakes
- Orchestrating everything. Notifications and analytics do not need an orchestrator.
- No timeouts. A task without
TimeoutSecondswaits very long. - Retrying business rejections. Retrying "insufficient funds" three times does not help.
- Using execution history as the system of record. It expires; keep status in your own table.
- Passing big payloads. The 256 KiB limit will bite in production. Pass references.
- Compensation that is not itself idempotent. Releasing a reservation twice must be safe.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
States.DataLimitExceeded | Payload over 256 KiB | Store data in S3 or a table and pass the key |
States.Runtime on a Task | Bad ResultPath or InputPath JSONPath | Test with the TestState API or Workflow Studio |
AccessDeniedException invoking a step | Execution role lacks lambda:InvokeFunction | Grant on the exact function ARN |
| Execution stuck at "Running" | Waiting on a task token with no timeout | Add TimeoutSeconds or HeartbeatSeconds |
| Throttled starts at peak | StartExecution rate bucket exhausted | Queue in front, add jitter, request a quota increase |
| Second start created a duplicate | Different execution names | Use payment ID as the name |
| History missing after three months | 90-day retention | Export audit facts to your own store |
Cost and cleanup
The C# runner is free. The deploy creates five tiny Lambda functions and one Standard state machine; at lab volumes the cost is effectively zero (Step Functions and Lambda free allowances apply ⚠ VERIFY). Run the deploy briefly and destroy it.
cdk destroy --profile learn-sandbox
terraform destroy
# Confirm nothing tagged for the course remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1Exercises
- 🟢 Guided: run
SagaRunner.cs. Add a third scenario where the Submit step always fails; predict the log lines, including which compensations run. - 🟡 Challenge: make the
UndoofReserveFundsfail permanently and confirm the outcome isNeedsManualIntervention. Then change the ASL soReleaseFundsfailure routes to a state that notifies operations before failing. - 🔴 Live problem: the design authority says "Step Functions is too expensive for 10 million payments a month". Build the cost comparison for Standard, Express and in-service orchestration with your own transition counts, and state what you would give up with each.
Assignment
Add architecture/payment-orchestration/ to your portfolio: the 15-section document, an ADR "orchestrate payments with Step Functions Standard, choreograph notifications", the ASL file deployed with CDK or Terraform, and the saga runner with unit tests covering timeout, retry and compensation failure.
Solutions
If Submit always fails (three attempts), the runner logs three Submit attempts, then Submit failed permanently: compensating 2 completed step(s). It pops Screen (no Undo, skipped) and ReserveFunds, logging release reservation. The outcome is Compensated. Why: compensation runs in reverse order for completed steps only; Submit never completed, so it is not undone.
With a permanently failing Undo, TryAsync returns false after three attempts and the runner returns NeedsManualIntervention. In ASL, add a Catch on ReleaseFunds with Next set to a Task that publishes to an SNS topic or SQS queue for operations, then Next to Failed. Why: compensation failure leaves money in a half-state, which needs a human, not silent success.
Compute transitions per payment from your ASL (count each state entered, including retries on average). Standard cost equals transitions times the per-transition price ⚠ VERIFY. Express cost is requests plus duration-based charges ⚠ VERIFY, which favours short flows. In-service orchestration has near-zero service cost but you build persistence, visibility and redrive. Present all three with the trade-off: Standard for the money path where the audit trail and redrive justify the cost; Express for short, high-volume, idempotent sub-flows; in-service only where state loss is acceptable or you persist transitions yourself.
Knowledge check
Answer: For independent reactions such as notifications and analytics, where loose coupling matters more than a single visible process.
Answer: Later steps may depend on earlier ones, so you undo the most recent first.
Answer: It can wait very long, effectively until the execution limit. Always set TimeoutSeconds, and HeartbeatSeconds for callbacks.
Answer: State input and output are limited to 256 KiB, and state is stored in execution history, so large or sensitive data should not travel in it.
Answer: No. Standard history is retained for 90 days after close. Keep authoritative status in your own database.
Interview questions
- Step Functions or code for a payment saga? Step Functions when you need durability, long waits, audit-visible history and redrive; code when the flow is short and latency-critical, persisting transitions if state loss is unacceptable.
- How do you prevent double processing on retry? Idempotent steps keyed by business ID, and idempotent execution names.
- How do you handle failure of a compensating action? Retry with backoff, then alarm and a visible failed state for operations; never swallow it.
- What are the main Step Functions limits you design around? 256 KiB payloads, 25,000 history events, start-rate and transition throttles, 90-day history retention.
- How do you roll out a new workflow version safely? Publish versions, shift an alias gradually, and let in-flight executions finish on their original definition.
Certification practice
A payment flow includes a possible manual review that can take up to two days. Which orchestration is most appropriate?
A. Step Functions Express with synchronous invocation. B. Step Functions Standard with a callback task token. C. A Lambda function that sleeps for two days. D. An SQS queue with a visibility timeout of two days.
Answer: B. Standard supports executions up to a year and callbacks. Express is limited to 5 minutes, Lambda has a maximum runtime of 15 minutes, and a two-day visibility timeout is not allowed and is not orchestration.
A Task state calls a Lambda function that sometimes throws a transient service error. What should the state definition include?
A. A Retry block with ErrorEquals, IntervalSeconds, MaxAttempts and BackoffRate.
B. A Wait state before the Task.
C. A Pass state after the Task.
D. A larger TimeoutSeconds only.
Answer: A. Retry handles transient errors with backoff. B and C do not retry, and D does not retry either; it only delays the failure.
An API must start a workflow per payment and avoid starting two executions for the same payment when the client retries. Which approach is correct?
A. Generate a random execution name each time. B. Use the payment ID as the Standard execution name. C. Disable client retries. D. Use Express Workflows only.
Answer: B. A deterministic name makes a retried start idempotent. A defeats it, C is unrealistic, and D does not address duplicate starts ⚠ VERIFY exact semantics for closed executions.
Completion checklist
- I can state when I would orchestrate and when I would choreograph
- I ran the saga runner and explained both scenarios
- My ASL gives every Task a timeout and separates retry from catch
- I compared Standard, Express and in-service cost for my own transition count
- I ran cleanup and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- Orchestrate the money path; choreograph reactions.
- Every step is idempotent, timed out and either retried (transient) or compensated (business failure).
- Know the limits: 256 KiB payload, 25,000 history events, 90-day history, start-rate throttles.
- Compensation failure is a real outcome that needs an alarm and a human.
- Hold policies and timeouts are risk decisions that need compliance and operations review.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Core banking integration
Connect cloud-native services to a legacy core banking system over hybrid networking and MQ-style messaging, using an anti-corruption gateway, stand-in reads, a circuit breaker and safe handling of unknown outcomes.
Event-driven banking architecture
Design domain events, per-domain EventBridge buses, schema governance and safe replay for a bank, keep personal data out of events, and check event contracts and duplicate delivery with a runnable C# validator.