Skip to Content
Cloud Engineering Academy
7. Data Engineering & ETL Pipelines

Orchestration: Step Functions, EventBridge and MWAA

Run Harbourline's transaction pipeline as an event-driven Step Functions workflow — triggered by S3 through EventBridge, waiting on a Glue job, retrying safely and alerting on failure — and know when EventBridge Scheduler, Glue workflows or Apache Airflow (MWAA) fit better.

intermediate⏱ 1 h 40 min🟡 Low costDEA-C01SAA-C03DVA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.StepFunctions 4.1.x) · Amazon.CDK.Lib 2.277.0 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain what orchestration adds to a pipeline (ordering, waiting, retries, error routing, audit history) and why it lives outside the Spark job.
  • Trigger a Step Functions workflow from an S3 upload using an EventBridge rule, and from a timetable using EventBridge Scheduler.
  • Write a state machine that runs a Glue job synchronously (.sync), retries throttling errors with back-off, catches failures and alerts.
  • Choose between Standard and Express workflows, and between Step Functions, Glue workflows and Amazon MWAA.
  • Test states without deploying a whole pipeline, using the TestState API.

Prerequisites

Why this matters in banking

In many banks the "pipeline" is a SQL Server Agent job at 02:00 that runs eleven SSIS packages one after another. When step 7 fails at 03:14, nobody knows until finance opens an empty report at 08:00, and the person on call has to work out which of steps 1–6 can be re-run safely. When a partner's file arrives late, the schedule runs on yesterday's data.

A bank needs pipelines that start when the data arrives, wait for long-running work without polling scripts, retry transient errors automatically, stop cleanly on real errors, tell a human straight away, and keep a complete history of what ran, with which inputs, and what happened — because the auditor will ask.

The idea in plain language

Orchestration is the "conductor" of a pipeline. It does not transform data itself; it decides what runs, in which order, what happens on success and on failure, and it records each step.

  • AWS Step Functions runs state machines: JSON (Amazon States Language, ASL) definitions of steps (states) such as Task, Choice, Parallel, Map, Wait, Succeed and Fail. Tasks can call 200+ AWS services directly. A Standard workflow can run for up to a year and keeps a full execution history; an Express workflow is for high-volume, short (up to 5 minutes) event processing.
  • Amazon EventBridge routes events: a rule matches events (for example S3 Object Created in bucket X under prefix incoming/) and sends them to a target such as a state machine.
  • EventBridge Scheduler starts targets on a timetable (cron or rate expressions) with time zones, flexible windows and retries — the replacement for "a cron job somewhere".
  • Amazon MWAA (Managed Workflows for Apache Airflow) runs Apache Airflow for teams that already write Python DAGs and need Airflow's operator ecosystem.

Analogy: the Glue job is the cashier counting the money. Step Functions is the branch manager's checklist: "when the van arrives (event), count the money (task), if the count is off by more than £0 call head office (choice + alert), file the slip (audit)". The checklist is kept, signed, for every van.

What problem the AWS service solves

NeedStep Functions featureNote
Wait for a Glue job without a polling scriptService integration glue:startJobRun.syncThe state waits (Standard workflows only) and fails if the job fails
Retry throttling, not business errorsRetry with ErrorEquals, IntervalSeconds, BackoffRate, MaxAttempts, JitterStrategyMatch specific error names first
Route failuresCatch → notify → FailResultPath keeps the original input next to the error
Process many filesMap state (inline or distributed)Distributed Map can iterate over S3 objects at large scale
Audit historyExecution history (Standard) + CloudWatch LogsStandard history is retained for 90 days after completion; ship logs for longer retention
Data passingJSONata (newer) or JSONPathJSONata is recommended for new workflows; this lesson uses JSONPath because the CDK lab uses it

Limits to know: execution history of a Standard workflow is limited to 25,000 events; state input/output payloads are limited to 256 KB — pass S3 keys, not data. Standard workflows bill per state transition; Express bill per request and duration.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Data Factory pipelineStep Functions state machine (or Glue workflow)Step Functions is general purpose: Glue, Lambda, ECS, Athena, DynamoDB, SNS, HTTP APIs
ADF storage event triggerEventBridge rule on S3 Object CreatedThe bucket must have EventBridge notifications turned on
ADF schedule trigger / SQL Agent scheduleEventBridge SchedulerTime-zone aware, flexible time windows, built-in retries and DLQ
Durable Functions orchestratorStep Functions Standard workflowDefined in ASL (JSON/YAML) or with CDK, not in C# code; visual graph in the console
Logic AppsStep Functions + EventBridge PipesSimilar "connectors" idea through service integrations
ADF Until/ForEach activitiesMap state, Choice + Wait loopDistributed Map reads lists straight from S3
Apache Airflow on Azure (Workflow Orchestration Manager)Amazon MWAABoth managed Airflow; MWAA environments bill per hour even when idle

Architecture

Security-boundary view. EventBridge assumes a role that may only states:StartExecution on this one state machine. The state machine's role may only start this Glue job, read its status and publish to the alerts topic. The Glue job runs with its own least-privilege role. Execution data is not written to logs (IncludeExecutionData = false) because inputs can contain file names that reveal partner identities.

Step-by-step implementation

Turn on EventBridge for the raw bucket

S3 sends events to EventBridge only when the bucket property Amazon EventBridge is on. The lab CDK stack sets EventBridgeEnabled = true.

Write the rule

Match source = aws.s3, detail-type = Object Created, the bucket name, and object.key with prefix incoming/. Anything else (for example the job's own outputs in other buckets) cannot trigger a loop.

Write the state machine

One Task state that calls Glue with .sync, passing --raw_bucket and --raw_key from the event. A Retry for Glue throttling and concurrency errors, a Catch that publishes to SNS and ends in Fail.

Make re-runs safe

Orchestration retries are only safe if the job is idempotent. The lab's Glue job derives its run id from bucket + key + version id and exits early if the audit record already exists.

Add a schedule for time-based work

Daily reports (for example refreshing a Quick Sight dataset or running a reconciliation query) use EventBridge Scheduler with Europe/London time zone so British Summer Time is handled for you.

Test

Use the console's Test state feature or the TestState API to run the NotifyFailure state with a sample input, then upload a sample file and watch the execution graph.

AWS Console instructions

  1. S3 → hl-m07-raw-… → Properties → Amazon EventBridge → Edit → On.
  2. Step Functions → State machines → Create state machine → Blank. Switch to Code, paste the ASL below, choose type Standard, and under Config select logging level ALL with Include execution data unticked, and an encrypted log group.
  3. Amazon EventBridge → Rules → Create rule on the default bus. Event pattern: paste the JSON below. Target: Step Functions state machine hl-m07-transaction-pipeline; let EventBridge create a role.
  4. EventBridge → Scheduler → Schedules → Create schedule: recurring, cron 0 6 * * ? *, time zone Europe/London, flexible time window Off, target Templated target → Amazon Athena StartQueryExecution (or any universal target), retry policy 3 attempts, dead-letter queue on.
  5. Upload transactions_messy.csv to incoming/sftp/ and open Executions to watch the run.

AWS CLI

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)

# The event pattern used by the rule (also handy for testing patterns).
cat > pattern.json <<EOF
{ "source": ["aws.s3"], "detail-type": ["Object Created"],
  "detail": { "bucket": { "name": ["hl-m07-raw-${ACCOUNT}"] },
              "object": { "key": [ { "prefix": "incoming/" } ] } } }
EOF
aws events test-event-pattern --event-pattern file://pattern.json \
  --event '{"id":"1","account":"'"$ACCOUNT"'","source":"aws.s3","time":"2026-11-18T06:00:00Z","region":"us-east-1","resources":[],"detail-type":"Object Created","detail":{"bucket":{"name":"hl-m07-raw-'"$ACCOUNT"'"},"object":{"key":"incoming/sftp/a.csv"}}}'

# Start an execution by hand with a fake S3 event (useful before the rule exists).
aws stepfunctions start-execution \
  --state-machine-arn "arn:aws:states:us-east-1:${ACCOUNT}:stateMachine:hl-m07-transaction-pipeline" \
  --name "manual-$(date +%Y%m%d%H%M%S)" \
  --input '{"detail":{"bucket":{"name":"hl-m07-raw-'"$ACCOUNT"'"},"object":{"key":"incoming/sftp/transactions_messy.csv"}}}'

# Test one state in isolation (no state machine run needed).
aws stepfunctions test-state \
  --definition '{"Type":"Pass","Parameters":{"file.$":"$.detail.object.key"},"End":true}' \
  --role-arn "arn:aws:iam::${ACCOUNT}:role/hl-m07-sfn-test" \
  --input '{"detail":{"object":{"key":"incoming/sftp/a.csv"}}}'

# A daily schedule at 06:00 London time that starts the reconciliation report query.
aws scheduler create-schedule --name hl-m07-daily-recon \
  --schedule-expression "cron(0 6 * * ? *)" --schedule-expression-timezone Europe/London \
  --flexible-time-window Mode=OFF \
  --target '{"Arn":"arn:aws:scheduler:::aws-sdk:athena:startQueryExecution","RoleArn":"arn:aws:iam::'"$ACCOUNT"':role/hl-m07-scheduler","Input":"{\"QueryString\":\"SELECT * FROM hl_m07_gold.v_reconciliation WHERE status = '\''BREAK'\''\",\"WorkGroup\":\"hl-m07-analysts\"}"}'

Infrastructure as Code

From labs/m07-transaction-pipeline/infra/cdk/Harbourline.DataPipeline.Infra/DataLakeStack.cs:

var runEtl = new GlueStartJobRun(this, "ValidateAndTransform", new GlueStartJobRunProps
{
    GlueJobName = EtlJobName,
    IntegrationPattern = IntegrationPattern.RUN_JOB,       // ".sync"
    Arguments = TaskInput.FromObject(new Dictionary<string, object>
    {
        ["--raw_bucket"] = JsonPath.StringAt("$.detail.bucket.name"),
        ["--raw_key"] = JsonPath.StringAt("$.detail.object.key"),
    }),
    ResultPath = "$.glue",
});
runEtl.AddRetry(new RetryProps
{
    Errors = ["Glue.ConcurrentRunsExceededException", "Glue.AWSGlueException"],
    Interval = Duration.Seconds(30), MaxAttempts = 3, BackoffRate = 2,
});
var notifyFailure = new SnsPublish(this, "NotifyFailure", new SnsPublishProps
{
    Topic = alerts,
    Message = TaskInput.FromObject(new Dictionary<string, object>
    {
        ["file"] = JsonPath.StringAt("$.detail.object.key"),
        ["error"] = JsonPath.StringAt("$.error.Error"),
        ["execution"] = JsonPath.StringAt("$$.Execution.Id"),
    }),
    ResultPath = JsonPath.DISCARD,
});
runEtl.AddCatch(notifyFailure.Next(new Fail(this, "RunFailed")), new CatchProps { ResultPath = "$.error" });

var workflow = new StateMachine(this, "Workflow", new StateMachineProps
{
    StateMachineName = "hl-m07-transaction-pipeline",
    StateMachineType = StateMachineType.STANDARD,
    DefinitionBody = DefinitionBody.FromChainable(runEtl.Next(new Succeed(this, "RunSucceeded"))),
    Logs = new LogOptions { Destination = workflowLogs, Level = SfnLogLevel.ALL, IncludeExecutionData = false },
    TracingEnabled = true,
});

_ = new Rule(this, "RawObjectCreated", new RuleProps
{
    EventPattern = new EventPattern
    {
        Source = ["aws.s3"],
        DetailType = ["Object Created"],
        Detail = new Dictionary<string, object>
        {
            ["bucket"] = new Dictionary<string, object> { ["name"] = new[] { raw.BucketName } },
            ["object"] = new Dictionary<string, object> { ["key"] = new object[] { new Dictionary<string, object> { ["prefix"] = "incoming/" } } },
        },
    },
    Targets = [new SfnStateMachine(workflow)],
});

Complete code

An operations tool in C# that lists today's pipeline executions, shows failures with their error cause, and can redrive a failed execution (Step Functions redrive restarts a failed Standard execution from the failed state, keeping the successful steps).

PipelineRuns.cs
// dotnet add package AWSSDK.StepFunctions (v4), plus AWSSDK.SSO and AWSSDK.SSOOIDC for the SSO profile
using Amazon.StepFunctions;
using Amazon.StepFunctions.Model;

var stateMachineArn = args.Length > 0
    ? args[0]
    : throw new ArgumentException("usage: PipelineRuns <state-machine-arn> [--redrive <execution-arn>]");

using var sfn = new AmazonStepFunctionsClient(); // AWS_PROFILE=learn-sandbox

if (args.Length == 3 && args[1] == "--redrive")
{
    // Only for failures that are now fixed (for example a missing permission). The Glue job is idempotent.
    var redrive = await sfn.RedriveExecutionAsync(new RedriveExecutionRequest { ExecutionArn = args[2] });
    Console.WriteLine($"Redriven at {redrive.RedriveDate:u}");
    return 0;
}

var since = DateTime.UtcNow.Date;
string? token = null;
do
{
    var page = await sfn.ListExecutionsAsync(new ListExecutionsRequest
    {
        StateMachineArn = stateMachineArn,
        MaxResults = 100,
        NextToken = token,
    });

    foreach (var e in page.Executions.Where(e => e.StartDate >= since))
    {
        Console.Write($"{e.StartDate:HH:mm:ss} {e.Status,-10} {e.Name}");
        if (e.Status == ExecutionStatus.FAILED)
        {
            var detail = await sfn.DescribeExecutionAsync(new DescribeExecutionRequest { ExecutionArn = e.ExecutionArn });
            Console.Write($"  error={detail.Error} cause={Truncate(detail.Cause, 120)}");
        }

        Console.WriteLine();
    }

    token = page.NextToken;
}
while (token is not null);

return 0;

static string Truncate(string? s, int max) => s is null ? "" : s.Length <= max ? s : s[..max] + "…";

Code explained line by line

  1. GlueStartJobRun with IntegrationPattern.RUN_JOB — compiles to arn:aws:states:::glue:startJobRun.sync. Step Functions starts the job and waits for its terminal state; a failed or timed-out job becomes a task failure (States.TaskFailed).
  2. JsonPath.StringAt("$.detail.bucket.name") — reads values from the EventBridge event, which is the execution input. Glue receives them as --raw_bucket and --raw_key.
  3. ResultPath = "$.glue" — adds the Glue result to the input instead of replacing it, so later states still see $.detail.
  4. AddRetry with named errors — retries concurrency and transient Glue errors only. Data errors (DQ_FAILED, SCHEMA_REJECTED) are not retried: retrying bad data gives the same bad result.
  5. "JitterStrategy": "FULL" (ASL tab) — randomises retry delays so many executions do not retry at the same instant.
  6. AddCatch(... ResultPath = "$.error") — on any remaining error, keep the input and add the error, then go to NotifyFailure.
  7. $$.Execution.Id — the double dollar reads the context object; the execution ARN lets the on-call engineer open the run directly.
  8. ResultPath = JsonPath.DISCARD — the SNS response is not needed downstream.
  9. IncludeExecutionData = false — logs show state transitions and errors but not payloads.
  10. RedriveExecutionAsync (C# tool) — restarts a failed execution from the failed state. Use it after fixing the cause; it reuses the original input, so idempotency matters.
  11. ListExecutionsAsync with NextToken — pagination; the console shows the same list.

Glue workflows, MWAA and when to use what

OptionStrengthsWeaknessesChoose when
Step FunctionsServerless, pay per transition, any AWS service, visual history, .sync waits, redriveASL is another language; 256 KB payloadsDefault for AWS-native pipelines, mixed services (Glue + Lambda + Athena + SNS)
Glue workflows and triggersBuilt into Glue; chains crawlers and jobs; conditional triggersGlue-only; limited error routing and alertingA small, Glue-only chain (crawler → job → crawler)
Amazon MWAAAirflow DAGs in Python, huge operator library, backfills, familiar to data teamsEnvironment bills per hour even when idle; you manage DAG code, plugins and versionsThe bank already runs Airflow, or needs Airflow's scheduling and backfill semantics across many systems
EventBridge SchedulerTime-zone aware schedules, retries, DLQStarts things; does not orchestrate stepsAnything that is "run X at time T"

Free alternative for Airflow: run Apache Airflow locally with the official Docker Compose file from the Airflow project and write the same pipeline as a DAG; never leave an MWAA environment running for learning.

Testing locally

AWS documents Step Functions Local as unsupported and without feature parity, and points to the TestState API instead. TestState runs a single state's definition against real AWS integrations (it needs the states:TestState permission and network access) without creating a state machine. For offline unit tests, keep logic out of the state machine: the Glue job's rules are tested with pytest, the Lambda handlers with xUnit, and the state machine is a thin, readable sequence that you test with TestState and one end-to-end upload.

Security considerations

⚠ Learning shortcut — not for production

The lab's state machine role is created by CDK grants and the alerts go to an e-mail subscription. In production: one role per state machine with exactly the Glue job ARN, the SNS topic ARN and the KMS key; alerts go to the incident management tool via SNS → HTTPS or a chat integration; and the RunFailed path also writes an audit record so a failed run is visible in the same report as successful ones.

  • Least-privilege trigger roles: the EventBridge target role may only states:StartExecution on one ARN.
  • No secrets in execution input. Inputs are stored in execution history. Pass references (secret names, S3 keys), never values.
  • Encrypt the log group and, for Standard workflows, consider a customer managed key for the state machine's data at rest ⚠ VERIFY the current Step Functions KMS options for your design.
  • Loop prevention: the rule matches only incoming/ in the raw bucket; the job writes to other buckets.

Banking use case

Saltmarsh Payroll uploads a file at 21:30. EventBridge starts the workflow within seconds; the Glue job validates, masks and reconciles; if reconciliation shows a BREAK, the run succeeds technically but the reconciliation view flags it, and the 06:00 scheduled query e-mails the operations team the list of breaks. If the Glue job fails — for example the partner changed the header — the workflow catches SCHEMA_REJECTED, posts the file name and execution link to the alerts topic, and the on-call engineer sees exactly which file failed and why. Every execution's history is the evidence auditors ask for.

Common mistakes

  • Polling Glue from Lambda in a loop → pays for idle Lambda time and adds code. Use .sync.
  • Retrying States.ALL → retries data errors three times, tripling cost and noise. Retry named transient errors only.
  • Putting large payloads in state input → hits 256 KB. Pass S3 keys.
  • Express workflow for a 20-minute Glue job → Express runs at most 5 minutes and does not support .sync job waits. Use Standard.
  • Forgetting the bucket's EventBridge setting → the rule never fires.
  • Schedules in UTC for UK business processes → reports shift by an hour twice a year. Use Scheduler's time zone.

Troubleshooting

SymptomLikely causeFix
No execution after uploadEventBridge off on bucket; key outside incoming/; rule disabledCheck bucket property and rule pattern with test-event-pattern
Execution fails with States.Runtime on a pathThe JSONPath does not exist in the inputInspect the input in execution history; fix the path
Glue.ConcurrentRunsExceededExceptionMore files than MaxConcurrentRunsRetry with back-off (configured) or raise the limit
Task fails immediately with AccessDeniedState machine role lacks glue:StartJobRun/GetJobRunAdd both on the job ARN
Alert e-mail never arrivesSubscription not confirmed, or KMS key policy blocks SNSConfirm subscription; allow the publishing role to use the key
Redrive not allowedExecution not failed, too old, or Express typeRedrive applies to failed Standard executions within the allowed period ⚠ VERIFY current window

Cost and cleanup

Step Functions Standard is billed per state transition (the lab uses about 3–4 per file; the first 4,000 transitions per month are in the always-free tier ⚠ VERIFY current free tier on the pricing page). EventBridge rules for AWS service events are free to match; Scheduler invocations have a free monthly allowance. The cost of a run is dominated by the Glue job (about USD 0.03). MWAA bills per environment-hour (hundreds of dollars a month even when idle) — design-only in this module.

# Lab stack: cdk destroy removes rule, state machine, topic and log group.
cd labs/m07-transaction-pipeline/infra/cdk && cdk destroy HlM07-DataLake --profile learn-sandbox
# If you created the schedule by hand:
aws scheduler delete-schedule --name hl-m07-daily-recon --profile learn-sandbox

Exercises

  • 🟢 Guided: Add a Choice state after the Glue job that reads a flag from the input ($.dryRun) and skips publishing. Test it with TestState.
  • 🟡 Challenge: Turn the workflow into a daily batch workflow: a Distributed Map state over every object under incoming/ from yesterday, with MaxConcurrency 3, running the same Glue job per file.
  • 🔴 Live problem: A partner re-sent yesterday's file with the same name "to be safe". Finance now sees doubled totals in a downstream system that reads curated data directly. Explain how the lab pipeline behaves, why your gold reports are fine, and what you change so downstream consumers are protected too.

Assignment

Export your state machine definition, a screenshot of one successful and one failed execution graph, and a one-page runbook: what the alert means, how to read the execution, when to redrive, and when to ask the partner for a corrected file.

Solutions

Knowledge check

Interview questions

  • "How do you make a pipeline restartable?" — Idempotent steps (deterministic outputs, run ids, conditional writes), orchestration with retries for transient errors only, redrive from the failed step, and audit records written last.
  • "Event-driven or scheduled?" — Event-driven for data arrival (process as soon as it lands); scheduled for time-based business processes (06:00 report). Most banks need both.
  • "How do you alert on pipeline failures?" — Catch in the workflow, publish a structured message (file, error, execution link) to SNS, plus CloudWatch alarms on ExecutionsFailed as a safety net.
  • "Step Functions vs Airflow?" — Serverless and AWS-integrated versus Python DAGs and ecosystem; cost model per transition versus per environment-hour.

Certification practice

Completion checklist

  • I can explain what orchestration adds beyond the job itself
  • I wrote (or read line by line) a state machine with .sync, Retry and Catch
  • I tested an event pattern and a single state without a full deployment
  • I can choose between Step Functions, Glue workflows, MWAA and Scheduler
  • I destroyed the stack and any hand-made schedules

Key takeaways

  • Orchestration owns order, waiting, retries, error routing and history; the job owns the data logic.
  • S3 → EventBridge → Step Functions Standard → Glue .sync is the event-driven pattern; Scheduler covers time-based work with time zones.
  • Retry named transient errors with back-off and jitter; catch everything else and alert with context.
  • Idempotent jobs make retries and redrive safe.
  • MWAA is for Airflow shops and costs money while idle; Step Functions Local is unsupported — use TestState.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page