Orchestration: Step Functions, EventBridge and MWAA
Run Harbourline's transaction pipeline as an event-driven Step Functions workflow — triggered by S3 through EventBridge, waiting on a Glue job, retrying safely and alerting on failure — and know when EventBridge Scheduler, Glue workflows or Apache Airflow (MWAA) fit better.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.StepFunctions 4.1.x) · Amazon.CDK.Lib 2.277.0 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain what orchestration adds to a pipeline (ordering, waiting, retries, error routing, audit history) and why it lives outside the Spark job.
- Trigger a Step Functions workflow from an S3 upload using an EventBridge rule, and from a timetable using EventBridge Scheduler.
- Write a state machine that runs a Glue job synchronously (
.sync), retries throttling errors with back-off, catches failures and alerts. - Choose between Standard and Express workflows, and between Step Functions, Glue workflows and Amazon MWAA.
- Test states without deploying a whole pipeline, using the TestState API.
Prerequisites
- Module 7 overview, S3 data lake design, Glue Data Catalog and crawlers
- Module 8: Messaging for EventBridge basics
- Tools: AWS CLI v2 (
learn-sandbox), Docker, optionally the CDK CLI
Why this matters in banking
In many banks the "pipeline" is a SQL Server Agent job at 02:00 that runs eleven SSIS packages one after another. When step 7 fails at 03:14, nobody knows until finance opens an empty report at 08:00, and the person on call has to work out which of steps 1–6 can be re-run safely. When a partner's file arrives late, the schedule runs on yesterday's data.
A bank needs pipelines that start when the data arrives, wait for long-running work without polling scripts, retry transient errors automatically, stop cleanly on real errors, tell a human straight away, and keep a complete history of what ran, with which inputs, and what happened — because the auditor will ask.
The idea in plain language
Orchestration is the "conductor" of a pipeline. It does not transform data itself; it decides what runs, in which order, what happens on success and on failure, and it records each step.
- AWS Step Functions runs state machines: JSON (Amazon States Language, ASL) definitions of steps (states) such as Task, Choice, Parallel, Map, Wait, Succeed and Fail. Tasks can call 200+ AWS services directly. A Standard workflow can run for up to a year and keeps a full execution history; an Express workflow is for high-volume, short (up to 5 minutes) event processing.
- Amazon EventBridge routes events: a rule matches events (for example S3 Object Created in bucket X under prefix
incoming/) and sends them to a target such as a state machine. - EventBridge Scheduler starts targets on a timetable (cron or rate expressions) with time zones, flexible windows and retries — the replacement for "a cron job somewhere".
- Amazon MWAA (Managed Workflows for Apache Airflow) runs Apache Airflow for teams that already write Python DAGs and need Airflow's operator ecosystem.
Analogy: the Glue job is the cashier counting the money. Step Functions is the branch manager's checklist: "when the van arrives (event), count the money (task), if the count is off by more than £0 call head office (choice + alert), file the slip (audit)". The checklist is kept, signed, for every van.
What problem the AWS service solves
| Need | Step Functions feature | Note |
|---|---|---|
| Wait for a Glue job without a polling script | Service integration glue:startJobRun.sync | The state waits (Standard workflows only) and fails if the job fails |
| Retry throttling, not business errors | Retry with ErrorEquals, IntervalSeconds, BackoffRate, MaxAttempts, JitterStrategy | Match specific error names first |
| Route failures | Catch → notify → Fail | ResultPath keeps the original input next to the error |
| Process many files | Map state (inline or distributed) | Distributed Map can iterate over S3 objects at large scale |
| Audit history | Execution history (Standard) + CloudWatch Logs | Standard history is retained for 90 days after completion; ship logs for longer retention |
| Data passing | JSONata (newer) or JSONPath | JSONata is recommended for new workflows; this lesson uses JSONPath because the CDK lab uses it |
Limits to know: execution history of a Standard workflow is limited to 25,000 events; state input/output payloads are limited to 256 KB — pass S3 keys, not data. Standard workflows bill per state transition; Express bill per request and duration.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Data Factory pipeline | Step Functions state machine (or Glue workflow) | Step Functions is general purpose: Glue, Lambda, ECS, Athena, DynamoDB, SNS, HTTP APIs |
| ADF storage event trigger | EventBridge rule on S3 Object Created | The bucket must have EventBridge notifications turned on |
| ADF schedule trigger / SQL Agent schedule | EventBridge Scheduler | Time-zone aware, flexible time windows, built-in retries and DLQ |
| Durable Functions orchestrator | Step Functions Standard workflow | Defined in ASL (JSON/YAML) or with CDK, not in C# code; visual graph in the console |
| Logic Apps | Step Functions + EventBridge Pipes | Similar "connectors" idea through service integrations |
| ADF Until/ForEach activities | Map state, Choice + Wait loop | Distributed Map reads lists straight from S3 |
| Apache Airflow on Azure (Workflow Orchestration Manager) | Amazon MWAA | Both managed Airflow; MWAA environments bill per hour even when idle |
Architecture
Security-boundary view. EventBridge assumes a role that may only states:StartExecution on this one state machine. The state machine's role may only start this Glue job, read its status and publish to the alerts topic. The Glue job runs with its own least-privilege role. Execution data is not written to logs (IncludeExecutionData = false) because inputs can contain file names that reveal partner identities.
Step-by-step implementation
Turn on EventBridge for the raw bucket
S3 sends events to EventBridge only when the bucket property Amazon EventBridge is on. The lab CDK stack sets EventBridgeEnabled = true.
Write the rule
Match source = aws.s3, detail-type = Object Created, the bucket name, and object.key with prefix incoming/. Anything else (for example the job's own outputs in other buckets) cannot trigger a loop.
Write the state machine
One Task state that calls Glue with .sync, passing --raw_bucket and --raw_key from the event. A Retry for Glue throttling and concurrency errors, a Catch that publishes to SNS and ends in Fail.
Make re-runs safe
Orchestration retries are only safe if the job is idempotent. The lab's Glue job derives its run id from bucket + key + version id and exits early if the audit record already exists.
Add a schedule for time-based work
Daily reports (for example refreshing a Quick Sight dataset or running a reconciliation query) use EventBridge Scheduler with Europe/London time zone so British Summer Time is handled for you.
Test
Use the console's Test state feature or the TestState API to run the NotifyFailure state with a sample input, then upload a sample file and watch the execution graph.
AWS Console instructions
- S3 → hl-m07-raw-… → Properties → Amazon EventBridge → Edit → On.
- Step Functions → State machines → Create state machine → Blank. Switch to Code, paste the ASL below, choose type Standard, and under Config select logging level ALL with Include execution data unticked, and an encrypted log group.
- Amazon EventBridge → Rules → Create rule on the default bus. Event pattern: paste the JSON below. Target: Step Functions state machine
hl-m07-transaction-pipeline; let EventBridge create a role. - EventBridge → Scheduler → Schedules → Create schedule: recurring, cron
0 6 * * ? *, time zone Europe/London, flexible time window Off, target Templated target → Amazon Athena StartQueryExecution (or any universal target), retry policy 3 attempts, dead-letter queue on. - Upload
transactions_messy.csvtoincoming/sftp/and open Executions to watch the run.
AWS CLI
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
# The event pattern used by the rule (also handy for testing patterns).
cat > pattern.json <<EOF
{ "source": ["aws.s3"], "detail-type": ["Object Created"],
"detail": { "bucket": { "name": ["hl-m07-raw-${ACCOUNT}"] },
"object": { "key": [ { "prefix": "incoming/" } ] } } }
EOF
aws events test-event-pattern --event-pattern file://pattern.json \
--event '{"id":"1","account":"'"$ACCOUNT"'","source":"aws.s3","time":"2026-11-18T06:00:00Z","region":"us-east-1","resources":[],"detail-type":"Object Created","detail":{"bucket":{"name":"hl-m07-raw-'"$ACCOUNT"'"},"object":{"key":"incoming/sftp/a.csv"}}}'
# Start an execution by hand with a fake S3 event (useful before the rule exists).
aws stepfunctions start-execution \
--state-machine-arn "arn:aws:states:us-east-1:${ACCOUNT}:stateMachine:hl-m07-transaction-pipeline" \
--name "manual-$(date +%Y%m%d%H%M%S)" \
--input '{"detail":{"bucket":{"name":"hl-m07-raw-'"$ACCOUNT"'"},"object":{"key":"incoming/sftp/transactions_messy.csv"}}}'
# Test one state in isolation (no state machine run needed).
aws stepfunctions test-state \
--definition '{"Type":"Pass","Parameters":{"file.$":"$.detail.object.key"},"End":true}' \
--role-arn "arn:aws:iam::${ACCOUNT}:role/hl-m07-sfn-test" \
--input '{"detail":{"object":{"key":"incoming/sftp/a.csv"}}}'
# A daily schedule at 06:00 London time that starts the reconciliation report query.
aws scheduler create-schedule --name hl-m07-daily-recon \
--schedule-expression "cron(0 6 * * ? *)" --schedule-expression-timezone Europe/London \
--flexible-time-window Mode=OFF \
--target '{"Arn":"arn:aws:scheduler:::aws-sdk:athena:startQueryExecution","RoleArn":"arn:aws:iam::'"$ACCOUNT"':role/hl-m07-scheduler","Input":"{\"QueryString\":\"SELECT * FROM hl_m07_gold.v_reconciliation WHERE status = '\''BREAK'\''\",\"WorkGroup\":\"hl-m07-analysts\"}"}'Infrastructure as Code
From labs/m07-transaction-pipeline/infra/cdk/Harbourline.DataPipeline.Infra/DataLakeStack.cs:
var runEtl = new GlueStartJobRun(this, "ValidateAndTransform", new GlueStartJobRunProps
{
GlueJobName = EtlJobName,
IntegrationPattern = IntegrationPattern.RUN_JOB, // ".sync"
Arguments = TaskInput.FromObject(new Dictionary<string, object>
{
["--raw_bucket"] = JsonPath.StringAt("$.detail.bucket.name"),
["--raw_key"] = JsonPath.StringAt("$.detail.object.key"),
}),
ResultPath = "$.glue",
});
runEtl.AddRetry(new RetryProps
{
Errors = ["Glue.ConcurrentRunsExceededException", "Glue.AWSGlueException"],
Interval = Duration.Seconds(30), MaxAttempts = 3, BackoffRate = 2,
});
var notifyFailure = new SnsPublish(this, "NotifyFailure", new SnsPublishProps
{
Topic = alerts,
Message = TaskInput.FromObject(new Dictionary<string, object>
{
["file"] = JsonPath.StringAt("$.detail.object.key"),
["error"] = JsonPath.StringAt("$.error.Error"),
["execution"] = JsonPath.StringAt("$$.Execution.Id"),
}),
ResultPath = JsonPath.DISCARD,
});
runEtl.AddCatch(notifyFailure.Next(new Fail(this, "RunFailed")), new CatchProps { ResultPath = "$.error" });
var workflow = new StateMachine(this, "Workflow", new StateMachineProps
{
StateMachineName = "hl-m07-transaction-pipeline",
StateMachineType = StateMachineType.STANDARD,
DefinitionBody = DefinitionBody.FromChainable(runEtl.Next(new Succeed(this, "RunSucceeded"))),
Logs = new LogOptions { Destination = workflowLogs, Level = SfnLogLevel.ALL, IncludeExecutionData = false },
TracingEnabled = true,
});
_ = new Rule(this, "RawObjectCreated", new RuleProps
{
EventPattern = new EventPattern
{
Source = ["aws.s3"],
DetailType = ["Object Created"],
Detail = new Dictionary<string, object>
{
["bucket"] = new Dictionary<string, object> { ["name"] = new[] { raw.BucketName } },
["object"] = new Dictionary<string, object> { ["key"] = new object[] { new Dictionary<string, object> { ["prefix"] = "incoming/" } } },
},
},
Targets = [new SfnStateMachine(workflow)],
});Complete code
An operations tool in C# that lists today's pipeline executions, shows failures with their error cause, and can redrive a failed execution (Step Functions redrive restarts a failed Standard execution from the failed state, keeping the successful steps).
// dotnet add package AWSSDK.StepFunctions (v4), plus AWSSDK.SSO and AWSSDK.SSOOIDC for the SSO profile
using Amazon.StepFunctions;
using Amazon.StepFunctions.Model;
var stateMachineArn = args.Length > 0
? args[0]
: throw new ArgumentException("usage: PipelineRuns <state-machine-arn> [--redrive <execution-arn>]");
using var sfn = new AmazonStepFunctionsClient(); // AWS_PROFILE=learn-sandbox
if (args.Length == 3 && args[1] == "--redrive")
{
// Only for failures that are now fixed (for example a missing permission). The Glue job is idempotent.
var redrive = await sfn.RedriveExecutionAsync(new RedriveExecutionRequest { ExecutionArn = args[2] });
Console.WriteLine($"Redriven at {redrive.RedriveDate:u}");
return 0;
}
var since = DateTime.UtcNow.Date;
string? token = null;
do
{
var page = await sfn.ListExecutionsAsync(new ListExecutionsRequest
{
StateMachineArn = stateMachineArn,
MaxResults = 100,
NextToken = token,
});
foreach (var e in page.Executions.Where(e => e.StartDate >= since))
{
Console.Write($"{e.StartDate:HH:mm:ss} {e.Status,-10} {e.Name}");
if (e.Status == ExecutionStatus.FAILED)
{
var detail = await sfn.DescribeExecutionAsync(new DescribeExecutionRequest { ExecutionArn = e.ExecutionArn });
Console.Write($" error={detail.Error} cause={Truncate(detail.Cause, 120)}");
}
Console.WriteLine();
}
token = page.NextToken;
}
while (token is not null);
return 0;
static string Truncate(string? s, int max) => s is null ? "" : s.Length <= max ? s : s[..max] + "…";Code explained line by line
GlueStartJobRunwithIntegrationPattern.RUN_JOB— compiles toarn:aws:states:::glue:startJobRun.sync. Step Functions starts the job and waits for its terminal state; a failed or timed-out job becomes a task failure (States.TaskFailed).JsonPath.StringAt("$.detail.bucket.name")— reads values from the EventBridge event, which is the execution input. Glue receives them as--raw_bucketand--raw_key.ResultPath = "$.glue"— adds the Glue result to the input instead of replacing it, so later states still see$.detail.AddRetrywith named errors — retries concurrency and transient Glue errors only. Data errors (DQ_FAILED,SCHEMA_REJECTED) are not retried: retrying bad data gives the same bad result."JitterStrategy": "FULL"(ASL tab) — randomises retry delays so many executions do not retry at the same instant.AddCatch(... ResultPath = "$.error")— on any remaining error, keep the input and add the error, then go toNotifyFailure.$$.Execution.Id— the double dollar reads the context object; the execution ARN lets the on-call engineer open the run directly.ResultPath = JsonPath.DISCARD— the SNS response is not needed downstream.IncludeExecutionData = false— logs show state transitions and errors but not payloads.RedriveExecutionAsync(C# tool) — restarts a failed execution from the failed state. Use it after fixing the cause; it reuses the original input, so idempotency matters.ListExecutionsAsyncwithNextToken— pagination; the console shows the same list.
Glue workflows, MWAA and when to use what
| Option | Strengths | Weaknesses | Choose when |
|---|---|---|---|
| Step Functions | Serverless, pay per transition, any AWS service, visual history, .sync waits, redrive | ASL is another language; 256 KB payloads | Default for AWS-native pipelines, mixed services (Glue + Lambda + Athena + SNS) |
| Glue workflows and triggers | Built into Glue; chains crawlers and jobs; conditional triggers | Glue-only; limited error routing and alerting | A small, Glue-only chain (crawler → job → crawler) |
| Amazon MWAA | Airflow DAGs in Python, huge operator library, backfills, familiar to data teams | Environment bills per hour even when idle; you manage DAG code, plugins and versions | The bank already runs Airflow, or needs Airflow's scheduling and backfill semantics across many systems |
| EventBridge Scheduler | Time-zone aware schedules, retries, DLQ | Starts things; does not orchestrate steps | Anything that is "run X at time T" |
Free alternative for Airflow: run Apache Airflow locally with the official Docker Compose file from the Airflow project and write the same pipeline as a DAG; never leave an MWAA environment running for learning.
Testing locally
AWS documents Step Functions Local as unsupported and without feature parity, and points to the TestState API instead. TestState runs a single state's definition against real AWS integrations (it needs the states:TestState permission and network access) without creating a state machine. For offline unit tests, keep logic out of the state machine: the Glue job's rules are tested with pytest, the Lambda handlers with xUnit, and the state machine is a thin, readable sequence that you test with TestState and one end-to-end upload.
Security considerations
⚠ Learning shortcut — not for production
The lab's state machine role is created by CDK grants and the alerts go to an e-mail subscription. In production: one role per state machine with exactly the Glue job ARN, the SNS topic ARN and the KMS key; alerts go to the incident management tool via SNS → HTTPS or a chat integration; and the RunFailed path also writes an audit record so a failed run is visible in the same report as successful ones.
- Least-privilege trigger roles: the EventBridge target role may only
states:StartExecutionon one ARN. - No secrets in execution input. Inputs are stored in execution history. Pass references (secret names, S3 keys), never values.
- Encrypt the log group and, for Standard workflows, consider a customer managed key for the state machine's data at rest ⚠ VERIFY the current Step Functions KMS options for your design.
- Loop prevention: the rule matches only
incoming/in the raw bucket; the job writes to other buckets.
Banking use case
Saltmarsh Payroll uploads a file at 21:30. EventBridge starts the workflow within seconds; the Glue job validates, masks and reconciles; if reconciliation shows a BREAK, the run succeeds technically but the reconciliation view flags it, and the 06:00 scheduled query e-mails the operations team the list of breaks. If the Glue job fails — for example the partner changed the header — the workflow catches SCHEMA_REJECTED, posts the file name and execution link to the alerts topic, and the on-call engineer sees exactly which file failed and why. Every execution's history is the evidence auditors ask for.
Common mistakes
- Polling Glue from Lambda in a loop → pays for idle Lambda time and adds code. Use
.sync. - Retrying
States.ALL→ retries data errors three times, tripling cost and noise. Retry named transient errors only. - Putting large payloads in state input → hits 256 KB. Pass S3 keys.
- Express workflow for a 20-minute Glue job → Express runs at most 5 minutes and does not support
.syncjob waits. Use Standard. - Forgetting the bucket's EventBridge setting → the rule never fires.
- Schedules in UTC for UK business processes → reports shift by an hour twice a year. Use Scheduler's time zone.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| No execution after upload | EventBridge off on bucket; key outside incoming/; rule disabled | Check bucket property and rule pattern with test-event-pattern |
Execution fails with States.Runtime on a path | The JSONPath does not exist in the input | Inspect the input in execution history; fix the path |
Glue.ConcurrentRunsExceededException | More files than MaxConcurrentRuns | Retry with back-off (configured) or raise the limit |
Task fails immediately with AccessDenied | State machine role lacks glue:StartJobRun/GetJobRun | Add both on the job ARN |
| Alert e-mail never arrives | Subscription not confirmed, or KMS key policy blocks SNS | Confirm subscription; allow the publishing role to use the key |
| Redrive not allowed | Execution not failed, too old, or Express type | Redrive applies to failed Standard executions within the allowed period ⚠ VERIFY current window |
Cost and cleanup
Step Functions Standard is billed per state transition (the lab uses about 3–4 per file; the first 4,000 transitions per month are in the always-free tier ⚠ VERIFY current free tier on the pricing page). EventBridge rules for AWS service events are free to match; Scheduler invocations have a free monthly allowance. The cost of a run is dominated by the Glue job (about USD 0.03). MWAA bills per environment-hour (hundreds of dollars a month even when idle) — design-only in this module.
# Lab stack: cdk destroy removes rule, state machine, topic and log group.
cd labs/m07-transaction-pipeline/infra/cdk && cdk destroy HlM07-DataLake --profile learn-sandbox
# If you created the schedule by hand:
aws scheduler delete-schedule --name hl-m07-daily-recon --profile learn-sandboxExercises
- 🟢 Guided: Add a
Choicestate after the Glue job that reads a flag from the input ($.dryRun) and skips publishing. Test it with TestState. - 🟡 Challenge: Turn the workflow into a daily batch workflow: a Distributed
Mapstate over every object underincoming/from yesterday, withMaxConcurrency3, running the same Glue job per file. - 🔴 Live problem: A partner re-sent yesterday's file with the same name "to be safe". Finance now sees doubled totals in a downstream system that reads curated data directly. Explain how the lab pipeline behaves, why your gold reports are fine, and what you change so downstream consumers are protected too.
Assignment
Export your state machine definition, a screenshot of one successful and one failed execution graph, and a one-page runbook: what the alert means, how to read the execution, when to redrive, and when to ask the partner for a corrected file.
Solutions
"CheckDryRun": {
"Type": "Choice",
"Choices": [ { "Variable": "$.dryRun", "BooleanEquals": true, "Next": "RunSucceeded" } ],
"Default": "NotifyCompletion"
}Test with aws stepfunctions test-state --definition '<the Choice JSON>' --input '{"dryRun":true}' and check nextState in the response. A Choice state needs no role to test because it calls no service. If $.dryRun may be missing, add "IsPresent": true checks first, otherwise the state fails with a runtime error.
Use a Map state with ItemReader reading s3:listObjectsV2 for Prefix: incoming/<yesterday>/, ItemProcessor containing the existing ValidateAndTransform task (with Arguments built from $.Key), MaxConcurrency: 3 to stay within the Glue job's MaxConcurrentRuns, and ToleratedFailurePercentage: 0 so any failed file fails the batch. Trigger it with EventBridge Scheduler at 02:00 Europe/London. Each child run is still one file, so idempotency and audit records work unchanged.
Same name but a new upload creates a new S3 version, so the lab job computes a new run id and processes it: curated now contains the rows twice, with different ingest_run_id values. Gold per-run tables are correct per run, but anything summing curated across runs double counts. Fixes: (1) de-duplicate on transaction_id and idempotency_key across runs — for example a curated Iceberg table with MERGE on transaction_id; (2) compute a content hash of the file and reject a byte-identical file already processed (record it in the audit table); (3) give downstream consumers a view that selects the latest version of each transaction_id. Tell the partner that resends must use a new batch id.
Knowledge check
Answer: .sync makes the state wait for the job to finish and fail if it fails, so errors are caught and alerted, and the next step runs only after success. Without it the workflow "succeeds" while the job is still running.
Answer: Standard: runs longer than 5 minutes, uses .sync, needs exactly-once execution semantics and a full history for audit. Express suits high-volume, short, event-processing flows.
Answer: In tested code (the job or a Lambda). Keep the state machine about flow: run, wait, branch on outcomes, retry, alert. Rules in ASL are hard to unit test.
Answer: When a team already writes Airflow DAGs, needs Airflow features (backfills, sensors across many systems, operator ecosystem), and accepts paying for an always-on environment.
Interview questions
- "How do you make a pipeline restartable?" — Idempotent steps (deterministic outputs, run ids, conditional writes), orchestration with retries for transient errors only, redrive from the failed step, and audit records written last.
- "Event-driven or scheduled?" — Event-driven for data arrival (process as soon as it lands); scheduled for time-based business processes (06:00 report). Most banks need both.
- "How do you alert on pipeline failures?" — Catch in the workflow, publish a structured message (file, error, execution link) to SNS, plus CloudWatch alarms on
ExecutionsFailedas a safety net. - "Step Functions vs Airflow?" — Serverless and AWS-integrated versus Python DAGs and ecosystem; cost model per transition versus per environment-hour.
Certification practice
A data engineer must run an AWS Glue job whenever a partner uploads a file to an S3 prefix, notify the operations team if the job fails, and keep a visual history of each run with minimal custom code. Which solution is best?
A. An S3 event notification that invokes a Lambda function, which starts the Glue job and polls it every 30 seconds.
B. An EventBridge rule on S3 Object Created events targeting a Step Functions Standard workflow that runs the Glue job with the .sync integration and publishes to SNS in a Catch.
C. A cron job on an EC2 instance that lists the bucket every minute.
D. A Glue crawler scheduled every hour.
Answer: B. Event-driven, waits for completion, built-in error routing and history. A adds polling code and Lambda time limits. C is a server and polling. D catalogues data but does not run the job or alert.
A Step Functions task that starts a Glue job occasionally fails with Glue.ConcurrentRunsExceededException during busy periods, and sometimes fails because input data is invalid. How should retries be configured?
A. Retry States.ALL 10 times with no back-off.
B. Retry only Glue.ConcurrentRunsExceededException with exponential back-off and jitter, and catch other errors to notify operators.
C. Disable retries and re-run failed executions manually.
D. Increase the Glue job timeout.
Answer: B. Retry transient errors with back-off; route data errors to people. A wastes cost on errors that will not fix themselves. C adds toil. D does not address either error.
A UK bank must generate a report at 06:00 London time every day, including during British Summer Time, with retries if the target is unavailable. Which approach requires the least effort?
A. An EventBridge rule with cron(0 6 * * ? *) in UTC.
B. EventBridge Scheduler with the expression cron(0 6 * * ? *), time zone Europe/London, a retry policy and a dead-letter queue.
C. A Windows Task Scheduler job on an EC2 instance.
D. A Lambda function that sleeps until 06:00.
Answer: B. Scheduler supports time zones and retries natively. A drifts by an hour in summer. C adds a server. D cannot run that long and wastes money.
Completion checklist
- I can explain what orchestration adds beyond the job itself
- I wrote (or read line by line) a state machine with
.sync,RetryandCatch - I tested an event pattern and a single state without a full deployment
- I can choose between Step Functions, Glue workflows, MWAA and Scheduler
- I destroyed the stack and any hand-made schedules
Key takeaways
- Orchestration owns order, waiting, retries, error routing and history; the job owns the data logic.
- S3 → EventBridge → Step Functions Standard → Glue
.syncis the event-driven pattern; Scheduler covers time-based work with time zones. - Retry named transient errors with back-off and jitter; catch everything else and alert with context.
- Idempotent jobs make retries and redrive safe.
- MWAA is for Airflow shops and costs money while idle; Step Functions Local is unsupported — use TestState.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Streaming ingestion: Kinesis Data Streams, Data Firehose and MSK
Move Harbourline transactions from batch files to a near-real-time stream: produce to Kinesis from C#, consume with a .NET Lambda that reports partial batch failures, deliver to S3 with Data Firehose, and know when Amazon MSK is the better fit.
Change data capture from SQL Server with AWS DMS
Stream inserts, updates and deletes from Harbourline's SQL Server core-banking tables into the S3 data lake with AWS DMS full load plus CDC, then merge them into a current-state table that copes with late and out-of-order changes.