Playbook: end-of-day batch overrunning its window
Find the slow stage of a nightly batch, tell data skew from saturation from waiting, buy time without skipping controls, and add progress-based alarms so the next overrun is caught at 01:00 instead of 05:00.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · AWS Step Functions · CloudWatch embedded metric format · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this playbook you can:
- Break an overrunning nightly batch into stages and shards, and identify the long pole (the one slice of work that decides the finish time).
- Classify a slowdown as skew, saturation, waiting or growth, using specific metrics for each.
- Choose safe ways to buy time (more parallelism, deferring non-critical stages) and recognise the ways that look attractive but break controls.
- Instrument a .NET batch worker with stage timers that emit CloudWatch embedded-metric-format (EMF) lines, and analyse shard durations for skew.
- Create progress-based alarms ("behind schedule") instead of failure-only alarms.
- Communicate an overrun to downstream owners with a revised estimate and an audit record.
Prerequisites
- Troubleshooting production issues methodically
- CloudWatch metrics and EMF and Logs Insights queries
- Alarms and composite alarms
- Performance optimisation
- Playbook: duplicate payment on client retry, because re-running stages safely depends on idempotency
- Tools: AWS CLI v2 with the
learn-sandboxprofile. The local reproduction is plain .NET 10, no AWS needed
Why this matters in banking
Harbourline Bank's end-of-day (EOD) batch starts at 22:00 once the day's cut-off passes. It posts interest and fees, generates statements, reconciles against Core Banking and produces a file for the regulatory reporting team, who need it by 05:30 to meet their own deadline. One Thursday the batch is still on stage three at 04:40. Nothing has failed: every log line is green. The batch is simply slower than the window.
A late batch is a business incident without an error message. Account balances shown in the app at 07:00 may be stale. Statements arrive late. A regulatory file can miss its deadline, which is a conversation with a regulator nobody wants. The skill here is to treat time as the failure: measure where it goes, decide quickly what is safe to defer, and tell people early. Engineers who only look for exceptions find nothing and lose an hour.
The idea in plain language
A batch is a pipeline of stages, and each stage is split into shards (slices of the data, for example by account ID range) that run in parallel. The finish time of a stage is the time of its slowest shard; the finish time of the batch is the sum of its stages (plus any waiting). So the batch can be late for four different reasons:
| Class | What it means | Signature |
|---|---|---|
| Skew | Work is unevenly split. One shard has a giant account or customer | One shard's duration is many times the median. Most workers sit idle waiting for it |
| Saturation | A shared resource is full: database CPU, IOPS, connection limit, DynamoDB capacity, Lambda concurrency | Every shard slows down together. A resource graph is flat at its ceiling |
| Waiting | Workers are blocked by something else: a lock, a partner API, a queue, a missing upstream file | Low CPU everywhere, long gaps in logs, timeouts, retries |
| Growth | The batch is simply doing more work than last year | Nightly stage durations trend steadily upward, no sudden change |
Analogy: A coach-load of 40 guests checks out of a hotel. The checkout desks are the workers. If all desks are busy and the queue is long, that is saturation. If 39 guests finish in five minutes and one guest disputes a minibar bill for an hour, that is skew. If the desks are idle because the till system is down, that is waiting. If the hotel simply took on more guests than last year, that is growth. The fix for each is different, and adding desks only helps the first.
What problem the AWS services solve
AWS gives you the observation tools and the elasticity to change parallelism quickly. It does not make an inefficient batch fast.
- AWS Step Functions records every stage with start and end times in its execution history, so the long pole is visible without any code. A Distributed Map state can fan out to a very large number of child executions for sharded work (⚠ VERIFY the current concurrency maximum). A Standard workflow execution history is capped at 25,000 events (⚠ VERIFY), which is one reason large fan-outs use Distributed Map instead of an inline Map.
- Amazon CloudWatch metrics for the data stores (RDS, DynamoDB, SQS, Lambda, ECS) show which resource saturates. Performance Insights (or the current Database Insights naming, ⚠ VERIFY) shows what an RDS database is waiting on.
- CloudWatch embedded metric format (EMF) lets your own worker emit stage metrics as structured log lines with no extra API calls.
- Amazon EventBridge Scheduler starts the batch at a fixed time, and AWS Batch or ECS supply compute you can scale out.
Limits and sharp edges to respect during an overrun:
- Scaling out can move the bottleneck. Doubling Lambda concurrency or ECS tasks doubles the load on the database, which may then throttle or run out of connections. Check the connection limit and CPU headroom first.
- Lambda account concurrency defaults to 1,000 concurrent executions per Region for many accounts, and new accounts can start lower (⚠ VERIFY current default). A quota increase request takes time, so it is not an emergency tool.
- DynamoDB limits a single partition to about 1,000 write units and 3,000 read units per second (⚠ VERIFY). A hot key is skew at the storage layer, and on-demand mode does not remove it.
- Instance changes mid-run (for example resizing an RDS instance) can cause a failover or restart. That is rarely a good mid-batch move.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| SQL Agent job with steps and a job history | Step Functions state machine with execution history | History is per state, with input and output you can inspect. No server to babysit |
| SSIS package with parallel data flows | Step Functions Distributed Map or AWS Glue | Shards are separate child executions, so skew shows up as one slow child |
| Azure Data Factory pipeline monitor | Step Functions console plus CloudWatch | You build the "behind schedule" alarm yourself |
| Azure Batch pool autoscale | AWS Batch, ECS service auto scaling, Lambda concurrency | Scaling is quick, but each layer has quotas and shared downstream limits |
Azure Monitor Application Insights custom metrics (TrackMetric) | CloudWatch EMF log lines | A single JSON line in stdout becomes a metric. Dimensions affect cost, so keep them few |
| SQL Server Query Store and wait stats | RDS Performance Insights top waits | Both show where the database spends time, which is how you see lock or IO waits |
Architecture
The batch and the checkpoints that matter
Decision tree: why is it slow?
Step-by-step implementation
Minutes 0 to 10: locate the stage and set the clock
- Open the Step Functions execution and note which state is running and when it started. Compare with the same stage on the last five nights (durations are in the history).
- Compute how late you are: the remaining work divided by the current rate gives a projected finish time. Write it down with the time you calculated it. A rough number is enough: "stage 3 is 40 percent done after 70 minutes, so about 105 minutes remain".
- Decide who needs to hear this now: the regulatory reporting owner and the operations manager. Send a short message with the projection and the time of your next update. Silence is the worst option.
Minutes 10 to 25: classify the slowdown
Use the decision tree above, and prove each branch with a number.
- Skew: list shard durations for the running stage. Anything over about three times the median deserves a look.
- Saturation: check the shared resources at the same time window (queries are in the CLI section): database CPU and connections, IO, DynamoDB throttles, Lambda throttles and concurrency, ECS task CPU and memory.
- Waiting: look for gaps. If workers log nothing for minutes, find what they were waiting for: a lock, a partner API call, an SQS message that never arrived.
- Growth: chart
StageDurationSecondsfor 30 nights. A slow upward line points to growth, and no overnight fix will cure it; you need capacity planning and a code change.
Minutes 25 to 45: buy time safely
Pick from this list in order. Each item has a rule.
| Move | Safe when | Danger |
|---|---|---|
| Raise parallelism for the slow stage | The database or store has headroom (CPU below about 60 percent, connections well under the limit) | Moves the bottleneck and can throttle the whole account |
| Add workers only for the unfinished shards | Shards are independent and idempotent | Do not start a second copy of a shard that is still running |
| Split the long-pole shard and run the pieces in parallel | The unit of work can be divided (by date range, by account sub-range) | Never split in a way that changes totals or ordering the stage depends on |
| Defer pre-agreed non-critical stages (analytics exports, marketing extracts) | They are listed as deferrable in the runbook and the owners agreed in advance | Never defer reconciliation, sanctions screening or the regulatory file without written sign-off |
| Pause other heavy database users (reporting queries, backups outside the managed window) | You can identify them in Performance Insights | Do not kill sessions you do not understand |
What you must not do: kill the batch and restart it from the top, skip a control stage to save time, edit data by hand to unblock a shard, or deploy new code to the running batch without the normal approval. A restart from zero costs the work already done, and a skipped control is an audit finding.
After the window: root cause and prevention
Write the cause by class (skew, saturation, waiting or growth), prove it with the numbers you captured, then fix it so it cannot recur: re-shard, add the alarm, fix the lock, plan capacity. The Exercises walk through a skew case.
AWS Console instructions
- Console → Step Functions → State machines →
hl-eod-batch→ Executions → (running execution). The Graph view colours states by status. Open Execution event history to read each state's start and end time. - Console → CloudWatch → Metrics → All metrics →
Harbourline/Batch. GraphStageDurationSecondsbyStagefor the last 30 days. - Console → RDS → Databases → your cluster → Monitoring. Check CPUUtilization, DatabaseConnections and ReadIOPS/WriteIOPS. For waits, open Performance Insights and group the load by Waits.
- Console → DynamoDB → Tables → Monitor. Check Throttled requests and Consumed write capacity.
- Console → Lambda → Functions → Monitor. Check Throttles and Concurrent executions.
- Console → CloudWatch → Logs → Logs Insights. Run the shard queries below.
AWS CLI
# 1. Find the running executions and see which state they are in.
aws stepfunctions list-executions \
--state-machine-arn arn:aws:states:us-east-1:111122223333:stateMachine:hl-eod-batch \
--status-filter RUNNING --profile learn-sandbox
# 2. Read the history of one execution: state names and timestamps (compact view).
aws stepfunctions get-execution-history \
--execution-arn <execution-arn> --max-results 200 \
--query 'events[?type==`TaskStateEntered` || type==`TaskStateExited`].[timestamp,type,stateEnteredEventDetails.name,stateExitedEventDetails.name]' \
--output table --profile learn-sandbox
# 3. Database pressure during the batch window.
aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name CPUUtilization \
--dimensions Name=DBClusterIdentifier,Value=hl-core-aurora \
--start-time $(date -u -d '4 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
--period 300 --statistics Average Maximum --profile learn-sandbox
aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name DatabaseConnections \
--dimensions Name=DBClusterIdentifier,Value=hl-core-aurora \
--start-time $(date -u -d '4 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
--period 300 --statistics Maximum --profile learn-sandbox
# 4. DynamoDB write throttles (sum per 5 minutes).
aws cloudwatch get-metric-statistics --namespace AWS/DynamoDB --metric-name WriteThrottleEvents \
--dimensions Name=TableName,Value=hl-ledger \
--start-time $(date -u -d '4 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
--period 300 --statistics Sum --profile learn-sandbox
# 5. Backlog in the work queue: how old is the oldest unprocessed shard message?
aws cloudwatch get-metric-statistics --namespace AWS/SQS --metric-name ApproximateAgeOfOldestMessage \
--dimensions Name=QueueName,Value=hl-eod-statements \
--start-time $(date -u -d '4 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
--period 300 --statistics Maximum --profile learn-sandbox
# 6. Raise parallelism of the statements service (ECS) for the unfinished work only.
aws ecs update-service --cluster hl-batch --service hl-statements-worker --desired-count 24 --profile learn-sandboxLogs Insights queries for shards (the worker logs the EMF lines shown below):
# Slowest shards in the current run
fields runId, stage, shardId, StageDurationSeconds, recordCount
| filter runId = "2026-11-12" and stage = "statements"
| sort StageDurationSeconds desc
| limit 10
# Skew indicator per stage: max versus median
fields stage, StageDurationSeconds
| filter runId = "2026-11-12"
| stats pct(StageDurationSeconds, 50) as median, max(StageDurationSeconds) as slowest, count(*) as shards by stageInfrastructure as Code
Alarm on a progress metric rather than on errors. The worker publishes PercentComplete (0 to 100) for each stage; the alarm fires when the batch is behind the expected curve at a given hour. A simpler start is an alarm on stage duration, shown here.
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.CloudWatch.Actions;
using Amazon.CDK.AWS.SNS;
using Constructs;
public class BatchAlarmsStack : Stack
{
public BatchAlarmsStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
{
var topic = new Topic(this, "BatchOps", new TopicProps { TopicName = "hl-batch-ops" });
var statementsDuration = new Metric(new MetricProps
{
Namespace = "Harbourline/Batch",
MetricName = "StageDurationSeconds",
DimensionsMap = new Dictionary<string, string> { ["Stage"] = "statements" },
Statistic = "Maximum",
Period = Duration.Minutes(5)
});
var alarm = new Alarm(this, "StatementsSlow", new AlarmProps
{
AlarmName = "hl-eod-statements-shard-slow",
AlarmDescription = "Any statements shard took over 45 minutes. Open the EOD batch playbook.",
Metric = statementsDuration,
Threshold = 45 * 60,
EvaluationPeriods = 1,
ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING
});
alarm.AddAlarmAction(new SnsAction(topic));
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-eod");
}
}Complete code
Two small pieces: a stage timer that emits an EMF line when a shard finishes, and a skew analyser you can run on the durations afterwards (or in a unit test).
using System.Diagnostics;
using System.Text.Json;
namespace Harbourline.Batch;
/// <summary>Times one shard of one stage and writes a CloudWatch EMF line to stdout.</summary>
public sealed class StageTimer : IDisposable
{
private readonly Stopwatch _sw = Stopwatch.StartNew();
private readonly string _runId, _stage, _shardId;
private readonly TextWriter _out;
public long RecordCount { get; set; }
public StageTimer(string runId, string stage, string shardId, TextWriter? output = null)
{
_runId = runId; _stage = stage; _shardId = shardId;
_out = output ?? Console.Out;
}
public void Dispose()
{
_sw.Stop();
var line = new Dictionary<string, object>
{
["_aws"] = new Dictionary<string, object>
{
["Timestamp"] = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(),
["CloudWatchMetrics"] = new[]
{
new Dictionary<string, object>
{
["Namespace"] = "Harbourline/Batch",
// Only Stage is a dimension. shardId and runId stay as plain log fields
// so they do not create thousands of billable metric series.
["Dimensions"] = new[] { new[] { "Stage" } },
["Metrics"] = new[]
{
new Dictionary<string, object> { ["Name"] = "StageDurationSeconds", ["Unit"] = "Seconds" },
new Dictionary<string, object> { ["Name"] = "RecordCount", ["Unit"] = "Count" }
}
}
}
},
["Stage"] = _stage,
["runId"] = _runId,
["shardId"] = _shardId,
["StageDurationSeconds"] = Math.Round(_sw.Elapsed.TotalSeconds, 3),
["RecordCount"] = RecordCount
};
_out.WriteLine(JsonSerializer.Serialize(line));
}
}namespace Harbourline.Batch;
public sealed record ShardTiming(string ShardId, double Seconds, long Records);
public sealed record SkewReport(double Median, double Slowest, double Ratio, string SlowestShard,
string Verdict, double? SecondsSavedIfPerfectlyBalanced);
public static class SkewAnalyzer
{
public static SkewReport Analyze(IReadOnlyList<ShardTiming> shards, double skewRatioThreshold = 3.0)
{
if (shards.Count == 0) throw new ArgumentException("No shards.", nameof(shards));
var ordered = shards.OrderBy(s => s.Seconds).ToList();
var median = ordered.Count % 2 == 1
? ordered[ordered.Count / 2].Seconds
: (ordered[ordered.Count / 2 - 1].Seconds + ordered[ordered.Count / 2].Seconds) / 2;
var slowest = ordered[^1];
var ratio = median > 0 ? slowest.Seconds / median : double.PositiveInfinity;
// If work were split evenly, the stage would take about the mean shard time.
var mean = ordered.Average(s => s.Seconds);
var saved = slowest.Seconds - mean;
var verdict = ratio >= skewRatioThreshold
? $"SKEW: shard {slowest.ShardId} is {ratio:F1}x the median. Split it or use work stealing."
: "No significant skew. If the stage is still slow, look at saturation or waiting.";
return new SkewReport(median, slowest.Seconds, ratio, slowest.ShardId, verdict,
ratio >= skewRatioThreshold ? saved : null);
}
}A tiny test that doubles as documentation:
using Harbourline.Batch;
using Xunit;
public class SkewAnalyzerTests
{
[Fact]
public void One_giant_account_is_reported_as_skew()
{
var shards = Enumerable.Range(0, 63).Select(i => new ShardTiming($"{i:D2}", 600, 50_000)).ToList();
shards.Add(new ShardTiming("63", 7_200, 2_000_000)); // a corporate customer with millions of rows
var report = SkewAnalyzer.Analyze(shards);
Assert.Equal("63", report.SlowestShard);
Assert.True(report.Ratio >= 3.0);
Assert.StartsWith("SKEW", report.Verdict);
Assert.NotNull(report.SecondsSavedIfPerfectlyBalanced);
}
[Fact]
public void Evenly_loaded_shards_are_not_skew()
{
var shards = Enumerable.Range(0, 8).Select(i => new ShardTiming($"{i}", 900 + i, 10_000)).ToList();
Assert.DoesNotContain("SKEW", SkewAnalyzer.Analyze(shards).Verdict);
}
}Code explained line by line
StageTimerisIDisposable, sousing var t = new StageTimer(...)times a whole block and guarantees the line is written even if the block returns early.RecordCountlets the worker add how many records it handled, so you can tell a slow shard that is big from a slow shard that is stuck (long duration with few records).- The
_awsobject is the EMF envelope. CloudWatch reads it from the log line and creates metrics from the named fields, with no API call from your code. - Only
Stageis a dimension. AddingshardIdas a dimension would create one metric series per shard per night, which is expensive and unreadable. KeepshardIdas a plain field so Logs Insights can still filter on it. SkewAnalyzeruses the median, not the mean, as the baseline, because one huge shard drags the mean up and hides itself.SecondsSavedIfPerfectlyBalancedis the slowest shard's time minus the mean: a rough estimate of how much faster the stage would be if the work were evenly split. It is an upper bound because shards cannot be split perfectly.- The tests build a realistic bad case (63 normal shards and one corporate account with 40 times the data) and assert the analyser names the right shard.
Security considerations
⚠ Learning shortcut — not for production
Under pressure it is tempting to give an engineer a broad role (for example AdministratorAccess or direct database write access) "just for tonight" to unblock the batch. Do not. Prepare a break-glass, time-boxed role in advance with only the actions the runbook needs (Step Functions StartExecution, StopExecution, ECS UpdateService, read-only metrics) and require approval and an audit record when it is used.
- Controls are part of the batch. Reconciliation, sanctions batch screening and the regulatory file are controls, not optional steps. Deferring them needs a documented, approved decision, not an engineer's judgement at 03:00.
- Do not edit data by hand to unblock a shard. Manual changes bypass the checks and leave no consistent trail. Fix the cause or hand the shard to the owner.
- Stage logs must not contain personal data. Log shard IDs, counts and timings, not account numbers or names. This also keeps the analysis available to a wider group.
- Scaling up changes the blast radius. More concurrent workers mean more concurrent credentials in use and more downstream calls. Keep each worker on its own least-privilege task role.
- Keep the incident timeline (who changed parallelism or deferred a stage, when and on whose approval). Your change management and audit teams decide what evidence they need; this lesson does not claim any regulatory standard is met.
Banking use case
Harbourline's statements stage ran 90 minutes in January and 310 minutes in November. The stage timers showed 63 shards at about ten minutes each and one shard taking 4 hours: a business customer with several million transactions in a year, hashed to the same shard as other accounts. The team did three things. First, tonight: they split that shard into 16 date-range pieces and ran them in parallel, because statements are independent per date range, and the stage finished 3 hours earlier. Second, this month: they changed sharding from account-hash to work units capped at 50,000 records, placed on an SQS queue, so workers pull the next unit when free (work stealing). Third, permanently: they alarmed on progress at 01:00 and 03:00 ("stage 3 must be 50 percent done by 01:00"), so the next overrun is a 01:00 page, not a 04:40 discovery.
Common mistakes
- Only alarming on failure. A late batch throws no exception. Alarm on duration and on progress against schedule.
- Adding workers to a skew problem. More workers do not shorten the one long shard.
- Adding workers to a saturated database. It makes things worse. Check CPU and connections first.
- Restarting from the top. Throws away completed work and may double-apply stages that are not idempotent.
- Averaging. The mean hides the long pole. Always look at the median and the maximum.
- Silence. Not telling downstream owners early. A revised estimate sent at 02:00 is a plan; a surprise at 05:15 is an incident.
- High-cardinality metric dimensions. Putting shard IDs or run IDs in dimensions makes the dashboard slow and the bill large.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| One shard far slower than the rest | Data skew (a giant account or a hot key) | Split the unit of work, cap work-unit size, use a queue so free workers pull more work |
| All shards slow after a release | New query without an index, or a plan change | Performance Insights top SQL, compare the plan, roll back the release if needed |
| Throughput flat while CPU is low | Waiting on a lock or a partner API | Check blocked sessions and outbound call latency. See Partner API timeouts |
ProvisionedThroughputExceededException or high WriteThrottleEvents | DynamoDB capacity or hot partition | Spread keys, use write sharding for hot keys, raise provisioned capacity or move to on-demand |
Lambda Throttles rising, queue age growing | Account or function concurrency limit reached | Raise reserved concurrency for the function within the account quota, or request a quota increase ahead of time |
| ECS tasks restarting during the batch | Memory limit exceeded as data grows | See ECS task keeps restarting |
| Database connections near the limit | Each worker opens its own pool | Cap the pool per worker, use RDS Proxy (⚠ VERIFY suitability for your engine), or reduce parallelism |
| No stage timing data | Timers not deployed, or EMF line malformed | Check the log group for the JSON lines. The _aws envelope must be valid JSON on one line |
Cost and cleanup
Cost label: free. The local skew exercise needs only .NET 10. The optional sandbox alarm costs about ten cents per month per standard alarm and EMF custom metrics cost a small amount per metric series (⚠ VERIFY current CloudWatch pricing). Delete the stack when you finish.
# CDK
cdk destroy BatchAlarmsStack --profile learn-sandbox
# Or by hand
aws cloudwatch delete-alarms --alarm-names hl-eod-statements-shard-slow --profile learn-sandbox
aws sns delete-topic --topic-arn arn:aws:sns:us-east-1:111122223333:hl-batch-ops --profile learn-sandbox
# Confirm nothing tagged for this lesson remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-eod --profile learn-sandboxExercises
- 🟢 Guided: Add
StageTimerandSkewAnalyzerto a console project. Generate 64 fake shard timings, make one 12 times larger, run the analyser and read the verdict. Print the JSON line from aStageTimerand check it is one valid line. - 🟡 Challenge: Write a function
ProjectFinish(DateTimeOffset start, double fractionDone, DateTimeOffset now)that returns the projected finish time. Add a rule that returns "behind schedule" if the projection is later than a deadline you pass in, with a 10 percent safety margin. Unit test it with three scenarios. - 🔴 Live problem: 03:20. The statements stage is 35 percent done. The regulatory file is due at 05:30 and depends on stage 4 (reconciliation, about 60 minutes) and stage 5 (about 20 minutes). Aurora CPU is 35 percent, connections are at 40 percent of the limit, and ECS shows 8 statements workers all busy. The product owner asks you to "skip reconciliation, we can do it tomorrow". Decide what you do and what you say.
Assignment
Add a batch-overrun section to your portfolio incident template: the four classes (skew, saturation, waiting, growth), the first-ten-minutes checklist and the "deferrable stages" table with owners. Commit incident-reports/eod-batch-overrun.md containing a worked example with fictional numbers and the skew analyser output.
Solutions
With 63 shards at about 600 seconds and one at 7,200, the median is 600 and the ratio 12, so the verdict starts with SKEW and names the slow shard. The estimated saving is roughly the slowest time minus the mean (about 6,500 seconds) which is an upper bound on what rebalancing could win. Why: the median ignores the outlier, so the ratio is a stable skew signal.
Elapsed is now - start. If fractionDone is 0.35 after 70 minutes, the total estimate is 70 / 0.35 = 200 minutes, so finish is start + 200 minutes. Guard fractionDone <= 0 (return unknown). Behind schedule when projectedFinish > deadline - 0.1 * (deadline - start). Tests: on track, borderline and clearly late. Why: a linear projection is crude (work is rarely uniform), but it is good enough to trigger a conversation, and writing it down forces a number into the room.
Remaining time: 05:30 minus 03:20 is 130 minutes. Stage 3 is 35 percent done; if it took (say) 100 minutes so far, projecting the rest takes about 185 more minutes, which would miss the deadline by a long way even before stages 4 and 5. Resources show headroom (CPU 35 percent, connections 40 percent), so this is not saturation, and workers are all busy, so it is probably growth or a big shard: add workers for the unfinished shards (for example 8 to 24) and check for a long-pole shard to split. Do not skip reconciliation on your own authority: it is a control, and the regulatory file depends on it. Instead say: "I can't skip a control without sign-off from the head of operations and finance control. Here is the projected time with and without extra workers. I'm adding workers now. I'll update everyone at 03:50. If the projection still misses 05:30, we escalate to the regulatory reporting owner for a decision on a late or partial submission." Why: you offer an action that reduces risk, give a number, name the decision owner and set the next update time.
Knowledge check
Answer: Skew. Split the slow shard into independent pieces and run them in parallel (and, longer term, cap work-unit size). Adding workers will not help much because the stage ends when its slowest shard ends.
Answer: Saturation. Adding workers adds load to a full database and makes it worse. Reduce parallelism, fix the heavy queries or add database capacity at a safe time.
Answer: The mean is pulled towards the outlier, so it hides the skew. The median stays near the typical shard, so the slowest-to-median ratio exposes the outlier.
Answer: Only stages listed as deferrable in the runbook and agreed by their owners in advance. Control stages (reconciliation, screening, regulatory outputs) need explicit sign-off.
Answer: Each distinct dimension value creates a separate metric series, which is costly and hard to chart. Keep it as a log field so you can still query it in Logs Insights.
Interview questions
- The nightly batch is going to miss its window. What do you do in the first 15 minutes? Locate the running stage, compute a projected finish, tell downstream owners with the estimate and next update time, then classify the slowdown (skew, saturation, waiting, growth) with specific metrics.
- How do you tell skew from saturation? Skew: one shard far above the median while resources have headroom. Saturation: all shards slow together and a shared resource is at its ceiling.
- Which mitigations are safe, and which are not? Safe: more parallelism where headroom exists, splitting the long-pole shard, deferring pre-agreed non-critical stages. Unsafe: restarting from scratch, skipping control stages, manual data edits, unapproved code changes.
- How would you design the batch so overruns are caught early? Stage and shard timers, a duration trend per stage, and progress-based alarms at fixed hours, not only failure alarms.
- Why make batch stages idempotent? So a stage can be re-run or resumed after failure without double-applying postings (the lesson of the duplicate payment playbook).
Certification practice
A nightly job on Amazon ECS processes 64 data shards in parallel against an Amazon Aurora database. The job now finishes after the deadline. CloudWatch shows Aurora CPU at 30 percent and 63 tasks finishing in 10 minutes, but one task runs for four hours. What is the most likely cause and best remediation?
A. Database saturation; increase the Aurora instance size. B. Data skew in one shard; split the large shard's work into smaller units. C. A network ACL is blocking traffic; open all ports. D. The task definition's CPU is too low for all tasks; double it for every task.
Answer: B. Low database CPU and one outlier task indicate uneven work. A is not supported by the metrics. C would affect all tasks, not one. D would not help the single oversized unit and wastes money on the 63 that finish quickly.
A batch processing system uses AWS Lambda to process messages from an SQS queue. During the nightly peak, messages wait for hours and the function shows throttles. The database is lightly loaded. Which action is most appropriate?
A. Increase the queue's visibility timeout to 12 hours. B. Raise the function's reserved concurrency (and the account concurrency quota if needed), after confirming the database can take the extra load. C. Replace SQS with SNS. D. Disable the dead-letter queue.
Answer: B. Throttles with spare database capacity point to a Lambda concurrency limit. A does not add capacity, C changes the messaging model without addressing throughput and D removes error handling.
A team wants to be warned at 01:00 when a nightly batch is unlikely to finish by 05:00, even though no step has failed. Which solution is best?
A. An alarm on the Step Functions ExecutionsFailed metric.
B. A custom metric for percent complete (for example through EMF) and an alarm that fires if it is below a threshold at a scheduled time.
C. A CloudTrail trail for Step Functions API calls.
D. An AWS Config rule on the state machine.
Answer: B. A progress metric compared with the expected curve detects slowness. A only fires on failure, C records API activity and D evaluates configuration, not runtime progress.
Completion checklist
- I can name the four classes of batch slowdown and one metric that proves each
- I can explain why more workers do not fix skew
- I know which stages must never be skipped without sign-off
- I ran the skew analyser exercise and its tests
- I can write a one-line projected-finish message for downstream owners
- I ran the cleanup steps and confirmed nothing tagged
m17-playbook-eodremains
Key takeaways
- A batch can be late without any error. Treat time as the failure and measure by stage and shard.
- Classify before acting: skew, saturation, waiting or growth each have a different fix.
- Buy time with parallelism where there is headroom and with pre-agreed deferrals. Never skip controls or restart from zero.
- Alarm on progress against the schedule, not only on failure.
- Tell downstream owners early, with a number and a time for the next update.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Playbook: duplicate payment on client retry
Trace how one customer instruction became two payments, prove it from logs and the transactions table, reverse the duplicate safely with an auditable correcting entry, and close the gap with idempotency keys and conditional writes.
Playbook: partner API timeouts
Diagnose a partner API that times out by splitting the call into DNS, connect, TLS and response time, checking the egress path and the partner's limits, and choosing timeouts, retries and a circuit breaker that fail safe for a bank.