Skip to Content
Cloud Engineering Academy
17. Daily-Work Playbooks & Career

Playbook: end-of-day batch overrunning its window

Find the slow stage of a nightly batch, tell data skew from saturation from waiting, buy time without skipping controls, and add progress-based alarms so the next overrun is caught at 01:00 instead of 05:00.

intermediate⏱ 1 h 15 min🟢 FreeSOA-C03SAA-C03DOP-C02

Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · AWS Step Functions · CloudWatch embedded metric format · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this playbook you can:

  • Break an overrunning nightly batch into stages and shards, and identify the long pole (the one slice of work that decides the finish time).
  • Classify a slowdown as skew, saturation, waiting or growth, using specific metrics for each.
  • Choose safe ways to buy time (more parallelism, deferring non-critical stages) and recognise the ways that look attractive but break controls.
  • Instrument a .NET batch worker with stage timers that emit CloudWatch embedded-metric-format (EMF) lines, and analyse shard durations for skew.
  • Create progress-based alarms ("behind schedule") instead of failure-only alarms.
  • Communicate an overrun to downstream owners with a revised estimate and an audit record.

Prerequisites

Why this matters in banking

Harbourline Bank's end-of-day (EOD) batch starts at 22:00 once the day's cut-off passes. It posts interest and fees, generates statements, reconciles against Core Banking and produces a file for the regulatory reporting team, who need it by 05:30 to meet their own deadline. One Thursday the batch is still on stage three at 04:40. Nothing has failed: every log line is green. The batch is simply slower than the window.

A late batch is a business incident without an error message. Account balances shown in the app at 07:00 may be stale. Statements arrive late. A regulatory file can miss its deadline, which is a conversation with a regulator nobody wants. The skill here is to treat time as the failure: measure where it goes, decide quickly what is safe to defer, and tell people early. Engineers who only look for exceptions find nothing and lose an hour.

The idea in plain language

A batch is a pipeline of stages, and each stage is split into shards (slices of the data, for example by account ID range) that run in parallel. The finish time of a stage is the time of its slowest shard; the finish time of the batch is the sum of its stages (plus any waiting). So the batch can be late for four different reasons:

ClassWhat it meansSignature
SkewWork is unevenly split. One shard has a giant account or customerOne shard's duration is many times the median. Most workers sit idle waiting for it
SaturationA shared resource is full: database CPU, IOPS, connection limit, DynamoDB capacity, Lambda concurrencyEvery shard slows down together. A resource graph is flat at its ceiling
WaitingWorkers are blocked by something else: a lock, a partner API, a queue, a missing upstream fileLow CPU everywhere, long gaps in logs, timeouts, retries
GrowthThe batch is simply doing more work than last yearNightly stage durations trend steadily upward, no sudden change

Analogy: A coach-load of 40 guests checks out of a hotel. The checkout desks are the workers. If all desks are busy and the queue is long, that is saturation. If 39 guests finish in five minutes and one guest disputes a minibar bill for an hour, that is skew. If the desks are idle because the till system is down, that is waiting. If the hotel simply took on more guests than last year, that is growth. The fix for each is different, and adding desks only helps the first.

What problem the AWS services solve

AWS gives you the observation tools and the elasticity to change parallelism quickly. It does not make an inefficient batch fast.

  • AWS Step Functions records every stage with start and end times in its execution history, so the long pole is visible without any code. A Distributed Map state can fan out to a very large number of child executions for sharded work (⚠ VERIFY the current concurrency maximum). A Standard workflow execution history is capped at 25,000 events (⚠ VERIFY), which is one reason large fan-outs use Distributed Map instead of an inline Map.
  • Amazon CloudWatch metrics for the data stores (RDS, DynamoDB, SQS, Lambda, ECS) show which resource saturates. Performance Insights (or the current Database Insights naming, ⚠ VERIFY) shows what an RDS database is waiting on.
  • CloudWatch embedded metric format (EMF) lets your own worker emit stage metrics as structured log lines with no extra API calls.
  • Amazon EventBridge Scheduler starts the batch at a fixed time, and AWS Batch or ECS supply compute you can scale out.

Limits and sharp edges to respect during an overrun:

  • Scaling out can move the bottleneck. Doubling Lambda concurrency or ECS tasks doubles the load on the database, which may then throttle or run out of connections. Check the connection limit and CPU headroom first.
  • Lambda account concurrency defaults to 1,000 concurrent executions per Region for many accounts, and new accounts can start lower (⚠ VERIFY current default). A quota increase request takes time, so it is not an emergency tool.
  • DynamoDB limits a single partition to about 1,000 write units and 3,000 read units per second (⚠ VERIFY). A hot key is skew at the storage layer, and on-demand mode does not remove it.
  • Instance changes mid-run (for example resizing an RDS instance) can cause a failover or restart. That is rarely a good mid-batch move.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
SQL Agent job with steps and a job historyStep Functions state machine with execution historyHistory is per state, with input and output you can inspect. No server to babysit
SSIS package with parallel data flowsStep Functions Distributed Map or AWS GlueShards are separate child executions, so skew shows up as one slow child
Azure Data Factory pipeline monitorStep Functions console plus CloudWatchYou build the "behind schedule" alarm yourself
Azure Batch pool autoscaleAWS Batch, ECS service auto scaling, Lambda concurrencyScaling is quick, but each layer has quotas and shared downstream limits
Azure Monitor Application Insights custom metrics (TrackMetric)CloudWatch EMF log linesA single JSON line in stdout becomes a metric. Dimensions affect cost, so keep them few
SQL Server Query Store and wait statsRDS Performance Insights top waitsBoth show where the database spends time, which is how you see lock or IO waits

Architecture

The batch and the checkpoints that matter

Decision tree: why is it slow?

Step-by-step implementation

Minutes 0 to 10: locate the stage and set the clock

  1. Open the Step Functions execution and note which state is running and when it started. Compare with the same stage on the last five nights (durations are in the history).
  2. Compute how late you are: the remaining work divided by the current rate gives a projected finish time. Write it down with the time you calculated it. A rough number is enough: "stage 3 is 40 percent done after 70 minutes, so about 105 minutes remain".
  3. Decide who needs to hear this now: the regulatory reporting owner and the operations manager. Send a short message with the projection and the time of your next update. Silence is the worst option.

Minutes 10 to 25: classify the slowdown

Use the decision tree above, and prove each branch with a number.

  • Skew: list shard durations for the running stage. Anything over about three times the median deserves a look.
  • Saturation: check the shared resources at the same time window (queries are in the CLI section): database CPU and connections, IO, DynamoDB throttles, Lambda throttles and concurrency, ECS task CPU and memory.
  • Waiting: look for gaps. If workers log nothing for minutes, find what they were waiting for: a lock, a partner API call, an SQS message that never arrived.
  • Growth: chart StageDurationSeconds for 30 nights. A slow upward line points to growth, and no overnight fix will cure it; you need capacity planning and a code change.

Minutes 25 to 45: buy time safely

Pick from this list in order. Each item has a rule.

MoveSafe whenDanger
Raise parallelism for the slow stageThe database or store has headroom (CPU below about 60 percent, connections well under the limit)Moves the bottleneck and can throttle the whole account
Add workers only for the unfinished shardsShards are independent and idempotentDo not start a second copy of a shard that is still running
Split the long-pole shard and run the pieces in parallelThe unit of work can be divided (by date range, by account sub-range)Never split in a way that changes totals or ordering the stage depends on
Defer pre-agreed non-critical stages (analytics exports, marketing extracts)They are listed as deferrable in the runbook and the owners agreed in advanceNever defer reconciliation, sanctions screening or the regulatory file without written sign-off
Pause other heavy database users (reporting queries, backups outside the managed window)You can identify them in Performance InsightsDo not kill sessions you do not understand

What you must not do: kill the batch and restart it from the top, skip a control stage to save time, edit data by hand to unblock a shard, or deploy new code to the running batch without the normal approval. A restart from zero costs the work already done, and a skipped control is an audit finding.

After the window: root cause and prevention

Write the cause by class (skew, saturation, waiting or growth), prove it with the numbers you captured, then fix it so it cannot recur: re-shard, add the alarm, fix the lock, plan capacity. The Exercises walk through a skew case.

AWS Console instructions

  1. Console → Step Functions → State machines → hl-eod-batch → Executions → (running execution). The Graph view colours states by status. Open Execution event history to read each state's start and end time.
  2. Console → CloudWatch → Metrics → All metrics → Harbourline/Batch. Graph StageDurationSeconds by Stage for the last 30 days.
  3. Console → RDS → Databases → your cluster → Monitoring. Check CPUUtilization, DatabaseConnections and ReadIOPS/WriteIOPS. For waits, open Performance Insights and group the load by Waits.
  4. Console → DynamoDB → Tables → Monitor. Check Throttled requests and Consumed write capacity.
  5. Console → Lambda → Functions → Monitor. Check Throttles and Concurrent executions.
  6. Console → CloudWatch → Logs → Logs Insights. Run the shard queries below.

AWS CLI

# 1. Find the running executions and see which state they are in.
aws stepfunctions list-executions \
  --state-machine-arn arn:aws:states:us-east-1:111122223333:stateMachine:hl-eod-batch \
  --status-filter RUNNING --profile learn-sandbox

# 2. Read the history of one execution: state names and timestamps (compact view).
aws stepfunctions get-execution-history \
  --execution-arn <execution-arn> --max-results 200 \
  --query 'events[?type==`TaskStateEntered` || type==`TaskStateExited`].[timestamp,type,stateEnteredEventDetails.name,stateExitedEventDetails.name]' \
  --output table --profile learn-sandbox

# 3. Database pressure during the batch window.
aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name CPUUtilization \
  --dimensions Name=DBClusterIdentifier,Value=hl-core-aurora \
  --start-time $(date -u -d '4 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --statistics Average Maximum --profile learn-sandbox

aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name DatabaseConnections \
  --dimensions Name=DBClusterIdentifier,Value=hl-core-aurora \
  --start-time $(date -u -d '4 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --statistics Maximum --profile learn-sandbox

# 4. DynamoDB write throttles (sum per 5 minutes).
aws cloudwatch get-metric-statistics --namespace AWS/DynamoDB --metric-name WriteThrottleEvents \
  --dimensions Name=TableName,Value=hl-ledger \
  --start-time $(date -u -d '4 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --statistics Sum --profile learn-sandbox

# 5. Backlog in the work queue: how old is the oldest unprocessed shard message?
aws cloudwatch get-metric-statistics --namespace AWS/SQS --metric-name ApproximateAgeOfOldestMessage \
  --dimensions Name=QueueName,Value=hl-eod-statements \
  --start-time $(date -u -d '4 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --statistics Maximum --profile learn-sandbox

# 6. Raise parallelism of the statements service (ECS) for the unfinished work only.
aws ecs update-service --cluster hl-batch --service hl-statements-worker --desired-count 24 --profile learn-sandbox

Logs Insights queries for shards (the worker logs the EMF lines shown below):

# Slowest shards in the current run
fields runId, stage, shardId, StageDurationSeconds, recordCount
| filter runId = "2026-11-12" and stage = "statements"
| sort StageDurationSeconds desc
| limit 10

# Skew indicator per stage: max versus median
fields stage, StageDurationSeconds
| filter runId = "2026-11-12"
| stats pct(StageDurationSeconds, 50) as median, max(StageDurationSeconds) as slowest, count(*) as shards by stage

Infrastructure as Code

Alarm on a progress metric rather than on errors. The worker publishes PercentComplete (0 to 100) for each stage; the alarm fires when the batch is behind the expected curve at a given hour. A simpler start is an alarm on stage duration, shown here.

using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.CloudWatch.Actions;
using Amazon.CDK.AWS.SNS;
using Constructs;

public class BatchAlarmsStack : Stack
{
    public BatchAlarmsStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
    {
        var topic = new Topic(this, "BatchOps", new TopicProps { TopicName = "hl-batch-ops" });

        var statementsDuration = new Metric(new MetricProps
        {
            Namespace = "Harbourline/Batch",
            MetricName = "StageDurationSeconds",
            DimensionsMap = new Dictionary<string, string> { ["Stage"] = "statements" },
            Statistic = "Maximum",
            Period = Duration.Minutes(5)
        });

        var alarm = new Alarm(this, "StatementsSlow", new AlarmProps
        {
            AlarmName = "hl-eod-statements-shard-slow",
            AlarmDescription = "Any statements shard took over 45 minutes. Open the EOD batch playbook.",
            Metric = statementsDuration,
            Threshold = 45 * 60,
            EvaluationPeriods = 1,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
            TreatMissingData = TreatMissingData.NOT_BREACHING
        });
        alarm.AddAlarmAction(new SnsAction(topic));

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-eod");
    }
}

Complete code

Two small pieces: a stage timer that emits an EMF line when a shard finishes, and a skew analyser you can run on the durations afterwards (or in a unit test).

StageTimer.cs
using System.Diagnostics;
using System.Text.Json;

namespace Harbourline.Batch;

/// <summary>Times one shard of one stage and writes a CloudWatch EMF line to stdout.</summary>
public sealed class StageTimer : IDisposable
{
    private readonly Stopwatch _sw = Stopwatch.StartNew();
    private readonly string _runId, _stage, _shardId;
    private readonly TextWriter _out;

    public long RecordCount { get; set; }

    public StageTimer(string runId, string stage, string shardId, TextWriter? output = null)
    {
        _runId = runId; _stage = stage; _shardId = shardId;
        _out = output ?? Console.Out;
    }

    public void Dispose()
    {
        _sw.Stop();
        var line = new Dictionary<string, object>
        {
            ["_aws"] = new Dictionary<string, object>
            {
                ["Timestamp"] = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(),
                ["CloudWatchMetrics"] = new[]
                {
                    new Dictionary<string, object>
                    {
                        ["Namespace"] = "Harbourline/Batch",
                        // Only Stage is a dimension. shardId and runId stay as plain log fields
                        // so they do not create thousands of billable metric series.
                        ["Dimensions"] = new[] { new[] { "Stage" } },
                        ["Metrics"] = new[]
                        {
                            new Dictionary<string, object> { ["Name"] = "StageDurationSeconds", ["Unit"] = "Seconds" },
                            new Dictionary<string, object> { ["Name"] = "RecordCount", ["Unit"] = "Count" }
                        }
                    }
                }
            },
            ["Stage"] = _stage,
            ["runId"] = _runId,
            ["shardId"] = _shardId,
            ["StageDurationSeconds"] = Math.Round(_sw.Elapsed.TotalSeconds, 3),
            ["RecordCount"] = RecordCount
        };
        _out.WriteLine(JsonSerializer.Serialize(line));
    }
}
SkewAnalyzer.cs
namespace Harbourline.Batch;

public sealed record ShardTiming(string ShardId, double Seconds, long Records);

public sealed record SkewReport(double Median, double Slowest, double Ratio, string SlowestShard,
                                string Verdict, double? SecondsSavedIfPerfectlyBalanced);

public static class SkewAnalyzer
{
    public static SkewReport Analyze(IReadOnlyList<ShardTiming> shards, double skewRatioThreshold = 3.0)
    {
        if (shards.Count == 0) throw new ArgumentException("No shards.", nameof(shards));

        var ordered = shards.OrderBy(s => s.Seconds).ToList();
        var median = ordered.Count % 2 == 1
            ? ordered[ordered.Count / 2].Seconds
            : (ordered[ordered.Count / 2 - 1].Seconds + ordered[ordered.Count / 2].Seconds) / 2;
        var slowest = ordered[^1];
        var ratio = median > 0 ? slowest.Seconds / median : double.PositiveInfinity;

        // If work were split evenly, the stage would take about the mean shard time.
        var mean = ordered.Average(s => s.Seconds);
        var saved = slowest.Seconds - mean;

        var verdict = ratio >= skewRatioThreshold
            ? $"SKEW: shard {slowest.ShardId} is {ratio:F1}x the median. Split it or use work stealing."
            : "No significant skew. If the stage is still slow, look at saturation or waiting.";

        return new SkewReport(median, slowest.Seconds, ratio, slowest.ShardId, verdict,
                              ratio >= skewRatioThreshold ? saved : null);
    }
}

A tiny test that doubles as documentation:

SkewAnalyzerTests.cs
using Harbourline.Batch;
using Xunit;

public class SkewAnalyzerTests
{
    [Fact]
    public void One_giant_account_is_reported_as_skew()
    {
        var shards = Enumerable.Range(0, 63).Select(i => new ShardTiming($"{i:D2}", 600, 50_000)).ToList();
        shards.Add(new ShardTiming("63", 7_200, 2_000_000));   // a corporate customer with millions of rows

        var report = SkewAnalyzer.Analyze(shards);

        Assert.Equal("63", report.SlowestShard);
        Assert.True(report.Ratio >= 3.0);
        Assert.StartsWith("SKEW", report.Verdict);
        Assert.NotNull(report.SecondsSavedIfPerfectlyBalanced);
    }

    [Fact]
    public void Evenly_loaded_shards_are_not_skew()
    {
        var shards = Enumerable.Range(0, 8).Select(i => new ShardTiming($"{i}", 900 + i, 10_000)).ToList();
        Assert.DoesNotContain("SKEW", SkewAnalyzer.Analyze(shards).Verdict);
    }
}

Code explained line by line

  1. StageTimer is IDisposable, so using var t = new StageTimer(...) times a whole block and guarantees the line is written even if the block returns early.
  2. RecordCount lets the worker add how many records it handled, so you can tell a slow shard that is big from a slow shard that is stuck (long duration with few records).
  3. The _aws object is the EMF envelope. CloudWatch reads it from the log line and creates metrics from the named fields, with no API call from your code.
  4. Only Stage is a dimension. Adding shardId as a dimension would create one metric series per shard per night, which is expensive and unreadable. Keep shardId as a plain field so Logs Insights can still filter on it.
  5. SkewAnalyzer uses the median, not the mean, as the baseline, because one huge shard drags the mean up and hides itself.
  6. SecondsSavedIfPerfectlyBalanced is the slowest shard's time minus the mean: a rough estimate of how much faster the stage would be if the work were evenly split. It is an upper bound because shards cannot be split perfectly.
  7. The tests build a realistic bad case (63 normal shards and one corporate account with 40 times the data) and assert the analyser names the right shard.

Security considerations

⚠ Learning shortcut — not for production

Under pressure it is tempting to give an engineer a broad role (for example AdministratorAccess or direct database write access) "just for tonight" to unblock the batch. Do not. Prepare a break-glass, time-boxed role in advance with only the actions the runbook needs (Step Functions StartExecution, StopExecution, ECS UpdateService, read-only metrics) and require approval and an audit record when it is used.

  • Controls are part of the batch. Reconciliation, sanctions batch screening and the regulatory file are controls, not optional steps. Deferring them needs a documented, approved decision, not an engineer's judgement at 03:00.
  • Do not edit data by hand to unblock a shard. Manual changes bypass the checks and leave no consistent trail. Fix the cause or hand the shard to the owner.
  • Stage logs must not contain personal data. Log shard IDs, counts and timings, not account numbers or names. This also keeps the analysis available to a wider group.
  • Scaling up changes the blast radius. More concurrent workers mean more concurrent credentials in use and more downstream calls. Keep each worker on its own least-privilege task role.
  • Keep the incident timeline (who changed parallelism or deferred a stage, when and on whose approval). Your change management and audit teams decide what evidence they need; this lesson does not claim any regulatory standard is met.

Banking use case

Harbourline's statements stage ran 90 minutes in January and 310 minutes in November. The stage timers showed 63 shards at about ten minutes each and one shard taking 4 hours: a business customer with several million transactions in a year, hashed to the same shard as other accounts. The team did three things. First, tonight: they split that shard into 16 date-range pieces and ran them in parallel, because statements are independent per date range, and the stage finished 3 hours earlier. Second, this month: they changed sharding from account-hash to work units capped at 50,000 records, placed on an SQS queue, so workers pull the next unit when free (work stealing). Third, permanently: they alarmed on progress at 01:00 and 03:00 ("stage 3 must be 50 percent done by 01:00"), so the next overrun is a 01:00 page, not a 04:40 discovery.

Common mistakes

  • Only alarming on failure. A late batch throws no exception. Alarm on duration and on progress against schedule.
  • Adding workers to a skew problem. More workers do not shorten the one long shard.
  • Adding workers to a saturated database. It makes things worse. Check CPU and connections first.
  • Restarting from the top. Throws away completed work and may double-apply stages that are not idempotent.
  • Averaging. The mean hides the long pole. Always look at the median and the maximum.
  • Silence. Not telling downstream owners early. A revised estimate sent at 02:00 is a plan; a surprise at 05:15 is an incident.
  • High-cardinality metric dimensions. Putting shard IDs or run IDs in dimensions makes the dashboard slow and the bill large.

Troubleshooting

SymptomLikely causeFix
One shard far slower than the restData skew (a giant account or a hot key)Split the unit of work, cap work-unit size, use a queue so free workers pull more work
All shards slow after a releaseNew query without an index, or a plan changePerformance Insights top SQL, compare the plan, roll back the release if needed
Throughput flat while CPU is lowWaiting on a lock or a partner APICheck blocked sessions and outbound call latency. See Partner API timeouts
ProvisionedThroughputExceededException or high WriteThrottleEventsDynamoDB capacity or hot partitionSpread keys, use write sharding for hot keys, raise provisioned capacity or move to on-demand
Lambda Throttles rising, queue age growingAccount or function concurrency limit reachedRaise reserved concurrency for the function within the account quota, or request a quota increase ahead of time
ECS tasks restarting during the batchMemory limit exceeded as data growsSee ECS task keeps restarting
Database connections near the limitEach worker opens its own poolCap the pool per worker, use RDS Proxy (⚠ VERIFY suitability for your engine), or reduce parallelism
No stage timing dataTimers not deployed, or EMF line malformedCheck the log group for the JSON lines. The _aws envelope must be valid JSON on one line

Cost and cleanup

Cost label: free. The local skew exercise needs only .NET 10. The optional sandbox alarm costs about ten cents per month per standard alarm and EMF custom metrics cost a small amount per metric series (⚠ VERIFY current CloudWatch pricing). Delete the stack when you finish.

# CDK
cdk destroy BatchAlarmsStack --profile learn-sandbox

# Or by hand
aws cloudwatch delete-alarms --alarm-names hl-eod-statements-shard-slow --profile learn-sandbox
aws sns delete-topic --topic-arn arn:aws:sns:us-east-1:111122223333:hl-batch-ops --profile learn-sandbox

# Confirm nothing tagged for this lesson remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-eod --profile learn-sandbox

Exercises

  • 🟢 Guided: Add StageTimer and SkewAnalyzer to a console project. Generate 64 fake shard timings, make one 12 times larger, run the analyser and read the verdict. Print the JSON line from a StageTimer and check it is one valid line.
  • 🟡 Challenge: Write a function ProjectFinish(DateTimeOffset start, double fractionDone, DateTimeOffset now) that returns the projected finish time. Add a rule that returns "behind schedule" if the projection is later than a deadline you pass in, with a 10 percent safety margin. Unit test it with three scenarios.
  • 🔴 Live problem: 03:20. The statements stage is 35 percent done. The regulatory file is due at 05:30 and depends on stage 4 (reconciliation, about 60 minutes) and stage 5 (about 20 minutes). Aurora CPU is 35 percent, connections are at 40 percent of the limit, and ECS shows 8 statements workers all busy. The product owner asks you to "skip reconciliation, we can do it tomorrow". Decide what you do and what you say.

Assignment

Add a batch-overrun section to your portfolio incident template: the four classes (skew, saturation, waiting, growth), the first-ten-minutes checklist and the "deferrable stages" table with owners. Commit incident-reports/eod-batch-overrun.md containing a worked example with fictional numbers and the skew analyser output.

Solutions

Knowledge check

Interview questions

  • The nightly batch is going to miss its window. What do you do in the first 15 minutes? Locate the running stage, compute a projected finish, tell downstream owners with the estimate and next update time, then classify the slowdown (skew, saturation, waiting, growth) with specific metrics.
  • How do you tell skew from saturation? Skew: one shard far above the median while resources have headroom. Saturation: all shards slow together and a shared resource is at its ceiling.
  • Which mitigations are safe, and which are not? Safe: more parallelism where headroom exists, splitting the long-pole shard, deferring pre-agreed non-critical stages. Unsafe: restarting from scratch, skipping control stages, manual data edits, unapproved code changes.
  • How would you design the batch so overruns are caught early? Stage and shard timers, a duration trend per stage, and progress-based alarms at fixed hours, not only failure alarms.
  • Why make batch stages idempotent? So a stage can be re-run or resumed after failure without double-applying postings (the lesson of the duplicate payment playbook).

Certification practice

Completion checklist

  • I can name the four classes of batch slowdown and one metric that proves each
  • I can explain why more workers do not fix skew
  • I know which stages must never be skipped without sign-off
  • I ran the skew analyser exercise and its tests
  • I can write a one-line projected-finish message for downstream owners
  • I ran the cleanup steps and confirmed nothing tagged m17-playbook-eod remains

Key takeaways

  • A batch can be late without any error. Treat time as the failure and measure by stage and shard.
  • Classify before acting: skew, saturation, waiting or growth each have a different fix.
  • Buy time with parallelism where there is headroom and with pre-agreed deferrals. Never skip controls or restart from zero.
  • Alarm on progress against the schedule, not only on failure.
  • Tell downstream owners early, with a number and a time for the next update.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page