Cut-over and rollback planning
Plan the night the bank switches over: choose a cut-over pattern, schedule freeze windows, define go/no-go criteria and rollback triggers before you need them, reconcile data, shift traffic gradually with weighted DNS, and assemble the evidence a change board and a regulator will ask for.
Assumes: .NET 10 LTS · AWS CLI v2 · Amazon Route 53 · Amazon CloudWatch · AWS CDK v2 (C#) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Choose a cut-over pattern (big bang, phased, parallel run, weighted traffic shift) and justify it for a banking workload.
- Write a timeline-based runbook with owners, durations, checkpoints and a clearly marked point of no return.
- Define go/no-go criteria and rollback triggers as testable rules, and encode them so a script can evaluate them.
- Plan data reconciliation and the data-divergence problem that makes rollback harder after go-live.
- Shift traffic gradually with weighted Route 53 records, a low TTL and a health signal, using CDK (C#) or Terraform.
- Assemble the evidence pack for a change advisory board (CAB) and for regulatory conversations.
Prerequisites
- Rehosting with AWS Application Migration Service and SQL Server database migration — the mechanics this plan orchestrates
- Network migration and hybrid architecture — DNS and connectivity
- Route 53 and DNS
- Operational resilience and third-party risk — why a migration is a resilience event
- Database failover — failover thinking applies to rollback
- Tools: .NET 10 SDK or Docker. No AWS account is needed for the code.
Why this matters in banking
A cut-over is where every earlier decision meets reality at 2 a.m. on a Sunday. Banks have a particular problem: transactions do not stop at the migration boundary. A card authorisation approved in the new environment at 02:10 must still be there if you roll back at 02:40, or a customer is charged for something the old ledger has never heard of. Equally, a rollback plan that was never rehearsed is a hope, not a plan.
Change boards and regulators will ask: who approved this, what was tested, what does "success" mean, what makes you stop, how long can customers be affected, how do you know the data matches, and what happens to money in flight. In the UK, operational resilience expectations include staying within defined impact tolerances for important business services ⚠ VERIFY the current regulatory wording with your compliance team. Your cut-over plan is evidence that you can.
The idea in plain language
A cut-over has five phases:
- Prepare. Everything that can be done in advance is done: environments built, data synchronised, tests passed, people briefed.
- Freeze. Changes stop. No unrelated deployments, no schema changes, no batch schedule edits.
- Switch. Stop writes at the source, drain the last data, run reconciliation, point users at the new system.
- Verify and decide. Run business checks. Go or roll back, using criteria agreed beforehand.
- Settle. Keep the old system intact for a rollback window, watch closely, then decommission.
Two ideas matter most. The point of no return is the moment after which going back loses or corrupts data (for example, once new transactions exist only on the target). Plan to make it as late as possible, and make the decision to cross it explicit. And decide the rollback triggers while calm, not at 03:00 under pressure.
Analogy: Opening a new branch. You rehearse the opening, count the tills in both branches, and agree in advance that if the card machines are not working by 9:30 you reopen the old branch. You do not shut the old branch's vault until the new one has balanced for a week.
What problem the AWS services solve
The cut-over is mostly process, but several AWS features make a safe, gradual switch possible.
| Need | AWS feature | Honest limits |
|---|---|---|
| Gradual traffic shift | Route 53 weighted records with a low TTL | Clients and resolvers can cache beyond the TTL; a shift is not instant or exact |
| Automatic traffic protection | Route 53 health checks, ALB target group health | Route 53 health checks reach public endpoints; for private endpoints use CloudWatch alarm-based checks ⚠ VERIFY |
| Blue/green at the load balancer | ALB weighted target groups | Works for HTTP(S) traffic you route through the ALB |
| Rehost final sync and rollback | MGN test and cut-over launches; keep the source until finalise | After finalising, the replica is gone (see rehost lesson) |
| Data continuity | DMS CDC (and reverse replication if configured) | Reverse replication must be built and rehearsed; it is not automatic |
| Runbook execution and evidence | Systems Manager Automation and Run Command, CloudTrail, CloudWatch | Incident Manager and Change Manager are closed or in maintenance; use runbooks with SSM Automation and your ITSM tool ⚠ VERIFY replacements |
| Observability during the window | CloudWatch dashboards and alarms, X-Ray or OpenTelemetry | Build the dashboard before the night |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Traffic Manager weighted routing | Route 53 weighted routing | Same idea; Route 53 adds latency, failover and geolocation policies |
| Azure Front Door canary and slot swap | ALB weighted target groups, CloudFront continuous deployment | Slots do not exist; you run two target groups or two stacks |
| Azure Migrate cut-over and "complete migration" | MGN cut-over launch and finalise cutover | Finalising is irreversible for replication |
| Azure Site Recovery failover and failback | MGN with reverse planning, or Elastic Disaster Recovery failback | Failback is a design item on AWS too; plan it before go-live |
| ITIL change records in ServiceNow | Same ITSM, plus CloudTrail for AWS-side evidence | AWS CloudTrail events are your technical audit trail |
| Azure Monitor action groups | CloudWatch alarms, SNS and EventBridge | Alarms drive notifications and, if you choose, automated rollback steps |
Architecture
Security-boundary view. During cut-over the number of people with elevated access rises: a break-glass role for the migration team, access to both databases, and the ability to change DNS. Control it: time-boxed role assumption through IAM Identity Center, session recording where possible, CloudTrail logging on, a named approver for each irreversible step, and revocation of the elevated access immediately after the window. Keep two-person rules for the irreversible steps (switch, finalise, decommission).
Step-by-step implementation
Choose the cut-over pattern
| Pattern | How it works | Best for | Main risk |
|---|---|---|---|
| Big bang | Everything switches in one window | Small, tightly coupled systems with a long maintenance window | One shot; rollback is all or nothing |
| Phased by function or customer segment | Move a slice (staff first, then 1% of customers, then more) | Large user bases, tolerant of two systems side by side | Data and routing must handle both worlds |
| Parallel run | Both systems process the same inputs; compare outputs; the old one stays authoritative until proven | Calculation-heavy systems (interest, fees, statements) | Double the cost; careful side-effect suppression |
| Weighted traffic shift | Gradually move HTTP traffic with weighted DNS or load-balancer weights | Stateless front ends and APIs | Shared state (sessions, databases) must be consistent |
For a core ledger, a bank often combines them: parallel-run calculations, a phased move of segments, and a short big-bang for the write path with a rehearsed rollback.
Pick the window
Avoid month-end, quarter-end, payment scheme peaks, tax deadlines and public holidays. Align with the freeze calendar and with the payment scheme cut-off times. Set the freeze start (for example, 72 hours before) and the end (after the rollback window).
Write the runbook as a timeline
Every step has an ID, an owner, a duration, a command or checklist link, an expected result, and what to do on failure. Mark the point of no return and the rollback deadline in bold. The timeline below is the skeleton.
Define go/no-go criteria and rollback triggers
Write them as yes/no tests. Examples are in the next section and in the code. Agree them with business, risk and operations before the rehearsal, and not during it.
Rehearse at least twice, then run a dress rehearsal
Rehearsal one finds the missing steps. Rehearsal two checks the timings. The dress rehearsal uses the real roles and tools, including the rollback. Time everything, and compare against the available window.
Execute, observe, decide
Follow the runbook. Record decisions, times and evidence as you go. At each checkpoint the incident commander states "go" or "stop" aloud and it is logged.
Hold the rollback window, then settle
Keep the old system intact (read-only or powered off) for the agreed window, watching reconciliation and customer complaints. Only then finalise replication, decommission and release licences.
Sample timeline (core ledger, Saturday window)
| Time | Step | Owner | Checkpoint |
|---|---|---|---|
| T-72h | Change freeze starts; dress rehearsal results approved | Release manager | CAB approval recorded |
| T-24h | Readiness gate: replication lag, test evidence, staffing | Incident commander | Go/no-go meeting 1 |
| T-2h | Lower DNS TTL to 60 s (already done T-48h); confirm dashboards | Platform lead | TTL verified |
| T-0 | Stop batch scheduler; stop writes at source; announce maintenance | Operations | Writes stopped |
| T+10m | CDC lag reaches zero | DBA | Lag zero confirmed |
| T+25m | Reconciliation: counts, control totals, hashes | Data lead | Reconciliation clean |
| T+30m | Go/no-go meeting 2 (last rollback point without data divergence) | Incident commander | Decision logged |
| T+35m | Switch application connection strings and DNS weights to 100% new | Platform lead | Point of no return crossed once new writes land |
| T+60m | Smoke tests, payment test transactions, statement check | QA lead | Business sign-off |
| T+60 to T+8h | Hypercare; alarms armed as rollback triggers | On-call | Hourly checkpoint |
| T+14 days | Rollback window ends; finalise replication; decommission plan | Programme manager | CAB closure |
AWS Console instructions
- Route 53 → Hosted zones → the zone → Create record: Routing policy Weighted, a Record ID (set identifier) for each side, Weight, TTL 60, optional Health check. Repeat for old and new.
- CloudWatch → Dashboards → Create dashboard: error rates, latency percentiles, queue depth, DMS latency, reconciliation counters (as custom metrics).
- CloudWatch → Alarms → Create alarm: each rollback trigger is an alarm with an SNS action to the incident channel.
- Systems Manager → Automation: store runbook steps that can be scripted (for example, set DNS weights).
- MGN → Source servers: the cut-over and finalise actions from the previous lesson (finalise only after the rollback window).
AWS CLI
# Shift traffic: 90% old, 10% new. Run once per step. Changes are described in a JSON batch.
cat > shift-10.json <<'JSON'
{
"Comment": "Cut-over step: 10 percent to AWS",
"Changes": [
{ "Action": "UPSERT", "ResourceRecordSet": {
"Name": "portal.example.com", "Type": "CNAME", "SetIdentifier": "old-dc", "Weight": 90, "TTL": 60,
"ResourceRecords": [ { "Value": "portal-old.dc.example.com" } ] } },
{ "Action": "UPSERT", "ResourceRecordSet": {
"Name": "portal.example.com", "Type": "CNAME", "SetIdentifier": "aws", "Weight": 10, "TTL": 60,
"ResourceRecords": [ { "Value": "hl-portal-123456.us-east-1.elb.amazonaws.com" } ] } }
]
}
JSON
aws route53 change-resource-record-sets --hosted-zone-id Z0123456789ABCDEFGHIJ \
--change-batch file://shift-10.json --profile learn-sandbox
# Rollback is the same command with weights 100 and 0. Keep the file ready and rehearsed.
# A rollback-trigger alarm: ALB 5xx rate above 1 percent for 5 consecutive minutes (metric math omitted for brevity).
aws cloudwatch put-metric-alarm --alarm-name hl-m13-portal-5xx --namespace AWS/ApplicationELB \
--metric-name HTTPCode_Target_5XX_Count --dimensions Name=LoadBalancer,Value=app/hl-portal/0123456789abcdef \
--statistic Sum --period 60 --evaluation-periods 5 --threshold 50 \
--comparison-operator GreaterThanThreshold --treat-missing-data notBreaching \
--alarm-actions arn:aws:sns:us-east-1:123456789012:hl-m13-cutover \
--profile learn-sandbox --region us-east-1example.com names and the IDs are placeholders. A count threshold is a simplification; in practice use a rate with metric math so traffic volume does not distort the trigger.
Infrastructure as Code
Weighted DNS records for a gradual shift. Parameterise the AWS weight so the change is a one-line, reviewed pull request and the pipeline records who approved each step.
using Amazon.CDK;
using Amazon.CDK.AWS.Route53;
using Constructs;
namespace Harbourline.Migration.Infra;
public sealed class CutoverDnsStack : Stack
{
public CutoverDnsStack(Construct scope, string id, string hostedZoneId, string zoneName, int awsWeight,
IStackProps? props = null) : base(scope, id, props)
{
if (awsWeight is < 0 or > 100)
throw new ArgumentOutOfRangeException(nameof(awsWeight), "weight must be 0 to 100");
// Old estate keeps the remainder so the two weights always total 100.
_ = new CfnRecordSet(this, "Old", new CfnRecordSetProps
{
HostedZoneId = hostedZoneId,
Name = $"portal.{zoneName}",
Type = "CNAME",
SetIdentifier = "old-dc",
Weight = 100 - awsWeight,
Ttl = "60",
ResourceRecords = ["portal-old.dc.example.com"]
});
_ = new CfnRecordSet(this, "Aws", new CfnRecordSetProps
{
HostedZoneId = hostedZoneId,
Name = $"portal.{zoneName}",
Type = "CNAME",
SetIdentifier = "aws",
Weight = awsWeight,
Ttl = "60",
ResourceRecords = ["hl-portal-123456.us-east-1.elb.amazonaws.com"]
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m13-cutover");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "your-name");
}
}Run with cdk deploy -c awsWeight=10, read in your Program.cs with app.Node.TryGetContext("awsWeight"). A weight of 0 on the AWS record is the rehearsed rollback.
Complete code
Two small pure-C# components: a go/no-go evaluator (readiness criteria with evidence and waivers) and a rollback monitor (sustained-breach triggers). They are deliberately independent of AWS so you can run them in a rehearsal and attach their output to the change record.
namespace Harbourline.Migration.Cutover;
public enum CriterionStatus { NotStarted, Passed, Failed, Waived }
public sealed record Criterion(
string Id,
string Description,
bool Mandatory,
CriterionStatus Status,
string? EvidenceRef = null, // link to a test report, a reconciliation output, a ticket
string? WaivedBy = null); // named approver, required for a waiver
public enum Decision { Go, GoWithWaivers, NoGo }
public sealed record Assessment(Decision Decision, IReadOnlyList<string> Reasons);
public static class GoNoGoEvaluator
{
public static Assessment Evaluate(IEnumerable<Criterion> criteria)
{
var reasons = new List<string>();
var waivers = 0;
foreach (var c in criteria)
{
switch (c.Status)
{
case CriterionStatus.Passed:
// A pass without evidence is an opinion, not a pass.
if (string.IsNullOrWhiteSpace(c.EvidenceRef))
reasons.Add($"{c.Id}: passed but no evidence reference");
break;
case CriterionStatus.Waived:
if (c.Mandatory || string.IsNullOrWhiteSpace(c.WaivedBy) || string.IsNullOrWhiteSpace(c.EvidenceRef))
reasons.Add($"{c.Id}: waiver invalid (mandatory criteria cannot be waived; waivers need approver and evidence)");
else
waivers++;
break;
case CriterionStatus.Failed:
case CriterionStatus.NotStarted:
if (c.Mandatory) reasons.Add($"{c.Id}: mandatory criterion {c.Status}: {c.Description}");
else waivers++; // a failed optional criterion is carried as a known issue
break;
}
}
if (reasons.Count > 0) return new Assessment(Decision.NoGo, reasons);
return waivers > 0
? new Assessment(Decision.GoWithWaivers, [$"{waivers} known issue(s) accepted with approval"])
: new Assessment(Decision.Go, ["all criteria passed with evidence"]);
}
}namespace Harbourline.Migration.Cutover;
public enum Comparison { GreaterThan, LessThan }
// A trigger fires only when the breach is sustained for the whole window: one bad sample is noise.
public sealed record Trigger(string Metric, Comparison Op, double Threshold, TimeSpan Window, string Action);
public sealed record Sample(string Metric, DateTimeOffset At, double Value);
public sealed record Fired(Trigger Trigger, DateTimeOffset Since);
public static class RollbackMonitor
{
public static IReadOnlyList<Fired> Evaluate(
IReadOnlyList<Sample> samples, IEnumerable<Trigger> triggers, DateTimeOffset now)
{
var fired = new List<Fired>();
foreach (var t in triggers)
{
var inWindow = samples
.Where(s => s.Metric == t.Metric && s.At > now - t.Window && s.At <= now)
.OrderBy(s => s.At)
.ToList();
// We need data covering the window. Missing data is treated as a breach of monitoring,
// not as "all fine": a dead monitor must never look like a healthy system.
if (inWindow.Count < 2 || inWindow[0].At > now - t.Window + TimeSpan.FromSeconds(90))
{
fired.Add(new Fired(t with { Action = "MONITORING GAP: " + t.Action }, now));
continue;
}
var allBreach = inWindow.All(s => t.Op == Comparison.GreaterThan ? s.Value > t.Threshold : s.Value < t.Threshold);
if (allBreach) fired.Add(new Fired(t, inWindow[0].At));
}
return fired;
}
}using Harbourline.Migration.Cutover;
// Go/no-go for the core ledger cut-over.
var criteria = new[]
{
new Criterion("G1", "Reconciliation clean for all 1,826 business dates", true, CriterionStatus.Passed, "evidence/recon-2026-11-21.txt"),
new Criterion("G2", "Dress rehearsal completed within the window", true, CriterionStatus.Passed, "evidence/rehearsal-2.md"),
new Criterion("G3", "Rollback rehearsed and timed (under 20 minutes)", true, CriterionStatus.Passed, "evidence/rollback-drill.md"),
new Criterion("G4", "Replication lag under 60 seconds", true, CriterionStatus.Passed, "cw-dashboard/dms-latency"),
new Criterion("G5", "On-call staffed, incident commander named", true, CriterionStatus.Passed, "rota/2026-11-22"),
new Criterion("G6", "Customer notice published", false, CriterionStatus.NotStarted)
};
var assessment = GoNoGoEvaluator.Evaluate(criteria);
Console.WriteLine($"Decision: {assessment.Decision}");
foreach (var r in assessment.Reasons) Console.WriteLine($" - {r}");
// Rollback triggers during hypercare.
var now = DateTimeOffset.Parse("2026-11-22T03:00:00Z");
var triggers = new[]
{
new Trigger("http_5xx_rate_pct", Comparison.GreaterThan, 1.0, TimeSpan.FromMinutes(5), "Shift DNS weight to 0 for AWS"),
new Trigger("payment_success_pct", Comparison.LessThan, 99.0, TimeSpan.FromMinutes(10), "Invoke rollback runbook R-3"),
new Trigger("reconciliation_diffs", Comparison.GreaterThan, 0.0, TimeSpan.FromMinutes(2), "Stop switch; page data lead")
};
// Sample feed: five minutes of 5xx above the limit, healthy payments, reconciliation clean.
var samples = new List<Sample>();
for (var m = 0; m <= 10; m++)
{
var at = now.AddMinutes(-10 + m);
samples.Add(new Sample("http_5xx_rate_pct", at, m >= 4 ? 2.5 : 0.2));
samples.Add(new Sample("payment_success_pct", at, 99.8));
samples.Add(new Sample("reconciliation_diffs", at, 0));
}
foreach (var f in RollbackMonitor.Evaluate(samples, triggers, now))
Console.WriteLine($"TRIGGER: {f.Trigger.Metric} since {f.Since:HH:mm} -> {f.Trigger.Action}");
return assessment.Decision == Decision.NoGo ? 2 : 0;Expected output: Decision: GoWithWaivers (G6 is optional and not started), then one trigger line for http_5xx_rate_pct. The reconciliation_diffs trigger has a two-minute window and the samples are one minute apart, so it has enough data; it does not fire because the value is zero.
Code explained line by line
Criterioncarries anEvidenceRef. The evaluator treats a pass without evidence as a problem: the change board needs a link, not an assurance.- In
Evaluate, a mandatory criterion that is failed or not started producesNoGo. A mandatory criterion cannot be waived (a waiver on a mandatory item is reported as invalid). This encodes "no one can waive the reconciliation". - A failed non-mandatory criterion is counted as a known issue, so the result becomes
GoWithWaiversrather than a silentGo. Triggerincludes aWindow: the breach must hold for the entire window, which filters single spikes.- In
RollbackMonitor.Evaluate,inWindowtakes samples in(now - window, now]. If the data does not cover the window (fewer than two samples or the oldest is too recent), the monitor reports a monitoring gap. Absence of data is not evidence of health. t with { Action = ... }creates a modified copy of the record, leaving the original trigger unchanged.allBreachusesAll(...): every sample in the window must violate the threshold. The earliest sample's time is reported as the start of the breach, which is what you put in the incident timeline.- The program's return code is 2 for NoGo, so a pipeline stage can block the switch step.
Security considerations
⚠ Learning shortcut — not for production
The lesson's evaluator and monitor are simplified: criteria are in code, the samples are fabricated, and the DNS commands run with a broad profile. In production the criteria live in a controlled document with approvers, the metrics come from CloudWatch (and the source systems), the DNS change is made by an automation role with a narrow policy (only the cut-over records), and every irreversible step requires two named approvers. Break-glass roles are time-boxed, monitored and removed after the window.
- Separation of duties. The person who approves the go decision should not be the person who executes the switch. See Federation and segregation of duties.
- Evidence integrity. Store the evidence pack in a versioned bucket with Object Lock where policy requires (S3 protection, Object Lock and Macie). CloudTrail provides the technical record of who changed what.
- Data in two places. During the rollback window both systems hold production personal data. Apply the same access controls, retention and breach monitoring to the old system until it is decommissioned, and then securely wipe it, with evidence. UK GDPR obligations continue throughout; consult your data-protection officer.
- Customer communication. A maintenance window and a customer notice are part of the plan. If the migration changes where data is processed, review whether notifications, records of processing or regulator engagement are required. That is a compliance decision, not an engineering one.
Banking use case
Harbourline schedules the ledger cut-over for the third weekend of November, away from month-end. The go/no-go pack contains the reconciliation output for every business date, two rehearsal reports, a timed rollback drill (14 minutes) and the staffing rota. The decision at T+30 minutes is "go" on all five mandatory criteria; the customer-notice criterion is carried as a known issue with an approved plan. Traffic moves 10 percent, 50 percent and 100 percent over three hours with one-hour holds. At 03:10 the 5xx trigger fires on a misconfigured connection pool in one availability zone; the incident commander uses the rehearsed step to set the AWS weight to 50, fixes the pool, and resumes. Because the point of no return was not crossed (writes were still replicating both ways through reverse replication), no data was lost. The post-implementation review adds a pool-size check to the readiness gate.
Common mistakes
- Rollback as an afterthought. If it is not rehearsed and timed, it does not exist. Rehearse it first, and set a deadline for the last safe moment.
- Defining success after the event. Criteria written during the window drift toward "it looks fine". Write them in advance, with evidence.
- Ignoring DNS caching. Some clients ignore short TTLs. Lower the TTL days ahead and expect a tail of stale clients.
- Data divergence on rollback. Once new transactions exist only on the target, going back means replaying or reversing them. Prepare reverse replication or a replay strategy, and decide who owns it.
- No freeze. An unrelated deployment during the window makes any failure impossible to attribute.
- Skipping hypercare. Most problems appear on the first business day, with real peak load. Staff for it.
- Finalising too early. Removing the replica or the old system before the rollback window ends removes your last way back.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Traffic still reaches the old estate after the DNS change | Resolver or client caching beyond the TTL; sessions pinned | Wait out TTL tails, drain old endpoints, and use load-balancer-level routing where possible |
| Weighted records give no response for AWS | Health check failing or wrong target | Check the health check path and security groups; verify the ALB DNS name |
| Alarm never fires during a rehearsal | Wrong metric, dimension or missing-data treatment | Test the alarm by forcing the condition; set treat-missing-data deliberately |
| Reconciliation differs after cut-over | In-flight transactions, late CDC, or timezone boundary | Compare by business date with a cut-off, replay CDC, and check the time zone rule |
| Rollback restores service but customers see old balances | Writes happened on the target and were not reverse-replicated | Execute the data-divergence procedure: reverse-replicate or replay the target's writes |
| Go/no-go meeting overruns | Criteria unclear or evidence missing | Move criteria and evidence links into the checklist a day earlier; use the evaluator output |
Cost and cleanup
Cost: free. The code runs locally. Route 53 hosted zones cost USD 0.50 per month and weighted records cost a small amount per million queries ⚠ VERIFY, so the DNS exercise is optional and cheap; deploy only to a test zone you own and remove it the same day.
# Remove the test records (CDK or Terraform users: destroy the stack instead).
cdk destroy
# terraform destroy
# Delete the test alarm and the rollback batch files.
aws cloudwatch delete-alarms --alarm-names hl-m13-portal-5xx --profile learn-sandbox --region us-east-1
rm -f shift-10.json
# Confirm nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab --profile learn-sandboxExercises
- 🟢 Guided: Run the program. Change criterion G1 to
Failedand confirm the decision becomesNoGoand the reason names G1. Then change G6 toWaivedwith aWaivedByand anEvidenceRefand observe the result. - 🟡 Challenge: Add a rule that a
Passedcriterion whoseEvidenceRefdoes not start withevidence/,cw-dashboard/orrota/is reported as unverifiable. Add a trigger for DMS latency (greater than 120 seconds for 5 minutes) to the monitor and test it with fabricated samples. - 🔴 Live problem: At T+45 minutes, 200 card authorisations have been approved in AWS, the 5xx rate is 3 percent and rising, and the business asks you to "just roll back". The reverse replication has been running for only 10 minutes. Write the decision note: options (roll back with replay, push forward with a fix, partial traffic shift), the data-divergence risk of each, who must approve, and the first three actions.
Assignment
Write cutover-runbook-and-rollback.md for your fictional bookstore or bank application (this feeds Lab 13.5 and the module assignment): the pattern chosen and why, the timeline table, ten go/no-go criteria with evidence references, five rollback triggers with thresholds and windows, the point of no return, the data-divergence procedure and the evidence-pack contents.
Solutions
With G1 Failed: Decision: NoGo and the reason G1: mandatory criterion Failed: Reconciliation clean for all 1,826 business dates. With G6 Waived, a WaivedBy and an EvidenceRef: the result stays GoWithWaivers, because G6 is optional and the waiver is valid. If you instead mark a mandatory criterion such as G4 as Waived, the result is NoGo with "waiver invalid", because mandatory criteria cannot be waived in this model.
Why: a waiver must mean something specific. Optional criteria can be accepted as known issues by a named approver with evidence; mandatory criteria (reconciliation, rollback drill) protect customers' money and cannot be waived by the people running the migration. If your organisation allows that in an emergency, it needs a documented higher-level approval outside this tool.
Add string[] allowed = ["evidence/", "cw-dashboard/", "rota/"]; and in the Passed branch, if (!allowed.Any(a => c.EvidenceRef.StartsWith(a, StringComparison.Ordinal))) reasons.Add(...). For DMS latency use new Trigger("dms_latency_seconds", Comparison.GreaterThan, 120, TimeSpan.FromMinutes(5), "Pause switch; investigate replication") and generate six minutes of samples, one per minute, at 150 to see it fire, and at 90 to see it stay quiet.
Why: evidence that cannot be located later is not evidence. A trigger on replication latency protects the part of the plan most likely to silently break: the data.
Knowledge check
Answer: The moment after which going back would lose or corrupt data, typically when new transactions exist only on the target. Plan it as late as possible and cross it only with an explicit decision, because rollback options shrink sharply after it.
Answer: A single bad sample is usually noise. A sustained breach is a signal. The window filters spikes while still reacting quickly to a real fault.
Answer: Resolvers and clients cache records for the old TTL. If you lower it only at the switch, caches keep pointing at the old answer for the old TTL. Lower it days ahead and verify.
Answer: A dead monitor is indistinguishable from a healthy system if you treat silence as success. Failing closed makes missing data visible and forces a human decision.
Interview questions
- What goes into a go/no-go decision? Pre-agreed mandatory criteria with evidence (reconciliation, rehearsal, rollback drill, lag, staffing), named approvers, and a recorded decision at each checkpoint.
- How do you roll back a database migration after writes on the target? Only with reverse replication or a replay strategy rehearsed beforehand; otherwise the rollback loses transactions. It is better to plan the point of no return than to improvise.
- How do you reduce the blast radius of a cut-over? Phase by segment, shift traffic gradually with a health signal, keep a short TTL, hold hypercare staffing, and keep the old system intact for the rollback window.
Certification practice
A bank is cutting over a customer-facing API from its data centre to AWS. It wants to move traffic gradually, with the ability to return to the old environment within minutes if error rates rise. Which approach meets this?
A. Replace the DNS record with the new endpoint and delete the old environment immediately. B. Use Route 53 weighted records with a low TTL, shift traffic in steps, monitor with CloudWatch alarms, and keep the old environment until the rollback window ends. C. Use a single Route 53 simple record and change it manually when problems are reported. D. Move all traffic at once and rely on AWS support to roll back.
Answer: B. Weighted records with a low TTL allow gradual shifts and fast reversal; alarms provide the trigger; keeping the old environment preserves the way back. A removes rollback, C has no gradual control, D has no plan.
Which health-check approach is most suitable when the new application endpoint is in a private subnet and not reachable from the internet, but you still want DNS failover decisions based on its health?
A. A standard Route 53 HTTP health check to the private IP address. B. A CloudWatch alarm on application metrics, with a Route 53 health check that monitors the alarm state. C. No health check; rely on manual monitoring. D. An S3 event notification.
Answer: B. Route 53 health checkers are outside your VPC and cannot reach private endpoints; a health check that monitors a CloudWatch alarm works for private resources. A cannot connect, C is slow and error-prone, D is unrelated.
Completion checklist
- I can name the five phases of a cut-over and the point of no return
- I ran the evaluator and the monitor and understood each output
- I wrote at least five go/no-go criteria with evidence references
- I can explain how weighted DNS gives a rollback path and where it falls short
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- Decide go/no-go criteria and rollback triggers before the rehearsal, and write them as tests with evidence.
- The point of no return is a decision, not an accident; make it as late as possible.
- Rollback must be rehearsed, timed and have a data-divergence plan.
- Shift traffic gradually with a low TTL and health signals; expect DNS caching tails.
- Hold the rollback window and hypercare, and only then finalise and decommission.
Further reading
- Operational resilience and third-party risk — why a migration is a resilience event for a bank
- Module 13 overview — the module project and Capstone 6 build on this runbook
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Windows and SQL Server licensing on AWS
Understand license-included versus bring-your-own-licence for Windows Server and SQL Server on AWS, when Dedicated Hosts and License Mobility apply, how to model the cost of staying on Windows against moving to Linux or PostgreSQL, and how to track licences with AWS License Manager. Informational, not legal advice.
Module 13 cheat sheet
One-page revision summary for Microsoft-to-AWS Migration: services, Azure mapping, CLI commands, limits, decision rules, exam traps and cost traps.