Failover with Route 53: health checks, failover routing, TTLs and split brain
Design DNS-based Regional failover with Route 53 health checks and failover or weighted records, calculate how long failover really takes, understand the control-plane trap and Application Recovery Controller routing controls, and prevent split brain.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.Route53 4.1.1.2) · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Route 53 API in us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain how a Route 53 health check decides that an endpoint is healthy, including the checker share rule, the intervals and the three health check types.
- Build failover records (primary and secondary) with alias records and health checks, and compare them with weighted, latency and calculated-health designs.
- Calculate the worst-case failover time from probe interval, failure threshold and TTL, and say why client caching can make it longer.
- Explain the control-plane versus data-plane difference and why a failover that edits records during an outage is fragile.
- Describe Application Recovery Controller (ARC) routing controls and how they differ from ordinary health checks, and note the status of ARC capabilities.
- Describe split brain and use fencing (an epoch guard) plus a deliberate failback process to prevent it.
- Check a health check's status from C# and read it against the documented rule.
Prerequisites
- DNS and Route 53 — hosted zones, record types, alias records and routing policies in general
- The four DR strategies and Active-active vs active-passive
- ELB for resilience — the load balancers that the records point at
- Tools: Docker for the lab tests; for the optional hands-on part, a domain you control in a Route 53 hosted zone (or skip it: the lesson works as a design exercise)
Why this matters in banking
At 14:07 the Accounts API in the primary Region stops answering. At 14:08 an engineer opens the Route 53 console to change the DNS record to the DR Region. The console is slow. The change goes through at 14:15. Some customers move at once; others keep sending requests to the dead Region for another ten minutes because their resolver cached the old answer for 300 seconds, and a few mobile clients hold on to it for much longer.
At 14:20 the primary Region partially recovers, and the automatic health check marks it healthy again. Route 53 begins answering with the primary address once more, while the database in the DR Region has already been promoted and has accepted payments. Now two databases both believe they are the source of truth. Reconciling them takes the weekend.
This story contains the four lessons of this page: DNS failover is slower than it looks (detection plus TTL plus client caches), editing records during an incident uses the control plane, automatic failback is dangerous, and without a guard you can get split brain.
The idea in plain language
DNS failover means: clients ask DNS for the address of api.harbourline.example. Route 53 answers with the address of the primary Region while it is healthy, and with the address of the secondary Region when it is not. Switching Regions is just giving a different answer.
Three pieces make it work:
- Health checks (Route 53 probes your endpoint, or watches an alarm, and decides healthy or unhealthy).
- Routing policy (failover, weighted, latency and so on) that uses health to choose answers.
- TTL (how long clients may cache the answer).
Analogy: the bank prints "Branch for this district" on a signpost. Inspectors check the main branch every few seconds (health checks). If it is closed, the sign is repainted to point at the backup branch (routing policy). But drivers who read the old sign an hour ago keep going to the old address until they look again (TTL and caches). And if the main branch reopens and the sign is repainted back while the backup branch has already been taking deposits, you now have two branches with different records (split brain).
How a health check decides
From the Route 53 documentation:
- Health checkers in locations around the world probe the endpoint. You choose the interval: every 30 seconds (standard) or every 10 seconds (fast, which costs more) ⚠ VERIFY pricing.
- Each checker applies a failure threshold: it marks the endpoint unhealthy after that many consecutive failed checks (3 is the default in the console ⚠ VERIFY).
- Route 53 aggregates: more than 18 percent of checkers reporting healthy means the endpoint is healthy; 18 percent or fewer means unhealthy. The documentation says this value was chosen so that a network problem isolating the endpoint from some locations does not trigger failover, and that it might change.
- Response rules: for HTTP and HTTPS, Route 53 must make a TCP connection within 4 seconds, then the endpoint must answer with a 2xx or 3xx status within 2 seconds. TCP health checks need a connection within 10 seconds. HTTPS checks do not validate certificates, so an expired certificate does not fail them. String matching looks in the first 5,120 bytes of the response body.
- A new health check is considered healthy until it has enough data.
There are three types of health check:
| Type | What it watches | Use when |
|---|---|---|
| Endpoint | A public IP or domain name, path and port | The endpoint is reachable from the internet (an ALB, an API) |
| Calculated | Other health checks (up to 255 children); healthy if at least N children are healthy | You need AND/OR logic, for example "API healthy AND database path healthy" |
| CloudWatch alarm | The data stream behind an alarm (not the alarm state itself) | The endpoint is private, or the signal is a metric (error rate, replication lag). Cross-account alarms are not supported |
Because a health check follows the alarm's data stream, you cannot force it by calling SetAlarmState: nice to know when testing.
Routing policies for resilience
| Policy | How it chooses | Resilience role |
|---|---|---|
| Failover | Primary while healthy, otherwise secondary | Active-passive Regional failover |
| Weighted | Splits traffic by weight; unhealthy records are skipped | Active-active, gradual shifts, manual failover by changing weights (control-plane action) |
| Latency | Lowest-latency healthy Region | Active-active with locality |
| Geolocation / geoproximity | By user location or bias | Data residency and sovereignty routing |
| Multivalue answer | Up to eight healthy records | Simple client-side spreading |
For an alias record to an AWS resource such as an ALB, enable evaluate target health: Route 53 then uses the target's own health (for an ALB, whether it has healthy targets) so the record is withdrawn when the ALB is unhealthy, even without a separate health check. For anything you want to be tied to a specific probe, attach a health check to the record.
TTL, caches and what "failover time" means
worst-case failover time = detection time + TTL where detection time = check interval x failure threshold.
| Interval | Threshold | TTL | Worst case for well-behaved clients |
|---|---|---|---|
| 10 s | 3 | 60 s | 90 s |
| 30 s | 3 | 60 s | 150 s |
| 30 s | 3 | 300 s | 390 s |
| 10 s | 1 | 20 s | 30 s (and a false-alarm risk) |
Alias records to AWS resources use a TTL that Route 53 sets for you (60 seconds for ALB aliases ⚠ VERIFY). Resolvers and applications may ignore or extend TTLs: some resolvers enforce a minimum, and long-lived connections keep talking to the old address regardless of DNS. So the table is a lower bound. Measure with the clients you actually have: browsers, mobile apps, partner systems and third-party resolvers.
The control-plane trap and routing controls
Route 53 is a global service whose control plane (the API that creates and edits records and health checks) is hosted in us-east-1. The data plane (answering DNS queries, running health checks and switching answers based on them) is distributed and designed for far higher availability. The AWS DR guidance is therefore: prepare with the control plane, execute with the data plane. Automatic failover driven by health checks is a data-plane action. Editing a record or changing weights during an incident is a control-plane action, and may be slow or fail when you need it most.
For a manually initiated failover that is still data-plane based, AWS describes Amazon Application Recovery Controller (ARC) routing controls: you create a routing control (a switch with On and Off) in a cluster. Each routing control is linked to a Route 53 health check. You attach that health check to your failover record. Flipping the routing control changes the health check's state, and Route 53 changes its answers accordingly. You flip the switch through the cluster's data-plane endpoints, which are highly available (a cluster has five endpoints in different Regions, and you should try all of them) ⚠ VERIFY the current endpoint count and Regions. This is the pattern for "a human decides, and the action is reliable even when a Region is down".
A note on ARC's other features: zonal shift and zonal autoshift (AZ level, covered in ELB for resilience) and readiness check. The module overview records that Readiness Check is in maintenance and closed to new customers (announced 2026-03-31); other ARC capabilities such as routing control are not listed as affected ⚠ VERIFY the current status before building anything on routing controls. Routing controls are billed per cluster-hour ⚠ VERIFY, so keep a lab cluster for minutes, not days.
Global Accelerator is the other common option: two static anycast IP addresses front endpoints in several Regions; its health checks and traffic dials move traffic without DNS caching, at the price of the service itself. It avoids DNS TTL problems but traffic-dial changes are control-plane operations, as noted in AWS's DR guidance.
Split brain and fencing
Split brain is two copies of the system both believing they are the primary and both accepting writes. It happens when failover is triggered by an observation (a health check cannot reach the primary) rather than a fact (the primary is dead and fenced). Typical triggers:
- The primary is alive but cut off from the health checkers (a partial network failure).
- The health check flaps: it fails, the DR database is promoted, then the check recovers and traffic returns to the old primary.
- An automatic failback sends traffic to the old Region before data has been reconciled.
- Clients with cached DNS keep writing to the old primary while new clients write to the new one.
Protection comes from several layers:
- One decision point. Prefer a deliberate, scripted, human-approved Regional failover (routing control or runbook) over fully automatic failover. Reserve automation for AZ-level events.
- Promote the data layer before moving traffic, and treat promotion as one-way. For Aurora Global Database, a failover detaches the old primary from the global cluster; it does not rejoin by itself.
- Fence the old primary. Stop it, make it read-only, remove its network access or revoke its credentials, and use an epoch (fencing token): every writer carries the epoch it was promoted under, and the data layer rejects writes with an older epoch. The lab's
EpochFencemodels the rule. - No automatic failback. Failback is a planned change: re-establish replication in the opposite direction, verify, then switch during a quiet window. Keep the failover health check "sticky" by using a routing control rather than a probe that can flip back.
- Reconcile. If writes happened on both sides, you need a documented reconciliation process before reopening.
What problem the AWS service solves
Route 53 gives you a managed, globally distributed authoritative DNS service with health-checked routing, so you can express "send customers to the healthy Region" as configuration, and have it run on the highly available data plane. ARC routing controls add a reliable manual override.
Honest limits:
- DNS is a blunt instrument. It affects new lookups only; existing connections, long-lived clients, resolvers that ignore TTLs and hard-coded IP addresses are not moved.
- Health checks are shallow and external. They see what the internet sees. They cannot see a corrupted database or a failing payment scheme, and they cannot probe private endpoints directly (use a CloudWatch alarm check, or a public canary).
- Automatic Regional failover is risky for stateful systems because of false positives and split brain.
- Control plane in us-east-1. Preparing records is a control-plane activity; make sure the failover record set already exists.
- Cost. Health checks are billed per check, with extras for fast intervals, HTTPS and string matching ⚠ VERIFY; ARC clusters are billed hourly ⚠ VERIFY.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Traffic Manager (priority, weighted, performance) | Route 53 failover, weighted, latency policies | Health is evaluated by Route 53 checkers, calculated checks, or alarm streams |
| Traffic Manager endpoint monitor | Route 53 health check | 30 s or 10 s intervals, 18 percent checker rule, HTTPS checks do not validate certificates |
| Traffic Manager profile TTL | Record TTL (alias TTL set by Route 53) | Same caching problem on both clouds |
| Azure Front Door origin failover (no DNS TTL on the data path) | CloudFront origin groups, Global Accelerator | Anycast and edge termination avoid DNS cache delays |
| Manually disabling an endpoint in a profile | ARC routing control | Switches live on a dedicated highly available data plane |
| Azure SQL failover group read/write listener | Aurora Global Database failover plus a Route 53 record | The database endpoint does not follow a Regional failover by itself; you re-point or use a global endpoint pattern ⚠ VERIFY |
| Azure Monitor alert driving Traffic Manager | CloudWatch alarm-based Route 53 health check | A way to fail over on a metric (error rate, replication lag), not just reachability |
Architecture
The timeline of a DNS failover and where the time goes:
Security-boundary view: protect the ability to change routing. In the primary account, only a break-glass role can touch the failover records and routing controls; ARC cluster endpoints need IAM permissions per call. Health check endpoints must reveal nothing sensitive. In the DR Region, the secondary stack runs with its own IAM and KMS; the DNS records (and the hosted zone) live in a protected account that survives the loss of either workload Region.
Step-by-step implementation
Decide the failover model
For the Accounts API (warm standby, RTO 30 minutes), choose human-approved failover through an ARC routing control for Regional events, and keep automatic behaviour only for AZ-level issues inside each Region. Write down who may flip the control.
Make the endpoints health-checkable
Each Region's ALB exposes /health/ready (the local-only readiness from ELB for resilience) on a public listener restricted as far as possible, or you use a CloudWatch-alarm-based health check against an internal canary that tests the real request path.
Create the health checks
An endpoint health check per Region (interval 10 s if the extra cost is acceptable, failure threshold 3, inverted off), optionally a calculated check that combines the endpoint check with a replication-lag alarm check, so the DR Region is only considered a valid target when its data is fresh.
Create the failover records
Two A alias records for the same name with set identifiers primary and secondary, failover type PRIMARY and SECONDARY, the primary tied to its health check with evaluate target health on. Keep TTLs low for any non-alias record.
Add the routing control (optional, paid)
Create an ARC cluster and a control panel with one routing control per Region, attach each routing control's health check to its record, and store the five cluster endpoints and the failover script in the runbook.
Calculate and test the failover time
Use Route53HealthModel.WorstCaseFailoverTime for the paper number, then measure with real clients while you flip the control in a pre-production environment. Record the result.
Design the guard rails
Promote the database first, then switch traffic; fence the old primary (epoch, read-only, network cut-off); no automatic failback; a reconciliation step; an alarm on "both Regions report writer".
AWS Console instructions
- Health checks: Console → Route 53 → Health checks → Create health check. Choose Endpoint, Status of other health checks (calculated) or State of CloudWatch alarm. Set the request interval (Standard (30 seconds) or Fast (10 seconds)) and the failure threshold in the advanced configuration.
- Failover records: Console → Route 53 → Hosted zones → your zone → Create record → Routing policy: Failover. Choose Primary or Secondary, set the record ID (set identifier), attach the health check, and for alias records turn on Evaluate target health.
- ARC: Console → Route 53 Application Recovery Controller → Routing control (create a cluster, a control panel and routing controls; copy the cluster endpoints). Labels can differ, so search for "routing control" ⚠ VERIFY.
AWS CLI
PROFILE=learn-sandbox
ZONE_ID=Z0123456789EXAMPLE # your hosted zone ID
NAME=api.example.com # a name you control in that zone
# 1. Create an endpoint health check against the primary ALB (fast checks, 3 failures).
CALLER=hl-m12-$(date +%s)
aws route53 create-health-check --caller-reference "$CALLER" --profile $PROFILE \
--health-check-config '{
"Type": "HTTPS",
"FullyQualifiedDomainName": "primary-alb.us-east-1.elb.amazonaws.com",
"Port": 443,
"ResourcePath": "/health/ready",
"RequestInterval": 10,
"FailureThreshold": 3,
"EnableSNI": true
}'
# 2. Failover record set: PRIMARY with the health check, SECONDARY without (the last resort).
cat > failover-records.json <<'JSON'
{
"Comment": "Harbourline Accounts API: Regional failover",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "api.example.com", "Type": "A", "SetIdentifier": "primary", "Failover": "PRIMARY",
"HealthCheckId": "<health-check-id-from-step-1>",
"AliasTarget": { "HostedZoneId": "Z35SXDOTRQ7X7K", "DNSName": "primary-alb.us-east-1.elb.amazonaws.com", "EvaluateTargetHealth": true }
}
},
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "api.example.com", "Type": "A", "SetIdentifier": "secondary", "Failover": "SECONDARY",
"AliasTarget": { "HostedZoneId": "Z1H1FL5HABSF5", "DNSName": "secondary-alb.us-west-2.elb.amazonaws.com", "EvaluateTargetHealth": true }
}
}
]
}
JSON
aws route53 change-resource-record-sets --hosted-zone-id $ZONE_ID --change-batch file://failover-records.json --profile $PROFILE
# 3. Watch health: how each checker region sees the endpoint.
aws route53 get-health-check-status --health-check-id <health-check-id> --profile $PROFILE
# 4. See what DNS is actually answering (the resolver's view, subject to caching).
dig +short api.example.comThe ALB hosted-zone IDs in the JSON (Z35SXDOTRQ7X7K, Z1H1FL5HABSF5) are example values for the two Regions: always read the correct hosted-zone ID for your load balancer from aws elbv2 describe-load-balancers (field CanonicalHostedZoneId). The commands manage real DNS records: use a test name in a zone you own, and delete the records and the health check afterwards.
Infrastructure as Code
The lab stack includes an optional Terraform health check and PRIMARY record (enabled when you set hosted_zone_id and record_name). The pattern for the pair is:
// Failover pair for an ALB in each Region (illustrative; needs a hosted zone you own).
using Amazon.CDK.AWS.Route53;
using Amazon.CDK.AWS.Route53.Targets;
var zone = HostedZone.FromHostedZoneAttributes(this, "Zone", new HostedZoneAttributes
{
HostedZoneId = "Z0123456789EXAMPLE",
ZoneName = "example.com",
});
var primaryCheck = new CfnHealthCheck(this, "PrimaryCheck", new CfnHealthCheckProps
{
HealthCheckConfig = new CfnHealthCheck.HealthCheckConfigProperty
{
Type = "HTTPS",
FullyQualifiedDomainName = primaryAlb.LoadBalancerDnsName,
Port = 443,
ResourcePath = "/health/ready",
RequestInterval = 10,
FailureThreshold = 3,
EnableSni = true,
},
});
new CfnRecordSet(this, "Primary", new CfnRecordSetProps
{
HostedZoneId = zone.HostedZoneId,
Name = "api.example.com",
Type = "A",
SetIdentifier = "primary",
Failover = "PRIMARY",
HealthCheckId = primaryCheck.AttrHealthCheckId,
AliasTarget = new CfnRecordSet.AliasTargetProperty
{
DnsName = primaryAlb.LoadBalancerDnsName,
HostedZoneId = primaryAlb.LoadBalancerCanonicalHostedZoneId,
EvaluateTargetHealth = true,
},
});
// Repeat for "secondary" with Failover = "SECONDARY" and the DR Region's ALB (no health check required).Complete code
Three pieces from the lab: the health-check arithmetic, the fencing guard, and a read-only C# report of a real health check. All three are in labs/m12-resilient-account-api, and the first two are covered by Route53HealthModelTests and EpochFenceTests.
namespace Harbourline.Resilience;
public static class Route53HealthModel
{
/// <summary>The documented healthy-checker share (more than 18 percent means healthy).</summary>
public const double HealthyCheckerThreshold = 0.18;
public static bool IsEndpointHealthy(int healthyCheckers, int totalCheckers)
{
if (totalCheckers < 0 || healthyCheckers < 0 || healthyCheckers > totalCheckers)
{
throw new ArgumentOutOfRangeException(nameof(healthyCheckers), "Healthy checkers must be between 0 and the total.");
}
// No observations yet: Route 53 considers a new health check healthy until it has enough data.
return totalCheckers == 0 || (double)healthyCheckers / totalCheckers > HealthyCheckerThreshold;
}
public static TimeSpan DetectionTime(TimeSpan checkInterval, int failureThreshold)
{
if (failureThreshold < 1) throw new ArgumentOutOfRangeException(nameof(failureThreshold));
return checkInterval * failureThreshold;
}
// Detection, then one TTL for cached answers. Clients that ignore TTLs add more.
public static TimeSpan WorstCaseFailoverTime(TimeSpan checkInterval, int failureThreshold, TimeSpan ttl) =>
DetectionTime(checkInterval, failureThreshold) + ttl;
}namespace Harbourline.Resilience;
public sealed class EpochFence
{
private long _epoch;
public EpochFence(long initialEpoch = 1)
{
if (initialEpoch < 1) throw new ArgumentOutOfRangeException(nameof(initialEpoch));
_epoch = initialEpoch;
}
public long Current => Interlocked.Read(ref _epoch);
// Called by the failover runbook when a new writer is promoted. Epochs only move forward.
public bool TryAdvance(long newEpoch)
{
long seen;
do
{
seen = Interlocked.Read(ref _epoch);
if (newEpoch <= seen) return false; // stale or duplicate failover command
}
while (Interlocked.CompareExchange(ref _epoch, newEpoch, seen) != seen);
return true;
}
// True only for the current epoch. Writes from older epochs must be rejected.
public bool AllowsWriteFrom(long writerEpoch) => writerEpoch == Current;
}using Amazon.Route53;
using Amazon.Route53.Model;
using Harbourline.Resilience;
static async Task<int> RunHealthCheckReportAsync(string healthCheckId)
{
// Route 53 is a global service: its API endpoint is in us-east-1 regardless of where you run the tool.
using var client = new AmazonRoute53Client(Amazon.RegionEndpoint.USEast1);
var response = await client.GetHealthCheckStatusAsync(new GetHealthCheckStatusRequest { HealthCheckId = healthCheckId });
var observations = response.HealthCheckObservations;
foreach (var o in observations.OrderBy(o => o.Region?.Value))
{
Console.WriteLine($"{o.Region?.Value,-16} {o.StatusReport?.Status}");
}
var healthy = observations.Count(o => o.StatusReport?.Status?.StartsWith("Success", StringComparison.OrdinalIgnoreCase) == true);
var verdict = Route53HealthModel.IsEndpointHealthy(healthy, observations.Count) ? "HEALTHY" : "UNHEALTHY";
Console.WriteLine($"{healthy} of {observations.Count} checkers report success -> {verdict} (Route 53 rule: more than 18 percent)");
return verdict == "HEALTHY" ? 0 : 1;
}Run it with dotnet run --project src/Harbourline.Resilience.Tools -- r53 <health-check-id> after aws sso login --profile learn-sandbox. The status text returned by Route 53 begins with Success for a passing probe (for example, Success: HTTP Status Code 200, OK) ⚠ VERIFY against a live response, since the tool relies on that prefix.
Code explained line by line
HealthyCheckerThreshold = 0.18is the documented line, andIsEndpointHealthycompares the healthy share with it using "more than". At exactly 18 percent the endpoint counts as unhealthy, as the lab testExactly_18_percent_is_not_enoughshows.totalCheckers == 0returns healthy, because Route 53 treats a new health check as healthy until it has data. Tests lock this in so nobody "fixes" it.DetectionTimemultiplies the interval by the failure threshold: 10 s x 3 = 30 s.WorstCaseFailoverTimeadds the TTL, which is the arithmetic from the table earlier.- In
EpochFence.TryAdvance, a compare-and-swap loop makes the epoch advance atomically: of twenty simultaneous commands for epoch 2, exactly one returnstrue. This is what a DynamoDB conditional write (attribute_not_existsor "epoch less than new epoch") gives you in a real system. AllowsWriteFrom(writerEpoch)is the fence. After the failover advances the epoch to 2, a zombie primary presenting epoch 1 is refused, even if clients with cached DNS still reach it.- The health check report creates the Route 53 client with
RegionEndpoint.USEast1because Route 53's API endpoint is global and served from us-east-1. GetHealthCheckStatusAsyncreturns one observation per checker location. The tool counts the observations whose status starts withSuccess, then applies the same 18 percent rule so that you can compare your own reading with what Route 53 decided.
Security considerations
- Who can change DNS can redirect customers. Treat the hosted zone and failover records as critical assets. Restrict
route53:ChangeResourceRecordSetsto a deployment pipeline and a break-glass role, enable CloudTrail data and management events, and consider a Service Control Policy or Route 53 DNSSEC (⚠ VERIFY the current DNSSEC options) for the zone. - Failover is also a way to attack availability. An attacker who can fail health checks (for example, by making the primary appear unhealthy) can force traffic onto the secondary. Keep health endpoints simple, rate-limited and not dependent on attacker-influenced inputs.
- Health check endpoints leak less if they are boring. Return a short status; never include versions, hostnames or dependency states.
- HTTPS health checks do not validate certificates. Monitor certificate expiry separately (ACM expiry alarms, canaries).
- Break-glass for routing controls. Require MFA, log every state change, and rehearse with the same role.
- Split brain is an integrity incident. Treat any evidence of two writers as a security and data-integrity event for the incident process (Incident response).
⚠ Learning shortcut — not for production
The lab's optional Route 53 part creates only a PRIMARY record and a health check pointing at a single ALB, and uses plain HTTP to avoid certificate setup. A production failover design has both records, HTTPS everywhere, a protected hosted zone in a separate account, a documented decision to fail over, a fencing mechanism, and a tested failback. Fast (10-second) health checks, ARC clusters and string matching cost extra: check prices before turning them on, and delete lab records and health checks when you finish.
Banking use case
Harbourline's Accounts API runs warm standby: us-east-1 primary, us-west-2 secondary (a design exercise region for this course; a real UK bank would choose Regions its data-protection officer approves). The agreed failover design:
| Aspect | Decision |
|---|---|
| Routing | Failover records for api.harbourline.example, ALB aliases, evaluate target health on |
| Health checks | Endpoint check on /health/ready every 10 s, threshold 3; calculated check that also requires "Aurora Global replication lag under 60 s" for the secondary |
| TTL | Alias default; clients and partner resolvers tested; mobile app retries with a fresh lookup after repeated connection failures |
| Regional failover | Human-approved, executed by a script that (1) fences the old primary (epoch advance and security-group lockdown), (2) promotes Aurora in the DR Region, (3) scales the DR service, (4) flips the ARC routing control, (5) verifies with synthetic payments |
| Failback | Planned change, never automatic; replication re-established and verified first |
| Evidence | Quarterly drill; measured RTO and RPO recorded; "worst case on paper" 90 s plus runbook time |
Worst case on paper for DNS alone with 10 s checks, 3 failures and a 60 s TTL is 90 seconds; the full runbook is in the tens of minutes, because the human and the database dominate the timeline. The numbers are fictional teaching examples.
Common mistakes
- A long TTL on failover records. Failover takes as long as the TTL on top of detection.
- Editing records during the incident instead of having failover driven by pre-created health-checked records or a routing control.
- Automatic failback. It reintroduces the old primary before data is reconciled.
- Health checks that depend on deep dependencies and cause needless failovers, or that are so shallow they never detect a real data-path failure. Choose deliberately.
- No evaluation of the secondary's health. Failing over to a Region that cannot serve is worse than staying.
- Forgetting private endpoints. Route 53 checkers cannot reach private IPs: use a CloudWatch alarm check on a canary instead.
- Assuming DNS moves everyone. Long-lived connections, IP-pinned partners and DNS-ignoring clients stay behind.
- No fencing. The old primary keeps accepting writes from cached clients.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Health check shows unhealthy but the site works | Health checkers cannot reach the endpoint (security group, WAF, path) | Allow Route 53 health checker IP ranges; use a path that does not need authentication; check the response code and the 2 s response limit |
| Health check shows healthy while the site is broken | The path is shallow (static page) | Use a path that exercises what matters, local to the task, or add a calculated or alarm-based check |
| Traffic did not move after failover | TTL and client caches, or a sticky client | Measure with real clients; lower TTLs ahead of time; add client retry with re-resolution |
| Secondary record never used | Primary check never failed, or the secondary is not allowed to answer | Check statuses; test by inverting the check in a lab; verify record types and names match |
| Route 53 API call fails or is slow during an incident | Control plane dependency | Use routing controls or pre-created records; avoid manual edits in the runbook |
| Both Regions accepted writes | Fencing missing, DNS flap, automatic failback | Add epoch fencing, read-only old primary, sticky failover, no automatic failback |
get-health-check-status shows few observations | New check or recently changed | Wait for data; remember new checks start healthy |
| ARC routing control update fails against one endpoint | That cluster endpoint's Region is impaired | Retry each of the five cluster endpoints |
Cost and cleanup
The reading and the lab tests are free. The optional hands-on part costs a little: a hosted zone is billed monthly, health checks are billed per check per month (fast intervals and HTTPS extras cost more), and an ARC cluster is billed per hour ⚠ VERIFY current prices in the Pricing Calculator. If you create these, delete them the same day.
PROFILE=learn-sandbox
# Delete the failover records you created (use the same change batch with Action DELETE), then the health checks.
sed 's/"UPSERT"/"DELETE"/' failover-records.json > delete-records.json
aws route53 change-resource-record-sets --hosted-zone-id $ZONE_ID --change-batch file://delete-records.json --profile $PROFILE
aws route53 delete-health-check --health-check-id <health-check-id> --profile $PROFILE
# Terraform route: remove the optional pieces by unsetting the variables or destroying the stack.
cd labs/m12-resilient-account-api/infra/terraform && terraform destroy
# Confirm nothing tagged for this lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m12-ha \
--profile $PROFILE --region us-east-1 --query "ResourceTagMappingList[].ResourceARN"If you created an ARC cluster, delete the routing controls, control panels and the cluster in the console; clusters bill by the hour.
Exercises
- 🟢 Guided: Run
Route53HealthModelTestsandEpochFenceTests. Fill in a table of interval, threshold, TTL and the worst-case failover time for five combinations. - 🟡 Challenge: Harbourline wants a failover that completes in under 60 seconds for well-behaved clients. Choose the interval, threshold and TTL, state the false-alarm trade-off, and explain what Global Accelerator would change (and what it would not).
- 🔴 Live problem: "Last quarter's drill showed that after we flipped DNS to the DR Region, about 15 percent of mobile clients kept using the primary for 40 minutes, and for 5 of those minutes both databases accepted writes." Diagnose both problems and write the fix list for client behaviour, DNS design, fencing and the runbook.
Assignment
Write the Regional failover runbook for Harbourline's Accounts API in docs/failover-runbook.md (start from dr-plan-template.md in the lab): decision criteria, the ordered steps with owners and measured or estimated minutes, the guard rails (fencing, no automatic failback), the verification checks, the failback plan and a drill schedule. Compute the slack against the RTO with FailoverBudget.
Solutions
Examples: 10 s x 3 + 60 s = 90 s; 30 s x 3 + 60 s = 150 s; 30 s x 3 + 300 s = 390 s; 10 s x 2 + 30 s = 50 s; 10 s x 1 + 20 s = 30 s. Run the epoch tests and notice that exactly one of twenty simultaneous failover commands wins. That is the property that stops two operators (or a script and an operator) from promoting two different Regions.
For under 60 seconds with well-behaved clients: for example interval 10 s, threshold 2 (20 s detection) and a TTL of 30 s, giving 50 s. The trade-off is a higher false-alarm rate (two missed 10 s probes), and low TTLs increase resolver traffic. Global Accelerator removes DNS caching from the picture (clients keep the same static anycast IPs and the service moves traffic using endpoint health), so client-side DNS delay disappears. It does not remove the need for the data layer to be promoted, it adds its own cost, and traffic-dial changes are control-plane actions ⚠ VERIFY current behaviour. DNS alone cannot be relied on for sub-minute failover with arbitrary clients.
Slow clients (15 percent for 40 minutes): clients cached the old answer well beyond the TTL or held long-lived connections or hard-coded IPs. Fix: client-side retry that re-resolves DNS after repeated failures and enforces a maximum connection lifetime, a lower TTL on non-alias names, partner communication about DNS behaviour, and consider Global Accelerator or edge termination for mobile traffic. Split brain for 5 minutes: the old primary was alive and still reachable by cached clients while the DR database accepted writes. Fix: fence the old primary as step one of the runbook (epoch advance and rejection of older epochs, read-only mode, security group lockdown), promote the database before moving traffic, use a routing control so the decision is single and sticky, ban automatic failback, and add an alarm and a reconciliation procedure. Record the drill findings and re-test.
Knowledge check
Answer: Health checkers in multiple locations probe the endpoint; Route 53 considers it healthy when more than 18 percent of checkers report healthy. For HTTP(S) the checker must connect within 4 seconds and get a 2xx or 3xx response within 2 seconds. The documentation says the 18 percent value might change.
Answer: About 90 seconds: 30 seconds to detect (10 x 3) plus up to one TTL (60 s) for cached answers. Clients that ignore TTLs or hold long-lived connections take longer.
Answer: Editing records uses the Route 53 control plane (hosted in us-east-1), which has lower availability design goals than the data plane. Health-check-driven answers and ARC routing control state changes use the data plane, which is designed to keep working during large events.
Answer: Two copies of the system both act as primary and accept writes. Prevent it with a single failover decision point, promoting the data layer before traffic, fencing the old primary (read-only, network cut-off, epoch or fencing token), and avoiding automatic failback until data is reconciled.
Answer: Route 53 health checkers are outside your VPC and cannot reach private addresses. Use a CloudWatch alarm-based health check (for example on a canary that tests the internal endpoint) instead.
Interview questions
- "How would you design Regional failover for a customer-facing API?" Health-checked failover records or Global Accelerator, readiness endpoints local to each task, a calculated check that includes data freshness, a human-approved failover script with fencing and a routing control, low TTLs, client retry behaviour, no automatic failback, and a measured drill.
- "Why not fail over automatically on any failed health check?" Health checks can fail for reasons other than the primary being down (partial network failures, flapping), and Regional failover of a stateful system risks split brain and data loss. Automatic action suits AZ-level events; Regional events usually need a decision.
- "What is the difference between a routing control and a health check?" A health check observes an endpoint or alarm. A routing control is a manual switch on a highly available data plane, linked to a health check, so you decide traffic placement reliably even during an outage.
- "What is fencing?" Making sure the old primary cannot write after failover, with a monotonically increasing epoch token, read-only mode or removal of its network and credentials.
Certification practice
A company runs an application in two Regions. It wants Route 53 to send users to the primary Region and automatically switch to the secondary Region if the primary becomes unhealthy. Which configuration meets this?
A. A weighted routing policy with equal weights and no health checks. B. A failover routing policy with a health check on the primary record. C. A geolocation routing policy. D. A simple routing policy with two values.
Answer: B. Failover routing with a health check on the primary returns the secondary when the primary is unhealthy. A and D do not evaluate health, and C routes by user location.
An operations team needs a Route 53 health check for an application endpoint that is only reachable inside a VPC. Which approach works?
A. An endpoint health check on the private IP address. B. A health check based on a CloudWatch alarm that monitors a metric emitted by an internal canary. C. A calculated health check with no children. D. A TCP health check with a longer interval.
Answer: B. Route 53 health checkers cannot reach private IP addresses, so use a CloudWatch alarm-based health check driven by a metric from inside the VPC. A and D cannot reach the endpoint; C has no signal.
A financial company wants to fail over its application to a second Region on a manual decision. It needs the failover action itself to remain reliable even if one Region is experiencing a large-scale event, and it does not want to depend on editing DNS records during the event. Which solution is most appropriate?
A. Change the record values in the Route 53 console during the event. B. Use Amazon Application Recovery Controller routing controls linked to Route 53 health checks that are attached to the failover records. C. Increase the TTL to 24 hours. D. Use weighted records and update the weights through the API.
Answer: B. Routing controls are data-plane switches on a highly available cluster endpoint set; their linked health checks change Route 53's answers without control-plane record edits. A and D use the control plane. C worsens failover time.
Completion checklist
- I can explain the 18 percent rule, the intervals and the three health check types
- I can calculate worst-case failover time from interval, threshold and TTL
- I can explain control plane versus data plane for Route 53 and what a routing control adds
- I ran
Route53HealthModelTestsandEpochFenceTests - I wrote a failover runbook with fencing and no automatic failback
- I deleted any records, health checks or ARC resources I created
Key takeaways
- DNS failover time is detection plus TTL, plus whatever clients add. Measure it with real clients.
- Prepare with the control plane, execute with the data plane: health-check-driven answers or ARC routing controls, not console edits during the incident.
- Prefer human-approved Regional failover with fencing; never fail back automatically.
- Split brain is prevented by single decision, promote-then-switch ordering, and fencing the old primary.
- Health checks must be reachable, cheap and honest about what they check.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Active-active vs active-passive: consistency, conflicts and single-writer ledgers
Understand what changes when two Regions both accept writes: replication lag, conflict resolution, last-writer-wins, strong consistency, and why most banking ledgers stay single-writer with a hot or warm second Region.
Lab: Resilient Account API (resilience pipeline, multi-AZ scaling and a DR plan)
Prove a .NET balance API survives a flaky dependency with a fault-injecting test backend, optionally deploy a three-AZ ECS service with target-tracking scaling and drill an AZ failure, then size a warm-standby DR plan for an RTO of 30 minutes and an RPO of 5 minutes.