Skip to Content
Cloud Engineering Academy
12. High Availability & Disaster Recovery

Availability vocabulary: HA, fault tolerance, DR, RTO and RPO

Speak precisely about failure: separate high availability from fault tolerance and disaster recovery, turn a target such as 99.95% into minutes of allowed downtime, and set RTO, RPO and impact-tolerance numbers a bank can test.

beginnerโฑ 1 h 10 min๐ŸŸข FreeSAA-C03SOA-C03CLF-C02

Assumes: .NET 10 LTS ยท AWS SDK for .NET v4 ยท AWS CLI v2 ยท Region us-east-1 (DR Region us-west-2 for design exercises). Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain the difference between high availability, fault tolerance, resilience and disaster recovery in one sentence each, and say which one a requirement is really asking for.
  • Convert an availability target ("three nines", 99.95%) into minutes of downtime per month and per year, and explain why a chain of services is less available than its weakest link.
  • Define RTO, RPO, MTTR and impact tolerance, and show how they relate.
  • Classify a failure by its blast radius (instance, Availability Zone, Region, account, data) and name the control that addresses each.
  • Explain control plane versus data plane, and why recovery plans should lean on the data plane.
  • Run the lab's availability and DR calculators and read their output critically.

Prerequisites

Why this matters in banking

Harbourline Bank's card-authorisation service fails for 47 minutes on a Saturday evening. The post-incident review produces four different sentences from four different people:

  • The CTO: "We are highly available, we run in three Availability Zones."
  • The head of operations: "We need to be able to fail over within an hour."
  • The risk officer: "Our impact tolerance for authorisations is 30 minutes, so we breached it."
  • The database lead: "We lost nothing, the data was replicated."

All four are talking about different things, and several are quietly wrong. Running in three AZs is high availability, not disaster recovery. "Within an hour" is a recovery time with no recovery point attached. A breached impact tolerance is a regulatory event, not just an engineering metric. "Replicated" says nothing about what happens when the bad data replicates too.

This lesson gives you a shared, precise vocabulary. Everything else in this module (scaling, Multi-AZ, resilience patterns, DR strategies, failover) is a decision that depends on these numbers. If the numbers are vague, the architecture will be either dangerously weak or absurdly expensive.

The idea in plain language

Start with five words that people use as if they were synonyms.

Availability is the share of time a service does its job correctly when asked. It is a ratio, usually written as a percentage ("99.95%") and measured over a window (a month, a year). It says nothing about why the remaining time was lost.

High availability (HA) means designing so that the failure of a component (a server, a disk, a whole building) does not take the service down, or takes it down only briefly. HA usually works by redundancy plus automatic detection and replacement. A brief blip while traffic moves is acceptable.

Fault tolerance is the stricter version: the service continues with no visible interruption when a component fails. It is more expensive because redundancy must be active all the time (for example, synchronous copies of every write). Most banking systems are highly available, and only a few critical paths are truly fault tolerant.

Resilience is the broader word: the ability to withstand, adapt to and recover from disruption, including slow dependencies, bad deployments, overload and human error. A resilient system degrades gracefully instead of falling over.

Disaster recovery (DR) is what you do when the primary location is lost or unusable for a long time: a Region outage, a destroyed data centre, ransomware, a corrupted database. DR assumes HA was not enough. It is measured by RTO and RPO.

Analogy: a bank branch has two tills (HA: if one jams, the other serves you), a generator that starts by itself (fault tolerance for power, because you never notice the cut), and a second branch across town with yesterday's keys and records (DR, used only if the building is closed for days). Resilience is the staff training that makes the branch stay open, with a queue and a smile, when the card machines are slow.

The two numbers that define DR

  • RTO (recovery time objective): the longest you are willing to be down. "RTO 30 minutes" means service must be restored within 30 minutes of the disaster starting.
  • RPO (recovery point objective): the longest period of data you are willing to lose, measured backwards from the disaster. "RPO 5 minutes" means you accept losing at most the last 5 minutes of writes. RPO is set by how often data is copied, not by how fast you restore.

Two related words appear constantly in operations:

  • MTTR (mean time to recover/repair): how long recovery actually takes on average. RTO is the goal, MTTR is what you measure in drills.
  • MTBF (mean time between failures): how often failure happens. Availability is roughly MTBF divided by MTBF plus MTTR, which is why fast recovery improves availability as much as fewer failures does.

RTO and RPO are business decisions, not engineering ones

Engineering can say what each RTO and RPO costs. Only the business can say how much downtime and data loss it can tolerate. In regulated banking the starting point is the impact tolerance of an important business service: the maximum tolerable disruption, usually a duration. Engineering then sets an RTO inside that tolerance, with margin, because detection, decision and communication all consume time before recovery even starts. If the tolerance for card authorisation is 4 hours, an RTO of 4 hours leaves no room for anything going wrong. An RTO of 1 hour does.

Blast radius: where did it fail?

Different failures need different defences. Think of concentric circles, smallest to largest:

Blast radiusExamplePrimary defenceCovered in
A process or containerMemory leak, crashHealth checks, restart, auto scalingAuto Scaling everywhere
An instance or hostHardware failureReplace automatically, stateless designSame lesson
An Availability ZonePower, cooling or network failure in one data centre groupSpread across at least 2 (better 3) AZsMulti-AZ by default
A dependencyCore Banking slow or erroringTimeouts, retries, breakers, bulkheadsResilience patterns in .NET
A RegionLarge-scale event affecting many services in the RegionDR in another RegionThe four DR strategies
A bad changeFaulty deployment or configurationProgressive delivery, rollback, feature flagsModule 9
DataCorruption, accidental delete, ransomwarePoint-in-time recovery, immutable backups, isolated copiesAWS Backup lesson in this module
An account or identityCompromised credentialsAccount separation, SCPs, break-glassModule 5

Notice what replication does not defend against: a bad UPDATE replicates to every copy within a second. Replication protects against loss of a place; backups and point-in-time recovery protect against loss of correctness.

What problem the AWS service solves

This lesson is about vocabulary, so there is no single service. But three AWS ideas anchor the vocabulary, and each has sharp edges.

1. Regions and Availability Zones are the unit of failure design. An AZ is one or more discrete data centres with independent power, cooling and networking, in a Region. AZs in a Region are connected by low-latency links, so you can replicate synchronously between them. Regions are separate: independent infrastructure, separate service endpoints, and (importantly) data never leaves a Region unless you copy it. That makes the AZ your HA boundary and the Region your DR boundary.

2. Static stability. A statically stable design keeps working when a dependency is impaired without needing to do anything: no scaling out, no failover API call, no configuration change. Example: run enough tasks in each of three AZs that losing one AZ still leaves capacity for peak traffic, rather than relying on Auto Scaling to launch replacements during the incident (launching needs the control plane, which may be unavailable or slow exactly when you need it).

3. Control plane versus data plane. The data plane does the real-time work (serving a request, answering a DNS query, a health check flipping a DNS answer). The control plane configures things (creating instances, changing a record, promoting a database). Data planes are designed for higher availability than control planes, so the rule for recovery is: prepare with the control plane, execute with the data plane. For example, a Route 53 health check automatically flipping a DNS answer is a data-plane action; manually editing a record during an outage is a control-plane action and may fail.

Limitations to be honest about:

  • Availability numbers in SLAs are not your availability. An AWS service SLA is a financial credit for that service; your system's availability is the product of everything in the request path, plus your own bugs and deployments.
  • Independence is an assumption. The parallel-availability formula below only holds when failures are independent. A shared deployment pipeline, a shared library bug or a shared certificate expiry fails every copy at once.
  • Nines are not RTO. A system can have excellent annual availability and still breach an impact tolerance with one long outage.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Availability Zones, zone-redundant servicesAvailability Zones, Multi-AZ optionsSame idea. On AWS, AZ names are mapped per account (us-east-1a in your account may not be us-east-1a in another); use AZ IDs such as use1-az1 to compare
Paired RegionsNo fixed pairs; you choose the DR RegionChoose for distance, regulation (UK/EU data), service availability and latency
Azure SLA tables and composite SLAsPer-service SLAs; you compute composite availability yourselfAWS publishes per-service SLAs; the composite arithmetic is the same
Recovery Time/Point Objective in Azure Site RecoveryRTO/RPO in AWS Elastic Disaster Recovery, AWS Backup, Resilience Hub policiesResilience Hub lets you express targets as a policy and assess a workload against them
Business Continuity Plan documentsImpact tolerances and runbooks, tested by game daysUK/EU regulators expect tested plans, not documents
.NET IHealthCheck and Application Insights availability testsELB/Route 53 health checks, CloudWatch syntheticsHealth checks are what trigger failover, so what they check is a design decision

Architecture

The same service looks different at each level of protection. The diagram shows which failures each layer absorbs.

Time also has a shape. A disaster produces a timeline, and RTO and RPO are two different measurements on it:

Step-by-step implementation

You will turn the vocabulary into numbers using the lab library Harbourline.Resilience (see Lab: Resilient Account API). No AWS account is needed.

Convert a target into minutes

Take the availability target for the Account API: 99.95% a month. A 30-day month has 43,200 minutes. Allowed downtime is 0.05% of that, which is 21.6 minutes. Write that number on the whiteboard; it is more useful than "three and a half nines".

Multiply down the request path

List what a balance request touches: ALB (99.99%), the API tasks (your own design, say 99.95%), the database (99.95%), Core Banking (99.9%). In series, all must work:

0.9999 x 0.9995 x 0.9995 x 0.999 = 0.9979, or about 99.79%, or roughly 90 minutes of downtime a month. The chain is worse than every link. To reach 99.95% you must remove the weakest link from the hard path, for example by caching the last known balance so Core Banking is no longer in series (lesson on caching, later in this module).

Check the independence assumption

Two identical deployments, each 99.9%, in two Regions give 1 - (0.001 x 0.001) = 99.9999% if failures are independent. Ask: do both share a deployment pipeline, a DNS provider, an identity provider, a certificate, a code base? Each shared thing is a place where the failures are correlated, so the real number is lower than the formula.

Set RTO and RPO with the business

Run a short workshop for one service. Ask three questions: how long can customers be unable to do this before real harm occurs (the impact tolerance)? How many minutes of transactions can we lose or reconstruct? What would each extra nine cost? Record the answers in the DR plan template at labs/m12-resilient-account-api/docs/dr-plan-template.md.

Classify the failures you will rehearse

For each row in the blast-radius table above, write which of your services is protected, by what control, and how you would prove it. Rows with no proof are your gaps.

AWS Console instructions

There is nothing to configure here, but you can look at your own account's failure boundaries. Console โ†’ EC2 โ†’ Dashboard โ†’ Service health and Console โ†’ EC2 โ†’ Settings โ†’ Zones show the Availability Zones for your Region and their zone IDs. Notice the zone ID (for example use1-az1); it is stable across accounts, the letter is not.

AWS CLI

# List this Region's Availability Zones with their stable zone IDs.
aws ec2 describe-availability-zones --region us-east-1 --profile learn-sandbox \
  --query "AvailabilityZones[].{Name:ZoneName,Id:ZoneId,State:State}" --output table

# List every Region available to the account (some are opt-in), to think about DR Region candidates.
aws ec2 describe-regions --all-regions --profile learn-sandbox \
  --query "Regions[].{Region:RegionName,OptIn:OptInStatus}" --output table

Infrastructure as Code

Not applicable: this lesson defines vocabulary and produces numbers. The infrastructure that turns those numbers into a design starts in Auto Scaling everywhere and the lab stack.

Complete code

The lab library holds the arithmetic. This is the console snippet used in the guided exercise; AvailabilityMath and FailoverBudget live in labs/m12-resilient-account-api/src/Harbourline.Resilience.

Program.cs (scratch console)
using Harbourline.Resilience;

// 1. What does a target mean in minutes?
var month = TimeSpan.FromDays(30);
foreach (var target in new[] { 0.99, 0.999, 0.9995, 0.9999 })
{
    var allowed = AvailabilityMath.AllowedDowntime(target, month);
    Console.WriteLine($"{target:P3} -> {allowed.TotalMinutes,6:F1} minutes a month");
}

// 2. A serial request path: ALB, API, database, Core Banking.
var path = AvailabilityMath.Serial(0.9999, 0.9995, 0.9995, 0.999);
Console.WriteLine($"Request path: {path:P3} -> {AvailabilityMath.AllowedDowntime(path, month).TotalMinutes:F0} minutes a month");

// 3. Two independent Regions (an upper bound: real failures are correlated).
var twoRegions = AvailabilityMath.Parallel(0.999, 0.999);
Console.WriteLine($"Two independent 99.9% Regions: {twoRegions:P4}");

// 4. Will a runbook fit inside the RTO?
RunbookStep[] runbook =
[
    new("Detect and declare", TimeSpan.FromMinutes(10)),
    new("Promote database",   TimeSpan.FromMinutes(5)),
    new("Scale compute",      TimeSpan.FromMinutes(12)),
    new("Switch traffic",     TimeSpan.FromMinutes(6)),
];
var slack = FailoverBudget.Slack(runbook, rto: TimeSpan.FromMinutes(30));
Console.WriteLine($"Runbook total {FailoverBudget.Total(runbook).TotalMinutes} min; slack against RTO 30 min: {slack.TotalMinutes} min");

Expected output (the percentage formatting rounds):

99.000% ->  432.0 minutes a month
99.900% ->   43.2 minutes a month
99.950% ->   21.6 minutes a month
99.990% ->    4.3 minutes a month
Request path: 99.790% -> 91 minutes a month
Two independent 99.9% Regions: 99.9999%
Runbook total 33 min; slack against RTO 30 min: -3 min

Code explained line by line

  1. AvailabilityMath.AllowedDowntime(target, month) multiplies the period by 1 - availability. It rejects values outside 0 to 1, because passing 99.9 instead of 0.999 is the classic bug.
  2. AvailabilityMath.Serial(...) multiplies the fractions. The result is always at most the smallest input, which is the point of the exercise.
  3. AvailabilityMath.Parallel(...) computes 1 - product(1 - a): the chance that all copies fail at once is the product of their failure probabilities. The doc comment on the method states the independence assumption, and the tests lock in the numbers.
  4. RunbookStep is a name plus a duration. FailoverBudget.Total adds them, and Slack subtracts from the RTO. A negative result tells you on paper, before a disaster, that the plan cannot meet its target.
  5. The runbook sums to 33 minutes against an RTO of 30: the tool shows the overrun before anyone writes it into a regulator-facing document.

Security considerations

Resilience is a security property. Under the CIA triad, availability is the "A", and attackers target it deliberately (DDoS, ransomware, destructive access).

  • Availability targets are a control objective. Record them with an owner, and review them when the service's importance changes.
  • DR copies are attack surface. A second Region holds a copy of customer data, so it needs the same encryption (customer-managed KMS keys per Region), access control and logging as the primary.
  • Ransomware and malicious deletion defeat replication. Plan immutable, isolated backups (a later lesson covers Vault Lock and cross-account copies).
  • Break-glass access to run a failover must be tested, audited and held by named roles, not by a shared admin user.

โš  Learning shortcut โ€” not for production

The lab calculators use illustrative planning numbers (for example "pilot light: RTO 1 hour"). They help you reason, but they are not AWS guarantees and not Harbourline's approved values. In a real bank, RTO and RPO come from the business impact analysis and are signed off by the service owner and risk function, and the achieved values come from measured drills. Nothing in this module makes a system "compliant"; legal, compliance and operational-resilience teams decide that.

Banking use case

Harbourline classifies its services into tiers so that it does not pay active-active prices for everything:

TierExample serviceAvailability targetRTORPO
0Card authorisation, Faster Payments gateway99.99%minutesseconds to near zero
1Online and mobile banking, Accounts API99.95%30 to 60 minutesabout 5 minutes
2Notifications, statements99.9%4 hours1 hour
3Internal reporting, data platform99.5%24 hours24 hours

Every tier maps to a DR strategy in the four DR strategies. The Accounts API sits in tier 1, which is why the module project asks for an RTO of 30 minutes and an RPO of 5 minutes. All figures here are fictional examples, not recommendations.

Common mistakes

  • Saying "highly available" when the requirement is DR. Three AZs do nothing for a Region event or for corrupted data. Ask "what is the worst thing we must survive?" first.
  • Quoting nines without a window. 99.9% a year allows about 8.8 hours in one go; 99.9% a month allows about 43 minutes. Always state the period.
  • Setting RTO and RPO from engineering comfort. They are business decisions with engineering price tags.
  • Forgetting detection and decision time. The clock starts at the disaster, not at the moment someone decides to fail over.
  • Equating replication with backup. Replication copies mistakes instantly.
  • Counting on the control plane during an incident. If your runbook says "create the load balancer", it depends on the very APIs that may be degraded.
  • Treating the SLA as the design. The SLA is a refund policy.

Troubleshooting

SymptomLikely causeFix
Your availability calculation exceeds the best single componentYou used the parallel formula for a serial chainMultiply in series when every part is required
Two "independent" Regions fail together in drillsShared dependency (pipeline, DNS, identity, one shared bug)List shared components and treat each as a correlated failure
RTO target "met" in the plan but missed in the drillDetection, decision and DNS/cache time were left outTime every step in a drill and use the measured numbers
describe-availability-zones returns different letters than a colleague's accountAZ names are shuffled per accountCompare zone IDs (use1-az1)
AllowedDowntime throws ArgumentOutOfRangeExceptionA percentage (99.9) was passed instead of a fraction (0.999)Pass fractions

Cost and cleanup

This lesson costs nothing: the calculators run locally and the CLI commands are read-only and free. There is nothing to tear down. If you ran the optional CLI commands, no resources were created.

# Nothing to delete. Confirm no lab resources exist from other lessons:
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --profile learn-sandbox --region us-east-1 --query "ResourceTagMappingList[].ResourceARN"

Exercises

  • ๐ŸŸข Guided: Run the scratch console above (or the lab tests in PlanningTests). Fill in a table of 99%, 99.5%, 99.9%, 99.95%, 99.99% and 99.999% against minutes per month and per year.
  • ๐ŸŸก Challenge: Harbourline's "view balance" path is ALB 99.99%, API 99.95%, Aurora 99.99%, Core Banking 99.9%, and an identity provider at 99.99%. Compute the serial availability, then show the effect of caching the last known balance so that Core Banking is only needed 5% of the time. State your assumption clearly.
  • ๐Ÿ”ด Live problem: The head of payments says: "We need 99.999% and RTO zero for Faster Payments submission." Prepare the three questions you will ask before agreeing to anything, and the two things you will say the number cannot cover (hint: correlated failures, data corruption, and scheme or third-party outages that are outside your control).

Assignment

Create docs/service-tiers.md in your portfolio repository: four tiers for a bank of your choosing (use Harbourline), with availability, RTO, RPO, impact-tolerance notes and one sentence on what each tier will not survive. Add a one-page "vocabulary card" that defines HA, fault tolerance, resilience, DR, RTO, RPO, MTTR and impact tolerance in your own words.

Solutions

Knowledge check

Interview questions

  • "What is the difference between HA and DR?" HA keeps the service running through component failures inside a Region, usually automatically and with seconds of impact. DR restores service after the loss of a whole location or after a data disaster, measured by RTO and RPO, and often involves a deliberate decision and a runbook.
  • "How would you decide an RTO for a payments service?" Start from the business impact tolerance and regulatory expectations, subtract detection and decision time and a safety margin, price the architectures that meet the result, and agree the number with the service owner. Then prove it in a drill.
  • "Why is 99.99% not simply 'add another server'?" About 52 minutes a year includes every deployment, certificate expiry, dependency outage and human mistake. The weakest serial link and correlated failures dominate, so the work is in removing hard dependencies and making recovery fast.
  • "What is static stability?" Staying healthy when a dependency fails without taking an action: pre-provisioned capacity in remaining AZs rather than relying on launching more during the incident.

Certification practice

Completion checklist

  • I can explain HA, fault tolerance, resilience and DR to a colleague in two sentences
  • I can convert a percentage target to minutes without a calculator for 99.9% and 99.99%
  • I ran the lab's planning tests or the scratch console and understood every number
  • I wrote tiers with RTO and RPO for at least four Harbourline services
  • I confirmed nothing tagged Project=cea-lab is left in the sandbox account

Key takeaways

  • HA, fault tolerance, resilience and DR answer different questions. Name the failure you must survive before choosing a design.
  • RTO is about time down; RPO is about data lost. Both are business decisions, set inside an impact tolerance with margin.
  • Chains are worse than their weakest link, copies help only when failures are independent, and replication does not protect against bad data.
  • Prepare with the control plane, recover with the data plane, and prove it in a drill.

Further reading

Independent learning resource โ€” not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page