Auto Scaling everywhere: EC2, ECS, Lambda, DynamoDB and Aurora
Configure scaling for compute and data layers, understand what each one scales on, and size minimum capacity so that losing an Availability Zone does not need a scale-out to survive.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.ApplicationAutoScaling 4.1.0.2) · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Name the scaling mechanism for EC2, ECS, Lambda, DynamoDB and Aurora, and say what signal each one reacts to.
- Choose between target tracking, step, scheduled and predictive scaling for a banking workload.
- Calculate the minimum capacity that survives the loss of one Availability Zone without scaling out first (static stability).
- Explain how scaling itself fails: quotas, cooldowns, cold starts, database connection limits and retry storms.
- Review the scaling settings of every ECS service in an account with a small C# tool.
Prerequisites
- Availability vocabulary — static stability and blast radius
- Scaling .NET apps — the basics of target tracking on ECS, statelessness and Lambda concurrency. This lesson does not repeat them; it adds the resilience view and the EC2, DynamoDB and Aurora layers.
- Tools: Docker for the lab tests; optional AWS CLI v2 with
learn-sandbox
Why this matters in banking
It is salary day, 09:00. Harbourline's Accounts API normally runs 6 tasks. Traffic triples in ten minutes. Two things happen at once: Auto Scaling starts adding tasks, and an Availability Zone has a network problem and its 2 tasks vanish from the load balancer. The service now has 4 tasks serving 3 times the normal load, while replacement tasks take 90 seconds to start, and each new task opens database connections that the database cannot afford. Customers see errors for four minutes.
None of those failures is exotic. Together they show the three ideas of this lesson: scaling is a control-plane reaction (it takes time), minimum capacity is your real protection (it is already running), and every scaled layer has a downstream limit (the database). A bank that treats auto scaling as "the cloud will handle it" has misunderstood where the safety actually comes from.
The idea in plain language
Scaling means changing capacity to match demand. Horizontal scaling adds or removes copies (more tasks); vertical scaling changes the size of one copy (bigger instance). Cloud resilience relies on horizontal scaling because copies can be spread across Availability Zones.
Auto scaling is a control loop: measure something, compare it with a target, change capacity, wait, repeat. You choose the measurement and the limits; AWS runs the loop.
Two numbers matter more than any policy:
- Minimum capacity is what runs all the time. It is your protection against a failure that happens faster than scaling can react.
- Maximum capacity is the ceiling that protects your dependencies (database, downstream APIs, bill).
Analogy: a bank branch rota. The permanent staff (minimum) are always on the floor and can cope with a lunchtime rush for a few minutes. The agency staff (scaled capacity) take 30 minutes to arrive. The branch manager must decide how many permanent staff to keep so that when one counter closes unexpectedly, the queue stays acceptable until the agency staff arrive.
Four ways to decide when to scale
| Policy | How it works | Best for | Watch out for |
|---|---|---|---|
| Target tracking | You pick a metric and a target ("keep CPU at 50%"); the service creates alarms and does the arithmetic | The default for stateless APIs | The metric must actually move with load; too-aggressive scale-in |
| Step scaling | Alarm thresholds map to specific capacity changes ("above 80% add 3, above 90% add 6") | Workloads that need big jumps or asymmetric reactions | You own the thresholds and the warm-up behaviour |
| Scheduled | Change min/max at known times | Predictable peaks: salary day, month-end batch, market open | Forgetting the schedule when the pattern changes |
| Predictive (EC2 Auto Scaling) | Forecasts load from history and scales ahead | Recurring daily or weekly cycles with slow start-up | Needs history; not for sudden unique events |
Use target tracking as the base, add scheduled actions for events you can predict, and keep generous minimums.
What problem the AWS service solves
The service family is AWS Auto Scaling-style capabilities spread across several products. They fall in two groups.
Group 1: EC2 Auto Scaling groups (ASGs). An ASG launches instances from a launch template, keeps the count between a minimum and maximum, replaces unhealthy instances, and spreads instances across the subnets (AZs) you give it. Notable resilience features:
- Health checks. By default the group uses EC2 status checks only. Turn on ELB health checks so an instance that is running but not serving is replaced. A health check grace period stops the group from killing a new instance while it is still booting.
- AZ rebalancing. The group tries to keep an equal number of instances per AZ. After an AZ recovers it launches in the emptier zones and terminates elsewhere, launching before terminating so capacity does not drop. Being at the maximum can stall this; AWS documents a temporary overshoot margin (by default 10 percent or one instance, whichever is greater).
- Instance refresh replaces instances gradually for AMI or template changes, with a minimum healthy percentage.
- Warm pools keep pre-initialised instances stopped or hibernated, to cut scale-out time for slow-booting .NET Framework images. ⚠ VERIFY current pricing and supported options before relying on them.
- Lifecycle hooks pause launch or termination so you can run steps (register with a service, drain connections).
Group 2: Application Auto Scaling is one control loop for many services: ECS service desired count, Lambda provisioned concurrency, DynamoDB table and index capacity, Aurora replica count, and more. You register a scalable target (resource, dimension, min, max) and attach scaling policies.
How each layer scales:
| Layer | What scales | Signal you usually use | Resilience notes |
|---|---|---|---|
| EC2 ASG | Instance count | Average CPU, ALB RequestCountPerTarget, a custom metric such as queue depth | Spread across at least 2, preferably 3 AZs; enable ELB health checks |
| ECS service | Task count (Fargate or EC2 capacity) | ECSServiceAverageCPUUtilization, ECSServiceAverageMemoryUtilization, ALBRequestCountPerTarget | Set the minimum to a multiple of the AZ count; enable service AZ rebalancing; use the deployment circuit breaker |
| Lambda | Concurrent executions (automatic) plus provisioned concurrency (optional scaling) | Utilisation of provisioned concurrency | Reserved concurrency is a cap and a guarantee for one function; the account has a shared default quota ⚠ VERIFY the current default and per-function scaling rate |
| DynamoDB | Read and write capacity | On-demand mode scales for you; in provisioned mode, target tracking on consumed/provisioned ratio | Replication (global tables) also consumes write capacity; throttling surfaces as ProvisionedThroughputExceededException |
| Aurora | Number of Aurora Replicas (Application Auto Scaling) and, for Serverless v2, ACUs inside a min-max range | Replica CPU or connections; Serverless v2 follows load automatically | Replicas help reads, not writes; the writer is still one instance; failover promotes a replica |
How scaling itself fails
Scaling is a reaction, so it fails in predictable ways:
- Too slow. Metrics take a minute or two to arrive, a new task needs time to pull an image and pass health checks, and a cooldown can delay the next step. A spike that doubles traffic in 30 seconds beats all of that. Defence: minimum capacity and load shedding.
- Too weak. The maximum is too low, the account quota is reached (vCPU quota, Fargate task quota, Lambda concurrency), or the AZ has no capacity of the instance type you asked for. Defence: quota headroom, flexible instance types, more AZs.
- Too much. Ten more tasks each open 20 database connections and the database falls over. Defence: cap maximum from the database budget, use a connection proxy, cap Lambda reserved concurrency. The
scaling-dotnet-appslesson covers the connection arithmetic. - Wrong signal. CPU is flat because the service is blocked waiting on a slow dependency; scaling out adds more waiting threads. Defence: scale on requests or queue depth, and fix the dependency with timeouts and breakers (see Resilience patterns in .NET).
- Flapping. Scale-in removes capacity right before the next wave. Defence: slow scale-in, longer cooldowns, scheduled minimums.
- It is the control plane. During a Regional event, launching new capacity may be slow or fail. Defence: static stability, do not rely on scale-out to survive.
Static stability: the minimum-capacity formula
If you run in N AZs and want to keep serving peak load after one AZ disappears, the remaining N - 1 AZs together must already hold peak capacity. If peak needs P tasks, run P / (N - 1) per surviving AZ, so in total N x P / (N - 1):
| AZs | Total tasks for peak 12 | Headroom over peak | Notes |
|---|---|---|---|
| 2 | 24 | 100% | Expensive: you pay for double |
| 3 | 18 | 50% | The usual sweet spot |
| 4 | 16 | 33% | Cheaper per unit of safety, more operational surface |
That is why banks tend to run three AZs: the cost of being able to lose one drops from double to one and a half. You can accept less headroom if the business accepts degraded mode (for example, best-effort traffic is shed while a replacement AZ's tasks start), but make that an explicit decision.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| VM Scale Sets with autoscale rules | EC2 Auto Scaling groups with scaling policies | Target tracking is the main style; launch templates replace scale set models |
| Instance protection / upgrade policy | Instance refresh, scale-in protection, lifecycle hooks | Rolling replacement is a first-class operation |
| App Service autoscale (Plan scale-out) | ECS service auto scaling through Application Auto Scaling | You scale tasks; the capacity underneath is Fargate or an ASG |
| Functions Premium pre-warmed instances | Lambda provisioned concurrency | Billed while configured; can itself be scaled |
| Azure SQL Hyperscale replicas / serverless | Aurora Replicas and Aurora Serverless v2 | Replica count and ACU range are separate settings |
| Cosmos DB autoscale RU/s | DynamoDB on-demand or provisioned with auto scaling | On-demand needs no capacity planning; provisioned is cheaper at steady load |
| Autoscale settings with schedules and profiles | Scheduled actions and predictive scaling | Scheduled actions change min and max, not just the count |
| Azure Monitor autoscale cooldown | Cooldowns and instance warm-up | Cooldown and warm-up are separate ideas on EC2 |
Architecture
The security and reliability boundary view: scaling policies are IAM-governed configuration. Only the deployment role should be able to change minimum and maximum capacity, because min = 0 or max = min silently removes your resilience.
Step-by-step implementation
You will review scaling settings, plan capacity, and (optionally) deploy the lab ECS service.
Work out minimum capacity from peak
Pick the service's measured peak (tasks at the 99th percentile of normal load, not a guess). Apply the formula above, or run CapacityPlanner.TotalTasksToSurviveAzLoss(peak, azCount) from the lab library. For the Accounts API: peak 12 tasks in 3 AZs gives 18.
Decide where scaling sits between minimum and peak
If you cannot afford running 18 all day, run a lower minimum (for example 9) plus a scheduled action that raises the minimum to 18 before salary-day mornings, and accept degraded mode otherwise. Write the decision down.
Choose the policies
Use target tracking on requests per target (it follows demand directly) and CPU (it catches work-heavy requests). Make scale-out fast (short cooldown) and scale-in slow (5 minutes or more). Keep the maximum below what the database can serve.
Align health checks
For EC2: enable ELB health checks and set the grace period to the real start-up time. For ECS: set the service health-check grace period, a sensible target-group health check, and a stop timeout longer than the app's shutdown time (ELB for resilience).
Verify with a report, not a feeling
Run the C# scaling report (below) against your account to find services whose minimum is below the AZ count or whose maximum equals their minimum.
Rehearse
Load test, then (in the lab) stop tasks in one AZ and measure errors and recovery time. A scaling design you have not stressed is untested.
AWS Console instructions
- ECS service scaling: Console → ECS → Clusters → your cluster → Services → your service → Service auto scaling (a tab or section in the service view, depending on the console version). Set minimum and maximum tasks, then add a policy with a metric and target.
- EC2 Auto Scaling group: Console → EC2 → Auto Scaling Groups → your group → Automatic scaling for policies, Details → Health checks for the ELB setting and grace period, Instance management for activity and Activity tab for scaling history.
- DynamoDB: Console → DynamoDB → Tables → your table → Additional settings → Read/write capacity to switch modes and configure auto scaling.
- Lambda: Console → Lambda → your function → Configuration → Concurrency and recursion detection for reserved and provisioned concurrency.
Console labels move; if a label differs, search the page for "scaling" or "concurrency" ⚠ VERIFY against the current console.
AWS CLI
PROFILE=learn-sandbox
REGION=us-east-1
# Register an ECS service as a scalable target: never fewer than 3 tasks, never more than 9.
aws application-autoscaling register-scalable-target \
--service-namespace ecs \
--scalable-dimension ecs:service:DesiredCount \
--resource-id service/hl-m12-account-api/api \
--min-capacity 3 --max-capacity 9 \
--profile $PROFILE --region $REGION
# Target-tracking policy: keep average CPU near 50 percent. Scale in slowly (300 s), out quickly (60 s).
aws application-autoscaling put-scaling-policy \
--service-namespace ecs \
--scalable-dimension ecs:service:DesiredCount \
--resource-id service/hl-m12-account-api/api \
--policy-name hl-m12-cpu \
--policy-type TargetTrackingScaling \
--target-tracking-scaling-policy-configuration '{
"TargetValue": 50.0,
"PredefinedMetricSpecification": {"PredefinedMetricType": "ECSServiceAverageCPUUtilization"},
"ScaleOutCooldown": 60,
"ScaleInCooldown": 300
}' \
--profile $PROFILE --region $REGION
# Scheduled action: raise the minimum before salary-day mornings (cron in UTC; example only).
aws application-autoscaling put-scheduled-action \
--service-namespace ecs \
--scalable-dimension ecs:service:DesiredCount \
--resource-id service/hl-m12-account-api/api \
--scheduled-action-name hl-m12-salary-day-up \
--schedule "cron(30 7 25 * ? *)" \
--scalable-target-action MinCapacity=18,MaxCapacity=24 \
--profile $PROFILE --region $REGION
# See what is registered, and what scaling did recently.
aws application-autoscaling describe-scalable-targets --service-namespace ecs --profile $PROFILE --region $REGION
aws application-autoscaling describe-scaling-activities --service-namespace ecs --max-results 10 --profile $PROFILE --region $REGION
# Lambda: cap one function so it cannot exhaust the database, and keep 5 environments warm.
aws lambda put-function-concurrency --function-name hl-m12-statement-export --reserved-concurrent-executions 20 \
--profile $PROFILE --region $REGIONBecause the cron expression above is an example, check the exact cron() format and time zone behaviour in the Application Auto Scaling scheduled-scaling docs ⚠ VERIFY before using it.
Infrastructure as Code
The lab stack labs/m12-resilient-account-api/infra implements this lesson's ECS example in both tools.
// Excerpt from labs/m12-resilient-account-api/infra/cdk/Harbourline.Resilience.Cdk/AccountApiHaStack.cs
var service = new FargateService(this, "Service", new FargateServiceProps
{
Cluster = cluster,
TaskDefinition = taskDefinition,
DesiredCount = props.MinTasks, // 3 = one per AZ
MinHealthyPercent = 100,
MaxHealthyPercent = 200,
CircuitBreaker = new DeploymentCircuitBreaker { Rollback = true }, // bad deployments roll back
AvailabilityZoneRebalancing = AvailabilityZoneRebalancing.ENABLED, // re-spread tasks after an AZ recovers
// ... network settings omitted
});
var scaling = service.AutoScaleTaskCount(new EnableScalingProps { MinCapacity = 3, MaxCapacity = 9 });
scaling.ScaleOnCpuUtilization("Cpu", new CpuUtilizationScalingProps
{
TargetUtilizationPercent = 50,
ScaleOutCooldown = Duration.Seconds(60),
ScaleInCooldown = Duration.Minutes(5), // scale in slowly: removing capacity is the risky direction
});
scaling.ScaleOnRequestCount("Requests", new RequestCountScalingProps
{
RequestsPerTarget = 600, // per task, per minute
TargetGroup = targetGroup,
ScaleOutCooldown = Duration.Seconds(60),
ScaleInCooldown = Duration.Minutes(5),
});Complete code
Two pieces of code from the lab: the capacity planner, and a read-only scaling report that reviews every ECS service in an account.
namespace Harbourline.Resilience;
public static class CapacityPlanner
{
public static int TotalTasksToSurviveAzLoss(int peakTasks, int azCount)
{
if (peakTasks < 1) throw new ArgumentOutOfRangeException(nameof(peakTasks));
if (azCount < 2) throw new ArgumentOutOfRangeException(nameof(azCount), "Surviving an AZ loss needs at least 2 AZs.");
var perSurvivingAz = (int)Math.Ceiling((double)peakTasks / (azCount - 1));
return perSurvivingAz * azCount;
}
public static double HeadroomFraction(int peakTasks, int azCount) =>
(double)TotalTasksToSurviveAzLoss(peakTasks, azCount) / peakTasks - 1.0;
}using Amazon.ApplicationAutoScaling;
using Amazon.ApplicationAutoScaling.Model;
using Harbourline.Resilience.Tools;
var azCount = args.Length > 0 && int.TryParse(args[0], out var parsed) ? parsed : 3;
using var client = new AmazonApplicationAutoScalingClient();
var targets = new List<ScalingTarget>();
string? nextToken = null;
do
{
var page = await client.DescribeScalableTargetsAsync(new DescribeScalableTargetsRequest
{
ServiceNamespace = ServiceNamespace.Ecs,
ScalableDimension = ScalableDimension.EcsServiceDesiredCount,
NextToken = nextToken,
});
targets.AddRange(page.ScalableTargets.Select(t => new ScalingTarget(t.ResourceId, t.MinCapacity ?? 0, t.MaxCapacity ?? 0)));
nextToken = page.NextToken;
}
while (nextToken is not null);
Console.WriteLine($"{targets.Count} ECS scalable target(s) found; reviewing against {azCount} AZs.");
var findings = ScalingReport.Analyse(targets, azCount);
foreach (var f in findings.OrderByDescending(f => f.Severity))
{
Console.WriteLine($"[{f.Severity}] {f.ResourceId}: {f.Message}");
}
return findings.Any(f => f.Severity == Severity.Problem) ? 1 : 0;The rules live in ScalingReport.cs (pure functions, covered by ScalingTests). Only Program.cs needs AWS credentials.
Code explained line by line
Math.Ceiling((double)peakTasks / (azCount - 1))computes tasks per surviving AZ, rounding up because you cannot run a fraction of a task. Multiplying byazCountgives the total to deploy. Casting todoubleavoids integer division truncating the result.- The guard
azCount < 2encodes a truth: one AZ cannot survive its own loss. new AmazonApplicationAutoScalingClient()uses the default credential chain (the SSO profile throughAWS_PROFILE), so no keys appear in code.ServiceNamespace.EcsandScalableDimension.EcsServiceDesiredCountare SDK constants for the stringsecsandecs:service:DesiredCount.- The
do ... while (nextToken is not null)loop follows pagination, so accounts with many services are fully covered. t.MinCapacity ?? 0: the SDK models optional integers as nullable; a missing value is treated as zero so it is flagged.findings.OrderByDescending(f => f.Severity)prints problems first, and the process exits with code 1 when any problem exists, so you can run it as a pipeline gate.
Security considerations
- Scaling configuration is a resilience control. Restrict
application-autoscaling:RegisterScalableTarget,PutScalingPolicyand ASGUpdateAutoScalingGroupto deployment roles. A change from min 3 to min 0 is an outage waiting for a trigger. - Scaling is also a cost and denial-of-wallet control. An attacker or a retry storm that drives traffic can scale you to the maximum. Keep the maximum realistic, alarm on running tasks near maximum, and put a WAF rate limit in front (WAF and Shield).
- Least privilege for the scaling service. Application Auto Scaling uses a service-linked role; do not replace it with a broad custom role.
- Log scaling activity. Scaling events appear in CloudTrail and in the scaling activity history; include them in change-review evidence.
⚠ Learning shortcut — not for production
The lab uses a small minimum (3 tasks), a public placeholder container and tasks with public IPs so that a three-AZ service costs a few cents an hour. Production should use the capacity formula against measured peak, private subnets, and an alarm on "running tasks equals maximum". Choose the minimum with the service owner: it is a cost-versus-risk decision that the business must understand.
Banking use case
Harbourline's Accounts API has a monthly pattern: a 10 to 20 times peak on salary day mornings and a 3 times peak on Monday mornings. The design:
- Minimum 9 tasks all month (3 per AZ), scheduled up to 18 on the 25th to the 28th at 07:30 UTC, back down at 11:00.
- Target tracking on requests per target, CPU as a secondary signal.
- Maximum 30, calculated from the Aurora connection budget (30 tasks x 10 connections per task = 300 connections, below the database's limit with headroom for admin and the other services).
- Load shedding for report endpoints once in-flight requests pass a threshold (lab library
LoadShedder). - A weekly report from the C# tool lists any service whose minimum is below the AZ count. All figures are fictional teaching examples.
Common mistakes
- Setting minimum 1 or 2 "to save money". One AZ loss then takes a third or half of capacity, or all of it.
- Scaling on CPU when the service is I/O bound. CPU stays flat while latency climbs. Scale on requests per target or queue depth instead.
- No maximum, or a maximum above the database's limits. Scale-out then moves the outage to the database.
- Fast scale-in. Capacity vanishes between waves. Slow it down.
- Relying on scale-out to survive an AZ loss. It is too slow and uses the control plane at the worst time.
- Forgetting quotas. Raising Fargate, vCPU or Lambda quotas takes time. Request increases before the peak, and in the DR Region too.
- Provisioned DynamoDB without auto scaling for a spiky workload. Throttling during salary day. Use on-demand or auto scaling, and remember global-table replication consumes write capacity.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Service never scales out | Metric is flat (I/O bound) or the policy targets the wrong service or target group | Check the metric in CloudWatch; scale on requests per target or a custom metric |
| Scales out then immediately scales in | Cooldown too short or target too close to normal load | Lengthen scale-in cooldown; raise the target gap |
| Stuck at maximum with growing errors | Maximum too low or downstream saturated | Raise the maximum only after checking database connections; add load shedding |
ECS tasks stay in PENDING | Quota reached, no capacity in an AZ, or image pull failing | Check service events and quotas; use more subnets/AZs |
ProvisionedThroughputExceededException on DynamoDB | Provisioned mode without enough auto scaling; hot partition | Use on-demand or raise scaling limits; review key design |
| Tasks all land in one AZ after an incident | Rebalancing is off or capacity was short | Enable service AZ rebalancing; check deployment settings |
AccessDeniedException on register-scalable-target | Missing application-autoscaling:* or the ECS service-linked role does not exist | Add permissions; create the service-linked role if needed |
Cost and cleanup
Reading and planning cost nothing. Deploying the lab ECS service costs about USD 0.10 per hour for the Fargate tasks plus the ALB, so deploy, test and destroy within an hour (⚠ VERIFY current Fargate and ALB prices in the AWS Pricing Calculator). Scaling out to the maximum multiplies the task cost.
# Tear down the lab stack (use the tool you deployed with).
cd labs/m12-resilient-account-api/infra/cdk && cdk destroy -c owner=<your-name> --profile learn-sandbox
# or
cd labs/m12-resilient-account-api/infra/terraform && terraform destroy
# Remove any scheduled action you created by hand.
aws application-autoscaling delete-scheduled-action --service-namespace ecs \
--scalable-dimension ecs:service:DesiredCount --resource-id service/hl-m12-account-api/api \
--scheduled-action-name hl-m12-salary-day-up --profile learn-sandbox --region us-east-1
# Confirm nothing tagged for this lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m12-ha \
--profile learn-sandbox --region us-east-1 --query "ResourceTagMappingList[].ResourceARN"Exercises
- 🟢 Guided: Use
CapacityPlanner(or theScalingTestsdata) to fill in total tasks and headroom for a peak of 12 across 2, 3 and 4 AZs. Then run the lab tests. - 🟡 Challenge: Harbourline's payments service peaks at 20 tasks, each using 8 database connections, against a database limit of 400 connections with 80 reserved for administration and other services. Compute the largest safe maximum and the minimum needed to survive an AZ loss in 3 AZs. Explain the conflict, if any, and propose how to resolve it.
- 🔴 Live problem: "Last Friday the API returned errors for six minutes when traffic spiked. Auto scaling worked. What went wrong?" You have these facts: CPU averaged 35% throughout, tasks took 70 seconds to become healthy, and the database CPU hit 100%. Write your diagnosis and three changes.
Assignment
Add a docs/scaling-standard.md to your portfolio repo: a one-page Harbourline scaling standard covering minimum capacity rules (multiple of AZ count, headroom formula), required policies, maximum derived from dependency budgets, scale-in settings, and the review you run weekly (the C# report).
Solutions
2 AZs: 24 tasks, 100% headroom. 3 AZs: 18 tasks, 50% headroom. 4 AZs: ceil(12/3) = 4 per AZ so 16 tasks, 33% headroom. Each extra AZ lowers the price of surviving a loss, which is why three is common. The lab test Total_tasks_to_survive_one_az_loss encodes these values.
Connection budget: 400 - 80 = 320 available; at 8 connections per task the maximum is 320 / 8 = 40 tasks. Static stability: peak 20 in 3 AZs needs ceil(20/2) x 3 = 30 tasks running. 30 is within 40, so the minimum can be 30 without breaching the database, but a maximum of 40 leaves only 10 tasks of scale-out headroom beyond the AZ-loss configuration. If more headroom is needed, reduce connections per task (tune pool size), add a connection proxy, or scale the database. Alternatively accept degraded mode with a lower minimum and load shedding. The conflict is that safety capacity and burst capacity share one downstream budget.
Diagnosis: the bottleneck was the database, not compute. CPU stayed at 35% because tasks were waiting on slow queries, so the CPU-based policy never scaled enough; scaling out added tasks that opened more connections and made the database worse. The 70-second start-up meant reaction was too slow for the spike. Changes: (1) scale on requests per target and cap maximum from the database connection budget; (2) raise the minimum or use a scheduled action ahead of known peaks, and speed up start-up (smaller image, readiness check tuning); (3) add timeouts, a bulkhead and load shedding so the API returns fast 503s instead of queuing, and protect the database with a connection proxy or read replicas for read endpoints.
Knowledge check
Answer: Minimum capacity is already running. Scaling takes time (metrics, alarms, start-up) and uses the control plane, which may be degraded in an incident. A policy cannot help during the first minutes; capacity already in the remaining AZs can.
Answer: 18. Two surviving AZs must hold 12, so 6 each; with 3 AZs that is 6 x 3 = 18.
Answer: Turning on ELB health checks for the group (with a suitable grace period). EC2 status checks alone only detect infrastructure problems, not application failure.
Answer: It both guarantees that amount of concurrency for the function and caps it at that amount, so the function cannot consume the whole account pool and cannot exceed the cap (protecting downstream systems). It is different from provisioned concurrency, which keeps environments initialised and is billed while configured.
Answer: More tasks open more connections and send more requests to a saturated dependency such as a database. The fix is to cap the maximum from the dependency's budget, add timeouts and load shedding, and use a connection proxy.
Interview questions
- "How many instances do you run behind a load balancer and why?" At least two AZs, preferably three, with the minimum set so that losing one AZ still holds peak: total equals AZs times peak divided by AZs minus one. Policies handle the rest.
- "Target tracking or step scaling?" Target tracking for most APIs because AWS manages the alarms and the arithmetic; step scaling when you need asymmetric or very large reactions, and you accept owning thresholds.
- "What breaks when an autoscaled service gets a sudden 10x spike?" Time (start-up and metric delay), quotas, and downstream limits. Mitigate with minimum capacity, scheduled scaling, load shedding and dependency caps.
- "How do you scale DynamoDB safely?" On-demand for unpredictable load, or provisioned with auto scaling and a minimum; watch hot partitions and global-table replication writes.
Certification practice
A web tier on EC2 must keep serving full peak traffic after the loss of one of three Availability Zones, without waiting for new instances to launch. Peak requires 12 instances. What minimum number of instances should the Auto Scaling group run across the three zones?
A. 12 B. 15 C. 18 D. 36
Answer: C. Two zones must carry 12 instances (6 each), so each of the three zones runs 6, giving 18. A leaves no headroom; B (5 per zone) leaves 10 after a loss; D is far more than needed.
An Auto Scaling group behind an ALB keeps running an instance whose web server process has stopped responding. The instance passes EC2 status checks. What should the SysOps administrator change?
A. Increase the maximum size. B. Enable ELB health checks for the Auto Scaling group. C. Shorten the scale-in cooldown. D. Switch the group to scheduled scaling.
Answer: B. The group only uses EC2 status checks by default; ELB health checks make it replace instances that fail the load balancer's application-level check. The other options do not detect application failure.
A read-heavy reporting workload overwhelms an Aurora writer. The company wants capacity for reads to grow and shrink automatically, with no application rewrite beyond using the reader endpoint. Which approach fits?
A. Resize the writer to the largest instance. B. Add Aurora Replicas and configure Application Auto Scaling on the replica count. C. Increase DynamoDB capacity. D. Place the database in more subnets.
Answer: B. Aurora Replicas add read capacity behind the reader endpoint, and Application Auto Scaling adjusts their number. A is vertical, limited and disruptive. C is a different service. D does not add capacity.
Completion checklist
- I can state what each of EC2, ECS, Lambda, DynamoDB and Aurora scales on
- I can calculate the capacity needed to survive one AZ loss for 2, 3 and 4 AZs
- I ran the lab tests (including
ScalingTests) - I wrote a scaling standard for Harbourline services
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- Minimum capacity, not the scaling policy, protects you in the first minutes of a failure.
- Run three AZs and size
total = AZs x peak / (AZs - 1)if you want to lose one AZ without scaling. - Cap the maximum from your downstream budget, and scale in slowly.
- Scaling is a reaction through the control plane, so pair it with load shedding and resilience patterns.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Availability vocabulary: HA, fault tolerance, DR, RTO and RPO
Speak precisely about failure: separate high availability from fault tolerance and disaster recovery, turn a target such as 99.95% into minutes of allowed downtime, and set RTO, RPO and impact-tolerance numbers a bank can test.
Elastic Load Balancing for resilience: cross-zone, health checks and zonal shift
Use load balancer behaviour as a resilience tool: choose cross-zone settings, design health checks that detect real failure without causing outages, drain targets cleanly, and move traffic away from a sick Availability Zone.