ECS deployment strategies: rolling, blue/green, linear and canary
Choose and configure the right Amazon ECS deployment strategy for a .NET API, wire alarm-based and circuit-breaker rollback, use lifecycle hooks as release gates, and rehearse a failed release on purpose.
Assumes: .NET 10 LTS · Amazon.CDK.Lib 2.x (L1 CfnService) · Terraform AWS provider 6.x · AWS CLI v2 · Amazon ECS native deployment strategies (ROLLING, BLUE_GREEN, LINEAR, CANARY) · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain the four ECS-native deployment strategies (
ROLLING,BLUE_GREEN,LINEAR,CANARY) and pick one for a given banking workload. - Configure the deployment circuit breaker for rolling updates and CloudWatch alarm rollback for any strategy.
- Describe the blue/green lifecycle stages, and use a Lambda hook or a
PAUSEhook as a release gate. - Explain how ECS pins a task definition to an image digest, and why that matters for rollback and audit.
- Break a release on purpose with the lab's
Health__ForceUnreadysetting and watch ECS return traffic to the old revision. - Compare ECS-native strategies with the older CodeDeploy (
CODE_DEPLOY) controller.
Prerequisites
- Docker and Amazon ECR in pipelines — images referenced by digest and immutable tags.
- Rollback strategies — why some changes (database, messages) cannot simply be rolled back.
- Lambda deployments — the same ideas (versions, weighted aliases, alarms) for serverless.
- Module 3: .NET on AWS — an ASP.NET Core API on ECS Fargate behind an Application Load Balancer (ALB).
- Tools: AWS CLI v2 with the
learn-sandboxprofile; Docker; .NET 10 SDK (or Docker) for the local parts.
Why this matters in banking
A payments API at Harbourline handles thousands of requests a minute. A release that returns HTTP 500 to one request in twenty is not a dramatic outage, but it is a stream of failed payments, duplicate retries, customer calls and, in some jurisdictions, an incident report to a regulator. The deployment strategy decides how many customers meet a bad release, for how long, and how quickly the platform notices on its own.
Three habits separate a mature bank from the rest:
- Small blast radius first. Send a small slice of real traffic to the new version before everyone else gets it.
- Objective rollback triggers. A person at 2 a.m. should not be deciding whether error rate 1.4% is "bad". An alarm decides, and the platform acts.
- A rehearsed rollback. The first time you roll back must not be during an incident. This lesson makes you break a release deliberately, in a sandbox, and watch the safety net work.
The idea in plain language
Every time you deploy, ECS has to get from "version N is running" to "version N+1 is running". The strategies differ in two things: do the old and new versions run side by side and how does traffic move.
| Strategy | Old and new at once? | How traffic moves | Rollback speed | Extra cost while deploying |
|---|---|---|---|---|
ROLLING | Yes, briefly, mixed in one pool | Tasks are replaced a batch at a time; the load balancer sees new tasks as they pass health checks | Slow: ECS must replace tasks again | Small (extra tasks up to maximumPercent) |
BLUE_GREEN | Yes, two full copies | A full second copy (green) starts, is tested with test traffic, then production traffic flips to it at once | Fast: flip the listener back while blue is still alive during bake time | A full second copy of the service |
LINEAR | Yes, two full copies | Equal steps (for example 10% every 6 minutes) until 100% | Fast until the final step | Full second copy for the whole ramp |
CANARY | Yes, two full copies | A small percentage first, a bake period, then everything else at once | Fast | Full second copy for the canary period |
Analogy: a branch is moving from the old teller system to a new one. Rolling swaps one counter at a time while the queue keeps moving. Blue/green builds a complete second hall, lets staff test it with dummy customers, then re-points the front door in one go, and keeps the old hall open for an hour in case. Canary lets only the first few customers through the new hall and watches their faces. Linear admits a bit more every few minutes.
What problem ECS solves here (and where it stops)
Until 2025, Amazon ECS could do rolling updates itself, but blue/green and canary needed AWS CodeDeploy as the deployment controller. ECS now supports BLUE_GREEN, LINEAR and CANARY natively, with the ECS service scheduler doing the work. The deploymentConfiguration of the service takes a strategy field with the values ROLLING | BLUE_GREEN | LINEAR | CANARY.
What this gives you:
- Two target groups and listener rules: ECS manages the shift between a production target group and an alternate one, plus an optional test listener rule so you can reach green before customers do.
- Bake time: after production traffic has shifted, both revisions keep running for
bakeTimeInMinutesso a rollback is a traffic flip, not a redeploy. You must provide this value forBLUE_GREEN. - Alarm-based rollback: CloudWatch alarms attached to the deployment can fail it and roll back.
- Lifecycle hooks: Lambda functions or pause points at defined stages.
What it does not do:
- It does not roll back your database. See Rollback strategies.
- It does not compare two versions' business metrics for you. You decide which alarms matter.
- Blue/green, linear and canary need an ALB (or other supported load balancer / VPC Lattice target); they are traffic-shifting strategies. Check the ECS documentation for the current list of supported services and launch types ⚠ VERIFY.
- It costs money while two copies run.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| App Service deployment slots with swap | ECS BLUE_GREEN | A slot swap is a DNS/routing swap with warm-up. ECS keeps both revisions alive for the bake time and swaps via ALB listener rules |
| Slot swap with auto swap and health check | Bake time plus alarm rollback | Rollback is driven by CloudWatch alarms, not by a manual swap-back |
| Traffic routing percentage on slots ("testing in production") | CANARY and LINEAR | Percentage is applied at the ALB between two target groups |
AKS rolling update with maxSurge and maxUnavailable | ECS ROLLING with maximumPercent and minimumHealthyPercent | Same idea: how many extra tasks, how many may be missing |
Kubernetes readinessProbe | ALB target group health check, container health check | Health check path is what decides whether new tasks may take traffic |
| Azure DevOps release gate: "query Azure Monitor alerts" | ECS deployment alarms; lifecycle hook Lambda | The gate is part of the service deployment itself, not only of the pipeline |
| Azure Pipelines manual intervention | PAUSE lifecycle hook with ContinueServiceDeployment | A pause inside the deployment, not before it |
Architecture
The blue/green life of one deployment
Security-boundary view
The deploy role from the lab can register task definitions and update one service. The permission to move traffic between target groups belongs to a separate ECS infrastructure role that ECS assumes. A compromised pipeline therefore cannot rewrite ALB rules directly.
Step-by-step implementation
Decide the strategy per service, not per company
Use a short decision rule and write it in the service's runbook:
- Stateless internal API, many tasks, tolerant of a mixed pool:
ROLLINGwith circuit breaker and alarms. - Customer-facing API, real money:
CANARY(5 to 10%, 10 to 30 minutes) orLINEAR. - Change with a risky cut-over and an easy undo (for example a feature behind a gate you cannot flag):
BLUE_GREENwith aPAUSEhook for a human check. - Service with strict connection state or very long requests: think twice. Long-lived connections stay on the old tasks until they drain, so test the drain time.
Build the infrastructure once
Blue/green and canary need: an ALB, two target groups (blue and green, same port and health check), a production listener rule, an optional test listener rule (a different port or header condition), and an ECS infrastructure IAM role that lets ECS modify those rules. The Module 3 Fargate stack gives you the ALB and service; this lab adds the second target group and the role (see the Infrastructure section).
Make the health check mean something
The ALB probes /health/ready in the lab API. A readiness check that only returns 200 without touching a dependency will happily send traffic to a broken release. The lab's ReadinessCheck pings its repository, and you can force it to fail with the environment variable Health__ForceUnready=true for the rollback drill.
Create the alarms that decide
At minimum: HTTP 5xx count on the target group and p99 target response time. Alarms must exist before you reference them in the deployment configuration. Use TreatMissingData of notBreaching for 5xx counts (no traffic is not an error) but think carefully for latency alarms.
Enable the strategy
Update the service with the deployment configuration (shown in the AWS CLI section). The lab's deploy/ecs/service-deployment.json sets BLUE_GREEN, a 10-minute bake time, two alarms with rollback enabled and one PAUSE hook after test traffic.
Let the pipeline deploy by digest
The pipeline renders the task definition with an image digest (see the release workflow). ECS also resolves tags to digests by default, so every task of a revision runs identical bytes.
Rehearse a failure
Deploy a revision with Health__ForceUnready=true. Green tasks never pass the target group health check, the deployment fails, and traffic stays on blue. Then deploy a revision that is healthy at start but breaks under traffic (an alarm test), and watch the alarm-driven rollback.
Rolling updates and the circuit breaker
ROLLING is the default. Two numbers control the batch size:
minimumHealthyPercent(default 100 for a replica service): the lower bound of healthy tasks during the deployment, as a percentage ofdesiredCount, rounded up.maximumPercent(default 200 for a replica service): the upper bound of running plus pending tasks, rounded down.
With four tasks, 100/200 means ECS may start four new tasks before stopping any old ones: no loss of capacity, double the cost during the deploy. With 50/100 ECS stops two, starts two, then repeats: no extra cost, half the capacity mid-deploy. For a payments API, prefer 100/200.
Two failure detectors can end a rolling deployment early, and they can be combined (either one failing marks the deployment failed):
| Detector | Use it when | Triggers on |
|---|---|---|
Deployment circuit breaker (deploymentCircuitBreaker with enable and rollback) | You want to stop when new tasks cannot start or cannot become healthy | Repeated task launch failures or tasks that fail to reach steady state |
CloudWatch alarms (alarms with alarmNames, enable, rollback) | You want to stop on application metrics | Any named alarm entering ALARM during the deployment |
The circuit breaker applies only to the rolling-update type. For the traffic-shifting strategies, health checks and alarms are your detectors. For rolling deployments you can also configure early success criteria, which lets ECS declare success when a percentage of target tasks is healthy instead of waiting for full stability; use it carefully for payment services because it trades certainty for speed.
Blue/green in detail
Lifecycle stages. Each stage can host a hook, apart from the ones that are pure waiting:
| Stage | What is true | Hook allowed |
|---|---|---|
RECONCILE_SERVICE | Only when a deployment starts while more than one revision is active | Yes |
PRE_SCALE_UP | Green has not started; blue takes 100% of production traffic | Yes |
SCALE_UP | Green tasks are launching; no traffic to green | No |
POST_SCALE_UP | Green is running; no test traffic yet | Yes |
TEST_TRAFFIC_SHIFT | Test traffic moves from 0 to 100% to green; production stays on blue | Lambda only |
POST_TEST_TRAFFIC_SHIFT | Green takes 100% of test traffic | Yes |
PRE_PRODUCTION_TRAFFIC_SHIFT | Before production shift (before every step for linear and canary) | Yes |
PRODUCTION_TRAFFIC_SHIFT | Production traffic moves to green | Lambda only |
POST_PRODUCTION_TRAFFIC_SHIFT | Production shift complete | Yes |
BAKE_TIME | Blue and green both running after the shift | No |
CLEAN_UP | Blue scaled to zero; green is now the production revision | No |
Hook types. AWS_LAMBDA (the default) invokes a function you wrote, passes your hookDetails, and expects a success or failure answer. PAUSE stops the deployment until you call the ContinueServiceDeployment API, either to continue or to roll back. PAUSE cannot be used at TEST_TRAFFIC_SHIFT or PRODUCTION_TRAFFIC_SHIFT.
Time limits. A lifecycle stage can last up to 24 hours, and each traffic-shift step within PRODUCTION_TRAFFIC_SHIFT can as well; after that ECS fails and rolls back. AWS recommends staying below the limit. If the deployment is driven by CloudFormation there is an additional 36-hour limit for the whole deployment, and pause hooks can be configured for up to 20,160 minutes (14 days) with an overall 30-day deployment timeout.
Test traffic. The TEST_TRAFFIC_SHIFT stage is where you run your post-deployment smoke tests against green through the test listener, before a single customer sees it. A Lambda hook can call https://alb-test-listener/version, compare the commit SHA with what the pipeline expects (the lab API exposes /version), run three read-only business calls, and return success or failure.
Canary and linear
Both use the same machinery as blue/green but move production traffic in steps.
- Canary:
canaryConfigurationwithcanaryPercent(multiples of 0.1 from 0.1 to 100.0, default 5.0) andcanaryBakeTimeInMinutes(0 to 1440, default 10). Step 1 shifts the canary percentage, the service waits for the canary bake time, step 2 shifts the remainder. - Linear:
linearConfigurationwithstepPercent(multiples of 0.1 from 3.0 to 100.0, default 10.0) andstepBakeTimeInMinutes(0 to 1440, default 6). The bake time is not applied after reaching 100%.
PRE_PRODUCTION_TRAFFIC_SHIFT and PRODUCTION_TRAFFIC_SHIFT hooks run at every step, so a Lambda hook can evaluate error rate at 10%, 20%, and so on. The deployment-level bakeTimeInMinutes still applies after 100% so the old revision stays available.
Caveat on small percentages. A 5% canary on a service handling 20 requests per minute sees one request a minute. Pick a percentage and bake time that produce enough requests for your alarm to mean something. The lab includes DeploymentHealthGate (in src/Harbourline.Delivery), which encodes this: it holds the verdict until MinRequestsForVerdict requests have been seen.
Image digests and version consistency
By default ECS resolves the image tag in the task definition to a digest when the first task of a revision starts, then starts every other task of that revision from the same digest. The per-container versionConsistency setting controls this. Points to know:
- If digest resolution fails three times, the deployment continues without it; with the circuit breaker enabled the deployment fails instead.
- Resolution needs a recent enough ECS agent on EC2 and a minimum Fargate platform version; current Fargate platform versions satisfy it.
- Do not rely on this alone. The lab pipeline renders the digest into the task definition itself, so the task definition revision, the audit record and the running task all name the same bytes.
Comparing with the CodeDeploy controller
| Aspect | ECS-native (deploymentController: ECS) | CodeDeploy (CODE_DEPLOY) |
|---|---|---|
| Where config lives | ECS service deploymentConfiguration | CodeDeploy application, deployment group and AppSpec |
| Strategies | Rolling, blue/green, linear, canary | Blue/green with predefined linear and canary configurations |
| Rollback triggers | Alarms, circuit breaker (rolling), hooks | Alarms, deployment failure, manual stop |
| Hooks | Lambda or pause at lifecycle stages | AppSpec hooks (for example BeforeAllowTraffic) |
| Pipeline integration | UpdateService; any CI tool | CodePipeline CodeDeployToECS action or create-deployment |
| Existing services | Switch by updating the service | A service created with CODE_DEPLOY is not simply converted; check the migration guidance ⚠ VERIFY |
If you already run CodeDeploy for ECS and it works, you do not have to migrate on a deadline. For new services, the native strategies remove a whole set of moving parts (application, deployment group, AppSpec, a separate IAM role). The CodeDeploy details, including Lambda, are in Lambda deployments.
AWS Console instructions
- Console → ECS → Clusters → your cluster → Services → your service → Update service.
- Expand Deployment options. Choose the Deployment strategy: Rolling update, Blue/green, Linear or Canary.
- For blue/green: set Bake time (minutes). For canary or linear set the percentage and bake times.
- Under Load balancing, choose the production listener rule and the alternate target group, the optional test listener rule, and the infrastructure role for the load balancer.
- Under Deployment failure detection, enable Use CloudWatch alarm(s) and choose the alarms; enable Rollback on failures. For rolling updates also enable Use the Amazon ECS deployment circuit breaker.
- (Optional) Add lifecycle hooks: choose Lambda or pause, the stages, and the role.
- Update. Watch Deployments tab, then Events.
Console labels move between releases; if a label differs, search the page for the field names above ⚠ VERIFY.
AWS CLI
P=learn-sandbox
# 1. Switch the service to ECS-native blue/green using the lab's file.
# The file names the cluster, service, deployment configuration and target groups.
aws ecs update-service --cli-input-json file://deploy/ecs/service-deployment.json --profile $P
# 2. Deploy a new revision (the pipeline normally does this; here by hand).
aws ecs register-task-definition --cli-input-json file://taskdef-rendered.json --profile $P
aws ecs update-service --cluster hl-m09-cluster --service hl-m09-accounts-api \
--task-definition hl-m09-accounts-api --profile $P
# 3. Watch the service deployments (newer API; shows lifecycle stage and status).
aws ecs list-service-deployments --cluster hl-m09-cluster --service hl-m09-accounts-api --profile $P
aws ecs describe-service-deployments --service-deployment-arns <arn-from-previous-command> --profile $P
# 4. A PAUSE hook is waiting: continue (approve), or roll back.
aws ecs continue-service-deployment --service-deployment-arn <arn> --profile $P # ⚠ VERIFY exact CLI name and flags
# 5. Stop a bad deployment and roll back by hand.
aws ecs stop-service-deployment --service-deployment-arn <arn> --stop-type ROLLBACK --profile $P # ⚠ VERIFY flags
# 6. Which image digest is actually running?
aws ecs describe-tasks --cluster hl-m09-cluster \
--tasks $(aws ecs list-tasks --cluster hl-m09-cluster --service-name hl-m09-accounts-api --query 'taskArns[0]' --output text --profile $P) \
--query 'tasks[0].containers[0].[image,imageDigest]' --output text --profile $PThe lab's configuration file (already in the repository) looks like this; read it line by line below.
{
"cluster": "hl-m09-cluster",
"service": "hl-m09-accounts-api",
"deploymentConfiguration": {
"strategy": "BLUE_GREEN",
"bakeTimeInMinutes": 10,
"alarms": {
"alarmNames": ["hl-m09-accounts-api-5xx", "hl-m09-accounts-api-p99-latency"],
"enable": true,
"rollback": true
},
"lifecycleHooks": [
{ "targetType": "PAUSE", "lifecycleStages": ["POST_TEST_TRAFFIC_SHIFT"] }
]
}
}strategy: BLUE_GREENchooses the traffic-shifting strategy; replace withCANARYand addcanaryConfigurationto try a canary.bakeTimeInMinutes: 10keeps blue alive for ten minutes after the shift, so rollback is a listener flip.alarms.enableandalarms.rollbackboth true: an alarm firing fails the deployment and rolls back, rather than only reporting.- The
PAUSEhook afterPOST_TEST_TRAFFIC_SHIFTstops the deployment once green passes test traffic, until an authorised person continues it. In a regulated team that person is the second approver.
The loadBalancers block of the same file names both target groups, the production and test listener rules and the infrastructure role. The ARNs in the lab use the fictional account 111122223333.
Infrastructure as Code
The L2 FargateService construct may not yet expose every new deployment field. The robust pattern is to keep the L2 service and set the new properties through the underlying L1 CfnService with a property override. Property names follow the CloudFormation schema for AWS::ECS::Service DeploymentConfiguration ⚠ VERIFY against your CDK version.
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.ECS;
using Amazon.CDK.AWS.ElasticLoadBalancingV2;
using Constructs;
namespace Harbourline.Delivery.Infra;
public static class BlueGreenSettings
{
/// <summary>Alarms that decide whether the release stays. Created BEFORE the deployment references them.</summary>
public static (Alarm Errors, Alarm Latency) CreateAlarms(Construct scope, IApplicationTargetGroup blue)
{
var errors = new Alarm(scope, "Accounts5xx", new AlarmProps
{
AlarmName = "hl-m09-accounts-api-5xx",
Metric = ((ApplicationTargetGroup)blue).Metrics.HttpCodeTarget(
HttpCodeTarget.TARGET_5XX_COUNT,
new MetricOptions { Period = Duration.Minutes(1), Statistic = "Sum" }),
Threshold = 5,
EvaluationPeriods = 2,
DatapointsToAlarm = 2,
ComparisonOperator = ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING,
});
var latency = new Alarm(scope, "AccountsP99", new AlarmProps
{
AlarmName = "hl-m09-accounts-api-p99-latency",
Metric = ((ApplicationTargetGroup)blue).Metrics.TargetResponseTime(
new MetricOptions { Period = Duration.Minutes(1), Statistic = "p99" }),
Threshold = 0.8, // seconds
EvaluationPeriods = 3,
DatapointsToAlarm = 3,
TreatMissingData = TreatMissingData.NOT_BREACHING,
});
return (errors, latency);
}
/// <summary>Switch the service to ECS-native blue/green through the L1 resource.</summary>
public static void Apply(FargateService service, string[] alarmNames, int bakeMinutes)
{
var cfn = (CfnService)service.Node.DefaultChild!;
cfn.AddPropertyOverride("DeploymentConfiguration.Strategy", "BLUE_GREEN");
cfn.AddPropertyOverride("DeploymentConfiguration.BakeTimeInMinutes", bakeMinutes);
cfn.AddPropertyOverride("DeploymentConfiguration.Alarms", new Dictionary<string, object>
{
["AlarmNames"] = alarmNames,
["Enable"] = true,
["Rollback"] = true,
});
}
}Use Amazon.CDK.Tags.Of(this) (not Tags.Of) inside a Stack to apply the lab tags. The alternate target group, listener rules and the ECS infrastructure role are ordinary CDK resources; the infrastructure role trusts ecs.amazonaws.com and uses the AWS managed policy for ECS load balancer management ⚠ VERIFY exact managed-policy name in IAM → Policies.
Complete code
The runnable parts of this lesson live in labs/m09-delivery-pipeline: the API with its /health/ready and /version endpoints, the task definition and the deployment configuration, and the C# health gate that turns "is this canary healthy?" into code with unit tests. The gate is useful inside a Lambda lifecycle hook or a post-deploy pipeline step:
using Harbourline.Delivery.Deployments;
// Samples collected every minute for the GREEN target group (from CloudWatch).
var samples = new List<HealthSample>
{
new(DateTimeOffset.UtcNow.AddMinutes(-2), Requests: 120, ServerErrors: 0, P99LatencyMs: 310,
HealthyTargets: 2, DesiredTargets: 2, AlarmsInAlarm: []),
new(DateTimeOffset.UtcNow.AddMinutes(-1), Requests: 140, ServerErrors: 9, P99LatencyMs: 940,
HealthyTargets: 2, DesiredTargets: 2, AlarmsInAlarm: []),
};
var result = DeploymentHealthGate.Evaluate(samples, new GatePolicy());
Console.WriteLine($"{result.Decision}: {string.Join(" ", result.Reasons)}");
// Exit non-zero so a pipeline step (or Lambda hook) reports failure and ECS rolls back.
return result.Decision == GateDecision.Rollback ? 1 : 0;From the lab root you can also run the same logic on the sample files:
dotnet run --project src/Harbourline.Delivery -- gate samples/canary-healthy.json # Proceed, exit 0
dotnet run --project src/Harbourline.Delivery -- gate samples/canary-failing.json # Rollback, exit 1Code explained line by line
using Harbourline.Delivery.Deployments;imports the gate types from the lab's library project.HealthSampleis one observation of the new revision: request count, 5xx count, p99 latency, healthy and desired target counts and the names of alarms currently in ALARM.- The first sample is healthy. The second has 9 errors in 140 requests (about 6.4%) and a p99 of 940 ms, both above the default
GatePolicylimits (1% errors, 800 ms). EvaluatereturnsProceed,HoldorRollbackwith reasons. Any alarm in ALARM is an immediate rollback; too few requests givesHold, because an error rate computed from a handful of requests is noise.- Returning exit code 1 for
Rollbackmakes the calling step fail. In a Lambda hook the equivalent is returning the failure status the hook contract expects ⚠ VERIFY the hook response format in the ECS lifecycle hook documentation. - In
BlueGreenSettings.CreateAlarms,TreatMissingData.NOT_BREACHINGstops "no requests in the last minute" from counting as an error.DatapointsToAlarm = 2of two evaluation periods avoids paging on one noisy minute. - In
Apply,AddPropertyOverridewrites directly into the CloudFormation template for the service. It is an escape hatch: when the L2 construct gains first-class properties, move to them.
Security considerations
⚠ Learning shortcut — not for production
The lab uses one cluster and one service for both "test" and "prod" deployments, force-deletes the ECR repository on destroy and runs tasks with a public-facing ALB over HTTP for speed. In production: a separate AWS account per environment, ALB with TLS 1.2+ and a WAF web ACL, access logs on, deletion protection on the ALB, and the cluster inside private subnets with VPC endpoints.
- Least privilege for the deploy role. It needs
ecs:RegisterTaskDefinition,ecs:UpdateServiceon one service, the read calls to watch deployments, andiam:PassRolelimited to the two task roles with the conditioniam:PassedToService = ecs-tasks.amazonaws.com. It does not need load balancer or listener permissions: ECS uses the infrastructure role for that. - The infrastructure role should be assumable only by
ecs.amazonaws.comand scoped to the lab's load balancer where the managed policy allows ⚠ VERIFY. - Hooks are code that runs with a role. A Lambda hook role should read alarms and call only the test listener; do not give it access to production data.
- PAUSE hook approvers. Who may call
ContinueServiceDeployment? Restrict that action to an approver role, and log it (CloudTrail records the call). This is a segregation-of-duties control; see Release management and audit evidence. - Test listener exposure. The test listener lets someone reach green before release. Restrict its security group to the pipeline's egress addresses or an internal CIDR, not the internet.
- Image provenance. Deploy only digests that the build job scanned and attested; the lesson on pipeline security shows how to verify them before deployment.
- Compliance. Deployment controls support change-management evidence; they do not make you compliant. Your risk and compliance teams decide what "approved release" means.
Banking use case
Harbourline's Payments Initiation API has a 99.95% internal availability target and a rule: no release during the month-end settlement window without a second approver. The team chooses CANARY: 10% for 15 minutes, then everything, then a 20-minute deployment bake. A PAUSE hook is configured at POST_TEST_TRAFFIC_SHIFT: the pipeline runs smoke tests against the test listener, then waits. The on-call lead (not the author) calls ContinueServiceDeployment after checking dashboards. During the canary, alarms on 5xx and on a business metric (payment initiation failures per minute, published by the app as a custom CloudWatch metric) are attached with rollback on. All of it is recorded as part of the release evidence: deployment ARN, stage timeline, approver identity from CloudTrail, alarm states during bake.
Common mistakes
- Alarms created after the deployment config. The service cannot reference an alarm that does not exist; create alarms first, and give them the right dimensions (target group of the new revision).
- Alarm on the wrong target group. In blue/green, an alarm only on the blue target group may stay green while green fails. Alarm on metrics that cover the traffic the new revision receives, or on both groups.
- Readiness check that always says yes. Then the canary "passes" while the API cannot reach its database. Make readiness depend on what the service really needs, but not on optional dependencies.
- A bake time shorter than the alarm's evaluation window. If the alarm needs three minutes to evaluate and bake time is one minute, rollback comes too late.
- Mutable tags in the task definition.
:latestmakes rollback ambiguous. Use digests. - Forgetting database compatibility. Blue and green run at the same time against the same database; both versions must work with the schema. See expand/contract migrations in Rollback strategies.
- Terraform fighting the pipeline. Without
ignore_changesontask_definition, a plan reverts what the pipeline deployed.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Deployment stuck at PRE_SCALE_UP or never starts green tasks | Task cannot be placed: subnet or security group wrong, ENI limit, no capacity | Check service events and stoppedReason of the stopped tasks |
| Green tasks start then are replaced repeatedly | Container exits (bad configuration) or health check path returns non-200 | Read CloudWatch log group /ecs/hl-m09-accounts-api; test the image locally with the same env vars |
ReadinessCheck fails in AWS but not locally | Missing network path to the dependency or secret not injected | Check the task execution role, secret ARN, security groups |
| Deployment rolled back, nobody knows why | An alarm fired during the deployment | describe-service-deployments and the alarm history in CloudWatch |
| Alarm never fires during a real failure | Alarm on blue only, wrong dimension, or TreatMissingData hides it | Alarm on the new revision's target group; use missing or breaching for traffic-dependent signals only after reasoning |
PAUSE hook never continues | Nobody called ContinueServiceDeployment; approver lacks permission | Grant the approver role the action; remember the 24-hour stage limit |
AccessDenied on UpdateService in the pipeline | Deploy role missing iam:PassRole for the task roles | Add the two role ARNs with iam:PassedToService |
| Terraform wants to downgrade the task definition | Missing ignore_changes | Add it; let the pipeline own revisions |
| Old connections still hitting blue after the shift | Long-lived keep-alive connections | Wait for the ALB deregistration delay; tune it and the bake time |
Cost and cleanup
Cost: credits. During a blue/green or canary deployment you pay for two copies of the service. The lab service is two Fargate tasks of 0.25 vCPU and 0.5 GB each, so four tasks for roughly ten to fifteen minutes per deployment, plus the ALB hourly charge and LCU usage, plus CloudWatch alarms (about 0.10 USD per standard alarm per month, pro-rated) ⚠ VERIFY current prices. Deploy, test and destroy within an hour: a few cents for the tasks and a few cents for the ALB hour.
Teardown (order matters because the service holds the target groups):
P=learn-sandbox
aws ecs update-service --cluster hl-m09-cluster --service hl-m09-accounts-api --desired-count 0 --profile $P
aws ecs delete-service --cluster hl-m09-cluster --service hl-m09-accounts-api --force --profile $P
# If you created the stack with CDK or Terraform, destroy it that way instead:
# cdk destroy --all | terraform destroy
aws cloudwatch delete-alarms --alarm-names hl-m09-accounts-api-5xx hl-m09-accounts-api-p99-latency --profile $P
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m09-delivery-pipeline --profile $P --region us-east-1The last command lists anything still tagged for this lab. An ALB left running is the usual surprise on the bill.
Free alternative: do the design-only exercises. Run DeploymentHealthGate against samples/canary-healthy.json and samples/canary-failing.json, write the alarm definitions and deployment configuration, and review them with a colleague without deploying.
Exercises
- 🟢 Guided: Run the lab's tests and the
gatecommand on both samples. Then write (do not deploy) aCANARYdeployment configuration: 10% for 15 minutes with the two lab alarms and rollback on. - 🟡 Challenge: Add a third alarm for a business metric (
PaymentInitiationFailures, Sum over 1 minute, threshold 3) to the configuration and extend the health gate sample format with anAlarmsInAlarmentry that triggers rollback. Write the xUnit test. - 🔴 Live problem: Ticket CHG-104310: "Customers saw intermittent 502s for about 4 minutes after yesterday's Payments release; the dashboard showed green." The service used
ROLLINGwith100/200and no alarms. Propose a strategy and configuration change, name the exact signals you would alarm on, and explain what the 502s say about your health check.
Assignment
Write a deployment strategy decision record (one page, Markdown) for the Accounts API: chosen strategy, percentages, bake times, alarms with thresholds and the reasoning, who may continue a paused deployment, and the rollback rehearsal you ran or designed. Store it in your portfolio repository next to the pipeline files.
Solutions
{
"deploymentConfiguration": {
"strategy": "CANARY",
"bakeTimeInMinutes": 20,
"canaryConfiguration": { "canaryPercent": 10.0, "canaryBakeTimeInMinutes": 15 },
"alarms": {
"alarmNames": ["hl-m09-accounts-api-5xx", "hl-m09-accounts-api-p99-latency"],
"enable": true,
"rollback": true
}
}
}canaryPercent and canaryBakeTimeInMinutes are the documented field names of the canary configuration. The deployment-level bakeTimeInMinutes keeps blue alive after the full shift. The healthy sample returns Proceed (exit 0) and the failing sample Rollback (exit 1) because its error rate or alarm state is over the policy.
Add "hl-m09-accounts-api-payment-failures" to alarmNames, create the alarm on a custom metric the application publishes, and in the test add a sample with AlarmsInAlarm: ["hl-m09-accounts-api-payment-failures"] and assert GateDecision.Rollback and that the reason mentions the alarm name. Why: rule 1 of the gate treats any alarm in ALARM as an immediate rollback, regardless of request volume, which is correct for a business alarm that is itself a rate over a window.
Likely a health check that passed while the new tasks could not serve real traffic (for example they were ready before warm-up or a downstream connection pool was empty), and no alarm to stop the roll. Switch to CANARY (10% for 15 minutes) with alarms on target 5xx and on ALB 5xx (502s can come from the ALB itself when targets reset connections), plus p99 latency, with rollback enabled. Make readiness check a real dependency call and add a start-up grace period on the target group. Keep 100/200 capacity so the old revision remains available during the canary. State clearly that you cannot know the root cause without the logs and ALB metrics; the answer is the control, not a guess.
Knowledge check
Answer: The deployment circuit breaker. It applies only to the rolling update type. Alarms work for rolling and traffic-shifting strategies, and hooks belong to blue/green, linear and canary.
Answer: bakeTimeInMinutes, the time both revisions run after production traffic has shifted. Without it there is no safe window for a fast rollback.
Answer: TEST_TRAFFIC_SHIFT and PRODUCTION_TRAFFIC_SHIFT; those stages accept Lambda hooks only.
Answer: Resolution happens at deployment time and can be skipped after repeated failures. A digest in the task definition makes the audit record, the revision and the running task name the same bytes, and tags can be moved if the registry is mutable.
Answer: Green is a different target group. During the canary shift its errors are not counted in blue's metrics. Alarm on the signals that include the new revision's traffic.
Interview questions
- How do you decide between canary and blue/green? Canary when you want real traffic to validate gradually and can tolerate a mixed pool; blue/green when a clean cut-over with a test phase is simpler to reason about. Both need a bake time and alarms.
- How does rollback work in ECS blue/green? While the bake time runs, blue is still running, so rollback re-points the listener rules to blue. After clean-up, rollback means deploying the previous revision again.
- What cannot ECS roll back for you? Database changes, messages already processed, emails sent. Use expand/contract and idempotency.
- Where would you put a human approval inside the deployment? A
PAUSEhook after test traffic, withContinueServiceDeploymentrestricted to an approver role. - What is a good canary percentage? Enough requests to judge in the bake time, for example 5 to 10% on a busy service; larger on a quiet service.
Certification practice
A team runs an ECS service behind an ALB and wants to send 10% of production traffic to a new revision for 15 minutes, then all traffic, and to roll back automatically if the 5xx alarm fires. Which configuration meets this with the least custom code?
A. ROLLING strategy with maximumPercent 110 and the circuit breaker enabled.
B. CANARY strategy with a canary percentage of 10, a canary bake time of 15 minutes, and the alarm attached with rollback enabled.
C. BLUE_GREEN with a bake time of 15 minutes and no alarms.
D. A Lambda function that edits the ALB listener weights every minute.
Answer: B. Canary does the two-step shift and bake, and alarms with rollback provide automatic reversal. A has no traffic-percentage control and the circuit breaker detects launch failures, not 5xx rates. C shifts 100% at once. D is custom code the platform already provides.
A developer deploys a new task definition revision by tag latest, then needs to prove which image ran during an incident. What change best supports that?
A. Enable ECR scan on push. B. Reference the image by digest in the task definition and record the digest in the release record. C. Tag images with the date. D. Enable CloudWatch Container Insights.
Answer: B. A digest is immutable and identifies exact bytes. Scanning (A) and Container Insights (D) are useful but do not identify the image. A date tag (C) is still a tag; with a mutable repository it can be re-pointed.
An ECS Fargate service with four tasks uses rolling updates with minimumHealthyPercent 50 and maximumPercent 100. During deployment users see slow responses. What is the most likely cause and fix?
A. The ALB is misconfigured; add a second ALB.
B. Only two tasks serve traffic at a time during the deployment; raise minimumHealthyPercent to 100 and maximumPercent to 200.
C. Fargate throttles during deployments; use EC2.
D. Enable sticky sessions.
Answer: B. With 50/100 ECS stops half the tasks before starting replacements, halving capacity mid-deploy. 100/200 starts new tasks first. The others do not address reduced capacity.
Completion checklist
- I can explain the four strategies and their trade-offs in two sentences each
- I ran the health gate on both sample files and understand why the verdicts differ
- I wrote a canary configuration with alarms and rollback enabled
- I know which stages accept Lambda hooks and which accept
PAUSE - I ran the cleanup steps and confirmed no ALB, service or alarm is still billing
Key takeaways
- ECS now supports
ROLLING,BLUE_GREEN,LINEARandCANARYnatively; traffic-shifting strategies need an ALB with two target groups and an infrastructure role. - Rollback must be driven by objective signals (alarms, circuit breaker), and rehearsed.
- Bake time turns rollback into a traffic flip; digests make rollback and audit unambiguous.
- Hooks are release gates: Lambda for automated checks,
PAUSEfor human approval. - Neither strategy rolls back your database. Plan expand/contract changes.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Lab 9.1: a CI workflow with Testcontainers and LocalStack
Run the Accounts API test suite locally in Docker, read the ci.yml workflow job by job, add a LocalStack-backed integration test with Testcontainers, and download JUnit results as pipeline evidence, all without an AWS account or any AWS credentials.
Lambda deployments: versions, aliases and safe traffic shifting
Release a .NET 10 Lambda function the way a bank should: immutable versions, a stable alias, canary traffic through CodeDeploy, pre-traffic and post-traffic hooks written in C#, and automatic rollback on CloudWatch alarms.