Rollback strategies: code, configuration and data
Decide when to roll back and when to roll forward, wire automatic rollback for ECS, Lambda and CloudFormation, use feature flags as the fastest back-out, and change a banking database schema with expand/contract migrations that never block a rollback.
Assumes: .NET 10 LTS · EF Core 10 (Npgsql.EntityFrameworkCore.PostgreSQL 10.0.3) · Amazon.Extensions.Configuration.SystemsManager 7.1.1 · AWS AppConfig · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Classify every part of a release — code, configuration, infrastructure, schema, data, messages, external side effects — by whether it can be rolled back, and how.
- Choose between automatic rollback, manual rollback and roll-forward, and write the decision rule down before an incident, not during one.
- Configure automatic rollback on ECS (deployment circuit breaker and deployment alarms), CodeDeploy, and CloudFormation (rollback triggers).
- Use AWS AppConfig feature flags as a back-out mechanism that does not need a deployment.
- Change a database schema with expand/contract (parallel change) migrations in EF Core, and enforce the rule with a test that blocks destructive migrations in the wrong release.
Prerequisites
- Lambda deployments — aliases, CodeDeploy and alarm-based rollback.
- ECS deployment strategies and CodeBuild, CodePipeline and CodeDeploy.
- Module 3: Configuration done right — configuration providers in ASP.NET Core.
- Comfortable with EF Core migrations from your .NET work.
- Tools: .NET 10 SDK (or Docker), AWS CLI v2 with the
learn-sandboxprofile.
Why this matters in banking
A change-advisory board at any bank asks the same question about every release: "What is the back-out plan?" Too often the answer is "redeploy the previous build", and too often that answer is wrong.
Here is a Harbourline incident report, lightly fictionalised. Release 4.12 of the Accounts API renamed the column beneficiaries.name to display_name in a single EF Core migration and shipped the code that used it. Twenty minutes after go-live, a different bug in the same release caused intermittent 500 errors on the "add beneficiary" screen. The on-call engineer did the textbook thing: rolled the ECS service back to release 4.11. Release 4.11 immediately crashed on every request that touched beneficiaries, because the column it expected — name — no longer existed. The "rollback" turned a partial outage into a full one. Restoring the database was ruled out because 6,000 payments had been booked since go-live. The team spent three hours writing a hot-fix forward under pressure.
The code rollback worked perfectly. The data could not follow it. This lesson is about designing releases so that a rollback is always possible for code, and so that changes which cannot be rolled back — schema, data, messages already sent to other banks — are made in a way that never needs one.
Regulators care too. UK and EU operational-resilience rules (for example the PRA/FCA operational-resilience framework and the EU Digital Operational Resilience Act) expect firms to manage ICT change risk and to stay within impact tolerances for important business services. A tested back-out plan is part of the evidence. Whether your plan meets a specific rule is a question for your risk and compliance colleagues, not for the pipeline.
The idea in plain language
Rollback means "return to the last known-good state". Roll-forward means "fix the problem with a new release instead". Both are legitimate. The trick is knowing which parts of a release can go back:
| Part of a release | Can it roll back? | How |
|---|---|---|
| Application code (container image, Lambda version) | Yes, cheaply | Point the service or alias at the previous image or version |
| Application configuration (feature flags, limits) | Yes, cheaply | Revert the configuration version; AppConfig can do it automatically |
| Infrastructure (CloudFormation, CDK, Terraform) | Usually | CloudFormation rolls back failed updates automatically; Terraform needs you to re-apply the previous state of the code |
| Database schema | Sometimes, dangerously | A "down" migration can drop columns that now hold new data |
| Database data written by the new release | Almost never | Restoring a backup also throws away every legitimate transaction since |
| Messages and events already published | No | A payment sent to Faster Payments, an email sent to a customer, a file sent to a partner — those happened |
| External contracts (API shapes other teams depend on) | Only if they never changed incompatibly | Version the API; keep old fields until all consumers move |
From that table come three rules that the rest of the lesson puts into practice:
- Make code and configuration rollback automatic and fast. Alarms and health checks decide; humans watch.
- Make everything that cannot roll back backward-compatible. Schema and contracts change in small steps that both the old and the new code can live with (expand/contract).
- Separate "deploy" from "release". Ship the new code switched off behind a feature flag. Turning the flag off is the fastest rollback of all.
Analogy: a bank moving a branch to a new building does not close the old one the night before. It opens the new building, runs both for a while with staff who know both addresses, redirects customers gradually, and only hands back the old keys when nobody has visited for a month. If the new building floods on day two, customers simply go back to the old one. Expand/contract is the database version of keeping the old branch open.
What problem the AWS services solve
AWS gives you rollback mechanisms at each layer. None of them knows anything about your data — that part is always your design.
| Layer | AWS mechanism | What triggers it | Notes |
|---|---|---|---|
| ECS rolling update | Deployment circuit breaker with rollback: true | Tasks fail to reach a steady state (fail to start or fail health checks) repeatedly | Catches crashes and failing health checks; does not catch a healthy-looking task that returns wrong answers |
| ECS rolling update | Deployment alarms with rollback: true | A CloudWatch alarm you nominate goes into ALARM during the deployment | Catches business symptoms (5xx rate, latency, rejection rate) |
| ECS blue/green (native, since July 2025) or CodeDeploy blue/green | Alarms, lifecycle hooks and a bake time during which blue stays running | Alarm or failed hook during traffic shift or bake | Rollback is a traffic switch back to blue, typically very fast |
| Lambda with CodeDeploy | Alarms and hooks (previous lesson) | Alarm or hook failure | Alias moves back to the previous version |
| CloudFormation | Automatic rollback of a failed stack operation; rollback triggers (up to 5 alarms, monitoring period 0–180 minutes) | Resource failure, or an alarm during or after the update | Only covers what is in the stack; data in an RDS instance is not "rolled back" |
| AWS AppConfig | Deployment strategy with CloudWatch alarm monitors | Alarm during a configuration deployment or bake | Reverts the configuration (including feature flags) without touching code |
| Aurora / RDS | Automated backups and point-in-time restore to a new cluster or instance | Manual | A disaster-recovery tool, not a release rollback tool: it loses all writes after the chosen time |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| App Service slot swap back | ECS blue/green traffic switch, Lambda alias repoint | Automated by alarms during the deployment rather than a manual swap |
| Azure DevOps release "redeploy previous release" | Re-run pipeline with previous image tag, or ECS/CodeDeploy automatic rollback | Prefer automatic rollback inside the deployment; keep the manual path as a rehearsed runbook |
Azure App Configuration feature flags + Microsoft.FeatureManagement | AWS AppConfig feature flags | AppConfig deploys configuration with a strategy (gradual, with bake time) and can roll back on alarms |
| ARM/Bicep deployment "rollback on error" | CloudFormation automatic rollback + rollback triggers | CloudFormation can also watch alarms for up to 180 minutes after the update |
| EF Core migrations run in the release pipeline | Same — EF Core migration bundles run as a pipeline step or ECS task | AWS has no special tool here; the expand/contract discipline is identical |
| Azure SQL point-in-time restore | Aurora/RDS point-in-time restore | Both restore to a new database; neither is a release rollback |
Architecture
The diagram shows the key design decision: code and configuration have automatic back-out paths; the schema does not, so every schema change must be safe for both the previous and the current code.
The decision flow during an incident
Write this flow into the runbook for every service. During an incident, nobody should be inventing it.
Step-by-step implementation
Write the rollback plan per release type
For each service, list the release types you make (code only; code + config; code + additive schema; contract change) and the back-out for each. Attach the table to the service's README. Releases that do not fit any row are "normal changes" that need an explicit, reviewed back-out plan.
Turn on automatic rollback for code
On ECS services: enable the deployment circuit breaker with rollback, and nominate deployment alarms (5xx rate on the target group, p99 latency, a business metric). On Lambda: CodeDeploy with alarms (previous lesson). On CloudFormation stacks that contain the service: add rollback triggers with a monitoring period long enough to see real traffic, for example 15 minutes.
Put risky behaviour behind a feature flag
Create an AppConfig application, environment and feature flag configuration profile. Deploy flag changes with a gradual deployment strategy and attach the same CloudWatch alarms as monitors, so a flag that causes errors is turned off again automatically.
Split every schema change into expand and contract
Expand (release N): add new tables, new nullable columns, new indexes; write to both old and new; read from old. Migrate (between releases): backfill in small batches. Switch (release N+1): read from new; still write both. Contract (release N+2 or later, once N+1 is stable): stop writing the old column, then drop it. Each step is compatible with the release before it, so code rollback stays safe.
Run migrations as their own pipeline step
Build an EF Core migration bundle (dotnet ef migrations bundle) in CI, store it with the image, and run it as a one-off ECS task with a dedicated database role before the service deploy. Never let the application apply migrations at start-up in production: ten tasks starting at once means ten processes racing to alter the same table.
Enforce the rule with a test
Add a unit test that inspects every migration and fails if a destructive operation (drop column, drop table, rename, type change) appears in a migration that is not explicitly marked as a contract step with a change reference. Code review then focuses on the marked migrations.
Rehearse
Once per quarter, in the test account, deploy a deliberately broken release and confirm each automatic rollback fires. Record the time to recovery. That number goes into the service's operational-resilience evidence.
AWS Console instructions
ECS — turn on circuit breaker and deployment alarms:
- Console → Elastic Container Service → Clusters →
hl-m09-cluster→ Services →hl-m09-accounts-api→ Update service. - Under Deployment options, keep Rolling update. Expand Deployment failure detection.
- Tick Use the Amazon ECS deployment circuit breaker and Rollback on failures.
- Tick Use CloudWatch alarm(s), choose your alarms, and tick Rollback on failures for alarms too.
- Update. The Deployments tab now shows the rollback state for each deployment.
CloudFormation — rollback triggers on an update:
- Console → CloudFormation → Stacks → your stack → Update.
- On Configure stack options → Rollback configuration, enter a monitoring time (0–180 minutes) and add each alarm ARN.
AppConfig — feature flag with alarm rollback:
- Console → AWS AppConfig → Applications → Create application (
hl-m09-accounts). - Configuration profiles and feature flags → Create → Feature flag, name it
accounts-flags, add a flaguseNewBeneficiaryScreen(off). - Environments → Create environment (
prod), and under Monitors add the CloudWatch alarm ARN plus a role that lets AppConfig read alarms. - Start deployment, choosing a gradual deployment strategy (for example
AppConfig.Linear20PercentEvery6Minutes) ⚠ VERIFY the predefined strategy names in your Region.
AWS CLI
P=learn-sandbox
# 1. ECS: enable the circuit breaker with rollback and attach deployment alarms.
aws ecs update-service --cluster hl-m09-cluster --service hl-m09-accounts-api \
--deployment-configuration '{
"deploymentCircuitBreaker": {"enable": true, "rollback": true},
"alarms": {"alarmNames": ["hl-m09-accounts-5xx-rate","hl-m09-accounts-p99"], "enable": true, "rollback": true},
"maximumPercent": 200, "minimumHealthyPercent": 100
}' --profile $P
# 2. ECS: see the state of the latest deployments, including rollbacks.
aws ecs describe-services --cluster hl-m09-cluster --services hl-m09-accounts-api \
--query 'services[0].deployments[].[id,status,rolloutState,rolloutStateReason,taskDefinition]' \
--output table --profile $P
# 3. ECS manual rollback: point the service back at the previous task-definition revision.
aws ecs update-service --cluster hl-m09-cluster --service hl-m09-accounts-api \
--task-definition hl-m09-accounts-api:41 --profile $P
# 4. CloudFormation: update a stack with a rollback trigger watched for 15 minutes afterwards.
aws cloudformation update-stack --stack-name hl-m09-accounts \
--use-previous-template --capabilities CAPABILITY_NAMED_IAM \
--rollback-configuration "RollbackTriggers=[{Arn=arn:aws:cloudwatch:us-east-1:111122223333:alarm:hl-m09-accounts-5xx-rate,Type=AWS::CloudWatch::Alarm}],MonitoringTimeInMinutes=15" \
--profile $P
# 5. CloudFormation: an update stuck in UPDATE_ROLLBACK_FAILED can be continued,
# skipping resources you have fixed by hand.
aws cloudformation continue-update-rollback --stack-name hl-m09-accounts --profile $P
# 6. AppConfig: turn a feature flag off by deploying the previous configuration version.
aws appconfig start-deployment --application-id abc1234 --environment-id env5678 \
--configuration-profile-id prof9012 --configuration-version 3 \
--deployment-strategy-id AppConfig.AllAtOnce --profile $P
# 7. AppConfig: stop a configuration deployment in progress (rolls it back).
aws appconfig stop-deployment --application-id abc1234 --environment-id env5678 \
--deployment-number 12 --profile $P
# 8. Aurora point-in-time restore (DISASTER RECOVERY, not release rollback):
# restores to a NEW cluster; the original keeps running.
aws rds restore-db-cluster-to-point-in-time --db-cluster-identifier hl-m09-accounts-restore \
--source-db-cluster-identifier hl-m09-accounts --restore-to-time 2026-11-18T09:55:00Z \
--profile $PThe IDs in steps 6 and 7 (abc1234 and so on) are placeholders; get yours with aws appconfig list-applications and the related list commands.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.AppConfig;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.ECS;
using Amazon.CDK.AWS.ECS.Patterns;
using Constructs;
namespace Harbourline.Accounts.Infra;
public sealed class AccountsServiceStack : Stack
{
public AccountsServiceStack(Construct scope, string id, AccountsServiceProps props)
: base(scope, id, props)
{
var service = new ApplicationLoadBalancedFargateService(this, "Api",
new ApplicationLoadBalancedFargateServiceProps
{
Cluster = props.Cluster,
ServiceName = "hl-m09-accounts-api",
DesiredCount = 2,
TaskImageOptions = new ApplicationLoadBalancedTaskImageOptions
{
Image = ContainerImage.FromEcrRepository(props.Repository, props.ImageTag),
ContainerPort = 8080
},
// Rolling update with automatic rollback when tasks cannot become healthy.
CircuitBreaker = new DeploymentCircuitBreaker { Enable = true, Rollback = true },
MinHealthyPercent = 100
});
// A business-level symptom: 5xx responses from the target group.
var fiveXx = new Alarm(this, "Api5xx", new AlarmProps
{
AlarmName = "hl-m09-accounts-5xx-rate",
Metric = service.TargetGroup.Metrics.HttpCodeTarget(
Amazon.CDK.AWS.ElasticLoadBalancingV2.HttpCodeTarget.TARGET_5XX_COUNT,
new MetricOptions { Period = Duration.Minutes(1), Statistic = "Sum" }),
Threshold = 5,
EvaluationPeriods = 2,
TreatMissingData = TreatMissingData.NOT_BREACHING
});
// Roll the ECS deployment back if the alarm fires while it is in progress.
service.Service.EnableDeploymentAlarms(new[] { fiveXx.AlarmName },
new DeploymentAlarmOptions { Behavior = AlarmBehavior.ROLLBACK_ON_ALARM });
// Feature flags in AppConfig, deployed gradually and watched by the same alarm.
var app = new Application(this, "AppConfigApp", new ApplicationProps
{
ApplicationName = "hl-m09-accounts"
});
var env = new Amazon.CDK.AWS.AppConfig.Environment(this, "Prod", new EnvironmentProps
{
Application = app,
EnvironmentName = "prod",
Monitors = new[] { Amazon.CDK.AWS.AppConfig.Monitor.FromCloudWatchAlarm(fiveXx) }
});
new HostedConfiguration(this, "Flags", new HostedConfigurationProps
{
Application = app,
Name = "accounts-flags",
Type = ConfigurationType.FEATURE_FLAGS,
Content = ConfigurationContent.FromInlineJson("""
{
"version": "1",
"flags": { "useNewBeneficiaryScreen": { "name": "useNewBeneficiaryScreen" } },
"values": { "useNewBeneficiaryScreen": { "enabled": false } }
}
"""),
DeployTo = new[] { env },
DeploymentStrategy = new Amazon.CDK.AWS.AppConfig.DeploymentStrategy(this, "Gradual", new DeploymentStrategyProps
{
RolloutStrategy = RolloutStrategy.Linear(new RolloutStrategyProps
{
GrowthFactor = 20,
DeploymentDuration = Duration.Minutes(10),
FinalBakeTime = Duration.Minutes(10)
})
})
});
}
}
public sealed class AccountsServiceProps : StackProps
{
public required ICluster Cluster { get; init; }
public required Amazon.CDK.AWS.ECR.IRepository Repository { get; init; }
public required string ImageTag { get; init; }
}The AppConfig L2 constructs shown (Application, Environment, HostedConfiguration, DeploymentStrategy, Monitor) are in Amazon.CDK.AWS.AppConfig in recent CDK v2 releases ⚠ VERIFY property names against your Amazon.CDK.Lib version. The feature-flag JSON format (version, flags, values) is the AppConfig AWS.AppConfig.FeatureFlags schema.
Complete code
Three pieces of runnable C#: a three-phase expand/contract migration in EF Core, a test that blocks destructive migrations outside a contract release, and an ASP.NET Core endpoint that reads an AppConfig feature flag.
1. Expand/contract in EF Core: renaming beneficiaries.name safely
namespace Harbourline.Accounts.Data;
public class Beneficiary
{
public required string BeneficiaryId { get; init; }
public required string CustomerId { get; init; }
public required string Iban { get; set; }
public DateTime CreatedAt { get; init; }
// OLD column. Still written in N and N+1 so that release N-1 keeps working after a rollback.
[Obsolete("Use DisplayName. Removed in the contract release (CHG-2481).")]
public string? Name { get; set; }
// NEW column. Nullable during expand; becomes required in the contract release.
public string? DisplayName { get; set; }
/// <summary>Single place that writes both columns during the transition.</summary>
public void SetDisplayName(string value)
{
DisplayName = value;
#pragma warning disable CS0618 // deliberate dual write
Name = value;
#pragma warning restore CS0618
}
/// <summary>Release N reads old-then-new; release N+1 reads new-then-old.</summary>
public string EffectiveDisplayName =>
#pragma warning disable CS0618
DisplayName ?? Name ?? string.Empty;
#pragma warning restore CS0618
}using Microsoft.EntityFrameworkCore.Migrations;
namespace Harbourline.Accounts.Data.Migrations;
/// <summary>EXPAND (release N): add the new column. Old code ignores it; new code dual-writes.</summary>
public partial class ExpandBeneficiaryDisplayName : Migration
{
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AddColumn<string>(
name: "display_name",
table: "beneficiaries",
type: "character varying(140)",
maxLength: 140,
nullable: true); // nullable: rows written by old code are still valid
}
protected override void Down(MigrationBuilder migrationBuilder)
{
// Safe to reverse: nothing depends on display_name yet except release N itself.
migrationBuilder.DropColumn(name: "display_name", table: "beneficiaries");
}
}-- Backfill in small batches so locks stay short and replication lag stays low.
-- Re-run until it reports 0 rows. Safe to run while both releases are live.
WITH batch AS (
SELECT beneficiary_id
FROM beneficiaries
WHERE display_name IS NULL AND name IS NOT NULL
LIMIT 5000
FOR UPDATE SKIP LOCKED
)
UPDATE beneficiaries b
SET display_name = b.name
FROM batch
WHERE b.beneficiary_id = batch.beneficiary_id;using Microsoft.EntityFrameworkCore.Migrations;
namespace Harbourline.Accounts.Data.Migrations;
/// <summary>
/// CONTRACT (release N+2): only after N+1 has been stable and no release older than
/// N+1 can be rolled back to. Marked so the safety test allows the destructive operation.
/// </summary>
[ContractPhase("CHG-2481")]
public partial class ContractBeneficiaryName : Migration
{
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AlterColumn<string>(
name: "display_name", table: "beneficiaries",
type: "character varying(140)", maxLength: 140, nullable: false,
oldClrType: typeof(string), oldType: "character varying(140)",
oldMaxLength: 140, oldNullable: true);
migrationBuilder.DropColumn(name: "name", table: "beneficiaries");
}
protected override void Down(MigrationBuilder migrationBuilder)
{
// Re-creating the column restores the SHAPE, not lost data: copy it back explicitly.
migrationBuilder.AddColumn<string>(
name: "name", table: "beneficiaries",
type: "character varying(140)", maxLength: 140, nullable: true);
migrationBuilder.Sql("UPDATE beneficiaries SET name = display_name;");
migrationBuilder.AlterColumn<string>(
name: "display_name", table: "beneficiaries",
type: "character varying(140)", maxLength: 140, nullable: true,
oldClrType: typeof(string), oldType: "character varying(140)",
oldMaxLength: 140, oldNullable: false);
}
}2. The migration safety test
namespace Harbourline.Accounts.Data;
/// <summary>
/// Marks a migration as a deliberate CONTRACT step. The change reference links to the
/// approved change record that confirmed no older release can be rolled back to.
/// </summary>
[AttributeUsage(AttributeTargets.Class, AllowMultiple = false)]
public sealed class ContractPhaseAttribute(string changeReference) : Attribute
{
public string ChangeReference { get; } = changeReference;
}using System.Reflection;
using Harbourline.Accounts.Data;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Infrastructure;
using Microsoft.EntityFrameworkCore.Migrations;
using Microsoft.EntityFrameworkCore.Migrations.Operations;
using Xunit;
public class MigrationSafetyTests
{
// Operations that can break the PREVIOUS release if it is rolled back to.
private static readonly Type[] Destructive =
[
typeof(DropColumnOperation), typeof(DropTableOperation), typeof(RenameColumnOperation),
typeof(RenameTableOperation), typeof(AlterColumnOperation), typeof(DropSchemaOperation)
];
[Fact]
public void Destructive_operations_only_appear_in_contract_phase_migrations()
{
// No database connection is opened: we only build the model and read migrations.
var options = new DbContextOptionsBuilder<AccountsDbContext>()
.UseNpgsql("Host=unused;Database=unused")
.Options;
using var db = new AccountsDbContext(options);
var assembly = db.GetService<IMigrationsAssembly>();
var provider = db.Database.ProviderName!;
var violations = new List<string>();
foreach (var (id, typeInfo) in assembly.Migrations)
{
if (typeInfo.GetCustomAttribute<ContractPhaseAttribute>() is { } contract)
{
Assert.Matches(@"^CHG-\d+$", contract.ChangeReference);
continue; // reviewed and approved as a contract step
}
var migration = assembly.CreateMigration(typeInfo, provider);
foreach (var op in migration.UpOperations)
{
if (Destructive.Contains(op.GetType()))
violations.Add($"{id}: {op.GetType().Name}");
}
}
Assert.True(violations.Count == 0,
"Destructive operations outside a [ContractPhase] migration:\n" + string.Join("\n", violations));
}
}3. Reading a feature flag from AppConfig in ASP.NET Core
using Harbourline.Accounts.Data;
var builder = WebApplication.CreateBuilder(args);
// Load the "accounts-flags" feature-flag profile from AWS AppConfig and refresh it
// every 30 seconds. The task role needs appconfig:StartConfigurationSession and
// appconfig:GetLatestConfiguration on this application only.
builder.Configuration.AddAppConfig(
applicationId: builder.Configuration["AppConfig:ApplicationId"]!,
environmentId: builder.Configuration["AppConfig:EnvironmentId"]!,
configProfileId: builder.Configuration["AppConfig:FlagsProfileId"]!,
reloadAfter: TimeSpan.FromSeconds(30));
var app = builder.Build();
app.MapGet("/customers/{customerId}/beneficiaries", async (
string customerId, IConfiguration config, IBeneficiaryReader reader, CancellationToken ct) =>
{
// Feature flags arrive as { "useNewBeneficiaryScreen": { "enabled": true } },
// which the configuration provider flattens to "useNewBeneficiaryScreen:enabled".
var useNew = config.GetValue<bool>("useNewBeneficiaryScreen:enabled");
var items = useNew
? await reader.ListWithDisplayNameAsync(customerId, ct) // new code path
: await reader.ListLegacyAsync(customerId, ct); // known-good path
return Results.Ok(items);
});
app.Run();AddAppConfig comes from the Amazon.Extensions.Configuration.SystemsManager package (7.1.1). IBeneficiaryReader is your existing data-access interface; the point is that both code paths ship in the same release and the flag chooses between them at runtime.
Code explained line by line
Beneficiary.cs
[Obsolete(...)] public string? Name— the compiler now warns anyone who touches the old column, and the message names the change that will remove it.public string? DisplayName— nullable while rows written by the old code exist.SetDisplayName— the only place that writes; it writes both columns. Dual writes are what make rollback to the previous release safe: the old release readsname, which is still current.EffectiveDisplayName— reads new-then-old, so rows not yet backfilled still display correctly.
ExpandBeneficiaryDisplayName
AddColumn ... nullable: true— adding a nullable column without a default is a fast, metadata-only change in PostgreSQL, so it does not lock the table for long.Downdrops the column — acceptable here, because no released code depends on it before release N.
Backfill SQL
LIMIT 5000 ... FOR UPDATE SKIP LOCKED— small batches, skipping rows another session holds, so it never blocks live traffic for long.WHERE display_name IS NULL AND name IS NOT NULL— idempotent: re-running only touches rows still to do.
ContractBeneficiaryName
[ContractPhase("CHG-2481")]— the marker the safety test looks for. The reference points to a change record confirming that no release older than N+1 is a rollback target.AlterColumn ... nullable: falsethenDropColumn("name")— the destructive part, deliberately isolated in its own release.Downrecreatesnameand copies data back fromdisplay_name. A down migration that only re-adds a column would restore the shape but lose the values; that is how "rollbacks" silently corrupt data.
MigrationSafetyTests
Destructive— the operation types that can break the previous release.AlterColumnOperationis included because type and nullability changes can break old code too.UseNpgsql("Host=unused...")— builds the model without connecting; the test runs in CI with no database.db.GetService<IMigrationsAssembly>()— EF Core's internal service that knows every migration in the assembly.assembly.CreateMigration(typeInfo, provider)thenmigration.UpOperations— EF Core builds the list of operations theUpmethod would run, without executing SQL.- The
ContractPhasebranch checks the change reference format, so nobody can wave a migration through with an empty string. - The final assertion prints every violation, which makes the failure message a review checklist.
Program.cs
AddAppConfig(...)— adds AppConfig as a configuration source; values refresh on the interval you set.config.GetValue<bool>("useNewBeneficiaryScreen:enabled")— reads the flag through normalIConfiguration, so the code has no AWS-specific feature-flag dependency.- Both branches exist in the same build. Turning the flag off in AppConfig is a rollback that needs no deployment, no image and no database change.
Security considerations
⚠ Learning shortcut — not for production
In this lesson the migration bundle runs with the same database credentials as the application. In production, run migrations with a separate database role that has DDL rights, retrieved from Secrets Manager only by the migration task's IAM role. The application's role should have only DML rights (select, insert, update), so a compromised application cannot drop tables.
⚠ Learning shortcut — not for production
Anyone who can deploy an AppConfig configuration version can switch production behaviour without a code review. In production, restrict appconfig:StartDeployment on the prod environment to the pipeline role, require flag changes to go through a pull request on the flag JSON, and alert on manual deployments via CloudTrail.
- Rollback is a production change. A manual rollback still needs to be recorded: who, when, why, which version. Use the same pipeline (run with the previous artefact) rather than console clicks, so the record is automatic.
- Previous images must still exist. An ECR lifecycle policy that keeps only the last 3 images can delete your rollback target after a busy week. Keep at least the images currently deployed in any environment plus a margin, and make images immutable (tag immutability on).
- Previous images may have known vulnerabilities. Rolling back to a build with a critical CVE that was fixed in the new release is a risk decision; the runbook should say who can accept it.
- Data protection. A point-in-time restore creates a full copy of customer data in a new cluster. It needs the same encryption, network isolation and deletion plan as production; under GDPR / UK GDPR, an untracked copy of personal data is a problem in itself.
Banking use case
Harbourline's platform team publishes a release classification that every service team uses. A release's class determines the rollback approach and the approval route.
| Class | Contents | Back-out | Approval |
|---|---|---|---|
| A — code only | Image or Lambda version change, no schema, no contract change | Automatic (circuit breaker, alarms, CodeDeploy) | Standard change; pipeline evidence is enough |
| B — code + flags | New behaviour shipped off, enabled via AppConfig | Flag off (automatic on alarm) then code rollback if needed | Standard change for deploy; flag enablement recorded separately |
| C — expand | Additive schema change + dual-writing code | Code rollback safe; schema stays | Standard change if the migration safety test passes |
| D — contract | Destructive schema or breaking contract change | Roll forward only | Normal change, reviewed by the database owner and service owner; scheduled outside payment cut-off windows |
The incident from the opening would have been a class D change disguised as class A. With the safety test in place, the rename migration would have failed CI because it was not marked [ContractPhase], and the reviewer would have asked for the expand/contract split.
Common mistakes
- Treating "redeploy the previous build" as the whole back-out plan → data and schema don't follow the code → classify every release and plan per class.
- Applying EF Core migrations at application start-up → race conditions with many tasks, and the app's role needs DDL rights → run a migration bundle as a separate task.
- Writing
Downmigrations that only restore structure → rolling back silently loses data → either copy data back explicitly or declare the migration irreversible. - Circuit breaker without alarms → healthy-looking tasks returning wrong answers are never rolled back → add business alarms as deployment alarms.
- Alarms already in
ALARMbefore the deployment → immediate rollback of a good release, or alarms ignored → check alarm states as a pipeline step before deploying. - Feature flags that never get removed → two code paths forever, both needing tests → give every flag an owner and a removal date.
- Using point-in-time restore as a rollback → loses every legitimate transaction since the restore point → keep PITR for disasters and corruption, not releases.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
ECS deployment shows rolloutState: FAILED with "circuit breaker" reason | New tasks failed to start or failed health checks repeatedly | Check stopped-task reasons (describe-tasks) and container logs; the service is already back on the previous task definition |
| ECS deployment rolled back but the app looked fine | A deployment alarm was in ALARM from an earlier, unrelated event | Check alarm history; add a pre-deploy "all alarms OK" check |
CloudFormation stack in UPDATE_ROLLBACK_FAILED | A resource couldn't return to its old state (deleted outside CloudFormation, or a dependency changed) | Fix the resource, then continue-update-rollback, skipping resources if needed |
| CloudFormation rolled back 10 minutes after "success" | A rollback trigger alarm fired during the monitoring period | Expected behaviour; investigate the alarm. Reduce the period only if the alarm is noisy |
AppConfig deployment status ROLLED_BACK | A monitor alarm went into ALARM during deployment or bake | Check the alarm; fix the flag or the code, then redeploy |
| Old release crashes after rollback with "column does not exist" | A destructive migration shipped with the new release | Roll forward; add the migration safety test so it cannot happen again |
MigrationSafetyTests fails on an old migration | The test was added after destructive migrations already existed | Mark historical ones with [ContractPhase("CHG-legacy")] after review, or start the check from a baseline migration ID |
Cost and cleanup
Cost: everything in the C# section runs locally and in CI for free. The ECS, CloudFormation and AppConfig settings themselves cost nothing extra. AppConfig charges per configuration request and per configuration received ⚠ VERIFY current AppConfig pricing; a lab with one task polling every 30 seconds costs a few cents per month at most. If you try the ECS steps, the Fargate tasks and load balancer are billed hourly — deploy, test and destroy within an hour. Aurora point-in-time restore creates a new, billed cluster: only try it in a deploy-and-destroy session, and delete the restored cluster immediately. Label: free for the core exercises.
P=learn-sandbox
# Delete the AppConfig resources (environments and profiles first, then the application).
APP=$(aws appconfig list-applications --query "Items[?Name=='hl-m09-accounts'].Id" --output text --profile $P)
for e in $(aws appconfig list-environments --application-id $APP --query 'Items[].Id' --output text --profile $P); do
aws appconfig delete-environment --application-id $APP --environment-id $e --profile $P; done
for c in $(aws appconfig list-configuration-profiles --application-id $APP --query 'Items[].Id' --output text --profile $P); do
for v in $(aws appconfig list-hosted-configuration-versions --application-id $APP --configuration-profile-id $c --query 'Items[].VersionNumber' --output text --profile $P); do
aws appconfig delete-hosted-configuration-version --application-id $APP --configuration-profile-id $c --version-number $v --profile $P; done
aws appconfig delete-configuration-profile --application-id $APP --configuration-profile-id $c --profile $P; done
aws appconfig delete-application --application-id $APP --profile $P
# If you deployed the CDK stack: destroy it.
cdk destroy --profile $P
# If you restored an Aurora cluster: delete its instances first, then the cluster, without a final snapshot.
aws rds delete-db-cluster --db-cluster-identifier hl-m09-accounts-restore --skip-final-snapshot --profile $P
# Confirm nothing tagged for this lesson remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m09-l9 \
--query 'ResourceTagMappingList[].ResourceARN' --profile $PExercises
- 🟢 Guided: Add
ContractPhaseAttributeandMigrationSafetyTeststo a solution of your own with at least two EF Core migrations. Write a third migration that renames a column, run the tests and watch them fail. Rewrite it as an expand step and watch them pass. - 🟡 Challenge: Design the expand/contract sequence for changing
transactions.amountfromnumeric(18,2)in pence-free pounds to a pairamount_minor bigint+currency_exponent smallint. List each release, what the code reads and writes, the backfill, and the exact point after which code rollback to the original release is no longer allowed. - 🔴 Live problem: At 10:20 a release of the Payments Service goes out. At 10:35 the operations team reports that 2% of SEPA payments are being sent with the wrong end-to-end ID format and some have already been accepted by the clearing system. The ECS deployment succeeded and no alarms fired. Write the first 30 minutes of the incident response: rollback or roll-forward, what to do about the payments already sent, and what you change in the pipeline afterwards.
Assignment
Write a ROLLBACK.md for one of your portfolio services containing: the release classes table adapted to your service; the decision flow diagram (Mermaid); the automatic rollback settings with their alarms; one rehearsal record (date, what was broken, time to recovery); and the migration safety test running in CI.
Solutions
The rename migration generated by EF Core contains a RenameColumnOperation, which the test reports as 2026…_RenameX: RenameColumnOperation. Rewriting it as expand means: one migration with AddColumn (nullable), entity code that dual-writes, and a separate later migration — marked [ContractPhase("CHG-…")] — that drops the old column. Why: after the expand migration, both the old release (reads the old column) and the new release (reads new, falls back to old) work against the same schema, so the ECS circuit breaker can roll back code at any time without a schema crash.
Release N (expand): add nullable amount_minor bigint and currency_exponent smallint; code writes amount and both new columns (amount_minor = amount * 10^exponent, exponent 2 for GBP and EUR); reads amount. Backfill: batch-convert existing rows; verify with a reconciliation query that amount_minor / 10^exponent = amount for every row — a bank should keep this query's output as evidence. Release N+1 (switch): read new columns; still write all three. Release N+2 (contract): stop writing amount; set the new columns NOT NULL. Release N+3 (contract, final): drop amount. Rollback to N is allowed until N+2 ships, because only from N+2 onwards does amount stop being correct. Every downstream consumer — the data lake, reconciliation jobs, reports — must have switched before N+3; that is usually the slowest part.
First 10 minutes: stop the bleeding. If the new behaviour is behind a flag, turn it off; otherwise roll back the code (this is a class A release, so code rollback is safe). Stopping new bad payments is more urgent than diagnosing. Payments already sent: they cannot be rolled back — they are with the clearing system. Identify them precisely (logs with correlation IDs, payments table by created_at and release version), inform the payments operations team, and follow the scheme's exception and investigation process. This is a business and possibly regulatory incident, not just a technical one. Afterwards: add a pre-traffic or post-deploy check that validates end-to-end ID format against the ISO 20022 rules; add a business metric on format violations with a deployment alarm; consider a longer canary for the Payments Service. Record the incident and link it to the release evidence.
Knowledge check
Answer: It restores to a new cluster and discards every write made after the restore time — including legitimate customer transactions. It is a disaster-recovery tool for corruption or loss, not a way to undo a release.
Answer: It detects tasks that repeatedly fail to start or fail health checks during a deployment and (with rollback: true) returns the service to the last completed deployment. It misses tasks that are healthy but behave wrongly — for that, add deployment alarms on error rates or business metrics.
Answer: Only once no release that reads or writes the old column is running or could be rolled back to — typically two releases after the expand step, and only after every downstream consumer has moved.
Answer: It changes behaviour without a deployment: no new image, no task replacement, no schema involvement. With AppConfig monitors it can even switch itself back automatically when an alarm fires.
Answer: Normal rollback happens when a resource fails to create or update. Rollback triggers also watch up to five CloudWatch alarms during the update and for a monitoring period (up to 180 minutes) afterwards, rolling the stack back if any alarm goes into ALARM.
Interview questions
- "What is your rollback strategy?" — It depends on what changed. Code and configuration roll back automatically on alarms; schema changes are expand/contract so code rollback is always safe; destructive schema and contract changes are roll-forward only and go through normal change approval. External side effects (payments sent) are handled by business processes, not by the pipeline.
- "How do you run EF Core migrations in a pipeline?" — Build a migration bundle in CI, run it as a one-off task with a DDL-capable role before deploying the service, keep migrations additive in normal releases, and enforce that with a test. Never migrate at app start-up in production.
- "Roll back or roll forward?" — Roll back when the previous state is known-good and reachable without data loss, which should be the default for class A/B/C releases. Roll forward when the release crossed a one-way door (destructive schema, contract change, external effects) or when the fix is trivial and well-tested — and always under incident change control.
- "How do feature flags interact with audit?" — A flag flip changes production behaviour, so it needs the same controls as a deployment: who changed it, when, approved by whom, and visible in CloudTrail and AppConfig deployment history.
Certification practice
An ECS service on Fargate uses rolling updates. Some releases start successfully and pass load-balancer health checks but return HTTP 500 for a specific API path. The team wants such deployments to roll back automatically. What should they do?
A. Enable the deployment circuit breaker with rollback.
B. Create a CloudWatch alarm on the target group's 5xx count and configure it as an ECS deployment alarm with rollback enabled.
C. Increase the health-check grace period.
D. Set minimumHealthyPercent to 100.
Answer: B. Deployment alarms catch symptoms that health checks don't. A only reacts to tasks that fail to start or fail health checks — these tasks are healthy. C delays health checks and makes things worse. D keeps capacity during the deployment but does not detect errors.
A team must rename a column used by an ASP.NET Core API. The API is deployed to ECS with automatic rollback enabled. Which approach keeps automatic rollback safe?
A. Rename the column and deploy the new code in the same release.
B. Add the new column, write to both, backfill, switch reads in a later release, and drop the old column in a final release.
C. Rename the column and rely on the EF Core Down migration if a rollback occurs.
D. Take a snapshot before deploying and restore it on rollback.
Answer: B. Expand/contract keeps the schema compatible with both releases. A breaks the previous release on rollback. C — ECS rollback does not run migrations. D loses all writes since the snapshot.
A company wants to enable a new pricing rule gradually for its payments API and automatically disable it if the API's error rate rises, without redeploying the application. Which solution meets these requirements?
A. Store the rule in an environment variable and update the ECS task definition. B. Use AWS AppConfig feature flags with a linear deployment strategy and a CloudWatch alarm configured as an environment monitor. C. Store the rule in an S3 object and have the application poll it. D. Use Parameter Store and a Lambda function that reverts the parameter when an alarm fires.
Answer: B. AppConfig deploys configuration gradually and rolls it back when a monitor alarm fires. A requires a new deployment. C and D need custom rollout and rollback logic.
A developer wants CloudFormation to roll back a stack update if a latency alarm goes into ALARM within 20 minutes after the update completes. What should they configure?
A. A stack policy.
B. Termination protection.
C. A rollback configuration with the alarm as a rollback trigger and a monitoring time of 20 minutes.
D. A DeletionPolicy: Retain on the resources.
Answer: C. Rollback triggers watch alarms during and after the update for up to 180 minutes. A prevents updates to resources; B prevents stack deletion; D affects deletion behaviour only.
Completion checklist
- I can classify a release's contents by whether each part can roll back
- I configured (or can configure) ECS circuit breaker and deployment alarms with rollback
- I can explain CloudFormation rollback triggers and their limits
- I wrote an expand/contract plan for a schema change
- The migration safety test runs in my CI and blocks unmarked destructive migrations
- I removed any AppConfig, ECS or restored database resources from the sandbox
Key takeaways
- Code and configuration roll back cheaply and should do so automatically on alarms; data, schema, contracts and external side effects do not.
- Expand/contract keeps every schema change compatible with the previous release, so code rollback is always safe.
- Feature flags separate deploying from releasing and give you the fastest, deployment-free back-out.
- A circuit breaker sees crashes; deployment alarms see symptoms. Use both.
- Point-in-time restore is for disasters, not releases. Destructive changes are roll-forward only and go through normal change approval.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Lambda deployments: versions, aliases and safe traffic shifting
Release a .NET 10 Lambda function the way a bank should: immutable versions, a stable alias, canary traffic through CodeDeploy, pre-traffic and post-traffic hooks written in C#, and automatic rollback on CloudWatch alarms.
Environment separation: one AWS account per environment
Separate dev, test, staging and prod into their own AWS accounts under AWS Organizations, deploy into each with its own OIDC role from GitHub Actions, promote the same image digest from one environment to the next, and add guardrails so a pipeline can never deploy to the wrong account.