Skip to Content
Cloud Engineering Academy
9. DevOps & CI/CD

Rollback strategies: code, configuration and data

Decide when to roll back and when to roll forward, wire automatic rollback for ECS, Lambda and CloudFormation, use feature flags as the fastest back-out, and change a banking database schema with expand/contract migrations that never block a rollback.

advanced⏱ 2 h 🟢 FreeDOP-C02DVA-C03

Assumes: .NET 10 LTS · EF Core 10 (Npgsql.EntityFrameworkCore.PostgreSQL 10.0.3) · Amazon.Extensions.Configuration.SystemsManager 7.1.1 · AWS AppConfig · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Classify every part of a release — code, configuration, infrastructure, schema, data, messages, external side effects — by whether it can be rolled back, and how.
  • Choose between automatic rollback, manual rollback and roll-forward, and write the decision rule down before an incident, not during one.
  • Configure automatic rollback on ECS (deployment circuit breaker and deployment alarms), CodeDeploy, and CloudFormation (rollback triggers).
  • Use AWS AppConfig feature flags as a back-out mechanism that does not need a deployment.
  • Change a database schema with expand/contract (parallel change) migrations in EF Core, and enforce the rule with a test that blocks destructive migrations in the wrong release.

Prerequisites

Why this matters in banking

A change-advisory board at any bank asks the same question about every release: "What is the back-out plan?" Too often the answer is "redeploy the previous build", and too often that answer is wrong.

Here is a Harbourline incident report, lightly fictionalised. Release 4.12 of the Accounts API renamed the column beneficiaries.name to display_name in a single EF Core migration and shipped the code that used it. Twenty minutes after go-live, a different bug in the same release caused intermittent 500 errors on the "add beneficiary" screen. The on-call engineer did the textbook thing: rolled the ECS service back to release 4.11. Release 4.11 immediately crashed on every request that touched beneficiaries, because the column it expected — name — no longer existed. The "rollback" turned a partial outage into a full one. Restoring the database was ruled out because 6,000 payments had been booked since go-live. The team spent three hours writing a hot-fix forward under pressure.

The code rollback worked perfectly. The data could not follow it. This lesson is about designing releases so that a rollback is always possible for code, and so that changes which cannot be rolled back — schema, data, messages already sent to other banks — are made in a way that never needs one.

Regulators care too. UK and EU operational-resilience rules (for example the PRA/FCA operational-resilience framework and the EU Digital Operational Resilience Act) expect firms to manage ICT change risk and to stay within impact tolerances for important business services. A tested back-out plan is part of the evidence. Whether your plan meets a specific rule is a question for your risk and compliance colleagues, not for the pipeline.

The idea in plain language

Rollback means "return to the last known-good state". Roll-forward means "fix the problem with a new release instead". Both are legitimate. The trick is knowing which parts of a release can go back:

Part of a releaseCan it roll back?How
Application code (container image, Lambda version)Yes, cheaplyPoint the service or alias at the previous image or version
Application configuration (feature flags, limits)Yes, cheaplyRevert the configuration version; AppConfig can do it automatically
Infrastructure (CloudFormation, CDK, Terraform)UsuallyCloudFormation rolls back failed updates automatically; Terraform needs you to re-apply the previous state of the code
Database schemaSometimes, dangerouslyA "down" migration can drop columns that now hold new data
Database data written by the new releaseAlmost neverRestoring a backup also throws away every legitimate transaction since
Messages and events already publishedNoA payment sent to Faster Payments, an email sent to a customer, a file sent to a partner — those happened
External contracts (API shapes other teams depend on)Only if they never changed incompatiblyVersion the API; keep old fields until all consumers move

From that table come three rules that the rest of the lesson puts into practice:

  1. Make code and configuration rollback automatic and fast. Alarms and health checks decide; humans watch.
  2. Make everything that cannot roll back backward-compatible. Schema and contracts change in small steps that both the old and the new code can live with (expand/contract).
  3. Separate "deploy" from "release". Ship the new code switched off behind a feature flag. Turning the flag off is the fastest rollback of all.

Analogy: a bank moving a branch to a new building does not close the old one the night before. It opens the new building, runs both for a while with staff who know both addresses, redirects customers gradually, and only hands back the old keys when nobody has visited for a month. If the new building floods on day two, customers simply go back to the old one. Expand/contract is the database version of keeping the old branch open.

What problem the AWS services solve

AWS gives you rollback mechanisms at each layer. None of them knows anything about your data — that part is always your design.

LayerAWS mechanismWhat triggers itNotes
ECS rolling updateDeployment circuit breaker with rollback: trueTasks fail to reach a steady state (fail to start or fail health checks) repeatedlyCatches crashes and failing health checks; does not catch a healthy-looking task that returns wrong answers
ECS rolling updateDeployment alarms with rollback: trueA CloudWatch alarm you nominate goes into ALARM during the deploymentCatches business symptoms (5xx rate, latency, rejection rate)
ECS blue/green (native, since July 2025) or CodeDeploy blue/greenAlarms, lifecycle hooks and a bake time during which blue stays runningAlarm or failed hook during traffic shift or bakeRollback is a traffic switch back to blue, typically very fast
Lambda with CodeDeployAlarms and hooks (previous lesson)Alarm or hook failureAlias moves back to the previous version
CloudFormationAutomatic rollback of a failed stack operation; rollback triggers (up to 5 alarms, monitoring period 0–180 minutes)Resource failure, or an alarm during or after the updateOnly covers what is in the stack; data in an RDS instance is not "rolled back"
AWS AppConfigDeployment strategy with CloudWatch alarm monitorsAlarm during a configuration deployment or bakeReverts the configuration (including feature flags) without touching code
Aurora / RDSAutomated backups and point-in-time restore to a new cluster or instanceManualA disaster-recovery tool, not a release rollback tool: it loses all writes after the chosen time

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
App Service slot swap backECS blue/green traffic switch, Lambda alias repointAutomated by alarms during the deployment rather than a manual swap
Azure DevOps release "redeploy previous release"Re-run pipeline with previous image tag, or ECS/CodeDeploy automatic rollbackPrefer automatic rollback inside the deployment; keep the manual path as a rehearsed runbook
Azure App Configuration feature flags + Microsoft.FeatureManagementAWS AppConfig feature flagsAppConfig deploys configuration with a strategy (gradual, with bake time) and can roll back on alarms
ARM/Bicep deployment "rollback on error"CloudFormation automatic rollback + rollback triggersCloudFormation can also watch alarms for up to 180 minutes after the update
EF Core migrations run in the release pipelineSame — EF Core migration bundles run as a pipeline step or ECS taskAWS has no special tool here; the expand/contract discipline is identical
Azure SQL point-in-time restoreAurora/RDS point-in-time restoreBoth restore to a new database; neither is a release rollback

Architecture

The diagram shows the key design decision: code and configuration have automatic back-out paths; the schema does not, so every schema change must be safe for both the previous and the current code.

The decision flow during an incident

Write this flow into the runbook for every service. During an incident, nobody should be inventing it.

Step-by-step implementation

Write the rollback plan per release type

For each service, list the release types you make (code only; code + config; code + additive schema; contract change) and the back-out for each. Attach the table to the service's README. Releases that do not fit any row are "normal changes" that need an explicit, reviewed back-out plan.

Turn on automatic rollback for code

On ECS services: enable the deployment circuit breaker with rollback, and nominate deployment alarms (5xx rate on the target group, p99 latency, a business metric). On Lambda: CodeDeploy with alarms (previous lesson). On CloudFormation stacks that contain the service: add rollback triggers with a monitoring period long enough to see real traffic, for example 15 minutes.

Put risky behaviour behind a feature flag

Create an AppConfig application, environment and feature flag configuration profile. Deploy flag changes with a gradual deployment strategy and attach the same CloudWatch alarms as monitors, so a flag that causes errors is turned off again automatically.

Split every schema change into expand and contract

Expand (release N): add new tables, new nullable columns, new indexes; write to both old and new; read from old. Migrate (between releases): backfill in small batches. Switch (release N+1): read from new; still write both. Contract (release N+2 or later, once N+1 is stable): stop writing the old column, then drop it. Each step is compatible with the release before it, so code rollback stays safe.

Run migrations as their own pipeline step

Build an EF Core migration bundle (dotnet ef migrations bundle) in CI, store it with the image, and run it as a one-off ECS task with a dedicated database role before the service deploy. Never let the application apply migrations at start-up in production: ten tasks starting at once means ten processes racing to alter the same table.

Enforce the rule with a test

Add a unit test that inspects every migration and fails if a destructive operation (drop column, drop table, rename, type change) appears in a migration that is not explicitly marked as a contract step with a change reference. Code review then focuses on the marked migrations.

Rehearse

Once per quarter, in the test account, deploy a deliberately broken release and confirm each automatic rollback fires. Record the time to recovery. That number goes into the service's operational-resilience evidence.

AWS Console instructions

ECS — turn on circuit breaker and deployment alarms:

  1. Console → Elastic Container Service → Clusters → hl-m09-cluster → Services → hl-m09-accounts-api → Update service.
  2. Under Deployment options, keep Rolling update. Expand Deployment failure detection.
  3. Tick Use the Amazon ECS deployment circuit breaker and Rollback on failures.
  4. Tick Use CloudWatch alarm(s), choose your alarms, and tick Rollback on failures for alarms too.
  5. Update. The Deployments tab now shows the rollback state for each deployment.

CloudFormation — rollback triggers on an update:

  1. Console → CloudFormation → Stacks → your stack → Update.
  2. On Configure stack options → Rollback configuration, enter a monitoring time (0–180 minutes) and add each alarm ARN.

AppConfig — feature flag with alarm rollback:

  1. Console → AWS AppConfig → Applications → Create application (hl-m09-accounts).
  2. Configuration profiles and feature flags → Create → Feature flag, name it accounts-flags, add a flag useNewBeneficiaryScreen (off).
  3. Environments → Create environment (prod), and under Monitors add the CloudWatch alarm ARN plus a role that lets AppConfig read alarms.
  4. Start deployment, choosing a gradual deployment strategy (for example AppConfig.Linear20PercentEvery6Minutes) ⚠ VERIFY the predefined strategy names in your Region.

AWS CLI

P=learn-sandbox

# 1. ECS: enable the circuit breaker with rollback and attach deployment alarms.
aws ecs update-service --cluster hl-m09-cluster --service hl-m09-accounts-api \
  --deployment-configuration '{
    "deploymentCircuitBreaker": {"enable": true, "rollback": true},
    "alarms": {"alarmNames": ["hl-m09-accounts-5xx-rate","hl-m09-accounts-p99"], "enable": true, "rollback": true},
    "maximumPercent": 200, "minimumHealthyPercent": 100
  }' --profile $P

# 2. ECS: see the state of the latest deployments, including rollbacks.
aws ecs describe-services --cluster hl-m09-cluster --services hl-m09-accounts-api \
  --query 'services[0].deployments[].[id,status,rolloutState,rolloutStateReason,taskDefinition]' \
  --output table --profile $P

# 3. ECS manual rollback: point the service back at the previous task-definition revision.
aws ecs update-service --cluster hl-m09-cluster --service hl-m09-accounts-api \
  --task-definition hl-m09-accounts-api:41 --profile $P

# 4. CloudFormation: update a stack with a rollback trigger watched for 15 minutes afterwards.
aws cloudformation update-stack --stack-name hl-m09-accounts \
  --use-previous-template --capabilities CAPABILITY_NAMED_IAM \
  --rollback-configuration "RollbackTriggers=[{Arn=arn:aws:cloudwatch:us-east-1:111122223333:alarm:hl-m09-accounts-5xx-rate,Type=AWS::CloudWatch::Alarm}],MonitoringTimeInMinutes=15" \
  --profile $P

# 5. CloudFormation: an update stuck in UPDATE_ROLLBACK_FAILED can be continued,
#    skipping resources you have fixed by hand.
aws cloudformation continue-update-rollback --stack-name hl-m09-accounts --profile $P

# 6. AppConfig: turn a feature flag off by deploying the previous configuration version.
aws appconfig start-deployment --application-id abc1234 --environment-id env5678 \
  --configuration-profile-id prof9012 --configuration-version 3 \
  --deployment-strategy-id AppConfig.AllAtOnce --profile $P

# 7. AppConfig: stop a configuration deployment in progress (rolls it back).
aws appconfig stop-deployment --application-id abc1234 --environment-id env5678 \
  --deployment-number 12 --profile $P

# 8. Aurora point-in-time restore (DISASTER RECOVERY, not release rollback):
#    restores to a NEW cluster; the original keeps running.
aws rds restore-db-cluster-to-point-in-time --db-cluster-identifier hl-m09-accounts-restore \
  --source-db-cluster-identifier hl-m09-accounts --restore-to-time 2026-11-18T09:55:00Z \
  --profile $P

The IDs in steps 6 and 7 (abc1234 and so on) are placeholders; get yours with aws appconfig list-applications and the related list commands.

Infrastructure as Code

AccountsServiceStack.cs (excerpt)
using Amazon.CDK;
using Amazon.CDK.AWS.AppConfig;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.ECS;
using Amazon.CDK.AWS.ECS.Patterns;
using Constructs;

namespace Harbourline.Accounts.Infra;

public sealed class AccountsServiceStack : Stack
{
    public AccountsServiceStack(Construct scope, string id, AccountsServiceProps props)
        : base(scope, id, props)
    {
        var service = new ApplicationLoadBalancedFargateService(this, "Api",
            new ApplicationLoadBalancedFargateServiceProps
            {
                Cluster = props.Cluster,
                ServiceName = "hl-m09-accounts-api",
                DesiredCount = 2,
                TaskImageOptions = new ApplicationLoadBalancedTaskImageOptions
                {
                    Image = ContainerImage.FromEcrRepository(props.Repository, props.ImageTag),
                    ContainerPort = 8080
                },
                // Rolling update with automatic rollback when tasks cannot become healthy.
                CircuitBreaker = new DeploymentCircuitBreaker { Enable = true, Rollback = true },
                MinHealthyPercent = 100
            });

        // A business-level symptom: 5xx responses from the target group.
        var fiveXx = new Alarm(this, "Api5xx", new AlarmProps
        {
            AlarmName = "hl-m09-accounts-5xx-rate",
            Metric = service.TargetGroup.Metrics.HttpCodeTarget(
                Amazon.CDK.AWS.ElasticLoadBalancingV2.HttpCodeTarget.TARGET_5XX_COUNT,
                new MetricOptions { Period = Duration.Minutes(1), Statistic = "Sum" }),
            Threshold = 5,
            EvaluationPeriods = 2,
            TreatMissingData = TreatMissingData.NOT_BREACHING
        });

        // Roll the ECS deployment back if the alarm fires while it is in progress.
        service.Service.EnableDeploymentAlarms(new[] { fiveXx.AlarmName },
            new DeploymentAlarmOptions { Behavior = AlarmBehavior.ROLLBACK_ON_ALARM });

        // Feature flags in AppConfig, deployed gradually and watched by the same alarm.
        var app = new Application(this, "AppConfigApp", new ApplicationProps
        {
            ApplicationName = "hl-m09-accounts"
        });
        var env = new Amazon.CDK.AWS.AppConfig.Environment(this, "Prod", new EnvironmentProps
        {
            Application = app,
            EnvironmentName = "prod",
            Monitors = new[] { Amazon.CDK.AWS.AppConfig.Monitor.FromCloudWatchAlarm(fiveXx) }
        });
        new HostedConfiguration(this, "Flags", new HostedConfigurationProps
        {
            Application = app,
            Name = "accounts-flags",
            Type = ConfigurationType.FEATURE_FLAGS,
            Content = ConfigurationContent.FromInlineJson("""
            {
              "version": "1",
              "flags":  { "useNewBeneficiaryScreen": { "name": "useNewBeneficiaryScreen" } },
              "values": { "useNewBeneficiaryScreen": { "enabled": false } }
            }
            """),
            DeployTo = new[] { env },
            DeploymentStrategy = new Amazon.CDK.AWS.AppConfig.DeploymentStrategy(this, "Gradual", new DeploymentStrategyProps
            {
                RolloutStrategy = RolloutStrategy.Linear(new RolloutStrategyProps
                {
                    GrowthFactor = 20,
                    DeploymentDuration = Duration.Minutes(10),
                    FinalBakeTime = Duration.Minutes(10)
                })
            })
        });
    }
}

public sealed class AccountsServiceProps : StackProps
{
    public required ICluster Cluster { get; init; }
    public required Amazon.CDK.AWS.ECR.IRepository Repository { get; init; }
    public required string ImageTag { get; init; }
}

The AppConfig L2 constructs shown (Application, Environment, HostedConfiguration, DeploymentStrategy, Monitor) are in Amazon.CDK.AWS.AppConfig in recent CDK v2 releases ⚠ VERIFY property names against your Amazon.CDK.Lib version. The feature-flag JSON format (version, flags, values) is the AppConfig AWS.AppConfig.FeatureFlags schema.

Complete code

Three pieces of runnable C#: a three-phase expand/contract migration in EF Core, a test that blocks destructive migrations outside a contract release, and an ASP.NET Core endpoint that reads an AppConfig feature flag.

1. Expand/contract in EF Core: renaming beneficiaries.name safely

src/Harbourline.Accounts.Data/Beneficiary.cs (release N and N+1)
namespace Harbourline.Accounts.Data;

public class Beneficiary
{
    public required string BeneficiaryId { get; init; }
    public required string CustomerId { get; init; }
    public required string Iban { get; set; }
    public DateTime CreatedAt { get; init; }

    // OLD column. Still written in N and N+1 so that release N-1 keeps working after a rollback.
    [Obsolete("Use DisplayName. Removed in the contract release (CHG-2481).")]
    public string? Name { get; set; }

    // NEW column. Nullable during expand; becomes required in the contract release.
    public string? DisplayName { get; set; }

    /// <summary>Single place that writes both columns during the transition.</summary>
    public void SetDisplayName(string value)
    {
        DisplayName = value;
#pragma warning disable CS0618 // deliberate dual write
        Name = value;
#pragma warning restore CS0618
    }

    /// <summary>Release N reads old-then-new; release N+1 reads new-then-old.</summary>
    public string EffectiveDisplayName =>
#pragma warning disable CS0618
        DisplayName ?? Name ?? string.Empty;
#pragma warning restore CS0618
}
src/Harbourline.Accounts.Data/Migrations/20261118090000_ExpandBeneficiaryDisplayName.cs
using Microsoft.EntityFrameworkCore.Migrations;

namespace Harbourline.Accounts.Data.Migrations;

/// <summary>EXPAND (release N): add the new column. Old code ignores it; new code dual-writes.</summary>
public partial class ExpandBeneficiaryDisplayName : Migration
{
    protected override void Up(MigrationBuilder migrationBuilder)
    {
        migrationBuilder.AddColumn<string>(
            name: "display_name",
            table: "beneficiaries",
            type: "character varying(140)",
            maxLength: 140,
            nullable: true);   // nullable: rows written by old code are still valid
    }

    protected override void Down(MigrationBuilder migrationBuilder)
    {
        // Safe to reverse: nothing depends on display_name yet except release N itself.
        migrationBuilder.DropColumn(name: "display_name", table: "beneficiaries");
    }
}
ops/backfill-beneficiary-display-name.sql (run between N and N+1, repeatable)
-- Backfill in small batches so locks stay short and replication lag stays low.
-- Re-run until it reports 0 rows. Safe to run while both releases are live.
WITH batch AS (
  SELECT beneficiary_id
  FROM beneficiaries
  WHERE display_name IS NULL AND name IS NOT NULL
  LIMIT 5000
  FOR UPDATE SKIP LOCKED
)
UPDATE beneficiaries b
SET display_name = b.name
FROM batch
WHERE b.beneficiary_id = batch.beneficiary_id;
src/Harbourline.Accounts.Data/Migrations/20261209090000_ContractBeneficiaryName.cs
using Microsoft.EntityFrameworkCore.Migrations;

namespace Harbourline.Accounts.Data.Migrations;

/// <summary>
/// CONTRACT (release N+2): only after N+1 has been stable and no release older than
/// N+1 can be rolled back to. Marked so the safety test allows the destructive operation.
/// </summary>
[ContractPhase("CHG-2481")]
public partial class ContractBeneficiaryName : Migration
{
    protected override void Up(MigrationBuilder migrationBuilder)
    {
        migrationBuilder.AlterColumn<string>(
            name: "display_name", table: "beneficiaries",
            type: "character varying(140)", maxLength: 140, nullable: false,
            oldClrType: typeof(string), oldType: "character varying(140)",
            oldMaxLength: 140, oldNullable: true);

        migrationBuilder.DropColumn(name: "name", table: "beneficiaries");
    }

    protected override void Down(MigrationBuilder migrationBuilder)
    {
        // Re-creating the column restores the SHAPE, not lost data: copy it back explicitly.
        migrationBuilder.AddColumn<string>(
            name: "name", table: "beneficiaries",
            type: "character varying(140)", maxLength: 140, nullable: true);
        migrationBuilder.Sql("UPDATE beneficiaries SET name = display_name;");
        migrationBuilder.AlterColumn<string>(
            name: "display_name", table: "beneficiaries",
            type: "character varying(140)", maxLength: 140, nullable: true,
            oldClrType: typeof(string), oldType: "character varying(140)",
            oldMaxLength: 140, oldNullable: false);
    }
}

2. The migration safety test

src/Harbourline.Accounts.Data/ContractPhaseAttribute.cs
namespace Harbourline.Accounts.Data;

/// <summary>
/// Marks a migration as a deliberate CONTRACT step. The change reference links to the
/// approved change record that confirmed no older release can be rolled back to.
/// </summary>
[AttributeUsage(AttributeTargets.Class, AllowMultiple = false)]
public sealed class ContractPhaseAttribute(string changeReference) : Attribute
{
    public string ChangeReference { get; } = changeReference;
}
tests/Harbourline.Accounts.Data.Tests/MigrationSafetyTests.cs
using System.Reflection;
using Harbourline.Accounts.Data;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Infrastructure;
using Microsoft.EntityFrameworkCore.Migrations;
using Microsoft.EntityFrameworkCore.Migrations.Operations;
using Xunit;

public class MigrationSafetyTests
{
    // Operations that can break the PREVIOUS release if it is rolled back to.
    private static readonly Type[] Destructive =
    [
        typeof(DropColumnOperation), typeof(DropTableOperation), typeof(RenameColumnOperation),
        typeof(RenameTableOperation), typeof(AlterColumnOperation), typeof(DropSchemaOperation)
    ];

    [Fact]
    public void Destructive_operations_only_appear_in_contract_phase_migrations()
    {
        // No database connection is opened: we only build the model and read migrations.
        var options = new DbContextOptionsBuilder<AccountsDbContext>()
            .UseNpgsql("Host=unused;Database=unused")
            .Options;
        using var db = new AccountsDbContext(options);

        var assembly = db.GetService<IMigrationsAssembly>();
        var provider = db.Database.ProviderName!;
        var violations = new List<string>();

        foreach (var (id, typeInfo) in assembly.Migrations)
        {
            if (typeInfo.GetCustomAttribute<ContractPhaseAttribute>() is { } contract)
            {
                Assert.Matches(@"^CHG-\d+$", contract.ChangeReference);
                continue; // reviewed and approved as a contract step
            }

            var migration = assembly.CreateMigration(typeInfo, provider);
            foreach (var op in migration.UpOperations)
            {
                if (Destructive.Contains(op.GetType()))
                    violations.Add($"{id}: {op.GetType().Name}");
            }
        }

        Assert.True(violations.Count == 0,
            "Destructive operations outside a [ContractPhase] migration:\n" + string.Join("\n", violations));
    }
}

3. Reading a feature flag from AppConfig in ASP.NET Core

src/Harbourline.Accounts.Api/Program.cs (excerpt)
using Harbourline.Accounts.Data;

var builder = WebApplication.CreateBuilder(args);

// Load the "accounts-flags" feature-flag profile from AWS AppConfig and refresh it
// every 30 seconds. The task role needs appconfig:StartConfigurationSession and
// appconfig:GetLatestConfiguration on this application only.
builder.Configuration.AddAppConfig(
    applicationId: builder.Configuration["AppConfig:ApplicationId"]!,
    environmentId: builder.Configuration["AppConfig:EnvironmentId"]!,
    configProfileId: builder.Configuration["AppConfig:FlagsProfileId"]!,
    reloadAfter: TimeSpan.FromSeconds(30));

var app = builder.Build();

app.MapGet("/customers/{customerId}/beneficiaries", async (
        string customerId, IConfiguration config, IBeneficiaryReader reader, CancellationToken ct) =>
    {
        // Feature flags arrive as { "useNewBeneficiaryScreen": { "enabled": true } },
        // which the configuration provider flattens to "useNewBeneficiaryScreen:enabled".
        var useNew = config.GetValue<bool>("useNewBeneficiaryScreen:enabled");

        var items = useNew
            ? await reader.ListWithDisplayNameAsync(customerId, ct)   // new code path
            : await reader.ListLegacyAsync(customerId, ct);           // known-good path
        return Results.Ok(items);
    });

app.Run();

AddAppConfig comes from the Amazon.Extensions.Configuration.SystemsManager package (7.1.1). IBeneficiaryReader is your existing data-access interface; the point is that both code paths ship in the same release and the flag chooses between them at runtime.

Code explained line by line

Beneficiary.cs

  1. [Obsolete(...)] public string? Name — the compiler now warns anyone who touches the old column, and the message names the change that will remove it.
  2. public string? DisplayName — nullable while rows written by the old code exist.
  3. SetDisplayName — the only place that writes; it writes both columns. Dual writes are what make rollback to the previous release safe: the old release reads name, which is still current.
  4. EffectiveDisplayName — reads new-then-old, so rows not yet backfilled still display correctly.

ExpandBeneficiaryDisplayName

  1. AddColumn ... nullable: true — adding a nullable column without a default is a fast, metadata-only change in PostgreSQL, so it does not lock the table for long.
  2. Down drops the column — acceptable here, because no released code depends on it before release N.

Backfill SQL

  1. LIMIT 5000 ... FOR UPDATE SKIP LOCKED — small batches, skipping rows another session holds, so it never blocks live traffic for long.
  2. WHERE display_name IS NULL AND name IS NOT NULL — idempotent: re-running only touches rows still to do.

ContractBeneficiaryName

  1. [ContractPhase("CHG-2481")] — the marker the safety test looks for. The reference points to a change record confirming that no release older than N+1 is a rollback target.
  2. AlterColumn ... nullable: false then DropColumn("name") — the destructive part, deliberately isolated in its own release.
  3. Down recreates name and copies data back from display_name. A down migration that only re-adds a column would restore the shape but lose the values; that is how "rollbacks" silently corrupt data.

MigrationSafetyTests

  1. Destructive — the operation types that can break the previous release. AlterColumnOperation is included because type and nullability changes can break old code too.
  2. UseNpgsql("Host=unused...") — builds the model without connecting; the test runs in CI with no database.
  3. db.GetService<IMigrationsAssembly>() — EF Core's internal service that knows every migration in the assembly.
  4. assembly.CreateMigration(typeInfo, provider) then migration.UpOperations — EF Core builds the list of operations the Up method would run, without executing SQL.
  5. The ContractPhase branch checks the change reference format, so nobody can wave a migration through with an empty string.
  6. The final assertion prints every violation, which makes the failure message a review checklist.

Program.cs

  1. AddAppConfig(...) — adds AppConfig as a configuration source; values refresh on the interval you set.
  2. config.GetValue<bool>("useNewBeneficiaryScreen:enabled") — reads the flag through normal IConfiguration, so the code has no AWS-specific feature-flag dependency.
  3. Both branches exist in the same build. Turning the flag off in AppConfig is a rollback that needs no deployment, no image and no database change.

Security considerations

⚠ Learning shortcut — not for production

In this lesson the migration bundle runs with the same database credentials as the application. In production, run migrations with a separate database role that has DDL rights, retrieved from Secrets Manager only by the migration task's IAM role. The application's role should have only DML rights (select, insert, update), so a compromised application cannot drop tables.

⚠ Learning shortcut — not for production

Anyone who can deploy an AppConfig configuration version can switch production behaviour without a code review. In production, restrict appconfig:StartDeployment on the prod environment to the pipeline role, require flag changes to go through a pull request on the flag JSON, and alert on manual deployments via CloudTrail.

  • Rollback is a production change. A manual rollback still needs to be recorded: who, when, why, which version. Use the same pipeline (run with the previous artefact) rather than console clicks, so the record is automatic.
  • Previous images must still exist. An ECR lifecycle policy that keeps only the last 3 images can delete your rollback target after a busy week. Keep at least the images currently deployed in any environment plus a margin, and make images immutable (tag immutability on).
  • Previous images may have known vulnerabilities. Rolling back to a build with a critical CVE that was fixed in the new release is a risk decision; the runbook should say who can accept it.
  • Data protection. A point-in-time restore creates a full copy of customer data in a new cluster. It needs the same encryption, network isolation and deletion plan as production; under GDPR / UK GDPR, an untracked copy of personal data is a problem in itself.

Banking use case

Harbourline's platform team publishes a release classification that every service team uses. A release's class determines the rollback approach and the approval route.

ClassContentsBack-outApproval
A — code onlyImage or Lambda version change, no schema, no contract changeAutomatic (circuit breaker, alarms, CodeDeploy)Standard change; pipeline evidence is enough
B — code + flagsNew behaviour shipped off, enabled via AppConfigFlag off (automatic on alarm) then code rollback if neededStandard change for deploy; flag enablement recorded separately
C — expandAdditive schema change + dual-writing codeCode rollback safe; schema staysStandard change if the migration safety test passes
D — contractDestructive schema or breaking contract changeRoll forward onlyNormal change, reviewed by the database owner and service owner; scheduled outside payment cut-off windows

The incident from the opening would have been a class D change disguised as class A. With the safety test in place, the rename migration would have failed CI because it was not marked [ContractPhase], and the reviewer would have asked for the expand/contract split.

Common mistakes

  • Treating "redeploy the previous build" as the whole back-out plan → data and schema don't follow the code → classify every release and plan per class.
  • Applying EF Core migrations at application start-up → race conditions with many tasks, and the app's role needs DDL rights → run a migration bundle as a separate task.
  • Writing Down migrations that only restore structure → rolling back silently loses data → either copy data back explicitly or declare the migration irreversible.
  • Circuit breaker without alarms → healthy-looking tasks returning wrong answers are never rolled back → add business alarms as deployment alarms.
  • Alarms already in ALARM before the deployment → immediate rollback of a good release, or alarms ignored → check alarm states as a pipeline step before deploying.
  • Feature flags that never get removed → two code paths forever, both needing tests → give every flag an owner and a removal date.
  • Using point-in-time restore as a rollback → loses every legitimate transaction since the restore point → keep PITR for disasters and corruption, not releases.

Troubleshooting

SymptomLikely causeFix
ECS deployment shows rolloutState: FAILED with "circuit breaker" reasonNew tasks failed to start or failed health checks repeatedlyCheck stopped-task reasons (describe-tasks) and container logs; the service is already back on the previous task definition
ECS deployment rolled back but the app looked fineA deployment alarm was in ALARM from an earlier, unrelated eventCheck alarm history; add a pre-deploy "all alarms OK" check
CloudFormation stack in UPDATE_ROLLBACK_FAILEDA resource couldn't return to its old state (deleted outside CloudFormation, or a dependency changed)Fix the resource, then continue-update-rollback, skipping resources if needed
CloudFormation rolled back 10 minutes after "success"A rollback trigger alarm fired during the monitoring periodExpected behaviour; investigate the alarm. Reduce the period only if the alarm is noisy
AppConfig deployment status ROLLED_BACKA monitor alarm went into ALARM during deployment or bakeCheck the alarm; fix the flag or the code, then redeploy
Old release crashes after rollback with "column does not exist"A destructive migration shipped with the new releaseRoll forward; add the migration safety test so it cannot happen again
MigrationSafetyTests fails on an old migrationThe test was added after destructive migrations already existedMark historical ones with [ContractPhase("CHG-legacy")] after review, or start the check from a baseline migration ID

Cost and cleanup

Cost: everything in the C# section runs locally and in CI for free. The ECS, CloudFormation and AppConfig settings themselves cost nothing extra. AppConfig charges per configuration request and per configuration received ⚠ VERIFY current AppConfig pricing; a lab with one task polling every 30 seconds costs a few cents per month at most. If you try the ECS steps, the Fargate tasks and load balancer are billed hourly — deploy, test and destroy within an hour. Aurora point-in-time restore creates a new, billed cluster: only try it in a deploy-and-destroy session, and delete the restored cluster immediately. Label: free for the core exercises.

P=learn-sandbox
# Delete the AppConfig resources (environments and profiles first, then the application).
APP=$(aws appconfig list-applications --query "Items[?Name=='hl-m09-accounts'].Id" --output text --profile $P)
for e in $(aws appconfig list-environments --application-id $APP --query 'Items[].Id' --output text --profile $P); do
  aws appconfig delete-environment --application-id $APP --environment-id $e --profile $P; done
for c in $(aws appconfig list-configuration-profiles --application-id $APP --query 'Items[].Id' --output text --profile $P); do
  for v in $(aws appconfig list-hosted-configuration-versions --application-id $APP --configuration-profile-id $c --query 'Items[].VersionNumber' --output text --profile $P); do
    aws appconfig delete-hosted-configuration-version --application-id $APP --configuration-profile-id $c --version-number $v --profile $P; done
  aws appconfig delete-configuration-profile --application-id $APP --configuration-profile-id $c --profile $P; done
aws appconfig delete-application --application-id $APP --profile $P

# If you deployed the CDK stack: destroy it.
cdk destroy --profile $P

# If you restored an Aurora cluster: delete its instances first, then the cluster, without a final snapshot.
aws rds delete-db-cluster --db-cluster-identifier hl-m09-accounts-restore --skip-final-snapshot --profile $P

# Confirm nothing tagged for this lesson remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m09-l9 \
  --query 'ResourceTagMappingList[].ResourceARN' --profile $P

Exercises

  • 🟢 Guided: Add ContractPhaseAttribute and MigrationSafetyTests to a solution of your own with at least two EF Core migrations. Write a third migration that renames a column, run the tests and watch them fail. Rewrite it as an expand step and watch them pass.
  • 🟡 Challenge: Design the expand/contract sequence for changing transactions.amount from numeric(18,2) in pence-free pounds to a pair amount_minor bigint + currency_exponent smallint. List each release, what the code reads and writes, the backfill, and the exact point after which code rollback to the original release is no longer allowed.
  • 🔴 Live problem: At 10:20 a release of the Payments Service goes out. At 10:35 the operations team reports that 2% of SEPA payments are being sent with the wrong end-to-end ID format and some have already been accepted by the clearing system. The ECS deployment succeeded and no alarms fired. Write the first 30 minutes of the incident response: rollback or roll-forward, what to do about the payments already sent, and what you change in the pipeline afterwards.

Assignment

Write a ROLLBACK.md for one of your portfolio services containing: the release classes table adapted to your service; the decision flow diagram (Mermaid); the automatic rollback settings with their alarms; one rehearsal record (date, what was broken, time to recovery); and the migration safety test running in CI.

Solutions

Knowledge check

Interview questions

  • "What is your rollback strategy?" — It depends on what changed. Code and configuration roll back automatically on alarms; schema changes are expand/contract so code rollback is always safe; destructive schema and contract changes are roll-forward only and go through normal change approval. External side effects (payments sent) are handled by business processes, not by the pipeline.
  • "How do you run EF Core migrations in a pipeline?" — Build a migration bundle in CI, run it as a one-off task with a DDL-capable role before deploying the service, keep migrations additive in normal releases, and enforce that with a test. Never migrate at app start-up in production.
  • "Roll back or roll forward?" — Roll back when the previous state is known-good and reachable without data loss, which should be the default for class A/B/C releases. Roll forward when the release crossed a one-way door (destructive schema, contract change, external effects) or when the fix is trivial and well-tested — and always under incident change control.
  • "How do feature flags interact with audit?" — A flag flip changes production behaviour, so it needs the same controls as a deployment: who changed it, when, approved by whom, and visible in CloudTrail and AppConfig deployment history.

Certification practice

Completion checklist

  • I can classify a release's contents by whether each part can roll back
  • I configured (or can configure) ECS circuit breaker and deployment alarms with rollback
  • I can explain CloudFormation rollback triggers and their limits
  • I wrote an expand/contract plan for a schema change
  • The migration safety test runs in my CI and blocks unmarked destructive migrations
  • I removed any AppConfig, ECS or restored database resources from the sandbox

Key takeaways

  • Code and configuration roll back cheaply and should do so automatically on alarms; data, schema, contracts and external side effects do not.
  • Expand/contract keeps every schema change compatible with the previous release, so code rollback is always safe.
  • Feature flags separate deploying from releasing and give you the fastest, deployment-free back-out.
  • A circuit breaker sees crashes; deployment alarms see symptoms. Use both.
  • Point-in-time restore is for disasters, not releases. Destructive changes are roll-forward only and go through normal change approval.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page