Skip to Content
Cloud Engineering Academy
9. DevOps & CI/CD

Environment separation: one AWS account per environment

Separate dev, test, staging and prod into their own AWS accounts under AWS Organizations, deploy into each with its own OIDC role from GitHub Actions, promote the same image digest from one environment to the next, and add guardrails so a pipeline can never deploy to the wrong account.

intermediate⏱ 1 h 50 min🟢 FreeDOP-C02SCS-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SecurityToken 4.x) · GitHub Actions (aws-actions/configure-aws-credentials v6.3.0) · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain why banks separate environments by AWS account, not by naming convention, VPC or tag.
  • Design an AWS Organizations layout for Harbourline's delivery pipeline: workload accounts per environment, a shared tooling account, and security accounts.
  • Choose between two cross-account deploy patterns — OIDC directly into each account and a central tooling role that chains into target roles — and explain the trade-offs.
  • Build once, promote many: move the same image digest from test to staging to prod, with a cross-account ECR policy.
  • Add guardrails: OIDC trust policies bound to GitHub environments, an SCP that limits who may deploy in prod, and a C# pre-flight check that refuses to deploy to the wrong account.

Prerequisites

Why this matters in banking

Harbourline's first cloud team put dev, test and prod for the Accounts API in one AWS account, separated by name prefixes: dev-accounts-api, test-accounts-api, prod-accounts-api. It looked tidy. Then:

  • A developer testing a clean-up script ran aws ecs delete-service against a list built with a *accounts-api* filter. The prod service was in the list.
  • A load test in "test" hit the account-level Lambda concurrency quota and throttled the prod payment validator, because quotas are per account and Region.
  • Internal audit asked "who can change production?" The honest answer was "everyone with access to the account", because IAM policies with name-based conditions had gaps nobody could prove were closed.
  • The cost report could not split prod cost from experiments without perfect tagging, which nobody had.

An AWS account is the strongest isolation boundary AWS offers: separate IAM, separate quotas, separate billing line, separate resource namespace, and it is the unit that SCPs, CloudTrail organisation trails and AWS Config aggregators work with. For a bank, account separation is also how you show segregation of duties concretely: developers have broad access in dev, read-only access in prod, and the only thing that can change prod is the pipeline's prod role, after approval.

The idea in plain language

Instead of one AWS account with everything inside it, you have one account per environment for each workload (or group of workloads), plus a few shared accounts. AWS Organizations groups them, applies guardrails to whole groups at once (SCPs) and pays one combined bill.

The pipeline then needs a way to deploy into several accounts. The modern way is no stored keys anywhere: GitHub Actions presents a short-lived OIDC token, and each environment's AWS account has its own IAM role that trusts only tokens for that repository and that GitHub environment. A job running in the test environment can only become the test role; a job running in prod — which only starts after a required reviewer approves — can only become the prod role.

What moves between environments is the artefact, not the build. You build the container image once, record its digest (sha256:…, an unchangeable fingerprint of its content), and deploy exactly that digest to test, then staging, then prod. If prod runs something, it is byte-for-byte what test ran.

Analogy: a bank does not keep the training branch, the trial branch and the real branch in one building with labels on the doors. They are separate buildings with separate keys. A courier (the pipeline) carries the same sealed bag (the image digest) from one building to the next, and at each door shows a pass that only works for that door, issued that day, and — for the real branch — counter-signed by a manager.

What problem the AWS services solve

Service / featureWhat it does for environment separationLimits and sharp edges
AWS OrganizationsGroups accounts into OUs, applies SCPs and RCPs, consolidates billingSCPs never grant permissions; they only cap them. They do not apply to the management account
AWS Control TowerSets up a landing zone: log archive and audit accounts, guardrails (controls), Account FactoryOpinionated; uses AWS Config and CloudTrail, which have their own cost
IAM OIDC identity provider (per account)Lets GitHub's OIDC tokens be exchanged for role credentials in that accountOne provider per issuer URL per account; the trust policy, not the provider, is what restricts who can assume a role
STS role chainingA tooling-account role assumes a role in a target accountChained sessions are limited to one hour; every hop needs both a trust policy and an sts:AssumeRole permission
ECR repository policies and replicationLet workload accounts pull images from a shared repository, or replicate images into each accountPull access is granted to the account's principals; the ECS task execution role still needs ecr:GetAuthorizationToken in its own account
CDK bootstrap --trustLets a pipeline account deploy CDK stacks into a target accountThe --cloudformation-execution-policies you choose decide what CloudFormation may do in that account

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Subscription per environmentAWS account per environmentAn AWS account is also the IAM, quota and namespace boundary; there is no tenant-wide RBAC across accounts
Management groupsOrganizational units (OUs)OUs nest up to five levels below the root
Azure Policy (deny effect)Service control policies (SCPs)SCPs restrict IAM actions; for resource configuration checks you add AWS Config rules or Control Tower controls
Azure Landing ZonesAWS Control Tower landing zoneSimilar idea; Control Tower creates log archive and audit accounts by default
Service connection per subscription (workload identity federation)OIDC provider + IAM role per accountSame federation idea; the AWS trust policy can bind to a GitHub environment through the sub claim
ACR shared across subscriptionsECR in a shared account + repository policy, or ECR cross-account replicationPull permission is in a resource policy on the repository
Azure DevOps environment approvalsGitHub environment required reviewersApproval gates which job can request the OIDC token for that environment

Architecture

Promotion flow

Two cross-account patterns

A. OIDC directly into each accountB. Tooling account + role chaining
HowEach workload account has the GitHub OIDC provider and a deploy role trusting one repo + environmentOnly the tooling account trusts GitHub; its role assumes hl-deploy-* roles in workload accounts
StrengthSimplest trust chain; the GitHub environment approval maps directly to the prod roleOne place to manage GitHub trust; works the same for CodePipeline
WeaknessThe OIDC provider and roles must be created in every account (automate with StackSets or Terraform)Chained sessions are capped at one hour; two policies to get right per hop; a compromise of the tooling role reaches every account it can chain into
Use whenGitHub Actions is your orchestrator (this module's default)CodePipeline in a tooling account, or a platform team that wants central control

Step-by-step implementation

Agree the account map

List environments and accounts and write them into the repository (for example deploy/environments.json): name, account ID, Region, OU, purpose, data classification. This file is what the pre-flight check in the code section reads. In this course you have one sandbox account, so treat the multi-account parts as a design exercise and simulate them with the sandbox.

Create one deploy role per environment

In each workload account, create hl-deploy-<env> with a trust policy that allows sts:AssumeRoleWithWebIdentity from the GitHub OIDC provider only when aud is sts.amazonaws.com and sub is exactly repo:harbourline/accounts-api:environment:<env>. Give it only the permissions that environment's deployment needs.

Create matching GitHub environments

In the repository: Settings → Environments → test, staging, prod. For prod, add required reviewers, turn on Prevent self-review, and restrict deployment branches to main. Store each environment's role ARN as an environment variable (DEPLOY_ROLE_ARN) — role ARNs are identifiers, not secrets.

Share images from one registry

Keep ECR in the shared-services account. Add a repository policy allowing the workload accounts to pull. Turn on tag immutability. Deploy by digest (repo@sha256:…), never by a moving tag such as latest.

Add organisation guardrails

Attach an SCP to the Prod OU that denies changing ECS services, Lambda aliases and CloudFormation stacks to every principal except the deploy role and the break-glass role. Attach a Region-restriction SCP to all workload OUs. Use RCPs or resource policies to stop prod data stores being reachable from non-prod principals.

Add a pre-flight check to every deploy job

Before any deployment command, the job calls sts:GetCallerIdentity and compares the account ID with the account map entry for its environment. If they differ, the job stops. This catches a mis-copied role ARN before it does damage.

Keep production data out of lower environments

Test and staging use synthetic data from the dataset generator (seed 20261117). Any exception — for example, a masked extract for a performance test — needs a data-protection review under GDPR / UK GDPR and a documented deletion date.

AWS Console instructions

Organizations:

  1. Sign in to the management account → Console → AWS Organizations → AWS accounts.
  2. Actions → Create new organizational unit under Root: Security, Infrastructure, Workloads, Sandbox; under Workloads, create NonProd and Prod.
  3. Add an AWS account → Create an AWS account for each environment (requires a unique email per account — use plus-addressing on a mailbox you control).
  4. Policies → Service control policies → Create policy, paste the SCP below, then Targets → Attach to the Prod OU.

OIDC provider and role in a workload account:

  1. Console → IAM → Identity providers → Add provider → OpenID Connect. Provider URL https://token.actions.githubusercontent.com, audience sts.amazonaws.com.
  2. IAM → Roles → Create role → Web identity, choose the provider and audience, then on the trust policy page add the sub condition for the repository and environment. Name it hl-deploy-prod.

GitHub environment:

  1. GitHub → repository → Settings → Environments → New environment → prod.
  2. Tick Required reviewers (up to six people or teams), tick Prevent self-review, and under Deployment branches and tags choose Selected branches and tags → main.
  3. Add the variable DEPLOY_ROLE_ARN with the prod role ARN.

AWS CLI

P=learn-sandbox

# 1. Organizations: show the OU tree and which accounts live where (run in the management account).
ROOT=$(aws organizations list-roots --query 'Roots[0].Id' --output text --profile mgmt)
aws organizations list-organizational-units-for-parent --parent-id $ROOT --profile mgmt
aws organizations list-accounts --query 'Accounts[].[Id,Name,Status]' --output table --profile mgmt

# 2. Create the GitHub OIDC provider in a workload account (once per account).
aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com --profile $P

# 3. Create the per-environment deploy role with the trust policy below.
aws iam create-role --role-name hl-deploy-test \
  --assume-role-policy-document file://trust-test.json \
  --max-session-duration 3600 \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m09-l10 Key=Environment,Value=learn --profile $P

# 4. Prove the guard works: simulate whether the role's permissions allow a prod-style action.
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::111122223333:role/hl-deploy-test \
  --action-names ecs:UpdateService iam:CreateUser --profile $P

# 5. ECR in the shared account: allow workload accounts to pull, and make tags immutable.
aws ecr put-image-tag-mutability --repository-name hl-accounts-api \
  --image-tag-mutability IMMUTABLE --profile shared
aws ecr set-repository-policy --repository-name hl-accounts-api \
  --policy-text file://ecr-pull-policy.json --profile shared

# 6. Resolve a tag to its digest once, then deploy the digest everywhere.
aws ecr describe-images --repository-name hl-accounts-api --image-ids imageTag=4.13.0 \
  --query 'imageDetails[0].imageDigest' --output text --profile shared

# 7. CDK: let the tooling/pipeline account deploy into a target account,
#    with a narrow execution policy instead of AdministratorAccess.
cdk bootstrap aws://444455556666/us-east-1 \
  --trust 777788889999 \
  --cloudformation-execution-policies arn:aws:iam::444455556666:policy/hl-cfn-exec-accounts \
  --profile prod-admin

The trust policy for hl-deploy-test (file trust-test.json):

trust-test.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GitHubTestEnvironmentOnly",
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:harbourline/accounts-api:environment:test"
        }
      }
    }
  ]
}

The cross-account ECR pull policy (file ecr-pull-policy.json):

ecr-pull-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WorkloadAccountsPull",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::111122223333:root",
          "arn:aws:iam::222233334444:root",
          "arn:aws:iam::444455556666:root"
        ]
      },
      "Action": [
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchCheckLayerAvailability"
      ]
    }
  ]
}

arn:aws:iam::<account>:root in a resource policy means "this account may grant the permission to its own principals"; each workload account still decides, through IAM, which of its roles (the ECS task execution role) actually gets to pull.

Infrastructure as Code

deploy_roles.tf — applied once per workload account
variable "environment" { type = string }          # "test", "staging" or "prod"
variable "github_repo" { type = string }          # "harbourline/accounts-api"

resource "aws_iam_openid_connect_provider" "github" {
  url            = "https://token.actions.githubusercontent.com"
  client_id_list = ["sts.amazonaws.com"]
}

data "aws_iam_policy_document" "trust" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }
    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:sub"
      values   = ["repo:${var.github_repo}:environment:${var.environment}"]
    }
  }
}

resource "aws_iam_role" "deploy" {
  name                 = "hl-deploy-${var.environment}"
  assume_role_policy   = data.aws_iam_policy_document.trust.json
  max_session_duration = 3600
  tags = {
    Project     = "cea-lab"
    Lab         = "m09-l10"
    Environment = var.environment
  }
}

# Only what an ECS deploy of this service needs, in this account.
data "aws_iam_policy_document" "deploy" {
  statement {
    sid       = "UpdateThisService"
    actions   = ["ecs:UpdateService", "ecs:DescribeServices"]
    resources = ["arn:aws:ecs:us-east-1:${data.aws_caller_identity.me.account_id}:service/hl-${var.environment}-cluster/hl-accounts-api"]
  }
  statement {
    sid       = "RegisterTaskDefinitions"
    actions   = ["ecs:RegisterTaskDefinition", "ecs:DescribeTaskDefinition"]
    resources = ["*"]   # these actions do not support resource-level permissions
  }
  statement {
    sid       = "PassOnlyTheTaskRoles"
    actions   = ["iam:PassRole"]
    resources = [
      "arn:aws:iam::${data.aws_caller_identity.me.account_id}:role/hl-accounts-api-task",
      "arn:aws:iam::${data.aws_caller_identity.me.account_id}:role/hl-accounts-api-exec"
    ]
    condition {
      test     = "StringEquals"
      variable = "iam:PassedToService"
      values   = ["ecs-tasks.amazonaws.com"]
    }
  }
}

data "aws_caller_identity" "me" {}

resource "aws_iam_role_policy" "deploy" {
  role   = aws_iam_role.deploy.id
  policy = data.aws_iam_policy_document.deploy.json
}

Apply this module to every workload account with a provider alias per account, or roll it out with CloudFormation StackSets from the management account. Either way, the role definition lives in Git and is reviewed like any other code.

Complete code

1. The account map and a C# pre-flight guard

deploy/environments.json
{
  "environments": {
    "test":    { "accountId": "111122223333", "region": "us-east-1", "dataClass": "synthetic" },
    "staging": { "accountId": "222233334444", "region": "us-east-1", "dataClass": "synthetic" },
    "prod":    { "accountId": "444455556666", "region": "us-east-1", "dataClass": "customer" }
  }
}
tools/DeployGuard/Program.cs
using System.Text.Json;
using Amazon.SecurityToken;
using Amazon.SecurityToken.Model;

namespace Harbourline.Platform.DeployGuard;

public sealed record EnvironmentEntry(string AccountId, string Region, string DataClass);
public sealed record EnvironmentMap(Dictionary<string, EnvironmentEntry> Environments);

public static class Guard
{
    /// <summary>
    /// Returns null when it is safe to deploy, or a human-readable reason to stop.
    /// </summary>
    public static async Task<string?> CheckAsync(
        IAmazonSecurityTokenService sts, EnvironmentMap map, string environment, string region,
        CancellationToken ct = default)
    {
        if (!map.Environments.TryGetValue(environment, out var expected))
            return $"Unknown environment '{environment}'. Known: {string.Join(", ", map.Environments.Keys)}.";

        if (!string.Equals(expected.Region, region, StringComparison.Ordinal))
            return $"Region mismatch: '{environment}' deploys to {expected.Region}, job is configured for {region}.";

        var me = await sts.GetCallerIdentityAsync(new GetCallerIdentityRequest(), ct);

        if (!string.Equals(me.Account, expected.AccountId, StringComparison.Ordinal))
            return $"Account mismatch: '{environment}' is {expected.AccountId}, credentials are for {me.Account}.";

        // The assumed-role ARN looks like arn:aws:sts::<acct>:assumed-role/<role>/<session>.
        var expectedRole = $":assumed-role/hl-deploy-{environment}/";
        if (!me.Arn.Contains(expectedRole, StringComparison.Ordinal))
            return $"Role mismatch: expected hl-deploy-{environment}, got {me.Arn}.";

        return null;
    }
}

public static class Program
{
    public static async Task<int> Main(string[] args)
    {
        if (args.Length != 2)
        {
            Console.Error.WriteLine("Usage: DeployGuard <environments.json> <environment>");
            return 2;
        }

        var map = JsonSerializer.Deserialize<EnvironmentMap>(
            await File.ReadAllTextAsync(args[0]),
            new JsonSerializerOptions { PropertyNameCaseInsensitive = true })
            ?? throw new InvalidOperationException("Empty environment map.");

        var region = Environment.GetEnvironmentVariable("AWS_REGION") ?? "unset";
        using var sts = new AmazonSecurityTokenServiceClient();

        var problem = await Guard.CheckAsync(sts, map, args[1], region);
        if (problem is not null)
        {
            // GitHub Actions turns this line into a red annotation on the run.
            Console.WriteLine($"::error title=Deploy guard::{problem}");
            return 1;
        }

        Console.WriteLine($"Deploy guard passed for '{args[1]}'.");
        return 0;
    }
}
tests/DeployGuard.Tests/GuardTests.cs
using Amazon.SecurityToken;
using Amazon.SecurityToken.Model;
using Harbourline.Platform.DeployGuard;
using NSubstitute;
using Xunit;

public class GuardTests
{
    private static readonly EnvironmentMap Map = new(new()
    {
        ["test"] = new("111122223333", "us-east-1", "synthetic"),
        ["prod"] = new("444455556666", "us-east-1", "customer")
    });

    private static IAmazonSecurityTokenService StsReturning(string account, string role)
    {
        var sts = Substitute.For<IAmazonSecurityTokenService>();
        sts.GetCallerIdentityAsync(Arg.Any<GetCallerIdentityRequest>(), Arg.Any<CancellationToken>())
           .Returns(new GetCallerIdentityResponse
           {
               Account = account,
               Arn = $"arn:aws:sts::{account}:assumed-role/{role}/GitHubActions"
           });
        return sts;
    }

    [Fact]
    public async Task Passes_when_account_region_and_role_match()
    {
        var result = await Guard.CheckAsync(StsReturning("444455556666", "hl-deploy-prod"), Map, "prod", "us-east-1");
        Assert.Null(result);
    }

    [Fact]
    public async Task Stops_prod_deploy_with_test_credentials()
    {
        var result = await Guard.CheckAsync(StsReturning("111122223333", "hl-deploy-test"), Map, "prod", "us-east-1");
        Assert.Contains("Account mismatch", result);
    }

    [Fact]
    public async Task Stops_when_role_is_not_the_environment_deploy_role()
    {
        var result = await Guard.CheckAsync(StsReturning("444455556666", "PlatformAdmin"), Map, "prod", "us-east-1");
        Assert.Contains("Role mismatch", result);
    }

    [Fact]
    public async Task Stops_on_wrong_region()
    {
        var result = await Guard.CheckAsync(StsReturning("444455556666", "hl-deploy-prod"), Map, "prod", "eu-west-2");
        Assert.Contains("Region mismatch", result);
    }
}

2. The promotion workflow

.github/workflows/promote.yml
name: build-and-promote
on:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write

jobs:
  build:
    runs-on: ubuntu-24.04
    environment: build                       # trusted only by the ECR-push role in the shared account
    outputs:
      digest: ${{ steps.push.outputs.digest }}
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
        with:
          role-to-assume: ${{ vars.ECR_PUSH_ROLE_ARN }}
          aws-region: us-east-1
      - id: ecr
        uses: aws-actions/amazon-ecr-login@7290e214a610b32bc0eef81748948b6d466a8783 # v2.2.0
      - id: push
        env:
          REPO: ${{ steps.ecr.outputs.registry }}/hl-accounts-api
        run: |
          docker build -t "$REPO:${GITHUB_SHA}" .
          docker push "$REPO:${GITHUB_SHA}"
          # The digest identifies the exact bytes pushed; tags can move, digests cannot.
          DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' "$REPO:${GITHUB_SHA}" | cut -d@ -f2)
          echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"

  deploy:
    needs: build
    strategy:
      max-parallel: 1                         # environments in order, never in parallel
      matrix:
        environment: [test, staging, prod]
    runs-on: ubuntu-24.04
    environment: ${{ matrix.environment }}    # prod waits here for a required reviewer
    concurrency: deploy-${{ matrix.environment }}
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: actions/setup-dotnet@0f4a354dd5547a28c70b7853dbd95d5b67886e73 # v6.1.0
        with:
          dotnet-version: "10.0.x"
      - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
        with:
          role-to-assume: ${{ vars.DEPLOY_ROLE_ARN }}   # a different ARN per GitHub environment
          aws-region: us-east-1
      - name: Pre-flight guard
        run: dotnet run --project tools/DeployGuard -- deploy/environments.json ${{ matrix.environment }}
      - name: Deploy the same digest
        run: ./deploy/ecs-deploy.sh "${{ matrix.environment }}" "${{ needs.build.outputs.digest }}"

A matrix with max-parallel: 1 runs the environments in the listed order in practice, but GitHub does not formally guarantee matrix ordering ⚠ VERIFY; many teams instead write three explicit jobs (deploy-test, deploy-staging with needs: deploy-test, deploy-prod with needs: deploy-staging), which also lets you add environment-specific test steps. Use the explicit form for production pipelines.

Code explained line by line

Guard.CheckAsync

  1. map.Environments.TryGetValue(environment, ...) — refuses unknown environment names (a typo like prdo stops the job rather than falling through).
  2. Region check — the job's AWS_REGION must match the account map; deploying to the right account in the wrong Region is still a wrong deployment.
  3. sts.GetCallerIdentityAsync — the one STS call that needs no permissions; it returns the account and the ARN of the credentials in use.
  4. Account comparison — catches a role ARN copied from the wrong GitHub environment.
  5. ":assumed-role/hl-deploy-{environment}/" — catches credentials from a different role in the right account (for example an admin session left in an environment variable on a self-hosted runner).
  6. Returns a reason string rather than throwing, which makes the tests readable and the output clear.

Program.Main

  1. PropertyNameCaseInsensitive = true — the JSON uses camelCase while the records use PascalCase.
  2. ::error title=Deploy guard::... — GitHub Actions workflow command that shows a red annotation on the run summary.
  3. Exit code 1 fails the step, so the deploy step never runs.

Workflow

  1. permissions: id-token: write — without it, the job cannot request an OIDC token at all.
  2. environment: build / environment: ${{ matrix.environment }} — the GitHub environment name becomes part of the token's sub claim, which is what each AWS trust policy checks.
  3. outputs.digest — the digest flows from the build job to every deploy job; there is no rebuild.
  4. concurrency: deploy-<env> — two runs can never deploy to the same environment at once.
  5. vars.DEPLOY_ROLE_ARN — an environment-scoped variable, different for each environment.

Security considerations

⚠ Learning shortcut — not for production

In this course everything runs in the single learn-sandbox account, and the "test", "staging" and "prod" roles are three roles in that one account. That demonstrates the trust policies and the guard, but provides none of the isolation this lesson is about. In production, each environment is a separate account in its own OU, with its own SCPs, and nobody has standing write access to the prod account.

⚠ Learning shortcut — not for production

Pattern A creates an OIDC provider in every account by hand in this lesson. In production, create providers and deploy roles from code (StackSets or Terraform) owned by the platform team, so a workload team cannot quietly widen a trust policy.

  • Never use a wildcard sub such as repo:harbourline/* or repo:harbourline/accounts-api:* for a prod role. Any branch, any pull request workflow, or any repository in the organisation could then assume it.
  • The management account deploys nothing. Keep it for Organizations and billing only; SCPs do not apply to it.
  • Logs go to the log-archive account. An organisation CloudTrail delivering to a bucket in a separate account means a compromised prod account cannot erase its own trail. See CloudTrail for banks.
  • Network separation follows account separation. No VPC peering or Transit Gateway route between prod and non-prod workload VPCs unless an architecture review approves a specific, logged path.
  • Break-glass. Keep one emergency permission set for prod, require MFA, alert on every use, and review each use afterwards. See Federation and segregation of duties.

Banking use case

Harbourline's account vending process, owned by the platform team, creates four accounts for each new product team: hl-<team>-dev, -test, -staging and -prod. Each is created with:

Baseline itemDevTest / StagingProd
Human access (IAM Identity Center)Developers: PowerUser-styleDevelopers: read-only + run testsDevelopers: read-only; SRE: read-only + break-glass
Deploy rolehl-deploy-dev (branch builds allowed)hl-deploy-test, hl-deploy-staging (main only)hl-deploy-prod (main, approved, self-review prevented)
DataSyntheticSynthetic, production-like volumeCustomer data
SCPsRegion limit, no IAM users+ no internet-facing resources without a tag approval+ changes only via pipeline, deletion protection required
Budget alert£200 / month£500 / monthPer capacity plan

The change-advisory board can now answer "who can change production?" with a short list: the hl-deploy-prod role (reachable only from an approved prod job on main), the CloudFormation execution role it passes, and the break-glass permission set. Each is in code, and each use is in CloudTrail in an account the workload team cannot touch.

Common mistakes

  • Separating environments by tag or name prefix only → one IAM mistake reaches prod; quotas are shared → use accounts.
  • Rebuilding the image for each environment → prod runs something that was never tested → build once, promote the digest.
  • One deploy role trusted by all GitHub environments → the test job can deploy to prod → one role per environment, sub bound to that environment.
  • Role ARNs stored as repository-level variables → every job sees the prod ARN; with a loose trust policy that is enough → use environment-scoped variables and tight trust policies.
  • ECR lifecycle policy deleting images still running in prod → rollback target or even scale-out fails → keep images by environment tag or by age long enough to cover your rollback window.
  • Copying production data into staging "just for a test" → a GDPR / UK GDPR issue and a likely audit finding → synthetic data by default; any exception reviewed and time-limited.

Troubleshooting

SymptomLikely causeFix
Not authorized to perform sts:AssumeRoleWithWebIdentitysub in the trust policy does not match the token (job not in an environment, wrong repo name or case, pull-request run)Print the claims in a debug job or check the sub format repo:owner/repo:environment:name; make sure the job declares environment:
Credentials could not be loaded in a jobMissing permissions: id-token: writeAdd it at workflow or job level
ECS tasks in prod fail with CannotPullContainerError ... 403Repository policy does not include the prod account, or the task execution role lacks ecr:GetAuthorizationToken / pull actionsFix the repository policy in the shared account and the execution role in the prod account
Chained role session expires mid-deployRole chaining caps sessions at one hourSplit long deploys, or use pattern A (OIDC directly into the target account)
AccessDenied with "explicit deny in a service control policy"An SCP blocks the action for this principal or RegionCheck which SCP applies to the account's OU; confirm the role name matches the exemption pattern
CDK deploy to target account fails with "not bootstrapped" or trust errorsTarget not bootstrapped with --trust for the pipeline accountRun cdk bootstrap in the target with --trust and narrow execution policies
Deploy guard reports "Account mismatch"DEPLOY_ROLE_ARN copied into the wrong GitHub environmentFix the variable; the guard did its job

Cost and cleanup

Cost: AWS Organizations, IAM roles, OIDC providers and SCPs are free. Creating extra member accounts is free, but each account's own resources bill normally, and Control Tower uses AWS Config and CloudTrail, which have charges. Closed accounts remain in a suspended state for 90 days before permanent closure. Label: free — this lesson is designed as a design exercise plus sandbox simulation: three roles in the one sandbox account. Do not create new accounts just for this lesson.

P=learn-sandbox
# Delete the simulated per-environment roles (inline policies first).
for env in test staging prod; do
  for pol in $(aws iam list-role-policies --role-name hl-deploy-$env --query 'PolicyNames[]' --output text --profile $P); do
    aws iam delete-role-policy --role-name hl-deploy-$env --policy-name "$pol" --profile $P
  done
  aws iam delete-role --role-name hl-deploy-$env --profile $P
done

# Keep the GitHub OIDC provider if other lessons use it; otherwise remove it:
# aws iam delete-open-id-connect-provider --open-id-connect-provider-arn arn:aws:iam::<acct>:oidc-provider/token.actions.githubusercontent.com --profile $P

# Confirm nothing tagged for this lesson remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m09-l10 \
  --query 'ResourceTagMappingList[].ResourceARN' --profile $P

Exercises

  • 🟢 Guided: In the sandbox, create hl-deploy-test and hl-deploy-prod with the trust policies above (pointing at your own repository). Create GitHub environments test and prod. Run a workflow job in test that tries to assume the prod role and confirm it fails; then run the deploy guard in each job.
  • 🟡 Challenge: Write the Terraform (or CDK) to roll the deploy role out to three accounts from one configuration, using provider aliases (or three stack instances with different Env). Add an output listing each role ARN, and a check that fails terraform plan if any trust policy's sub contains *.
  • 🔴 Live problem: A product owner asks for "a copy of last month's production transactions in staging so we can reproduce a reconciliation bug". Write your reply: what you would offer instead, under what conditions an exception could be granted, who must approve it, and how the data would be protected and deleted.

Assignment

Produce an ENVIRONMENTS.md for your portfolio project with: the OU and account diagram (Mermaid); a table of environments with account ID placeholders, Region, data class and who has which access; the trust policy for each deploy role; the promotion workflow; and the SCP you would attach to the prod OU, with a sentence explaining each statement.

Solutions

Knowledge check

Interview questions

  • "How would you structure AWS accounts for a team with dev, test and prod?" — An organisation with Security (log archive, audit), Infrastructure (shared ECR/networking), Workloads split into NonProd and Prod OUs, and Sandbox. One account per environment per workload or domain. SCPs per OU. The management account runs nothing.
  • "How does your pipeline deploy into prod without long-lived keys?" — GitHub OIDC; each account has a role trusting only repo:<org>/<repo>:environment:<env>; the prod environment has required reviewers with self-review prevented; the role can only update the specific service and pass specific roles.
  • "How do you make sure prod runs what you tested?" — Build once, push to a shared registry with immutable tags, pass the digest between jobs, deploy by digest, and record the digest in the release evidence.
  • "What stops someone changing prod by hand?" — Read-only human access, an SCP that denies change actions to anyone but the deploy and CloudFormation execution roles, monitored break-glass, and CloudTrail delivered to a separate log-archive account.

Certification practice

Completion checklist

  • I can explain why environments are separated by account, with at least three concrete reasons
  • I drew an OU and account layout for Harbourline
  • I created per-environment deploy roles with sub conditions bound to GitHub environments
  • I saw a test job fail to assume the prod role
  • I can explain build-once, promote-by-digest
  • I deleted the simulated roles from the sandbox

Key takeaways

  • The AWS account is the isolation boundary for IAM, quotas, billing and guardrails; use one per environment.
  • One deploy role per environment, trusted only by repo:<org>/<repo>:environment:<env>; protect prod with required reviewers and self-review prevention.
  • Build once; promote the image digest through every environment.
  • SCPs make "only the pipeline changes prod" an organisation-level rule, not a convention.
  • Keep production data in production; synthetic data everywhere else.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page