Environment separation: one AWS account per environment
Separate dev, test, staging and prod into their own AWS accounts under AWS Organizations, deploy into each with its own OIDC role from GitHub Actions, promote the same image digest from one environment to the next, and add guardrails so a pipeline can never deploy to the wrong account.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SecurityToken 4.x) · GitHub Actions (aws-actions/configure-aws-credentials v6.3.0) · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain why banks separate environments by AWS account, not by naming convention, VPC or tag.
- Design an AWS Organizations layout for Harbourline's delivery pipeline: workload accounts per environment, a shared tooling account, and security accounts.
- Choose between two cross-account deploy patterns — OIDC directly into each account and a central tooling role that chains into target roles — and explain the trade-offs.
- Build once, promote many: move the same image digest from test to staging to prod, with a cross-account ECR policy.
- Add guardrails: OIDC trust policies bound to GitHub environments, an SCP that limits who may deploy in prod, and a C# pre-flight check that refuses to deploy to the wrong account.
Prerequisites
- Module 1: Accounts and AWS Organizations — what an account, OU and SCP are.
- Permission boundaries, SCPs and RCPs — how SCPs are evaluated.
- GitHub Actions with OIDC — the provider, the trust policy and the
subclaim. - Tools: AWS CLI v2 with the
learn-sandboxprofile; a GitHub repository where you can create environments.
Why this matters in banking
Harbourline's first cloud team put dev, test and prod for the Accounts API in one AWS account, separated by name prefixes: dev-accounts-api, test-accounts-api, prod-accounts-api. It looked tidy. Then:
- A developer testing a clean-up script ran
aws ecs delete-serviceagainst a list built with a*accounts-api*filter. The prod service was in the list. - A load test in "test" hit the account-level Lambda concurrency quota and throttled the prod payment validator, because quotas are per account and Region.
- Internal audit asked "who can change production?" The honest answer was "everyone with access to the account", because IAM policies with name-based conditions had gaps nobody could prove were closed.
- The cost report could not split prod cost from experiments without perfect tagging, which nobody had.
An AWS account is the strongest isolation boundary AWS offers: separate IAM, separate quotas, separate billing line, separate resource namespace, and it is the unit that SCPs, CloudTrail organisation trails and AWS Config aggregators work with. For a bank, account separation is also how you show segregation of duties concretely: developers have broad access in dev, read-only access in prod, and the only thing that can change prod is the pipeline's prod role, after approval.
The idea in plain language
Instead of one AWS account with everything inside it, you have one account per environment for each workload (or group of workloads), plus a few shared accounts. AWS Organizations groups them, applies guardrails to whole groups at once (SCPs) and pays one combined bill.
The pipeline then needs a way to deploy into several accounts. The modern way is no stored keys anywhere: GitHub Actions presents a short-lived OIDC token, and each environment's AWS account has its own IAM role that trusts only tokens for that repository and that GitHub environment. A job running in the test environment can only become the test role; a job running in prod — which only starts after a required reviewer approves — can only become the prod role.
What moves between environments is the artefact, not the build. You build the container image once, record its digest (sha256:…, an unchangeable fingerprint of its content), and deploy exactly that digest to test, then staging, then prod. If prod runs something, it is byte-for-byte what test ran.
Analogy: a bank does not keep the training branch, the trial branch and the real branch in one building with labels on the doors. They are separate buildings with separate keys. A courier (the pipeline) carries the same sealed bag (the image digest) from one building to the next, and at each door shows a pass that only works for that door, issued that day, and — for the real branch — counter-signed by a manager.
What problem the AWS services solve
| Service / feature | What it does for environment separation | Limits and sharp edges |
|---|---|---|
| AWS Organizations | Groups accounts into OUs, applies SCPs and RCPs, consolidates billing | SCPs never grant permissions; they only cap them. They do not apply to the management account |
| AWS Control Tower | Sets up a landing zone: log archive and audit accounts, guardrails (controls), Account Factory | Opinionated; uses AWS Config and CloudTrail, which have their own cost |
| IAM OIDC identity provider (per account) | Lets GitHub's OIDC tokens be exchanged for role credentials in that account | One provider per issuer URL per account; the trust policy, not the provider, is what restricts who can assume a role |
| STS role chaining | A tooling-account role assumes a role in a target account | Chained sessions are limited to one hour; every hop needs both a trust policy and an sts:AssumeRole permission |
| ECR repository policies and replication | Let workload accounts pull images from a shared repository, or replicate images into each account | Pull access is granted to the account's principals; the ECS task execution role still needs ecr:GetAuthorizationToken in its own account |
CDK bootstrap --trust | Lets a pipeline account deploy CDK stacks into a target account | The --cloudformation-execution-policies you choose decide what CloudFormation may do in that account |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Subscription per environment | AWS account per environment | An AWS account is also the IAM, quota and namespace boundary; there is no tenant-wide RBAC across accounts |
| Management groups | Organizational units (OUs) | OUs nest up to five levels below the root |
| Azure Policy (deny effect) | Service control policies (SCPs) | SCPs restrict IAM actions; for resource configuration checks you add AWS Config rules or Control Tower controls |
| Azure Landing Zones | AWS Control Tower landing zone | Similar idea; Control Tower creates log archive and audit accounts by default |
| Service connection per subscription (workload identity federation) | OIDC provider + IAM role per account | Same federation idea; the AWS trust policy can bind to a GitHub environment through the sub claim |
| ACR shared across subscriptions | ECR in a shared account + repository policy, or ECR cross-account replication | Pull permission is in a resource policy on the repository |
| Azure DevOps environment approvals | GitHub environment required reviewers | Approval gates which job can request the OIDC token for that environment |
Architecture
Promotion flow
Two cross-account patterns
| A. OIDC directly into each account | B. Tooling account + role chaining | |
|---|---|---|
| How | Each workload account has the GitHub OIDC provider and a deploy role trusting one repo + environment | Only the tooling account trusts GitHub; its role assumes hl-deploy-* roles in workload accounts |
| Strength | Simplest trust chain; the GitHub environment approval maps directly to the prod role | One place to manage GitHub trust; works the same for CodePipeline |
| Weakness | The OIDC provider and roles must be created in every account (automate with StackSets or Terraform) | Chained sessions are capped at one hour; two policies to get right per hop; a compromise of the tooling role reaches every account it can chain into |
| Use when | GitHub Actions is your orchestrator (this module's default) | CodePipeline in a tooling account, or a platform team that wants central control |
Step-by-step implementation
Agree the account map
List environments and accounts and write them into the repository (for example deploy/environments.json): name, account ID, Region, OU, purpose, data classification. This file is what the pre-flight check in the code section reads. In this course you have one sandbox account, so treat the multi-account parts as a design exercise and simulate them with the sandbox.
Create one deploy role per environment
In each workload account, create hl-deploy-<env> with a trust policy that allows sts:AssumeRoleWithWebIdentity from the GitHub OIDC provider only when aud is sts.amazonaws.com and sub is exactly repo:harbourline/accounts-api:environment:<env>. Give it only the permissions that environment's deployment needs.
Create matching GitHub environments
In the repository: Settings → Environments → test, staging, prod. For prod, add required reviewers, turn on Prevent self-review, and restrict deployment branches to main. Store each environment's role ARN as an environment variable (DEPLOY_ROLE_ARN) — role ARNs are identifiers, not secrets.
Share images from one registry
Keep ECR in the shared-services account. Add a repository policy allowing the workload accounts to pull. Turn on tag immutability. Deploy by digest (repo@sha256:…), never by a moving tag such as latest.
Add organisation guardrails
Attach an SCP to the Prod OU that denies changing ECS services, Lambda aliases and CloudFormation stacks to every principal except the deploy role and the break-glass role. Attach a Region-restriction SCP to all workload OUs. Use RCPs or resource policies to stop prod data stores being reachable from non-prod principals.
Add a pre-flight check to every deploy job
Before any deployment command, the job calls sts:GetCallerIdentity and compares the account ID with the account map entry for its environment. If they differ, the job stops. This catches a mis-copied role ARN before it does damage.
Keep production data out of lower environments
Test and staging use synthetic data from the dataset generator (seed 20261117). Any exception — for example, a masked extract for a performance test — needs a data-protection review under GDPR / UK GDPR and a documented deletion date.
AWS Console instructions
Organizations:
- Sign in to the management account → Console → AWS Organizations → AWS accounts.
- Actions → Create new organizational unit under Root:
Security,Infrastructure,Workloads,Sandbox; underWorkloads, createNonProdandProd. - Add an AWS account → Create an AWS account for each environment (requires a unique email per account — use plus-addressing on a mailbox you control).
- Policies → Service control policies → Create policy, paste the SCP below, then Targets → Attach to the
ProdOU.
OIDC provider and role in a workload account:
- Console → IAM → Identity providers → Add provider → OpenID Connect. Provider URL
https://token.actions.githubusercontent.com, audiencests.amazonaws.com. - IAM → Roles → Create role → Web identity, choose the provider and audience, then on the trust policy page add the
subcondition for the repository and environment. Name ithl-deploy-prod.
GitHub environment:
- GitHub → repository → Settings → Environments → New environment →
prod. - Tick Required reviewers (up to six people or teams), tick Prevent self-review, and under Deployment branches and tags choose Selected branches and tags →
main. - Add the variable
DEPLOY_ROLE_ARNwith the prod role ARN.
AWS CLI
P=learn-sandbox
# 1. Organizations: show the OU tree and which accounts live where (run in the management account).
ROOT=$(aws organizations list-roots --query 'Roots[0].Id' --output text --profile mgmt)
aws organizations list-organizational-units-for-parent --parent-id $ROOT --profile mgmt
aws organizations list-accounts --query 'Accounts[].[Id,Name,Status]' --output table --profile mgmt
# 2. Create the GitHub OIDC provider in a workload account (once per account).
aws iam create-open-id-connect-provider \
--url https://token.actions.githubusercontent.com \
--client-id-list sts.amazonaws.com --profile $P
# 3. Create the per-environment deploy role with the trust policy below.
aws iam create-role --role-name hl-deploy-test \
--assume-role-policy-document file://trust-test.json \
--max-session-duration 3600 \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m09-l10 Key=Environment,Value=learn --profile $P
# 4. Prove the guard works: simulate whether the role's permissions allow a prod-style action.
aws iam simulate-principal-policy \
--policy-source-arn arn:aws:iam::111122223333:role/hl-deploy-test \
--action-names ecs:UpdateService iam:CreateUser --profile $P
# 5. ECR in the shared account: allow workload accounts to pull, and make tags immutable.
aws ecr put-image-tag-mutability --repository-name hl-accounts-api \
--image-tag-mutability IMMUTABLE --profile shared
aws ecr set-repository-policy --repository-name hl-accounts-api \
--policy-text file://ecr-pull-policy.json --profile shared
# 6. Resolve a tag to its digest once, then deploy the digest everywhere.
aws ecr describe-images --repository-name hl-accounts-api --image-ids imageTag=4.13.0 \
--query 'imageDetails[0].imageDigest' --output text --profile shared
# 7. CDK: let the tooling/pipeline account deploy into a target account,
# with a narrow execution policy instead of AdministratorAccess.
cdk bootstrap aws://444455556666/us-east-1 \
--trust 777788889999 \
--cloudformation-execution-policies arn:aws:iam::444455556666:policy/hl-cfn-exec-accounts \
--profile prod-adminThe trust policy for hl-deploy-test (file trust-test.json):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GitHubTestEnvironmentOnly",
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:harbourline/accounts-api:environment:test"
}
}
}
]
}The cross-account ECR pull policy (file ecr-pull-policy.json):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "WorkloadAccountsPull",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::111122223333:root",
"arn:aws:iam::222233334444:root",
"arn:aws:iam::444455556666:root"
]
},
"Action": [
"ecr:BatchGetImage",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchCheckLayerAvailability"
]
}
]
}arn:aws:iam::<account>:root in a resource policy means "this account may grant the permission to its own principals"; each workload account still decides, through IAM, which of its roles (the ECS task execution role) actually gets to pull.
Infrastructure as Code
variable "environment" { type = string } # "test", "staging" or "prod"
variable "github_repo" { type = string } # "harbourline/accounts-api"
resource "aws_iam_openid_connect_provider" "github" {
url = "https://token.actions.githubusercontent.com"
client_id_list = ["sts.amazonaws.com"]
}
data "aws_iam_policy_document" "trust" {
statement {
actions = ["sts:AssumeRoleWithWebIdentity"]
principals {
type = "Federated"
identifiers = [aws_iam_openid_connect_provider.github.arn]
}
condition {
test = "StringEquals"
variable = "token.actions.githubusercontent.com:aud"
values = ["sts.amazonaws.com"]
}
condition {
test = "StringEquals"
variable = "token.actions.githubusercontent.com:sub"
values = ["repo:${var.github_repo}:environment:${var.environment}"]
}
}
}
resource "aws_iam_role" "deploy" {
name = "hl-deploy-${var.environment}"
assume_role_policy = data.aws_iam_policy_document.trust.json
max_session_duration = 3600
tags = {
Project = "cea-lab"
Lab = "m09-l10"
Environment = var.environment
}
}
# Only what an ECS deploy of this service needs, in this account.
data "aws_iam_policy_document" "deploy" {
statement {
sid = "UpdateThisService"
actions = ["ecs:UpdateService", "ecs:DescribeServices"]
resources = ["arn:aws:ecs:us-east-1:${data.aws_caller_identity.me.account_id}:service/hl-${var.environment}-cluster/hl-accounts-api"]
}
statement {
sid = "RegisterTaskDefinitions"
actions = ["ecs:RegisterTaskDefinition", "ecs:DescribeTaskDefinition"]
resources = ["*"] # these actions do not support resource-level permissions
}
statement {
sid = "PassOnlyTheTaskRoles"
actions = ["iam:PassRole"]
resources = [
"arn:aws:iam::${data.aws_caller_identity.me.account_id}:role/hl-accounts-api-task",
"arn:aws:iam::${data.aws_caller_identity.me.account_id}:role/hl-accounts-api-exec"
]
condition {
test = "StringEquals"
variable = "iam:PassedToService"
values = ["ecs-tasks.amazonaws.com"]
}
}
}
data "aws_caller_identity" "me" {}
resource "aws_iam_role_policy" "deploy" {
role = aws_iam_role.deploy.id
policy = data.aws_iam_policy_document.deploy.json
}Apply this module to every workload account with a provider alias per account, or roll it out with CloudFormation StackSets from the management account. Either way, the role definition lives in Git and is reviewed like any other code.
Complete code
1. The account map and a C# pre-flight guard
{
"environments": {
"test": { "accountId": "111122223333", "region": "us-east-1", "dataClass": "synthetic" },
"staging": { "accountId": "222233334444", "region": "us-east-1", "dataClass": "synthetic" },
"prod": { "accountId": "444455556666", "region": "us-east-1", "dataClass": "customer" }
}
}using System.Text.Json;
using Amazon.SecurityToken;
using Amazon.SecurityToken.Model;
namespace Harbourline.Platform.DeployGuard;
public sealed record EnvironmentEntry(string AccountId, string Region, string DataClass);
public sealed record EnvironmentMap(Dictionary<string, EnvironmentEntry> Environments);
public static class Guard
{
/// <summary>
/// Returns null when it is safe to deploy, or a human-readable reason to stop.
/// </summary>
public static async Task<string?> CheckAsync(
IAmazonSecurityTokenService sts, EnvironmentMap map, string environment, string region,
CancellationToken ct = default)
{
if (!map.Environments.TryGetValue(environment, out var expected))
return $"Unknown environment '{environment}'. Known: {string.Join(", ", map.Environments.Keys)}.";
if (!string.Equals(expected.Region, region, StringComparison.Ordinal))
return $"Region mismatch: '{environment}' deploys to {expected.Region}, job is configured for {region}.";
var me = await sts.GetCallerIdentityAsync(new GetCallerIdentityRequest(), ct);
if (!string.Equals(me.Account, expected.AccountId, StringComparison.Ordinal))
return $"Account mismatch: '{environment}' is {expected.AccountId}, credentials are for {me.Account}.";
// The assumed-role ARN looks like arn:aws:sts::<acct>:assumed-role/<role>/<session>.
var expectedRole = $":assumed-role/hl-deploy-{environment}/";
if (!me.Arn.Contains(expectedRole, StringComparison.Ordinal))
return $"Role mismatch: expected hl-deploy-{environment}, got {me.Arn}.";
return null;
}
}
public static class Program
{
public static async Task<int> Main(string[] args)
{
if (args.Length != 2)
{
Console.Error.WriteLine("Usage: DeployGuard <environments.json> <environment>");
return 2;
}
var map = JsonSerializer.Deserialize<EnvironmentMap>(
await File.ReadAllTextAsync(args[0]),
new JsonSerializerOptions { PropertyNameCaseInsensitive = true })
?? throw new InvalidOperationException("Empty environment map.");
var region = Environment.GetEnvironmentVariable("AWS_REGION") ?? "unset";
using var sts = new AmazonSecurityTokenServiceClient();
var problem = await Guard.CheckAsync(sts, map, args[1], region);
if (problem is not null)
{
// GitHub Actions turns this line into a red annotation on the run.
Console.WriteLine($"::error title=Deploy guard::{problem}");
return 1;
}
Console.WriteLine($"Deploy guard passed for '{args[1]}'.");
return 0;
}
}using Amazon.SecurityToken;
using Amazon.SecurityToken.Model;
using Harbourline.Platform.DeployGuard;
using NSubstitute;
using Xunit;
public class GuardTests
{
private static readonly EnvironmentMap Map = new(new()
{
["test"] = new("111122223333", "us-east-1", "synthetic"),
["prod"] = new("444455556666", "us-east-1", "customer")
});
private static IAmazonSecurityTokenService StsReturning(string account, string role)
{
var sts = Substitute.For<IAmazonSecurityTokenService>();
sts.GetCallerIdentityAsync(Arg.Any<GetCallerIdentityRequest>(), Arg.Any<CancellationToken>())
.Returns(new GetCallerIdentityResponse
{
Account = account,
Arn = $"arn:aws:sts::{account}:assumed-role/{role}/GitHubActions"
});
return sts;
}
[Fact]
public async Task Passes_when_account_region_and_role_match()
{
var result = await Guard.CheckAsync(StsReturning("444455556666", "hl-deploy-prod"), Map, "prod", "us-east-1");
Assert.Null(result);
}
[Fact]
public async Task Stops_prod_deploy_with_test_credentials()
{
var result = await Guard.CheckAsync(StsReturning("111122223333", "hl-deploy-test"), Map, "prod", "us-east-1");
Assert.Contains("Account mismatch", result);
}
[Fact]
public async Task Stops_when_role_is_not_the_environment_deploy_role()
{
var result = await Guard.CheckAsync(StsReturning("444455556666", "PlatformAdmin"), Map, "prod", "us-east-1");
Assert.Contains("Role mismatch", result);
}
[Fact]
public async Task Stops_on_wrong_region()
{
var result = await Guard.CheckAsync(StsReturning("444455556666", "hl-deploy-prod"), Map, "prod", "eu-west-2");
Assert.Contains("Region mismatch", result);
}
}2. The promotion workflow
name: build-and-promote
on:
push:
branches: [main]
permissions:
contents: read
id-token: write
jobs:
build:
runs-on: ubuntu-24.04
environment: build # trusted only by the ECR-push role in the shared account
outputs:
digest: ${{ steps.push.outputs.digest }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
with:
role-to-assume: ${{ vars.ECR_PUSH_ROLE_ARN }}
aws-region: us-east-1
- id: ecr
uses: aws-actions/amazon-ecr-login@7290e214a610b32bc0eef81748948b6d466a8783 # v2.2.0
- id: push
env:
REPO: ${{ steps.ecr.outputs.registry }}/hl-accounts-api
run: |
docker build -t "$REPO:${GITHUB_SHA}" .
docker push "$REPO:${GITHUB_SHA}"
# The digest identifies the exact bytes pushed; tags can move, digests cannot.
DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' "$REPO:${GITHUB_SHA}" | cut -d@ -f2)
echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"
deploy:
needs: build
strategy:
max-parallel: 1 # environments in order, never in parallel
matrix:
environment: [test, staging, prod]
runs-on: ubuntu-24.04
environment: ${{ matrix.environment }} # prod waits here for a required reviewer
concurrency: deploy-${{ matrix.environment }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-dotnet@0f4a354dd5547a28c70b7853dbd95d5b67886e73 # v6.1.0
with:
dotnet-version: "10.0.x"
- uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
with:
role-to-assume: ${{ vars.DEPLOY_ROLE_ARN }} # a different ARN per GitHub environment
aws-region: us-east-1
- name: Pre-flight guard
run: dotnet run --project tools/DeployGuard -- deploy/environments.json ${{ matrix.environment }}
- name: Deploy the same digest
run: ./deploy/ecs-deploy.sh "${{ matrix.environment }}" "${{ needs.build.outputs.digest }}"A matrix with max-parallel: 1 runs the environments in the listed order in practice, but GitHub does not formally guarantee matrix ordering ⚠ VERIFY; many teams instead write three explicit jobs (deploy-test, deploy-staging with needs: deploy-test, deploy-prod with needs: deploy-staging), which also lets you add environment-specific test steps. Use the explicit form for production pipelines.
Code explained line by line
Guard.CheckAsync
map.Environments.TryGetValue(environment, ...)— refuses unknown environment names (a typo likeprdostops the job rather than falling through).- Region check — the job's
AWS_REGIONmust match the account map; deploying to the right account in the wrong Region is still a wrong deployment. sts.GetCallerIdentityAsync— the one STS call that needs no permissions; it returns the account and the ARN of the credentials in use.- Account comparison — catches a role ARN copied from the wrong GitHub environment.
":assumed-role/hl-deploy-{environment}/"— catches credentials from a different role in the right account (for example an admin session left in an environment variable on a self-hosted runner).- Returns a reason string rather than throwing, which makes the tests readable and the output clear.
Program.Main
PropertyNameCaseInsensitive = true— the JSON uses camelCase while the records use PascalCase.::error title=Deploy guard::...— GitHub Actions workflow command that shows a red annotation on the run summary.- Exit code
1fails the step, so the deploy step never runs.
Workflow
permissions: id-token: write— without it, the job cannot request an OIDC token at all.environment: build/environment: ${{ matrix.environment }}— the GitHub environment name becomes part of the token'ssubclaim, which is what each AWS trust policy checks.outputs.digest— the digest flows from the build job to every deploy job; there is no rebuild.concurrency: deploy-<env>— two runs can never deploy to the same environment at once.vars.DEPLOY_ROLE_ARN— an environment-scoped variable, different for each environment.
Security considerations
⚠ Learning shortcut — not for production
In this course everything runs in the single learn-sandbox account, and the "test", "staging" and "prod" roles are three roles in that one account. That demonstrates the trust policies and the guard, but provides none of the isolation this lesson is about. In production, each environment is a separate account in its own OU, with its own SCPs, and nobody has standing write access to the prod account.
⚠ Learning shortcut — not for production
Pattern A creates an OIDC provider in every account by hand in this lesson. In production, create providers and deploy roles from code (StackSets or Terraform) owned by the platform team, so a workload team cannot quietly widen a trust policy.
- Never use a wildcard
subsuch asrepo:harbourline/*orrepo:harbourline/accounts-api:*for a prod role. Any branch, any pull request workflow, or any repository in the organisation could then assume it. - The management account deploys nothing. Keep it for Organizations and billing only; SCPs do not apply to it.
- Logs go to the log-archive account. An organisation CloudTrail delivering to a bucket in a separate account means a compromised prod account cannot erase its own trail. See CloudTrail for banks.
- Network separation follows account separation. No VPC peering or Transit Gateway route between prod and non-prod workload VPCs unless an architecture review approves a specific, logged path.
- Break-glass. Keep one emergency permission set for prod, require MFA, alert on every use, and review each use afterwards. See Federation and segregation of duties.
Banking use case
Harbourline's account vending process, owned by the platform team, creates four accounts for each new product team: hl-<team>-dev, -test, -staging and -prod. Each is created with:
| Baseline item | Dev | Test / Staging | Prod |
|---|---|---|---|
| Human access (IAM Identity Center) | Developers: PowerUser-style | Developers: read-only + run tests | Developers: read-only; SRE: read-only + break-glass |
| Deploy role | hl-deploy-dev (branch builds allowed) | hl-deploy-test, hl-deploy-staging (main only) | hl-deploy-prod (main, approved, self-review prevented) |
| Data | Synthetic | Synthetic, production-like volume | Customer data |
| SCPs | Region limit, no IAM users | + no internet-facing resources without a tag approval | + changes only via pipeline, deletion protection required |
| Budget alert | £200 / month | £500 / month | Per capacity plan |
The change-advisory board can now answer "who can change production?" with a short list: the hl-deploy-prod role (reachable only from an approved prod job on main), the CloudFormation execution role it passes, and the break-glass permission set. Each is in code, and each use is in CloudTrail in an account the workload team cannot touch.
Common mistakes
- Separating environments by tag or name prefix only → one IAM mistake reaches prod; quotas are shared → use accounts.
- Rebuilding the image for each environment → prod runs something that was never tested → build once, promote the digest.
- One deploy role trusted by all GitHub environments → the
testjob can deploy to prod → one role per environment,subbound to that environment. - Role ARNs stored as repository-level variables → every job sees the prod ARN; with a loose trust policy that is enough → use environment-scoped variables and tight trust policies.
- ECR lifecycle policy deleting images still running in prod → rollback target or even scale-out fails → keep images by environment tag or by age long enough to cover your rollback window.
- Copying production data into staging "just for a test" → a GDPR / UK GDPR issue and a likely audit finding → synthetic data by default; any exception reviewed and time-limited.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Not authorized to perform sts:AssumeRoleWithWebIdentity | sub in the trust policy does not match the token (job not in an environment, wrong repo name or case, pull-request run) | Print the claims in a debug job or check the sub format repo:owner/repo:environment:name; make sure the job declares environment: |
Credentials could not be loaded in a job | Missing permissions: id-token: write | Add it at workflow or job level |
ECS tasks in prod fail with CannotPullContainerError ... 403 | Repository policy does not include the prod account, or the task execution role lacks ecr:GetAuthorizationToken / pull actions | Fix the repository policy in the shared account and the execution role in the prod account |
| Chained role session expires mid-deploy | Role chaining caps sessions at one hour | Split long deploys, or use pattern A (OIDC directly into the target account) |
AccessDenied with "explicit deny in a service control policy" | An SCP blocks the action for this principal or Region | Check which SCP applies to the account's OU; confirm the role name matches the exemption pattern |
| CDK deploy to target account fails with "not bootstrapped" or trust errors | Target not bootstrapped with --trust for the pipeline account | Run cdk bootstrap in the target with --trust and narrow execution policies |
| Deploy guard reports "Account mismatch" | DEPLOY_ROLE_ARN copied into the wrong GitHub environment | Fix the variable; the guard did its job |
Cost and cleanup
Cost: AWS Organizations, IAM roles, OIDC providers and SCPs are free. Creating extra member accounts is free, but each account's own resources bill normally, and Control Tower uses AWS Config and CloudTrail, which have charges. Closed accounts remain in a suspended state for 90 days before permanent closure. Label: free — this lesson is designed as a design exercise plus sandbox simulation: three roles in the one sandbox account. Do not create new accounts just for this lesson.
P=learn-sandbox
# Delete the simulated per-environment roles (inline policies first).
for env in test staging prod; do
for pol in $(aws iam list-role-policies --role-name hl-deploy-$env --query 'PolicyNames[]' --output text --profile $P); do
aws iam delete-role-policy --role-name hl-deploy-$env --policy-name "$pol" --profile $P
done
aws iam delete-role --role-name hl-deploy-$env --profile $P
done
# Keep the GitHub OIDC provider if other lessons use it; otherwise remove it:
# aws iam delete-open-id-connect-provider --open-id-connect-provider-arn arn:aws:iam::<acct>:oidc-provider/token.actions.githubusercontent.com --profile $P
# Confirm nothing tagged for this lesson remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m09-l10 \
--query 'ResourceTagMappingList[].ResourceARN' --profile $PExercises
- 🟢 Guided: In the sandbox, create
hl-deploy-testandhl-deploy-prodwith the trust policies above (pointing at your own repository). Create GitHub environmentstestandprod. Run a workflow job intestthat tries to assume the prod role and confirm it fails; then run the deploy guard in each job. - 🟡 Challenge: Write the Terraform (or CDK) to roll the deploy role out to three accounts from one configuration, using provider aliases (or three stack instances with different
Env). Add an output listing each role ARN, and a check that failsterraform planif any trust policy'ssubcontains*. - 🔴 Live problem: A product owner asks for "a copy of last month's production transactions in staging so we can reproduce a reconciliation bug". Write your reply: what you would offer instead, under what conditions an exception could be granted, who must approve it, and how the data would be protected and deleted.
Assignment
Produce an ENVIRONMENTS.md for your portfolio project with: the OU and account diagram (Mermaid); a table of environments with account ID placeholders, Region, data class and who has which access; the trust policy for each deploy role; the promotion workflow; and the SCP you would attach to the prod OU, with a sentence explaining each statement.
Solutions
The test job's token carries sub = repo:<you>/<repo>:environment:test. The prod role's trust policy requires ...:environment:prod, so STS rejects the request with AccessDenied on AssumeRoleWithWebIdentity. The deploy guard in the test job passes (right account and role), and in the prod job — after approval — also passes. Why it matters: the environment approval in GitHub and the IAM trust policy in AWS reinforce each other. Even if someone edits the workflow to use the prod ARN in a test job, AWS refuses.
In Terraform, declare provider "aws" { alias = "test" ... assume_role { role_arn = ... } } for each account and instantiate a deploy_role module three times, passing providers = { aws = aws.test } and the environment name. For the wildcard check, add a precondition in the module's role resource: condition = !strcontains(local.sub, "*") with a clear error_message. Why: the guard is in the code that creates the role, so a reviewer doesn't have to spot a * in a JSON string — the plan fails before anything changes.
Offer first: generate a synthetic dataset with the same shape and volume (the course generator, plus "messy" variants), or reproduce the bug with the specific transaction pattern described in anonymised terms. If real data is genuinely necessary: a documented business justification; a data-protection impact review (the DPO's office); the minimum fields (pseudonymise or tokenise names, IBANs and references); a separate, access-controlled location with the same encryption and logging as prod; named people with access; and a deletion date with evidence of deletion. The approval sits with the data owner and data protection, not with the engineering team. Under GDPR / UK GDPR, "it's only staging" is not a lawful basis.
Knowledge check
Answer: IAM (principals and policies), service quotas (e.g. Lambda concurrency), billing, and the resource namespace; it is also the unit SCPs and organisation-wide logging operate on.
Answer: A tag can be moved to different content (unless immutability is enforced everywhere); a digest is a hash of the content and always means the same bytes. Promoting the digest proves prod runs what test tested.
Answer: A chained role session is limited to one hour, and every hop needs correct trust and permission policies; the tooling role also becomes a high-value target because it can reach every account it chains into.
Answer: Only tokens issued for jobs in that repository that declare the prod GitHub environment can assume the role. Combined with required reviewers on the environment, the job cannot obtain the token until approved.
Answer: SCPs do not apply to the management account, and it controls the whole organisation. Anything running there sits outside your guardrails and next to the most powerful permissions you have.
Interview questions
- "How would you structure AWS accounts for a team with dev, test and prod?" — An organisation with Security (log archive, audit), Infrastructure (shared ECR/networking), Workloads split into NonProd and Prod OUs, and Sandbox. One account per environment per workload or domain. SCPs per OU. The management account runs nothing.
- "How does your pipeline deploy into prod without long-lived keys?" — GitHub OIDC; each account has a role trusting only
repo:<org>/<repo>:environment:<env>; the prod environment has required reviewers with self-review prevented; the role can only update the specific service and pass specific roles. - "How do you make sure prod runs what you tested?" — Build once, push to a shared registry with immutable tags, pass the digest between jobs, deploy by digest, and record the digest in the release evidence.
- "What stops someone changing prod by hand?" — Read-only human access, an SCP that denies change actions to anyone but the deploy and CloudFormation execution roles, monitored break-glass, and CloudTrail delivered to a separate log-archive account.
Certification practice
A company uses GitHub Actions to deploy to separate test and production AWS accounts. Security requires that no long-term credentials are stored in GitHub and that only workflow jobs approved for production can deploy to the production account. What should a DevOps engineer do?
A. Create an IAM user in each account and store access keys as GitHub environment secrets.
B. Create an IAM OIDC provider for GitHub in each account and a role in each whose trust policy requires the matching sub value for the repository and GitHub environment; protect the production environment with required reviewers.
C. Create one role in the test account trusted by GitHub that has AdministratorAccess in the production account.
D. Use AWS Systems Manager Parameter Store to hold access keys and read them in the workflow.
Answer: B. Short-lived OIDC credentials bound to repository and environment, with approval on the environment. A and D use long-term keys. C is not possible as stated (a role's permissions apply in its own account) and would grossly over-privilege.
An organisation wants to ensure that, in all production accounts, only the pipeline's deployment role can update ECS services, including for administrators. Which solution meets this requirement?
A. An IAM permissions boundary on the deployment role.
B. An SCP attached to the production OU that denies ecs:UpdateService unless aws:PrincipalArn matches the deployment role.
C. An AWS Config rule that detects ECS service changes.
D. A resource-based policy on the ECS cluster.
Answer: B. SCPs apply to every principal in member accounts, including administrators. A limits the deployment role, not others. C detects after the fact. D — ECS clusters do not support resource-based policies for this purpose.
A team builds a container image in a shared services account and deploys it to test, staging and production accounts. Auditors require evidence that production runs the same image that passed testing. What is the BEST approach?
A. Rebuild the image in each account from the same Git commit.
B. Use the latest tag in every environment.
C. Enable tag immutability, deploy the image by digest in every environment, allow pull access with an ECR repository policy, and record the digest in the release record.
D. Copy the image tarball to S3 and load it in each account.
Answer: C. The digest is a content hash; immutability prevents tag reuse. A can produce different bytes (base image updates, non-reproducible builds). B is a moving pointer. D is manual and loses registry features such as scanning.
Completion checklist
- I can explain why environments are separated by account, with at least three concrete reasons
- I drew an OU and account layout for Harbourline
- I created per-environment deploy roles with
subconditions bound to GitHub environments - I saw a test job fail to assume the prod role
- I can explain build-once, promote-by-digest
- I deleted the simulated roles from the sandbox
Key takeaways
- The AWS account is the isolation boundary for IAM, quotas, billing and guardrails; use one per environment.
- One deploy role per environment, trusted only by
repo:<org>/<repo>:environment:<env>; protect prod with required reviewers and self-review prevention. - Build once; promote the image digest through every environment.
- SCPs make "only the pipeline changes prod" an organisation-level rule, not a convention.
- Keep production data in production; synthetic data everywhere else.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Rollback strategies: code, configuration and data
Decide when to roll back and when to roll forward, wire automatic rollback for ECS, Lambda and CloudFormation, use feature flags as the fastest back-out, and change a banking database schema with expand/contract migrations that never block a rollback.
Lab 9.2: an OIDC deploy role that a fork cannot assume
Evaluate real GitHub OIDC tokens against IAM trust policies offline, prove that a fork, a pull request, another repository and the wrong audience are all refused, lint dangerous policies, then create the provider and three least-privilege roles with Terraform.