Infrastructure automation in the pipeline: CDK and Terraform with approval gates
Run CDK and Terraform from GitHub Actions the way a bank needs: plan on every pull request with a read-only role, apply only from main after a protected-environment approval, apply exactly the plan that was reviewed, and detect drift.
Assumes: .NET 10 LTS · Amazon.CDK.Lib 2.x · Cdklabs.CdkNag 3.x · Terraform 1.14 · AWS provider 6.x · GitHub Actions · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain why infrastructure changes need a plan, review, apply flow in a pipeline, and why applying a different plan than the one reviewed is a control failure.
- Build a GitHub Actions workflow that runs
cdk difforterraform planon a pull request with a read-only OIDC role and posts the result for reviewers. - Build the apply workflow: it runs from
mainonly, waits for a protected environment approval, uses a separate write role and applies a saved plan. - Configure Terraform remote state in S3 with native locking, and explain what must be protected around it.
- Add policy-as-code gates (cdk-nag, Checkov or Open Policy Agent) that fail the pipeline before any apply.
- Detect configuration drift on a schedule and decide what to do about it.
Prerequisites
- Environment separation — one AWS account per environment and cross-account deploy roles.
- GitHub Actions with OIDC — roles per repository and environment, no stored keys.
- Module 10: Infrastructure as code — CDK and Terraform fundamentals (can be studied in parallel).
- Tools: AWS CLI v2, .NET 10 SDK, Node.js (for the CDK CLI), Terraform 1.10 or later; Docker is optional.
Why this matters in banking
Application code gets reviewed. Infrastructure often does not, and yet one line of infrastructure can open a database to the internet, turn off encryption or delete a log bucket. Two incidents come up again and again in post-mortems:
- "It was a small change." Someone ran
terraform applyfrom a laptop with a local state file, against production, after "just a quick fix" in the console. The plan was never reviewed, and the state no longer matched reality for the next person. - "The plan said something different." A pull request showed a harmless plan. By the time the apply ran, the base branch had moved and the apply deleted a security group rule that other services depended on.
A bank's change-management policy usually says the same thing in more formal words: the change that is approved must be the change that is applied, by an identity that is not a person, with a record. This lesson turns that sentence into workflow files.
The idea in plain language
Infrastructure as code (IaC) describes cloud resources in files. Two tools matter in this course: AWS CDK (you write C#, it produces AWS CloudFormation templates) and Terraform (you write HCL, it talks to AWS APIs directly and remembers what it built in a state file).
Both have a "preview, then execute" pair of commands:
| Step | CDK | Terraform | What it does |
|---|---|---|---|
| Build | cdk synth | terraform validate | Compile your code into something checkable; no AWS changes |
| Preview | cdk diff | terraform plan | Compare desired state with what exists; no changes |
| Execute | cdk deploy | terraform apply | Make the changes |
A pipeline automates the pair and puts people and policy between them:
- On a pull request, run build and preview. A reviewer reads the preview, not just the code.
- After the merge, on
main, save the plan (or the cloud assembly), wait for approval in a protected environment, then execute exactly that. - On a schedule, preview again with no code change. Any difference is drift.
Analogy: a building-works permit. The architect submits drawings (the plan). The inspector approves those drawings. The builder must build those drawings, and an inspector returns later to check the building still matches them. Nobody is allowed to approve one set of drawings and build another.
What problem the AWS services and tools solve
| Tool | Role in the pipeline | Limits and sharp edges |
|---|---|---|
| AWS CDK | Synthesises CloudFormation; cdk diff shows template and IAM-impact changes | Deployment goes through CloudFormation and the CDK bootstrap roles in the target account; each account and Region must be bootstrapped |
| AWS CloudFormation | Executes changes with rollback; supports change sets and stack policies | A change set is the CloudFormation form of a plan; stack rollback does not undo data deletion |
| Terraform | Plan and apply against AWS APIs with remote state | State can contain secret values in clear text; state locking and access control are your responsibility |
| S3 backend | Stores Terraform state with versioning, encryption and locking | Native S3 locking (use_lockfile) needs a recent Terraform ⚠ VERIFY the minimum version and whether DynamoDB locking is deprecated in your version |
| cdk-nag | Checks CDK constructs against rule packs (for example AWS Solutions) at synth time | Rules produce noise; suppressions must be reviewed. v3 uses an acknowledgment API, not the older suppressions API |
| Checkov, tfsec-style scanners, OPA/Conftest | Policy checks on Terraform plans and templates | Tools differ in rule sets; pin versions, treat policy as code with its own review |
| IAM Access Analyzer policy checks | Validate IAM policy documents (for example CheckNoNewAccess) | Requires AWS credentials, so run in a job with a read-only role ⚠ VERIFY exact CLI names |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS / pipeline equivalent | What's different |
|---|---|---|
Bicep or ARM what-if | cdk diff, terraform plan, CloudFormation change set | Same idea. Terraform's saved plan file is a stronger guarantee than ARM what-if, which only previews |
| ARM deployment stacks / deny assignments | CloudFormation stack policies, SCPs | Stack policies stop updates to named resources; SCPs stop whole classes of actions in an account |
| Azure DevOps environment approvals | GitHub environment with required reviewers | Approval is attached to the job that applies, and to the OIDC subject |
| Azure Policy (audit/deny at deployment) | cdk-nag, Checkov, OPA before apply; AWS Config and SCPs after | Policy runs in the pipeline (shift left) and again in the account (defence in depth) |
Terraform with azurerm backend (blob + lease) | Terraform S3 backend with native locking | Same pattern, different storage; both need strict access control to the state |
| Service connection with Contributor on the subscription | Separate plan (read-only) and apply (write) roles per environment | Least privilege splits previewing from changing |
Architecture
Security-boundary view
The two roles differ in who can assume them, not just in what they can do. The plan role's trust policy accepts the pull-request subject; the apply role's accepts only the protected environment. A pull request from a fork cannot even obtain the plan role if you restrict trust to your repository's subjects.
Step-by-step implementation
Create the state bucket and the roles (once per account)
For Terraform, create an S3 bucket with versioning, KMS encryption, public access blocked and a TLS-only bucket policy. For both tools, create two OIDC roles per environment: hl-iac-plan and hl-iac-apply. The OIDC details are in GitHub Actions with OIDC; the one rule to add here is that the plan role is trusted by pull-request jobs and the apply role only by the protected environment.
Make formatting, validation and policy checks mandatory
These need no AWS credentials, so they run first and fail fast: terraform fmt -check, terraform validate, cdk synth, cdk-nag (inside synth), Checkov on the Terraform directory.
Run the preview on every pull request
Use the plan role, run terraform plan -out=tfplan or cdk diff, convert the result to text and post it as a pull-request comment or job summary. Make the check required in branch protection.
Require infrastructure owners as reviewers
Add /infra/ and the workflow files to CODEOWNERS so that a platform or security engineer must approve changes (see Git and GitHub for regulated teams). IAM, KMS, network and logging changes deserve a second pair of eyes whatever the size.
Apply from main, in an environment, using the saved plan
The apply job needs the plan job, declares environment: prod (so required reviewers must approve) and runs terraform apply tfplan. Terraform refuses to apply a saved plan if the state has changed since the plan was made, which closes the gap in the second incident above.
Schedule drift detection
A nightly job runs the plan against main with no changes and exits non-zero when it finds any. The job posts to a channel or opens an issue. A person decides: re-apply (undo the manual change) or codify it.
Record the evidence
Keep the plan text, the approver and the apply log as build artefacts or in the evidence bucket from Release management and audit evidence. An auditor asking "who changed this security group and who approved it" should find the run in under a minute.
AWS Console instructions
Most of this lesson is pipeline code, but you do the one-time set-up and the inspection in the console.
State bucket: Console → S3 → Create bucket. Name hl-tfstate-<account-id>; Bucket Versioning: Enable; Default encryption: SSE-KMS with a customer managed key; Block all public access: on. Add tags Project=cea-lab, Lab=m09-iac, Environment=learn.
Roles: Console → IAM → Roles → Create role → Web identity; provider token.actions.githubusercontent.com, audience sts.amazonaws.com. Create hl-iac-plan, then edit the trust policy to the pull-request and branch subjects (see the OIDC lesson for the exact sub values). Attach ReadOnlyAccess plus an inline policy for the state bucket (s3:GetObject, s3:ListBucket, and for locking s3:PutObject, s3:DeleteObject on the lock file only).
Inspect a CDK change set: Console → CloudFormation → Stacks → your stack → Change sets. A cdk deploy creates a change set, shows what will be replaced or deleted, and executes it.
Environment approvals: in GitHub, Repository → Settings → Environments → prod → Required reviewers, enable Prevent self-review, and restrict deployment branches to main.
AWS CLI
P=learn-sandbox
# 1. Create the state bucket (versioned and encrypted). The name must be globally unique.
ACCOUNT=$(aws sts get-caller-identity --query Account --output text --profile $P)
aws s3api create-bucket --bucket hl-tfstate-$ACCOUNT --region us-east-1 --profile $P
aws s3api put-bucket-versioning --bucket hl-tfstate-$ACCOUNT \
--versioning-configuration Status=Enabled --profile $P
aws s3api put-public-access-block --bucket hl-tfstate-$ACCOUNT \
--public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true \
--profile $P
# 2. Bootstrap CDK in the account (creates the deploy, file-publishing and CloudFormation execution roles).
npx cdk bootstrap aws://$ACCOUNT/us-east-1 --profile $P
# 3. Preview a CDK change as a CloudFormation change set without executing it.
npx cdk deploy --no-execute --profile $P # creates the change set only
# 4. List change sets for a stack and read one.
aws cloudformation list-change-sets --stack-name Hl-M09-Iac --profile $P
aws cloudformation describe-change-set --stack-name Hl-M09-Iac --change-set-name <name> --profile $P
# 5. Terraform: init, plan to a file, show it as JSON for policy checks, apply that exact file.
terraform init -backend-config=backend.hcl
terraform plan -out=tfplan -input=false
terraform show -json tfplan > tfplan.json
terraform apply -input=false tfplan
# 6. Drift check: exit code 0 = no changes, 2 = changes present, 1 = error.
terraform plan -detailed-exitcode -input=false; echo "exit=$?"Infrastructure as Code
The pipeline itself is the topic, so the IaC here has two parts: the state backend and the roles the pipeline uses.
CDK needs no state backend: CloudFormation holds the state. What you must create is the plan role and apply role. Here they are in C#, with a permission boundary on the apply role so even a compromised pipeline cannot create administrators.
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Constructs;
namespace Harbourline.Delivery.Infra;
public sealed class IacPipelineRoles : Construct
{
public Role Plan { get; }
public Role Apply { get; }
public IacPipelineRoles(Construct scope, string id, string repo, string accountId) : base(scope, id)
{
var provider = OpenIdConnectProvider.FromOpenIdConnectProviderArn(this, "GitHub",
$"arn:aws:iam::{accountId}:oidc-provider/token.actions.githubusercontent.com");
// Plan role: trusted by the repository's pull requests and main. Read-only.
Plan = new Role(this, "Plan", new RoleProps
{
RoleName = "hl-iac-plan",
MaxSessionDuration = Duration.Hours(1),
AssumedBy = new WebIdentityPrincipal(provider.OpenIdConnectProviderArn, new Dictionary<string, object>
{
["StringEquals"] = new Dictionary<string, object>
{
["token.actions.githubusercontent.com:aud"] = "sts.amazonaws.com",
},
["StringLike"] = new Dictionary<string, object>
{
// pull_request and the main branch: two exact subjects, not a wildcard on the repo.
["token.actions.githubusercontent.com:sub"] = new[]
{
$"repo:{repo}:pull_request",
$"repo:{repo}:ref:refs/heads/main",
},
},
}),
ManagedPolicies = [ManagedPolicy.FromAwsManagedPolicyName("ReadOnlyAccess")],
});
// Apply role: ONLY the protected environment can assume it.
var boundary = new ManagedPolicy(this, "ApplyBoundary", new ManagedPolicyProps
{
Statements =
[
new PolicyStatement(new PolicyStatementProps
{
Effect = Effect.ALLOW,
Actions = ["s3:*", "dynamodb:*", "ecs:*", "elasticloadbalancing:*", "logs:*", "cloudwatch:*", "ec2:*"],
Resources = ["*"],
}),
new PolicyStatement(new PolicyStatementProps
{
Effect = Effect.DENY,
Actions = ["iam:CreateUser", "iam:CreateAccessKey", "organizations:*", "account:*"],
Resources = ["*"],
}),
],
});
Apply = new Role(this, "Apply", new RoleProps
{
RoleName = "hl-iac-apply",
MaxSessionDuration = Duration.Hours(1),
PermissionsBoundary = boundary,
AssumedBy = new WebIdentityPrincipal(provider.OpenIdConnectProviderArn, new Dictionary<string, object>
{
["StringEquals"] = new Dictionary<string, object>
{
["token.actions.githubusercontent.com:aud"] = "sts.amazonaws.com",
["token.actions.githubusercontent.com:sub"] = $"repo:{repo}:environment:prod",
},
}),
});
}
}For CDK itself, the apply role does not need broad rights: cdk deploy assumes the bootstrap roles. Give the apply role only sts:AssumeRole on the cdk-hnb659fds-* roles in the account (the default bootstrap qualifier is hnb659fds) ⚠ VERIFY the qualifier in your account.
Complete code
The pull-request workflow
name: iac-plan
on:
pull_request:
paths: ["infra/**", ".github/workflows/iac-*.yml"]
permissions:
contents: read
id-token: write # OIDC token for the plan role
pull-requests: write # post the plan as a comment
concurrency:
group: iac-plan-${{ github.ref }}
cancel-in-progress: true
jobs:
checks:
name: Format, validate, policy
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: hashicorp/setup-terraform@<pin-a-full-commit-sha> # ⚠ VERIFY the current release and pin by SHA
with:
terraform_version: "1.14.0"
- run: terraform -chdir=infra/terraform fmt -check -recursive
- run: terraform -chdir=infra/terraform init -backend=false
- run: terraform -chdir=infra/terraform validate
- name: Checkov (policy as code) on the Terraform files
run: |
pip install checkov==<pinned-version>
checkov -d infra/terraform --framework terraform --quiet --compact
plan:
name: Terraform plan (read-only role)
needs: checks
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
with:
role-to-assume: ${{ vars.IAC_PLAN_ROLE_ARN }}
role-session-name: gh-${{ github.run_id }}-plan
aws-region: us-east-1
- uses: hashicorp/setup-terraform@<pin-a-full-commit-sha> # ⚠ VERIFY
with:
terraform_version: "1.14.0"
terraform_wrapper: false
- name: Init and plan
working-directory: infra/terraform
run: |
terraform init -input=false -backend-config=backend.hcl
terraform plan -input=false -lock=false -out=tfplan
terraform show -no-color tfplan > plan.txt
terraform show -json tfplan > plan.json
- name: Summarise the plan for reviewers
run: |
{
echo "## Terraform plan"
echo '```'
head -c 60000 infra/terraform/plan.txt
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- name: Block destructive changes to protected resource types
run: |
jq -e '[.resource_changes[] | select(.change.actions | index("delete"))
| select(.type | test("aws_kms_key|aws_s3_bucket$|aws_cloudwatch_log_group|aws_db_instance"))] | length == 0' \
infra/terraform/plan.json
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: pr-plan
path: |
infra/terraform/plan.txt
infra/terraform/plan.json
retention-days: 30The plan from a pull request is for people to read. It is not the one applied: the base branch may move before merge.
The apply workflow
name: iac-apply
on:
push:
branches: [main]
paths: ["infra/**"]
permissions:
contents: read
concurrency:
group: iac-apply-prod
cancel-in-progress: false # never cancel an apply half-way
jobs:
plan:
name: Plan on main and save it
runs-on: ubuntu-latest
timeout-minutes: 20
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
with:
role-to-assume: ${{ vars.IAC_PLAN_ROLE_ARN }}
aws-region: us-east-1
- uses: hashicorp/setup-terraform@<pin-a-full-commit-sha> # ⚠ VERIFY
with:
terraform_version: "1.14.0"
terraform_wrapper: false
- working-directory: infra/terraform
run: |
terraform init -input=false -backend-config=backend.hcl
terraform plan -input=false -out=tfplan
terraform show -no-color tfplan > plan.txt
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: main-plan
path: |
infra/terraform/tfplan
infra/terraform/plan.txt
retention-days: 90
apply:
name: Apply the saved plan (approval required)
needs: plan
runs-on: ubuntu-latest
timeout-minutes: 60
environment: prod # required reviewers; subject becomes repo:...:environment:prod
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: main-plan
path: infra/terraform
- uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
with:
role-to-assume: ${{ vars.IAC_APPLY_ROLE_ARN }}
role-session-name: gh-${{ github.run_id }}-apply
aws-region: us-east-1
- uses: hashicorp/setup-terraform@<pin-a-full-commit-sha> # ⚠ VERIFY
with:
terraform_version: "1.14.0"
terraform_wrapper: false
- name: Apply exactly the plan the reviewer saw
working-directory: infra/terraform
run: |
terraform init -input=false -backend-config=backend.hcl
terraform apply -input=false tfplanThe reviewer who approves the prod environment sees the plan job's output in the run before approving. Terraform refuses tfplan if the state changed since it was created ("saved plan is stale"), so a concurrent change cannot sneak in.
The CDK variant
name: cdk-deploy
on:
push:
branches: [main]
paths: ["infra/cdk/**"]
permissions:
contents: read
jobs:
diff:
runs-on: ubuntu-latest
permissions: { contents: read, id-token: write }
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with: { persist-credentials: false }
- uses: actions/setup-dotnet@0f4a354dd5547a28c70b7853dbd95d5b67886e73 # v6.1.0
with: { global-json-file: global.json }
- uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
with:
role-to-assume: ${{ vars.IAC_PLAN_ROLE_ARN }}
aws-region: us-east-1
- name: Synth (cdk-nag runs here and fails on errors) and diff
working-directory: infra/cdk
run: |
npx cdk@2 synth --quiet
npx cdk@2 diff --fail 2>&1 | tee diff.txt || true # diff exits 1 when there are differences; reviewers read diff.txt
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with: { name: cdk-diff, path: infra/cdk/diff.txt }
deploy:
needs: diff
runs-on: ubuntu-latest
environment: prod
permissions: { contents: read, id-token: write }
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with: { persist-credentials: false }
- uses: actions/setup-dotnet@0f4a354dd5547a28c70b7853dbd95d5b67886e73 # v6.1.0
with: { global-json-file: global.json }
- uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
with:
role-to-assume: ${{ vars.IAC_APPLY_ROLE_ARN }}
aws-region: us-east-1
- name: Deploy (broadening IAM or security-group changes still prompt; no --require-approval never)
working-directory: infra/cdk
run: npx cdk@2 deploy --all --require-approval broadening --ciA note on --require-approval: in a non-interactive pipeline a prompt cannot be answered, so a broadening change fails the job instead of silently proceeding. That is the safe behaviour for IAM and security-group widening; resolve it by reviewing the diff and, for approved changes, redesigning so the change is explicit. Use never only for dev accounts.
Code explained line by line
on: pull_requestwithpaths:limits the workflow to infrastructure changes. Application-only changes do not pay for a plan.permissions: id-token: writelets the job request a GitHub OIDC token. It grants nothing in AWS by itself; the role's trust policy decides.terraform_wrapper: falsestops the setup action wrapping Terraform's output, which matters when you pipeterraform show -json.terraform plan -out=tfplanwrites the plan to a file.terraform show -json tfplanrenders it for machines: policy tools and thejqdestructive-change check read this form.- The
jq -eexpression exits non-zero when a plan deletes a KMS key, a bucket, a log group or a database. A human can still approve such a change, but not by accident; add an explicit label or ticket check if you want the exception recorded. - In
iac-apply.yml,concurrencywithcancel-in-progress: falseserialises applies and never kills one mid-run. environment: prodon the apply job is what pauses it for reviewers and changes the OIDCsubtorepo:<owner>/<repo>:environment:prod, which only the apply role trusts.terraform apply -input=false tfplanapplies the file, not the code; it does not re-plan.- In the CDK workflow,
cdk diff --failexits non-zero if differences exist; the|| truekeeps the job going so reviewers can read the output. In the C# stacks, cdk-nag runs duringsynthas an aspect; errors fail the job before any AWS call.
Security considerations
⚠ Learning shortcut — not for production
The lab-style examples keep the plan role trusted by main and use -lock=false for pull-request plans to avoid giving a read-only role write access to the lock file, run Terraform with a single state for a whole environment and let the apply role keep broad service permissions inside a boundary. Production: one state per bounded context (network, security, workloads), the apply role scoped to what that state manages, state in a dedicated account with access limited to the pipeline roles, and a reviewed break-glass process.
- The state file is sensitive. It can hold secrets in clear text (database passwords, keys). Encrypt it with a customer managed KMS key, restrict who can read it, and keep secrets out of Terraform where possible; use Secrets Manager-managed passwords instead. See Secrets in CI/CD.
- Pull requests from forks. They must not receive the plan role. Either disable workflows for fork PRs that touch
infra/, require approval for fork workflows, or use trust conditions that only name your repository's own subjects. Never usepull_request_targetto check out and run untrusted code. - Plan output can leak data. Mark sensitive variables
sensitive = true; plan text goes into logs and comments. - Treat the plan as an input to review, not a rubber stamp. Large plans hide dangerous lines. Cap plan size or split states so reviewers can actually read what they approve.
- Stack policies and SCPs. A CloudFormation stack policy can deny updates to critical resources (a log bucket, a KMS key) unless an explicit override is supplied; an organisation SCP can deny deleting CloudTrail or disabling encryption entirely, as a backstop to anything the pipeline does.
- Break-glass. Define who may apply from outside the pipeline in an emergency, how it is monitored (CloudTrail alert on console or CLI changes by humans in prod) and how the change is reconciled into code afterwards.
- Compliance. These controls produce evidence of review and approval; they do not make an environment compliant. Your risk team defines what is required.
Banking use case
Harbourline's platform team owns the network and logging stacks used by all payment services. Policy: no human has write access to production infrastructure. Engineers open pull requests; the plan role posts the plan; CODEOWNERS require a platform engineer and, for IAM or KMS changes, a security engineer. On merge, the pipeline re-plans, and a release manager approves the prod environment. The apply role, bound to a permission boundary that forbids creating IAM users or access keys, applies the saved plan. A nightly drift job pages the platform channel if anyone changed anything by hand; the break-glass permission set is monitored, and every use produces a ticket that must end in a pull request. Examiners ask "show me how a security group in production got its current rule"; the answer is a pull request, a plan, an approver and a run.
Common mistakes
- Applying code, not the plan.
terraform applywithout a plan file re-plans at apply time; what runs may differ from what was reviewed. Always save and apply the plan. - One role for everything. A pull-request job with write access means any contributor can change production by editing a workflow. Separate roles, separate trust conditions.
- State on a laptop or in a public bucket. Remote, encrypted, versioned, private; with locking.
cdk deploy --require-approval neverin prod to "fix the pipeline hanging". The prompt was telling you something about IAM.- Suppressing cdk-nag rules without a reason. Every acknowledgment needs a justification and a reviewer.
- Drift ignored. If the nightly job is red for three weeks, people stop trusting it. Triage or delete the manual change.
- Pipeline deploys infra and app in one job. Keep separate cadences. A slow infrastructure approval should not block a code fix.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Not authorized to perform sts:AssumeRoleWithWebIdentity in the plan job | Subject does not match the plan role's trust (for example pull_request vs branch) | Print the token sub; compare with the trust policy |
Error acquiring the state lock | Another run holds the lock or a crashed run left a lock object | Wait; if certain no run is active, terraform force-unlock <id> after confirming with the team |
Saved plan is stale on apply | State changed between plan and apply (another apply or drift) | Re-run the plan job and approve the new plan |
cdk diff fails with Need to perform AWS calls... lookup | Missing lookup role permission or context values not committed | Allow sts:AssumeRole on the lookup role; commit cdk.context.json |
CDK deploy fails: This deployment will make potentially sensitive changes | --require-approval broadening in a CI session | Review and, if approved, adjust the change; do not switch to never in prod |
| Checkov fails on a resource you must create | Genuine finding or false positive | Fix, or add a suppression with a ticket reference in a comment and CODEOWNERS review |
| Plan shows replacement of a resource nobody changed | Provider upgrade changed defaults, or drift | Pin provider versions; read the changed attribute in the plan |
Terraform cannot read state: AccessDenied | Plan role lacks bucket or KMS access | Grant kms:Decrypt on the state key and s3:GetObject/s3:ListBucket |
Cost and cleanup
Cost: free. GitHub-hosted runners for private repositories use your plan's included minutes ⚠ VERIFY current allowance; public repositories are free. A state bucket with a few small objects costs cents per month. IAM roles, OIDC providers, CloudFormation and CDK bootstrap have no charge; the resources you deploy do. The exercises below use only an S3 bucket and an IAM role, so there is nothing to bill beyond storage.
Teardown:
P=learn-sandbox
# Terraform-managed resources
terraform destroy
# State bucket: versioned, so delete all versions before the bucket
aws s3api list-object-versions --bucket hl-tfstate-$ACCOUNT --profile $P
aws s3 rb s3://hl-tfstate-$ACCOUNT --force --profile $P # deletes current objects; for versioned buckets remove versions and delete markers first
# CDK stacks
npx cdk destroy --all --profile $P
# Roles created for the lab
aws iam delete-role-policy --role-name hl-iac-plan --policy-name state-access --profile $P 2>/dev/null
aws iam detach-role-policy --role-name hl-iac-plan --policy-arn arn:aws:iam::aws:policy/ReadOnlyAccess --profile $P
aws iam delete-role --role-name hl-iac-plan --profile $P
aws iam delete-role --role-name hl-iac-apply --profile $P
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m09-iac --profile $P --region us-east-1Free alternative: do the design exercises and run terraform validate, terraform fmt -check and terraform plan against a local backend in Docker (hashicorp/terraform:1.14) without any AWS credentials, using a provider configuration with skip_credentials_validation for a read-free sample ⚠ VERIFY the flags for your provider version.
Exercises
- 🟢 Guided: Write the
iac-plan.ymlworkflow for a Terraform folder that creates one S3 bucket. Make fmt, validate and the plan steps required checks. Open a pull request in your own sandbox repository and read the job summary. - 🟡 Challenge: Add a job that fails when the plan deletes any resource of type
aws_s3_bucketoraws_kms_key, then prove it by writing a plan that removes a bucket. Add a shortREADMEnote explaining who may approve an exception and how it is recorded. - 🔴 Live problem: Ticket CHG-104388: "A security group in prod now allows 0.0.0.0/0 on port 5432. Nobody has a change record. Last night's drift job was red; it has been red for a month." Describe your first hour: stop the exposure, find who and how, bring the code and the account back in line, and what pipeline changes prevent a repeat.
Assignment
Add an infra/ pipeline to your portfolio repository: plan on pull request with a read-only role, apply on main after an environment approval using a saved plan, plus a nightly drift workflow. Include a short docs/infra-change-process.md that says who reviews what, how emergencies are handled and how plans are retained.
Solutions
Use the checks and plan jobs from the pull-request workflow above with your bucket module, set the repository variable for the plan role ARN, and in Settings → Rules (or branch protection) mark Format, validate, policy and Terraform plan (read-only role) as required status checks. The summary shows + aws_s3_bucket.records as a create. Why required checks: without them, a plan that fails does not stop a merge.
The jq -e step shown in the pull-request workflow does it: select resource_changes whose change.actions includes delete and whose type matches the protected list, then assert the count is zero. To test, remove the bucket from the Terraform file; the step fails with exit code 1. For the exception process: a label infra-delete-approved plus a CODEOWNERS review from security, and the check reads the label through the GitHub API; the approved exception is then visible in the pull request history.
Minute 0 to 15, stop the exposure. Apply the known good configuration from main or, if faster and approved by the incident lead, remove the rule through the break-glass permission set; record who did it. Check VPC Flow Logs or ALB and database logs for connections from unknown sources during the window. Find the cause. CloudTrail: AuthorizeSecurityGroupIngress for that group, who and from where (console, CLI, role). Reconcile. Run the plan: it should show the rule being removed; if the change was legitimate it belongs in code through a pull request. Prevent. Remove human write access in prod (SCP plus permission sets), alert on any non-pipeline identity changing security groups, add an AWS Config rule or SCP for 0.0.0.0/0 on database ports, and make the drift job page a person instead of staying silent. Never claim the exposure was harmless without evidence from logs.
Knowledge check
Answer: Without a plan file, apply re-plans against the current state and code, so what runs may differ from what the reviewer saw. A saved plan is exactly what was approved, and Terraform rejects it if the state has changed since.
Answer: The protected environment subject, repo:OWNER/REPO:environment:prod, so only a job that passed the environment's required reviewers can assume it. Trusting a branch alone would let any workflow change that runs on that branch bypass the approval.
Answer: Exit code 2 (0 means no changes, 1 means an error). That is the signal a drift job uses.
Answer: It disables the safety check for IAM and security-group broadening, so widening access needs no explicit confirmation, and in a pipeline nobody sees it. Use broadening in prod and review the diff instead.
Answer: State can contain secrets in clear text, and anyone who can write state can make Terraform believe resources exist or not, changing what the next apply does. Version it, encrypt it and limit it to the pipeline roles.
Interview questions
- How do you make sure the change reviewed is the change applied? Save the plan on main, apply that file in an approved environment; Terraform rejects a stale plan. For CDK, deploy the synthesised assembly or re-diff in the same run and restrict approval prompts.
- Why two roles, plan and apply? Previews run on every pull request, including from less-trusted contexts, so they get read-only access. Writes are reserved for the protected environment.
- How do you handle drift? A scheduled plan with exit code detection, an owner for each alert, and a rule: either revert the manual change or codify it through a pull request.
- Where do policy checks belong? Before apply (cdk-nag, Checkov, OPA on the plan JSON), and again in the account (SCPs, AWS Config) as defence in depth.
- What do you do with secrets in Terraform state? Avoid them, use managed passwords, encrypt and lock down the state bucket.
Certification practice
A company wants infrastructure changes reviewed before they are applied and wants the applied change to be identical to the reviewed one. They use Terraform in GitHub Actions. Which design satisfies this?
A. Run terraform apply -auto-approve on every merge to main.
B. Run terraform plan on pull requests; on main, save a plan, require environment approval, then apply the saved plan.
C. Give developers the apply role so they can run apply from laptops after review.
D. Run terraform apply in a pull-request workflow using a read-only role.
Answer: B. The saved plan guarantees identical changes and the environment adds approval. A skips review of the final plan, C bypasses the pipeline and audit trail, and D cannot work with a read-only role and applies unreviewed code.
A bank deploys CDK stacks into separate accounts from a central pipeline. Which approach follows least privilege?
A. Store an administrator access key for each account in the CI system. B. Use OIDC to assume a deploy role per account that can only assume that account's CDK bootstrap roles. C. Give the CI role AdministratorAccess in every account. D. Deploy from an EC2 instance with an admin instance profile.
Answer: B. Short-lived credentials, scoped to deploy through bootstrap roles. A and C are broad or long-lived; D concentrates admin power in a long-running machine.
A scheduled job runs terraform plan -detailed-exitcode nightly and returns exit code 2. What does that indicate?
A. The plan failed with an error. B. There are no changes. C. The deployed infrastructure differs from the code; there are changes to apply. D. The state is locked.
Answer: C. Exit 0 means no changes, 1 an error, 2 differences. A lock error returns 1.
Completion checklist
- I can explain the plan, review, apply flow and the saved-plan guarantee
- I wrote a pull-request workflow with a read-only role and a required check
- I wrote an apply workflow bound to a protected environment
- I added at least one policy-as-code gate that fails the build
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- The change approved must be the change applied: save the plan and apply that file.
- Separate roles for previewing and changing, bound by different OIDC subjects.
- Policy gates run before apply; SCPs and Config catch what slips through.
- State is sensitive: encrypt, version, lock and restrict it.
- Drift detection turns manual changes into visible, owned work.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Pipeline security: protect the machine that deploys your bank
Treat the delivery pipeline as a privileged, attackable system: stop script injection and untrusted pull requests, pin actions and base images, scope tokens and IAM roles, sign and attest artefacts, produce SBOMs, and audit workflows with a small C# linter plus zizmor.
Release management and audit evidence: from ticket to commit to deployment
Turn change approval, segregation of duties and traceability into pipeline code: a release record that links ticket, pull request, commit, image digest, approvals and scans, a release policy that blocks bad releases, and tamper-resistant evidence storage.