Skip to Content
Cloud Engineering Academy
9. DevOps & CI/CD

Infrastructure automation in the pipeline: CDK and Terraform with approval gates

Run CDK and Terraform from GitHub Actions the way a bank needs: plan on every pull request with a read-only role, apply only from main after a protected-environment approval, apply exactly the plan that was reviewed, and detect drift.

advanced⏱ 1 h 45 min🟢 FreeDOP-C02SAP-C02

Assumes: .NET 10 LTS · Amazon.CDK.Lib 2.x · Cdklabs.CdkNag 3.x · Terraform 1.14 · AWS provider 6.x · GitHub Actions · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain why infrastructure changes need a plan, review, apply flow in a pipeline, and why applying a different plan than the one reviewed is a control failure.
  • Build a GitHub Actions workflow that runs cdk diff or terraform plan on a pull request with a read-only OIDC role and posts the result for reviewers.
  • Build the apply workflow: it runs from main only, waits for a protected environment approval, uses a separate write role and applies a saved plan.
  • Configure Terraform remote state in S3 with native locking, and explain what must be protected around it.
  • Add policy-as-code gates (cdk-nag, Checkov or Open Policy Agent) that fail the pipeline before any apply.
  • Detect configuration drift on a schedule and decide what to do about it.

Prerequisites

Why this matters in banking

Application code gets reviewed. Infrastructure often does not, and yet one line of infrastructure can open a database to the internet, turn off encryption or delete a log bucket. Two incidents come up again and again in post-mortems:

  1. "It was a small change." Someone ran terraform apply from a laptop with a local state file, against production, after "just a quick fix" in the console. The plan was never reviewed, and the state no longer matched reality for the next person.
  2. "The plan said something different." A pull request showed a harmless plan. By the time the apply ran, the base branch had moved and the apply deleted a security group rule that other services depended on.

A bank's change-management policy usually says the same thing in more formal words: the change that is approved must be the change that is applied, by an identity that is not a person, with a record. This lesson turns that sentence into workflow files.

The idea in plain language

Infrastructure as code (IaC) describes cloud resources in files. Two tools matter in this course: AWS CDK (you write C#, it produces AWS CloudFormation templates) and Terraform (you write HCL, it talks to AWS APIs directly and remembers what it built in a state file).

Both have a "preview, then execute" pair of commands:

StepCDKTerraformWhat it does
Buildcdk synthterraform validateCompile your code into something checkable; no AWS changes
Previewcdk diffterraform planCompare desired state with what exists; no changes
Executecdk deployterraform applyMake the changes

A pipeline automates the pair and puts people and policy between them:

  1. On a pull request, run build and preview. A reviewer reads the preview, not just the code.
  2. After the merge, on main, save the plan (or the cloud assembly), wait for approval in a protected environment, then execute exactly that.
  3. On a schedule, preview again with no code change. Any difference is drift.

Analogy: a building-works permit. The architect submits drawings (the plan). The inspector approves those drawings. The builder must build those drawings, and an inspector returns later to check the building still matches them. Nobody is allowed to approve one set of drawings and build another.

What problem the AWS services and tools solve

ToolRole in the pipelineLimits and sharp edges
AWS CDKSynthesises CloudFormation; cdk diff shows template and IAM-impact changesDeployment goes through CloudFormation and the CDK bootstrap roles in the target account; each account and Region must be bootstrapped
AWS CloudFormationExecutes changes with rollback; supports change sets and stack policiesA change set is the CloudFormation form of a plan; stack rollback does not undo data deletion
TerraformPlan and apply against AWS APIs with remote stateState can contain secret values in clear text; state locking and access control are your responsibility
S3 backendStores Terraform state with versioning, encryption and lockingNative S3 locking (use_lockfile) needs a recent Terraform ⚠ VERIFY the minimum version and whether DynamoDB locking is deprecated in your version
cdk-nagChecks CDK constructs against rule packs (for example AWS Solutions) at synth timeRules produce noise; suppressions must be reviewed. v3 uses an acknowledgment API, not the older suppressions API
Checkov, tfsec-style scanners, OPA/ConftestPolicy checks on Terraform plans and templatesTools differ in rule sets; pin versions, treat policy as code with its own review
IAM Access Analyzer policy checksValidate IAM policy documents (for example CheckNoNewAccess)Requires AWS credentials, so run in a job with a read-only role ⚠ VERIFY exact CLI names

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS / pipeline equivalentWhat's different
Bicep or ARM what-ifcdk diff, terraform plan, CloudFormation change setSame idea. Terraform's saved plan file is a stronger guarantee than ARM what-if, which only previews
ARM deployment stacks / deny assignmentsCloudFormation stack policies, SCPsStack policies stop updates to named resources; SCPs stop whole classes of actions in an account
Azure DevOps environment approvalsGitHub environment with required reviewersApproval is attached to the job that applies, and to the OIDC subject
Azure Policy (audit/deny at deployment)cdk-nag, Checkov, OPA before apply; AWS Config and SCPs afterPolicy runs in the pipeline (shift left) and again in the account (defence in depth)
Terraform with azurerm backend (blob + lease)Terraform S3 backend with native lockingSame pattern, different storage; both need strict access control to the state
Service connection with Contributor on the subscriptionSeparate plan (read-only) and apply (write) roles per environmentLeast privilege splits previewing from changing

Architecture

Security-boundary view

The two roles differ in who can assume them, not just in what they can do. The plan role's trust policy accepts the pull-request subject; the apply role's accepts only the protected environment. A pull request from a fork cannot even obtain the plan role if you restrict trust to your repository's subjects.

Step-by-step implementation

Create the state bucket and the roles (once per account)

For Terraform, create an S3 bucket with versioning, KMS encryption, public access blocked and a TLS-only bucket policy. For both tools, create two OIDC roles per environment: hl-iac-plan and hl-iac-apply. The OIDC details are in GitHub Actions with OIDC; the one rule to add here is that the plan role is trusted by pull-request jobs and the apply role only by the protected environment.

Make formatting, validation and policy checks mandatory

These need no AWS credentials, so they run first and fail fast: terraform fmt -check, terraform validate, cdk synth, cdk-nag (inside synth), Checkov on the Terraform directory.

Run the preview on every pull request

Use the plan role, run terraform plan -out=tfplan or cdk diff, convert the result to text and post it as a pull-request comment or job summary. Make the check required in branch protection.

Require infrastructure owners as reviewers

Add /infra/ and the workflow files to CODEOWNERS so that a platform or security engineer must approve changes (see Git and GitHub for regulated teams). IAM, KMS, network and logging changes deserve a second pair of eyes whatever the size.

Apply from main, in an environment, using the saved plan

The apply job needs the plan job, declares environment: prod (so required reviewers must approve) and runs terraform apply tfplan. Terraform refuses to apply a saved plan if the state has changed since the plan was made, which closes the gap in the second incident above.

Schedule drift detection

A nightly job runs the plan against main with no changes and exits non-zero when it finds any. The job posts to a channel or opens an issue. A person decides: re-apply (undo the manual change) or codify it.

Record the evidence

Keep the plan text, the approver and the apply log as build artefacts or in the evidence bucket from Release management and audit evidence. An auditor asking "who changed this security group and who approved it" should find the run in under a minute.

AWS Console instructions

Most of this lesson is pipeline code, but you do the one-time set-up and the inspection in the console.

State bucket: Console → S3 → Create bucket. Name hl-tfstate-<account-id>; Bucket Versioning: Enable; Default encryption: SSE-KMS with a customer managed key; Block all public access: on. Add tags Project=cea-lab, Lab=m09-iac, Environment=learn.

Roles: Console → IAM → Roles → Create role → Web identity; provider token.actions.githubusercontent.com, audience sts.amazonaws.com. Create hl-iac-plan, then edit the trust policy to the pull-request and branch subjects (see the OIDC lesson for the exact sub values). Attach ReadOnlyAccess plus an inline policy for the state bucket (s3:GetObject, s3:ListBucket, and for locking s3:PutObject, s3:DeleteObject on the lock file only).

Inspect a CDK change set: Console → CloudFormation → Stacks → your stack → Change sets. A cdk deploy creates a change set, shows what will be replaced or deleted, and executes it.

Environment approvals: in GitHub, Repository → Settings → Environments → prod → Required reviewers, enable Prevent self-review, and restrict deployment branches to main.

AWS CLI

P=learn-sandbox

# 1. Create the state bucket (versioned and encrypted). The name must be globally unique.
ACCOUNT=$(aws sts get-caller-identity --query Account --output text --profile $P)
aws s3api create-bucket --bucket hl-tfstate-$ACCOUNT --region us-east-1 --profile $P
aws s3api put-bucket-versioning --bucket hl-tfstate-$ACCOUNT \
  --versioning-configuration Status=Enabled --profile $P
aws s3api put-public-access-block --bucket hl-tfstate-$ACCOUNT \
  --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true \
  --profile $P

# 2. Bootstrap CDK in the account (creates the deploy, file-publishing and CloudFormation execution roles).
npx cdk bootstrap aws://$ACCOUNT/us-east-1 --profile $P

# 3. Preview a CDK change as a CloudFormation change set without executing it.
npx cdk deploy --no-execute --profile $P        # creates the change set only

# 4. List change sets for a stack and read one.
aws cloudformation list-change-sets --stack-name Hl-M09-Iac --profile $P
aws cloudformation describe-change-set --stack-name Hl-M09-Iac --change-set-name <name> --profile $P

# 5. Terraform: init, plan to a file, show it as JSON for policy checks, apply that exact file.
terraform init -backend-config=backend.hcl
terraform plan -out=tfplan -input=false
terraform show -json tfplan > tfplan.json
terraform apply -input=false tfplan

# 6. Drift check: exit code 0 = no changes, 2 = changes present, 1 = error.
terraform plan -detailed-exitcode -input=false; echo "exit=$?"

Infrastructure as Code

The pipeline itself is the topic, so the IaC here has two parts: the state backend and the roles the pipeline uses.

CDK needs no state backend: CloudFormation holds the state. What you must create is the plan role and apply role. Here they are in C#, with a permission boundary on the apply role so even a compromised pipeline cannot create administrators.

IacPipelineRoles.cs
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Constructs;

namespace Harbourline.Delivery.Infra;

public sealed class IacPipelineRoles : Construct
{
    public Role Plan { get; }
    public Role Apply { get; }

    public IacPipelineRoles(Construct scope, string id, string repo, string accountId) : base(scope, id)
    {
        var provider = OpenIdConnectProvider.FromOpenIdConnectProviderArn(this, "GitHub",
            $"arn:aws:iam::{accountId}:oidc-provider/token.actions.githubusercontent.com");

        // Plan role: trusted by the repository's pull requests and main. Read-only.
        Plan = new Role(this, "Plan", new RoleProps
        {
            RoleName = "hl-iac-plan",
            MaxSessionDuration = Duration.Hours(1),
            AssumedBy = new WebIdentityPrincipal(provider.OpenIdConnectProviderArn, new Dictionary<string, object>
            {
                ["StringEquals"] = new Dictionary<string, object>
                {
                    ["token.actions.githubusercontent.com:aud"] = "sts.amazonaws.com",
                },
                ["StringLike"] = new Dictionary<string, object>
                {
                    // pull_request and the main branch: two exact subjects, not a wildcard on the repo.
                    ["token.actions.githubusercontent.com:sub"] = new[]
                    {
                        $"repo:{repo}:pull_request",
                        $"repo:{repo}:ref:refs/heads/main",
                    },
                },
            }),
            ManagedPolicies = [ManagedPolicy.FromAwsManagedPolicyName("ReadOnlyAccess")],
        });

        // Apply role: ONLY the protected environment can assume it.
        var boundary = new ManagedPolicy(this, "ApplyBoundary", new ManagedPolicyProps
        {
            Statements =
            [
                new PolicyStatement(new PolicyStatementProps
                {
                    Effect = Effect.ALLOW,
                    Actions = ["s3:*", "dynamodb:*", "ecs:*", "elasticloadbalancing:*", "logs:*", "cloudwatch:*", "ec2:*"],
                    Resources = ["*"],
                }),
                new PolicyStatement(new PolicyStatementProps
                {
                    Effect = Effect.DENY,
                    Actions = ["iam:CreateUser", "iam:CreateAccessKey", "organizations:*", "account:*"],
                    Resources = ["*"],
                }),
            ],
        });

        Apply = new Role(this, "Apply", new RoleProps
        {
            RoleName = "hl-iac-apply",
            MaxSessionDuration = Duration.Hours(1),
            PermissionsBoundary = boundary,
            AssumedBy = new WebIdentityPrincipal(provider.OpenIdConnectProviderArn, new Dictionary<string, object>
            {
                ["StringEquals"] = new Dictionary<string, object>
                {
                    ["token.actions.githubusercontent.com:aud"] = "sts.amazonaws.com",
                    ["token.actions.githubusercontent.com:sub"] = $"repo:{repo}:environment:prod",
                },
            }),
        });
    }
}

For CDK itself, the apply role does not need broad rights: cdk deploy assumes the bootstrap roles. Give the apply role only sts:AssumeRole on the cdk-hnb659fds-* roles in the account (the default bootstrap qualifier is hnb659fds) ⚠ VERIFY the qualifier in your account.

Complete code

The pull-request workflow

.github/workflows/iac-plan.yml
name: iac-plan

on:
  pull_request:
    paths: ["infra/**", ".github/workflows/iac-*.yml"]

permissions:
  contents: read
  id-token: write        # OIDC token for the plan role
  pull-requests: write   # post the plan as a comment

concurrency:
  group: iac-plan-${{ github.ref }}
  cancel-in-progress: true

jobs:
  checks:
    name: Format, validate, policy
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: hashicorp/setup-terraform@<pin-a-full-commit-sha> # ⚠ VERIFY the current release and pin by SHA
        with:
          terraform_version: "1.14.0"
      - run: terraform -chdir=infra/terraform fmt -check -recursive
      - run: terraform -chdir=infra/terraform init -backend=false
      - run: terraform -chdir=infra/terraform validate
      - name: Checkov (policy as code) on the Terraform files
        run: |
          pip install checkov==<pinned-version>
          checkov -d infra/terraform --framework terraform --quiet --compact

  plan:
    name: Terraform plan (read-only role)
    needs: checks
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
        with:
          role-to-assume: ${{ vars.IAC_PLAN_ROLE_ARN }}
          role-session-name: gh-${{ github.run_id }}-plan
          aws-region: us-east-1
      - uses: hashicorp/setup-terraform@<pin-a-full-commit-sha> # ⚠ VERIFY
        with:
          terraform_version: "1.14.0"
          terraform_wrapper: false
      - name: Init and plan
        working-directory: infra/terraform
        run: |
          terraform init -input=false -backend-config=backend.hcl
          terraform plan -input=false -lock=false -out=tfplan
          terraform show -no-color tfplan > plan.txt
          terraform show -json tfplan > plan.json
      - name: Summarise the plan for reviewers
        run: |
          {
            echo "## Terraform plan"
            echo '```'
            head -c 60000 infra/terraform/plan.txt
            echo '```'
          } >> "$GITHUB_STEP_SUMMARY"
      - name: Block destructive changes to protected resource types
        run: |
          jq -e '[.resource_changes[] | select(.change.actions | index("delete"))
                  | select(.type | test("aws_kms_key|aws_s3_bucket$|aws_cloudwatch_log_group|aws_db_instance"))] | length == 0' \
            infra/terraform/plan.json
      - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
        with:
          name: pr-plan
          path: |
            infra/terraform/plan.txt
            infra/terraform/plan.json
          retention-days: 30

The plan from a pull request is for people to read. It is not the one applied: the base branch may move before merge.

The apply workflow

.github/workflows/iac-apply.yml
name: iac-apply

on:
  push:
    branches: [main]
    paths: ["infra/**"]

permissions:
  contents: read

concurrency:
  group: iac-apply-prod
  cancel-in-progress: false      # never cancel an apply half-way

jobs:
  plan:
    name: Plan on main and save it
    runs-on: ubuntu-latest
    timeout-minutes: 20
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
        with:
          role-to-assume: ${{ vars.IAC_PLAN_ROLE_ARN }}
          aws-region: us-east-1
      - uses: hashicorp/setup-terraform@<pin-a-full-commit-sha> # ⚠ VERIFY
        with:
          terraform_version: "1.14.0"
          terraform_wrapper: false
      - working-directory: infra/terraform
        run: |
          terraform init -input=false -backend-config=backend.hcl
          terraform plan -input=false -out=tfplan
          terraform show -no-color tfplan > plan.txt
      - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
        with:
          name: main-plan
          path: |
            infra/terraform/tfplan
            infra/terraform/plan.txt
          retention-days: 90

  apply:
    name: Apply the saved plan (approval required)
    needs: plan
    runs-on: ubuntu-latest
    timeout-minutes: 60
    environment: prod            # required reviewers; subject becomes repo:...:environment:prod
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
        with:
          name: main-plan
          path: infra/terraform
      - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
        with:
          role-to-assume: ${{ vars.IAC_APPLY_ROLE_ARN }}
          role-session-name: gh-${{ github.run_id }}-apply
          aws-region: us-east-1
      - uses: hashicorp/setup-terraform@<pin-a-full-commit-sha> # ⚠ VERIFY
        with:
          terraform_version: "1.14.0"
          terraform_wrapper: false
      - name: Apply exactly the plan the reviewer saw
        working-directory: infra/terraform
        run: |
          terraform init -input=false -backend-config=backend.hcl
          terraform apply -input=false tfplan

The reviewer who approves the prod environment sees the plan job's output in the run before approving. Terraform refuses tfplan if the state changed since it was created ("saved plan is stale"), so a concurrent change cannot sneak in.

The CDK variant

.github/workflows/cdk-deploy.yml
name: cdk-deploy

on:
  push:
    branches: [main]
    paths: ["infra/cdk/**"]

permissions:
  contents: read

jobs:
  diff:
    runs-on: ubuntu-latest
    permissions: { contents: read, id-token: write }
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with: { persist-credentials: false }
      - uses: actions/setup-dotnet@0f4a354dd5547a28c70b7853dbd95d5b67886e73 # v6.1.0
        with: { global-json-file: global.json }
      - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
        with:
          role-to-assume: ${{ vars.IAC_PLAN_ROLE_ARN }}
          aws-region: us-east-1
      - name: Synth (cdk-nag runs here and fails on errors) and diff
        working-directory: infra/cdk
        run: |
          npx cdk@2 synth --quiet
          npx cdk@2 diff --fail 2>&1 | tee diff.txt || true   # diff exits 1 when there are differences; reviewers read diff.txt
      - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
        with: { name: cdk-diff, path: infra/cdk/diff.txt }

  deploy:
    needs: diff
    runs-on: ubuntu-latest
    environment: prod
    permissions: { contents: read, id-token: write }
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with: { persist-credentials: false }
      - uses: actions/setup-dotnet@0f4a354dd5547a28c70b7853dbd95d5b67886e73 # v6.1.0
        with: { global-json-file: global.json }
      - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
        with:
          role-to-assume: ${{ vars.IAC_APPLY_ROLE_ARN }}
          aws-region: us-east-1
      - name: Deploy (broadening IAM or security-group changes still prompt; no --require-approval never)
        working-directory: infra/cdk
        run: npx cdk@2 deploy --all --require-approval broadening --ci

A note on --require-approval: in a non-interactive pipeline a prompt cannot be answered, so a broadening change fails the job instead of silently proceeding. That is the safe behaviour for IAM and security-group widening; resolve it by reviewing the diff and, for approved changes, redesigning so the change is explicit. Use never only for dev accounts.

Code explained line by line

  1. on: pull_request with paths: limits the workflow to infrastructure changes. Application-only changes do not pay for a plan.
  2. permissions: id-token: write lets the job request a GitHub OIDC token. It grants nothing in AWS by itself; the role's trust policy decides.
  3. terraform_wrapper: false stops the setup action wrapping Terraform's output, which matters when you pipe terraform show -json.
  4. terraform plan -out=tfplan writes the plan to a file. terraform show -json tfplan renders it for machines: policy tools and the jq destructive-change check read this form.
  5. The jq -e expression exits non-zero when a plan deletes a KMS key, a bucket, a log group or a database. A human can still approve such a change, but not by accident; add an explicit label or ticket check if you want the exception recorded.
  6. In iac-apply.yml, concurrency with cancel-in-progress: false serialises applies and never kills one mid-run.
  7. environment: prod on the apply job is what pauses it for reviewers and changes the OIDC sub to repo:<owner>/<repo>:environment:prod, which only the apply role trusts.
  8. terraform apply -input=false tfplan applies the file, not the code; it does not re-plan.
  9. In the CDK workflow, cdk diff --fail exits non-zero if differences exist; the || true keeps the job going so reviewers can read the output. In the C# stacks, cdk-nag runs during synth as an aspect; errors fail the job before any AWS call.

Security considerations

⚠ Learning shortcut — not for production

The lab-style examples keep the plan role trusted by main and use -lock=false for pull-request plans to avoid giving a read-only role write access to the lock file, run Terraform with a single state for a whole environment and let the apply role keep broad service permissions inside a boundary. Production: one state per bounded context (network, security, workloads), the apply role scoped to what that state manages, state in a dedicated account with access limited to the pipeline roles, and a reviewed break-glass process.

  • The state file is sensitive. It can hold secrets in clear text (database passwords, keys). Encrypt it with a customer managed KMS key, restrict who can read it, and keep secrets out of Terraform where possible; use Secrets Manager-managed passwords instead. See Secrets in CI/CD.
  • Pull requests from forks. They must not receive the plan role. Either disable workflows for fork PRs that touch infra/, require approval for fork workflows, or use trust conditions that only name your repository's own subjects. Never use pull_request_target to check out and run untrusted code.
  • Plan output can leak data. Mark sensitive variables sensitive = true; plan text goes into logs and comments.
  • Treat the plan as an input to review, not a rubber stamp. Large plans hide dangerous lines. Cap plan size or split states so reviewers can actually read what they approve.
  • Stack policies and SCPs. A CloudFormation stack policy can deny updates to critical resources (a log bucket, a KMS key) unless an explicit override is supplied; an organisation SCP can deny deleting CloudTrail or disabling encryption entirely, as a backstop to anything the pipeline does.
  • Break-glass. Define who may apply from outside the pipeline in an emergency, how it is monitored (CloudTrail alert on console or CLI changes by humans in prod) and how the change is reconciled into code afterwards.
  • Compliance. These controls produce evidence of review and approval; they do not make an environment compliant. Your risk team defines what is required.

Banking use case

Harbourline's platform team owns the network and logging stacks used by all payment services. Policy: no human has write access to production infrastructure. Engineers open pull requests; the plan role posts the plan; CODEOWNERS require a platform engineer and, for IAM or KMS changes, a security engineer. On merge, the pipeline re-plans, and a release manager approves the prod environment. The apply role, bound to a permission boundary that forbids creating IAM users or access keys, applies the saved plan. A nightly drift job pages the platform channel if anyone changed anything by hand; the break-glass permission set is monitored, and every use produces a ticket that must end in a pull request. Examiners ask "show me how a security group in production got its current rule"; the answer is a pull request, a plan, an approver and a run.

Common mistakes

  • Applying code, not the plan. terraform apply without a plan file re-plans at apply time; what runs may differ from what was reviewed. Always save and apply the plan.
  • One role for everything. A pull-request job with write access means any contributor can change production by editing a workflow. Separate roles, separate trust conditions.
  • State on a laptop or in a public bucket. Remote, encrypted, versioned, private; with locking.
  • cdk deploy --require-approval never in prod to "fix the pipeline hanging". The prompt was telling you something about IAM.
  • Suppressing cdk-nag rules without a reason. Every acknowledgment needs a justification and a reviewer.
  • Drift ignored. If the nightly job is red for three weeks, people stop trusting it. Triage or delete the manual change.
  • Pipeline deploys infra and app in one job. Keep separate cadences. A slow infrastructure approval should not block a code fix.

Troubleshooting

SymptomLikely causeFix
Not authorized to perform sts:AssumeRoleWithWebIdentity in the plan jobSubject does not match the plan role's trust (for example pull_request vs branch)Print the token sub; compare with the trust policy
Error acquiring the state lockAnother run holds the lock or a crashed run left a lock objectWait; if certain no run is active, terraform force-unlock <id> after confirming with the team
Saved plan is stale on applyState changed between plan and apply (another apply or drift)Re-run the plan job and approve the new plan
cdk diff fails with Need to perform AWS calls... lookupMissing lookup role permission or context values not committedAllow sts:AssumeRole on the lookup role; commit cdk.context.json
CDK deploy fails: This deployment will make potentially sensitive changes--require-approval broadening in a CI sessionReview and, if approved, adjust the change; do not switch to never in prod
Checkov fails on a resource you must createGenuine finding or false positiveFix, or add a suppression with a ticket reference in a comment and CODEOWNERS review
Plan shows replacement of a resource nobody changedProvider upgrade changed defaults, or driftPin provider versions; read the changed attribute in the plan
Terraform cannot read state: AccessDeniedPlan role lacks bucket or KMS accessGrant kms:Decrypt on the state key and s3:GetObject/s3:ListBucket

Cost and cleanup

Cost: free. GitHub-hosted runners for private repositories use your plan's included minutes ⚠ VERIFY current allowance; public repositories are free. A state bucket with a few small objects costs cents per month. IAM roles, OIDC providers, CloudFormation and CDK bootstrap have no charge; the resources you deploy do. The exercises below use only an S3 bucket and an IAM role, so there is nothing to bill beyond storage.

Teardown:

P=learn-sandbox
# Terraform-managed resources
terraform destroy
# State bucket: versioned, so delete all versions before the bucket
aws s3api list-object-versions --bucket hl-tfstate-$ACCOUNT --profile $P
aws s3 rb s3://hl-tfstate-$ACCOUNT --force --profile $P   # deletes current objects; for versioned buckets remove versions and delete markers first
# CDK stacks
npx cdk destroy --all --profile $P
# Roles created for the lab
aws iam delete-role-policy --role-name hl-iac-plan --policy-name state-access --profile $P 2>/dev/null
aws iam detach-role-policy --role-name hl-iac-plan --policy-arn arn:aws:iam::aws:policy/ReadOnlyAccess --profile $P
aws iam delete-role --role-name hl-iac-plan --profile $P
aws iam delete-role --role-name hl-iac-apply --profile $P
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m09-iac --profile $P --region us-east-1

Free alternative: do the design exercises and run terraform validate, terraform fmt -check and terraform plan against a local backend in Docker (hashicorp/terraform:1.14) without any AWS credentials, using a provider configuration with skip_credentials_validation for a read-free sample ⚠ VERIFY the flags for your provider version.

Exercises

  • 🟢 Guided: Write the iac-plan.yml workflow for a Terraform folder that creates one S3 bucket. Make fmt, validate and the plan steps required checks. Open a pull request in your own sandbox repository and read the job summary.
  • 🟡 Challenge: Add a job that fails when the plan deletes any resource of type aws_s3_bucket or aws_kms_key, then prove it by writing a plan that removes a bucket. Add a short README note explaining who may approve an exception and how it is recorded.
  • 🔴 Live problem: Ticket CHG-104388: "A security group in prod now allows 0.0.0.0/0 on port 5432. Nobody has a change record. Last night's drift job was red; it has been red for a month." Describe your first hour: stop the exposure, find who and how, bring the code and the account back in line, and what pipeline changes prevent a repeat.

Assignment

Add an infra/ pipeline to your portfolio repository: plan on pull request with a read-only role, apply on main after an environment approval using a saved plan, plus a nightly drift workflow. Include a short docs/infra-change-process.md that says who reviews what, how emergencies are handled and how plans are retained.

Solutions

Knowledge check

Interview questions

  • How do you make sure the change reviewed is the change applied? Save the plan on main, apply that file in an approved environment; Terraform rejects a stale plan. For CDK, deploy the synthesised assembly or re-diff in the same run and restrict approval prompts.
  • Why two roles, plan and apply? Previews run on every pull request, including from less-trusted contexts, so they get read-only access. Writes are reserved for the protected environment.
  • How do you handle drift? A scheduled plan with exit code detection, an owner for each alert, and a rule: either revert the manual change or codify it through a pull request.
  • Where do policy checks belong? Before apply (cdk-nag, Checkov, OPA on the plan JSON), and again in the account (SCPs, AWS Config) as defence in depth.
  • What do you do with secrets in Terraform state? Avoid them, use managed passwords, encrypt and lock down the state bucket.

Certification practice

Completion checklist

  • I can explain the plan, review, apply flow and the saved-plan guarantee
  • I wrote a pull-request workflow with a read-only role and a required check
  • I wrote an apply workflow bound to a protected environment
  • I added at least one policy-as-code gate that fails the build
  • I ran the cleanup steps and confirmed nothing is still billing

Key takeaways

  • The change approved must be the change applied: save the plan and apply that file.
  • Separate roles for previewing and changing, bound by different OIDC subjects.
  • Policy gates run before apply; SCPs and Config catch what slips through.
  • State is sensitive: encrypt, version, lock and restrict it.
  • Drift detection turns manual changes into visible, owned work.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page