Skip to Content
Cloud Engineering Academy
10. Infrastructure as Code

Terraform state

Move Terraform state into an encrypted, versioned S3 bucket with native lock files, see locking stop two applies racing each other, use workspaces carefully, and bring existing resources under management with import, moved and removed blocks.

intermediate⏱ 2 h 30 min🟡 Low costDOP-C02SOA-C03

Assumes: Terraform 1.17.2 · Terraform AWS provider 6.73.0 (~> 6.0) · S3 backend with use_lockfile · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain what Terraform state contains, why it exists, and why it is sensitive.
  • Bootstrap a state bucket with a customer managed KMS key, versioning, Block Public Access and a TLS-only policy.
  • Configure the S3 backend with encrypt, kms_key_id and use_lockfile, and migrate local state into it.
  • Demonstrate state locking with two terminals, and know when (and when not) to use force-unlock.
  • Choose between workspaces and separate state per environment or account.
  • Use import, moved and removed blocks, and the terraform state commands, to change what Terraform manages without destroying anything.
  • Write a least-privilege IAM policy for the identity that reads and writes state.

Prerequisites

  • Terraform fundamentals — the workflow, HCL and the tf Docker function used again here.
  • Module 5: Security — KMS key policies and S3 bucket policies.
  • Tools: Docker Desktop, AWS CLI v2 with the learn-sandbox profile, two PowerShell windows for the locking experiment.

Why this matters in banking

Picture two engineers at Harbourline. Priya is adding a CloudWatch alarm to the Payments environment. Tom is tightening a security group in the same environment. Both have the configuration checked out, both run terraform apply at almost the same moment, and both have their own copy of the state file on their laptops. Each apply writes its own version of the state. Whichever finishes second overwrites the other's record. The next plan thinks Priya's alarm does not exist and tries to create it again, or worse, thinks Tom's rule was never there.

Now add three more realities of a bank:

  • The state file lists every resource ID in the account, and can contain database connection strings, generated passwords and private keys. It is sensitive data and must be encrypted, access-controlled and logged.
  • An auditor will ask, "If this file is corrupted, how do you recover?" The answer must be "S3 versioning, tested", not "Tom has a copy".
  • The bank already has hundreds of resources created by hand or by older tools. Terraform must be able to adopt them without rebuilding them.

This lesson fixes all three.

The idea in plain language

State is Terraform's memory. For every resource address in your code (aws_s3_bucket.audit) it stores the real-world ID (hl-m10-l8-audit-111122223333) and the last attributes it saw. Terraform needs it to know which real objects belong to this configuration, to work out dependencies when deleting, and to make plans fast.

A backend decides where the state lives. The default is a local file. The S3 backend stores it as an object in a bucket you own.

Locking stops two runs from changing the same state at once. With use_lockfile = true, Terraform writes a small .tflock object next to the state using S3 conditional writes. If the lock object already exists, the second run waits or fails with a message saying who holds the lock.

Analogy: state is the bank's asset register for a branch: every safe, ATM and alarm panel with its serial number. The backend is the fireproof cabinet the register lives in. The lock is the sign-out sheet on the cabinet: only one person may update the register at a time, and everyone can see who has it.

What problem the AWS services and backend solve

NeedHow it is met
Shared, durable storageS3 object, 11 nines durability design
Recovery from corruption or a bad applyS3 versioning on the state bucket
ConfidentialitySSE-KMS with a customer managed key, Block Public Access, TLS-only bucket policy, narrow IAM
Integrity under concurrencyuse_lockfile = true (S3 conditional writes)
AuditCloudTrail management events, plus S3 data events on the bucket if you need object-level logging

DynamoDB locking is deprecated. Older guides create a DynamoDB table with a LockID partition key and set dynamodb_table in the backend. HashiCorp's S3 backend documentation now marks DynamoDB-based locking as deprecated and says it will be removed in a future minor version. You can set both use_lockfile and dynamodb_table during a migration; new configurations should use only use_lockfile.

Honest limitations:

  • The state bucket has to exist before any configuration can use it — a chicken-and-egg problem you solve once with a small bootstrap configuration.
  • Locking prevents simultaneous writes; it does not prevent someone with S3 permissions from editing or deleting the state object directly. IAM and versioning cover that.
  • State is not a backup of your data. It records that a database exists, not what is in it.
  • sensitive = true hides values from output only. Secret values still end up in state unless the resource supports write-only arguments or you avoid passing secrets through Terraform.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS / Terraform equivalentWhat's different
azurerm backend in a Storage Account containers3 backend in an S3 bucketSame idea; different settings
Blob lease used as the state lockuse_lockfile — a .tflock object written with S3 conditional writesNo extra service; DynamoDB tables are no longer needed
Storage Account encryption with a customer-managed key in Key VaultSSE-KMS with a customer managed KMS key, kms_key_id in the backendKMS key policy and IAM must both allow the caller
Blob versioning / soft deleteS3 versioning plus a noncurrent-version lifecycle ruleRestore by copying an older version over the current one
ARM deployments have no client-side stateCloudFormation keeps state inside the serviceTerraform is the odd one out: you manage state
az resource import into Bicep (not supported as such)import block and terraform plan -generate-config-outTerraform can generate starter HCL for imported resources

Architecture

Locking sequence

Security-boundary view

  • Who can read state = who can read secrets in it. Grant s3:GetObject on state keys and kms:Decrypt on the key only to the pipeline roles and a small break-glass group.
  • Separate buckets (or at least prefixes and policies) per environment. A developer who can plan in learn must not be able to read prod state.
  • Production pattern: state buckets live in a dedicated tooling or shared-services account, and workload pipelines assume a role there just for state. The landing-zone lesson shows where that account sits.

Step-by-step implementation

Bootstrap the state bucket

Create a folder bootstrap/ with the configuration from Complete code. It uses local state because the bucket does not exist yet. Apply it once. Keep its tiny state file safe (or, after creation, migrate the bootstrap itself into the bucket it created — a common and acceptable pattern).

Point a workload configuration at the bucket

Reuse the configuration from Terraform fundamentals in a folder audit/. Add an empty backend "s3" {} block and a backend.hcl file with the settings. Keeping the settings in a separate file (a partial configuration) lets the same code use different buckets per environment.

Migrate local state into S3

Run tf init -backend-config=backend.hcl -migrate-state. Terraform detects the new backend, asks whether to copy the existing state and, when you answer yes, uploads it. Check with aws s3 ls s3://<bucket>/audit/learn/.

Race two applies

Open a second PowerShell window in the same folder. Change noncurrent_days to 45. In window A run tf apply and stop at the "Enter a value" prompt. In window B run tf plan. Window B fails with Error acquiring the state lock and prints the lock ID, the user and the time. Answer no in window A; the lock is released and window B's plan now succeeds. Add -lock-timeout=60s to make a run wait rather than fail.

Try a workspace

Run tf workspace new test, then tf plan. The plan wants to create everything again, because the test workspace has its own, empty state stored at env:/test/audit/learn/terraform.tfstate. Run tf workspace select default and tf workspace delete test. You will decide below whether workspaces suit a bank.

Import an existing resource

Create a bucket by hand with the CLI (simulating a "ClickOps" resource), then adopt it with an import block and -generate-config-out. Review the generated HCL, tidy it, and apply. The plan must say 1 to import, 0 to add, 0 to destroy.

Refactor without destroying

Rename a resource address with a moved block, and stop managing a resource with a removed block. Both changes must plan with zero creates and zero destroys.

Recover a previous state version

List the state object's versions with aws s3api list-object-versions. Download the previous version to see that recovery is possible. Do not overwrite the current state in this exercise — just prove you could.

Tear down in the right order

Destroy workload configurations first (their state lives in the bucket), then destroy the bootstrap configuration. The KMS key goes into a 7-day pending-deletion window, during which it is not billed.

AWS Console instructions

  • Console → S3 → hl-m10-tfstate-<account>-us-east-1 → Objects: browse audit/learn/. While window A waits at its prompt you can see terraform.tfstate.tflock; after it finishes, the lock object is gone.
  • Select terraform.tfstate → Versions toggle on: every apply creates a new version.
  • Properties → Default encryption: SSE-KMS with alias/hl-m10-tfstate and Bucket Key enabled.
  • Console → KMS → Customer managed keys → alias/hl-m10-tfstate: check Key rotation is on and review the key policy.
  • Console → CloudTrail → Event history: filter by Event source kms.amazonaws.com to see GenerateDataKey and Decrypt calls made by Terraform's S3 reads and writes (Bucket Keys reduce how many you see).

AWS CLI

# What's in the state prefix?
aws s3 ls s3://hl-m10-tfstate-111122223333-us-east-1/audit/learn/ --profile learn-sandbox

# Every version of the state file (newest first)
aws s3api list-object-versions \
  --bucket hl-m10-tfstate-111122223333-us-east-1 \
  --prefix audit/learn/terraform.tfstate \
  --query "Versions[].{Id:VersionId,Latest:IsLatest,Modified:LastModified}" \
  --output table --profile learn-sandbox

# Download an older version to inspect (never edit state by hand)
aws s3api get-object \
  --bucket hl-m10-tfstate-111122223333-us-east-1 \
  --key audit/learn/terraform.tfstate \
  --version-id "PASTE_VERSION_ID" previous.tfstate --profile learn-sandbox

# Create a "hand-made" bucket for the import exercise
aws s3api create-bucket --bucket hl-m10-l9-legacy-111122223333 \
  --region us-east-1 --profile learn-sandbox
aws s3api put-bucket-tagging --bucket hl-m10-l9-legacy-111122223333 \
  --tagging 'TagSet=[{Key=Project,Value=cea-lab},{Key=Lab,Value=m10-l9},{Key=Environment,Value=learn},{Key=Owner,Value=yourname}]' \
  --profile learn-sandbox

Terraform state commands

tf state list                                # every address in state
tf state show aws_s3_bucket.audit            # attributes of one resource
tf plan -refresh-only                        # show drift without proposing config changes
tf apply -refresh-only                       # accept reality into state (use with care)
tf force-unlock LOCK_ID                      # only when you are SURE no run holds the lock
tf state pull > snapshot.tfstate             # read-only local copy for inspection

Prefer the declarative moved, removed and import blocks over the imperative state mv, state rm and terraform import commands: blocks go through code review and appear in the plan; commands change state immediately with no review.

Infrastructure as Code

The same idea in the other tools in this module, for comparison:

terraform {
  backend "s3" {
    bucket       = "hl-m10-tfstate-111122223333-us-east-1"
    key          = "audit/learn/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    kms_key_id   = "alias/hl-m10-tfstate"
    use_lockfile = true
  }
}

Complete code

Bootstrap configuration

bootstrap/main.tf
terraform {
  required_version = ">= 1.14.0, < 2.0.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
  # Local state on purpose: the bucket this creates does not exist yet
}

provider "aws" {
  region = "us-east-1"
  default_tags {
    tags = {
      Project     = "cea-lab"
      Lab         = "m10-l9"
      Environment = "learn"
      Owner       = var.owner
      ManagedBy   = "terraform"
    }
  }
}

variable "owner" {
  type = string
}

variable "state_writer_role_arns" {
  description = "Extra role ARNs allowed to use the key (e.g. a CI deploy role). Your SSO role works through IAM."
  type        = list(string)
  default     = []
}

data "aws_caller_identity" "current" {}
data "aws_region" "current" {}

locals {
  account_id  = data.aws_caller_identity.current.account_id
  bucket_name = "hl-m10-tfstate-${local.account_id}-${data.aws_region.current.region}"
}

# --- KMS key for state ---------------------------------------------------
data "aws_iam_policy_document" "key" {
  # Account root: lets IAM policies in this account grant use of the key
  statement {
    sid       = "EnableIamPolicies"
    actions   = ["kms:*"]
    resources = ["*"]
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${local.account_id}:root"]
    }
  }

  dynamic "statement" {
    for_each = length(var.state_writer_role_arns) > 0 ? [1] : []
    content {
      sid       = "AllowStateWriters"
      actions   = ["kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey", "kms:DescribeKey"]
      resources = ["*"]
      principals {
        type        = "AWS"
        identifiers = var.state_writer_role_arns
      }
      condition {
        test     = "StringEquals"
        variable = "kms:ViaService"
        values   = ["s3.${data.aws_region.current.region}.amazonaws.com"]
      }
    }
  }
}

resource "aws_kms_key" "state" {
  description             = "Encrypts Terraform state for Harbourline learning labs"
  enable_key_rotation     = true
  deletion_window_in_days = 7
  policy                  = data.aws_iam_policy_document.key.json
}

resource "aws_kms_alias" "state" {
  name          = "alias/hl-m10-tfstate"
  target_key_id = aws_kms_key.state.key_id
}

# --- State bucket --------------------------------------------------------
resource "aws_s3_bucket" "state" {
  bucket = local.bucket_name
  # No force_destroy: deleting state by accident must be hard
}

resource "aws_s3_bucket_ownership_controls" "state" {
  bucket = aws_s3_bucket.state.id
  rule {
    object_ownership = "BucketOwnerEnforced"
  }
}

resource "aws_s3_bucket_public_access_block" "state" {
  bucket                  = aws_s3_bucket.state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.state.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "state" {
  bucket = aws_s3_bucket.state.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.state.arn
    }
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "state" {
  bucket = aws_s3_bucket.state.id
  rule {
    id     = "keep-90-days-of-history"
    status = "Enabled"
    filter {}
    noncurrent_version_expiration {
      noncurrent_days = 90
    }
  }
  depends_on = [aws_s3_bucket_versioning.state]
}

data "aws_iam_policy_document" "bucket" {
  statement {
    sid     = "DenyInsecureTransport"
    effect  = "Deny"
    actions = ["s3:*"]
    principals {
      type        = "*"
      identifiers = ["*"]
    }
    resources = [aws_s3_bucket.state.arn, "${aws_s3_bucket.state.arn}/*"]
    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }

  statement {
    sid     = "DenyUnencryptedUploads"
    effect  = "Deny"
    actions = ["s3:PutObject"]
    principals {
      type        = "*"
      identifiers = ["*"]
    }
    resources = ["${aws_s3_bucket.state.arn}/*"]
    condition {
      test     = "StringNotEquals"
      variable = "s3:x-amz-server-side-encryption"
      values   = ["aws:kms"]
    }
  }
}

resource "aws_s3_bucket_policy" "state" {
  bucket     = aws_s3_bucket.state.id
  policy     = data.aws_iam_policy_document.bucket.json
  depends_on = [aws_s3_bucket_public_access_block.state]
}

output "state_bucket" {
  value = aws_s3_bucket.state.bucket
}

output "state_kms_key_arn" {
  value = aws_kms_key.state.arn
}

Workload configuration using the backend

audit/backend.tf
terraform {
  # Partial configuration: values come from backend.hcl at init time
  backend "s3" {}
}
audit/backend.hcl
bucket       = "hl-m10-tfstate-111122223333-us-east-1"
key          = "audit/learn/terraform.tfstate"
region       = "us-east-1"
encrypt      = true
kms_key_id   = "alias/hl-m10-tfstate"
use_lockfile = true
# Copy local state into the bucket (answers 'yes' when prompted)
tf init -backend-config=backend.hcl -migrate-state

Least-privilege policy for a state user

state-access-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListStatePrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::hl-m10-tfstate-111122223333-us-east-1",
      "Condition": { "StringLike": { "s3:prefix": ["audit/learn/*"] } }
    },
    {
      "Sid": "ReadWriteState",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::hl-m10-tfstate-111122223333-us-east-1/audit/learn/terraform.tfstate"
    },
    {
      "Sid": "ManageLockFile",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::hl-m10-tfstate-111122223333-us-east-1/audit/learn/terraform.tfstate.tflock"
    },
    {
      "Sid": "UseStateKey",
      "Effect": "Allow",
      "Action": ["kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey"],
      "Resource": "arn:aws:kms:us-east-1:111122223333:key/REPLACE-WITH-KEY-ID"
    }
  ]
}

Import, moved and removed blocks

audit/adopt.tf
# 1. Adopt a bucket somebody created by hand
import {
  to = aws_s3_bucket.legacy
  id = "hl-m10-l9-legacy-111122223333"
}

# 2. We renamed aws_s3_bucket.audit to aws_s3_bucket.evidence in main.tf.
#    Tell Terraform it is the same object, so nothing is destroyed.
moved {
  from = aws_s3_bucket.audit
  to   = aws_s3_bucket.evidence
}

# 3. Hand a resource over to another team's configuration:
#    forget it in this state, but do NOT delete it in AWS.
removed {
  from = aws_s3_bucket_lifecycle_configuration.old_rule

  lifecycle {
    destroy = false
  }
}
# Generate starter HCL for every import block that has no resource yet
tf plan -generate-config-out=generated.tf
# Review generated.tf, remove computed/defaulted arguments, move it into main.tf, then:
tf plan -out=tfplan   # expect: 1 to import, 0 to add, 0 to change, 0 to destroy
tf apply tfplan

Code explained line by line

Bootstrap

  1. No backend block — the bootstrap uses local state because it creates the bucket the others depend on.
  2. data "aws_region" "current" and .region — in AWS provider v6 the Region attribute of this data source is region; older code used .name, which v6 deprecates ⚠ VERIFY against the provider changelog if you see a deprecation warning.
  3. statement "EnableIamPolicies" with the account root principal — the standard key-policy statement that delegates to IAM: identities in this account can use the key if their IAM policy allows it. Without it you can lock yourself out of the key.
  4. dynamic "statement" — generates the AllowStateWriters statement only if you pass role ARNs. dynamic blocks create repeated nested blocks from a collection.
  5. kms:ViaService = s3.us-east-1.amazonaws.com — the role may use the key only through S3, not by calling KMS directly to decrypt arbitrary data.
  6. enable_key_rotation = true — yearly automatic rotation of the key material; old ciphertext still decrypts.
  7. deletion_window_in_days = 7 — the shortest waiting period. During the window the key is not billed and can be recovered.
  8. The bucket has no force_destroy. Destroying the bootstrap with state objects still inside should fail; you will empty the bucket deliberately at the end.
  9. kms_master_key_id = aws_kms_key.state.arn — default encryption with your customer managed key, so access needs both S3 and KMS permission.
  10. noncurrent_version_expiration { noncurrent_days = 90 } — 90 days of state history; long enough to recover from a bad apply noticed late, short enough to limit storage.
  11. DenyUnencryptedUploads — refuses any PutObject that does not request SSE-KMS. Terraform's backend with encrypt = true sends the header, so it passes; a careless manual upload does not.

Backend

  1. backend "s3" {} with values in backend.hcl — backend blocks cannot use variables, so partial configuration is the standard way to vary bucket and key per environment.
  2. key = "audit/learn/terraform.tfstate" — use a <component>/<environment>/ pattern. It keeps state small (one component per state) and lets IAM policies grant per-prefix access.
  3. encrypt = true with kms_key_id — the backend writes the state and lock objects with SSE-KMS using that key.
  4. use_lockfile = true — locking via a terraform.tfstate.tflock object next to the state. The lock needs s3:GetObject, s3:PutObject and s3:DeleteObject on that key.

IAM policy

  1. s3:ListBucket restricted with s3:prefix — the backend lists to find state and workspaces; this stops it browsing other teams' prefixes.
  2. Separate statements for the state object and the lock object — the state needs read and write; the lock also needs delete.
  3. kms:Encrypt, kms:Decrypt, kms:GenerateDataKey — the three KMS actions the S3 backend documentation lists when kms_key_id is set.

Adopt and refactor

  1. import { to = …, id = … } — tells the next plan to read the existing object into state at that address. The id format depends on the resource type (a bucket name here; a security group ID such as sg-… for security groups). The import happens during apply, so it is reviewed like any other change.
  2. -generate-config-out=generated.tf — Terraform writes HCL for imported resources that have no resource block yet. The output is a starting point: it includes every argument, many of which are defaults you should delete.
  3. moved { from, to } — records a rename in code. Anyone applying later gets the same move automatically. Keep moved blocks for a while (or permanently in shared modules) so every consumer picks them up.
  4. removed { from = …, lifecycle { destroy = false } } — removes the object from state without deleting it in AWS. Without destroy = false, deleting a resource block means "destroy it".

Workspaces vs separate state

A workspace is a named, separate state for the same configuration and backend. terraform.workspace is available as a value in your code.

ApproachHowGood forRisk in a bank
CLI workspacesterraform workspace new test; state at env:/test/<key>Short-lived copies: a feature branch's preview environment, a test of a moduleAll environments share one backend, one bucket policy and usually one set of credentials. Running apply in the wrong workspace hits the wrong environment. The current workspace is invisible in the code
Separate root folders or separate backend files per environmentenvs/test/backend.hcl, envs/prod/backend.hcl, each with its own key — or its own bucket in its own accountLong-lived dev, test and prodMore files, but each environment can have separate credentials, a separate state bucket and separate approvals

Harbourline's rule: production state lives in the production account's (or a tooling account's) dedicated bucket, accessed only by the production deploy role. Workspaces are allowed only for ephemeral environments. The next-but-one lesson, Environment configuration and versioning, builds the folder layout.

Secrets and state

Some resources return or accept secret values: an RDS master password, a generated random_password, a TLS private key from the tls provider. Unless the resource supports write-only arguments (Terraform 1.11 and later, for example password_wo on some database resources ⚠ VERIFY per resource in the provider docs), the value is stored in state.

Safer patterns, in order of preference:

  1. Let AWS generate and hold the secret. For RDS, set manage_master_user_password = true; RDS creates and rotates the password in Secrets Manager and Terraform never sees it.
  2. Use write-only arguments or ephemeral resources where the provider supports them, so the value is used during apply and never persisted.
  3. Create the secret container in Terraform, put the value in later through a separate, audited process.
  4. If a secret must pass through Terraform, accept that the state is secret-bearing and restrict it accordingly (which you should do anyway).

Security considerations

⚠ Learning shortcut — not for production

In this lesson the state bucket, the KMS key and the workload all live in the same sandbox account, and your SSO role has broad rights to all of them. In production, put state buckets in a dedicated tooling account (or one bucket per environment account), allow only the pipeline's deploy roles to read and write each prefix, keep human access to a break-glass role with MFA and alerting, and turn on CloudTrail S3 data events for the state bucket so every read of state is recorded.

  • Never edit state by hand. If you must repair state, use moved, removed, import or, as a last resort, terraform state commands — after terraform state pull to a backup.
  • force-unlock is a break-glass action. Only use it after confirming the run that holds the lock is truly dead (check the CI job and the lock's Who and Created fields). Unlocking during a live apply can corrupt state.
  • Versioning is your undo button. Test restoring a previous version once a quarter; record the test as audit evidence.
  • Deny s3:DeleteObject on the state key to everyone except a break-glass role if your team's process allows it — the lock file still needs delete, so scope the deny to *.tfstate keys only.
  • Sensitive values in state — see the section above. Classify the state bucket at the same level as the most sensitive value it can contain.

Banking use case

Harbourline's platform team runs one state bucket per environment account (hl-tfstate-<account>-eu-west-2), each encrypted with its own customer managed key. The production key policy allows kms:Decrypt only to the production deploy role and the break-glass role. Each component — network, payments, ledger-db, observability — has its own state key, so a mistake in the observability plan cannot touch the database's state, and plans stay fast.

When the Data Platform team migrated 140 hand-built Glue jobs and S3 buckets, they did it with import blocks generated by a script from an inventory CSV, reviewed in batches of 20, each batch's plan showing only imports. No resource was recreated. The pull requests became the evidence that the migration did not change production.

Common mistakes

  • Creating the state bucket inside the configuration that uses it → chicken and egg; destroying that configuration deletes its own state → use a separate bootstrap configuration.
  • Copying old guides that create a DynamoDB lock table → extra cost and a deprecated feature → use_lockfile = true.
  • One giant state for the whole bank → slow plans, every change risks everything, everyone needs access to all of it → split state by component and environment.
  • Using workspaces for prod vs dev with the same credentials → one wrong workspace select and you apply to prod → separate backends and roles for long-lived environments.
  • terraform state rm in a terminal to "fix" a plan → unreviewed and easy to forget, so the next person's plan recreates or orphans the resource → use a removed block.
  • Assuming sensitive = true protects state → it only masks CLI output → avoid secrets in state, and protect state anyway.
  • Forgetting KMS permissions → AccessDenied on GenerateDataKey when the backend writes → add the three KMS actions for the state key.

Troubleshooting

SymptomLikely causeFix
Error acquiring the state lock … PreconditionFailedAnother run holds the lockWait, or use -lock-timeout=5m; check CI for a running job
Lock never released after a crash or killed CI jobThe process died before deleting .tflockConfirm no run is active, then terraform force-unlock LOCK_ID
Backend configuration changed on initYou edited backend settingsinit -reconfigure (start fresh) or init -migrate-state (copy state) — choose deliberately
AccessDenied when writing stateMissing kms:GenerateDataKey or s3:PutObject on the keyCheck both IAM and the KMS key policy
AccessDenied on the .tflock objectMissing s3:DeleteObject or s3:PutObject for the lock keyAdd the lock-file statement
Plan wants to create everythingWrong workspace, wrong key, or init pointed at a different bucketterraform workspace show; check backend.hcl
Import fails with Cannot import non-existent remote objectWrong id format or wrong Region/accountCheck the resource's import ID format in the provider docs
Generated config has errors on planIt includes conflicting or read-only argumentsDelete the arguments named in the error; keep only what you mean to manage
BucketNotEmpty destroying bootstrapVersioned state objects remainDelete all object versions first (see cleanup)

Cost and cleanup

ItemCost
Customer managed KMS key$1 per month, prorated hourly — about $0.0014 per hour. Not billed while pending deletion
KMS requestsWithin the 20,000 requests per month free tier for this lesson
S3 storage for a few state versionsFractions of a cent
S3 requests for lockingFractions of a cent
Lesson total if destroyed the same dayAbout $0.01

Free alternative: use the AWS managed key (sse_algorithm = "aws:kms" without kms_master_key_id, and no kms_key_id in the backend). You lose key-policy control, which is why the lesson uses a customer managed key.

# 1. Destroy workloads first (their state is in the bucket)
Set-Location audit; tf destroy; Set-Location ..

# 2. Delete the hand-made legacy bucket if you used a removed block instead of destroying it
aws s3api delete-bucket --bucket hl-m10-l9-legacy-111122223333 --profile learn-sandbox

# 3. Empty the versioned state bucket (all versions and delete markers)
$b = "hl-m10-tfstate-111122223333-us-east-1"
$v = aws s3api list-object-versions --bucket $b --profile learn-sandbox --output json | ConvertFrom-Json
foreach ($o in @($v.Versions) + @($v.DeleteMarkers)) {
  if ($o) { aws s3api delete-object --bucket $b --key $o.Key --version-id $o.VersionId --profile learn-sandbox | Out-Null }
}

# 4. Destroy the bootstrap (KMS key enters 7-day pending deletion)
Set-Location bootstrap; tf destroy; Set-Location ..

# 5. Confirm nothing is left
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m10-l9 `
  --profile learn-sandbox --region us-east-1 --query "ResourceTagMappingList[].ResourceARN"

The KMS key may still appear in the tagging API while pending deletion; that is expected and not billed.

Exercises

  • 🟢 Guided: Bootstrap the state bucket, migrate the audit configuration's state into it, run the two-terminal locking experiment, and list the state object's versions.
  • 🟡 Challenge: Create a security group by hand with the CLI in your default VPC. Adopt it with an import block and -generate-config-out. Tidy the generated HCL until the plan shows only the import. Then rename its address with a moved block and confirm the plan shows a move and nothing else.
  • 🔴 Live problem: It is 18:40. The nightly CI job that applies the payments configuration failed with Error acquiring the state lock. The lock info says Who: runner@ci-7f3a, Created: 14:02. A developer in the chat says "just force-unlock it". What do you check before you do anything, and what do you do if you find the 14:02 job is still running?

Assignment

Add a bootstrap/ folder to your harbourline-iac-terraform repository and convert the audit configuration to the S3 backend with partial configuration. Add docs/state-runbook.md covering: where state lives per environment, who can read it, how to recover a previous version (tested, with the date), and the force-unlock procedure with its pre-checks.

Solutions

Knowledge check

Interview questions

  • "What is in Terraform state and how do you protect it?" — Resource addresses mapped to real IDs, all known attributes, outputs and dependency metadata — which can include secrets. I keep it in a versioned, KMS-encrypted S3 bucket with Block Public Access, TLS-only policy, per-prefix IAM, native lock files, and CloudTrail data events, and I avoid putting secrets through Terraform where possible.
  • "Two applies ran at once and state is now wrong. What do you do?" — Stop further applies, pull and save the current state, compare with the previous S3 version, run a plan -refresh-only to see the gap between state and reality, and repair with import/removed/moved blocks or by restoring the previous version followed by a refresh. Then fix the root cause: locking was off, or two backends pointed at different keys.
  • "Workspaces or folders for environments?" — Folders (or separate backend configurations) with separate credentials for long-lived environments, because isolation and explicitness matter more than fewer files. Workspaces for short-lived copies.
  • "How do you bring 200 hand-made resources under Terraform?" — Inventory them, generate import blocks, use -generate-config-out for starter HCL, tidy it, and review plans that contain imports only — in batches. Nothing is recreated, and the PRs are the evidence.
  • "Why is DynamoDB no longer needed for locking?" — S3 supports conditional writes, so the backend can create a lock object atomically in the same bucket. HashiCorp has deprecated DynamoDB locking in the S3 backend.

Certification practice

Completion checklist

  • I can explain why Terraform needs state and why it is sensitive
  • I bootstrapped an encrypted, versioned state bucket and migrated state into it
  • I saw the lock stop a second run, and I know the force-unlock pre-checks
  • I adopted a resource with an import block and renamed one with moved
  • I emptied and destroyed everything, and the tagging check shows only the pending-deletion KMS key

Key takeaways

  • State maps your code to real resources. Store it in S3 with versioning, SSE-KMS, Block Public Access, a TLS-only policy and narrow IAM.
  • Use use_lockfile = true; DynamoDB locking is deprecated.
  • Split state by component and environment; use separate backends and credentials for long-lived environments, workspaces only for short-lived ones.
  • Change what Terraform manages with import, moved and removed blocks so the change is reviewed.
  • Keep secrets out of state where you can; protect state as if it contains them anyway.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page