Terraform state
Move Terraform state into an encrypted, versioned S3 bucket with native lock files, see locking stop two applies racing each other, use workspaces carefully, and bring existing resources under management with import, moved and removed blocks.
Assumes: Terraform 1.17.2 · Terraform AWS provider 6.73.0 (~> 6.0) · S3 backend with use_lockfile · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain what Terraform state contains, why it exists, and why it is sensitive.
- Bootstrap a state bucket with a customer managed KMS key, versioning, Block Public Access and a TLS-only policy.
- Configure the S3 backend with
encrypt,kms_key_idanduse_lockfile, and migrate local state into it. - Demonstrate state locking with two terminals, and know when (and when not) to use
force-unlock. - Choose between workspaces and separate state per environment or account.
- Use
import,movedandremovedblocks, and theterraform statecommands, to change what Terraform manages without destroying anything. - Write a least-privilege IAM policy for the identity that reads and writes state.
Prerequisites
- Terraform fundamentals — the workflow, HCL and the
tfDocker function used again here. - Module 5: Security — KMS key policies and S3 bucket policies.
- Tools: Docker Desktop, AWS CLI v2 with the
learn-sandboxprofile, two PowerShell windows for the locking experiment.
Why this matters in banking
Picture two engineers at Harbourline. Priya is adding a CloudWatch alarm to the Payments environment. Tom is tightening a security group in the same environment. Both have the configuration checked out, both run terraform apply at almost the same moment, and both have their own copy of the state file on their laptops. Each apply writes its own version of the state. Whichever finishes second overwrites the other's record. The next plan thinks Priya's alarm does not exist and tries to create it again, or worse, thinks Tom's rule was never there.
Now add three more realities of a bank:
- The state file lists every resource ID in the account, and can contain database connection strings, generated passwords and private keys. It is sensitive data and must be encrypted, access-controlled and logged.
- An auditor will ask, "If this file is corrupted, how do you recover?" The answer must be "S3 versioning, tested", not "Tom has a copy".
- The bank already has hundreds of resources created by hand or by older tools. Terraform must be able to adopt them without rebuilding them.
This lesson fixes all three.
The idea in plain language
State is Terraform's memory. For every resource address in your code (aws_s3_bucket.audit) it stores the real-world ID (hl-m10-l8-audit-111122223333) and the last attributes it saw. Terraform needs it to know which real objects belong to this configuration, to work out dependencies when deleting, and to make plans fast.
A backend decides where the state lives. The default is a local file. The S3 backend stores it as an object in a bucket you own.
Locking stops two runs from changing the same state at once. With use_lockfile = true, Terraform writes a small .tflock object next to the state using S3 conditional writes. If the lock object already exists, the second run waits or fails with a message saying who holds the lock.
Analogy: state is the bank's asset register for a branch: every safe, ATM and alarm panel with its serial number. The backend is the fireproof cabinet the register lives in. The lock is the sign-out sheet on the cabinet: only one person may update the register at a time, and everyone can see who has it.
What problem the AWS services and backend solve
| Need | How it is met |
|---|---|
| Shared, durable storage | S3 object, 11 nines durability design |
| Recovery from corruption or a bad apply | S3 versioning on the state bucket |
| Confidentiality | SSE-KMS with a customer managed key, Block Public Access, TLS-only bucket policy, narrow IAM |
| Integrity under concurrency | use_lockfile = true (S3 conditional writes) |
| Audit | CloudTrail management events, plus S3 data events on the bucket if you need object-level logging |
DynamoDB locking is deprecated. Older guides create a DynamoDB table with a LockID partition key and set dynamodb_table in the backend. HashiCorp's S3 backend documentation now marks DynamoDB-based locking as deprecated and says it will be removed in a future minor version. You can set both use_lockfile and dynamodb_table during a migration; new configurations should use only use_lockfile.
Honest limitations:
- The state bucket has to exist before any configuration can use it — a chicken-and-egg problem you solve once with a small bootstrap configuration.
- Locking prevents simultaneous writes; it does not prevent someone with S3 permissions from editing or deleting the state object directly. IAM and versioning cover that.
- State is not a backup of your data. It records that a database exists, not what is in it.
sensitive = truehides values from output only. Secret values still end up in state unless the resource supports write-only arguments or you avoid passing secrets through Terraform.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS / Terraform equivalent | What's different |
|---|---|---|
azurerm backend in a Storage Account container | s3 backend in an S3 bucket | Same idea; different settings |
| Blob lease used as the state lock | use_lockfile — a .tflock object written with S3 conditional writes | No extra service; DynamoDB tables are no longer needed |
| Storage Account encryption with a customer-managed key in Key Vault | SSE-KMS with a customer managed KMS key, kms_key_id in the backend | KMS key policy and IAM must both allow the caller |
| Blob versioning / soft delete | S3 versioning plus a noncurrent-version lifecycle rule | Restore by copying an older version over the current one |
| ARM deployments have no client-side state | CloudFormation keeps state inside the service | Terraform is the odd one out: you manage state |
az resource import into Bicep (not supported as such) | import block and terraform plan -generate-config-out | Terraform can generate starter HCL for imported resources |
Architecture
Locking sequence
Security-boundary view
- Who can read state = who can read secrets in it. Grant
s3:GetObjecton state keys andkms:Decrypton the key only to the pipeline roles and a small break-glass group. - Separate buckets (or at least prefixes and policies) per environment. A developer who can plan in
learnmust not be able to readprodstate. - Production pattern: state buckets live in a dedicated tooling or shared-services account, and workload pipelines assume a role there just for state. The landing-zone lesson shows where that account sits.
Step-by-step implementation
Bootstrap the state bucket
Create a folder bootstrap/ with the configuration from Complete code. It uses local state because the bucket does not exist yet. Apply it once. Keep its tiny state file safe (or, after creation, migrate the bootstrap itself into the bucket it created — a common and acceptable pattern).
Point a workload configuration at the bucket
Reuse the configuration from Terraform fundamentals in a folder audit/. Add an empty backend "s3" {} block and a backend.hcl file with the settings. Keeping the settings in a separate file (a partial configuration) lets the same code use different buckets per environment.
Migrate local state into S3
Run tf init -backend-config=backend.hcl -migrate-state. Terraform detects the new backend, asks whether to copy the existing state and, when you answer yes, uploads it. Check with aws s3 ls s3://<bucket>/audit/learn/.
Race two applies
Open a second PowerShell window in the same folder. Change noncurrent_days to 45. In window A run tf apply and stop at the "Enter a value" prompt. In window B run tf plan. Window B fails with Error acquiring the state lock and prints the lock ID, the user and the time. Answer no in window A; the lock is released and window B's plan now succeeds. Add -lock-timeout=60s to make a run wait rather than fail.
Try a workspace
Run tf workspace new test, then tf plan. The plan wants to create everything again, because the test workspace has its own, empty state stored at env:/test/audit/learn/terraform.tfstate. Run tf workspace select default and tf workspace delete test. You will decide below whether workspaces suit a bank.
Import an existing resource
Create a bucket by hand with the CLI (simulating a "ClickOps" resource), then adopt it with an import block and -generate-config-out. Review the generated HCL, tidy it, and apply. The plan must say 1 to import, 0 to add, 0 to destroy.
Refactor without destroying
Rename a resource address with a moved block, and stop managing a resource with a removed block. Both changes must plan with zero creates and zero destroys.
Recover a previous state version
List the state object's versions with aws s3api list-object-versions. Download the previous version to see that recovery is possible. Do not overwrite the current state in this exercise — just prove you could.
Tear down in the right order
Destroy workload configurations first (their state lives in the bucket), then destroy the bootstrap configuration. The KMS key goes into a 7-day pending-deletion window, during which it is not billed.
AWS Console instructions
- Console → S3 →
hl-m10-tfstate-<account>-us-east-1→ Objects: browseaudit/learn/. While window A waits at its prompt you can seeterraform.tfstate.tflock; after it finishes, the lock object is gone. - Select
terraform.tfstate→ Versions toggle on: every apply creates a new version. - Properties → Default encryption: SSE-KMS with
alias/hl-m10-tfstateand Bucket Key enabled. - Console → KMS → Customer managed keys →
alias/hl-m10-tfstate: check Key rotation is on and review the key policy. - Console → CloudTrail → Event history: filter by Event source
kms.amazonaws.comto seeGenerateDataKeyandDecryptcalls made by Terraform's S3 reads and writes (Bucket Keys reduce how many you see).
AWS CLI
# What's in the state prefix?
aws s3 ls s3://hl-m10-tfstate-111122223333-us-east-1/audit/learn/ --profile learn-sandbox
# Every version of the state file (newest first)
aws s3api list-object-versions \
--bucket hl-m10-tfstate-111122223333-us-east-1 \
--prefix audit/learn/terraform.tfstate \
--query "Versions[].{Id:VersionId,Latest:IsLatest,Modified:LastModified}" \
--output table --profile learn-sandbox
# Download an older version to inspect (never edit state by hand)
aws s3api get-object \
--bucket hl-m10-tfstate-111122223333-us-east-1 \
--key audit/learn/terraform.tfstate \
--version-id "PASTE_VERSION_ID" previous.tfstate --profile learn-sandbox
# Create a "hand-made" bucket for the import exercise
aws s3api create-bucket --bucket hl-m10-l9-legacy-111122223333 \
--region us-east-1 --profile learn-sandbox
aws s3api put-bucket-tagging --bucket hl-m10-l9-legacy-111122223333 \
--tagging 'TagSet=[{Key=Project,Value=cea-lab},{Key=Lab,Value=m10-l9},{Key=Environment,Value=learn},{Key=Owner,Value=yourname}]' \
--profile learn-sandboxTerraform state commands
tf state list # every address in state
tf state show aws_s3_bucket.audit # attributes of one resource
tf plan -refresh-only # show drift without proposing config changes
tf apply -refresh-only # accept reality into state (use with care)
tf force-unlock LOCK_ID # only when you are SURE no run holds the lock
tf state pull > snapshot.tfstate # read-only local copy for inspectionPrefer the declarative moved, removed and import blocks over the imperative state mv, state rm and terraform import commands: blocks go through code review and appear in the plan; commands change state immediately with no review.
Infrastructure as Code
The same idea in the other tools in this module, for comparison:
terraform {
backend "s3" {
bucket = "hl-m10-tfstate-111122223333-us-east-1"
key = "audit/learn/terraform.tfstate"
region = "us-east-1"
encrypt = true
kms_key_id = "alias/hl-m10-tfstate"
use_lockfile = true
}
}Complete code
Bootstrap configuration
terraform {
required_version = ">= 1.14.0, < 2.0.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 6.0"
}
}
# Local state on purpose: the bucket this creates does not exist yet
}
provider "aws" {
region = "us-east-1"
default_tags {
tags = {
Project = "cea-lab"
Lab = "m10-l9"
Environment = "learn"
Owner = var.owner
ManagedBy = "terraform"
}
}
}
variable "owner" {
type = string
}
variable "state_writer_role_arns" {
description = "Extra role ARNs allowed to use the key (e.g. a CI deploy role). Your SSO role works through IAM."
type = list(string)
default = []
}
data "aws_caller_identity" "current" {}
data "aws_region" "current" {}
locals {
account_id = data.aws_caller_identity.current.account_id
bucket_name = "hl-m10-tfstate-${local.account_id}-${data.aws_region.current.region}"
}
# --- KMS key for state ---------------------------------------------------
data "aws_iam_policy_document" "key" {
# Account root: lets IAM policies in this account grant use of the key
statement {
sid = "EnableIamPolicies"
actions = ["kms:*"]
resources = ["*"]
principals {
type = "AWS"
identifiers = ["arn:aws:iam::${local.account_id}:root"]
}
}
dynamic "statement" {
for_each = length(var.state_writer_role_arns) > 0 ? [1] : []
content {
sid = "AllowStateWriters"
actions = ["kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey", "kms:DescribeKey"]
resources = ["*"]
principals {
type = "AWS"
identifiers = var.state_writer_role_arns
}
condition {
test = "StringEquals"
variable = "kms:ViaService"
values = ["s3.${data.aws_region.current.region}.amazonaws.com"]
}
}
}
}
resource "aws_kms_key" "state" {
description = "Encrypts Terraform state for Harbourline learning labs"
enable_key_rotation = true
deletion_window_in_days = 7
policy = data.aws_iam_policy_document.key.json
}
resource "aws_kms_alias" "state" {
name = "alias/hl-m10-tfstate"
target_key_id = aws_kms_key.state.key_id
}
# --- State bucket --------------------------------------------------------
resource "aws_s3_bucket" "state" {
bucket = local.bucket_name
# No force_destroy: deleting state by accident must be hard
}
resource "aws_s3_bucket_ownership_controls" "state" {
bucket = aws_s3_bucket.state.id
rule {
object_ownership = "BucketOwnerEnforced"
}
}
resource "aws_s3_bucket_public_access_block" "state" {
bucket = aws_s3_bucket.state.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
resource "aws_s3_bucket_versioning" "state" {
bucket = aws_s3_bucket.state.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_server_side_encryption_configuration" "state" {
bucket = aws_s3_bucket.state.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "aws:kms"
kms_master_key_id = aws_kms_key.state.arn
}
bucket_key_enabled = true
}
}
resource "aws_s3_bucket_lifecycle_configuration" "state" {
bucket = aws_s3_bucket.state.id
rule {
id = "keep-90-days-of-history"
status = "Enabled"
filter {}
noncurrent_version_expiration {
noncurrent_days = 90
}
}
depends_on = [aws_s3_bucket_versioning.state]
}
data "aws_iam_policy_document" "bucket" {
statement {
sid = "DenyInsecureTransport"
effect = "Deny"
actions = ["s3:*"]
principals {
type = "*"
identifiers = ["*"]
}
resources = [aws_s3_bucket.state.arn, "${aws_s3_bucket.state.arn}/*"]
condition {
test = "Bool"
variable = "aws:SecureTransport"
values = ["false"]
}
}
statement {
sid = "DenyUnencryptedUploads"
effect = "Deny"
actions = ["s3:PutObject"]
principals {
type = "*"
identifiers = ["*"]
}
resources = ["${aws_s3_bucket.state.arn}/*"]
condition {
test = "StringNotEquals"
variable = "s3:x-amz-server-side-encryption"
values = ["aws:kms"]
}
}
}
resource "aws_s3_bucket_policy" "state" {
bucket = aws_s3_bucket.state.id
policy = data.aws_iam_policy_document.bucket.json
depends_on = [aws_s3_bucket_public_access_block.state]
}
output "state_bucket" {
value = aws_s3_bucket.state.bucket
}
output "state_kms_key_arn" {
value = aws_kms_key.state.arn
}Workload configuration using the backend
terraform {
# Partial configuration: values come from backend.hcl at init time
backend "s3" {}
}bucket = "hl-m10-tfstate-111122223333-us-east-1"
key = "audit/learn/terraform.tfstate"
region = "us-east-1"
encrypt = true
kms_key_id = "alias/hl-m10-tfstate"
use_lockfile = true# Copy local state into the bucket (answers 'yes' when prompted)
tf init -backend-config=backend.hcl -migrate-stateLeast-privilege policy for a state user
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListStatePrefix",
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::hl-m10-tfstate-111122223333-us-east-1",
"Condition": { "StringLike": { "s3:prefix": ["audit/learn/*"] } }
},
{
"Sid": "ReadWriteState",
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject"],
"Resource": "arn:aws:s3:::hl-m10-tfstate-111122223333-us-east-1/audit/learn/terraform.tfstate"
},
{
"Sid": "ManageLockFile",
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
"Resource": "arn:aws:s3:::hl-m10-tfstate-111122223333-us-east-1/audit/learn/terraform.tfstate.tflock"
},
{
"Sid": "UseStateKey",
"Effect": "Allow",
"Action": ["kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey"],
"Resource": "arn:aws:kms:us-east-1:111122223333:key/REPLACE-WITH-KEY-ID"
}
]
}Import, moved and removed blocks
# 1. Adopt a bucket somebody created by hand
import {
to = aws_s3_bucket.legacy
id = "hl-m10-l9-legacy-111122223333"
}
# 2. We renamed aws_s3_bucket.audit to aws_s3_bucket.evidence in main.tf.
# Tell Terraform it is the same object, so nothing is destroyed.
moved {
from = aws_s3_bucket.audit
to = aws_s3_bucket.evidence
}
# 3. Hand a resource over to another team's configuration:
# forget it in this state, but do NOT delete it in AWS.
removed {
from = aws_s3_bucket_lifecycle_configuration.old_rule
lifecycle {
destroy = false
}
}# Generate starter HCL for every import block that has no resource yet
tf plan -generate-config-out=generated.tf
# Review generated.tf, remove computed/defaulted arguments, move it into main.tf, then:
tf plan -out=tfplan # expect: 1 to import, 0 to add, 0 to change, 0 to destroy
tf apply tfplanCode explained line by line
Bootstrap
- No
backendblock — the bootstrap uses local state because it creates the bucket the others depend on. data "aws_region" "current"and.region— in AWS provider v6 the Region attribute of this data source isregion; older code used.name, which v6 deprecates ⚠ VERIFY against the provider changelog if you see a deprecation warning.statement "EnableIamPolicies"with the account root principal — the standard key-policy statement that delegates to IAM: identities in this account can use the key if their IAM policy allows it. Without it you can lock yourself out of the key.dynamic "statement"— generates theAllowStateWritersstatement only if you pass role ARNs.dynamicblocks create repeated nested blocks from a collection.kms:ViaService = s3.us-east-1.amazonaws.com— the role may use the key only through S3, not by calling KMS directly to decrypt arbitrary data.enable_key_rotation = true— yearly automatic rotation of the key material; old ciphertext still decrypts.deletion_window_in_days = 7— the shortest waiting period. During the window the key is not billed and can be recovered.- The bucket has no
force_destroy. Destroying the bootstrap with state objects still inside should fail; you will empty the bucket deliberately at the end. kms_master_key_id = aws_kms_key.state.arn— default encryption with your customer managed key, so access needs both S3 and KMS permission.noncurrent_version_expiration { noncurrent_days = 90 }— 90 days of state history; long enough to recover from a bad apply noticed late, short enough to limit storage.DenyUnencryptedUploads— refuses anyPutObjectthat does not request SSE-KMS. Terraform's backend withencrypt = truesends the header, so it passes; a careless manual upload does not.
Backend
backend "s3" {}with values inbackend.hcl— backend blocks cannot use variables, so partial configuration is the standard way to vary bucket and key per environment.key = "audit/learn/terraform.tfstate"— use a<component>/<environment>/pattern. It keeps state small (one component per state) and lets IAM policies grant per-prefix access.encrypt = truewithkms_key_id— the backend writes the state and lock objects with SSE-KMS using that key.use_lockfile = true— locking via aterraform.tfstate.tflockobject next to the state. The lock needss3:GetObject,s3:PutObjectands3:DeleteObjecton that key.
IAM policy
s3:ListBucketrestricted withs3:prefix— the backend lists to find state and workspaces; this stops it browsing other teams' prefixes.- Separate statements for the state object and the lock object — the state needs read and write; the lock also needs delete.
kms:Encrypt,kms:Decrypt,kms:GenerateDataKey— the three KMS actions the S3 backend documentation lists whenkms_key_idis set.
Adopt and refactor
import { to = …, id = … }— tells the next plan to read the existing object into state at that address. Theidformat depends on the resource type (a bucket name here; a security group ID such assg-…for security groups). The import happens duringapply, so it is reviewed like any other change.-generate-config-out=generated.tf— Terraform writes HCL for imported resources that have noresourceblock yet. The output is a starting point: it includes every argument, many of which are defaults you should delete.moved { from, to }— records a rename in code. Anyone applying later gets the same move automatically. Keepmovedblocks for a while (or permanently in shared modules) so every consumer picks them up.removed { from = …, lifecycle { destroy = false } }— removes the object from state without deleting it in AWS. Withoutdestroy = false, deleting aresourceblock means "destroy it".
Workspaces vs separate state
A workspace is a named, separate state for the same configuration and backend. terraform.workspace is available as a value in your code.
| Approach | How | Good for | Risk in a bank |
|---|---|---|---|
| CLI workspaces | terraform workspace new test; state at env:/test/<key> | Short-lived copies: a feature branch's preview environment, a test of a module | All environments share one backend, one bucket policy and usually one set of credentials. Running apply in the wrong workspace hits the wrong environment. The current workspace is invisible in the code |
| Separate root folders or separate backend files per environment | envs/test/backend.hcl, envs/prod/backend.hcl, each with its own key — or its own bucket in its own account | Long-lived dev, test and prod | More files, but each environment can have separate credentials, a separate state bucket and separate approvals |
Harbourline's rule: production state lives in the production account's (or a tooling account's) dedicated bucket, accessed only by the production deploy role. Workspaces are allowed only for ephemeral environments. The next-but-one lesson, Environment configuration and versioning, builds the folder layout.
Secrets and state
Some resources return or accept secret values: an RDS master password, a generated random_password, a TLS private key from the tls provider. Unless the resource supports write-only arguments (Terraform 1.11 and later, for example password_wo on some database resources ⚠ VERIFY per resource in the provider docs), the value is stored in state.
Safer patterns, in order of preference:
- Let AWS generate and hold the secret. For RDS, set
manage_master_user_password = true; RDS creates and rotates the password in Secrets Manager and Terraform never sees it. - Use write-only arguments or ephemeral resources where the provider supports them, so the value is used during apply and never persisted.
- Create the secret container in Terraform, put the value in later through a separate, audited process.
- If a secret must pass through Terraform, accept that the state is secret-bearing and restrict it accordingly (which you should do anyway).
Security considerations
⚠ Learning shortcut — not for production
In this lesson the state bucket, the KMS key and the workload all live in the same sandbox account, and your SSO role has broad rights to all of them. In production, put state buckets in a dedicated tooling account (or one bucket per environment account), allow only the pipeline's deploy roles to read and write each prefix, keep human access to a break-glass role with MFA and alerting, and turn on CloudTrail S3 data events for the state bucket so every read of state is recorded.
- Never edit state by hand. If you must repair state, use
moved,removed,importor, as a last resort,terraform statecommands — afterterraform state pullto a backup. force-unlockis a break-glass action. Only use it after confirming the run that holds the lock is truly dead (check the CI job and the lock'sWhoandCreatedfields). Unlocking during a live apply can corrupt state.- Versioning is your undo button. Test restoring a previous version once a quarter; record the test as audit evidence.
- Deny
s3:DeleteObjecton the state key to everyone except a break-glass role if your team's process allows it — the lock file still needs delete, so scope the deny to*.tfstatekeys only. - Sensitive values in state — see the section above. Classify the state bucket at the same level as the most sensitive value it can contain.
Banking use case
Harbourline's platform team runs one state bucket per environment account (hl-tfstate-<account>-eu-west-2), each encrypted with its own customer managed key. The production key policy allows kms:Decrypt only to the production deploy role and the break-glass role. Each component — network, payments, ledger-db, observability — has its own state key, so a mistake in the observability plan cannot touch the database's state, and plans stay fast.
When the Data Platform team migrated 140 hand-built Glue jobs and S3 buckets, they did it with import blocks generated by a script from an inventory CSV, reviewed in batches of 20, each batch's plan showing only imports. No resource was recreated. The pull requests became the evidence that the migration did not change production.
Common mistakes
- Creating the state bucket inside the configuration that uses it → chicken and egg; destroying that configuration deletes its own state → use a separate bootstrap configuration.
- Copying old guides that create a DynamoDB lock table → extra cost and a deprecated feature →
use_lockfile = true. - One giant state for the whole bank → slow plans, every change risks everything, everyone needs access to all of it → split state by component and environment.
- Using workspaces for prod vs dev with the same credentials → one wrong
workspace selectand you apply to prod → separate backends and roles for long-lived environments. terraform state rmin a terminal to "fix" a plan → unreviewed and easy to forget, so the next person's plan recreates or orphans the resource → use aremovedblock.- Assuming
sensitive = trueprotects state → it only masks CLI output → avoid secrets in state, and protect state anyway. - Forgetting KMS permissions →
AccessDeniedonGenerateDataKeywhen the backend writes → add the three KMS actions for the state key.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Error acquiring the state lock … PreconditionFailed | Another run holds the lock | Wait, or use -lock-timeout=5m; check CI for a running job |
| Lock never released after a crash or killed CI job | The process died before deleting .tflock | Confirm no run is active, then terraform force-unlock LOCK_ID |
Backend configuration changed on init | You edited backend settings | init -reconfigure (start fresh) or init -migrate-state (copy state) — choose deliberately |
AccessDenied when writing state | Missing kms:GenerateDataKey or s3:PutObject on the key | Check both IAM and the KMS key policy |
AccessDenied on the .tflock object | Missing s3:DeleteObject or s3:PutObject for the lock key | Add the lock-file statement |
| Plan wants to create everything | Wrong workspace, wrong key, or init pointed at a different bucket | terraform workspace show; check backend.hcl |
Import fails with Cannot import non-existent remote object | Wrong id format or wrong Region/account | Check the resource's import ID format in the provider docs |
| Generated config has errors on plan | It includes conflicting or read-only arguments | Delete the arguments named in the error; keep only what you mean to manage |
BucketNotEmpty destroying bootstrap | Versioned state objects remain | Delete all object versions first (see cleanup) |
Cost and cleanup
| Item | Cost |
|---|---|
| Customer managed KMS key | $1 per month, prorated hourly — about $0.0014 per hour. Not billed while pending deletion |
| KMS requests | Within the 20,000 requests per month free tier for this lesson |
| S3 storage for a few state versions | Fractions of a cent |
| S3 requests for locking | Fractions of a cent |
| Lesson total if destroyed the same day | About $0.01 |
Free alternative: use the AWS managed key (sse_algorithm = "aws:kms" without kms_master_key_id, and no kms_key_id in the backend). You lose key-policy control, which is why the lesson uses a customer managed key.
# 1. Destroy workloads first (their state is in the bucket)
Set-Location audit; tf destroy; Set-Location ..
# 2. Delete the hand-made legacy bucket if you used a removed block instead of destroying it
aws s3api delete-bucket --bucket hl-m10-l9-legacy-111122223333 --profile learn-sandbox
# 3. Empty the versioned state bucket (all versions and delete markers)
$b = "hl-m10-tfstate-111122223333-us-east-1"
$v = aws s3api list-object-versions --bucket $b --profile learn-sandbox --output json | ConvertFrom-Json
foreach ($o in @($v.Versions) + @($v.DeleteMarkers)) {
if ($o) { aws s3api delete-object --bucket $b --key $o.Key --version-id $o.VersionId --profile learn-sandbox | Out-Null }
}
# 4. Destroy the bootstrap (KMS key enters 7-day pending deletion)
Set-Location bootstrap; tf destroy; Set-Location ..
# 5. Confirm nothing is left
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m10-l9 `
--profile learn-sandbox --region us-east-1 --query "ResourceTagMappingList[].ResourceARN"The KMS key may still appear in the tagging API while pending deletion; that is expected and not billed.
Exercises
- 🟢 Guided: Bootstrap the state bucket, migrate the
auditconfiguration's state into it, run the two-terminal locking experiment, and list the state object's versions. - 🟡 Challenge: Create a security group by hand with the CLI in your default VPC. Adopt it with an
importblock and-generate-config-out. Tidy the generated HCL until the plan shows only the import. Then rename its address with amovedblock and confirm the plan shows a move and nothing else. - 🔴 Live problem: It is 18:40. The nightly CI job that applies the
paymentsconfiguration failed withError acquiring the state lock. The lock info saysWho: runner@ci-7f3a,Created: 14:02. A developer in the chat says "just force-unlock it". What do you check before you do anything, and what do you do if you find the 14:02 job is still running?
Assignment
Add a bootstrap/ folder to your harbourline-iac-terraform repository and convert the audit configuration to the S3 backend with partial configuration. Add docs/state-runbook.md covering: where state lives per environment, who can read it, how to recover a previous version (tested, with the date), and the force-unlock procedure with its pre-checks.
Solutions
After tf init -backend-config=backend.hcl -migrate-state, Terraform prints a message that it successfully configured the backend and asks: Do you want to copy existing state to the new backend? Answer yes. Delete the local terraform.tfstate and terraform.tfstate.backup afterwards — they are now stale and confusing.
In the locking test, window B prints a block like:
Error: Error acquiring the state lock
Lock Info:
ID: 3c1f…
Path: hl-m10-tfstate-111122223333-us-east-1/audit/learn/terraform.tfstate
Operation: OperationTypeApply
Who: root@4e2a9c1b
Version: 1.17.2
Created: 2026-11-18 10:14:03 UTCWho shows root@<container-id> because Terraform runs as root inside the container. In CI, set meaningful runner names so the lock tells you which job holds it.
aws ec2 create-security-group --group-name hl-m10-l9-legacy-sg \
--description "Hand-made SG for import exercise" \
--vpc-id vpc-0123456789abcdef0 --profile learn-sandboximport {
to = aws_security_group.legacy
id = "sg-0abc1234def567890"
}tf plan -generate-config-out=generated.tf produces a full aws_security_group block, including egress (the default allow-all rule), revoke_rules_on_delete, tags and tags_all. Keep name, description, vpc_id and the egress you actually want; delete tags_all (computed) and empty or defaulted arguments. The plan should read Plan: 1 to import, 0 to add, 0 to change, 0 to destroy. If it also shows a tag change, that is default_tags adding ManagedBy — acceptable and visible.
Then rename the block to aws_security_group.payments_legacy and add:
moved {
from = aws_security_group.legacy
to = aws_security_group.payments_legacy
}The plan shows # aws_security_group.legacy has moved to aws_security_group.payments_legacy and zero changes. You can now remove the import block; it has done its job. Keep the moved block until every environment has applied it.
Before touching the lock:
- Look up the CI job that started at 14:02 (runner
ci-7f3a). Is it still running, hung, or dead? A job that ran for 4.5 hours is suspicious but may be stuck waiting on a resource (for example an RDS modification). - Check CloudTrail for API calls from the deploy role since 14:02 — are changes still happening?
- Pull a backup:
terraform state pull > payments-backup-1840.tfstate(reading does not need the lock).
If the 14:02 job is still running: do not force-unlock. Find out what it is waiting for. Cancel the job cleanly from CI if necessary, which lets Terraform release the lock, or wait for its timeout. Force-unlocking while it still writes would let a second apply interleave with it and corrupt the state.
If it is confirmed dead: run terraform force-unlock <ID> with the exact lock ID, then a plan (not apply) to see whether the dead run left anything half-done. Record the incident: why did a job hang, and should CI jobs have a hard timeout?
Knowledge check
Answer: use_lockfile = true. It creates a lock object named after the state key with a .tflock suffix, written with S3 conditional writes so only one run can create it. DynamoDB-based locking (dynamodb_table) is deprecated.
Answer: So a corrupted or wrongly written state can be restored from a previous version. It also protects against accidental deletion. Versioning plus a noncurrent-version lifecycle rule gives recovery without unlimited storage growth.
Answer: At <workspace_key_prefix>/<workspace name>/<key>, with env: as the default prefix — for example env:/test/audit/learn/terraform.tfstate.
Answer: Deleting the resource block makes the next plan destroy the real object. A removed block with lifecycle { destroy = false } makes Terraform forget the object in state but leave it running in AWS — used when handing a resource to another configuration or tool.
Answer: The backend is configured during init, before variables are evaluated. Use partial configuration: an empty or minimal backend "s3" block plus -backend-config=file.hcl (or -backend-config="key=value") at init time.
Answer: kms:Encrypt, kms:Decrypt and kms:GenerateDataKey on that key — granted through IAM and allowed by the key policy.
Interview questions
- "What is in Terraform state and how do you protect it?" — Resource addresses mapped to real IDs, all known attributes, outputs and dependency metadata — which can include secrets. I keep it in a versioned, KMS-encrypted S3 bucket with Block Public Access, TLS-only policy, per-prefix IAM, native lock files, and CloudTrail data events, and I avoid putting secrets through Terraform where possible.
- "Two applies ran at once and state is now wrong. What do you do?" — Stop further applies, pull and save the current state, compare with the previous S3 version, run a
plan -refresh-onlyto see the gap between state and reality, and repair withimport/removed/movedblocks or by restoring the previous version followed by a refresh. Then fix the root cause: locking was off, or two backends pointed at different keys. - "Workspaces or folders for environments?" — Folders (or separate backend configurations) with separate credentials for long-lived environments, because isolation and explicitness matter more than fewer files. Workspaces for short-lived copies.
- "How do you bring 200 hand-made resources under Terraform?" — Inventory them, generate
importblocks, use-generate-config-outfor starter HCL, tidy it, and review plans that contain imports only — in batches. Nothing is recreated, and the PRs are the evidence. - "Why is DynamoDB no longer needed for locking?" — S3 supports conditional writes, so the backend can create a lock object atomically in the same bucket. HashiCorp has deprecated DynamoDB locking in the S3 backend.
Certification practice
A team stores Terraform state in S3. Twice this month, two pipeline runs applied the same configuration at once and the state lost track of resources. The team wants the simplest supported fix with no extra services. What should they do?
A. Create a DynamoDB table with a LockID key and set dynamodb_table in the backend.
B. Set use_lockfile = true in the S3 backend configuration.
C. Enable S3 Object Lock in compliance mode on the state bucket.
D. Serialise the pipelines by giving them the same CodeBuild project.
Answer: B. Native S3 lock files prevent concurrent writes without another service. A works but uses a deprecated mechanism and an extra service. C is wrong: Object Lock makes objects immutable for a retention period — Terraform could not update the state at all. D does not guarantee serialisation (other pipelines or humans can still run) and is not a lock.
An administrator must make sure Terraform state in S3 cannot be read by developers who have read access to most S3 buckets in the account, while the deploy role can still use it. Which combination is MOST effective? (Choose TWO.)
A. Encrypt the state with a customer managed KMS key whose key policy grants decrypt only to the deploy role and a break-glass role.
B. Mark all Terraform outputs as sensitive.
C. Add a bucket policy that denies s3:GetObject on state keys to every principal except the deploy and break-glass roles.
D. Enable S3 Transfer Acceleration.
E. Store the state in the default workspace only.
Answer: A and C. Both layers independently stop developers: without KMS decrypt the object cannot be read, and the bucket policy denies the read before KMS is involved. B only masks CLI output; values remain in state. D is about upload speed. E has nothing to do with access control.
An organisation has 60 security groups created manually. It wants Terraform to manage them without any downtime and with every change reviewed in a pull request. What should the engineer do?
A. Write matching resource blocks and run terraform apply; Terraform detects existing groups by name.
B. Add import blocks for each group, run terraform plan -generate-config-out, review and tidy the generated configuration, and apply a plan that shows only imports.
C. Delete the groups and let Terraform recreate them in a maintenance window.
D. Run terraform import for each group from an engineer's laptop, then commit the state file.
Answer: B. Import blocks are declarative and reviewed, and the plan proves nothing is created or destroyed. A is wrong: Terraform does not adopt existing objects by name; it tries to create new ones (and duplicate names in the same VPC fail). C causes downtime. D bypasses review and commits state to Git.
Completion checklist
- I can explain why Terraform needs state and why it is sensitive
- I bootstrapped an encrypted, versioned state bucket and migrated state into it
- I saw the lock stop a second run, and I know the
force-unlockpre-checks - I adopted a resource with an
importblock and renamed one withmoved - I emptied and destroyed everything, and the tagging check shows only the pending-deletion KMS key
Key takeaways
- State maps your code to real resources. Store it in S3 with versioning, SSE-KMS, Block Public Access, a TLS-only policy and narrow IAM.
- Use
use_lockfile = true; DynamoDB locking is deprecated. - Split state by component and environment; use separate backends and credentials for long-lived environments, workspaces only for short-lived ones.
- Change what Terraform manages with
import,movedandremovedblocks so the change is reviewed. - Keep secrets out of state where you can; protect state as if it contains them anyway.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Terraform fundamentals
Write, plan and apply your first Terraform configuration for AWS: an encrypted, private audit-evidence bucket for Harbourline Bank. You learn providers, resources, data sources, variables, outputs and the init, plan, apply and destroy workflow.
Terraform modules
Package a secure S3 bucket as a reusable Terraform module with validated inputs, outputs and a terraform test suite, version it with Git tags, call it from registry and Git sources, and decide when a module is worth writing at all.