Incident response on AWS: runbooks, containment, forensics and evidence
Prepare for and run security incidents the way a bank must: a lifecycle and roles agreed in advance, runbooks for a leaked access key and a suspicious data download, isolating a compromised EC2 instance or credential without destroying evidence, preserving evidence with integrity, and knowing when legal and compliance must decide on notification.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.IdentityManagement 4.1.1.2, AWSSDK.EC2 4.1.4) · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Describe the incident-response lifecycle (prepare; detect and analyse; contain, eradicate and recover; post-incident activity) and the roles a bank needs.
- Run two core runbooks end to end: "leaked access key" and "suspicious data download".
- Contain compromised credentials correctly: deactivate IAM user keys, revoke active role sessions with a
aws:TokenIssueTimedeny, and decide what to do with a KMS key. - Isolate a compromised EC2 instance without destroying evidence: isolation security group, termination protection, removal from Auto Scaling and load balancers, EBS snapshots, and memory capture considerations.
- Preserve evidence with hashes, chain of custody and an Object Lock–protected forensic bucket in a dedicated account.
- Build response automation from currently available services (EventBridge, SNS, Systems Manager Automation) — Systems Manager Incident Manager is closed to new customers — and explain where the AWS Security Incident Response service fits.
- Explain why the decision to notify the ICO, the FCA/PRA or customers belongs to legal, compliance and the DPO, and what technical facts they will need from you.
Prerequisites
- CloudTrail for banks — the primary source of evidence
- Security Hub, GuardDuty, Inspector and Detective — where incidents are detected
- IAM in depth and IAM Identity Center, MFA and temporary credentials — sessions and credentials
- Tools: AWS CLI v2 (
learn-sandboxprofile), .NET 10 SDK
Why this matters in banking
Every bank will have security incidents. What regulators, customers and boards judge is how the bank responds: how fast it detected the problem, whether it contained it without making things worse, whether it can say precisely what data was affected, and whether it notified the right people on time.
Two scenarios recur in cloud environments:
- Leaked access key. A developer commits a long-lived IAM user key to a public repository. Automated scanners run by attackers find leaked keys within minutes. The attacker lists buckets, creates new users for persistence and launches crypto-mining instances in unused Regions.
- Suspicious data download. A legitimate role, used from an unusual location, downloads 40,000 objects from the customer-statements bucket at 02:00. Is it a breach, a misconfigured batch job, or an insider?
For a UK bank, a personal data breach may need to be reported to the ICO within 72 hours of becoming aware of it, where it's likely to result in a risk to people's rights and freedoms, and to affected individuals where the risk is high. Operational and security incidents may also need to be reported to the FCA and PRA under their own rules. Those decisions are legal and compliance decisions. Your job as an engineer is to make sure they have accurate facts quickly — which is only possible if logging, runbooks and access were prepared beforehand.
The idea in plain language
Incident response (IR) is the organised way to handle a security event so that damage is limited, evidence is kept, and the business recovers.
The widely used NIST lifecycle has four phases:
- Preparation — roles, contact lists, runbooks, tooling, access (break-glass roles), logging and practice (game days).
- Detection and analysis — a finding arrives; you triage it, decide whether it's an incident, scope it and record a timeline.
- Containment, eradication and recovery — stop the bleeding (short-term containment), remove the attacker's foothold (eradication), restore service safely (recovery).
- Post-incident activity — lessons learned, control improvements, evidence for regulators.
Two principles dominate in the cloud:
- Contain before you clean, preserve before you change. Terminating a compromised instance feels decisive but destroys memory and disk evidence. Isolate it first, snapshot it, then decide.
- Credentials are the perimeter. Most AWS incidents involve misuse of credentials. Containment usually means disabling keys and revoking sessions, not pulling network cables.
Analogy: If a branch's vault is broken into, you don't bulldoze the branch. You lock the doors so nobody else gets in (containment), keep everything as it was for the police (evidence), change every key and code (eradication), reopen with extra guards (recovery), and then write up what failed so other branches can fix it (lessons learned). The incident manager is the branch manager who coordinates, not the person who dusts for fingerprints.
What problem the AWS services solve
| Need | AWS building block (status checked 2026-11-18) | Notes |
|---|---|---|
| Detect | GuardDuty, Security Hub, Macie, CloudWatch alarms | See the detection lesson |
| Notify and coordinate | EventBridge → SNS (pager, chat, email); your ticketing tool | Systems Manager Incident Manager closed to new customers on 2025-11-07 — don't build new designs on it |
| Automate containment | Systems Manager Automation runbooks; Lambda; Step Functions | Runbooks are versioned documents with approval steps |
| Investigate | CloudTrail + Athena, Detective, VPC Flow Logs, CloudWatch Logs Insights | Evidence must be retained before you need it |
| Preserve evidence | EBS snapshots, S3 with Object Lock in a forensic account, KMS | Copy snapshots to the forensic account |
| Expert help | AWS Security Incident Response (managed service: triage of findings, case management, access to the AWS Customer Incident Response Team) | Metered pricing since November 2025: first 10,000 findings per month free, then per-finding tiers; included for Enterprise Support customers ⚠ VERIFY terms for your support plan |
| Revoke sessions | IAM inline deny on aws:TokenIssueTime ("Revoke active sessions" in the console) | Applies to role sessions issued before a time |
| Stop credential use | UpdateAccessKey to Inactive; AWSDenyAll managed policy; SCPs for whole accounts | Keep the key record for investigation; delete later |
Honest limitations:
- There's no single AWS "IR button". You assemble detection, automation and process yourself.
- Security groups don't always cut existing connections. Tracked connections may continue after you change an instance's security groups; untracked ones stop. For immediate isolation, combine the security-group change with a network ACL deny, or follow AWS's documented approach ⚠ VERIFY the current guidance in the EC2 security group connection-tracking documentation.
- Revoking role sessions doesn't affect sessions issued after the revocation time — so if the attacker can still assume the role (stolen user key, compromised CI pipeline), you must remove that path too.
- Memory is volatile. If you need memory forensics, you must capture it from the running instance with tools you've prepared in advance (for example an approved memory-capture tool in an SSM document). After stop or terminate, it's gone.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Microsoft Sentinel incidents and playbooks (Logic Apps) | Security Hub / GuardDuty findings + EventBridge + SSM Automation / Step Functions | Code-first runbooks; many banks still use their SIEM's case management |
| Entra ID "Revoke sessions" | IAM "Revoke active sessions" (inline deny on aws:TokenIssueTime) | Applies to IAM role sessions; Identity Center users also need their portal session and permission set access removed |
| Disable a service principal secret | UpdateAccessKey --status Inactive | Keys are inactive, not deleted, until the investigation is done |
| Defender for Endpoint "Isolate device" | Isolation security group + NACL + termination protection | No agent needed, but connection tracking matters |
| Azure disk snapshots for forensics | EBS snapshots copied to a forensic account | Share with a KMS key the forensic account can use |
| Azure Immutable Blob for evidence | S3 Object Lock (compliance mode in production) | Same WORM idea |
| Microsoft Incident Response (DART) | AWS Security Incident Response / AWS CIRT | Managed service with metered pricing |
Architecture
Trust boundaries: responders act through a pre-provisioned IR role in every account (assumable only from the security tooling account, with MFA, logged and alerted). Evidence leaves the affected account for a forensic account that the attacker can't reach even if they hold admin in the workload account.
The lifecycle as a timeline
Step-by-step implementation
This section is the preparation work; the runbooks follow.
Define roles and severities
Name an incident manager (coordinates, decides), technical lead (investigates), communications lead, and scribe (timeline). Define severity levels with examples, and who must be told at each level — including the DPO, legal and compliance for any incident that may involve personal data.
Pre-provision access
Create an hl-incident-responder role in every account via StackSets or Control Tower customisations, trusted only by the security tooling account, requiring MFA, with permissions for containment and read-only investigation. Alert on every assumption. Test it quarterly.
Make sure evidence exists before you need it
Organisation CloudTrail with data events for sensitive buckets, VPC Flow Logs for production VPCs, DNS query logs, load balancer and WAF logs, application logs with correlation IDs — all retained in the log-archive account.
Build the forensic account
A dedicated account with an evidence bucket (Object Lock, SSE-KMS, versioning), a KMS key that workload accounts can use to share snapshots, an isolated VPC with no internet access for analysis instances, and access limited to the forensic team.
Write and automate runbooks
Start with the two runbooks below. Automate the safe, repeatable steps (deactivate key, isolate instance, snapshot) with SSM Automation and a containment tool; keep human decision points explicit (approval steps).
Practise
Run a tabletop exercise each quarter and a technical game day twice a year (for example with GuardDuty sample findings or a planted canary key). Measure time to detect, time to contain, and gaps.
Runbook IR-01: leaked access key
| # | Action | Command / tool | Why |
|---|---|---|---|
| 1 | Open the incident, start the timeline | Ticket + chat channel | Every action must be recorded with time and actor |
| 2 | Identify the key's owner and account | aws iam get-access-key-last-used --access-key-id AKIA... (in the account that owns it) | Returns the user name, last service and Region used |
| 3 | Deactivate the key (don't delete yet) | aws iam update-access-key --user-name U --access-key-id AKIA... --status Inactive | Stops new requests; keeps the key record for investigation |
| 4 | Block the user completely | aws iam attach-user-policy --user-name U --policy-arn arn:aws:iam::aws:policy/AWSDenyAll | The attacker may have created a second key or a console password |
| 5 | Find everything the key did | Athena on CloudTrail: useridentity.accesskeyid = 'AKIA...', all Regions, from first leak time | Scope; look for CreateUser, CreateAccessKey, AssumeRole, RunInstances, PutBucketPolicy, data events |
| 6 | Follow assumed roles | Athena: events where useridentity.sessioncontext.sessionissuer.arn matches roles the key assumed | Attackers pivot via roles |
| 7 | Revoke those role sessions | "Revoke active sessions" (IAM console) or the containment tool | Temporary credentials stay valid until expiry otherwise |
| 8 | Remove persistence | Delete attacker-created users, keys, roles, policies, Lambda functions, EC2 instances (snapshot first), unknown SSH keys | Eradication |
| 9 | Check every Region | aws ec2 describe-instances --region <each>; GuardDuty in all Regions | Crypto-mining in unused Regions is common |
| 10 | Assess data access | S3 data events, KMS Decrypt events, RDS logs | Facts for the DPO: what data, how many people, which categories |
| 11 | Hand facts to DPO/legal/compliance | Written summary with timeline and uncertainty | They decide on ICO/regulator/customer notification |
| 12 | Recover and improve | Replace long-lived keys with roles/OIDC; secrets scanning in CI; SCP preventing IAM user key creation | Root cause |
Runbook IR-04: suspicious data download
| # | Action | Detail |
|---|---|---|
| 1 | Triage the signal | GuardDuty S3 finding (e.g. anomalous behaviour), Macie alert, or a CloudWatch alarm on GetObject volume |
| 2 | Identify the principal and session | CloudTrail S3 data events: userIdentity.arn, sessionContext, sourceIPAddress, userAgent |
| 3 | Decide: authorised or not? | Check change tickets, batch schedules, the owning team — within minutes, not hours |
| 4 | If unauthorised: contain | Revoke the role's older sessions; if a user, deactivate keys; add a bucket policy deny for that principal; consider blocking the source IP via WAF/NACL if relevant |
| 5 | Preserve evidence | Export the relevant CloudTrail records and S3 server access logs to the evidence bucket with SHA-256 hashes |
| 6 | Scope the data | List object keys accessed; map to data classification and number of customers; check whether objects were encrypted with keys the attacker could also use (KMS Decrypt events) |
| 7 | Inform DPO, legal, compliance | Facts, uncertainties and timeline; they decide on notifications |
| 8 | Eradicate and recover | Fix the access path (over-broad role, leaked credentials, vulnerable app) |
AWS Console instructions
- Revoke role sessions: Console → IAM → Roles → (role) → Revoke sessions (sometimes labelled Revoke active sessions). This adds an inline policy (named
AWSRevokeOlderSessions) that denies all actions for sessions issued before now. - Deactivate a key: IAM → Users → (user) → Security credentials → Access keys → Actions → Deactivate.
- Isolate an instance: EC2 → Instances → (instance) → Actions → Security → Change security groups → remove all, add
hl-ir-isolation(no inbound or outbound rules). Actions → Instance settings → Change termination protection → Enable. - Snapshot volumes: EC2 → Instances → (instance) → Actions → Image and templates → Create snapshot(s) ⚠ VERIFY the menu path; or Elastic Block Store → Snapshots → Create snapshot → Instance.
- Detach from Auto Scaling: EC2 → Auto Scaling groups → (group) → Instance management → Detach (choose to replace the instance so capacity is kept).
- Disable a KMS key (only on decision): KMS → Customer managed keys → (key) → Key actions → Disable.
AWS CLI
PROFILE=learn-sandbox
REGION=us-east-1
# ---------- IR-01: leaked access key ----------
KEY=AKIAIOSFODNN7EXAMPLE
aws iam get-access-key-last-used --access-key-id $KEY --profile $PROFILE
USER=$(aws iam get-access-key-last-used --access-key-id $KEY --query UserName --output text --profile $PROFILE)
aws iam update-access-key --user-name $USER --access-key-id $KEY --status Inactive --profile $PROFILE
aws iam attach-user-policy --user-name $USER --policy-arn arn:aws:iam::aws:policy/AWSDenyAll --profile $PROFILE
aws iam tag-user --user-name $USER --tags Key=IncidentId,Value=INC-2026-0042 --profile $PROFILE
# Other credentials the attacker may have added
aws iam list-access-keys --user-name $USER --profile $PROFILE
aws iam get-login-profile --user-name $USER --profile $PROFILE # error = no console password (good)
# ---------- Revoke a role's active sessions ----------
NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
aws iam put-role-policy --role-name hl-batch-recon-role --policy-name AWSRevokeOlderSessions --profile $PROFILE \
--policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":["*"],"Resource":["*"],
"Condition":{"DateLessThan":{"aws:TokenIssueTime":"'"$NOW"'"}}}]}'
# ---------- IR-03: isolate an EC2 instance ----------
IID=i-0123456789abcdef0
VPC=$(aws ec2 describe-instances --instance-ids $IID --query "Reservations[0].Instances[0].VpcId" --output text --profile $PROFILE --region $REGION)
SG=$(aws ec2 create-security-group --group-name hl-ir-isolation-$IID --description "IR isolation INC-2026-0042" \
--vpc-id $VPC --query GroupId --output text --profile $PROFILE --region $REGION)
# New SGs allow all outbound by default — remove it so the SG has no rules at all
aws ec2 revoke-security-group-egress --group-id $SG --ip-permissions '[{"IpProtocol":"-1","IpRanges":[{"CidrIp":"0.0.0.0/0"}]}]' \
--profile $PROFILE --region $REGION
aws ec2 modify-instance-attribute --instance-id $IID --groups $SG --profile $PROFILE --region $REGION
aws ec2 modify-instance-attribute --instance-id $IID --disable-api-termination --profile $PROFILE --region $REGION
aws ec2 modify-instance-attribute --instance-id $IID --no-disable-api-stop --profile $PROFILE --region $REGION # keep stop possible
# If it belongs to an Auto Scaling group: detach and let the ASG replace it
# aws autoscaling detach-instances --instance-ids $IID --auto-scaling-group-name hl-recon-asg --no-should-decrement-desired-capacity
# If it's behind a target group: deregister
# aws elbv2 deregister-targets --target-group-arn <tg-arn> --targets Id=$IID
# Snapshot every attached volume (crash-consistent), tagged for the incident
aws ec2 create-snapshots --instance-specification InstanceId=$IID,ExcludeBootVolume=false \
--description "INC-2026-0042 evidence $IID" --copy-tags-from-source volume \
--tag-specifications 'ResourceType=snapshot,Tags=[{Key=IncidentId,Value=INC-2026-0042},{Key=Evidence,Value=true}]' \
--profile $PROFILE --region $REGION
aws ec2 create-tags --resources $IID --tags Key=IncidentId,Value=INC-2026-0042 Key=Quarantine,Value=true \
--profile $PROFILE --region $REGION
# Optional: ask GuardDuty Malware Protection to scan the instance's volumes on demand
# aws guardduty start-malware-scan --resource-arn arn:aws:ec2:us-east-1:111122223333:instance/$IID
# ---------- Preserve evidence with hashes ----------
aws ec2 get-console-output --instance-id $IID --latest --output text --profile $PROFILE --region $REGION > console-$IID.txt
sha256sum console-$IID.txt > console-$IID.txt.sha256
aws s3 cp console-$IID.txt s3://hl-forensic-evidence-222233334444/INC-2026-0042/ --profile $PROFILE
aws s3 cp console-$IID.txt.sha256 s3://hl-forensic-evidence-222233334444/INC-2026-0042/ --profile $PROFILEThe evidence-bucket path and account are examples; in production the copy is made by a role that can write to the forensic account but not delete.
Infrastructure as Code
Two pieces of preparation as code: an SSM Automation runbook that isolates an instance (isolation SG, termination protection, snapshots, tags), and an evidence bucket with Object Lock.
using Amazon.CDK;
using Amazon.CDK.AWS.S3;
using Amazon.CDK.AWS.SSM;
using Constructs;
namespace Harbourline.Security.Infra;
public sealed class IncidentResponseStack : Stack
{
public IncidentResponseStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
Tags.SetTag("Project", "cea-lab");
Tags.SetTag("Lab", "m05-l21");
Tags.SetTag("Environment", "learn");
_ = new Bucket(this, "Evidence", new BucketProps
{
BucketName = $"hl-m05-evidence-{Account}",
BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
Encryption = BucketEncryption.S3_MANAGED, // production: KMS key in the forensic account
EnforceSSL = true,
Versioned = true,
ObjectLockEnabled = true,
// LEARNING SHORTCUT: governance, 1 day. Production: compliance mode, retention set by legal.
ObjectLockDefaultRetention = ObjectLockRetention.Governance(Duration.Days(1)),
RemovalPolicy = RemovalPolicy.RETAIN,
});
var content = new Dictionary<string, object>
{
["schemaVersion"] = "0.3",
["description"] = "IR-03: isolate an EC2 instance and snapshot its volumes. Does not stop or terminate.",
["assumeRole"] = "{{ AutomationAssumeRole }}",
["parameters"] = new Dictionary<string, object>
{
["InstanceId"] = new Dictionary<string, object> { ["type"] = "String" },
["IsolationSecurityGroupId"] = new Dictionary<string, object> { ["type"] = "String" },
["IncidentId"] = new Dictionary<string, object> { ["type"] = "String", ["allowedPattern"] = "^INC-[0-9]{4}-[0-9]{4}$" },
["AutomationAssumeRole"] = new Dictionary<string, object> { ["type"] = "String" },
},
["mainSteps"] = new object[]
{
new Dictionary<string, object>
{
["name"] = "ApproveIsolation",
["action"] = "aws:approve",
["inputs"] = new Dictionary<string, object>
{
["Approvers"] = new[] { "arn:aws:iam::" + Account + ":role/hl-incident-manager" },
["Message"] = "Isolate {{ InstanceId }} for {{ IncidentId }}?",
["MinRequiredApprovals"] = 1,
},
},
new Dictionary<string, object>
{
["name"] = "SwapSecurityGroups",
["action"] = "aws:executeAwsApi",
["inputs"] = new Dictionary<string, object>
{
["Service"] = "ec2", ["Api"] = "ModifyInstanceAttribute",
["InstanceId"] = "{{ InstanceId }}", ["Groups"] = new[] { "{{ IsolationSecurityGroupId }}" },
},
},
new Dictionary<string, object>
{
["name"] = "EnableTerminationProtection",
["action"] = "aws:executeAwsApi",
["inputs"] = new Dictionary<string, object>
{
["Service"] = "ec2", ["Api"] = "ModifyInstanceAttribute",
["InstanceId"] = "{{ InstanceId }}",
["DisableApiTermination"] = new Dictionary<string, object> { ["Value"] = true },
},
},
new Dictionary<string, object>
{
["name"] = "SnapshotVolumes",
["action"] = "aws:executeAwsApi",
["inputs"] = new Dictionary<string, object>
{
["Service"] = "ec2", ["Api"] = "CreateSnapshots",
["InstanceSpecification"] = new Dictionary<string, object> { ["InstanceId"] = "{{ InstanceId }}" },
["Description"] = "{{ IncidentId }} evidence {{ InstanceId }}",
["CopyTagsFromSource"] = "volume",
},
["outputs"] = new object[]
{
new Dictionary<string, object> { ["Name"] = "SnapshotIds", ["Selector"] = "$.Snapshots..SnapshotId", ["Type"] = "StringList" },
},
},
new Dictionary<string, object>
{
["name"] = "TagInstance",
["action"] = "aws:createTags",
["inputs"] = new Dictionary<string, object>
{
["ResourceType"] = "EC2",
["ResourceIds"] = new[] { "{{ InstanceId }}" },
["Tags"] = new object[]
{
new Dictionary<string, string> { ["Key"] = "IncidentId", ["Value"] = "{{ IncidentId }}" },
new Dictionary<string, string> { ["Key"] = "Quarantine", ["Value"] = "true" },
},
},
},
},
};
_ = new CfnDocument(this, "IsolateInstanceRunbook", new CfnDocumentProps
{
Name = "HL-IR03-IsolateEC2Instance",
DocumentType = "Automation",
DocumentFormat = "JSON",
Content = content,
});
}
}The automation role (AutomationAssumeRole) needs ec2:ModifyInstanceAttribute, ec2:CreateSnapshots, ec2:CreateSnapshot, ec2:CreateTags and ec2:DescribeInstances — scoped with conditions where possible ⚠ VERIFY the exact permissions CreateSnapshots needs in your account setup.
Start the runbook with:
aws ssm start-automation-execution --document-name HL-IR03-IsolateEC2Instance --profile learn-sandbox \
--parameters InstanceId=i-0123456789abcdef0,IsolationSecurityGroupId=sg-0aaaabbbbccccdddd,IncidentId=INC-2026-0042,AutomationAssumeRole=arn:aws:iam::111122223333:role/hl-ir-automationComplete code
The containment tool is what an on-call engineer runs under pressure. Its design rules:
- Dry-run by default. It prints the exact plan; nothing changes until you add
--execute. - Plan, then act. Each command builds an ordered list of steps; the executor runs them one by one and stops on the first failure.
- Evidence by design. Every step is written to a JSON-lines journal with UTC timestamps, the operator's identity and the outcome, and the journal's SHA-256 is printed at the end for chain of custody.
- No destructive actions. It deactivates, denies, isolates and snapshots. It never deletes keys or terminates instances.
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<RootNamespace>Harbourline.Security.Containment</RootNamespace>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AWSSDK.IdentityManagement" Version="4.1.1.2" />
<PackageReference Include="AWSSDK.EC2" Version="4.1.4" />
<PackageReference Include="AWSSDK.SecurityToken" Version="4.1.0.2" />
</ItemGroup>
</Project>using Amazon.EC2;
using Amazon.EC2.Model;
using Amazon.IdentityManagement;
using Amazon.IdentityManagement.Model;
namespace Harbourline.Security.Containment;
public sealed record Step(string Description, Func<CancellationToken, Task<string>> Action);
public static class Plans
{
public const string DenyAllPolicyArn = "arn:aws:iam::aws:policy/AWSDenyAll";
public static string RevokeOlderSessionsPolicy(DateTimeOffset cutoff) => $$"""
{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":["*"],"Resource":["*"],
"Condition":{"DateLessThan":{"aws:TokenIssueTime":"{{cutoff.UtcDateTime:yyyy-MM-ddTHH:mm:ssZ}}"} } }]}
""";
// IR-01: leaked IAM user access key.
public static IReadOnlyList<Step> LeakedKey(IAmazonIdentityManagementService iam, string userName, string accessKeyId, string incidentId) =>
[
new($"Deactivate access key {accessKeyId} of user {userName}", async ct =>
{
await iam.UpdateAccessKeyAsync(new UpdateAccessKeyRequest
{
UserName = userName, AccessKeyId = accessKeyId, Status = Amazon.IdentityManagement.StatusType.Inactive,
}, ct);
return "key inactive";
}),
new($"Attach {DenyAllPolicyArn} to user {userName}", async ct =>
{
await iam.AttachUserPolicyAsync(new AttachUserPolicyRequest { UserName = userName, PolicyArn = DenyAllPolicyArn }, ct);
return "deny-all attached";
}),
new($"Tag user {userName} with IncidentId={incidentId}", async ct =>
{
await iam.TagUserAsync(new TagUserRequest
{
UserName = userName,
Tags = [new Amazon.IdentityManagement.Model.Tag { Key = "IncidentId", Value = incidentId }],
}, ct);
return "tagged";
}),
new($"List remaining access keys of {userName} (attacker may have created more)", async ct =>
{
var keys = await iam.ListAccessKeysAsync(new ListAccessKeysRequest { UserName = userName }, ct);
var active = (keys.AccessKeyMetadata ?? []).Where(k => k.Status == Amazon.IdentityManagement.StatusType.Active).Select(k => k.AccessKeyId).ToList();
return active.Count == 0 ? "no other active keys" : $"STILL ACTIVE: {string.Join(", ", active)} — deactivate them";
}),
];
// Revoke all sessions of a role issued before 'cutoff'.
public static IReadOnlyList<Step> RevokeRoleSessions(IAmazonIdentityManagementService iam, string roleName, DateTimeOffset cutoff) =>
[
new($"Put inline policy AWSRevokeOlderSessions on role {roleName} (deny tokens issued before {cutoff:u})", async ct =>
{
await iam.PutRolePolicyAsync(new PutRolePolicyRequest
{
RoleName = roleName,
PolicyName = "AWSRevokeOlderSessions",
PolicyDocument = RevokeOlderSessionsPolicy(cutoff),
}, ct);
return "older sessions revoked";
}),
];
// IR-03: isolate an EC2 instance and preserve its disks. Never stops or terminates.
public static IReadOnlyList<Step> IsolateInstance(IAmazonEC2 ec2, string instanceId, string vpcId, string incidentId)
{
string? sgId = null;
return
[
new($"Create isolation security group in {vpcId}", async ct =>
{
var sg = await ec2.CreateSecurityGroupAsync(new CreateSecurityGroupRequest
{
GroupName = $"hl-ir-isolation-{instanceId}",
Description = $"IR isolation {incidentId}",
VpcId = vpcId,
}, ct);
sgId = sg.GroupId;
return sgId;
}),
new("Remove the default allow-all egress rule from the isolation group", async ct =>
{
await ec2.RevokeSecurityGroupEgressAsync(new RevokeSecurityGroupEgressRequest
{
GroupId = sgId,
IpPermissions = [new IpPermission { IpProtocol = "-1", Ipv4Ranges = [new IpRange { CidrIp = "0.0.0.0/0" }] }],
}, ct);
return "no inbound, no outbound";
}),
new($"Replace all security groups on {instanceId} with the isolation group", async ct =>
{
await ec2.ModifyInstanceAttributeAsync(new ModifyInstanceAttributeRequest { InstanceId = instanceId, Groups = [sgId!] }, ct);
return "isolated (check NACL for tracked connections)";
}),
new($"Enable termination protection on {instanceId}", async ct =>
{
await ec2.ModifyInstanceAttributeAsync(new ModifyInstanceAttributeRequest { InstanceId = instanceId, DisableApiTermination = true }, ct);
return "termination protected";
}),
new($"Snapshot all volumes of {instanceId}", async ct =>
{
var snaps = await ec2.CreateSnapshotsAsync(new CreateSnapshotsRequest
{
InstanceSpecification = new InstanceSpecification { InstanceId = instanceId, ExcludeBootVolume = false },
Description = $"{incidentId} evidence {instanceId}",
CopyTagsFromSource = CopyTagsFromSource.Volume,
TagSpecifications =
[
new TagSpecification
{
ResourceType = ResourceType.Snapshot,
Tags = [new Amazon.EC2.Model.Tag("IncidentId", incidentId), new Amazon.EC2.Model.Tag("Evidence", "true")],
},
],
}, ct);
return string.Join(", ", (snaps.Snapshots ?? []).Select(s => s.SnapshotId));
}),
new($"Tag {instanceId} as quarantined", async ct =>
{
await ec2.CreateTagsAsync(new CreateTagsRequest
{
Resources = [instanceId],
Tags = [new Amazon.EC2.Model.Tag("IncidentId", incidentId), new Amazon.EC2.Model.Tag("Quarantine", "true")],
}, ct);
return "tagged";
}),
];
}
}using System.Security.Cryptography;
using System.Text.Json;
namespace Harbourline.Security.Containment;
public sealed class Executor(TextWriter output, string journalPath, string operatorArn, TimeProvider clock)
{
public async Task<bool> RunAsync(string incidentId, IReadOnlyList<Step> plan, bool execute, CancellationToken ct = default)
{
output.WriteLine(execute ? "EXECUTING plan:" : "DRY RUN — nothing will change. Plan:");
for (var i = 0; i < plan.Count; i++) output.WriteLine($" {i + 1}. {plan[i].Description}");
if (!execute) return true;
await using var journal = new StreamWriter(journalPath, append: true);
foreach (var step in plan)
{
var started = clock.GetUtcNow();
string outcome; bool ok;
try { outcome = await step.Action(ct); ok = true; }
catch (Exception ex) { outcome = $"{ex.GetType().Name}: {ex.Message}"; ok = false; }
var entry = JsonSerializer.Serialize(new
{
incidentId, at = started, finished = clock.GetUtcNow(), operatorArn, step = step.Description, ok, outcome,
});
await journal.WriteLineAsync(entry);
output.WriteLine($" [{(ok ? "OK" : "FAILED")}] {step.Description} → {outcome}");
if (!ok) { output.WriteLine("Stopping: fix the failure, then re-run the remaining steps manually."); break; }
}
await journal.FlushAsync(ct);
journal.Close();
var hash = Convert.ToHexStringLower(SHA256.HashData(await File.ReadAllBytesAsync(journalPath, ct)));
output.WriteLine($"Journal {journalPath} SHA-256 {hash} — record this in the incident ticket.");
return true;
}
}using Amazon.EC2;
using Amazon.IdentityManagement;
using Amazon.SecurityToken;
using Amazon.SecurityToken.Model;
using Harbourline.Security.Containment;
// Usage:
// containment leaked-key <incident-id> <user-name> <access-key-id> [--execute]
// containment revoke-sessions <incident-id> <role-name> [--execute]
// containment isolate <incident-id> <instance-id> <vpc-id> [--execute]
if (args.Length < 3) { Console.Error.WriteLine("See usage in Program.cs"); return 2; }
var execute = args.Contains("--execute");
var a = args.Where(x => x != "--execute").ToArray();
var incidentId = a[1];
if (!System.Text.RegularExpressions.Regex.IsMatch(incidentId, "^INC-[0-9]{4}-[0-9]{4}$"))
{
Console.Error.WriteLine("Incident ID must look like INC-2026-0042");
return 2;
}
var who = await new AmazonSecurityTokenServiceClient().GetCallerIdentityAsync(new GetCallerIdentityRequest());
Console.WriteLine($"Operator: {who.Arn} in account {who.Account}");
IReadOnlyList<Step> plan = a[0] switch
{
"leaked-key" when a.Length == 4 => Plans.LeakedKey(new AmazonIdentityManagementServiceClient(), a[2], a[3], incidentId),
"revoke-sessions" when a.Length == 3 => Plans.RevokeRoleSessions(new AmazonIdentityManagementServiceClient(), a[2], DateTimeOffset.UtcNow),
"isolate" when a.Length == 4 => Plans.IsolateInstance(new AmazonEC2Client(), a[2], a[3], incidentId),
_ => [],
};
if (plan.Count == 0) { Console.Error.WriteLine("Unknown command or wrong arguments"); return 2; }
var executor = new Executor(Console.Out, $"{incidentId}-journal.jsonl", who.Arn, TimeProvider.System);
return await executor.RunAsync(incidentId, plan, execute) ? 0 : 1;A test that proves dry-run never touches AWS, and that the revoke policy is well-formed:
using System.Text.Json;
using Harbourline.Security.Containment;
using Xunit;
public sealed class ContainmentTests
{
[Fact]
public async Task Dry_run_executes_nothing()
{
var called = false;
IReadOnlyList<Step> plan = [new("Dangerous step", _ => { called = true; return Task.FromResult("done"); })];
var output = new StringWriter();
var executor = new Executor(output, Path.GetTempFileName(), "arn:aws:sts::111122223333:assumed-role/ir/test", TimeProvider.System);
await executor.RunAsync("INC-2026-0001", plan, execute: false, TestContext.Current.CancellationToken);
Assert.False(called);
Assert.Contains("DRY RUN", output.ToString());
}
[Fact]
public async Task Execution_stops_at_first_failure()
{
var secondRan = false;
IReadOnlyList<Step> plan =
[
new("fails", _ => throw new InvalidOperationException("boom")),
new("should not run", _ => { secondRan = true; return Task.FromResult("x"); }),
];
var executor = new Executor(new StringWriter(), Path.GetTempFileName(), "arn:test", TimeProvider.System);
await executor.RunAsync("INC-2026-0001", plan, execute: true, TestContext.Current.CancellationToken);
Assert.False(secondRan);
}
[Fact]
public void Revoke_policy_denies_tokens_before_cutoff()
{
var json = Plans.RevokeOlderSessionsPolicy(new DateTimeOffset(2026, 11, 18, 10, 30, 0, TimeSpan.Zero));
using var doc = JsonDocument.Parse(json);
var stmt = doc.RootElement.GetProperty("Statement")[0];
Assert.Equal("Deny", stmt.GetProperty("Effect").GetString());
Assert.Equal("2026-11-18T10:30:00Z",
stmt.GetProperty("Condition").GetProperty("DateLessThan").GetProperty("aws:TokenIssueTime").GetString());
}
}Code explained line by line
Step(Description, Action)— a plan is data: a readable description plus a delegate. That's what makes dry-run trivial and honest: you print exactly what would run.RevokeOlderSessionsPolicyproduces the same policy the console's "Revoke sessions" button creates: deny everything whenaws:TokenIssueTimeis earlier than the cutoff. Sessions issued after the cutoff still work — which is why you must also cut off whatever lets the attacker get new sessions.- The raw string
$$"""…"""lets the JSON's single braces stay literal while{{…}}interpolates the timestamp in ISO 8601 UTC format. LeakedKey— deactivate (not delete) the key, then attach the AWS managedAWSDenyAllpolicy so a second key or console password is useless too, then tag the user, then check for other active keys — attackers often create a backup key immediately.keys.AccessKeyMetadata ?? []— SDK v4 collections may benull; always guard.IsolateInstancekeeps the security-group ID in a captured variable (sgId) so later steps use the group created by the first step.- A new security group comes with an allow-all egress rule; the second step removes it so the group has no rules at all.
ModifyInstanceAttributewithGroupsreplaces all security groups on the primary network interface. Instances with several ENIs need each interface changed ⚠ VERIFY for multi-ENI instances.DisableApiTermination = truestops someone (or an Auto Scaling scale-in) terminating the evidence via the API.CreateSnapshotswith anInstanceSpecificationsnapshots every attached EBS volume at the same point in time (crash-consistent) and tags them as evidence.Executorprints the plan first, and in dry-run returns immediately — the test proves no action runs.- Each executed step writes one JSON line with start and finish times, operator ARN, description, success flag and outcome: an audit-quality action log.
- Execution stops at the first failure. In an incident, blindly continuing after a failure (for example snapshotting after isolation failed) creates confusion about the system's state.
- The journal's SHA-256 is printed so it can be recorded in the ticket; anyone can later prove the journal wasn't edited.
Program.csvalidates the incident ID format and records the operator's identity from STS — you always know who ran what.
Security considerations
⚠ Learning shortcut — not for production
The lab runs containment from your own sandbox role, keeps evidence in the same account with governance-mode Object Lock for one day, and uses a test instance and test user. In production: responders use a pre-provisioned, MFA-protected IR role assumed from the security tooling account; evidence goes to a separate forensic account with compliance-mode Object Lock and a dedicated KMS key; runbooks have approval steps; and every action is journalled in the incident record.
- Don't tip off an insider. For suspected insider cases, coordinate with HR and legal before visible containment actions; quietly preserve evidence first where the risk allows.
- Don't terminate, don't reboot. Both destroy volatile evidence. Isolate, snapshot, and only then decide.
- Decide deliberately about KMS keys. Disabling a customer managed key stops the attacker using it — and stops your own applications too, which may be a self-inflicted outage. Options in order of disruption: remove the attacker's access in the key policy and retire grants; revoke the compromised principal; disable the key; schedule deletion (never during an investigation — it can make evidence permanently unreadable). Rotation creates new key material but doesn't help with data keys the attacker already decrypted.
- Protect the IR role. It's powerful; alert on every use; restrict who can assume it; review its use after each incident.
- Evidence handling is personal-data processing. Forensic images contain customer data. Restrict access, log it, and define retention with legal.
- Communicate on a separate channel. If the attacker may have access to email or chat, use the out-of-band channel defined in your IR plan.
- Notification is not an engineering decision. Give the DPO, legal and compliance facts, timelines and uncertainties early; they decide what is reportable to the ICO, the FCA/PRA, card schemes or customers.
Banking use case
INC-2026-0042 (fictional), Saturday 14:05. GuardDuty raises UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS (High) for the role hl-batch-recon-role used by EC2 instances in the reconciliation account. The triage Lambda pages on-call.
| Time | Action | Who |
|---|---|---|
| 14:09 | On-call acknowledges, opens INC-2026-0042, declares severity 2, pages incident manager | On-call engineer |
| 14:15 | Detective/Athena: the role's credentials used from an external IP; ListBuckets, then GetObject on hl-prod-recon-exports (412 objects) | Technical lead |
| 14:18 | Containment tool: revoke-sessions for the role (dry-run reviewed, then --execute) | Technical lead, approved by IM |
| 14:22 | isolate for the two instances; snapshots taken; ASG detached with replacement | Technical lead |
| 14:40 | Root cause hypothesis: SSRF vulnerability in an internal reporting app on the instance exposing IMDSv1 credentials | Technical lead |
| 15:10 | DPO, legal and compliance briefed: 412 export files, each containing transaction summaries for business customers; personal data present for sole traders | Incident manager |
| 16:00 | Eradication: IMDSv2 enforced (HttpTokens=required) via launch template; app patched; new instances deployed | Platform team |
| Sunday | Forensic analysis of snapshots in the forensic account confirms no persistence | Forensics |
| Monday | Legal/DPO decide the breach is reportable to the ICO within 72 hours of awareness and prepare the notification; compliance assesses FCA reporting obligations | Legal, DPO, compliance |
| +2 weeks | Post-incident review: SCP requiring IMDSv2 for all new instances; Inspector P1 for SSRF-class findings; game day scheduled | Security |
Note what made the timeline possible: data events on the export bucket, GuardDuty everywhere, a pre-provisioned IR role, and a containment tool that had been tested.
Common mistakes
- Terminating the compromised instance immediately → evidence lost → isolate and snapshot first.
- Deleting the leaked key → can't correlate it later and loses metadata → deactivate; delete after the investigation.
- Revoking sessions but leaving the source → attacker gets fresh credentials → fix the path (stolen key, IMDSv1, CI secret).
- Only checking one Region → attacker resources elsewhere keep running → check every enabled Region; use an SCP to block unused Regions.
- No logging until the incident → can't scope data access → enable data events and flow logs before you need them.
- Engineers notifying regulators or customers on their own → inconsistent or premature statements → route through the incident manager, legal and DPO.
- Responders without access at 3 a.m. → delays while someone finds an admin → pre-provision and test the IR role.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Attacker activity continues after "Revoke sessions" | They obtain new sessions via another path (user key, instance role on a still-running instance) | Find the source in CloudTrail (sessionIssuer, sourceIPAddress); deactivate keys; isolate instances |
| Isolated instance still has an established SSH session | Connection tracking keeps existing tracked flows | Add a NACL deny for the subnet/host or follow AWS's documented isolation sequence |
UnauthorizedOperation on CreateSnapshots | IR role missing ec2:CreateSnapshots/ec2:CreateTags, or KMS key policy blocks encrypted volume snapshots | Add permissions; ensure the role can use the volume's KMS key |
| Snapshot can't be shared with the forensic account | Encrypted with an AWS managed key (can't be shared) | Copy the snapshot re-encrypted with a customer managed key that the forensic account can use, then share |
AccessDenied for PutRolePolicy on a role | SCP or permission boundary blocks IAM changes, or the role is protected (e.g. Control Tower roles) | Use the authorised IR role; follow the documented exception for protected roles |
| GuardDuty finding keeps reappearing | Root cause not fixed, or attacker persistence | Re-scope; check for new users/keys/roles/Lambda functions/instances |
| Can't tell which objects were downloaded | No S3 data events or server access logs | Enable them now for the future; use what's available (CloudFront logs, app logs) and record the uncertainty for the DPO |
Cost and cleanup
| Item (checked 2026-11-18) | Price | Lab exposure |
|---|---|---|
| IAM, revoke sessions, security groups, tags | Free | Free |
| Test EC2 instance (t4g.nano) | About USD 0.0042 per hour ⚠ VERIFY | 1 hour ≈ under USD 0.01 |
| EBS snapshots | About USD 0.05 per GB-month (incremental) | 8 GB for a day ≈ USD 0.01 |
| SSM Automation | Free tier covers lab use ⚠ VERIFY current Automation step pricing | Free |
| S3 evidence bucket | Cents | Cents |
| AWS Security Incident Response | First 10,000 findings per month free; then tiered per finding | Not enabled in the lab |
Free alternative: run the containment tool in dry-run against a fictional user, role and instance ID, and run the unit tests. Do the tabletop exercise in the Exercises section.
Lab practice (credits): create a test IAM user with one access key (never use it for anything), a t4g.nano instance in a sandbox VPC, then run each containment command with --execute.
PROFILE=learn-sandbox
REGION=us-east-1
# Remove the test user (detach policy, delete key, delete user)
aws iam detach-user-policy --user-name hl-m05-ir-test --policy-arn arn:aws:iam::aws:policy/AWSDenyAll --profile $PROFILE
aws iam delete-access-key --user-name hl-m05-ir-test --access-key-id <key-id> --profile $PROFILE
aws iam delete-user --user-name hl-m05-ir-test --profile $PROFILE
# Remove the revoke-sessions inline policy from the test role
aws iam delete-role-policy --role-name hl-m05-ir-test-role --policy-name AWSRevokeOlderSessions --profile $PROFILE
# Instance: remove termination protection, terminate, delete isolation SG and snapshots
aws ec2 modify-instance-attribute --instance-id $IID --no-disable-api-termination --profile $PROFILE --region $REGION
aws ec2 terminate-instances --instance-ids $IID --profile $PROFILE --region $REGION
aws ec2 wait instance-terminated --instance-ids $IID --profile $PROFILE --region $REGION
aws ec2 delete-security-group --group-id $SG --profile $PROFILE --region $REGION
for s in $(aws ec2 describe-snapshots --owner-ids self --filters Name=tag:IncidentId,Values=INC-2026-0042 \
--query "Snapshots[].SnapshotId" --output text --profile $PROFILE --region $REGION); do
aws ec2 delete-snapshot --snapshot-id $s --profile $PROFILE --region $REGION
done
# IaC: cdk destroy / terraform destroy (evidence bucket objects are deletable after the 1-day governance retention)
# Confirm
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--query "ResourceTagMappingList[].ResourceARN" --profile $PROFILE --region $REGIONExercises
- 🟢 Guided: Build and run the containment tool in dry-run for all three commands. Then, in your sandbox, create a test user with a key and a t4g.nano instance, and run
leaked-keyandisolatewith--execute. Confirm the key is inactive, the instance has only the isolation group, and snapshots exist. Record the journal hash. Clean up. - 🟡 Challenge: Add a
block-bucket-principalcommand that adds a bucket-policy statement denying all S3 actions on a given bucket to a given principal ARN, preserving existing statements. Make it dry-run by default and add a test for the policy merge. - 🔴 Live problem (tabletop): Friday 17:45. Macie reports that a bucket in the data-platform account containing a 2 GB CSV of customer names, dates of birth and IBANs became publicly readable 6 hours ago after a Terraform change. S3 server access logs show 37
GETrequests for the file from 5 IPs not owned by Harbourline. Run the incident: what do you do in the first hour, what facts do you give the DPO, and what changes afterwards?
Assignment
Create Harbourline's IR pack in your portfolio: severity matrix and roles; runbooks IR-01 (leaked key), IR-03 (compromised EC2) and IR-04 (suspicious data download) as Markdown checklists with exact commands; the SSM Automation document as IaC; the containment tool with tests; and a one-page "facts for the DPO" template (what happened, when, which data and how many people, containment status, uncertainties).
Solutions
Dry-run output for isolate lists six steps and changes nothing. With --execute, verify:
aws iam list-access-keys --user-name hl-m05-ir-test --query "AccessKeyMetadata[].Status" # ["Inactive"]
aws iam list-attached-user-policies --user-name hl-m05-ir-test # AWSDenyAll
aws ec2 describe-instances --instance-ids $IID --query "Reservations[0].Instances[0].SecurityGroups"
aws ec2 describe-instance-attribute --instance-id $IID --attribute disableApiTermination
aws ec2 describe-snapshots --owner-ids self --filters Name=tag:IncidentId,Values=INC-2026-0001
sha256sum INC-2026-0001-journal.jsonl # matches the hash the tool printedWhy each check matters: you verify outcomes, not just that commands returned success. In a real incident the scribe records each verification in the timeline.
Read the current policy with GetBucketPolicyAsync (handle NoSuchBucketPolicy as an empty policy), parse it with System.Text.Json.Nodes.JsonNode, append:
{ "Sid": "IRBlockINC2026xxxx", "Effect": "Deny", "Principal": { "AWS": "<principal-arn>" },
"Action": "s3:*", "Resource": ["arn:aws:s3:::<bucket>", "arn:aws:s3:::<bucket>/*"] }and write it back with PutBucketPolicyAsync. Test the merge function with an existing two-statement policy and assert three statements with the original two unchanged. Why a deny in the bucket policy: it applies even if the principal's own IAM policies allow access, and it's local to the data, so it's effective while you work out the principal's other access paths. Remember to remove it after recovery.
First 15 minutes: declare a severity-1 incident; incident manager, DPO and legal informed immediately (personal data and IBANs). Contain: re-enable Block Public Access on the bucket (and check account-level BPA); confirm public access is gone. Preserve: export S3 server access logs and CloudTrail records (the Terraform apply, the policy change) to the evidence bucket with hashes; snapshot the Terraform state and plan.
Scope (first hour): exact window of public exposure (from CloudTrail PutBucketPolicy/PutPublicAccessBlock times); every request to the object (server access logs: IP, time, bytes sent — did they download the full 2 GB?); how many customers are in the file and which data categories.
Facts for the DPO: what was exposed, for how long, to how many requests and IPs, whether full downloads happened, number of individuals and data categories (names, dates of birth, IBANs — financial data increases risk), containment time, remaining uncertainties. The DPO and legal assess the 72-hour ICO notification and customer communication; fraud teams may need to monitor affected accounts.
Afterwards: root cause in the Terraform change (who approved, why the plan review missed it); preventive controls — account-level and organisation-level S3 Block Public Access, an SCP denying s3:PutBucketPublicAccessBlock changes, IaC scanning in the pipeline, Config rule with automatic remediation; post-incident review within two weeks.
Knowledge check
Answer: Deactivation stops its use immediately but keeps the key's record, which helps investigation and correlation in CloudTrail. You delete it after the investigation, once no longer needed.
Answer: It adds an inline policy (AWSRevokeOlderSessions) to the role that denies all actions for sessions whose aws:TokenIssueTime is before the time you revoked. New sessions issued afterwards are unaffected.
Answer: Swap to an isolation security group with no rules (plus NACL for tracked connections if needed); enable termination protection; detach from Auto Scaling and deregister from load balancers; capture memory if required and tooling is ready; snapshot all volumes; tag; copy evidence to the forensic account. Don't reboot or terminate.
Answer: There's no direct one-for-one replacement; new designs combine CloudWatch alarms, EventBridge, SNS, Systems Manager Automation runbooks and the organisation's own incident tooling, optionally with AWS Security Incident Response for triage and expert support.
Answer: The bank's DPO, legal and compliance functions, based on facts provided by the incident team. Engineers provide accurate, timely facts and uncertainties.
Interview questions
- "Walk me through your first 30 minutes after a leaked access key is reported." — Open the incident; identify owner via
get-access-key-last-used; deactivate the key; deny-all the user; check for other keys/console access; query CloudTrail by access key ID in all Regions; revoke sessions for any assumed roles; look for persistence; inform the incident manager and, if data may be involved, the DPO. - "Why not just terminate a compromised instance?" — It destroys memory and disk evidence you need to understand scope and root cause. Isolate, protect from termination, snapshot, then rebuild clean and terminate later under control.
- "How do you prepare for incidents in AWS?" — Logging in place (CloudTrail data events, flow logs), pre-provisioned IR roles, forensic account, runbooks automated with approvals, out-of-band comms, and regular game days.
- "What's the risk in disabling a KMS key during an incident?" — It breaks every legitimate workload using the key. Prefer targeted key-policy and grant changes; disable only when the business accepts the outage; never schedule deletion during an investigation.
Certification practice
GuardDuty reports that temporary credentials for an EC2 instance role are being used from an IP address outside AWS. The security team must immediately prevent further use of the stolen credentials while keeping the instance available for forensic analysis. Which actions should they take? (Choose TWO.)
- A. Delete the IAM role
- B. Revoke the role's active sessions by adding a policy that denies requests with an
aws:TokenIssueTimeearlier than the current time - C. Terminate the instance
- D. Isolate the instance with a security group that has no rules, enable termination protection and take EBS snapshots
- E. Rotate the KMS key used by the instance's EBS volumes
Answer: B and D. B invalidates the stolen session credentials; D contains the instance (the source of the credentials) while preserving evidence. A breaks other workloads and destroys useful context. C destroys evidence. E doesn't affect stolen role credentials.
A forensic team needs to analyse an EBS volume from a compromised instance in a separate forensic account. The volume is encrypted with the AWS managed key aws/ebs. What should the team do?
- A. Share the snapshot directly with the forensic account
- B. Create a snapshot, copy it re-encrypted with a customer managed KMS key whose key policy allows the forensic account, then share the copy
- C. Make the snapshot public temporarily
- D. Detach the volume and attach it to an instance in the forensic account
Answer: B. Snapshots encrypted with an AWS managed key can't be shared; re-encrypt with a shareable customer managed key. A fails for that reason. C exposes customer data. D — volumes can't be attached across accounts.
A company wants a repeatable, auditable way for on-call engineers to isolate EC2 instances during incidents, with manager approval before execution. Which solution fits best?
- A. A wiki page with CLI commands
- B. A Systems Manager Automation runbook with an
aws:approvestep followed byaws:executeAwsApisteps - C. An EC2 user data script
- D. A CloudFormation stack per incident
Answer: B. Automation runbooks are versioned, logged and support approval steps. A isn't enforced or auditable. C runs at boot. D is the wrong tool for imperative response actions.
A bank wants to ensure that, after a security incident, it can determine which objects were read from an S3 bucket containing customer documents. What must be in place BEFORE an incident?
- A. S3 Versioning
- B. CloudTrail data events (or S3 server access logging) for the bucket, retained in a separate account
- C. S3 Transfer Acceleration
- D. Default encryption with SSE-S3
Answer: B. Object-level reads are only recorded if data events or access logging are enabled in advance. A preserves versions but doesn't record reads. C speeds uploads. D encrypts but doesn't log access.
Completion checklist
- I can explain the IR lifecycle and the roles in a bank's incident team
- I can execute the leaked-key and suspicious-download runbooks step by step
- I revoked role sessions and isolated a test instance without terminating it
- I preserved evidence with hashes and know why it belongs in a separate forensic account
- I know which services are closed (Incident Manager) and what to use instead
- I cleaned up the test user, role policy, instance, security group and snapshots
Key takeaways
- Incident response is prepared before the incident: logging, access, runbooks, forensic account and practice.
- In AWS, credentials are the usual target: deactivate keys, deny-all the user, revoke role sessions — and close the path that issued them.
- Contain before you clean and preserve before you change: isolate, protect from termination, snapshot; don't terminate or reboot.
- Automate repeatable steps (SSM Automation with approvals, a dry-run-first tool) and journal every action with timestamps and hashes.
- Notification to the ICO, regulators and customers is decided by DPO, legal and compliance; engineers supply fast, accurate facts and honest uncertainty.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Vulnerability management: Amazon Inspector, patching SLAs and Patch Manager
Run vulnerability management the way a bank must: continuous Amazon Inspector scanning of EC2, ECR images and Lambda functions, risk-based prioritisation with CVSS, EPSS and reachability, patching SLAs with evidence, Systems Manager Patch Manager, and a governed exception process.
Backup security and key rotation: AWS Backup, Vault Lock, cross-account copies and zero-downtime rotation
Make Harbourline's backups survive ransomware and rogue administrators with AWS Backup vault policies, Vault Lock, logically air-gapped vaults and cross-account copies, and rotate KMS keys, secrets, access keys and signing keys without an outage.