Skip to Content
Cloud Engineering Academy
5. Security for Banking & Finance

Incident response on AWS: runbooks, containment, forensics and evidence

Prepare for and run security incidents the way a bank must: a lifecycle and roles agreed in advance, runbooks for a leaked access key and a suspicious data download, isolating a compromised EC2 instance or credential without destroying evidence, preserving evidence with integrity, and knowing when legal and compliance must decide on notification.

advanced⏱ 2 h 30 min🟡 Low costSCS-C03SOA-C03SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.IdentityManagement 4.1.1.2, AWSSDK.EC2 4.1.4) · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Describe the incident-response lifecycle (prepare; detect and analyse; contain, eradicate and recover; post-incident activity) and the roles a bank needs.
  • Run two core runbooks end to end: "leaked access key" and "suspicious data download".
  • Contain compromised credentials correctly: deactivate IAM user keys, revoke active role sessions with a aws:TokenIssueTime deny, and decide what to do with a KMS key.
  • Isolate a compromised EC2 instance without destroying evidence: isolation security group, termination protection, removal from Auto Scaling and load balancers, EBS snapshots, and memory capture considerations.
  • Preserve evidence with hashes, chain of custody and an Object Lock–protected forensic bucket in a dedicated account.
  • Build response automation from currently available services (EventBridge, SNS, Systems Manager Automation) — Systems Manager Incident Manager is closed to new customers — and explain where the AWS Security Incident Response service fits.
  • Explain why the decision to notify the ICO, the FCA/PRA or customers belongs to legal, compliance and the DPO, and what technical facts they will need from you.

Prerequisites

Why this matters in banking

Every bank will have security incidents. What regulators, customers and boards judge is how the bank responds: how fast it detected the problem, whether it contained it without making things worse, whether it can say precisely what data was affected, and whether it notified the right people on time.

Two scenarios recur in cloud environments:

  1. Leaked access key. A developer commits a long-lived IAM user key to a public repository. Automated scanners run by attackers find leaked keys within minutes. The attacker lists buckets, creates new users for persistence and launches crypto-mining instances in unused Regions.
  2. Suspicious data download. A legitimate role, used from an unusual location, downloads 40,000 objects from the customer-statements bucket at 02:00. Is it a breach, a misconfigured batch job, or an insider?

For a UK bank, a personal data breach may need to be reported to the ICO within 72 hours of becoming aware of it, where it's likely to result in a risk to people's rights and freedoms, and to affected individuals where the risk is high. Operational and security incidents may also need to be reported to the FCA and PRA under their own rules. Those decisions are legal and compliance decisions. Your job as an engineer is to make sure they have accurate facts quickly — which is only possible if logging, runbooks and access were prepared beforehand.

The idea in plain language

Incident response (IR) is the organised way to handle a security event so that damage is limited, evidence is kept, and the business recovers.

The widely used NIST lifecycle has four phases:

  1. Preparation — roles, contact lists, runbooks, tooling, access (break-glass roles), logging and practice (game days).
  2. Detection and analysis — a finding arrives; you triage it, decide whether it's an incident, scope it and record a timeline.
  3. Containment, eradication and recovery — stop the bleeding (short-term containment), remove the attacker's foothold (eradication), restore service safely (recovery).
  4. Post-incident activity — lessons learned, control improvements, evidence for regulators.

Two principles dominate in the cloud:

  • Contain before you clean, preserve before you change. Terminating a compromised instance feels decisive but destroys memory and disk evidence. Isolate it first, snapshot it, then decide.
  • Credentials are the perimeter. Most AWS incidents involve misuse of credentials. Containment usually means disabling keys and revoking sessions, not pulling network cables.

Analogy: If a branch's vault is broken into, you don't bulldoze the branch. You lock the doors so nobody else gets in (containment), keep everything as it was for the police (evidence), change every key and code (eradication), reopen with extra guards (recovery), and then write up what failed so other branches can fix it (lessons learned). The incident manager is the branch manager who coordinates, not the person who dusts for fingerprints.

What problem the AWS services solve

NeedAWS building block (status checked 2026-11-18)Notes
DetectGuardDuty, Security Hub, Macie, CloudWatch alarmsSee the detection lesson
Notify and coordinateEventBridge → SNS (pager, chat, email); your ticketing toolSystems Manager Incident Manager closed to new customers on 2025-11-07 — don't build new designs on it
Automate containmentSystems Manager Automation runbooks; Lambda; Step FunctionsRunbooks are versioned documents with approval steps
InvestigateCloudTrail + Athena, Detective, VPC Flow Logs, CloudWatch Logs InsightsEvidence must be retained before you need it
Preserve evidenceEBS snapshots, S3 with Object Lock in a forensic account, KMSCopy snapshots to the forensic account
Expert helpAWS Security Incident Response (managed service: triage of findings, case management, access to the AWS Customer Incident Response Team)Metered pricing since November 2025: first 10,000 findings per month free, then per-finding tiers; included for Enterprise Support customers ⚠ VERIFY terms for your support plan
Revoke sessionsIAM inline deny on aws:TokenIssueTime ("Revoke active sessions" in the console)Applies to role sessions issued before a time
Stop credential useUpdateAccessKey to Inactive; AWSDenyAll managed policy; SCPs for whole accountsKeep the key record for investigation; delete later

Honest limitations:

  • There's no single AWS "IR button". You assemble detection, automation and process yourself.
  • Security groups don't always cut existing connections. Tracked connections may continue after you change an instance's security groups; untracked ones stop. For immediate isolation, combine the security-group change with a network ACL deny, or follow AWS's documented approach ⚠ VERIFY the current guidance in the EC2 security group connection-tracking documentation.
  • Revoking role sessions doesn't affect sessions issued after the revocation time — so if the attacker can still assume the role (stolen user key, compromised CI pipeline), you must remove that path too.
  • Memory is volatile. If you need memory forensics, you must capture it from the running instance with tools you've prepared in advance (for example an approved memory-capture tool in an SSM document). After stop or terminate, it's gone.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Microsoft Sentinel incidents and playbooks (Logic Apps)Security Hub / GuardDuty findings + EventBridge + SSM Automation / Step FunctionsCode-first runbooks; many banks still use their SIEM's case management
Entra ID "Revoke sessions"IAM "Revoke active sessions" (inline deny on aws:TokenIssueTime)Applies to IAM role sessions; Identity Center users also need their portal session and permission set access removed
Disable a service principal secretUpdateAccessKey --status InactiveKeys are inactive, not deleted, until the investigation is done
Defender for Endpoint "Isolate device"Isolation security group + NACL + termination protectionNo agent needed, but connection tracking matters
Azure disk snapshots for forensicsEBS snapshots copied to a forensic accountShare with a KMS key the forensic account can use
Azure Immutable Blob for evidenceS3 Object Lock (compliance mode in production)Same WORM idea
Microsoft Incident Response (DART)AWS Security Incident Response / AWS CIRTManaged service with metered pricing

Architecture

Trust boundaries: responders act through a pre-provisioned IR role in every account (assumable only from the security tooling account, with MFA, logged and alerted). Evidence leaves the affected account for a forensic account that the attacker can't reach even if they hold admin in the workload account.

The lifecycle as a timeline

Step-by-step implementation

This section is the preparation work; the runbooks follow.

Define roles and severities

Name an incident manager (coordinates, decides), technical lead (investigates), communications lead, and scribe (timeline). Define severity levels with examples, and who must be told at each level — including the DPO, legal and compliance for any incident that may involve personal data.

Pre-provision access

Create an hl-incident-responder role in every account via StackSets or Control Tower customisations, trusted only by the security tooling account, requiring MFA, with permissions for containment and read-only investigation. Alert on every assumption. Test it quarterly.

Make sure evidence exists before you need it

Organisation CloudTrail with data events for sensitive buckets, VPC Flow Logs for production VPCs, DNS query logs, load balancer and WAF logs, application logs with correlation IDs — all retained in the log-archive account.

Build the forensic account

A dedicated account with an evidence bucket (Object Lock, SSE-KMS, versioning), a KMS key that workload accounts can use to share snapshots, an isolated VPC with no internet access for analysis instances, and access limited to the forensic team.

Write and automate runbooks

Start with the two runbooks below. Automate the safe, repeatable steps (deactivate key, isolate instance, snapshot) with SSM Automation and a containment tool; keep human decision points explicit (approval steps).

Practise

Run a tabletop exercise each quarter and a technical game day twice a year (for example with GuardDuty sample findings or a planted canary key). Measure time to detect, time to contain, and gaps.

Runbook IR-01: leaked access key

#ActionCommand / toolWhy
1Open the incident, start the timelineTicket + chat channelEvery action must be recorded with time and actor
2Identify the key's owner and accountaws iam get-access-key-last-used --access-key-id AKIA... (in the account that owns it)Returns the user name, last service and Region used
3Deactivate the key (don't delete yet)aws iam update-access-key --user-name U --access-key-id AKIA... --status InactiveStops new requests; keeps the key record for investigation
4Block the user completelyaws iam attach-user-policy --user-name U --policy-arn arn:aws:iam::aws:policy/AWSDenyAllThe attacker may have created a second key or a console password
5Find everything the key didAthena on CloudTrail: useridentity.accesskeyid = 'AKIA...', all Regions, from first leak timeScope; look for CreateUser, CreateAccessKey, AssumeRole, RunInstances, PutBucketPolicy, data events
6Follow assumed rolesAthena: events where useridentity.sessioncontext.sessionissuer.arn matches roles the key assumedAttackers pivot via roles
7Revoke those role sessions"Revoke active sessions" (IAM console) or the containment toolTemporary credentials stay valid until expiry otherwise
8Remove persistenceDelete attacker-created users, keys, roles, policies, Lambda functions, EC2 instances (snapshot first), unknown SSH keysEradication
9Check every Regionaws ec2 describe-instances --region <each>; GuardDuty in all RegionsCrypto-mining in unused Regions is common
10Assess data accessS3 data events, KMS Decrypt events, RDS logsFacts for the DPO: what data, how many people, which categories
11Hand facts to DPO/legal/complianceWritten summary with timeline and uncertaintyThey decide on ICO/regulator/customer notification
12Recover and improveReplace long-lived keys with roles/OIDC; secrets scanning in CI; SCP preventing IAM user key creationRoot cause

Runbook IR-04: suspicious data download

#ActionDetail
1Triage the signalGuardDuty S3 finding (e.g. anomalous behaviour), Macie alert, or a CloudWatch alarm on GetObject volume
2Identify the principal and sessionCloudTrail S3 data events: userIdentity.arn, sessionContext, sourceIPAddress, userAgent
3Decide: authorised or not?Check change tickets, batch schedules, the owning team — within minutes, not hours
4If unauthorised: containRevoke the role's older sessions; if a user, deactivate keys; add a bucket policy deny for that principal; consider blocking the source IP via WAF/NACL if relevant
5Preserve evidenceExport the relevant CloudTrail records and S3 server access logs to the evidence bucket with SHA-256 hashes
6Scope the dataList object keys accessed; map to data classification and number of customers; check whether objects were encrypted with keys the attacker could also use (KMS Decrypt events)
7Inform DPO, legal, complianceFacts, uncertainties and timeline; they decide on notifications
8Eradicate and recoverFix the access path (over-broad role, leaked credentials, vulnerable app)

AWS Console instructions

  • Revoke role sessions: Console → IAM → Roles → (role) → Revoke sessions (sometimes labelled Revoke active sessions). This adds an inline policy (named AWSRevokeOlderSessions) that denies all actions for sessions issued before now.
  • Deactivate a key: IAM → Users → (user) → Security credentials → Access keys → Actions → Deactivate.
  • Isolate an instance: EC2 → Instances → (instance) → Actions → Security → Change security groups → remove all, add hl-ir-isolation (no inbound or outbound rules). Actions → Instance settings → Change termination protection → Enable.
  • Snapshot volumes: EC2 → Instances → (instance) → Actions → Image and templates → Create snapshot(s) ⚠ VERIFY the menu path; or Elastic Block Store → Snapshots → Create snapshot → Instance.
  • Detach from Auto Scaling: EC2 → Auto Scaling groups → (group) → Instance management → Detach (choose to replace the instance so capacity is kept).
  • Disable a KMS key (only on decision): KMS → Customer managed keys → (key) → Key actions → Disable.

AWS CLI

PROFILE=learn-sandbox
REGION=us-east-1

# ---------- IR-01: leaked access key ----------
KEY=AKIAIOSFODNN7EXAMPLE
aws iam get-access-key-last-used --access-key-id $KEY --profile $PROFILE
USER=$(aws iam get-access-key-last-used --access-key-id $KEY --query UserName --output text --profile $PROFILE)

aws iam update-access-key --user-name $USER --access-key-id $KEY --status Inactive --profile $PROFILE
aws iam attach-user-policy --user-name $USER --policy-arn arn:aws:iam::aws:policy/AWSDenyAll --profile $PROFILE
aws iam tag-user --user-name $USER --tags Key=IncidentId,Value=INC-2026-0042 --profile $PROFILE
# Other credentials the attacker may have added
aws iam list-access-keys --user-name $USER --profile $PROFILE
aws iam get-login-profile --user-name $USER --profile $PROFILE   # error = no console password (good)

# ---------- Revoke a role's active sessions ----------
NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
aws iam put-role-policy --role-name hl-batch-recon-role --policy-name AWSRevokeOlderSessions --profile $PROFILE \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":["*"],"Resource":["*"],
    "Condition":{"DateLessThan":{"aws:TokenIssueTime":"'"$NOW"'"}}}]}'

# ---------- IR-03: isolate an EC2 instance ----------
IID=i-0123456789abcdef0
VPC=$(aws ec2 describe-instances --instance-ids $IID --query "Reservations[0].Instances[0].VpcId" --output text --profile $PROFILE --region $REGION)
SG=$(aws ec2 create-security-group --group-name hl-ir-isolation-$IID --description "IR isolation INC-2026-0042" \
  --vpc-id $VPC --query GroupId --output text --profile $PROFILE --region $REGION)
# New SGs allow all outbound by default — remove it so the SG has no rules at all
aws ec2 revoke-security-group-egress --group-id $SG --ip-permissions '[{"IpProtocol":"-1","IpRanges":[{"CidrIp":"0.0.0.0/0"}]}]' \
  --profile $PROFILE --region $REGION

aws ec2 modify-instance-attribute --instance-id $IID --groups $SG --profile $PROFILE --region $REGION
aws ec2 modify-instance-attribute --instance-id $IID --disable-api-termination --profile $PROFILE --region $REGION
aws ec2 modify-instance-attribute --instance-id $IID --no-disable-api-stop --profile $PROFILE --region $REGION  # keep stop possible
# If it belongs to an Auto Scaling group: detach and let the ASG replace it
# aws autoscaling detach-instances --instance-ids $IID --auto-scaling-group-name hl-recon-asg --no-should-decrement-desired-capacity
# If it's behind a target group: deregister
# aws elbv2 deregister-targets --target-group-arn <tg-arn> --targets Id=$IID

# Snapshot every attached volume (crash-consistent), tagged for the incident
aws ec2 create-snapshots --instance-specification InstanceId=$IID,ExcludeBootVolume=false \
  --description "INC-2026-0042 evidence $IID" --copy-tags-from-source volume \
  --tag-specifications 'ResourceType=snapshot,Tags=[{Key=IncidentId,Value=INC-2026-0042},{Key=Evidence,Value=true}]' \
  --profile $PROFILE --region $REGION
aws ec2 create-tags --resources $IID --tags Key=IncidentId,Value=INC-2026-0042 Key=Quarantine,Value=true \
  --profile $PROFILE --region $REGION

# Optional: ask GuardDuty Malware Protection to scan the instance's volumes on demand
# aws guardduty start-malware-scan --resource-arn arn:aws:ec2:us-east-1:111122223333:instance/$IID

# ---------- Preserve evidence with hashes ----------
aws ec2 get-console-output --instance-id $IID --latest --output text --profile $PROFILE --region $REGION > console-$IID.txt
sha256sum console-$IID.txt > console-$IID.txt.sha256
aws s3 cp console-$IID.txt s3://hl-forensic-evidence-222233334444/INC-2026-0042/ --profile $PROFILE
aws s3 cp console-$IID.txt.sha256 s3://hl-forensic-evidence-222233334444/INC-2026-0042/ --profile $PROFILE

The evidence-bucket path and account are examples; in production the copy is made by a role that can write to the forensic account but not delete.

Infrastructure as Code

Two pieces of preparation as code: an SSM Automation runbook that isolates an instance (isolation SG, termination protection, snapshots, tags), and an evidence bucket with Object Lock.

IncidentResponseStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.S3;
using Amazon.CDK.AWS.SSM;
using Constructs;

namespace Harbourline.Security.Infra;

public sealed class IncidentResponseStack : Stack
{
    public IncidentResponseStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        Tags.SetTag("Project", "cea-lab");
        Tags.SetTag("Lab", "m05-l21");
        Tags.SetTag("Environment", "learn");

        _ = new Bucket(this, "Evidence", new BucketProps
        {
            BucketName = $"hl-m05-evidence-{Account}",
            BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
            Encryption = BucketEncryption.S3_MANAGED,   // production: KMS key in the forensic account
            EnforceSSL = true,
            Versioned = true,
            ObjectLockEnabled = true,
            // LEARNING SHORTCUT: governance, 1 day. Production: compliance mode, retention set by legal.
            ObjectLockDefaultRetention = ObjectLockRetention.Governance(Duration.Days(1)),
            RemovalPolicy = RemovalPolicy.RETAIN,
        });

        var content = new Dictionary<string, object>
        {
            ["schemaVersion"] = "0.3",
            ["description"] = "IR-03: isolate an EC2 instance and snapshot its volumes. Does not stop or terminate.",
            ["assumeRole"] = "{{ AutomationAssumeRole }}",
            ["parameters"] = new Dictionary<string, object>
            {
                ["InstanceId"] = new Dictionary<string, object> { ["type"] = "String" },
                ["IsolationSecurityGroupId"] = new Dictionary<string, object> { ["type"] = "String" },
                ["IncidentId"] = new Dictionary<string, object> { ["type"] = "String", ["allowedPattern"] = "^INC-[0-9]{4}-[0-9]{4}$" },
                ["AutomationAssumeRole"] = new Dictionary<string, object> { ["type"] = "String" },
            },
            ["mainSteps"] = new object[]
            {
                new Dictionary<string, object>
                {
                    ["name"] = "ApproveIsolation",
                    ["action"] = "aws:approve",
                    ["inputs"] = new Dictionary<string, object>
                    {
                        ["Approvers"] = new[] { "arn:aws:iam::" + Account + ":role/hl-incident-manager" },
                        ["Message"] = "Isolate {{ InstanceId }} for {{ IncidentId }}?",
                        ["MinRequiredApprovals"] = 1,
                    },
                },
                new Dictionary<string, object>
                {
                    ["name"] = "SwapSecurityGroups",
                    ["action"] = "aws:executeAwsApi",
                    ["inputs"] = new Dictionary<string, object>
                    {
                        ["Service"] = "ec2", ["Api"] = "ModifyInstanceAttribute",
                        ["InstanceId"] = "{{ InstanceId }}", ["Groups"] = new[] { "{{ IsolationSecurityGroupId }}" },
                    },
                },
                new Dictionary<string, object>
                {
                    ["name"] = "EnableTerminationProtection",
                    ["action"] = "aws:executeAwsApi",
                    ["inputs"] = new Dictionary<string, object>
                    {
                        ["Service"] = "ec2", ["Api"] = "ModifyInstanceAttribute",
                        ["InstanceId"] = "{{ InstanceId }}",
                        ["DisableApiTermination"] = new Dictionary<string, object> { ["Value"] = true },
                    },
                },
                new Dictionary<string, object>
                {
                    ["name"] = "SnapshotVolumes",
                    ["action"] = "aws:executeAwsApi",
                    ["inputs"] = new Dictionary<string, object>
                    {
                        ["Service"] = "ec2", ["Api"] = "CreateSnapshots",
                        ["InstanceSpecification"] = new Dictionary<string, object> { ["InstanceId"] = "{{ InstanceId }}" },
                        ["Description"] = "{{ IncidentId }} evidence {{ InstanceId }}",
                        ["CopyTagsFromSource"] = "volume",
                    },
                    ["outputs"] = new object[]
                    {
                        new Dictionary<string, object> { ["Name"] = "SnapshotIds", ["Selector"] = "$.Snapshots..SnapshotId", ["Type"] = "StringList" },
                    },
                },
                new Dictionary<string, object>
                {
                    ["name"] = "TagInstance",
                    ["action"] = "aws:createTags",
                    ["inputs"] = new Dictionary<string, object>
                    {
                        ["ResourceType"] = "EC2",
                        ["ResourceIds"] = new[] { "{{ InstanceId }}" },
                        ["Tags"] = new object[]
                        {
                            new Dictionary<string, string> { ["Key"] = "IncidentId", ["Value"] = "{{ IncidentId }}" },
                            new Dictionary<string, string> { ["Key"] = "Quarantine", ["Value"] = "true" },
                        },
                    },
                },
            },
        };

        _ = new CfnDocument(this, "IsolateInstanceRunbook", new CfnDocumentProps
        {
            Name = "HL-IR03-IsolateEC2Instance",
            DocumentType = "Automation",
            DocumentFormat = "JSON",
            Content = content,
        });
    }
}

The automation role (AutomationAssumeRole) needs ec2:ModifyInstanceAttribute, ec2:CreateSnapshots, ec2:CreateSnapshot, ec2:CreateTags and ec2:DescribeInstances — scoped with conditions where possible ⚠ VERIFY the exact permissions CreateSnapshots needs in your account setup.

Start the runbook with:

aws ssm start-automation-execution --document-name HL-IR03-IsolateEC2Instance --profile learn-sandbox \
  --parameters InstanceId=i-0123456789abcdef0,IsolationSecurityGroupId=sg-0aaaabbbbccccdddd,IncidentId=INC-2026-0042,AutomationAssumeRole=arn:aws:iam::111122223333:role/hl-ir-automation

Complete code

The containment tool is what an on-call engineer runs under pressure. Its design rules:

  • Dry-run by default. It prints the exact plan; nothing changes until you add --execute.
  • Plan, then act. Each command builds an ordered list of steps; the executor runs them one by one and stops on the first failure.
  • Evidence by design. Every step is written to a JSON-lines journal with UTC timestamps, the operator's identity and the outcome, and the journal's SHA-256 is printed at the end for chain of custody.
  • No destructive actions. It deactivates, denies, isolates and snapshots. It never deletes keys or terminates instances.
Containment/Containment.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <RootNamespace>Harbourline.Security.Containment</RootNamespace>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="AWSSDK.IdentityManagement" Version="4.1.1.2" />
    <PackageReference Include="AWSSDK.EC2" Version="4.1.4" />
    <PackageReference Include="AWSSDK.SecurityToken" Version="4.1.0.2" />
  </ItemGroup>
</Project>
Containment/Plans.cs
using Amazon.EC2;
using Amazon.EC2.Model;
using Amazon.IdentityManagement;
using Amazon.IdentityManagement.Model;

namespace Harbourline.Security.Containment;

public sealed record Step(string Description, Func<CancellationToken, Task<string>> Action);

public static class Plans
{
    public const string DenyAllPolicyArn = "arn:aws:iam::aws:policy/AWSDenyAll";

    public static string RevokeOlderSessionsPolicy(DateTimeOffset cutoff) => $$"""
        {"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":["*"],"Resource":["*"],
          "Condition":{"DateLessThan":{"aws:TokenIssueTime":"{{cutoff.UtcDateTime:yyyy-MM-ddTHH:mm:ssZ}}"} } }]}
        """;

    // IR-01: leaked IAM user access key.
    public static IReadOnlyList<Step> LeakedKey(IAmazonIdentityManagementService iam, string userName, string accessKeyId, string incidentId) =>
    [
        new($"Deactivate access key {accessKeyId} of user {userName}", async ct =>
        {
            await iam.UpdateAccessKeyAsync(new UpdateAccessKeyRequest
            {
                UserName = userName, AccessKeyId = accessKeyId, Status = Amazon.IdentityManagement.StatusType.Inactive,
            }, ct);
            return "key inactive";
        }),
        new($"Attach {DenyAllPolicyArn} to user {userName}", async ct =>
        {
            await iam.AttachUserPolicyAsync(new AttachUserPolicyRequest { UserName = userName, PolicyArn = DenyAllPolicyArn }, ct);
            return "deny-all attached";
        }),
        new($"Tag user {userName} with IncidentId={incidentId}", async ct =>
        {
            await iam.TagUserAsync(new TagUserRequest
            {
                UserName = userName,
                Tags = [new Amazon.IdentityManagement.Model.Tag { Key = "IncidentId", Value = incidentId }],
            }, ct);
            return "tagged";
        }),
        new($"List remaining access keys of {userName} (attacker may have created more)", async ct =>
        {
            var keys = await iam.ListAccessKeysAsync(new ListAccessKeysRequest { UserName = userName }, ct);
            var active = (keys.AccessKeyMetadata ?? []).Where(k => k.Status == Amazon.IdentityManagement.StatusType.Active).Select(k => k.AccessKeyId).ToList();
            return active.Count == 0 ? "no other active keys" : $"STILL ACTIVE: {string.Join(", ", active)} — deactivate them";
        }),
    ];

    // Revoke all sessions of a role issued before 'cutoff'.
    public static IReadOnlyList<Step> RevokeRoleSessions(IAmazonIdentityManagementService iam, string roleName, DateTimeOffset cutoff) =>
    [
        new($"Put inline policy AWSRevokeOlderSessions on role {roleName} (deny tokens issued before {cutoff:u})", async ct =>
        {
            await iam.PutRolePolicyAsync(new PutRolePolicyRequest
            {
                RoleName = roleName,
                PolicyName = "AWSRevokeOlderSessions",
                PolicyDocument = RevokeOlderSessionsPolicy(cutoff),
            }, ct);
            return "older sessions revoked";
        }),
    ];

    // IR-03: isolate an EC2 instance and preserve its disks. Never stops or terminates.
    public static IReadOnlyList<Step> IsolateInstance(IAmazonEC2 ec2, string instanceId, string vpcId, string incidentId)
    {
        string? sgId = null;
        return
        [
            new($"Create isolation security group in {vpcId}", async ct =>
            {
                var sg = await ec2.CreateSecurityGroupAsync(new CreateSecurityGroupRequest
                {
                    GroupName = $"hl-ir-isolation-{instanceId}",
                    Description = $"IR isolation {incidentId}",
                    VpcId = vpcId,
                }, ct);
                sgId = sg.GroupId;
                return sgId;
            }),
            new("Remove the default allow-all egress rule from the isolation group", async ct =>
            {
                await ec2.RevokeSecurityGroupEgressAsync(new RevokeSecurityGroupEgressRequest
                {
                    GroupId = sgId,
                    IpPermissions = [new IpPermission { IpProtocol = "-1", Ipv4Ranges = [new IpRange { CidrIp = "0.0.0.0/0" }] }],
                }, ct);
                return "no inbound, no outbound";
            }),
            new($"Replace all security groups on {instanceId} with the isolation group", async ct =>
            {
                await ec2.ModifyInstanceAttributeAsync(new ModifyInstanceAttributeRequest { InstanceId = instanceId, Groups = [sgId!] }, ct);
                return "isolated (check NACL for tracked connections)";
            }),
            new($"Enable termination protection on {instanceId}", async ct =>
            {
                await ec2.ModifyInstanceAttributeAsync(new ModifyInstanceAttributeRequest { InstanceId = instanceId, DisableApiTermination = true }, ct);
                return "termination protected";
            }),
            new($"Snapshot all volumes of {instanceId}", async ct =>
            {
                var snaps = await ec2.CreateSnapshotsAsync(new CreateSnapshotsRequest
                {
                    InstanceSpecification = new InstanceSpecification { InstanceId = instanceId, ExcludeBootVolume = false },
                    Description = $"{incidentId} evidence {instanceId}",
                    CopyTagsFromSource = CopyTagsFromSource.Volume,
                    TagSpecifications =
                    [
                        new TagSpecification
                        {
                            ResourceType = ResourceType.Snapshot,
                            Tags = [new Amazon.EC2.Model.Tag("IncidentId", incidentId), new Amazon.EC2.Model.Tag("Evidence", "true")],
                        },
                    ],
                }, ct);
                return string.Join(", ", (snaps.Snapshots ?? []).Select(s => s.SnapshotId));
            }),
            new($"Tag {instanceId} as quarantined", async ct =>
            {
                await ec2.CreateTagsAsync(new CreateTagsRequest
                {
                    Resources = [instanceId],
                    Tags = [new Amazon.EC2.Model.Tag("IncidentId", incidentId), new Amazon.EC2.Model.Tag("Quarantine", "true")],
                }, ct);
                return "tagged";
            }),
        ];
    }
}
Containment/Executor.cs
using System.Security.Cryptography;
using System.Text.Json;

namespace Harbourline.Security.Containment;

public sealed class Executor(TextWriter output, string journalPath, string operatorArn, TimeProvider clock)
{
    public async Task<bool> RunAsync(string incidentId, IReadOnlyList<Step> plan, bool execute, CancellationToken ct = default)
    {
        output.WriteLine(execute ? "EXECUTING plan:" : "DRY RUN — nothing will change. Plan:");
        for (var i = 0; i < plan.Count; i++) output.WriteLine($"  {i + 1}. {plan[i].Description}");
        if (!execute) return true;

        await using var journal = new StreamWriter(journalPath, append: true);
        foreach (var step in plan)
        {
            var started = clock.GetUtcNow();
            string outcome; bool ok;
            try { outcome = await step.Action(ct); ok = true; }
            catch (Exception ex) { outcome = $"{ex.GetType().Name}: {ex.Message}"; ok = false; }

            var entry = JsonSerializer.Serialize(new
            {
                incidentId, at = started, finished = clock.GetUtcNow(), operatorArn, step = step.Description, ok, outcome,
            });
            await journal.WriteLineAsync(entry);
            output.WriteLine($"  [{(ok ? "OK" : "FAILED")}] {step.Description} → {outcome}");
            if (!ok) { output.WriteLine("Stopping: fix the failure, then re-run the remaining steps manually."); break; }
        }

        await journal.FlushAsync(ct);
        journal.Close();
        var hash = Convert.ToHexStringLower(SHA256.HashData(await File.ReadAllBytesAsync(journalPath, ct)));
        output.WriteLine($"Journal {journalPath} SHA-256 {hash} — record this in the incident ticket.");
        return true;
    }
}
Containment/Program.cs
using Amazon.EC2;
using Amazon.IdentityManagement;
using Amazon.SecurityToken;
using Amazon.SecurityToken.Model;
using Harbourline.Security.Containment;

// Usage:
//   containment leaked-key <incident-id> <user-name> <access-key-id> [--execute]
//   containment revoke-sessions <incident-id> <role-name> [--execute]
//   containment isolate <incident-id> <instance-id> <vpc-id> [--execute]
if (args.Length < 3) { Console.Error.WriteLine("See usage in Program.cs"); return 2; }

var execute = args.Contains("--execute");
var a = args.Where(x => x != "--execute").ToArray();
var incidentId = a[1];
if (!System.Text.RegularExpressions.Regex.IsMatch(incidentId, "^INC-[0-9]{4}-[0-9]{4}$"))
{
    Console.Error.WriteLine("Incident ID must look like INC-2026-0042");
    return 2;
}

var who = await new AmazonSecurityTokenServiceClient().GetCallerIdentityAsync(new GetCallerIdentityRequest());
Console.WriteLine($"Operator: {who.Arn} in account {who.Account}");

IReadOnlyList<Step> plan = a[0] switch
{
    "leaked-key" when a.Length == 4 => Plans.LeakedKey(new AmazonIdentityManagementServiceClient(), a[2], a[3], incidentId),
    "revoke-sessions" when a.Length == 3 => Plans.RevokeRoleSessions(new AmazonIdentityManagementServiceClient(), a[2], DateTimeOffset.UtcNow),
    "isolate" when a.Length == 4 => Plans.IsolateInstance(new AmazonEC2Client(), a[2], a[3], incidentId),
    _ => [],
};
if (plan.Count == 0) { Console.Error.WriteLine("Unknown command or wrong arguments"); return 2; }

var executor = new Executor(Console.Out, $"{incidentId}-journal.jsonl", who.Arn, TimeProvider.System);
return await executor.RunAsync(incidentId, plan, execute) ? 0 : 1;

A test that proves dry-run never touches AWS, and that the revoke policy is well-formed:

Containment.Tests/ContainmentTests.cs
using System.Text.Json;
using Harbourline.Security.Containment;
using Xunit;

public sealed class ContainmentTests
{
    [Fact]
    public async Task Dry_run_executes_nothing()
    {
        var called = false;
        IReadOnlyList<Step> plan = [new("Dangerous step", _ => { called = true; return Task.FromResult("done"); })];
        var output = new StringWriter();
        var executor = new Executor(output, Path.GetTempFileName(), "arn:aws:sts::111122223333:assumed-role/ir/test", TimeProvider.System);

        await executor.RunAsync("INC-2026-0001", plan, execute: false, TestContext.Current.CancellationToken);

        Assert.False(called);
        Assert.Contains("DRY RUN", output.ToString());
    }

    [Fact]
    public async Task Execution_stops_at_first_failure()
    {
        var secondRan = false;
        IReadOnlyList<Step> plan =
        [
            new("fails", _ => throw new InvalidOperationException("boom")),
            new("should not run", _ => { secondRan = true; return Task.FromResult("x"); }),
        ];
        var executor = new Executor(new StringWriter(), Path.GetTempFileName(), "arn:test", TimeProvider.System);
        await executor.RunAsync("INC-2026-0001", plan, execute: true, TestContext.Current.CancellationToken);
        Assert.False(secondRan);
    }

    [Fact]
    public void Revoke_policy_denies_tokens_before_cutoff()
    {
        var json = Plans.RevokeOlderSessionsPolicy(new DateTimeOffset(2026, 11, 18, 10, 30, 0, TimeSpan.Zero));
        using var doc = JsonDocument.Parse(json);
        var stmt = doc.RootElement.GetProperty("Statement")[0];
        Assert.Equal("Deny", stmt.GetProperty("Effect").GetString());
        Assert.Equal("2026-11-18T10:30:00Z",
            stmt.GetProperty("Condition").GetProperty("DateLessThan").GetProperty("aws:TokenIssueTime").GetString());
    }
}

Code explained line by line

  1. Step(Description, Action) — a plan is data: a readable description plus a delegate. That's what makes dry-run trivial and honest: you print exactly what would run.
  2. RevokeOlderSessionsPolicy produces the same policy the console's "Revoke sessions" button creates: deny everything when aws:TokenIssueTime is earlier than the cutoff. Sessions issued after the cutoff still work — which is why you must also cut off whatever lets the attacker get new sessions.
  3. The raw string $$"""…""" lets the JSON's single braces stay literal while {{…}} interpolates the timestamp in ISO 8601 UTC format.
  4. LeakedKey — deactivate (not delete) the key, then attach the AWS managed AWSDenyAll policy so a second key or console password is useless too, then tag the user, then check for other active keys — attackers often create a backup key immediately.
  5. keys.AccessKeyMetadata ?? [] — SDK v4 collections may be null; always guard.
  6. IsolateInstance keeps the security-group ID in a captured variable (sgId) so later steps use the group created by the first step.
  7. A new security group comes with an allow-all egress rule; the second step removes it so the group has no rules at all.
  8. ModifyInstanceAttribute with Groups replaces all security groups on the primary network interface. Instances with several ENIs need each interface changed ⚠ VERIFY for multi-ENI instances.
  9. DisableApiTermination = true stops someone (or an Auto Scaling scale-in) terminating the evidence via the API.
  10. CreateSnapshots with an InstanceSpecification snapshots every attached EBS volume at the same point in time (crash-consistent) and tags them as evidence.
  11. Executor prints the plan first, and in dry-run returns immediately — the test proves no action runs.
  12. Each executed step writes one JSON line with start and finish times, operator ARN, description, success flag and outcome: an audit-quality action log.
  13. Execution stops at the first failure. In an incident, blindly continuing after a failure (for example snapshotting after isolation failed) creates confusion about the system's state.
  14. The journal's SHA-256 is printed so it can be recorded in the ticket; anyone can later prove the journal wasn't edited.
  15. Program.cs validates the incident ID format and records the operator's identity from STS — you always know who ran what.

Security considerations

⚠ Learning shortcut — not for production

The lab runs containment from your own sandbox role, keeps evidence in the same account with governance-mode Object Lock for one day, and uses a test instance and test user. In production: responders use a pre-provisioned, MFA-protected IR role assumed from the security tooling account; evidence goes to a separate forensic account with compliance-mode Object Lock and a dedicated KMS key; runbooks have approval steps; and every action is journalled in the incident record.

  • Don't tip off an insider. For suspected insider cases, coordinate with HR and legal before visible containment actions; quietly preserve evidence first where the risk allows.
  • Don't terminate, don't reboot. Both destroy volatile evidence. Isolate, snapshot, and only then decide.
  • Decide deliberately about KMS keys. Disabling a customer managed key stops the attacker using it — and stops your own applications too, which may be a self-inflicted outage. Options in order of disruption: remove the attacker's access in the key policy and retire grants; revoke the compromised principal; disable the key; schedule deletion (never during an investigation — it can make evidence permanently unreadable). Rotation creates new key material but doesn't help with data keys the attacker already decrypted.
  • Protect the IR role. It's powerful; alert on every use; restrict who can assume it; review its use after each incident.
  • Evidence handling is personal-data processing. Forensic images contain customer data. Restrict access, log it, and define retention with legal.
  • Communicate on a separate channel. If the attacker may have access to email or chat, use the out-of-band channel defined in your IR plan.
  • Notification is not an engineering decision. Give the DPO, legal and compliance facts, timelines and uncertainties early; they decide what is reportable to the ICO, the FCA/PRA, card schemes or customers.

Banking use case

INC-2026-0042 (fictional), Saturday 14:05. GuardDuty raises UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS (High) for the role hl-batch-recon-role used by EC2 instances in the reconciliation account. The triage Lambda pages on-call.

TimeActionWho
14:09On-call acknowledges, opens INC-2026-0042, declares severity 2, pages incident managerOn-call engineer
14:15Detective/Athena: the role's credentials used from an external IP; ListBuckets, then GetObject on hl-prod-recon-exports (412 objects)Technical lead
14:18Containment tool: revoke-sessions for the role (dry-run reviewed, then --execute)Technical lead, approved by IM
14:22isolate for the two instances; snapshots taken; ASG detached with replacementTechnical lead
14:40Root cause hypothesis: SSRF vulnerability in an internal reporting app on the instance exposing IMDSv1 credentialsTechnical lead
15:10DPO, legal and compliance briefed: 412 export files, each containing transaction summaries for business customers; personal data present for sole tradersIncident manager
16:00Eradication: IMDSv2 enforced (HttpTokens=required) via launch template; app patched; new instances deployedPlatform team
SundayForensic analysis of snapshots in the forensic account confirms no persistenceForensics
MondayLegal/DPO decide the breach is reportable to the ICO within 72 hours of awareness and prepare the notification; compliance assesses FCA reporting obligationsLegal, DPO, compliance
+2 weeksPost-incident review: SCP requiring IMDSv2 for all new instances; Inspector P1 for SSRF-class findings; game day scheduledSecurity

Note what made the timeline possible: data events on the export bucket, GuardDuty everywhere, a pre-provisioned IR role, and a containment tool that had been tested.

Common mistakes

  • Terminating the compromised instance immediately → evidence lost → isolate and snapshot first.
  • Deleting the leaked key → can't correlate it later and loses metadata → deactivate; delete after the investigation.
  • Revoking sessions but leaving the source → attacker gets fresh credentials → fix the path (stolen key, IMDSv1, CI secret).
  • Only checking one Region → attacker resources elsewhere keep running → check every enabled Region; use an SCP to block unused Regions.
  • No logging until the incident → can't scope data access → enable data events and flow logs before you need them.
  • Engineers notifying regulators or customers on their own → inconsistent or premature statements → route through the incident manager, legal and DPO.
  • Responders without access at 3 a.m. → delays while someone finds an admin → pre-provision and test the IR role.

Troubleshooting

SymptomLikely causeFix
Attacker activity continues after "Revoke sessions"They obtain new sessions via another path (user key, instance role on a still-running instance)Find the source in CloudTrail (sessionIssuer, sourceIPAddress); deactivate keys; isolate instances
Isolated instance still has an established SSH sessionConnection tracking keeps existing tracked flowsAdd a NACL deny for the subnet/host or follow AWS's documented isolation sequence
UnauthorizedOperation on CreateSnapshotsIR role missing ec2:CreateSnapshots/ec2:CreateTags, or KMS key policy blocks encrypted volume snapshotsAdd permissions; ensure the role can use the volume's KMS key
Snapshot can't be shared with the forensic accountEncrypted with an AWS managed key (can't be shared)Copy the snapshot re-encrypted with a customer managed key that the forensic account can use, then share
AccessDenied for PutRolePolicy on a roleSCP or permission boundary blocks IAM changes, or the role is protected (e.g. Control Tower roles)Use the authorised IR role; follow the documented exception for protected roles
GuardDuty finding keeps reappearingRoot cause not fixed, or attacker persistenceRe-scope; check for new users/keys/roles/Lambda functions/instances
Can't tell which objects were downloadedNo S3 data events or server access logsEnable them now for the future; use what's available (CloudFront logs, app logs) and record the uncertainty for the DPO

Cost and cleanup

Item (checked 2026-11-18)PriceLab exposure
IAM, revoke sessions, security groups, tagsFreeFree
Test EC2 instance (t4g.nano)About USD 0.0042 per hour ⚠ VERIFY1 hour ≈ under USD 0.01
EBS snapshotsAbout USD 0.05 per GB-month (incremental)8 GB for a day ≈ USD 0.01
SSM AutomationFree tier covers lab use ⚠ VERIFY current Automation step pricingFree
S3 evidence bucketCentsCents
AWS Security Incident ResponseFirst 10,000 findings per month free; then tiered per findingNot enabled in the lab

Free alternative: run the containment tool in dry-run against a fictional user, role and instance ID, and run the unit tests. Do the tabletop exercise in the Exercises section.

Lab practice (credits): create a test IAM user with one access key (never use it for anything), a t4g.nano instance in a sandbox VPC, then run each containment command with --execute.

PROFILE=learn-sandbox
REGION=us-east-1

# Remove the test user (detach policy, delete key, delete user)
aws iam detach-user-policy --user-name hl-m05-ir-test --policy-arn arn:aws:iam::aws:policy/AWSDenyAll --profile $PROFILE
aws iam delete-access-key --user-name hl-m05-ir-test --access-key-id <key-id> --profile $PROFILE
aws iam delete-user --user-name hl-m05-ir-test --profile $PROFILE

# Remove the revoke-sessions inline policy from the test role
aws iam delete-role-policy --role-name hl-m05-ir-test-role --policy-name AWSRevokeOlderSessions --profile $PROFILE

# Instance: remove termination protection, terminate, delete isolation SG and snapshots
aws ec2 modify-instance-attribute --instance-id $IID --no-disable-api-termination --profile $PROFILE --region $REGION
aws ec2 terminate-instances --instance-ids $IID --profile $PROFILE --region $REGION
aws ec2 wait instance-terminated --instance-ids $IID --profile $PROFILE --region $REGION
aws ec2 delete-security-group --group-id $SG --profile $PROFILE --region $REGION
for s in $(aws ec2 describe-snapshots --owner-ids self --filters Name=tag:IncidentId,Values=INC-2026-0042 \
           --query "Snapshots[].SnapshotId" --output text --profile $PROFILE --region $REGION); do
  aws ec2 delete-snapshot --snapshot-id $s --profile $PROFILE --region $REGION
done

# IaC: cdk destroy / terraform destroy (evidence bucket objects are deletable after the 1-day governance retention)

# Confirm
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --query "ResourceTagMappingList[].ResourceARN" --profile $PROFILE --region $REGION

Exercises

  • 🟢 Guided: Build and run the containment tool in dry-run for all three commands. Then, in your sandbox, create a test user with a key and a t4g.nano instance, and run leaked-key and isolate with --execute. Confirm the key is inactive, the instance has only the isolation group, and snapshots exist. Record the journal hash. Clean up.
  • 🟡 Challenge: Add a block-bucket-principal command that adds a bucket-policy statement denying all S3 actions on a given bucket to a given principal ARN, preserving existing statements. Make it dry-run by default and add a test for the policy merge.
  • 🔴 Live problem (tabletop): Friday 17:45. Macie reports that a bucket in the data-platform account containing a 2 GB CSV of customer names, dates of birth and IBANs became publicly readable 6 hours ago after a Terraform change. S3 server access logs show 37 GET requests for the file from 5 IPs not owned by Harbourline. Run the incident: what do you do in the first hour, what facts do you give the DPO, and what changes afterwards?

Assignment

Create Harbourline's IR pack in your portfolio: severity matrix and roles; runbooks IR-01 (leaked key), IR-03 (compromised EC2) and IR-04 (suspicious data download) as Markdown checklists with exact commands; the SSM Automation document as IaC; the containment tool with tests; and a one-page "facts for the DPO" template (what happened, when, which data and how many people, containment status, uncertainties).

Solutions

Knowledge check

Interview questions

  • "Walk me through your first 30 minutes after a leaked access key is reported." — Open the incident; identify owner via get-access-key-last-used; deactivate the key; deny-all the user; check for other keys/console access; query CloudTrail by access key ID in all Regions; revoke sessions for any assumed roles; look for persistence; inform the incident manager and, if data may be involved, the DPO.
  • "Why not just terminate a compromised instance?" — It destroys memory and disk evidence you need to understand scope and root cause. Isolate, protect from termination, snapshot, then rebuild clean and terminate later under control.
  • "How do you prepare for incidents in AWS?" — Logging in place (CloudTrail data events, flow logs), pre-provisioned IR roles, forensic account, runbooks automated with approvals, out-of-band comms, and regular game days.
  • "What's the risk in disabling a KMS key during an incident?" — It breaks every legitimate workload using the key. Prefer targeted key-policy and grant changes; disable only when the business accepts the outage; never schedule deletion during an investigation.

Certification practice

Completion checklist

  • I can explain the IR lifecycle and the roles in a bank's incident team
  • I can execute the leaked-key and suspicious-download runbooks step by step
  • I revoked role sessions and isolated a test instance without terminating it
  • I preserved evidence with hashes and know why it belongs in a separate forensic account
  • I know which services are closed (Incident Manager) and what to use instead
  • I cleaned up the test user, role policy, instance, security group and snapshots

Key takeaways

  • Incident response is prepared before the incident: logging, access, runbooks, forensic account and practice.
  • In AWS, credentials are the usual target: deactivate keys, deny-all the user, revoke role sessions — and close the path that issued them.
  • Contain before you clean and preserve before you change: isolate, protect from termination, snapshot; don't terminate or reboot.
  • Automate repeatable steps (SSM Automation with approvals, a dry-run-first tool) and journal every action with timestamps and hashes.
  • Notification to the ICO, regulators and customers is decided by DPO, legal and compliance; engineers supply fast, accurate facts and honest uncertainty.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page