Skip to Content
Cloud Engineering Academy
12. High Availability & Disaster Recovery

Chaos engineering with AWS Fault Injection Service

Plan and run a safe first chaos experiment: write a steady-state hypothesis, build an AWS Fault Injection Service experiment template that stops ECS tasks with a CloudWatch alarm as the stop condition, gate it with a pre-flight check in C#, and rehearse the same idea locally for free.

advanced⏱ 2 h 30 min🟡 Low costSOA-C03DOP-C02SAP-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.FIS) · AWS CLI v2 · AWS CDK v2 (C#) · Terraform AWS provider 6.x · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain chaos engineering as controlled experiments on a steady state, not random breakage.
  • Write a hypothesis, a blast radius and a stop condition for an experiment.
  • Create an AWS Fault Injection Service (FIS) experiment template with targets, actions, a stop-condition alarm and a least-privilege experiment role.
  • Gate an experiment with a C# pre-flight check and observe the steady state while it runs.
  • Run the same pattern locally and free with a fault-injecting HTTP handler.

Prerequisites

  • Module 11: Monitoring, because alarms are both your stop condition and your evidence.
  • Health checks and Deploy to Fargate, for the service you will break.
  • The auto scaling, load balancing and resilience-pattern lessons earlier in this module (see the module overview).
  • Tools: AWS CLI v2 (learn-sandbox), .NET 10 SDK. A running ECS service is needed only for the real AWS experiment.

Why this matters in banking

Harbourline's design says: "If one payment API task dies, the load balancer removes it and the service replaces it, with no failed payments." Everyone believes that. Nobody has watched it happen under load. Then a task is killed during a release and some payments fail because the client library has no retry, the load balancer's deregistration delay is longer than the client timeout, and the health check is too slow to notice.

Chaos engineering is the discipline of finding out on a Tuesday morning, on purpose, with a safety net, rather than on a Saturday night by accident. For a regulated firm there is a second benefit: an experiment record (hypothesis, what was injected, what happened, what was fixed) is evidence that resilience was tested, not assumed.

Reminder: FIS performs real actions on real resources. Start in a non-production environment, with small blast radius, and with an agreed stop condition. Production experiments need change approval and risk sign-off.

The idea in plain language

A chaos experiment follows the scientific method:

  1. Define steady state: a measurable signal that says the system is healthy (for example, payment success rate at least 99.9% and p99 latency under 800 ms).
  2. Form a hypothesis: "If we stop one of three API tasks, the steady state stays within limits."
  3. Inject a fault, small and controlled: stop one task.
  4. Observe: did the steady state hold?
  5. Learn and fix: if it did not, you found a weakness cheaply. If it did, you have evidence, and you can increase the blast radius next time.

Analogy: A fire drill. You don't set the building on fire. You pull the alarm at a planned time, with the fire marshal watching and a clear rule ("if someone is hurt, stop"), and you time how long evacuation takes. Afterwards you fix the blocked exit you discovered.

What problem FIS solves

AWS Fault Injection Service is a managed service for running fault injection experiments on AWS workloads. Its building blocks:

  • Experiment template: the blueprint, with actions, targets and stop conditions.
  • Action: what to do to a resource, such as stop ECS tasks, stop or terminate EC2 instances, reboot or fail over RDS, inject API errors, throttle or disrupt network connectivity. Actions have a duration or run until stopped, and can run in sequence or in parallel.
  • Target: which resources the action applies to, selected by ARN, tags and filters, with a selection mode (all, a count, or a percentage).
  • Stop condition: a CloudWatch alarm. If it goes into the ALARM state while the experiment runs, FIS stops the experiment.
  • Experiment role: an IAM role FIS assumes to perform the actions. Scope it to the minimum.
  • Scenarios: pre-built, multi-action experiment recipes in the FIS scenario library, for example for loss of an Availability Zone or cross-Region connectivity ⚠ VERIFY current scenario names.

DynamoDB global tables and several other services also publish FIS-compatible actions, such as pausing replication to simulate a Region isolation; see the global tables documentation for the current action list.

Honest limits:

  • Not every failure is injectable. FIS covers a defined list of actions; check the actions reference for your service ⚠ VERIFY.
  • It cannot tell you what to measure. Your alarms and dashboards define the steady state; poor observability means a poor experiment.
  • It costs money: billed per minute that an action runs, per target account ⚠ VERIFY the current rate on the FIS pricing page. Experiments are short, so the cost is small, but the resources you break may also bill while recovering.
  • Single-account by default: targets must be in the same account as the experiment unless you use multi-account experiments.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Chaos Studio experimentsFIS experiment templatesBoth define steps, targets and faults; FIS stop conditions are CloudWatch alarms
Chaos Studio target onboardingFIS targets by ARN or tag; some actions need an agent via Systems ManagerService-level actions (stop task, failover RDS) need no agent
Chaos Studio managed identityFIS experiment roleA normal IAM role with a trust policy for fis.amazonaws.com
Polly chaos strategies (Simmy)FIS for infrastructure faults, a fault-injecting handler for code-level faultsUse both: FIS for the platform, code-level chaos for dependency behaviour
Azure Monitor alerts as guardrailCloudWatch alarm as stop conditionThe alarm is attached to the template, not to a separate rule

Architecture

Security-boundary view: the experiment role can stop ECS tasks in one cluster and nothing else. The people who may start experiments are different from those who can edit templates (separation of duties), and every start is logged by CloudTrail. In a bank, templates live in version control and run through change approval; ad-hoc console experiments are for the sandbox only.

Step-by-step implementation

Define the steady state and the hypothesis

Write it down before touching AWS. Example: "Payment success rate stays at least 99.9% and p99 latency below 800 ms while one of three Accounts API tasks is stopped, because the ALB removes the unhealthy target within 30 seconds and clients retry once."

Make the steady state observable

Create CloudWatch alarms: one on the ALB's HTTPCode_Target_5XX_Count or a custom success-rate metric, and one on TargetResponseTime p99. The stop-condition alarm should fire a little before the business impact would breach the tolerance.

Choose a small blast radius

Target one task (COUNT(1)) in one service, in a sandbox, during working hours with the team present. Never start with "all".

Create the experiment role

A role trusted by fis.amazonaws.com with ecs:StopTask, ecs:ListTasks, ecs:DescribeTasks and tag:GetResources, restricted by resource and condition to the lab cluster ⚠ VERIFY the exact action list in the FIS actions reference.

Create the experiment template

Define the target (tasks of the lab service), the action aws:ecs:stop-task, and the stop condition. Review it like code.

Run the pre-flight gate, start the load, start the experiment

The gate checks that the alarm is OK, the target is tagged as a lab resource, the blast radius is within limits and a human is on call. Start a steady load, then start the experiment and watch.

Record the result and fix

Compare observations with the hypothesis. Write down what happened, what you changed (retry, health check interval, deregistration delay, minimum task count) and the date you will rerun it.

AWS Console instructions

  • Create a template: Console → AWS Fault Injection Service → Experiment templates → Create experiment template. Add an action (for example aws:ecs:stop-task), define the target (resource type, tag or ARN filter, selection mode such as Count with 1), add a stop condition (CloudWatch alarm), and choose the IAM role.
  • Start: open the template → Start experiment → add tags (Project=cea-lab) → confirm by typing start.
  • Watch: Experiments → select the run → Timeline shows each action's state. In CloudWatch open the dashboard with success rate, p99, and ECS running task count.
  • Stop early: Experiments → select the run → Stop experiment.
  • Check the guardrail: CloudWatch → Alarms → confirm the stop-condition alarm is OK before starting.

AWS CLI

# Create the experiment template from a JSON file (see below).
aws fis create-experiment-template --cli-input-json file://template.json \
  --region us-east-1 --profile learn-sandbox

# List templates.
aws fis list-experiment-templates \
  --query "experimentTemplates[].{Id:id,Description:description}" \
  --region us-east-1 --profile learn-sandbox

# Check the stop-condition alarm state (must be OK).
aws cloudwatch describe-alarms --alarm-names hl-m12-api-success-rate-low \
  --query "MetricAlarms[].StateValue" --region us-east-1 --profile learn-sandbox

# Start the experiment, tagged for cleanup and audit.
aws fis start-experiment --experiment-template-id EXT1a2b3c4d5e6f7 \
  --tags Project=cea-lab,Lab=m12-fis,Environment=learn \
  --region us-east-1 --profile learn-sandbox

# Poll the state: initiating, running, completed, stopped or failed.
aws fis get-experiment --id EXP1a2b3c4d5e6f7 \
  --query "experiment.{State:state.status,Reason:state.reason}" \
  --region us-east-1 --profile learn-sandbox

# Stop an experiment immediately (the emergency brake).
aws fis stop-experiment --id EXP1a2b3c4d5e6f7 --region us-east-1 --profile learn-sandbox

# See the running task count recover after the experiment.
aws ecs describe-services --cluster hl-m12-cluster --services hl-m12-accounts-api \
  --query "services[0].{Desired:desiredCount,Running:runningCount}" \
  --region us-east-1 --profile learn-sandbox

A minimal template.json that stops one task of a service, with a stop condition:

{
  "description": "hl-m12: stop 1 of 3 Accounts API tasks",
  "roleArn": "arn:aws:iam::111122223333:role/hl-m12-fis-role",
  "targets": {
    "ApiTasks": {
      "resourceType": "aws:ecs:task",
      "selectionMode": "COUNT(1)",
      "parameters": {
        "cluster": "hl-m12-cluster",
        "service": "hl-m12-accounts-api"
      }
    }
  },
  "actions": {
    "StopOneTask": {
      "actionId": "aws:ecs:stop-task",
      "description": "Stop one task",
      "targets": { "Tasks": "ApiTasks" }
    }
  },
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:us-east-1:111122223333:alarm:hl-m12-api-success-rate-low"
    }
  ],
  "tags": { "Project": "cea-lab", "Lab": "m12-fis", "Environment": "learn" }
}

The action ID, target key and target parameters follow the FIS actions reference at the time of writing, but they change as FIS adds actions, so confirm them with the reference or aws fis list-actions before relying on them ⚠ VERIFY. Template and experiment IDs and the account number are placeholders.

Infrastructure as Code

infra/cdk/FisStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.FIS;
using Amazon.CDK.AWS.IAM;
using Constructs;

namespace Harbourline.Ha.Infra;

public sealed class FisStack : Stack
{
    public FisStack(Construct scope, string id, string clusterName, string serviceName,
        string clusterArn, IAlarm stopAlarm, IStackProps props) : base(scope, id, props)
    {
        // Experiment role: FIS may stop tasks in ONE cluster, nothing else.
        var role = new Role(this, "FisRole", new RoleProps
        {
            RoleName = "hl-m12-fis-role",
            AssumedBy = new ServicePrincipal("fis.amazonaws.com")
        });
        role.AddToPolicy(new PolicyStatement(new PolicyStatementProps
        {
            Effect = Effect.ALLOW,
            Actions = new[] { "ecs:StopTask" },
            Resources = new[] { $"arn:aws:ecs:{Region}:{Account}:task/{clusterName}/*" }
        }));
        role.AddToPolicy(new PolicyStatement(new PolicyStatementProps
        {
            Effect = Effect.ALLOW,
            Actions = new[] { "ecs:ListTasks", "ecs:DescribeTasks", "tag:GetResources" },
            Resources = new[] { "*" } // list and describe actions do not support narrow resources ⚠ VERIFY
        }));

        _ = new CfnExperimentTemplate(this, "StopOneTask", new CfnExperimentTemplateProps
        {
            Description = "hl-m12: stop 1 of 3 Accounts API tasks",
            RoleArn = role.RoleArn,
            Targets = new Dictionary<string, object>
            {
                ["ApiTasks"] = new CfnExperimentTemplate.ExperimentTemplateTargetProperty
                {
                    ResourceType = "aws:ecs:task",
                    SelectionMode = "COUNT(1)",
                    Parameters = new Dictionary<string, string>
                    {
                        ["cluster"] = clusterName,
                        ["service"] = serviceName
                    }
                }
            },
            Actions = new Dictionary<string, object>
            {
                ["StopOneTask"] = new CfnExperimentTemplate.ExperimentTemplateActionProperty
                {
                    ActionId = "aws:ecs:stop-task",
                    Targets = new Dictionary<string, string> { ["Tasks"] = "ApiTasks" }
                }
            },
            StopConditions = new[]
            {
                new CfnExperimentTemplate.ExperimentTemplateStopConditionProperty
                {
                    Source = "aws:cloudwatch:alarm",
                    Value = stopAlarm.AlarmArn
                }
            },
            Tags = new Dictionary<string, string>
            {
                ["Project"] = "cea-lab", ["Lab"] = "m12-fis", ["Environment"] = "learn"
            }
        });
    }
}

The type of Targets and Actions on CfnExperimentTemplateProps is a map of property objects; if your CDK version complains, use the IResolvable overload or build the template from CfnJson/raw overrides ⚠ VERIFY. The clusterArn argument is shown for parity with the Terraform and can be used to tighten the role further.

Complete code

Three small pieces: a pre-flight gate, a steady-state probe that runs during the experiment, and a local fault-injecting handler that gives you the same learning for free.

src/Harbourline.Ha.Chaos/ExperimentGate.cs
namespace Harbourline.Ha.Chaos;

public sealed record ExperimentPlan(
    string Name,
    string Hypothesis,
    string Environment,           // "learn", "test", "prod"
    int TargetCount,
    int TotalTargets,
    bool HasStopCondition,
    string StopAlarmState,        // OK, ALARM, INSUFFICIENT_DATA
    bool HumanOnCall,
    bool ChangeApproved);

public sealed record GateResult(bool Allowed, IReadOnlyList<string> Problems);

public static class ExperimentGate
{
    /// <summary>Refuses to start an experiment that breaks the safety rules.</summary>
    public static GateResult Evaluate(ExperimentPlan p, double maxBlastRadius = 0.34)
    {
        var problems = new List<string>();

        if (string.IsNullOrWhiteSpace(p.Hypothesis))
            problems.Add("No written hypothesis.");
        if (!p.HasStopCondition)
            problems.Add("No stop condition: an experiment without a guardrail must not run.");
        if (p.StopAlarmState != "OK")
            problems.Add($"Stop-condition alarm is {p.StopAlarmState}, expected OK (system not in steady state).");
        if (p.TotalTargets <= 0 || (double)p.TargetCount / p.TotalTargets > maxBlastRadius)
            problems.Add($"Blast radius too large: {p.TargetCount} of {p.TotalTargets} targets (limit {maxBlastRadius:P0}).");
        if (!p.HumanOnCall)
            problems.Add("Nobody confirmed they are watching.");
        if (p.Environment == "prod" && !p.ChangeApproved)
            problems.Add("Production experiments need an approved change record.");

        return new GateResult(problems.Count == 0, problems);
    }
}
src/Harbourline.Ha.Chaos/SteadyStateProbe.cs
using System.Diagnostics;

namespace Harbourline.Ha.Chaos;

public sealed record ProbeResult(int Total, int Failures, double P99Ms)
{
    public double SuccessRate => Total == 0 ? 0 : 1.0 - (double)Failures / Total;
}

public static class SteadyStateProbe
{
    /// <summary>Calls the URL repeatedly for the given duration and summarises success rate and p99 latency.</summary>
    public static async Task<ProbeResult> RunAsync(HttpClient http, Uri url, TimeSpan duration,
        TimeSpan interval, CancellationToken ct)
    {
        var latencies = new List<double>();
        var failures = 0;
        var end = DateTime.UtcNow + duration;

        while (DateTime.UtcNow < end && !ct.IsCancellationRequested)
        {
            var sw = Stopwatch.StartNew();
            try
            {
                using var resp = await http.GetAsync(url, ct);
                if (!resp.IsSuccessStatusCode) failures++;
            }
            catch (Exception) when (!ct.IsCancellationRequested)
            {
                failures++;
            }
            latencies.Add(sw.Elapsed.TotalMilliseconds);
            await Task.Delay(interval, ct);
        }

        latencies.Sort();
        var p99 = latencies.Count == 0 ? 0 : latencies[(int)Math.Min(latencies.Count - 1, Math.Ceiling(latencies.Count * 0.99) - 1)];
        return new ProbeResult(latencies.Count, failures, p99);
    }

    public static bool HoldsSteadyState(ProbeResult r, double minSuccess = 0.999, double maxP99Ms = 800) =>
        r.Total > 0 && r.SuccessRate >= minSuccess && r.P99Ms <= maxP99Ms;
}
src/Harbourline.Ha.Chaos/FaultInjectingHandler.cs
namespace Harbourline.Ha.Chaos;

/// <summary>
/// A local chaos tool: wrap HttpClient with this handler to inject errors and latency.
/// Free, deterministic with a seed, and good for proving retry and circuit-breaker behaviour.
/// </summary>
public sealed class FaultInjectingHandler(HttpMessageHandler inner, double errorRate, TimeSpan extraLatency, int seed = 42)
    : DelegatingHandler(inner)
{
    private readonly Random _random = new(seed);
    private readonly object _gate = new();
    public bool Enabled { get; set; } = true;

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct)
    {
        if (Enabled)
        {
            bool fail;
            lock (_gate) fail = _random.NextDouble() < errorRate;

            if (extraLatency > TimeSpan.Zero) await Task.Delay(extraLatency, ct);
            if (fail) return new HttpResponseMessage(System.Net.HttpStatusCode.ServiceUnavailable)
            {
                RequestMessage = request,
                Content = new StringContent("injected fault")
            };
        }
        return await base.SendAsync(request, ct);
    }
}

A console runner ties the real AWS experiment to the gate and the probe:

src/Harbourline.Ha.Chaos/Program.cs
using Amazon;
using Amazon.CloudWatch;
using Amazon.CloudWatch.Model;
using Amazon.FIS;
using Amazon.FIS.Model;
using Harbourline.Ha.Chaos;

var templateId = args.ElementAtOrDefault(0) ?? throw new ArgumentException("Usage: <templateId> <url> <alarmName>");
var url = new Uri(args.ElementAtOrDefault(1) ?? throw new ArgumentException("missing url"));
var alarmName = args.ElementAtOrDefault(2) ?? "hl-m12-api-success-rate-low";

using var cw = new AmazonCloudWatchClient(RegionEndpoint.USEast1);
using var fis = new AmazonFISClient(RegionEndpoint.USEast1);
using var http = new HttpClient { Timeout = TimeSpan.FromSeconds(3) };

// 1. Read the guardrail alarm state.
var alarms = await cw.DescribeAlarmsAsync(new DescribeAlarmsRequest { AlarmNames = [alarmName] });
var alarmState = alarms.MetricAlarms.FirstOrDefault()?.StateValue?.Value ?? "INSUFFICIENT_DATA";

// 2. Pre-flight gate.
var plan = new ExperimentPlan(
    Name: "stop-one-api-task",
    Hypothesis: "One of three tasks can stop without breaching 99.9% success or 800 ms p99.",
    Environment: "learn",
    TargetCount: 1, TotalTargets: 3,
    HasStopCondition: true,
    StopAlarmState: alarmState,
    HumanOnCall: args.Contains("--i-am-watching"),
    ChangeApproved: false);

var gate = ExperimentGate.Evaluate(plan);
if (!gate.Allowed)
{
    foreach (var problem in gate.Problems) Console.WriteLine($"BLOCKED: {problem}");
    return 1;
}

// 3. Baseline, experiment, observe.
var baseline = await SteadyStateProbe.RunAsync(http, url, TimeSpan.FromSeconds(30), TimeSpan.FromMilliseconds(100), CancellationToken.None);
Console.WriteLine($"Baseline: {baseline.SuccessRate:P2} success, p99 {baseline.P99Ms:F0} ms");

var started = await fis.StartExperimentAsync(new StartExperimentRequest
{
    ExperimentTemplateId = templateId,
    ClientToken = Guid.NewGuid().ToString(),
    Tags = new Dictionary<string, string> { ["Project"] = "cea-lab", ["Lab"] = "m12-fis" }
});
var experimentId = started.Experiment.Id;
Console.WriteLine($"Started experiment {experimentId} at {DateTime.UtcNow:O}");

var during = await SteadyStateProbe.RunAsync(http, url, TimeSpan.FromSeconds(90), TimeSpan.FromMilliseconds(100), CancellationToken.None);
Console.WriteLine($"During:   {during.SuccessRate:P2} success, p99 {during.P99Ms:F0} ms");

// 4. Verdict against the hypothesis.
var held = SteadyStateProbe.HoldsSteadyState(during);
var state = (await fis.GetExperimentAsync(new GetExperimentRequest { Id = experimentId })).Experiment.State.Status.Value;
Console.WriteLine($"FIS state: {state}. Hypothesis {(held ? "HELD" : "REJECTED")}.");
return held ? 0 : 2;

Package references: AWSSDK.FIS, AWSSDK.CloudWatch (4.x; check NuGet for current versions). The request and response property names above follow the SDK's naming conventions ⚠ VERIFY against the installed package, since the FIS model types are generated from the service API.

Code explained line by line

  1. ExperimentPlan is the written plan as data: hypothesis, environment, blast radius, stop condition and approvals.
  2. ExperimentGate.Evaluate returns all problems, not just the first, so a reviewer sees everything that must be fixed. The default limit of about a third of targets means "no more than one AZ's worth" in a three-AZ design.
  3. The gate insists the alarm is currently OK: starting a chaos experiment while the system is already unhealthy is just a mess, and the stop condition would trip immediately.
  4. SteadyStateProbe.RunAsync samples the endpoint and collects latencies. The p99 index uses Math.Ceiling(count * 0.99) - 1, the nearest-rank method, clamped to the array.
  5. HoldsSteadyState encodes the hypothesis thresholds in code, so the verdict is not a matter of opinion.
  6. FaultInjectingHandler is a DelegatingHandler that randomly returns 503 and adds latency. It lets you prove, locally and for free, that your Microsoft.Extensions.Http.Resilience pipeline from the resilience-patterns lesson works under faults. A fixed seed makes the failure pattern repeatable in tests.
  7. In Program.cs the runner reads the alarm, runs the gate, measures a baseline, starts FIS with tags, measures during the experiment and prints a verdict. Exit code 2 means the hypothesis was rejected, which a pipeline can treat as a finding.
  8. ClientToken makes the start request idempotent, so a retry never starts two experiments.

Security considerations

⚠ Learning shortcut — not for production

The lab runs in a sandbox with broad ListTasks and DescribeTasks permissions, a single account, no change record and a human watching on a flag. A production programme adds change management, separate roles for authoring and running experiments, multi-account experiments from a central resilience account, experiment logging to a protected bucket, and notification to the incident channel.

  • The experiment role is a weapon. Scope ecs:StopTask to one cluster or tag, and require a tag condition on targets so a typo cannot reach production.
  • Who can start an experiment. Allow fis:StartExperiment for a small group, require MFA, and deny it in production accounts by SCP except for the resilience role.
  • Stop conditions are mandatory. An alarm with INSUFFICIENT_DATA does not protect you. Test the alarm itself.
  • Logging and evidence. Enable FIS experiment logging to CloudWatch Logs or S3, and CloudTrail data events for fis:*. Keep records with the resilience evidence.
  • Data and customers. Run production experiments on synthetic traffic first. If real customers are affected, tell the operations and communications teams in advance.
  • Third parties. You may only test components you control. Do not inject faults against a partner or vendor endpoint without agreement; for third-party dependencies, simulate faults on your side with the fault-injecting handler.

Banking use case

Harbourline's resilience programme runs a quarterly experiment calendar:

QuarterExperimentHypothesisEnvironment
Q1Stop 1 of 3 payment API tasksNo failed payments; recovery under 60 sTest, then production off-peak
Q2Fail over the Aurora cluster to a readerApplication reconnects within 60 sTest
Q3Inject 500 ms latency on calls to the sanctions-screening serviceCircuit breaker opens; payments queue and are retriedTest (local handler plus FIS where supported)
Q4Disrupt connectivity in one Availability Zone (FIS scenario)Remaining AZs carry the load; no customer-facing errorsTest, then game day

Findings go into a tracker with an owner and date. A typical first finding: "deregistration delay was 300 seconds, so the load balancer kept sending a trickle of requests to a stopping task". The fix goes into the next sprint and the experiment is repeated to confirm. The summary sheet is part of the operational-resilience evidence pack; Harbourline does not claim it equals regulatory compliance.

Common mistakes

  • Starting big. All targets, production, no stop condition. Start with one target in a sandbox.
  • No steady state. If you cannot say what "healthy" means in numbers, you cannot tell whether the experiment hurt.
  • Stop condition that cannot fire. An alarm with the wrong metric, dimension or missing data policy gives false confidence.
  • Testing only the infrastructure. Platform recovery is fine but the client has no retry. Include the client and the load.
  • Not fixing the finding. A chaos programme that produces reports and no fixes erodes trust. Track every finding to closure and rerun.
  • Chaos without people. Run it while the team watches; the first runs are learning exercises, not automation.
  • Forgetting cleanup. Some actions (such as stopping instances) leave resources in a changed state. Know how each recovers.

Troubleshooting

SymptomLikely causeFix
start-experiment fails with access deniedThe caller lacks fis:StartExperiment or iam:PassRole for the experiment roleGrant them, scoped to the role and template
Experiment failed immediatelyRole missing an action, or the target filter matches nothingRead the reason in get-experiment; check target parameters and role permissions
No resources targetedWrong cluster, service or tag filter; tasks not runningVerify with aws ecs list-tasks; correct the selection
Experiment stopped by the stop condition at onceThe alarm was already in ALARM, or its threshold is too tightFix the baseline first; tune the alarm
The service did not recoverDesired count was 1, no scheduler to replace tasks, or a deployment was in progressUse a service with desired count of at least 2 and healthy deployments
Alarm shows INSUFFICIENT_DATAMetric has no datapoints at low trafficRun a load generator, or set the alarm's missing-data handling deliberately
Probe shows errors before the experiment startsSystem unhealthy at baselineStop; the experiment's precondition is not met
Local handler never injects faultsEnabled false or error rate 0Check constructor arguments; use a different seed

Cost and cleanup

Cost: credits. FIS bills per minute an action runs (⚠ VERIFY the current per-action-minute rate), so a two-minute single-action experiment costs a tiny amount. The real cost is the ECS service and ALB you are testing against: a small Fargate service with three tasks and an ALB costs in the range of cents to a couple of dollars if torn down within the hour ⚠ VERIFY with the Pricing Calculator. Free alternative: use the FaultInjectingHandler and the gate and probe against a local container; no AWS account needed.

# Make sure no experiment is still running.
aws fis list-experiments --query "experiments[?state.status=='running'].id" \
  --region us-east-1 --profile learn-sandbox

# Delete the experiment template and role (CDK/Terraform: destroy the stack).
aws fis delete-experiment-template --id EXT1a2b3c4d5e6f7 --region us-east-1 --profile learn-sandbox
aws iam delete-role-policy --role-name hl-m12-fis-role --policy-name stop-lab-tasks --profile learn-sandbox
aws iam delete-role --role-name hl-m12-fis-role --profile learn-sandbox

# Delete the lab ECS service, ALB and alarms (or: cdk destroy --all / terraform destroy).
aws ecs update-service --cluster hl-m12-cluster --service hl-m12-accounts-api --desired-count 0 \
  --region us-east-1 --profile learn-sandbox
aws ecs delete-service --cluster hl-m12-cluster --service hl-m12-accounts-api --force \
  --region us-east-1 --profile learn-sandbox
aws cloudwatch delete-alarms --alarm-names hl-m12-api-success-rate-low --region us-east-1 --profile learn-sandbox

# Confirm nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --region us-east-1 --profile learn-sandbox --query "ResourceTagMappingList[].ResourceARN"

Exercises

  • 🟢 Guided: Write the hypothesis, steady-state metrics, blast radius and stop condition for "stop one of three tasks" on paper. Then run the local version: wrap an HttpClient in FaultInjectingHandler with a 30% error rate and 100 ms latency, and call it with and without a retry policy from the resilience-patterns lesson. Compare success rates.
  • 🟡 Challenge: Write xUnit v3 tests for ExperimentGate covering each rule (no stop condition, alarm not OK, blast radius too big, no human, prod without change approval) and for SteadyStateProbe.HoldsSteadyState at the threshold edges. Then, if you have a sandbox ECS service, run the real experiment from the CLI.
  • 🔴 Live problem: A product owner says "we cannot run chaos tests in production, customers might notice". Write a staged plan (synthetic traffic, canary tenant, off-peak window, limited percentage, automatic stop) with the approvals you need and the evidence you will show the risk committee.

Assignment

Produce an experiment card for one Harbourline service: steady-state metrics, hypothesis, injected fault, blast radius, stop condition, approvals, expected recovery, what you observed, and the fixes raised. Include the experiment template (CDK or Terraform) and the gate tests. Commit it to your portfolio repository.

Solutions

Knowledge check

Interview questions

  • What is chaos engineering and how is it different from testing? Controlled experiments on a running system to verify steady-state behaviour under real failure modes, rather than asserting expected outputs for given inputs. It tests assumptions about the interaction of components, retries, timeouts and humans.
  • How would you introduce chaos engineering in a regulated bank? Start in non-production with small blast radius, version-controlled templates, change approval, stop conditions and evidence retention; graduate to production off-peak with synthetic traffic, and involve risk and operations early.
  • What would you look at after a failed experiment? The timeline of the fault, ALB health checks and deregistration delay, client timeouts and retries, autoscaling reaction time, and alarms.
  • What are the risks of FIS itself? It acts on real resources, so broad roles, missing stop conditions or wrong targets can cause real outages. Mitigate with scoped roles, tag conditions, approvals and a human on call.

Certification practice

Completion checklist

  • I can state steady state, hypothesis, blast radius and stop condition for an experiment
  • I can explain targets, actions, stop conditions and the experiment role
  • I ran the local fault-injecting handler exercise (and, optionally, the FIS experiment)
  • I ran the cleanup steps and confirmed nothing is still billing

Key takeaways

  • Chaos engineering is the scientific method applied to resilience: hypothesis, controlled fault, observation, fix.
  • FIS gives controlled actions and alarm-based stop conditions; the experiment role is as powerful as it is scoped.
  • Start small: one target, a sandbox, a human watching.
  • Gate experiments in code and keep the record: it is resilience evidence.
  • Use local fault injection for code-level resilience and FIS for platform faults.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page