Chaos engineering with AWS Fault Injection Service
Plan and run a safe first chaos experiment: write a steady-state hypothesis, build an AWS Fault Injection Service experiment template that stops ECS tasks with a CloudWatch alarm as the stop condition, gate it with a pre-flight check in C#, and rehearse the same idea locally for free.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.FIS) · AWS CLI v2 · AWS CDK v2 (C#) · Terraform AWS provider 6.x · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain chaos engineering as controlled experiments on a steady state, not random breakage.
- Write a hypothesis, a blast radius and a stop condition for an experiment.
- Create an AWS Fault Injection Service (FIS) experiment template with targets, actions, a stop-condition alarm and a least-privilege experiment role.
- Gate an experiment with a C# pre-flight check and observe the steady state while it runs.
- Run the same pattern locally and free with a fault-injecting HTTP handler.
Prerequisites
- Module 11: Monitoring, because alarms are both your stop condition and your evidence.
- Health checks and Deploy to Fargate, for the service you will break.
- The auto scaling, load balancing and resilience-pattern lessons earlier in this module (see the module overview).
- Tools: AWS CLI v2 (
learn-sandbox), .NET 10 SDK. A running ECS service is needed only for the real AWS experiment.
Why this matters in banking
Harbourline's design says: "If one payment API task dies, the load balancer removes it and the service replaces it, with no failed payments." Everyone believes that. Nobody has watched it happen under load. Then a task is killed during a release and some payments fail because the client library has no retry, the load balancer's deregistration delay is longer than the client timeout, and the health check is too slow to notice.
Chaos engineering is the discipline of finding out on a Tuesday morning, on purpose, with a safety net, rather than on a Saturday night by accident. For a regulated firm there is a second benefit: an experiment record (hypothesis, what was injected, what happened, what was fixed) is evidence that resilience was tested, not assumed.
Reminder: FIS performs real actions on real resources. Start in a non-production environment, with small blast radius, and with an agreed stop condition. Production experiments need change approval and risk sign-off.
The idea in plain language
A chaos experiment follows the scientific method:
- Define steady state: a measurable signal that says the system is healthy (for example, payment success rate at least 99.9% and p99 latency under 800 ms).
- Form a hypothesis: "If we stop one of three API tasks, the steady state stays within limits."
- Inject a fault, small and controlled: stop one task.
- Observe: did the steady state hold?
- Learn and fix: if it did not, you found a weakness cheaply. If it did, you have evidence, and you can increase the blast radius next time.
Analogy: A fire drill. You don't set the building on fire. You pull the alarm at a planned time, with the fire marshal watching and a clear rule ("if someone is hurt, stop"), and you time how long evacuation takes. Afterwards you fix the blocked exit you discovered.
What problem FIS solves
AWS Fault Injection Service is a managed service for running fault injection experiments on AWS workloads. Its building blocks:
- Experiment template: the blueprint, with actions, targets and stop conditions.
- Action: what to do to a resource, such as stop ECS tasks, stop or terminate EC2 instances, reboot or fail over RDS, inject API errors, throttle or disrupt network connectivity. Actions have a duration or run until stopped, and can run in sequence or in parallel.
- Target: which resources the action applies to, selected by ARN, tags and filters, with a selection mode (all, a count, or a percentage).
- Stop condition: a CloudWatch alarm. If it goes into the ALARM state while the experiment runs, FIS stops the experiment.
- Experiment role: an IAM role FIS assumes to perform the actions. Scope it to the minimum.
- Scenarios: pre-built, multi-action experiment recipes in the FIS scenario library, for example for loss of an Availability Zone or cross-Region connectivity ⚠ VERIFY current scenario names.
DynamoDB global tables and several other services also publish FIS-compatible actions, such as pausing replication to simulate a Region isolation; see the global tables documentation for the current action list.
Honest limits:
- Not every failure is injectable. FIS covers a defined list of actions; check the actions reference for your service ⚠ VERIFY.
- It cannot tell you what to measure. Your alarms and dashboards define the steady state; poor observability means a poor experiment.
- It costs money: billed per minute that an action runs, per target account ⚠ VERIFY the current rate on the FIS pricing page. Experiments are short, so the cost is small, but the resources you break may also bill while recovering.
- Single-account by default: targets must be in the same account as the experiment unless you use multi-account experiments.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Chaos Studio experiments | FIS experiment templates | Both define steps, targets and faults; FIS stop conditions are CloudWatch alarms |
| Chaos Studio target onboarding | FIS targets by ARN or tag; some actions need an agent via Systems Manager | Service-level actions (stop task, failover RDS) need no agent |
| Chaos Studio managed identity | FIS experiment role | A normal IAM role with a trust policy for fis.amazonaws.com |
| Polly chaos strategies (Simmy) | FIS for infrastructure faults, a fault-injecting handler for code-level faults | Use both: FIS for the platform, code-level chaos for dependency behaviour |
| Azure Monitor alerts as guardrail | CloudWatch alarm as stop condition | The alarm is attached to the template, not to a separate rule |
Architecture
Security-boundary view: the experiment role can stop ECS tasks in one cluster and nothing else. The people who may start experiments are different from those who can edit templates (separation of duties), and every start is logged by CloudTrail. In a bank, templates live in version control and run through change approval; ad-hoc console experiments are for the sandbox only.
Step-by-step implementation
Define the steady state and the hypothesis
Write it down before touching AWS. Example: "Payment success rate stays at least 99.9% and p99 latency below 800 ms while one of three Accounts API tasks is stopped, because the ALB removes the unhealthy target within 30 seconds and clients retry once."
Make the steady state observable
Create CloudWatch alarms: one on the ALB's HTTPCode_Target_5XX_Count or a custom success-rate metric, and one on TargetResponseTime p99. The stop-condition alarm should fire a little before the business impact would breach the tolerance.
Choose a small blast radius
Target one task (COUNT(1)) in one service, in a sandbox, during working hours with the team present. Never start with "all".
Create the experiment role
A role trusted by fis.amazonaws.com with ecs:StopTask, ecs:ListTasks, ecs:DescribeTasks and tag:GetResources, restricted by resource and condition to the lab cluster ⚠ VERIFY the exact action list in the FIS actions reference.
Create the experiment template
Define the target (tasks of the lab service), the action aws:ecs:stop-task, and the stop condition. Review it like code.
Run the pre-flight gate, start the load, start the experiment
The gate checks that the alarm is OK, the target is tagged as a lab resource, the blast radius is within limits and a human is on call. Start a steady load, then start the experiment and watch.
Record the result and fix
Compare observations with the hypothesis. Write down what happened, what you changed (retry, health check interval, deregistration delay, minimum task count) and the date you will rerun it.
AWS Console instructions
- Create a template: Console → AWS Fault Injection Service → Experiment templates → Create experiment template. Add an action (for example
aws:ecs:stop-task), define the target (resource type, tag or ARN filter, selection mode such as Count with1), add a stop condition (CloudWatch alarm), and choose the IAM role. - Start: open the template → Start experiment → add tags (
Project=cea-lab) → confirm by typingstart. - Watch: Experiments → select the run → Timeline shows each action's state. In CloudWatch open the dashboard with success rate, p99, and ECS running task count.
- Stop early: Experiments → select the run → Stop experiment.
- Check the guardrail: CloudWatch → Alarms → confirm the stop-condition alarm is
OKbefore starting.
AWS CLI
# Create the experiment template from a JSON file (see below).
aws fis create-experiment-template --cli-input-json file://template.json \
--region us-east-1 --profile learn-sandbox
# List templates.
aws fis list-experiment-templates \
--query "experimentTemplates[].{Id:id,Description:description}" \
--region us-east-1 --profile learn-sandbox
# Check the stop-condition alarm state (must be OK).
aws cloudwatch describe-alarms --alarm-names hl-m12-api-success-rate-low \
--query "MetricAlarms[].StateValue" --region us-east-1 --profile learn-sandbox
# Start the experiment, tagged for cleanup and audit.
aws fis start-experiment --experiment-template-id EXT1a2b3c4d5e6f7 \
--tags Project=cea-lab,Lab=m12-fis,Environment=learn \
--region us-east-1 --profile learn-sandbox
# Poll the state: initiating, running, completed, stopped or failed.
aws fis get-experiment --id EXP1a2b3c4d5e6f7 \
--query "experiment.{State:state.status,Reason:state.reason}" \
--region us-east-1 --profile learn-sandbox
# Stop an experiment immediately (the emergency brake).
aws fis stop-experiment --id EXP1a2b3c4d5e6f7 --region us-east-1 --profile learn-sandbox
# See the running task count recover after the experiment.
aws ecs describe-services --cluster hl-m12-cluster --services hl-m12-accounts-api \
--query "services[0].{Desired:desiredCount,Running:runningCount}" \
--region us-east-1 --profile learn-sandboxA minimal template.json that stops one task of a service, with a stop condition:
{
"description": "hl-m12: stop 1 of 3 Accounts API tasks",
"roleArn": "arn:aws:iam::111122223333:role/hl-m12-fis-role",
"targets": {
"ApiTasks": {
"resourceType": "aws:ecs:task",
"selectionMode": "COUNT(1)",
"parameters": {
"cluster": "hl-m12-cluster",
"service": "hl-m12-accounts-api"
}
}
},
"actions": {
"StopOneTask": {
"actionId": "aws:ecs:stop-task",
"description": "Stop one task",
"targets": { "Tasks": "ApiTasks" }
}
},
"stopConditions": [
{
"source": "aws:cloudwatch:alarm",
"value": "arn:aws:cloudwatch:us-east-1:111122223333:alarm:hl-m12-api-success-rate-low"
}
],
"tags": { "Project": "cea-lab", "Lab": "m12-fis", "Environment": "learn" }
}The action ID, target key and target parameters follow the FIS actions reference at the time of writing, but they change as FIS adds actions, so confirm them with the reference or aws fis list-actions before relying on them ⚠ VERIFY. Template and experiment IDs and the account number are placeholders.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.FIS;
using Amazon.CDK.AWS.IAM;
using Constructs;
namespace Harbourline.Ha.Infra;
public sealed class FisStack : Stack
{
public FisStack(Construct scope, string id, string clusterName, string serviceName,
string clusterArn, IAlarm stopAlarm, IStackProps props) : base(scope, id, props)
{
// Experiment role: FIS may stop tasks in ONE cluster, nothing else.
var role = new Role(this, "FisRole", new RoleProps
{
RoleName = "hl-m12-fis-role",
AssumedBy = new ServicePrincipal("fis.amazonaws.com")
});
role.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
Effect = Effect.ALLOW,
Actions = new[] { "ecs:StopTask" },
Resources = new[] { $"arn:aws:ecs:{Region}:{Account}:task/{clusterName}/*" }
}));
role.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
Effect = Effect.ALLOW,
Actions = new[] { "ecs:ListTasks", "ecs:DescribeTasks", "tag:GetResources" },
Resources = new[] { "*" } // list and describe actions do not support narrow resources ⚠ VERIFY
}));
_ = new CfnExperimentTemplate(this, "StopOneTask", new CfnExperimentTemplateProps
{
Description = "hl-m12: stop 1 of 3 Accounts API tasks",
RoleArn = role.RoleArn,
Targets = new Dictionary<string, object>
{
["ApiTasks"] = new CfnExperimentTemplate.ExperimentTemplateTargetProperty
{
ResourceType = "aws:ecs:task",
SelectionMode = "COUNT(1)",
Parameters = new Dictionary<string, string>
{
["cluster"] = clusterName,
["service"] = serviceName
}
}
},
Actions = new Dictionary<string, object>
{
["StopOneTask"] = new CfnExperimentTemplate.ExperimentTemplateActionProperty
{
ActionId = "aws:ecs:stop-task",
Targets = new Dictionary<string, string> { ["Tasks"] = "ApiTasks" }
}
},
StopConditions = new[]
{
new CfnExperimentTemplate.ExperimentTemplateStopConditionProperty
{
Source = "aws:cloudwatch:alarm",
Value = stopAlarm.AlarmArn
}
},
Tags = new Dictionary<string, string>
{
["Project"] = "cea-lab", ["Lab"] = "m12-fis", ["Environment"] = "learn"
}
});
}
}The type of Targets and Actions on CfnExperimentTemplateProps is a map of property objects; if your CDK version complains, use the IResolvable overload or build the template from CfnJson/raw overrides ⚠ VERIFY. The clusterArn argument is shown for parity with the Terraform and can be used to tighten the role further.
Complete code
Three small pieces: a pre-flight gate, a steady-state probe that runs during the experiment, and a local fault-injecting handler that gives you the same learning for free.
namespace Harbourline.Ha.Chaos;
public sealed record ExperimentPlan(
string Name,
string Hypothesis,
string Environment, // "learn", "test", "prod"
int TargetCount,
int TotalTargets,
bool HasStopCondition,
string StopAlarmState, // OK, ALARM, INSUFFICIENT_DATA
bool HumanOnCall,
bool ChangeApproved);
public sealed record GateResult(bool Allowed, IReadOnlyList<string> Problems);
public static class ExperimentGate
{
/// <summary>Refuses to start an experiment that breaks the safety rules.</summary>
public static GateResult Evaluate(ExperimentPlan p, double maxBlastRadius = 0.34)
{
var problems = new List<string>();
if (string.IsNullOrWhiteSpace(p.Hypothesis))
problems.Add("No written hypothesis.");
if (!p.HasStopCondition)
problems.Add("No stop condition: an experiment without a guardrail must not run.");
if (p.StopAlarmState != "OK")
problems.Add($"Stop-condition alarm is {p.StopAlarmState}, expected OK (system not in steady state).");
if (p.TotalTargets <= 0 || (double)p.TargetCount / p.TotalTargets > maxBlastRadius)
problems.Add($"Blast radius too large: {p.TargetCount} of {p.TotalTargets} targets (limit {maxBlastRadius:P0}).");
if (!p.HumanOnCall)
problems.Add("Nobody confirmed they are watching.");
if (p.Environment == "prod" && !p.ChangeApproved)
problems.Add("Production experiments need an approved change record.");
return new GateResult(problems.Count == 0, problems);
}
}using System.Diagnostics;
namespace Harbourline.Ha.Chaos;
public sealed record ProbeResult(int Total, int Failures, double P99Ms)
{
public double SuccessRate => Total == 0 ? 0 : 1.0 - (double)Failures / Total;
}
public static class SteadyStateProbe
{
/// <summary>Calls the URL repeatedly for the given duration and summarises success rate and p99 latency.</summary>
public static async Task<ProbeResult> RunAsync(HttpClient http, Uri url, TimeSpan duration,
TimeSpan interval, CancellationToken ct)
{
var latencies = new List<double>();
var failures = 0;
var end = DateTime.UtcNow + duration;
while (DateTime.UtcNow < end && !ct.IsCancellationRequested)
{
var sw = Stopwatch.StartNew();
try
{
using var resp = await http.GetAsync(url, ct);
if (!resp.IsSuccessStatusCode) failures++;
}
catch (Exception) when (!ct.IsCancellationRequested)
{
failures++;
}
latencies.Add(sw.Elapsed.TotalMilliseconds);
await Task.Delay(interval, ct);
}
latencies.Sort();
var p99 = latencies.Count == 0 ? 0 : latencies[(int)Math.Min(latencies.Count - 1, Math.Ceiling(latencies.Count * 0.99) - 1)];
return new ProbeResult(latencies.Count, failures, p99);
}
public static bool HoldsSteadyState(ProbeResult r, double minSuccess = 0.999, double maxP99Ms = 800) =>
r.Total > 0 && r.SuccessRate >= minSuccess && r.P99Ms <= maxP99Ms;
}namespace Harbourline.Ha.Chaos;
/// <summary>
/// A local chaos tool: wrap HttpClient with this handler to inject errors and latency.
/// Free, deterministic with a seed, and good for proving retry and circuit-breaker behaviour.
/// </summary>
public sealed class FaultInjectingHandler(HttpMessageHandler inner, double errorRate, TimeSpan extraLatency, int seed = 42)
: DelegatingHandler(inner)
{
private readonly Random _random = new(seed);
private readonly object _gate = new();
public bool Enabled { get; set; } = true;
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct)
{
if (Enabled)
{
bool fail;
lock (_gate) fail = _random.NextDouble() < errorRate;
if (extraLatency > TimeSpan.Zero) await Task.Delay(extraLatency, ct);
if (fail) return new HttpResponseMessage(System.Net.HttpStatusCode.ServiceUnavailable)
{
RequestMessage = request,
Content = new StringContent("injected fault")
};
}
return await base.SendAsync(request, ct);
}
}A console runner ties the real AWS experiment to the gate and the probe:
using Amazon;
using Amazon.CloudWatch;
using Amazon.CloudWatch.Model;
using Amazon.FIS;
using Amazon.FIS.Model;
using Harbourline.Ha.Chaos;
var templateId = args.ElementAtOrDefault(0) ?? throw new ArgumentException("Usage: <templateId> <url> <alarmName>");
var url = new Uri(args.ElementAtOrDefault(1) ?? throw new ArgumentException("missing url"));
var alarmName = args.ElementAtOrDefault(2) ?? "hl-m12-api-success-rate-low";
using var cw = new AmazonCloudWatchClient(RegionEndpoint.USEast1);
using var fis = new AmazonFISClient(RegionEndpoint.USEast1);
using var http = new HttpClient { Timeout = TimeSpan.FromSeconds(3) };
// 1. Read the guardrail alarm state.
var alarms = await cw.DescribeAlarmsAsync(new DescribeAlarmsRequest { AlarmNames = [alarmName] });
var alarmState = alarms.MetricAlarms.FirstOrDefault()?.StateValue?.Value ?? "INSUFFICIENT_DATA";
// 2. Pre-flight gate.
var plan = new ExperimentPlan(
Name: "stop-one-api-task",
Hypothesis: "One of three tasks can stop without breaching 99.9% success or 800 ms p99.",
Environment: "learn",
TargetCount: 1, TotalTargets: 3,
HasStopCondition: true,
StopAlarmState: alarmState,
HumanOnCall: args.Contains("--i-am-watching"),
ChangeApproved: false);
var gate = ExperimentGate.Evaluate(plan);
if (!gate.Allowed)
{
foreach (var problem in gate.Problems) Console.WriteLine($"BLOCKED: {problem}");
return 1;
}
// 3. Baseline, experiment, observe.
var baseline = await SteadyStateProbe.RunAsync(http, url, TimeSpan.FromSeconds(30), TimeSpan.FromMilliseconds(100), CancellationToken.None);
Console.WriteLine($"Baseline: {baseline.SuccessRate:P2} success, p99 {baseline.P99Ms:F0} ms");
var started = await fis.StartExperimentAsync(new StartExperimentRequest
{
ExperimentTemplateId = templateId,
ClientToken = Guid.NewGuid().ToString(),
Tags = new Dictionary<string, string> { ["Project"] = "cea-lab", ["Lab"] = "m12-fis" }
});
var experimentId = started.Experiment.Id;
Console.WriteLine($"Started experiment {experimentId} at {DateTime.UtcNow:O}");
var during = await SteadyStateProbe.RunAsync(http, url, TimeSpan.FromSeconds(90), TimeSpan.FromMilliseconds(100), CancellationToken.None);
Console.WriteLine($"During: {during.SuccessRate:P2} success, p99 {during.P99Ms:F0} ms");
// 4. Verdict against the hypothesis.
var held = SteadyStateProbe.HoldsSteadyState(during);
var state = (await fis.GetExperimentAsync(new GetExperimentRequest { Id = experimentId })).Experiment.State.Status.Value;
Console.WriteLine($"FIS state: {state}. Hypothesis {(held ? "HELD" : "REJECTED")}.");
return held ? 0 : 2;Package references: AWSSDK.FIS, AWSSDK.CloudWatch (4.x; check NuGet for current versions). The request and response property names above follow the SDK's naming conventions ⚠ VERIFY against the installed package, since the FIS model types are generated from the service API.
Code explained line by line
ExperimentPlanis the written plan as data: hypothesis, environment, blast radius, stop condition and approvals.ExperimentGate.Evaluatereturns all problems, not just the first, so a reviewer sees everything that must be fixed. The default limit of about a third of targets means "no more than one AZ's worth" in a three-AZ design.- The gate insists the alarm is currently
OK: starting a chaos experiment while the system is already unhealthy is just a mess, and the stop condition would trip immediately. SteadyStateProbe.RunAsyncsamples the endpoint and collects latencies. The p99 index usesMath.Ceiling(count * 0.99) - 1, the nearest-rank method, clamped to the array.HoldsSteadyStateencodes the hypothesis thresholds in code, so the verdict is not a matter of opinion.FaultInjectingHandleris aDelegatingHandlerthat randomly returns 503 and adds latency. It lets you prove, locally and for free, that yourMicrosoft.Extensions.Http.Resiliencepipeline from the resilience-patterns lesson works under faults. A fixed seed makes the failure pattern repeatable in tests.- In
Program.csthe runner reads the alarm, runs the gate, measures a baseline, starts FIS with tags, measures during the experiment and prints a verdict. Exit code 2 means the hypothesis was rejected, which a pipeline can treat as a finding. ClientTokenmakes the start request idempotent, so a retry never starts two experiments.
Security considerations
⚠ Learning shortcut — not for production
The lab runs in a sandbox with broad ListTasks and DescribeTasks permissions, a single account, no change record and a human watching on a flag. A production programme adds change management, separate roles for authoring and running experiments, multi-account experiments from a central resilience account, experiment logging to a protected bucket, and notification to the incident channel.
- The experiment role is a weapon. Scope
ecs:StopTaskto one cluster or tag, and require a tag condition on targets so a typo cannot reach production. - Who can start an experiment. Allow
fis:StartExperimentfor a small group, require MFA, and deny it in production accounts by SCP except for the resilience role. - Stop conditions are mandatory. An alarm with
INSUFFICIENT_DATAdoes not protect you. Test the alarm itself. - Logging and evidence. Enable FIS experiment logging to CloudWatch Logs or S3, and CloudTrail data events for
fis:*. Keep records with the resilience evidence. - Data and customers. Run production experiments on synthetic traffic first. If real customers are affected, tell the operations and communications teams in advance.
- Third parties. You may only test components you control. Do not inject faults against a partner or vendor endpoint without agreement; for third-party dependencies, simulate faults on your side with the fault-injecting handler.
Banking use case
Harbourline's resilience programme runs a quarterly experiment calendar:
| Quarter | Experiment | Hypothesis | Environment |
|---|---|---|---|
| Q1 | Stop 1 of 3 payment API tasks | No failed payments; recovery under 60 s | Test, then production off-peak |
| Q2 | Fail over the Aurora cluster to a reader | Application reconnects within 60 s | Test |
| Q3 | Inject 500 ms latency on calls to the sanctions-screening service | Circuit breaker opens; payments queue and are retried | Test (local handler plus FIS where supported) |
| Q4 | Disrupt connectivity in one Availability Zone (FIS scenario) | Remaining AZs carry the load; no customer-facing errors | Test, then game day |
Findings go into a tracker with an owner and date. A typical first finding: "deregistration delay was 300 seconds, so the load balancer kept sending a trickle of requests to a stopping task". The fix goes into the next sprint and the experiment is repeated to confirm. The summary sheet is part of the operational-resilience evidence pack; Harbourline does not claim it equals regulatory compliance.
Common mistakes
- Starting big. All targets, production, no stop condition. Start with one target in a sandbox.
- No steady state. If you cannot say what "healthy" means in numbers, you cannot tell whether the experiment hurt.
- Stop condition that cannot fire. An alarm with the wrong metric, dimension or missing data policy gives false confidence.
- Testing only the infrastructure. Platform recovery is fine but the client has no retry. Include the client and the load.
- Not fixing the finding. A chaos programme that produces reports and no fixes erodes trust. Track every finding to closure and rerun.
- Chaos without people. Run it while the team watches; the first runs are learning exercises, not automation.
- Forgetting cleanup. Some actions (such as stopping instances) leave resources in a changed state. Know how each recovers.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
start-experiment fails with access denied | The caller lacks fis:StartExperiment or iam:PassRole for the experiment role | Grant them, scoped to the role and template |
Experiment failed immediately | Role missing an action, or the target filter matches nothing | Read the reason in get-experiment; check target parameters and role permissions |
| No resources targeted | Wrong cluster, service or tag filter; tasks not running | Verify with aws ecs list-tasks; correct the selection |
| Experiment stopped by the stop condition at once | The alarm was already in ALARM, or its threshold is too tight | Fix the baseline first; tune the alarm |
| The service did not recover | Desired count was 1, no scheduler to replace tasks, or a deployment was in progress | Use a service with desired count of at least 2 and healthy deployments |
Alarm shows INSUFFICIENT_DATA | Metric has no datapoints at low traffic | Run a load generator, or set the alarm's missing-data handling deliberately |
| Probe shows errors before the experiment starts | System unhealthy at baseline | Stop; the experiment's precondition is not met |
| Local handler never injects faults | Enabled false or error rate 0 | Check constructor arguments; use a different seed |
Cost and cleanup
Cost: credits. FIS bills per minute an action runs (⚠ VERIFY the current per-action-minute rate), so a two-minute single-action experiment costs a tiny amount. The real cost is the ECS service and ALB you are testing against: a small Fargate service with three tasks and an ALB costs in the range of cents to a couple of dollars if torn down within the hour ⚠ VERIFY with the Pricing Calculator. Free alternative: use the FaultInjectingHandler and the gate and probe against a local container; no AWS account needed.
# Make sure no experiment is still running.
aws fis list-experiments --query "experiments[?state.status=='running'].id" \
--region us-east-1 --profile learn-sandbox
# Delete the experiment template and role (CDK/Terraform: destroy the stack).
aws fis delete-experiment-template --id EXT1a2b3c4d5e6f7 --region us-east-1 --profile learn-sandbox
aws iam delete-role-policy --role-name hl-m12-fis-role --policy-name stop-lab-tasks --profile learn-sandbox
aws iam delete-role --role-name hl-m12-fis-role --profile learn-sandbox
# Delete the lab ECS service, ALB and alarms (or: cdk destroy --all / terraform destroy).
aws ecs update-service --cluster hl-m12-cluster --service hl-m12-accounts-api --desired-count 0 \
--region us-east-1 --profile learn-sandbox
aws ecs delete-service --cluster hl-m12-cluster --service hl-m12-accounts-api --force \
--region us-east-1 --profile learn-sandbox
aws cloudwatch delete-alarms --alarm-names hl-m12-api-success-rate-low --region us-east-1 --profile learn-sandbox
# Confirm nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--region us-east-1 --profile learn-sandbox --query "ResourceTagMappingList[].ResourceARN"Exercises
- 🟢 Guided: Write the hypothesis, steady-state metrics, blast radius and stop condition for "stop one of three tasks" on paper. Then run the local version: wrap an
HttpClientinFaultInjectingHandlerwith a 30% error rate and 100 ms latency, and call it with and without a retry policy from the resilience-patterns lesson. Compare success rates. - 🟡 Challenge: Write xUnit v3 tests for
ExperimentGatecovering each rule (no stop condition, alarm not OK, blast radius too big, no human, prod without change approval) and forSteadyStateProbe.HoldsSteadyStateat the threshold edges. Then, if you have a sandbox ECS service, run the real experiment from the CLI. - 🔴 Live problem: A product owner says "we cannot run chaos tests in production, customers might notice". Write a staged plan (synthetic traffic, canary tenant, off-peak window, limited percentage, automatic stop) with the approvals you need and the evidence you will show the risk committee.
Assignment
Produce an experiment card for one Harbourline service: steady-state metrics, hypothesis, injected fault, blast radius, stop condition, approvals, expected recovery, what you observed, and the fixes raised. Include the experiment template (CDK or Terraform) and the gate tests. Commit it to your portfolio repository.
Solutions
Steady state: success at least 99.9% and p99 under 800 ms for the payments endpoint, measured by the ALB metrics and your probe. Hypothesis: with three tasks, stopping one keeps both within limits because the ALB deregisters the failed target and the service starts a replacement. Blast radius: 1 of 3 tasks (33%), one service, sandbox. Stop condition: alarm on success rate below 99% for two consecutive minutes. Local run: with a 30% injected error rate and no retry, success is about 70%; with a retry policy that retries once with jitter, success rises to about 91% (1 minus 0.3 squared) and with two retries about 97%. The lesson: retries lift success dramatically, but also multiply load on a failing dependency, which is why circuit breakers and bulkheads exist.
Create a baseline valid plan (hypothesis present, stop condition true, alarm OK, 1 of 3 targets, human on call, environment learn) and assert Allowed is true. Then mutate one field per test: empty hypothesis, HasStopCondition false, alarm ALARM, TargetCount 2 of 3 (66%), HumanOnCall false, environment prod with ChangeApproved false, and assert Allowed is false and the problems list contains the expected text. Add a test with several faults at once and assert all problems are reported. For the probe, use a ProbeResult(1000, 1, 799) (passes) and ProbeResult(1000, 2, 799) (fails at 99.8%) to test the boundaries; the reason to test edges is that off-by-one errors in thresholds are exactly what silently weakens guardrails.
Knowledge check
Answer: Actions (what to do), targets (which resources), stop conditions (CloudWatch alarms that abort the experiment) and an IAM experiment role. Without stop conditions the experiment has no guardrail, and without a narrow role it can do more damage than intended.
Answer: The system must be in steady state to learn anything. If the alarm is already firing, the experiment stops immediately or, worse, masks existing problems.
Answer: The amount of the system an experiment can affect. Keep it small with a count or percentage selection mode, a single service, a non-production environment first, and short duration.
Answer: The handler injects faults inside your client code (errors and latency for a dependency), testing the application's resilience logic for free. FIS injects faults into real AWS infrastructure (stopped tasks, failovers, network disruption). Use both.
Answer: Treat it as a finding: record it, fix the weakness (retries, health check timing, capacity, deregistration delay), and rerun the same experiment to confirm the fix.
Interview questions
- What is chaos engineering and how is it different from testing? Controlled experiments on a running system to verify steady-state behaviour under real failure modes, rather than asserting expected outputs for given inputs. It tests assumptions about the interaction of components, retries, timeouts and humans.
- How would you introduce chaos engineering in a regulated bank? Start in non-production with small blast radius, version-controlled templates, change approval, stop conditions and evidence retention; graduate to production off-peak with synthetic traffic, and involve risk and operations early.
- What would you look at after a failed experiment? The timeline of the fault, ALB health checks and deregistration delay, client timeouts and retries, autoscaling reaction time, and alarms.
- What are the risks of FIS itself? It acts on real resources, so broad roles, missing stop conditions or wrong targets can cause real outages. Mitigate with scoped roles, tag conditions, approvals and a human on call.
Certification practice
A team wants to verify that an ECS service tolerates the loss of a task and that the experiment stops automatically if the error rate rises above 2%. Which combination should they use?
A. An FIS experiment template with the aws:ecs:stop-task action and a CloudWatch alarm on error rate as the stop condition.
B. A Lambda function that stops a random task every hour.
C. An Auto Scaling scheduled action that sets desired count to 0.
D. A manual aws ecs stop-task run in production at night.
Answer: A. FIS provides controlled actions, targets and alarm-based stop conditions with an auditable experiment record. B has no guardrail or hypothesis. C removes all capacity. D is unstructured and unguarded.
An FIS experiment stopped itself after 40 seconds although the ECS service was healthy. The stop-condition alarm shows INSUFFICIENT_DATA then ALARM. What is the most likely cause and fix?
A. The experiment role lacks ecs:StopTask; add it.
B. The alarm's metric has sparse data and the missing-data treatment sets it to ALARM; generate steady traffic and set an appropriate missing-data handling.
C. FIS limits experiments to 30 seconds.
D. The ECS service is in the wrong Region.
Answer: B. Alarms on sparse metrics can flip state, and a stop condition in ALARM aborts the experiment. A would give a failed action, not a stop. C is false. D would result in no targets, not an alarm stop.
A bank wants to prove that its workload survives the loss of an Availability Zone, with controls suitable for a regulated environment. Which approach is best?
A. Terminate half of the production instances during business hours without notice. B. Run an FIS experiment or scenario against a test environment first, with narrowly scoped roles, tagged targets, a CloudWatch stop condition and an approved change, then repeat in production off-peak with synthetic traffic. C. Rely on the AWS Service Level Agreement as evidence. D. Delete the standby database to see if alarms fire.
Answer: B. It stages the experiment with guardrails, approvals and evidence. A is uncontrolled. C is not evidence about your own design. D causes needless data risk.
Completion checklist
- I can state steady state, hypothesis, blast radius and stop condition for an experiment
- I can explain targets, actions, stop conditions and the experiment role
- I ran the local fault-injecting handler exercise (and, optionally, the FIS experiment)
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- Chaos engineering is the scientific method applied to resilience: hypothesis, controlled fault, observation, fix.
- FIS gives controlled actions and alarm-based stop conditions; the experiment role is as powerful as it is scoped.
- Start small: one target, a sandbox, a human watching.
- Gate experiments in code and keep the record: it is resilience evidence.
- Use local fault injection for code-level resilience and FIS for platform faults.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Performance and cost optimisation for resilient systems
Right-size auto-scaled fleets, move .NET services to Graviton, use Spot and Savings Plans safely, and put a price on idle disaster-recovery capacity with a small C# cost model so resilience spending is a decision, not a surprise.
AWS Resilience Hub and game days
Define a resilience policy with RTO and RPO targets, assess an application with AWS Resilience Hub, then run a realistic failover game day and turn the timings into evidence, using a C# policy checker and drill timeline analyser.