Troubleshooting production issues methodically: symptom, hypothesis, evidence, fix
Follow a repeatable process from a vague symptom to a proven root cause, using a worked 5xx investigation on an ECS service behind an Application Load Balancer, and build the dashboard and evidence-gathering tool that make the next investigation faster.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Apply a seven-step troubleshooting process (frame, scope, what changed, hypotheses, evidence, mitigate, prove the fix) and explain why each step exists.
- Tell an ALB-generated 5xx from a target-generated 5xx, and read 502, 503 and 504 as different clues.
- Use the RED and USE methods to choose which metrics to look at first.
- Work through a 5xx incident on ECS from the ALB down to a stopped task and a deployment, and write the root cause in one precise sentence.
- Build a triage dashboard in code and a C# tool that gathers the first-ten-minutes evidence automatically.
Prerequisites
- Performance bottlenecks in .NET on AWS, because the worked example reuses its vocabulary.
- Log aggregation and error monitoring for fingerprints and Logs Insights.
- CloudTrail for operations for the "what changed?" step.
- CloudWatch metrics and custom metrics, Alarms and composite alarms and Distributed tracing with OpenTelemetry and X-Ray for .NET (ADOT) for the signals you read.
- Tools: AWS CLI v2 with the
learn-sandboxSSO profile, .NET 10 SDK.
Why this matters in banking
At 10:42 the payments API starts returning errors to a few percent of requests. By 10:50 three people have three theories: "the database", "the partner", "the new release". Each person starts looking at a different dashboard, nobody writes anything down, and two engineers change things at the same time. By 11:30 the errors stop, and no one knows why. The post-incident review has nothing to say except "it recovered".
That outcome is the expensive one. It repeats, it hides real causes, and a regulator or internal audit will ask what you knew and when. A method does not make you smarter, but it stops you from skipping steps under pressure, makes the work shareable between people, and leaves a record that a post-incident review can use.
The idea in plain language
Troubleshooting is the scientific method on a clock. You observe, you guess, you test the guess against evidence, and you change one thing at a time. Most wasted hours come from three habits: fixing before understanding, changing several things at once, and trusting a theory instead of testing it.
The loop:
Analogy: a doctor does not operate because the patient says "I feel unwell". She asks when it started, checks vital signs, asks what changed (new medication?), lists likely causes, orders the test that best separates them, treats, then re-checks. The patient chart is the record.
What problem the AWS service solves
AWS does not troubleshoot for you, but it provides the instruments. The skill is knowing which instrument answers which question:
| Question | Instrument |
|---|---|
| Is the symptom real and how big is it? | ALB/API Gateway metrics, Application Signals SLO status, canary results |
| Who is affected? | Logs Insights by route, customer segment, Availability Zone, release |
| What changed? | CloudTrail (infrastructure), ECS deployment history, CI/CD records, AWS Config timeline |
| Where is time or failure concentrated? | Traces, service map, per-operation metrics |
| Is a resource saturated? | CPU, memory, connections, queue depth, throttles, thread-pool queue |
| What does the failing component say? | Application logs by fingerprint, ECS stopped-task reason, ALB access logs |
Two frameworks help choose what to look at first:
- RED for services: Rate (requests per second), Errors (failed fraction), Duration (latency distribution). Ask it of every service in the request path.
- USE for resources: Utilisation, Saturation, Errors of each resource (CPU, memory, disks, network, connection pools, queues).
Rule of thumb: RED tells you which service hurts; USE tells you which resource of that service is the cause.
Honest limits: metrics lag (most are one-minute data points, some arrive several minutes late), logs can be sampled or dropped, traces are sampled, and CloudTrail events can arrive minutes after the call. During an incident treat "no data" as unknown, not as "fine".
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Application Insights "Failures" and "Performance" blades | Application Signals service detail, Logs Insights over structured logs | You assemble the view from metrics, logs and traces |
| Azure Resource Health and Service Health | AWS Health Dashboard (account and public) | Check it early to rule out a provider event ⚠ VERIFY current naming |
| Activity Log "what changed?" | CloudTrail, AWS Config resource timeline, ECS deployment history | Three sources, each with different coverage |
| Azure Monitor "Diagnose and solve problems" | Logs Insights saved queries, runbooks, CloudWatch investigations (where enabled) | Treat any automated AI investigation as a hint, not as proof ⚠ VERIFY availability |
| Kusto queries over many tables | Logs Insights per log group | Fewer joins; correlation uses ids you log |
| Azure Load Balancer / App Gateway diagnostics | ALB metrics and access logs | ALB distinguishes its own 5xx from the target's 5xx |
kubectl describe / App Service logs | aws ecs describe-services, describe-tasks, stopped reason | ECS records why a task stopped for a short time only |
Architecture
The line between HTTPCode_ELB_5XX_Count (the load balancer itself produced the error: no healthy targets, target closed the connection, timeout) and HTTPCode_Target_5XX_Count (your application returned 5xx) is the first fork in most investigations.
| ALB code | Usual meaning | First place to look |
|---|---|---|
| 502 Bad Gateway | Target closed or sent an invalid response; task killed mid-request | ECS stopped-task reasons, application crashes, keep-alive timeouts |
| 503 Service Unavailable | No registered healthy targets | Target group health, task count, deployment in progress |
| 504 Gateway Timeout | Target did not respond before the ALB idle timeout | Slow dependency, thread-pool or connection-pool saturation |
| 500 from target | Application error (counted as target 5xx) | Application logs by fingerprint |
Step-by-step implementation: the worked investigation
The scenario is fictional but realistic. Harbourline's payments API runs on ECS Fargate behind an ALB. At 10:42 on a Wednesday the Application Signals availability SLO shows fast burn and the alarm pages the on-call engineer. The investigation follows the seven steps.
1. Frame the symptom
Write one sentence with facts only: "Since about 10:40 UTC, 3 to 6% of POST /payments requests return 5xx; reads look normal; the SLO burn rate is 40." Note who reported it, the first alert time and the severity. Do not write a cause. Open a shared incident note and start a timeline.
2. Scope it
Is it all routes or one? All Availability Zones or one? All customers or one segment? One release or all? Run one Logs Insights query grouped by route and one by Availability Zone or task id. Result in this case: only POST /payments, spread across both AZs, but concentrated on three of eight task ids. Concentration on a subset of instances is a strong clue: something is wrong with those instances, not with a shared dependency.
3. What changed?
Always ask first, because most incidents follow a change. Check the deployment history, infrastructure changes and traffic. The ECS service shows a deployment completed at 10:31: new task definition revision 47. CloudTrail confirms UpdateService called by the pipeline role at 10:29. Traffic is normal. The change is a prime suspect, but it is a suspect, not a verdict.
4. List hypotheses, ranked
| # | Hypothesis | Predicts | Cheapest test |
|---|---|---|---|
| H1 | Revision 47 has a code defect in the payments path | Errors on all tasks, a new fingerprint | Logs by fingerprint and release |
| H2 | Revision 47 changed resources and tasks are being killed | Errors on tasks that restart; stopped tasks with a kill reason; memory near the limit | describe-tasks stopped reason; memory metric |
| H3 | The database is saturated | DatabaseConnections at max; slow query spans | RDS metrics, trace spans |
| H4 | The partner API is slow | 504s with long partner spans | Dependency metrics |
Order by likelihood and cost of test. Test the cheapest discriminating evidence first.
5. Gather evidence
H3 test: RDS connections at 40% of the limit and query latency normal. H3 disproved. H4 test: partner spans normal, no 504 spike. H4 disproved. H1 test: no new fingerprint in the logs; the errors are 502 from the ALB, not 500 from the app. H1 unlikely. H2 test: describe-tasks for stopped tasks shows stoppedReason "OutOfMemoryError: Container killed due to memory usage" with exit code 137 for four tasks in the last 15 minutes. ECS MemoryUtilization for the service climbs to 100% before each stop. The task definition diff between revisions 46 and 47 shows the memory limit changed from 1024 to 512 MiB in the pipeline's parameters. H2 supported.
6. Mitigate first, then fix
Roll the service back to revision 46 (update-service --task-definition ...:46). Mitigation is not the fix; it stops the bleeding with the least risky action. Record the time. The tasks stop restarting, the ALB 502 count falls to zero within two minutes, and the SLO burn rate decays.
7. Prove the fix and write it down
Verify with the same metrics that showed the problem: 502 count at zero for 15 minutes, availability SLI back above the goal, no stopped tasks with exit code 137. Then the root cause sentence: "Revision 47 halved the container memory limit to 512 MiB through a pipeline parameter typo; under normal load the .NET process exceeded it, ECS killed tasks with exit code 137, and the ALB returned 502 for requests in flight on those tasks." The permanent fixes (validate resource parameters in the pipeline, alarm on stopped tasks with exit 137, add a deployment circuit breaker) go into the review.
AWS Console instructions
- ALB metrics: Console → EC2 → Load Balancers → your ALB → Monitoring. Compare
HTTPCode_ELB_5XX_CountwithHTTPCode_Target_5XX_Count,TargetResponseTime,UnHealthyHostCountandHealthyHostCount. - Target health: EC2 → Target Groups → your group → Targets (health status and reason codes).
- ECS events and deployments: ECS → Clusters → service → Events and Deployments tabs. Stopped tasks: Tasks tab → filter Stopped → open a task → Stopped reason and container Exit code. Stopped tasks are retained only for a limited time, so capture them early. ⚠ VERIFY retention window.
- Task definition diff: ECS → Task definitions → family → compare revisions (CPU, memory, environment, image).
- Logs Insights: CloudWatch → Logs → Logs Insights, with your saved queries.
- What changed: CloudTrail → Event history, filter by
EventSource = ecs.amazonaws.com; AWS Config → Resources → timeline, if Config is enabled. - Health: the AWS Health Dashboard for account events affecting your services. ⚠ VERIFY console path.
AWS CLI
# 0) Is it ALB-generated or target-generated? Sum of each over the last 30 minutes, per minute.
for m in HTTPCode_ELB_5XX_Count HTTPCode_Target_5XX_Count HTTPCode_ELB_502_Count HTTPCode_ELB_503_Count HTTPCode_ELB_504_Count; do
echo "== $m"
aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name $m \
--dimensions Name=LoadBalancer,Value=app/hl-lab-alb/0123456789abcdef \
--start-time $(date -u -d '-30 minutes' +%Y-%m-%dT%H:%M:%SZ) --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
--period 60 --statistics Sum --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text \
--profile learn-sandbox --region us-east-1
done
# 1) Scope: errors grouped by task id and route (structured logs carry both).
aws logs start-query --log-group-name /hl/payments/api \
--start-time $(date -u -d '-30 minutes' +%s) --end-time $(date -u +%s) \
--query-string 'filter level = "Error" or statusCode >= 500
| stats count() as errors by taskId, route | sort errors desc | limit 20' \
--profile learn-sandbox --region us-east-1
# (then) aws logs get-query-results --query-id <id> --profile learn-sandbox --region us-east-1
# 2) What changed? Recent deployments and service events.
aws ecs describe-services --cluster hl-lab --services hl-payments-api \
--query 'services[0].{deployments:deployments[].{id:id,status:status,rollout:rolloutState,taskDef:taskDefinition,created:createdAt},events:events[0:10].[createdAt,message]}' \
--profile learn-sandbox --region us-east-1
# 3) Who made the change? (event history, management events)
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=UpdateService \
--max-results 5 --profile learn-sandbox --region us-east-1
# 4) Why did tasks stop?
STOPPED=$(aws ecs list-tasks --cluster hl-lab --service-name hl-payments-api --desired-status STOPPED \
--query 'taskArns' --output text --profile learn-sandbox --region us-east-1)
aws ecs describe-tasks --cluster hl-lab --tasks $STOPPED \
--query 'tasks[].{stoppedAt:stoppedAt,reason:stoppedReason,exit:containers[0].exitCode,taskDef:taskDefinitionArn}' \
--profile learn-sandbox --region us-east-1
# 5) Compare task definition revisions (memory, cpu, env).
aws ecs describe-task-definition --task-definition hl-payments-api:46 \
--query 'taskDefinition.{cpu:cpu,mem:memory,cont:containerDefinitions[].{n:name,mem:memory,memRes:memoryReservation}}' \
--profile learn-sandbox --region us-east-1
aws ecs describe-task-definition --task-definition hl-payments-api:47 \
--query 'taskDefinition.{cpu:cpu,mem:memory,cont:containerDefinitions[].{n:name,mem:memory,memRes:memoryReservation}}' \
--profile learn-sandbox --region us-east-1
# 6) Mitigate: roll back to the last known-good revision (record the time in the incident note!).
aws ecs update-service --cluster hl-lab --service hl-payments-api \
--task-definition hl-payments-api:46 --profile learn-sandbox --region us-east-1
# 7) Prove it: same query as step 0, expect zeros, plus stable running count.
aws ecs describe-services --cluster hl-lab --services hl-payments-api \
--query 'services[0].{desired:desiredCount,running:runningCount,pending:pendingCount}' \
--profile learn-sandbox --region us-east-1Infrastructure as Code
A triage dashboard gives the on-call engineer the first four graphs of the investigation on one page: ALB 5xx split by origin, target health, latency, and ECS memory with stopped-task counts. Build it as code so it exists before the incident. Dimension strings are inputs, because they differ per environment.
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Constructs;
namespace Harbourline.Monitoring.Infra;
public sealed class TriageDashboardStack : Stack
{
public TriageDashboardStack(
Construct scope, string id,
string loadBalancerDimension, // e.g. app/hl-lab-alb/0123456789abcdef
string targetGroupDimension, // e.g. targetgroup/hl-payments/0123456789abcdef
string clusterName, string serviceName,
IStackProps? props = null) : base(scope, id, props)
{
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m11-l12");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Metric Alb(string name, string stat = "Sum", string? label = null) => new(new MetricProps
{
Namespace = "AWS/ApplicationELB",
MetricName = name,
Statistic = stat,
Period = Duration.Minutes(1),
Label = label ?? name,
DimensionsMap = new Dictionary<string, string> { ["LoadBalancer"] = loadBalancerDimension },
});
Metric AlbTg(string name, string stat = "Sum") => new(new MetricProps
{
Namespace = "AWS/ApplicationELB",
MetricName = name,
Statistic = stat,
Period = Duration.Minutes(1),
DimensionsMap = new Dictionary<string, string>
{
["LoadBalancer"] = loadBalancerDimension,
["TargetGroup"] = targetGroupDimension,
},
});
Metric Ecs(string name, string stat = "Average") => new(new MetricProps
{
Namespace = "AWS/ECS",
MetricName = name,
Statistic = stat,
Period = Duration.Minutes(1),
DimensionsMap = new Dictionary<string, string>
{
["ClusterName"] = clusterName,
["ServiceName"] = serviceName,
},
});
new Dashboard(this, "Triage", new DashboardProps
{
DashboardName = "hl-m11-triage",
Widgets = new[]
{
new IWidget[]
{
new GraphWidget(new GraphWidgetProps
{
Title = "5xx by origin: ALB-generated vs application-generated",
Left = new IMetric[]
{
Alb("HTTPCode_ELB_5XX_Count", label: "ALB-generated 5xx"),
Alb("HTTPCode_Target_5XX_Count", label: "Target 5xx"),
Alb("HTTPCode_ELB_502_Count", label: "502"),
Alb("HTTPCode_ELB_503_Count", label: "503"),
Alb("HTTPCode_ELB_504_Count", label: "504"),
},
Width = 12, Height = 6,
}),
new GraphWidget(new GraphWidgetProps
{
Title = "Target health",
Left = new IMetric[] { AlbTg("HealthyHostCount", "Minimum"), AlbTg("UnHealthyHostCount", "Maximum") },
Width = 12, Height = 6,
}),
},
new IWidget[]
{
new GraphWidget(new GraphWidgetProps
{
Title = "Latency (target response time)",
Left = new IMetric[]
{
Alb("TargetResponseTime", "p50", "p50"),
Alb("TargetResponseTime", "p99", "p99"),
},
Width = 12, Height = 6,
}),
new GraphWidget(new GraphWidgetProps
{
Title = "ECS CPU and memory (saturation)",
Left = new IMetric[] { Ecs("CPUUtilization"), Ecs("MemoryUtilization") },
LeftYAxis = new YAxisProps { Min = 0, Max = 100 },
Width = 12, Height = 6,
}),
},
new IWidget[]
{
new LogQueryWidget(new LogQueryWidgetProps
{
Title = "Errors by task and route (last hour)",
LogGroupNames = new[] { "/hl/payments/api" },
QueryString =
"filter level = \"Error\" | stats count() as errors by taskId, route | sort errors desc | limit 20",
Width = 24, Height = 6,
}),
},
},
});
}
}Complete code
A small evidence-gathering tool for the first ten minutes: it prints the ALB 5xx split, the recent ECS deployments and service events, and the reasons for recently stopped tasks, all on one page. It uses AWS SDK for .NET v4 and the default credential chain.
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<!-- Look up exact 4.x versions at https://api.nuget.org/v3-flatcontainer/<id>/index.json -->
<PackageReference Include="AWSSDK.CloudWatch" Version="4.*" />
<PackageReference Include="AWSSDK.ECS" Version="4.*" />
</ItemGroup>
</Project>using Amazon;
using Amazon.CloudWatch;
using Amazon.CloudWatch.Model;
using Amazon.ECS;
using Amazon.ECS.Model;
// Usage: dotnet run -- <cluster> <service> <alb-dimension> [minutes=30]
// e.g. dotnet run -- hl-lab hl-payments-api app/hl-lab-alb/0123456789abcdef 30
if (args.Length < 3)
{
Console.Error.WriteLine("Usage: Triage <cluster> <service> <alb-dimension> [minutes=30]");
return 1;
}
var (cluster, service, albDimension) = (args[0], args[1], args[2]);
var minutes = args.Length > 3 && int.TryParse(args[3], out var m) ? m : 30;
var region = RegionEndpoint.USEast1;
using var cw = new AmazonCloudWatchClient(region);
using var ecs = new AmazonECSClient(region);
Console.WriteLine($"=== TRIAGE {DateTime.UtcNow:u} cluster={cluster} service={service} window={minutes}m ===\n");
// ---- 1. Is it the load balancer or the app? ----
Console.WriteLine("1) 5xx by origin (sum over window)");
foreach (var metric in new[]
{
"HTTPCode_ELB_5XX_Count", "HTTPCode_Target_5XX_Count",
"HTTPCode_ELB_502_Count", "HTTPCode_ELB_503_Count", "HTTPCode_ELB_504_Count",
})
{
var res = await cw.GetMetricDataAsync(new GetMetricDataRequest
{
StartTime = DateTime.UtcNow.AddMinutes(-minutes),
EndTime = DateTime.UtcNow,
MetricDataQueries =
[
new MetricDataQuery
{
Id = "q1",
ReturnData = true,
MetricStat = new MetricStat
{
Period = 60,
Stat = "Sum",
Metric = new Amazon.CloudWatch.Model.Metric
{
Namespace = "AWS/ApplicationELB",
MetricName = metric,
Dimensions = [new Amazon.CloudWatch.Model.Dimension { Name = "LoadBalancer", Value = albDimension }],
},
},
},
],
});
var values = res.MetricDataResults?.FirstOrDefault()?.Values ?? [];
Console.WriteLine($" {metric,-28} total={values.Sum():F0} worst-minute={(values.Count == 0 ? 0 : values.Max()):F0}");
}
// ---- 2. What changed? ----
Console.WriteLine("\n2) Deployments and recent service events");
var svc = (await ecs.DescribeServicesAsync(new DescribeServicesRequest
{
Cluster = cluster,
Services = [service],
})).Services?.FirstOrDefault();
if (svc is null)
{
Console.WriteLine(" Service not found. Check cluster and service names.");
return 2;
}
Console.WriteLine($" desired={svc.DesiredCount} running={svc.RunningCount} pending={svc.PendingCount}");
foreach (var d in (svc.Deployments ?? []).OrderByDescending(d => d.CreatedAt))
{
Console.WriteLine($" deployment {d.Id}: {d.Status} rollout={d.RolloutState} taskdef={d.TaskDefinition?.Split('/').LastOrDefault()} created={d.CreatedAt:u}");
}
foreach (var e in (svc.Events ?? []).Take(8))
{
Console.WriteLine($" {e.CreatedAt:HH:mm:ss} {e.Message}");
}
// ---- 3. Why did tasks stop? ----
Console.WriteLine("\n3) Recently stopped tasks");
var stopped = await ecs.ListTasksAsync(new ListTasksRequest
{
Cluster = cluster,
ServiceName = service,
DesiredStatus = DesiredStatus.STOPPED,
});
if ((stopped.TaskArns ?? []).Count == 0)
{
Console.WriteLine(" none retained (stopped tasks are kept only briefly, so absence is not proof)");
}
else
{
var described = await ecs.DescribeTasksAsync(new DescribeTasksRequest
{
Cluster = cluster,
Tasks = stopped.TaskArns,
});
foreach (var t in (described.Tasks ?? []).OrderByDescending(t => t.StoppedAt).Take(10))
{
var exits = string.Join(",", (t.Containers ?? []).Select(c => c.ExitCode?.ToString() ?? "-"));
Console.WriteLine($" {t.StoppedAt:HH:mm:ss} exit=[{exits}] {t.StoppedReason} ({t.TaskDefinitionArn?.Split('/').LastOrDefault()})");
}
Console.WriteLine("\n Hint: exit 137 with 'OutOfMemoryError' means the container hit its memory limit.");
}
return 0;Code explained line by line
- The tuple deconstruction
var (cluster, service, albDimension) = (args[0], ...)gives the three required arguments readable names. - Credentials come from the default chain (SSO profile locally). The tool is read-only: it calls only
GetMetricData,DescribeServices,ListTasksandDescribeTasks, which map to a small read-only IAM policy. GetMetricDataAsyncis preferred over the olderGetMetricStatisticsfor new code: it supports several queries per call and metric math. EachMetricDataQueryneeds anId; here there is one query per metric to keep the output readable.- Fully qualified
Amazon.CloudWatch.Model.MetricandDimensionavoid ambiguity with otherMetric-like types in your project. ?? []guards the SDK v4 behaviour where list properties can be null when absent, instead of empty.- Sorting deployments by
CreatedAtshows the newest first. TwoPRIMARY/ACTIVEdeployments side by side usually means a rollout is in progress. - Service
Eventsare the short human-readable log ECS keeps ("service ... has reached a steady state", "task failed to start"). They are limited in length and age, so capture them early. DesiredStatus.STOPPEDlists tasks ECS still remembers. The tool printsStoppedReasonand each container'sExitCode; exit 137 is the signature of a SIGKILL, commonly OOM.- The final hint line is deliberately a hypothesis helper, not a verdict.
Security considerations
⚠ Learning shortcut — not for production
The lab runs the tool with your broad sandbox role and prints raw StoppedReason and event text to the terminal. In production, give responders a dedicated read-only incident role (metrics, logs, ECS describe, CloudTrail lookup) with MFA and session logging, and keep incident notes in a system with access control, because they contain operational detail and sometimes customer data.
- Break-glass access. Responders often need broader rights than their daily role. Use a time-limited, logged elevation process, not shared admin credentials, and review each use afterwards.
- Do not paste customer data into incident chat or tickets. Use correlation ids and fingerprints instead of names, IBANs or account numbers.
- Keep the timeline. Timestamps of actions are evidence for the post-incident review and for any regulator conversation; they are also a record of who changed what during the incident.
- Rollback is a change. A rollback can reintroduce a known vulnerability or an old schema assumption. Check that the rolled-back version is compatible with the current database schema before you do it.
- Avoid "fixing" with wider permissions or open security groups during an incident. Temporary exposures become permanent ones.
Banking use case
Harbourline treats troubleshooting as a skill with standard artefacts:
| Artefact | Content | Owner |
|---|---|---|
| Incident note | Symptom sentence, timeline of observations and actions with timestamps, current hypothesis list | Incident lead |
| Triage dashboard | The six graphs and the errors-by-task query, pre-built | Platform team |
| "What changed?" checklist | Pipeline deploys, CloudTrail infra changes, feature flags, traffic shape, partner status | On-call |
| Evidence tool | The C# triage command, version-controlled | Platform team |
| Hypothesis log | Each hypothesis with its test and result (supported, disproved, unknown) | Incident lead |
Two rules the team repeats: "mitigate first, understand properly after" when customers are affected and a safe mitigation exists (rollback, scale, fail over); and "one change at a time, recorded" so the effect of each action can be attributed.
Common mistakes
- Starting with the favourite theory. "It's always the database." Rank hypotheses and test the cheapest discriminating one first.
- Changing two things at once. You will not know which one helped, and you might hide a cause.
- Skipping "what changed?". Most incidents follow a deployment, a config change, a certificate expiry, a quota limit or a traffic change.
- Confusing ALB 5xx with target 5xx. They point to different layers.
- Reading averages. Averages hide the 3 of 8 tasks that are failing. Use percentiles and group by instance.
- Treating absence of data as health. Missing metrics may mean the thing that publishes them is dead.
- No written timeline. The review will rely on memory.
- Declaring victory early. Verify with the same evidence that revealed the problem, for long enough to cover a full traffic cycle.
Troubleshooting
This table is itself a quick triage index for common ALB and ECS symptoms.
| Symptom | Likely cause | Where to look first |
|---|---|---|
| 502 spikes aligned with task restarts | Tasks killed (OOM, failed health checks, deployments with no connection draining) | ECS stopped reasons, memory metric, deregistration_delay and stopTimeout |
503 with HealthyHostCount = 0 | No healthy targets: failing health check, bad deployment, no capacity | Target health reason codes, ECS events, health check path and port |
| 504 on slow routes | Dependency slow or saturated; app slower than ALB idle timeout | Traces, thread-pool queue, database connections |
| 500s from the target | Application exception | Logs by fingerprint and release |
| Errors only on some tasks | Bad instance, noisy neighbour, partial deployment | Group by task id and image tag |
| Errors only in one AZ | AZ-specific dependency or subnet issue | Group by AZ, check subnet routes, NAT, endpoints |
| New deployment stuck, tasks restart in a loop | Failing container start (missing secret, bad image, wrong port) | describe-tasks stopped reason; container logs; enable the deployment circuit breaker |
describe-tasks returns nothing for stopped tasks | Retention window elapsed | Use logs and ECS task state change events captured by EventBridge |
| CloudWatch graph shows gaps | No data points published, or period too short | Widen period, check publisher, treat missing as unknown |
| Query returns no rows | Wrong log group, time window or field names | Run fields @message with limit 5 to see what exists |
Cost and cleanup
| Item | Label | Estimate |
|---|---|---|
| The triage tool and CLI commands | free | GetMetricData and ECS describe calls are inexpensive; check the pricing page for metric API request charges ⚠ VERIFY |
| One CloudWatch dashboard | credits | Custom dashboards are billed per dashboard per month beyond the free allowance; delete after the lab ⚠ VERIFY price (commonly $3 per dashboard per month) |
| Reproducing the incident (Fargate tasks for one hour) | credits | A few cents for two small tasks ⚠ VERIFY |
Free alternative: run the investigation as a paper exercise using the scenario above, with the commands' outputs invented by you or supplied by your instructor, and do the C# tool and dashboard design without deploying. Or reproduce a smaller version locally with Docker: run the API with --memory 128m so Docker kills it for OOM, then read docker inspect for OOMKilled and exit code 137.
# Delete the dashboard and any lab stack
aws cloudwatch delete-dashboards --dashboard-names hl-m11-triage --profile learn-sandbox --region us-east-1
cdk destroy TriageDashboardStack --profile learn-sandbox # or: terraform destroy
# Confirm nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1 --query 'ResourceTagMappingList[].ResourceARN'Exercises
- 🟢 Guided: Take the worked investigation and rewrite it as a hypothesis log: four hypotheses, the test for each, the observed result and the verdict. Add the exact CLI command for each test.
- 🟡 Challenge: Extend the C# triage tool with a fourth section that prints
MemoryUtilizationandCPUUtilization(ECS service metrics) for the window, flagging any minute above 90%. Then add a--jsonflag that writes the whole report as a single JSON document for the incident note. - 🔴 Live problem: Ticket: "Errors 502 and 504 started at 14:05 and stopped by themselves at 14:20; no deployments." You have 30 minutes. Write the order in which you would check ALB metrics, target health history, ECS events, CloudTrail, RDS events, AWS Health and the partner status page, and what each result would rule in or out. Then write the root-cause sentence you would accept as sufficient to close the ticket, and one you would reject.
Assignment
Create a "first ten minutes" runbook page for the payments API: the symptom sentence template, the five scoping queries, the what-changed checklist, a hypothesis table pre-filled with the five most common causes for your stack (bad deploy, resource limit, dependency saturation, certificate or quota expiry, traffic spike), and the rollback command with its safety checks. Commit it next to the dashboard and the tool.
Solutions
A good log reads: H1 code defect, test: logs grouped by fingerprint and release, result: no new fingerprint, verdict: unlikely. H2 resource change kills tasks, test: describe-tasks stopped reasons plus memory metric, result: exit 137 on four tasks and memory at 100%, verdict: supported. H3 database saturation, test: RDS DatabaseConnections and trace spans, result: 40% of limit, normal spans, verdict: disproved. H4 partner latency, test: dependency metrics and 504 count, result: normal, verdict: disproved. Each verdict cites evidence, not opinion. Notice that H2 was tested because the scoping step showed errors concentrated on a few tasks, and the what-changed step showed a deployment.
Add a section that calls GetMetricDataAsync for namespace AWS/ECS, metric names MemoryUtilization and CPUUtilization, dimensions ClusterName and ServiceName, statistic Maximum, period 60. Loop over Timestamps and Values and print rows where the value exceeds 90. For --json, collect the sections into an anonymous object and call System.Text.Json.JsonSerializer.Serialize(report, new JsonSerializerOptions { WriteIndented = true }). Keep the default human-readable output unchanged.
Knowledge check
Answer: The load balancer itself generated the errors (no healthy target, target closed the connection, timeout), not your application returning 5xx. Look at target health, ECS stops and timeouts rather than application exceptions.
Answer: RED (rate, errors, duration) looks at services from the request viewpoint; USE (utilisation, saturation, errors) looks at resources. RED finds which service is unhealthy, USE finds which resource limits it.
Answer: Most production incidents follow a change (deployment, configuration, certificate, quota, traffic), so the change list is the cheapest, highest-probability source of hypotheses.
Answer: The container exceeded its memory limit and was killed (SIGKILL, 128 plus 9). Check task and container memory settings and the application's memory use.
Answer: When customers are affected and a low-risk mitigation exists (rollback, scale, failover), restoring service first limits impact; the root cause analysis then proceeds with less pressure, using the recorded evidence. Mitigation must itself be a single, recorded, reversible change.
Interview questions
- Walk me through how you would investigate a spike in 5xx errors. Frame the symptom, scope by route, instance, AZ and release, check what changed, list ranked hypotheses, test cheapest discriminating evidence first, mitigate with a rollback or scale if safe, verify with the same metrics, document root cause and prevention.
- How do you distinguish a slow database from thread-pool starvation? Compare database metrics and trace spans against CPU, thread-pool queue length and blocked stacks; the database shows long query spans, starvation shows time spent waiting for a thread.
- What do you do when the logs are empty during an incident? Treat it as a finding: check whether the logger, the agent or the task is dead, whether log group names or permissions changed, and use other signals (metrics, ALB access logs, traces).
- How would you prevent this class of incident recurring? Pipeline validation of resource parameters, deployment circuit breaker with rollback, alarms on stopped tasks and exit codes, and a canary on the payment flow.
Certification practice
Users report intermittent 502 errors from an application behind an ALB. CloudWatch shows HTTPCode_ELB_502_Count rising while HTTPCode_Target_5XX_Count is zero. ECS tasks restart several times an hour. What should the operations engineer check first?
A. The ALB security group outbound rules. B. The stopped-task reasons and container exit codes for the ECS service. C. The Route 53 health check. D. The RDS read replica lag.
Answer: B. 502s with zero target 5xx and restarting tasks suggest connections closed by killed tasks. A, C and D relate to different layers and do not explain restarts.
After a deployment, an API returns errors. A previous task definition revision is known to be healthy. What is the BEST immediate action to restore service while preserving evidence?
A. Delete the cluster and recreate it. B. Update the service to use the previous task definition revision, then investigate using retained logs and the stopped-task records. C. Increase the desired count to 100. D. Disable CloudTrail to reduce noise.
Answer: B. Roll back to a known-good revision with a single recorded change, and keep evidence for analysis. A destroys evidence. C may make it worse. D removes the audit trail.
A developer sees 504 responses from an ALB for requests that call a slow downstream API. Which metric combination best confirms that targets are slow rather than unavailable?
A. HTTPCode_ELB_504_Count rising with high TargetResponseTime and healthy HealthyHostCount.
B. HealthyHostCount at zero.
C. RejectedConnectionCount only.
D. S3 BucketSizeBytes.
Answer: A. 504 with healthy targets and long response times indicates slow targets. B points to 503. C relates to connection limits on the load balancer. D is unrelated.
Completion checklist
- I can recite the seven steps and say why each exists
- I can tell ALB-generated 5xx from target-generated 5xx, and 502, 503 and 504 from each other
- I worked through the 5xx scenario and wrote a one-sentence root cause
- I built (or designed) the triage dashboard and ran or read the triage tool
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- Method beats instinct under pressure: frame, scope, what changed, hypothesise, test, mitigate, prove.
- Group by instance, route, AZ and release before theorising; concentration is a clue.
- The ALB splits its own 5xx from your app's 5xx; start there.
- Mitigate with one recorded, reversible change; then fix properly.
- Pre-build the dashboard, the queries and the evidence tool, because you will not build them during the incident.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Performance bottlenecks in .NET on AWS: thread pool, connection pools, cold starts and N+1
Recognise the four classic .NET performance failures on AWS from their symptoms, prove each with evidence from dotnet-counters, CloudWatch and traces, and apply the right fix.
Incident response and on-call runbooks: severity, roles, communication and reviews
Design the human side of operations: severity levels, incident roles, communication cadence, runbooks that an engineer can follow at 3 a.m., a semi-automated rollback with Systems Manager Automation, and a blameless post-incident review.