Skip to Content
Cloud Engineering Academy
11. Monitoring, Logging & Troubleshooting

Troubleshooting production issues methodically: symptom, hypothesis, evidence, fix

Follow a repeatable process from a vague symptom to a proven root cause, using a worked 5xx investigation on an ECS service behind an Application Load Balancer, and build the dashboard and evidence-gathering tool that make the next investigation faster.

advanced⏱ 1 h 50 min🟢 FreeSOA-C03DOP-C02DVA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Apply a seven-step troubleshooting process (frame, scope, what changed, hypotheses, evidence, mitigate, prove the fix) and explain why each step exists.
  • Tell an ALB-generated 5xx from a target-generated 5xx, and read 502, 503 and 504 as different clues.
  • Use the RED and USE methods to choose which metrics to look at first.
  • Work through a 5xx incident on ECS from the ALB down to a stopped task and a deployment, and write the root cause in one precise sentence.
  • Build a triage dashboard in code and a C# tool that gathers the first-ten-minutes evidence automatically.

Prerequisites

Why this matters in banking

At 10:42 the payments API starts returning errors to a few percent of requests. By 10:50 three people have three theories: "the database", "the partner", "the new release". Each person starts looking at a different dashboard, nobody writes anything down, and two engineers change things at the same time. By 11:30 the errors stop, and no one knows why. The post-incident review has nothing to say except "it recovered".

That outcome is the expensive one. It repeats, it hides real causes, and a regulator or internal audit will ask what you knew and when. A method does not make you smarter, but it stops you from skipping steps under pressure, makes the work shareable between people, and leaves a record that a post-incident review can use.

The idea in plain language

Troubleshooting is the scientific method on a clock. You observe, you guess, you test the guess against evidence, and you change one thing at a time. Most wasted hours come from three habits: fixing before understanding, changing several things at once, and trusting a theory instead of testing it.

The loop:

Analogy: a doctor does not operate because the patient says "I feel unwell". She asks when it started, checks vital signs, asks what changed (new medication?), lists likely causes, orders the test that best separates them, treats, then re-checks. The patient chart is the record.

What problem the AWS service solves

AWS does not troubleshoot for you, but it provides the instruments. The skill is knowing which instrument answers which question:

QuestionInstrument
Is the symptom real and how big is it?ALB/API Gateway metrics, Application Signals SLO status, canary results
Who is affected?Logs Insights by route, customer segment, Availability Zone, release
What changed?CloudTrail (infrastructure), ECS deployment history, CI/CD records, AWS Config timeline
Where is time or failure concentrated?Traces, service map, per-operation metrics
Is a resource saturated?CPU, memory, connections, queue depth, throttles, thread-pool queue
What does the failing component say?Application logs by fingerprint, ECS stopped-task reason, ALB access logs

Two frameworks help choose what to look at first:

  • RED for services: Rate (requests per second), Errors (failed fraction), Duration (latency distribution). Ask it of every service in the request path.
  • USE for resources: Utilisation, Saturation, Errors of each resource (CPU, memory, disks, network, connection pools, queues).

Rule of thumb: RED tells you which service hurts; USE tells you which resource of that service is the cause.

Honest limits: metrics lag (most are one-minute data points, some arrive several minutes late), logs can be sampled or dropped, traces are sampled, and CloudTrail events can arrive minutes after the call. During an incident treat "no data" as unknown, not as "fine".

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Application Insights "Failures" and "Performance" bladesApplication Signals service detail, Logs Insights over structured logsYou assemble the view from metrics, logs and traces
Azure Resource Health and Service HealthAWS Health Dashboard (account and public)Check it early to rule out a provider event ⚠ VERIFY current naming
Activity Log "what changed?"CloudTrail, AWS Config resource timeline, ECS deployment historyThree sources, each with different coverage
Azure Monitor "Diagnose and solve problems"Logs Insights saved queries, runbooks, CloudWatch investigations (where enabled)Treat any automated AI investigation as a hint, not as proof ⚠ VERIFY availability
Kusto queries over many tablesLogs Insights per log groupFewer joins; correlation uses ids you log
Azure Load Balancer / App Gateway diagnosticsALB metrics and access logsALB distinguishes its own 5xx from the target's 5xx
kubectl describe / App Service logsaws ecs describe-services, describe-tasks, stopped reasonECS records why a task stopped for a short time only

Architecture

The line between HTTPCode_ELB_5XX_Count (the load balancer itself produced the error: no healthy targets, target closed the connection, timeout) and HTTPCode_Target_5XX_Count (your application returned 5xx) is the first fork in most investigations.

ALB codeUsual meaningFirst place to look
502 Bad GatewayTarget closed or sent an invalid response; task killed mid-requestECS stopped-task reasons, application crashes, keep-alive timeouts
503 Service UnavailableNo registered healthy targetsTarget group health, task count, deployment in progress
504 Gateway TimeoutTarget did not respond before the ALB idle timeoutSlow dependency, thread-pool or connection-pool saturation
500 from targetApplication error (counted as target 5xx)Application logs by fingerprint

Step-by-step implementation: the worked investigation

The scenario is fictional but realistic. Harbourline's payments API runs on ECS Fargate behind an ALB. At 10:42 on a Wednesday the Application Signals availability SLO shows fast burn and the alarm pages the on-call engineer. The investigation follows the seven steps.

1. Frame the symptom

Write one sentence with facts only: "Since about 10:40 UTC, 3 to 6% of POST /payments requests return 5xx; reads look normal; the SLO burn rate is 40." Note who reported it, the first alert time and the severity. Do not write a cause. Open a shared incident note and start a timeline.

2. Scope it

Is it all routes or one? All Availability Zones or one? All customers or one segment? One release or all? Run one Logs Insights query grouped by route and one by Availability Zone or task id. Result in this case: only POST /payments, spread across both AZs, but concentrated on three of eight task ids. Concentration on a subset of instances is a strong clue: something is wrong with those instances, not with a shared dependency.

3. What changed?

Always ask first, because most incidents follow a change. Check the deployment history, infrastructure changes and traffic. The ECS service shows a deployment completed at 10:31: new task definition revision 47. CloudTrail confirms UpdateService called by the pipeline role at 10:29. Traffic is normal. The change is a prime suspect, but it is a suspect, not a verdict.

4. List hypotheses, ranked

#HypothesisPredictsCheapest test
H1Revision 47 has a code defect in the payments pathErrors on all tasks, a new fingerprintLogs by fingerprint and release
H2Revision 47 changed resources and tasks are being killedErrors on tasks that restart; stopped tasks with a kill reason; memory near the limitdescribe-tasks stopped reason; memory metric
H3The database is saturatedDatabaseConnections at max; slow query spansRDS metrics, trace spans
H4The partner API is slow504s with long partner spansDependency metrics

Order by likelihood and cost of test. Test the cheapest discriminating evidence first.

5. Gather evidence

H3 test: RDS connections at 40% of the limit and query latency normal. H3 disproved. H4 test: partner spans normal, no 504 spike. H4 disproved. H1 test: no new fingerprint in the logs; the errors are 502 from the ALB, not 500 from the app. H1 unlikely. H2 test: describe-tasks for stopped tasks shows stoppedReason "OutOfMemoryError: Container killed due to memory usage" with exit code 137 for four tasks in the last 15 minutes. ECS MemoryUtilization for the service climbs to 100% before each stop. The task definition diff between revisions 46 and 47 shows the memory limit changed from 1024 to 512 MiB in the pipeline's parameters. H2 supported.

6. Mitigate first, then fix

Roll the service back to revision 46 (update-service --task-definition ...:46). Mitigation is not the fix; it stops the bleeding with the least risky action. Record the time. The tasks stop restarting, the ALB 502 count falls to zero within two minutes, and the SLO burn rate decays.

7. Prove the fix and write it down

Verify with the same metrics that showed the problem: 502 count at zero for 15 minutes, availability SLI back above the goal, no stopped tasks with exit code 137. Then the root cause sentence: "Revision 47 halved the container memory limit to 512 MiB through a pipeline parameter typo; under normal load the .NET process exceeded it, ECS killed tasks with exit code 137, and the ALB returned 502 for requests in flight on those tasks." The permanent fixes (validate resource parameters in the pipeline, alarm on stopped tasks with exit 137, add a deployment circuit breaker) go into the review.

AWS Console instructions

  • ALB metrics: Console → EC2 → Load Balancers → your ALB → Monitoring. Compare HTTPCode_ELB_5XX_Count with HTTPCode_Target_5XX_Count, TargetResponseTime, UnHealthyHostCount and HealthyHostCount.
  • Target health: EC2 → Target Groups → your group → Targets (health status and reason codes).
  • ECS events and deployments: ECS → Clusters → service → Events and Deployments tabs. Stopped tasks: Tasks tab → filter Stopped → open a task → Stopped reason and container Exit code. Stopped tasks are retained only for a limited time, so capture them early. ⚠ VERIFY retention window.
  • Task definition diff: ECS → Task definitions → family → compare revisions (CPU, memory, environment, image).
  • Logs Insights: CloudWatch → Logs → Logs Insights, with your saved queries.
  • What changed: CloudTrail → Event history, filter by EventSource = ecs.amazonaws.com; AWS Config → Resources → timeline, if Config is enabled.
  • Health: the AWS Health Dashboard for account events affecting your services. ⚠ VERIFY console path.

AWS CLI

# 0) Is it ALB-generated or target-generated? Sum of each over the last 30 minutes, per minute.
for m in HTTPCode_ELB_5XX_Count HTTPCode_Target_5XX_Count HTTPCode_ELB_502_Count HTTPCode_ELB_503_Count HTTPCode_ELB_504_Count; do
  echo "== $m"
  aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name $m \
    --dimensions Name=LoadBalancer,Value=app/hl-lab-alb/0123456789abcdef \
    --start-time $(date -u -d '-30 minutes' +%Y-%m-%dT%H:%M:%SZ) --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
    --period 60 --statistics Sum --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text \
    --profile learn-sandbox --region us-east-1
done

# 1) Scope: errors grouped by task id and route (structured logs carry both).
aws logs start-query --log-group-name /hl/payments/api \
  --start-time $(date -u -d '-30 minutes' +%s) --end-time $(date -u +%s) \
  --query-string 'filter level = "Error" or statusCode >= 500
    | stats count() as errors by taskId, route | sort errors desc | limit 20' \
  --profile learn-sandbox --region us-east-1
# (then) aws logs get-query-results --query-id <id> --profile learn-sandbox --region us-east-1

# 2) What changed? Recent deployments and service events.
aws ecs describe-services --cluster hl-lab --services hl-payments-api \
  --query 'services[0].{deployments:deployments[].{id:id,status:status,rollout:rolloutState,taskDef:taskDefinition,created:createdAt},events:events[0:10].[createdAt,message]}' \
  --profile learn-sandbox --region us-east-1

# 3) Who made the change? (event history, management events)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateService \
  --max-results 5 --profile learn-sandbox --region us-east-1

# 4) Why did tasks stop?
STOPPED=$(aws ecs list-tasks --cluster hl-lab --service-name hl-payments-api --desired-status STOPPED \
  --query 'taskArns' --output text --profile learn-sandbox --region us-east-1)
aws ecs describe-tasks --cluster hl-lab --tasks $STOPPED \
  --query 'tasks[].{stoppedAt:stoppedAt,reason:stoppedReason,exit:containers[0].exitCode,taskDef:taskDefinitionArn}' \
  --profile learn-sandbox --region us-east-1

# 5) Compare task definition revisions (memory, cpu, env).
aws ecs describe-task-definition --task-definition hl-payments-api:46 \
  --query 'taskDefinition.{cpu:cpu,mem:memory,cont:containerDefinitions[].{n:name,mem:memory,memRes:memoryReservation}}' \
  --profile learn-sandbox --region us-east-1
aws ecs describe-task-definition --task-definition hl-payments-api:47 \
  --query 'taskDefinition.{cpu:cpu,mem:memory,cont:containerDefinitions[].{n:name,mem:memory,memRes:memoryReservation}}' \
  --profile learn-sandbox --region us-east-1

# 6) Mitigate: roll back to the last known-good revision (record the time in the incident note!).
aws ecs update-service --cluster hl-lab --service hl-payments-api \
  --task-definition hl-payments-api:46 --profile learn-sandbox --region us-east-1

# 7) Prove it: same query as step 0, expect zeros, plus stable running count.
aws ecs describe-services --cluster hl-lab --services hl-payments-api \
  --query 'services[0].{desired:desiredCount,running:runningCount,pending:pendingCount}' \
  --profile learn-sandbox --region us-east-1

Infrastructure as Code

A triage dashboard gives the on-call engineer the first four graphs of the investigation on one page: ALB 5xx split by origin, target health, latency, and ECS memory with stopped-task counts. Build it as code so it exists before the incident. Dimension strings are inputs, because they differ per environment.

TriageDashboardStack.cs
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Constructs;

namespace Harbourline.Monitoring.Infra;

public sealed class TriageDashboardStack : Stack
{
    public TriageDashboardStack(
        Construct scope, string id,
        string loadBalancerDimension,  // e.g. app/hl-lab-alb/0123456789abcdef
        string targetGroupDimension,   // e.g. targetgroup/hl-payments/0123456789abcdef
        string clusterName, string serviceName,
        IStackProps? props = null) : base(scope, id, props)
    {
        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m11-l12");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");

        Metric Alb(string name, string stat = "Sum", string? label = null) => new(new MetricProps
        {
            Namespace = "AWS/ApplicationELB",
            MetricName = name,
            Statistic = stat,
            Period = Duration.Minutes(1),
            Label = label ?? name,
            DimensionsMap = new Dictionary<string, string> { ["LoadBalancer"] = loadBalancerDimension },
        });

        Metric AlbTg(string name, string stat = "Sum") => new(new MetricProps
        {
            Namespace = "AWS/ApplicationELB",
            MetricName = name,
            Statistic = stat,
            Period = Duration.Minutes(1),
            DimensionsMap = new Dictionary<string, string>
            {
                ["LoadBalancer"] = loadBalancerDimension,
                ["TargetGroup"] = targetGroupDimension,
            },
        });

        Metric Ecs(string name, string stat = "Average") => new(new MetricProps
        {
            Namespace = "AWS/ECS",
            MetricName = name,
            Statistic = stat,
            Period = Duration.Minutes(1),
            DimensionsMap = new Dictionary<string, string>
            {
                ["ClusterName"] = clusterName,
                ["ServiceName"] = serviceName,
            },
        });

        new Dashboard(this, "Triage", new DashboardProps
        {
            DashboardName = "hl-m11-triage",
            Widgets = new[]
            {
                new IWidget[]
                {
                    new GraphWidget(new GraphWidgetProps
                    {
                        Title = "5xx by origin: ALB-generated vs application-generated",
                        Left = new IMetric[]
                        {
                            Alb("HTTPCode_ELB_5XX_Count", label: "ALB-generated 5xx"),
                            Alb("HTTPCode_Target_5XX_Count", label: "Target 5xx"),
                            Alb("HTTPCode_ELB_502_Count", label: "502"),
                            Alb("HTTPCode_ELB_503_Count", label: "503"),
                            Alb("HTTPCode_ELB_504_Count", label: "504"),
                        },
                        Width = 12, Height = 6,
                    }),
                    new GraphWidget(new GraphWidgetProps
                    {
                        Title = "Target health",
                        Left = new IMetric[] { AlbTg("HealthyHostCount", "Minimum"), AlbTg("UnHealthyHostCount", "Maximum") },
                        Width = 12, Height = 6,
                    }),
                },
                new IWidget[]
                {
                    new GraphWidget(new GraphWidgetProps
                    {
                        Title = "Latency (target response time)",
                        Left = new IMetric[]
                        {
                            Alb("TargetResponseTime", "p50", "p50"),
                            Alb("TargetResponseTime", "p99", "p99"),
                        },
                        Width = 12, Height = 6,
                    }),
                    new GraphWidget(new GraphWidgetProps
                    {
                        Title = "ECS CPU and memory (saturation)",
                        Left = new IMetric[] { Ecs("CPUUtilization"), Ecs("MemoryUtilization") },
                        LeftYAxis = new YAxisProps { Min = 0, Max = 100 },
                        Width = 12, Height = 6,
                    }),
                },
                new IWidget[]
                {
                    new LogQueryWidget(new LogQueryWidgetProps
                    {
                        Title = "Errors by task and route (last hour)",
                        LogGroupNames = new[] { "/hl/payments/api" },
                        QueryString =
                            "filter level = \"Error\" | stats count() as errors by taskId, route | sort errors desc | limit 20",
                        Width = 24, Height = 6,
                    }),
                },
            },
        });
    }
}

Complete code

A small evidence-gathering tool for the first ten minutes: it prints the ALB 5xx split, the recent ECS deployments and service events, and the reasons for recently stopped tasks, all on one page. It uses AWS SDK for .NET v4 and the default credential chain.

Triage.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
  </PropertyGroup>
  <ItemGroup>
    <!-- Look up exact 4.x versions at https://api.nuget.org/v3-flatcontainer/<id>/index.json -->
    <PackageReference Include="AWSSDK.CloudWatch" Version="4.*" />
    <PackageReference Include="AWSSDK.ECS" Version="4.*" />
  </ItemGroup>
</Project>
Program.cs
using Amazon;
using Amazon.CloudWatch;
using Amazon.CloudWatch.Model;
using Amazon.ECS;
using Amazon.ECS.Model;

// Usage: dotnet run -- <cluster> <service> <alb-dimension> [minutes=30]
//   e.g. dotnet run -- hl-lab hl-payments-api app/hl-lab-alb/0123456789abcdef 30
if (args.Length < 3)
{
    Console.Error.WriteLine("Usage: Triage <cluster> <service> <alb-dimension> [minutes=30]");
    return 1;
}

var (cluster, service, albDimension) = (args[0], args[1], args[2]);
var minutes = args.Length > 3 && int.TryParse(args[3], out var m) ? m : 30;
var region = RegionEndpoint.USEast1;

using var cw = new AmazonCloudWatchClient(region);
using var ecs = new AmazonECSClient(region);

Console.WriteLine($"=== TRIAGE {DateTime.UtcNow:u}  cluster={cluster} service={service} window={minutes}m ===\n");

// ---- 1. Is it the load balancer or the app? ----
Console.WriteLine("1) 5xx by origin (sum over window)");
foreach (var metric in new[]
{
    "HTTPCode_ELB_5XX_Count", "HTTPCode_Target_5XX_Count",
    "HTTPCode_ELB_502_Count", "HTTPCode_ELB_503_Count", "HTTPCode_ELB_504_Count",
})
{
    var res = await cw.GetMetricDataAsync(new GetMetricDataRequest
    {
        StartTime = DateTime.UtcNow.AddMinutes(-minutes),
        EndTime = DateTime.UtcNow,
        MetricDataQueries =
        [
            new MetricDataQuery
            {
                Id = "q1",
                ReturnData = true,
                MetricStat = new MetricStat
                {
                    Period = 60,
                    Stat = "Sum",
                    Metric = new Amazon.CloudWatch.Model.Metric
                    {
                        Namespace = "AWS/ApplicationELB",
                        MetricName = metric,
                        Dimensions = [new Amazon.CloudWatch.Model.Dimension { Name = "LoadBalancer", Value = albDimension }],
                    },
                },
            },
        ],
    });

    var values = res.MetricDataResults?.FirstOrDefault()?.Values ?? [];
    Console.WriteLine($"   {metric,-28} total={values.Sum():F0}  worst-minute={(values.Count == 0 ? 0 : values.Max()):F0}");
}

// ---- 2. What changed? ----
Console.WriteLine("\n2) Deployments and recent service events");
var svc = (await ecs.DescribeServicesAsync(new DescribeServicesRequest
{
    Cluster = cluster,
    Services = [service],
})).Services?.FirstOrDefault();

if (svc is null)
{
    Console.WriteLine("   Service not found. Check cluster and service names.");
    return 2;
}

Console.WriteLine($"   desired={svc.DesiredCount} running={svc.RunningCount} pending={svc.PendingCount}");
foreach (var d in (svc.Deployments ?? []).OrderByDescending(d => d.CreatedAt))
{
    Console.WriteLine($"   deployment {d.Id}: {d.Status} rollout={d.RolloutState} taskdef={d.TaskDefinition?.Split('/').LastOrDefault()} created={d.CreatedAt:u}");
}
foreach (var e in (svc.Events ?? []).Take(8))
{
    Console.WriteLine($"   {e.CreatedAt:HH:mm:ss}  {e.Message}");
}

// ---- 3. Why did tasks stop? ----
Console.WriteLine("\n3) Recently stopped tasks");
var stopped = await ecs.ListTasksAsync(new ListTasksRequest
{
    Cluster = cluster,
    ServiceName = service,
    DesiredStatus = DesiredStatus.STOPPED,
});

if ((stopped.TaskArns ?? []).Count == 0)
{
    Console.WriteLine("   none retained (stopped tasks are kept only briefly, so absence is not proof)");
}
else
{
    var described = await ecs.DescribeTasksAsync(new DescribeTasksRequest
    {
        Cluster = cluster,
        Tasks = stopped.TaskArns,
    });

    foreach (var t in (described.Tasks ?? []).OrderByDescending(t => t.StoppedAt).Take(10))
    {
        var exits = string.Join(",", (t.Containers ?? []).Select(c => c.ExitCode?.ToString() ?? "-"));
        Console.WriteLine($"   {t.StoppedAt:HH:mm:ss}  exit=[{exits}]  {t.StoppedReason}  ({t.TaskDefinitionArn?.Split('/').LastOrDefault()})");
    }

    Console.WriteLine("\n   Hint: exit 137 with 'OutOfMemoryError' means the container hit its memory limit.");
}

return 0;

Code explained line by line

  1. The tuple deconstruction var (cluster, service, albDimension) = (args[0], ...) gives the three required arguments readable names.
  2. Credentials come from the default chain (SSO profile locally). The tool is read-only: it calls only GetMetricData, DescribeServices, ListTasks and DescribeTasks, which map to a small read-only IAM policy.
  3. GetMetricDataAsync is preferred over the older GetMetricStatistics for new code: it supports several queries per call and metric math. Each MetricDataQuery needs an Id; here there is one query per metric to keep the output readable.
  4. Fully qualified Amazon.CloudWatch.Model.Metric and Dimension avoid ambiguity with other Metric-like types in your project.
  5. ?? [] guards the SDK v4 behaviour where list properties can be null when absent, instead of empty.
  6. Sorting deployments by CreatedAt shows the newest first. Two PRIMARY/ACTIVE deployments side by side usually means a rollout is in progress.
  7. Service Events are the short human-readable log ECS keeps ("service ... has reached a steady state", "task failed to start"). They are limited in length and age, so capture them early.
  8. DesiredStatus.STOPPED lists tasks ECS still remembers. The tool prints StoppedReason and each container's ExitCode; exit 137 is the signature of a SIGKILL, commonly OOM.
  9. The final hint line is deliberately a hypothesis helper, not a verdict.

Security considerations

⚠ Learning shortcut — not for production

The lab runs the tool with your broad sandbox role and prints raw StoppedReason and event text to the terminal. In production, give responders a dedicated read-only incident role (metrics, logs, ECS describe, CloudTrail lookup) with MFA and session logging, and keep incident notes in a system with access control, because they contain operational detail and sometimes customer data.

  • Break-glass access. Responders often need broader rights than their daily role. Use a time-limited, logged elevation process, not shared admin credentials, and review each use afterwards.
  • Do not paste customer data into incident chat or tickets. Use correlation ids and fingerprints instead of names, IBANs or account numbers.
  • Keep the timeline. Timestamps of actions are evidence for the post-incident review and for any regulator conversation; they are also a record of who changed what during the incident.
  • Rollback is a change. A rollback can reintroduce a known vulnerability or an old schema assumption. Check that the rolled-back version is compatible with the current database schema before you do it.
  • Avoid "fixing" with wider permissions or open security groups during an incident. Temporary exposures become permanent ones.

Banking use case

Harbourline treats troubleshooting as a skill with standard artefacts:

ArtefactContentOwner
Incident noteSymptom sentence, timeline of observations and actions with timestamps, current hypothesis listIncident lead
Triage dashboardThe six graphs and the errors-by-task query, pre-builtPlatform team
"What changed?" checklistPipeline deploys, CloudTrail infra changes, feature flags, traffic shape, partner statusOn-call
Evidence toolThe C# triage command, version-controlledPlatform team
Hypothesis logEach hypothesis with its test and result (supported, disproved, unknown)Incident lead

Two rules the team repeats: "mitigate first, understand properly after" when customers are affected and a safe mitigation exists (rollback, scale, fail over); and "one change at a time, recorded" so the effect of each action can be attributed.

Common mistakes

  • Starting with the favourite theory. "It's always the database." Rank hypotheses and test the cheapest discriminating one first.
  • Changing two things at once. You will not know which one helped, and you might hide a cause.
  • Skipping "what changed?". Most incidents follow a deployment, a config change, a certificate expiry, a quota limit or a traffic change.
  • Confusing ALB 5xx with target 5xx. They point to different layers.
  • Reading averages. Averages hide the 3 of 8 tasks that are failing. Use percentiles and group by instance.
  • Treating absence of data as health. Missing metrics may mean the thing that publishes them is dead.
  • No written timeline. The review will rely on memory.
  • Declaring victory early. Verify with the same evidence that revealed the problem, for long enough to cover a full traffic cycle.

Troubleshooting

This table is itself a quick triage index for common ALB and ECS symptoms.

SymptomLikely causeWhere to look first
502 spikes aligned with task restartsTasks killed (OOM, failed health checks, deployments with no connection draining)ECS stopped reasons, memory metric, deregistration_delay and stopTimeout
503 with HealthyHostCount = 0No healthy targets: failing health check, bad deployment, no capacityTarget health reason codes, ECS events, health check path and port
504 on slow routesDependency slow or saturated; app slower than ALB idle timeoutTraces, thread-pool queue, database connections
500s from the targetApplication exceptionLogs by fingerprint and release
Errors only on some tasksBad instance, noisy neighbour, partial deploymentGroup by task id and image tag
Errors only in one AZAZ-specific dependency or subnet issueGroup by AZ, check subnet routes, NAT, endpoints
New deployment stuck, tasks restart in a loopFailing container start (missing secret, bad image, wrong port)describe-tasks stopped reason; container logs; enable the deployment circuit breaker
describe-tasks returns nothing for stopped tasksRetention window elapsedUse logs and ECS task state change events captured by EventBridge
CloudWatch graph shows gapsNo data points published, or period too shortWiden period, check publisher, treat missing as unknown
Query returns no rowsWrong log group, time window or field namesRun fields @message with limit 5 to see what exists

Cost and cleanup

ItemLabelEstimate
The triage tool and CLI commandsfreeGetMetricData and ECS describe calls are inexpensive; check the pricing page for metric API request charges ⚠ VERIFY
One CloudWatch dashboardcreditsCustom dashboards are billed per dashboard per month beyond the free allowance; delete after the lab ⚠ VERIFY price (commonly $3 per dashboard per month)
Reproducing the incident (Fargate tasks for one hour)creditsA few cents for two small tasks ⚠ VERIFY

Free alternative: run the investigation as a paper exercise using the scenario above, with the commands' outputs invented by you or supplied by your instructor, and do the C# tool and dashboard design without deploying. Or reproduce a smaller version locally with Docker: run the API with --memory 128m so Docker kills it for OOM, then read docker inspect for OOMKilled and exit code 137.

# Delete the dashboard and any lab stack
aws cloudwatch delete-dashboards --dashboard-names hl-m11-triage --profile learn-sandbox --region us-east-1
cdk destroy TriageDashboardStack --profile learn-sandbox   # or: terraform destroy

# Confirm nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --profile learn-sandbox --region us-east-1 --query 'ResourceTagMappingList[].ResourceARN'

Exercises

  • 🟢 Guided: Take the worked investigation and rewrite it as a hypothesis log: four hypotheses, the test for each, the observed result and the verdict. Add the exact CLI command for each test.
  • 🟡 Challenge: Extend the C# triage tool with a fourth section that prints MemoryUtilization and CPUUtilization (ECS service metrics) for the window, flagging any minute above 90%. Then add a --json flag that writes the whole report as a single JSON document for the incident note.
  • 🔴 Live problem: Ticket: "Errors 502 and 504 started at 14:05 and stopped by themselves at 14:20; no deployments." You have 30 minutes. Write the order in which you would check ALB metrics, target health history, ECS events, CloudTrail, RDS events, AWS Health and the partner status page, and what each result would rule in or out. Then write the root-cause sentence you would accept as sufficient to close the ticket, and one you would reject.

Assignment

Create a "first ten minutes" runbook page for the payments API: the symptom sentence template, the five scoping queries, the what-changed checklist, a hypothesis table pre-filled with the five most common causes for your stack (bad deploy, resource limit, dependency saturation, certificate or quota expiry, traffic spike), and the rollback command with its safety checks. Commit it next to the dashboard and the tool.

Solutions

Knowledge check

Interview questions

  • Walk me through how you would investigate a spike in 5xx errors. Frame the symptom, scope by route, instance, AZ and release, check what changed, list ranked hypotheses, test cheapest discriminating evidence first, mitigate with a rollback or scale if safe, verify with the same metrics, document root cause and prevention.
  • How do you distinguish a slow database from thread-pool starvation? Compare database metrics and trace spans against CPU, thread-pool queue length and blocked stacks; the database shows long query spans, starvation shows time spent waiting for a thread.
  • What do you do when the logs are empty during an incident? Treat it as a finding: check whether the logger, the agent or the task is dead, whether log group names or permissions changed, and use other signals (metrics, ALB access logs, traces).
  • How would you prevent this class of incident recurring? Pipeline validation of resource parameters, deployment circuit breaker with rollback, alarms on stopped tasks and exit codes, and a canary on the payment flow.

Certification practice

Completion checklist

  • I can recite the seven steps and say why each exists
  • I can tell ALB-generated 5xx from target-generated 5xx, and 502, 503 and 504 from each other
  • I worked through the 5xx scenario and wrote a one-sentence root cause
  • I built (or designed) the triage dashboard and ran or read the triage tool
  • I ran the cleanup steps and confirmed nothing is still billing

Key takeaways

  • Method beats instinct under pressure: frame, scope, what changed, hypothesise, test, mitigate, prove.
  • Group by instance, route, AZ and release before theorising; concentration is a clue.
  • The ALB splits its own 5xx from your app's 5xx; start there.
  • Mitigate with one recorded, reversible change; then fix properly.
  • Pre-build the dashboard, the queries and the evidence tool, because you will not build them during the incident.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page