Performance bottlenecks in .NET on AWS: thread pool, connection pools, cold starts and N+1
Recognise the four classic .NET performance failures on AWS from their symptoms, prove each with evidence from dotnet-counters, CloudWatch and traces, and apply the right fix.
Assumes: .NET 10 LTS · EF Core 10 · Npgsql · AWS SDK for .NET v4 · dotnet-counters · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain thread-pool starvation, connection-pool exhaustion, cold starts and N+1 queries in plain language, and say what each looks like on a dashboard.
- Reproduce thread-pool starvation with sync-over-async code, diagnose it with
dotnet-counters, and fix it. - Calculate whether your connection pools, instance counts and database limits add up, and know when to add RDS Proxy.
- Measure Lambda cold starts from the
REPORTlog line and choose between memory tuning, ReadyToRun, Native AOT, SnapStart and provisioned concurrency. - Detect N+1 queries by counting database commands per request, and fix them with EF Core projections,
Includeor batching.
Prerequisites
- CloudWatch metrics and custom metrics, Alarms and composite alarms and Distributed tracing with OpenTelemetry and X-Ray for .NET (ADOT). You will read traces and metrics in this lesson.
- Database performance optimisation for index and query-plan background.
- Lambda with C# and Deploy to Fargate.
- Tools: .NET 10 SDK,
dotnet-counters(dotnet tool install --global dotnet-counters), Docker, AWS CLI v2.
Why this matters in banking
Salary day, 09:00. Payments traffic triples. Dashboards show CPU at 22% on every task, memory is fine, the database is idle, and yet the p99 latency of POST /payments is 14 seconds and the load balancer is returning 504s. Nothing is obviously broken, so the team scales out from 4 to 12 tasks. Latency does not improve, and now the database is rejecting connections.
This is not unusual. In .NET services the expensive failures are usually resource-shaped, not CPU-shaped: you run out of threads, or connections, or time, or round trips, long before you run out of CPU. Each looks different and each has a different fix. Scaling out is the wrong first move for two of the four.
The idea in plain language
Think of a branch with a fixed number of cashiers (threads), a fixed number of vault keys (database connections), a staff room where cashiers must wake up before serving (cold start) and a manager who sends a runner to the vault once per customer instead of once per hundred customers (N+1).
| Problem | Plain-language version | Typical symptom |
|---|---|---|
| Thread-pool starvation | All cashiers are standing at the vault door waiting; new customers queue even though the building is empty | High latency, low CPU, rising queue length, timeouts |
| Connection-pool exhaustion | All vault keys are out; everyone waits for a key | Errors like "timeout obtaining a connection from the pool", database connection count flat at the maximum |
| Cold start | The cashier has to wake up, unpack and log in before the first customer | First request after idle or scale-out is slow; the rest are fast |
| N+1 queries | One query for the list, then one more for each item in the list | Latency grows with the number of rows returned; many near-identical SQL statements per request |
Analogy: starvation is a bank where every cashier phones head office and waits on hold instead of serving the next customer. The phone line is free to answer other calls; they just do not hang up and come back. Async code is hanging up and calling back.
What problem the AWS service solves
This lesson is mostly about your code, but AWS gives you the evidence and some of the levers:
- CloudWatch metrics show symptoms: ECS
CPUUtilization/MemoryUtilization, ALBTargetResponseTimeandHTTPCode_Target_5XX_Count, RDSDatabaseConnections, LambdaDuration,ConcurrentExecutionsand theREPORTline'sInit Duration. - Tracing (X-Ray, OpenTelemetry, Application Signals) shows where a request spent its time: waiting for a connection looks very different from a slow query.
- RDS Proxy pools and shares database connections for functions and many small tasks.
- Lambda features attack cold starts: memory tuning, SnapStart for supported runtimes, provisioned concurrency.
Honest limits:
- SnapStart supports Java 11 and later, Python 3.12 and later and .NET 8 and later managed runtimes (and matching base images). It works on published versions and aliases only, not
$LATEST, and does not support provisioned concurrency, EFS, or ephemeral storage above 512 MB. There is a charge for snapshot caching and restoration for non-Java runtimes. ⚠ VERIFY current pricing and the exact .NET 10 statement; the documentation says ".NET 8 and later". - Lambda .NET 10 runtime exists as
dotnet10. At the time of checkingdotnet8had reached its deprecation date of 2026-11-10 for the managed runtime, so plan upgrades. ⚠ VERIFY dates. - No tool fixes bad code. Autoscaling a starved service adds more starved instances.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Application Insights Profiler / Snapshot Debugger | Traces + dotnet-counters/dotnet-trace against the container; CodeGuru Profiler is not covered here | You usually attach diagnostic tools yourself in a container |
| Azure Functions cold start, Premium plan pre-warmed instances | Lambda cold start, provisioned concurrency, SnapStart | Billing and tuning differ; SnapStart snapshots the initialised environment |
| Azure SQL max connections / elastic pools | RDS max_connections by instance class, RDS Proxy | Connection limits are set by instance memory and engine parameters |
| App Service "Always On" | ECS minimum task count, Lambda provisioned concurrency | Idle shutdown works differently per service |
| Azure Monitor metrics for thread pool via Application Insights performance counters | dotnet-counters, System.Runtime meter exported by OpenTelemetry | You must export runtime metrics yourself |
| Azure Load Testing | A load script, k6 or Locust against your sandbox, or AWS-published distributed load testing solutions | Bring your own tool |
Architecture
Where the limits sit: threads per process (thread pool), connections per process (client pool), connections per database (server limit), and time to first request (cold start). A request fails or slows at the smallest of them.
Step-by-step implementation
Establish a baseline with runtime counters
Start the API locally, run a normal load, and record the thread count, queue length, GC counts and request rate. Without a baseline you cannot tell abnormal from normal.
Reproduce thread-pool starvation
Run the deliberately bad endpoint that blocks on async work. Watch queue length climb while CPU stays low.
Diagnose with dotnet-counters
Attach to the process, read the thread pool counters, then capture stacks (dotnet-stack report) to see the threads blocked in .Result or .Wait().
Fix it, then prove the fix
Make the call async end to end and re-run the same load. The queue length should stay near zero.
Add the three guardrails
A background service that logs thread-pool stats, an EF Core interceptor that counts commands per request, and CloudWatch alarms on database connections and Lambda init duration.
Check the arithmetic for connections
Multiply tasks (or Lambda concurrency) by Max Pool Size; compare with the database limit. Decide whether you need smaller pools, fewer tasks or RDS Proxy.
AWS Console instructions
- ECS: Console → ECS → Clusters → your cluster → Services → your service → Metrics tab for CPU and memory; Logs for application output. Compare with ALB Target group → Monitoring for
TargetResponseTimeand 5xx. - RDS: Console → RDS → Databases → your instance → Monitoring →
DatabaseConnections,CPUUtilization,FreeableMemory. Performance Insights (if enabled) shows top SQL and wait events. ⚠ VERIFY: Performance Insights availability and pricing changed over time; check the current page. - Lambda: Console → Lambda → your function → Monitor → View CloudWatch logs. In Logs Insights, run the cold start query below. Configuration → General for memory; Versions/Aliases for SnapStart and provisioned concurrency.
- CloudWatch: Metrics → All metrics to graph
AWS/RDS DatabaseConnectionsand your custom thread-pool metric on one chart with latency.
AWS CLI
# 1) Local diagnosis (no AWS needed). Find the process, then watch runtime counters.
dotnet-counters ps
dotnet-counters monitor --process-id <PID> --refresh-interval 1 \
--counters System.Runtime[threadpool-thread-count,threadpool-queue-length,threadpool-completed-items-count,monitor-lock-contention-count,cpu-usage,gc-heap-size]
# 2) See what the blocked threads are doing (dotnet-stack is a separate global tool).
dotnet-stack report --process-id <PID> | head -80
# 3) RDS connection count for the last hour, 1-minute average.
aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name DatabaseConnections \
--dimensions Name=DBInstanceIdentifier,Value=hl-lab-postgres \
--start-time $(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%SZ) --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
--period 60 --statistics Maximum \
--profile learn-sandbox --region us-east-1
# 4) Lambda cold starts: how many in the last hour, and how long did INIT take?
aws logs start-query --log-group-name /aws/lambda/hl-payments-fn \
--start-time $(date -u -d '-1 hour' +%s) --end-time $(date -u +%s) \
--query-string 'filter @type = "REPORT"
| stats count() as invocations,
count(@initDuration) as coldStarts,
avg(@initDuration) as avgInitMs,
max(@initDuration) as maxInitMs,
pct(@duration, 99) as p99Ms,
max(@maxMemoryUsed / 1024 / 1024) as maxMemMB
by bin(5m)' \
--profile learn-sandbox --region us-east-1
# 5) Publish a version and turn on SnapStart for published versions (supported for .NET 8 and later).
aws lambda update-function-configuration --function-name hl-payments-fn \
--snap-start ApplyOn=PublishedVersions --profile learn-sandbox --region us-east-1
aws lambda wait function-updated --function-name hl-payments-fn --profile learn-sandbox --region us-east-1
aws lambda publish-version --function-name hl-payments-fn --profile learn-sandbox --region us-east-1
# 6) How many database commands did one request run? (find it in your own structured logs)
aws logs start-query --log-group-name /hl/payments/api \
--start-time $(date -u -d '-30 minutes' +%s) --end-time $(date -u +%s) \
--query-string 'filter category = "RequestCompleted"
| stats avg(dbCommandCount) as avgCmds, max(dbCommandCount) as maxCmds, count() as requests by route
| sort maxCmds desc' \
--profile learn-sandbox --region us-east-1Counter names for dotnet-counters (threadpool-thread-count, threadpool-queue-length) come from the System.Runtime provider; run dotnet-counters list to confirm the names for your version. ⚠ VERIFY
Infrastructure as Code
The stack deploys a Lambda function with SnapStart and a published version alias, plus alarms for database connections and for a custom thread-pool queue metric. The RDS instance is assumed to exist (its identifier is passed in).
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.CloudWatch.Actions;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.SNS;
using Constructs;
namespace Harbourline.Monitoring.Infra;
public sealed class PerformanceGuardrailsStack : Stack
{
public PerformanceGuardrailsStack(Construct scope, string id, string dbInstanceId, IStackProps? props = null)
: base(scope, id, props)
{
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m11-l11");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
var topic = new Topic(this, "PerfTopic", new TopicProps { TopicName = "hl-m11-perf-alerts" });
// --- Lambda with SnapStart on a published version ---
var fn = new Function(this, "PaymentsFn", new FunctionProps
{
FunctionName = "hl-payments-fn",
// ⚠ VERIFY: newer CDK versions expose Runtime.DOTNET_10; this builds the runtime by name.
Runtime = new Runtime("dotnet10", RuntimeFamily.DOTNET_CORE),
Handler = "Harbourline.Payments.Lambda::Harbourline.Payments.Lambda.Function::FunctionHandler",
Code = Code.FromAsset("./publish"), // output of dotnet publish -c Release (ReadyToRun on)
MemorySize = 1024, // more memory = more CPU, which shortens INIT
Timeout = Duration.Seconds(10),
Architecture = Architecture.ARM_64, // Graviton, usually cheaper per ms ⚠ VERIFY for your workload
});
// SnapStart via the L1 property, so we do not depend on an L2 flag for .NET.
((CfnFunction)fn.Node.DefaultChild!).SnapStart = new CfnFunction.SnapStartProperty
{
ApplyOn = "PublishedVersions",
};
var version = fn.CurrentVersion; // a new version per code change
new Alias(this, "Live", new AliasProps { AliasName = "live", Version = version });
// --- Alarm: database connections approaching the server limit ---
var connections = new Metric(new MetricProps
{
Namespace = "AWS/RDS",
MetricName = "DatabaseConnections",
DimensionsMap = new System.Collections.Generic.Dictionary<string, string>
{
["DBInstanceIdentifier"] = dbInstanceId,
},
Statistic = "Maximum",
Period = Duration.Minutes(1),
});
new Alarm(this, "DbConnectionsHigh", new AlarmProps
{
AlarmName = "hl-db-connections-high",
AlarmDescription = "Connections above 80% of the budget. Check pools, leaks and task count. Runbook: pool exhaustion.",
Metric = connections,
Threshold = 80, // set to 80% of the real max_connections minus a reserve for admin tools
EvaluationPeriods = 3,
DatapointsToAlarm = 3,
ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
TreatMissingData = TreatMissingData.MISSING,
}).AddAlarmAction(new SnsAction(topic));
// --- Alarm: thread-pool queue length published by the app (see the C# below) ---
var queue = new Metric(new MetricProps
{
Namespace = "Harbourline/Runtime",
MetricName = "ThreadPoolQueueLength",
DimensionsMap = new System.Collections.Generic.Dictionary<string, string>
{
["Service"] = "harbourline-payments-api",
},
Statistic = "Maximum",
Period = Duration.Minutes(1),
});
new Alarm(this, "ThreadPoolQueueHigh", new AlarmProps
{
AlarmName = "hl-threadpool-queue-high",
AlarmDescription = "Thread pool work queue is growing. Check for sync-over-async. Runbook: thread starvation.",
Metric = queue,
Threshold = 50,
EvaluationPeriods = 3,
DatapointsToAlarm = 3,
ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING,
}).AddAlarmAction(new SnsAction(topic));
}
}Complete code
Four pieces: the deliberately broken endpoint and its fix, a runtime stats publisher, an EF Core interceptor that counts commands per request, and an N+1 example with its fixes. All data is fictional.
// Lab 11.4: reproduce thread-pool starvation. Run, then hit /bad with 200 concurrent requests.
using Harbourline.Observability;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHostedService<RuntimeStatsPublisher>();
builder.Services.AddSingleton<PartnerClient>();
var app = builder.Build();
// BAD: blocks a pool thread while waiting for async work (sync-over-async).
app.MapGet("/bad", (PartnerClient partner) =>
{
var rate = partner.GetRateAsync("GBP", "EUR").GetAwaiter().GetResult(); // blocks!
return Results.Ok(new { rate });
});
// GOOD: the thread goes back to the pool while the I/O is pending.
app.MapGet("/good", async (PartnerClient partner, CancellationToken ct) =>
{
var rate = await partner.GetRateAsync("GBP", "EUR", ct);
return Results.Ok(new { rate });
});
app.Run();
/// <summary>Simulates a slow partner FX-rate API.</summary>
public sealed class PartnerClient
{
public async Task<decimal> GetRateAsync(string from, string to, CancellationToken ct = default)
{
await Task.Delay(TimeSpan.FromMilliseconds(500), ct); // pretend network latency
return 1.1632m; // fictional rate
}
}using System.Text.Json;
namespace Harbourline.Observability;
/// <summary>
/// Every 10 seconds, logs thread-pool statistics as a CloudWatch Embedded Metric Format line.
/// CloudWatch turns the line into metrics: no PutMetricData call, no extra SDK dependency.
/// </summary>
public sealed class RuntimeStatsPublisher(ILogger<RuntimeStatsPublisher> logger) : BackgroundService
{
private static readonly JsonSerializerOptions Json = new() { WriteIndented = false };
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
using var timer = new PeriodicTimer(TimeSpan.FromSeconds(10));
while (await timer.WaitForNextTickAsync(stoppingToken))
{
var emf = new Dictionary<string, object>
{
["_aws"] = new Dictionary<string, object>
{
["Timestamp"] = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(),
["CloudWatchMetrics"] = new[]
{
new Dictionary<string, object>
{
["Namespace"] = "Harbourline/Runtime",
["Dimensions"] = new[] { new[] { "Service" } },
["Metrics"] = new[]
{
new Dictionary<string, string> { ["Name"] = "ThreadPoolQueueLength", ["Unit"] = "Count" },
new Dictionary<string, string> { ["Name"] = "ThreadPoolThreadCount", ["Unit"] = "Count" },
new Dictionary<string, string> { ["Name"] = "GcGen2Collections", ["Unit"] = "Count" },
},
},
},
},
["Service"] = "harbourline-payments-api",
["ThreadPoolQueueLength"] = ThreadPool.PendingWorkItemCount,
["ThreadPoolThreadCount"] = ThreadPool.ThreadCount,
["GcGen2Collections"] = GC.CollectionCount(2),
};
// One JSON object per line on stdout: the ECS awslogs driver ships it to CloudWatch Logs.
Console.Out.WriteLine(JsonSerializer.Serialize(emf, Json));
}
logger.LogInformation("Runtime stats publisher stopped");
}
}using System.Data.Common;
using Microsoft.EntityFrameworkCore.Diagnostics;
namespace Harbourline.Observability;
/// <summary>Counts database commands per HTTP request so N+1 patterns show up in logs and traces.</summary>
public sealed class DbCommandCounter(IHttpContextAccessor accessor) : DbCommandInterceptor
{
public const string ItemKey = "hl.db.command.count";
public override ValueTask<InterceptionResult<DbDataReader>> ReaderExecutingAsync(
DbCommand command, CommandEventData eventData,
InterceptionResult<DbDataReader> result, CancellationToken cancellationToken = default)
{
Increment();
return base.ReaderExecutingAsync(command, eventData, result, cancellationToken);
}
public override InterceptionResult<DbDataReader> ReaderExecuting(
DbCommand command, CommandEventData eventData, InterceptionResult<DbDataReader> result)
{
Increment();
return base.ReaderExecuting(command, eventData, result);
}
private void Increment()
{
var items = accessor.HttpContext?.Items;
if (items is null) return;
items[ItemKey] = (items.TryGetValue(ItemKey, out var v) && v is int n ? n : 0) + 1;
}
}
/// <summary>Middleware that logs the final count and warns when it looks like N+1.</summary>
public sealed class DbCommandCountMiddleware(RequestDelegate next, ILogger<DbCommandCountMiddleware> logger)
{
private const int WarnThreshold = 15;
public async Task InvokeAsync(HttpContext ctx)
{
await next(ctx);
var count = ctx.Items.TryGetValue(DbCommandCounter.ItemKey, out var v) && v is int n ? n : 0;
var route = (ctx.GetEndpoint() as RouteEndpoint)?.RoutePattern.RawText ?? "unknown";
if (count >= WarnThreshold)
{
logger.LogWarning("Possible N+1: {Route} ran {DbCommandCount} database commands", route, count);
}
else
{
logger.LogInformation("RequestCompleted {Route} {DbCommandCount}", route, count);
}
}
}using Microsoft.EntityFrameworkCore;
namespace Harbourline.Accounts.Data;
public sealed record StatementLine(string AccountId, DateOnly BookingDate, decimal Amount, string CounterpartyName);
public static class StatementQueries
{
// BAD: N+1. One query for accounts, then one query per account for its transactions.
// With lazy loading proxies enabled the loop below does the same thing without you noticing.
public static async Task<List<StatementLine>> LatestLinesNPlusOneAsync(
BankingDbContext db, string customerId, CancellationToken ct)
{
var accounts = await db.Accounts.Where(a => a.CustomerId == customerId).ToListAsync(ct); // 1 query
var lines = new List<StatementLine>();
foreach (var account in accounts) // N queries
{
var txs = await db.Transactions
.Where(t => t.AccountId == account.AccountId)
.OrderByDescending(t => t.BookingDate)
.Take(10)
.ToListAsync(ct);
lines.AddRange(txs.Select(t => new StatementLine(t.AccountId, t.BookingDate, t.Amount, t.CounterpartyName)));
}
return lines;
}
// GOOD: one round trip. Project only the columns you need, join in the database.
public static Task<List<StatementLine>> LatestLinesAsync(
BankingDbContext db, string customerId, CancellationToken ct) =>
db.Transactions
.AsNoTracking()
.Where(t => db.Accounts.Any(a => a.AccountId == t.AccountId && a.CustomerId == customerId))
.OrderByDescending(t => t.BookingDate)
.Take(50)
.Select(t => new StatementLine(t.AccountId, t.BookingDate, t.Amount, t.CounterpartyName))
.ToListAsync(ct);
}And the connection and HTTP client settings that avoid the other two classic failures:
// Connection string: size the pool deliberately and fail fast instead of queueing forever.
// Npgsql: Maximum Pool Size defaults to 100 per connection string, per process.
var cs = "Host=hl-lab-postgres.example.internal;Database=banking;Username=app;" +
"Maximum Pool Size=20;Minimum Pool Size=2;Timeout=5;Command Timeout=15;" +
"Application Name=hl-payments-api"; // password comes from Secrets Manager, never from code
builder.Services.AddHttpContextAccessor();
builder.Services.AddSingleton<DbCommandCounter>();
builder.Services.AddDbContextPool<BankingDbContext>((sp, o) =>
o.UseNpgsql(cs).AddInterceptors(sp.GetRequiredService<DbCommandCounter>()), poolSize: 64);
// One typed client via the factory: reuses sockets, recycles DNS. Never `new HttpClient()` per call.
builder.Services.AddHttpClient<PartnerClient>(c => c.Timeout = TimeSpan.FromSeconds(5))
.ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
{
PooledConnectionLifetime = TimeSpan.FromMinutes(5),
MaxConnectionsPerServer = 100,
});
// ...
app.UseMiddleware<DbCommandCountMiddleware>();Code explained line by line
GetAwaiter().GetResult()in/badblocks the current pool thread until the async delay finishes. Each request holds one thread for 500 ms doing nothing. With 200 concurrent requests, the pool runs out of ready threads.- The .NET thread pool adds threads slowly (it measures throughput and injects roughly one or two per second once the minimum is reached) so the queue grows faster than the pool, and latency climbs while CPU stays low. That is starvation.
/goodusesawait, so the thread returns to the pool during the delay; the same 200 requests are served by a handful of threads.RuntimeStatsPublisheremits a CloudWatch Embedded Metric Format line: the_awsblock says which fields are metrics and which are dimensions. Writing it to stdout is enough on ECS (the log driver ships it) and Lambda. The metric name in the alarm (ThreadPoolQueueLength) matches the field.ThreadPool.PendingWorkItemCountandThreadPool.ThreadCountare static properties onThreadPool(available since .NET Core 3.0).GC.CollectionCount(2)counts gen-2 collections; a steady climb points to memory pressure rather than starvation.PeriodicTimerreplacesTask.Delayloops with a cleaner, cancellation-aware tick.DbCommandCounterhooks into EF Core's command pipeline, counts reader executions on theHttpContext.Itemsdictionary and the middleware reads the total after the request finishes. The warning threshold of 15 is arbitrary; tune it to your slowest legitimate endpoint.- In
LatestLinesNPlusOneAsynctheforeachruns one query per account: 1 + N round trips. The fixed version is a single SQL statement, withAsNoTracking(no change tracking needed for read models) and a projection to a record. Maximum Pool Size=20limits each process to 20 connections;Timeout=5makes callers fail after five seconds waiting for a pooled connection instead of the default queueing behaviour, so exhaustion surfaces fast and loudly.AddDbContextPoolreusesDbContextinstances, reducing allocation. It does not change database connection pooling, which is the driver's job.AddHttpClient<PartnerClient>plusPooledConnectionLifetimekeeps connections reusable but periodically refreshed so DNS changes (for example blue/green partner endpoints) are picked up.
Security considerations
⚠ Learning shortcut — not for production
The lab connects to the database with a literal connection string, exposes /bad publicly on a sandbox host and runs diagnostic tools as a privileged user. In production, read credentials from Secrets Manager at start-up (or use IAM database authentication), never ship a deliberately broken endpoint, and gate diagnostic tooling behind break-glass access.
- Dumps are data.
dotnet-dumpanddotnet-gcdumpfiles contain process memory: connection strings, tokens, customer fields. Store them encrypted, restrict access, delete after use, and treat them as personal data under UK GDPR. - Diagnostic ports.
dotnet-countersanddotnet-traceuse a local IPC channel. Do not expose it over the network, and avoid addingSYS_PTRACEor shared PID namespaces to production containers casually. - Do not log SQL parameters. EF Core can log parameter values when sensitive data logging is on; keep
EnableSensitiveDataLoggingoff outside local development. - Alarms need runbooks, not just thresholds. An unowned alarm is a signal nobody acts on.
Banking use case
A summary of the four failures as they occurred on the Harbourline payments service during a rehearsal:
| Incident | Evidence | Root cause | Fix |
|---|---|---|---|
| Slow payments, CPU 20%, 504s | Thread count flat, queue length 400+, monitor-lock-contention low | A fraud-check client called .Result on an async SDK call | Made the path async; set Timeout on the client; added the queue-length alarm |
Timeout expired ... pool errors | RDS connections flat at 100, tasks at 12 | 12 tasks times default pool of 100 exceeded the database's limit; one handler leaked a connection | Pool size 20, using for the connection, RDS Proxy for Lambda callers |
| First request of the morning 6 s on the Lambda path | Init Duration of 4.8 s on cold starts only | Heavy startup: JSON source generators missing, DI container building every client eagerly | ReadyToRun, 1,024 MB, lazy clients, SnapStart on the live alias |
| Statement endpoint slow for customers with many accounts | 1 + N SQL statements per request, N up to 40 | Query inside a loop | Single projected query; command-count warning in logs |
The connection arithmetic that should live in every design review:
tasks (or Lambda concurrency) x Max Pool Size <= usable database connections - admin reserve
12 tasks x 20 = 240 vs db.t4g.medium PostgreSQL default max_connections of roughly 400-ish ⚠ VERIFY for your instance class
Lambda at 200 concurrency x 5 = 1,000 -> needs RDS Proxy or a lower per-function poolThe exact max_connections depends on the engine, parameter group and instance memory; look up the real value with SHOW max_connections; and in the RDS documentation, rather than trusting a rule of thumb. ⚠ VERIFY
Common mistakes
- Scaling out first. More tasks make connection pressure worse and do nothing for starvation. Diagnose before scaling.
.Result,.Wait()andGetAwaiter().GetResult()in request paths. AlsoTask.Runaround async code andasync void. Go async end to end.- Raising
ThreadPool.SetMinThreadsas the fix. It can hide the problem and burn memory. Use it as a stopgap while you remove the blocking. - Default pool sizes multiplied by many instances. 100 per process looks harmless until the arithmetic is done.
- Not disposing connections or DbContexts. Leaks show as pool exhaustion that never recovers until restart.
new HttpClient()per request. Causes socket exhaustion and stale DNS; useIHttpClientFactoryor a shared instance.- Lazy-loading proxies in API code. They hide N+1 behind property access.
- Fixing cold starts with provisioned concurrency before measuring. Measure
Init Durationfirst; most of it is often avoidable initialisation work. - Putting I/O in a static constructor of a Lambda function. Errors there fail every invocation of that environment; prefer explicit, logged initialisation.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| High latency, low CPU, queue length rising | Thread-pool starvation (sync-over-async, blocking I/O) | dotnet-stack report to find blocked frames; make the code async; add timeouts |
dotnet-counters shows no data | Wrong PID, container without shared diagnostics IPC, or tool version mismatch | Run the tool inside the same container (or share /tmp), match tool to runtime |
Timeout ... obtaining a connection from the pool | Pool exhausted by leaks or too few connections for the load | Dispose connections, size pool, check slow queries holding connections, add RDS Proxy |
too many connections from the database | Sum of pools exceeds server limit | Reduce pool size or instance count; use RDS Proxy; raise limit only with memory to match |
| Intermittent socket errors to a partner | Per-request HttpClient or stale connections | Use the factory, set PooledConnectionLifetime |
| Slow first request only | Cold start (Lambda INIT, or JIT on a new task) | Measure; ReadyToRun, trimming, more memory, SnapStart, warm-up on ECS before registering with the ALB |
Init Duration large even with SnapStart | Snapshot not applied (using $LATEST), or post-restore work heavy | Invoke the published version or alias; move per-invocation work out of init |
| Many similar SQL statements per request | N+1 | Count commands, then Include, projection or batch queries |
| Memory climbs, gen-2 GCs frequent | Large object allocations, caches without bounds | dotnet-gcdump, bound caches, check Server vs Workstation GC settings |
Cost and cleanup
| Item | Label | Estimate |
|---|---|---|
Local reproduction with dotnet-counters | free | Runs on your machine |
| Lambda function, alias and SnapStart | credits | A few invocations are well within free allowances; SnapStart caching is charged per hour per version for non-Java runtimes ⚠ VERIFY; delete versions after the lab |
| CloudWatch alarms (2) and EMF metrics | credits | Pennies for an hour |
| RDS instance for connection tests | paid | Use the lab database from Module 6 and stop or delete it; or skip and use local PostgreSQL in Docker |
Free alternative: do the whole thread-pool and N+1 reproduction locally. Run PostgreSQL in Docker, set Maximum Pool Size=5, and reproduce exhaustion by holding connections open in a loop. Run dotnet-counters against the local process. Only the SnapStart and alarm parts need an AWS account.
# Remove the stack
cdk destroy PerformanceGuardrailsStack --profile learn-sandbox # or: terraform destroy
# SnapStart caches snapshots per published version: delete old versions explicitly if you created any by hand.
aws lambda list-versions-by-function --function-name hl-payments-fn --profile learn-sandbox --region us-east-1
aws lambda delete-function --function-name hl-payments-fn --qualifier 1 --profile learn-sandbox --region us-east-1
# Confirm nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1 --query 'ResourceTagMappingList[].ResourceARN'Exercises
- 🟢 Guided (Lab 11.4): Run
Starvation/Program.cs. Withdotnet-countersattached, send 200 concurrent requests to/bad(use a loop ofcurlor a small C# load generator), record the queue length and p99 latency, then repeat for/good. Write down the three numbers that differ most. - 🟡 Challenge: Add a new endpoint that gets a customer's accounts and their last ten transactions each using the N+1 version, then convert it to a single query. Use
DbCommandCounteroutput to prove the number of commands dropped. - 🔴 Live problem: Ticket: "Lambda function
hl-payments-fnp99 is 5 seconds on Monday mornings but 120 ms the rest of the week." You have CloudWatch Logs and the code. Write the Logs Insights query, the three hypotheses you will test (cold starts after idle, connection setup, a Monday batch), the evidence for each, and the cheapest fix for each.
Assignment
Write a "performance triage card" for Harbourline on one page: for each of the four failures, the three signals that confirm it, the one command that proves it, the safe first fix and the dangerous first fix to avoid. Add the connection-arithmetic worksheet filled in for your own sandbox. Commit it with the guardrail IaC.
Solutions
Typical results for 200 concurrent requests with a 500 ms simulated call: on /bad, ThreadPool Thread Count climbs slowly, ThreadPool Queue Length reaches the tens or hundreds, p99 latency stretches to many seconds, CPU stays low. On /good, queue length stays near zero, thread count is small and p99 is close to 500 ms. The three differing numbers are queue length, p99 latency and thread count. The explanation: blocked threads are not available to start new work, and the pool grows by only a thread or two per second, so latency is dominated by waiting for a thread, not by the work.
Write the loop version (foreach account, query transactions) and watch the interceptor log N + 1 commands. Replace it with the single join/projection shown in Queries.cs, or use Include with AsSplitQuery() if you need full entities. The command count should drop to 1 (or 2 for split queries). Remember that ordering and Take per account need a window function or a per-account approach; if you need "last ten per account" exactly, a single query with ROW_NUMBER() through raw SQL or a lateral join is the correct tool, and the counter proves it ran once.
Knowledge check
Answer: Thread-pool starvation, usually from blocking calls on async work. CPU-bound problems show high CPU; a database problem would show database metrics.
Answer: The tasks can open up to 1,200 connections, so the database refuses new ones once it reaches 300. Reduce the pool size, reduce instances, or add RDS Proxy.
Answer: SnapStart. It works only on published versions and aliases (not $LATEST) and does not support provisioned concurrency, EFS or ephemeral storage above 512 MB.
Answer: Many near-identical SQL statements within a single request, with latency growing with the number of rows in the first query.
Answer: Each new task has the same blocking code, so it starves as well, and extra tasks multiply database connections. Fix the blocking calls first.
Interview questions
- How do you tell thread-pool starvation from a slow database? Starvation: low CPU, rising queue length, blocked frames in
dotnet-stack, database latency normal. Slow database: queue length normal, traces show long database spans, database metrics and wait events degraded. - What would you check if pool-timeout errors appear after a scale-out? The multiplication of instances by pool size against server connections, leaked connections, slow queries holding connections, and whether RDS Proxy is warranted.
- Name three ways to reduce Lambda .NET cold starts. Increase memory (more CPU), ReadyToRun or Native AOT with trimming, SnapStart on published versions, reduce eager initialisation, provisioned concurrency for strict latency needs.
- How do you find N+1 in an existing service? Count database commands per request (interceptor or trace span counts), look for repeated statements, and review loops that call the database.
Certification practice
A .NET Lambda function behind API Gateway has acceptable latency except the first request after idle periods, which takes about five seconds. Logs show Init Duration of 4.7 seconds. The team wants to reduce this without keeping instances warm full time. What is the BEST option?
A. Increase the function timeout. B. Enable Lambda SnapStart on a published version and invoke it through an alias. C. Move the function into a VPC. D. Switch to a larger EBS volume.
Answer: B. SnapStart restores from a snapshot of the initialised environment. A does not shorten init. C usually makes cold starts longer. D does not apply to Lambda.
An application on ECS shows rising latency and 504 errors, CloudWatch shows CPU at 18% and the RDS DatabaseConnections metric flat at its maximum. Application logs show timeouts obtaining a connection from the pool. Which action addresses the cause?
A. Increase the ALB idle timeout. B. Reduce per-task connection pool size and total tasks to fit the database limit, fix leaks, and consider RDS Proxy. C. Increase the task CPU. D. Enable cross-zone load balancing.
Answer: B. The database connection ceiling is saturated. A, C and D do not change connection demand.
A team wants an automatic alarm when a .NET service starts queueing work on its thread pool, without installing agents. Which approach is the simplest?
A. Write EMF log lines containing the thread pool queue length and alarm on the resulting metric. B. Poll the process from outside with SSH. C. Enable VPC Flow Logs. D. Use CloudTrail data events.
Answer: A. Embedded Metric Format turns a structured log line into a metric with no extra API calls. The others do not observe runtime internals.
Completion checklist
- I can explain the four failures and their symptoms in plain language
- I reproduced thread-pool starvation locally and watched it in
dotnet-counters - I can do the connection-pool arithmetic for my own deployment
- I know the SnapStart restrictions and how to read
Init Duration - I counted database commands per request and fixed an N+1 query
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- Most .NET performance failures on AWS are resource-shaped: threads, connections, init time and round trips, not CPU.
- Starvation looks like high latency with low CPU; async end to end is the fix, thread settings are a stopgap.
- Pools multiply: instances times pool size must fit under the database limit, or use RDS Proxy.
- Measure cold starts from
Init Durationbefore choosing a remedy. - Count commands per request to catch N+1 before customers do.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Log aggregation and error monitoring across accounts
Centralise logs from many AWS accounts without copying everything twice, group thousands of exception lines into a handful of distinct errors, and route the right alerts to the right people with severity-based paths.
Troubleshooting production issues methodically: symptom, hypothesis, evidence, fix
Follow a repeatable process from a vague symptom to a proven root cause, using a worked 5xx investigation on an ECS service behind an Application Load Balancer, and build the dashboard and evidence-gathering tool that make the next investigation faster.