Skip to Content
Cloud Engineering Academy
11. Monitoring, Logging & Troubleshooting

Performance bottlenecks in .NET on AWS: thread pool, connection pools, cold starts and N+1

Recognise the four classic .NET performance failures on AWS from their symptoms, prove each with evidence from dotnet-counters, CloudWatch and traces, and apply the right fix.

advanced⏱ 2 h 🟢 FreeSOA-C03DVA-C03DOP-C02

Assumes: .NET 10 LTS · EF Core 10 · Npgsql · AWS SDK for .NET v4 · dotnet-counters · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain thread-pool starvation, connection-pool exhaustion, cold starts and N+1 queries in plain language, and say what each looks like on a dashboard.
  • Reproduce thread-pool starvation with sync-over-async code, diagnose it with dotnet-counters, and fix it.
  • Calculate whether your connection pools, instance counts and database limits add up, and know when to add RDS Proxy.
  • Measure Lambda cold starts from the REPORT log line and choose between memory tuning, ReadyToRun, Native AOT, SnapStart and provisioned concurrency.
  • Detect N+1 queries by counting database commands per request, and fix them with EF Core projections, Include or batching.

Prerequisites

Why this matters in banking

Salary day, 09:00. Payments traffic triples. Dashboards show CPU at 22% on every task, memory is fine, the database is idle, and yet the p99 latency of POST /payments is 14 seconds and the load balancer is returning 504s. Nothing is obviously broken, so the team scales out from 4 to 12 tasks. Latency does not improve, and now the database is rejecting connections.

This is not unusual. In .NET services the expensive failures are usually resource-shaped, not CPU-shaped: you run out of threads, or connections, or time, or round trips, long before you run out of CPU. Each looks different and each has a different fix. Scaling out is the wrong first move for two of the four.

The idea in plain language

Think of a branch with a fixed number of cashiers (threads), a fixed number of vault keys (database connections), a staff room where cashiers must wake up before serving (cold start) and a manager who sends a runner to the vault once per customer instead of once per hundred customers (N+1).

ProblemPlain-language versionTypical symptom
Thread-pool starvationAll cashiers are standing at the vault door waiting; new customers queue even though the building is emptyHigh latency, low CPU, rising queue length, timeouts
Connection-pool exhaustionAll vault keys are out; everyone waits for a keyErrors like "timeout obtaining a connection from the pool", database connection count flat at the maximum
Cold startThe cashier has to wake up, unpack and log in before the first customerFirst request after idle or scale-out is slow; the rest are fast
N+1 queriesOne query for the list, then one more for each item in the listLatency grows with the number of rows returned; many near-identical SQL statements per request

Analogy: starvation is a bank where every cashier phones head office and waits on hold instead of serving the next customer. The phone line is free to answer other calls; they just do not hang up and come back. Async code is hanging up and calling back.

What problem the AWS service solves

This lesson is mostly about your code, but AWS gives you the evidence and some of the levers:

  • CloudWatch metrics show symptoms: ECS CPUUtilization/MemoryUtilization, ALB TargetResponseTime and HTTPCode_Target_5XX_Count, RDS DatabaseConnections, Lambda Duration, ConcurrentExecutions and the REPORT line's Init Duration.
  • Tracing (X-Ray, OpenTelemetry, Application Signals) shows where a request spent its time: waiting for a connection looks very different from a slow query.
  • RDS Proxy pools and shares database connections for functions and many small tasks.
  • Lambda features attack cold starts: memory tuning, SnapStart for supported runtimes, provisioned concurrency.

Honest limits:

  • SnapStart supports Java 11 and later, Python 3.12 and later and .NET 8 and later managed runtimes (and matching base images). It works on published versions and aliases only, not $LATEST, and does not support provisioned concurrency, EFS, or ephemeral storage above 512 MB. There is a charge for snapshot caching and restoration for non-Java runtimes. ⚠ VERIFY current pricing and the exact .NET 10 statement; the documentation says ".NET 8 and later".
  • Lambda .NET 10 runtime exists as dotnet10. At the time of checking dotnet8 had reached its deprecation date of 2026-11-10 for the managed runtime, so plan upgrades. ⚠ VERIFY dates.
  • No tool fixes bad code. Autoscaling a starved service adds more starved instances.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Application Insights Profiler / Snapshot DebuggerTraces + dotnet-counters/dotnet-trace against the container; CodeGuru Profiler is not covered hereYou usually attach diagnostic tools yourself in a container
Azure Functions cold start, Premium plan pre-warmed instancesLambda cold start, provisioned concurrency, SnapStartBilling and tuning differ; SnapStart snapshots the initialised environment
Azure SQL max connections / elastic poolsRDS max_connections by instance class, RDS ProxyConnection limits are set by instance memory and engine parameters
App Service "Always On"ECS minimum task count, Lambda provisioned concurrencyIdle shutdown works differently per service
Azure Monitor metrics for thread pool via Application Insights performance countersdotnet-counters, System.Runtime meter exported by OpenTelemetryYou must export runtime metrics yourself
Azure Load TestingA load script, k6 or Locust against your sandbox, or AWS-published distributed load testing solutionsBring your own tool

Architecture

Where the limits sit: threads per process (thread pool), connections per process (client pool), connections per database (server limit), and time to first request (cold start). A request fails or slows at the smallest of them.

Step-by-step implementation

Establish a baseline with runtime counters

Start the API locally, run a normal load, and record the thread count, queue length, GC counts and request rate. Without a baseline you cannot tell abnormal from normal.

Reproduce thread-pool starvation

Run the deliberately bad endpoint that blocks on async work. Watch queue length climb while CPU stays low.

Diagnose with dotnet-counters

Attach to the process, read the thread pool counters, then capture stacks (dotnet-stack report) to see the threads blocked in .Result or .Wait().

Fix it, then prove the fix

Make the call async end to end and re-run the same load. The queue length should stay near zero.

Add the three guardrails

A background service that logs thread-pool stats, an EF Core interceptor that counts commands per request, and CloudWatch alarms on database connections and Lambda init duration.

Check the arithmetic for connections

Multiply tasks (or Lambda concurrency) by Max Pool Size; compare with the database limit. Decide whether you need smaller pools, fewer tasks or RDS Proxy.

AWS Console instructions

  • ECS: Console → ECS → Clusters → your cluster → Services → your service → Metrics tab for CPU and memory; Logs for application output. Compare with ALB Target group → Monitoring for TargetResponseTime and 5xx.
  • RDS: Console → RDS → Databases → your instance → Monitoring → DatabaseConnections, CPUUtilization, FreeableMemory. Performance Insights (if enabled) shows top SQL and wait events. ⚠ VERIFY: Performance Insights availability and pricing changed over time; check the current page.
  • Lambda: Console → Lambda → your function → Monitor → View CloudWatch logs. In Logs Insights, run the cold start query below. Configuration → General for memory; Versions/Aliases for SnapStart and provisioned concurrency.
  • CloudWatch: Metrics → All metrics to graph AWS/RDS DatabaseConnections and your custom thread-pool metric on one chart with latency.

AWS CLI

# 1) Local diagnosis (no AWS needed). Find the process, then watch runtime counters.
dotnet-counters ps
dotnet-counters monitor --process-id <PID> --refresh-interval 1 \
  --counters System.Runtime[threadpool-thread-count,threadpool-queue-length,threadpool-completed-items-count,monitor-lock-contention-count,cpu-usage,gc-heap-size]

# 2) See what the blocked threads are doing (dotnet-stack is a separate global tool).
dotnet-stack report --process-id <PID> | head -80

# 3) RDS connection count for the last hour, 1-minute average.
aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name DatabaseConnections \
  --dimensions Name=DBInstanceIdentifier,Value=hl-lab-postgres \
  --start-time $(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%SZ) --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 60 --statistics Maximum \
  --profile learn-sandbox --region us-east-1

# 4) Lambda cold starts: how many in the last hour, and how long did INIT take?
aws logs start-query --log-group-name /aws/lambda/hl-payments-fn \
  --start-time $(date -u -d '-1 hour' +%s) --end-time $(date -u +%s) \
  --query-string 'filter @type = "REPORT"
    | stats count() as invocations,
            count(@initDuration) as coldStarts,
            avg(@initDuration) as avgInitMs,
            max(@initDuration) as maxInitMs,
            pct(@duration, 99) as p99Ms,
            max(@maxMemoryUsed / 1024 / 1024) as maxMemMB
      by bin(5m)' \
  --profile learn-sandbox --region us-east-1

# 5) Publish a version and turn on SnapStart for published versions (supported for .NET 8 and later).
aws lambda update-function-configuration --function-name hl-payments-fn \
  --snap-start ApplyOn=PublishedVersions --profile learn-sandbox --region us-east-1
aws lambda wait function-updated --function-name hl-payments-fn --profile learn-sandbox --region us-east-1
aws lambda publish-version --function-name hl-payments-fn --profile learn-sandbox --region us-east-1

# 6) How many database commands did one request run? (find it in your own structured logs)
aws logs start-query --log-group-name /hl/payments/api \
  --start-time $(date -u -d '-30 minutes' +%s) --end-time $(date -u +%s) \
  --query-string 'filter category = "RequestCompleted"
    | stats avg(dbCommandCount) as avgCmds, max(dbCommandCount) as maxCmds, count() as requests by route
    | sort maxCmds desc' \
  --profile learn-sandbox --region us-east-1

Counter names for dotnet-counters (threadpool-thread-count, threadpool-queue-length) come from the System.Runtime provider; run dotnet-counters list to confirm the names for your version. ⚠ VERIFY

Infrastructure as Code

The stack deploys a Lambda function with SnapStart and a published version alias, plus alarms for database connections and for a custom thread-pool queue metric. The RDS instance is assumed to exist (its identifier is passed in).

PerformanceGuardrailsStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.CloudWatch.Actions;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.SNS;
using Constructs;

namespace Harbourline.Monitoring.Infra;

public sealed class PerformanceGuardrailsStack : Stack
{
    public PerformanceGuardrailsStack(Construct scope, string id, string dbInstanceId, IStackProps? props = null)
        : base(scope, id, props)
    {
        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m11-l11");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");

        var topic = new Topic(this, "PerfTopic", new TopicProps { TopicName = "hl-m11-perf-alerts" });

        // --- Lambda with SnapStart on a published version ---
        var fn = new Function(this, "PaymentsFn", new FunctionProps
        {
            FunctionName = "hl-payments-fn",
            // ⚠ VERIFY: newer CDK versions expose Runtime.DOTNET_10; this builds the runtime by name.
            Runtime = new Runtime("dotnet10", RuntimeFamily.DOTNET_CORE),
            Handler = "Harbourline.Payments.Lambda::Harbourline.Payments.Lambda.Function::FunctionHandler",
            Code = Code.FromAsset("./publish"), // output of dotnet publish -c Release (ReadyToRun on)
            MemorySize = 1024,                  // more memory = more CPU, which shortens INIT
            Timeout = Duration.Seconds(10),
            Architecture = Architecture.ARM_64, // Graviton, usually cheaper per ms ⚠ VERIFY for your workload
        });

        // SnapStart via the L1 property, so we do not depend on an L2 flag for .NET.
        ((CfnFunction)fn.Node.DefaultChild!).SnapStart = new CfnFunction.SnapStartProperty
        {
            ApplyOn = "PublishedVersions",
        };

        var version = fn.CurrentVersion; // a new version per code change
        new Alias(this, "Live", new AliasProps { AliasName = "live", Version = version });

        // --- Alarm: database connections approaching the server limit ---
        var connections = new Metric(new MetricProps
        {
            Namespace = "AWS/RDS",
            MetricName = "DatabaseConnections",
            DimensionsMap = new System.Collections.Generic.Dictionary<string, string>
            {
                ["DBInstanceIdentifier"] = dbInstanceId,
            },
            Statistic = "Maximum",
            Period = Duration.Minutes(1),
        });

        new Alarm(this, "DbConnectionsHigh", new AlarmProps
        {
            AlarmName = "hl-db-connections-high",
            AlarmDescription = "Connections above 80% of the budget. Check pools, leaks and task count. Runbook: pool exhaustion.",
            Metric = connections,
            Threshold = 80, // set to 80% of the real max_connections minus a reserve for admin tools
            EvaluationPeriods = 3,
            DatapointsToAlarm = 3,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
            TreatMissingData = TreatMissingData.MISSING,
        }).AddAlarmAction(new SnsAction(topic));

        // --- Alarm: thread-pool queue length published by the app (see the C# below) ---
        var queue = new Metric(new MetricProps
        {
            Namespace = "Harbourline/Runtime",
            MetricName = "ThreadPoolQueueLength",
            DimensionsMap = new System.Collections.Generic.Dictionary<string, string>
            {
                ["Service"] = "harbourline-payments-api",
            },
            Statistic = "Maximum",
            Period = Duration.Minutes(1),
        });

        new Alarm(this, "ThreadPoolQueueHigh", new AlarmProps
        {
            AlarmName = "hl-threadpool-queue-high",
            AlarmDescription = "Thread pool work queue is growing. Check for sync-over-async. Runbook: thread starvation.",
            Metric = queue,
            Threshold = 50,
            EvaluationPeriods = 3,
            DatapointsToAlarm = 3,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
            TreatMissingData = TreatMissingData.NOT_BREACHING,
        }).AddAlarmAction(new SnsAction(topic));
    }
}

Complete code

Four pieces: the deliberately broken endpoint and its fix, a runtime stats publisher, an EF Core interceptor that counts commands per request, and an N+1 example with its fixes. All data is fictional.

Starvation/Program.cs
// Lab 11.4: reproduce thread-pool starvation. Run, then hit /bad with 200 concurrent requests.
using Harbourline.Observability;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHostedService<RuntimeStatsPublisher>();
builder.Services.AddSingleton<PartnerClient>();
var app = builder.Build();

// BAD: blocks a pool thread while waiting for async work (sync-over-async).
app.MapGet("/bad", (PartnerClient partner) =>
{
    var rate = partner.GetRateAsync("GBP", "EUR").GetAwaiter().GetResult(); // blocks!
    return Results.Ok(new { rate });
});

// GOOD: the thread goes back to the pool while the I/O is pending.
app.MapGet("/good", async (PartnerClient partner, CancellationToken ct) =>
{
    var rate = await partner.GetRateAsync("GBP", "EUR", ct);
    return Results.Ok(new { rate });
});

app.Run();

/// <summary>Simulates a slow partner FX-rate API.</summary>
public sealed class PartnerClient
{
    public async Task<decimal> GetRateAsync(string from, string to, CancellationToken ct = default)
    {
        await Task.Delay(TimeSpan.FromMilliseconds(500), ct); // pretend network latency
        return 1.1632m;                                        // fictional rate
    }
}
RuntimeStatsPublisher.cs
using System.Text.Json;

namespace Harbourline.Observability;

/// <summary>
/// Every 10 seconds, logs thread-pool statistics as a CloudWatch Embedded Metric Format line.
/// CloudWatch turns the line into metrics: no PutMetricData call, no extra SDK dependency.
/// </summary>
public sealed class RuntimeStatsPublisher(ILogger<RuntimeStatsPublisher> logger) : BackgroundService
{
    private static readonly JsonSerializerOptions Json = new() { WriteIndented = false };

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromSeconds(10));
        while (await timer.WaitForNextTickAsync(stoppingToken))
        {
            var emf = new Dictionary<string, object>
            {
                ["_aws"] = new Dictionary<string, object>
                {
                    ["Timestamp"] = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(),
                    ["CloudWatchMetrics"] = new[]
                    {
                        new Dictionary<string, object>
                        {
                            ["Namespace"] = "Harbourline/Runtime",
                            ["Dimensions"] = new[] { new[] { "Service" } },
                            ["Metrics"] = new[]
                            {
                                new Dictionary<string, string> { ["Name"] = "ThreadPoolQueueLength", ["Unit"] = "Count" },
                                new Dictionary<string, string> { ["Name"] = "ThreadPoolThreadCount", ["Unit"] = "Count" },
                                new Dictionary<string, string> { ["Name"] = "GcGen2Collections",    ["Unit"] = "Count" },
                            },
                        },
                    },
                },
                ["Service"] = "harbourline-payments-api",
                ["ThreadPoolQueueLength"] = ThreadPool.PendingWorkItemCount,
                ["ThreadPoolThreadCount"] = ThreadPool.ThreadCount,
                ["GcGen2Collections"] = GC.CollectionCount(2),
            };

            // One JSON object per line on stdout: the ECS awslogs driver ships it to CloudWatch Logs.
            Console.Out.WriteLine(JsonSerializer.Serialize(emf, Json));
        }

        logger.LogInformation("Runtime stats publisher stopped");
    }
}
DbCommandCounter.cs
using System.Data.Common;
using Microsoft.EntityFrameworkCore.Diagnostics;

namespace Harbourline.Observability;

/// <summary>Counts database commands per HTTP request so N+1 patterns show up in logs and traces.</summary>
public sealed class DbCommandCounter(IHttpContextAccessor accessor) : DbCommandInterceptor
{
    public const string ItemKey = "hl.db.command.count";

    public override ValueTask<InterceptionResult<DbDataReader>> ReaderExecutingAsync(
        DbCommand command, CommandEventData eventData,
        InterceptionResult<DbDataReader> result, CancellationToken cancellationToken = default)
    {
        Increment();
        return base.ReaderExecutingAsync(command, eventData, result, cancellationToken);
    }

    public override InterceptionResult<DbDataReader> ReaderExecuting(
        DbCommand command, CommandEventData eventData, InterceptionResult<DbDataReader> result)
    {
        Increment();
        return base.ReaderExecuting(command, eventData, result);
    }

    private void Increment()
    {
        var items = accessor.HttpContext?.Items;
        if (items is null) return;
        items[ItemKey] = (items.TryGetValue(ItemKey, out var v) && v is int n ? n : 0) + 1;
    }
}

/// <summary>Middleware that logs the final count and warns when it looks like N+1.</summary>
public sealed class DbCommandCountMiddleware(RequestDelegate next, ILogger<DbCommandCountMiddleware> logger)
{
    private const int WarnThreshold = 15;

    public async Task InvokeAsync(HttpContext ctx)
    {
        await next(ctx);

        var count = ctx.Items.TryGetValue(DbCommandCounter.ItemKey, out var v) && v is int n ? n : 0;
        var route = (ctx.GetEndpoint() as RouteEndpoint)?.RoutePattern.RawText ?? "unknown";

        if (count >= WarnThreshold)
        {
            logger.LogWarning("Possible N+1: {Route} ran {DbCommandCount} database commands", route, count);
        }
        else
        {
            logger.LogInformation("RequestCompleted {Route} {DbCommandCount}", route, count);
        }
    }
}
Queries.cs
using Microsoft.EntityFrameworkCore;

namespace Harbourline.Accounts.Data;

public sealed record StatementLine(string AccountId, DateOnly BookingDate, decimal Amount, string CounterpartyName);

public static class StatementQueries
{
    // BAD: N+1. One query for accounts, then one query per account for its transactions.
    // With lazy loading proxies enabled the loop below does the same thing without you noticing.
    public static async Task<List<StatementLine>> LatestLinesNPlusOneAsync(
        BankingDbContext db, string customerId, CancellationToken ct)
    {
        var accounts = await db.Accounts.Where(a => a.CustomerId == customerId).ToListAsync(ct); // 1 query
        var lines = new List<StatementLine>();

        foreach (var account in accounts)                                                         // N queries
        {
            var txs = await db.Transactions
                .Where(t => t.AccountId == account.AccountId)
                .OrderByDescending(t => t.BookingDate)
                .Take(10)
                .ToListAsync(ct);

            lines.AddRange(txs.Select(t => new StatementLine(t.AccountId, t.BookingDate, t.Amount, t.CounterpartyName)));
        }

        return lines;
    }

    // GOOD: one round trip. Project only the columns you need, join in the database.
    public static Task<List<StatementLine>> LatestLinesAsync(
        BankingDbContext db, string customerId, CancellationToken ct) =>
        db.Transactions
            .AsNoTracking()
            .Where(t => db.Accounts.Any(a => a.AccountId == t.AccountId && a.CustomerId == customerId))
            .OrderByDescending(t => t.BookingDate)
            .Take(50)
            .Select(t => new StatementLine(t.AccountId, t.BookingDate, t.Amount, t.CounterpartyName))
            .ToListAsync(ct);
}

And the connection and HTTP client settings that avoid the other two classic failures:

Program.cs (excerpt): pools and clients
// Connection string: size the pool deliberately and fail fast instead of queueing forever.
// Npgsql: Maximum Pool Size defaults to 100 per connection string, per process.
var cs = "Host=hl-lab-postgres.example.internal;Database=banking;Username=app;" +
         "Maximum Pool Size=20;Minimum Pool Size=2;Timeout=5;Command Timeout=15;" +
         "Application Name=hl-payments-api"; // password comes from Secrets Manager, never from code

builder.Services.AddHttpContextAccessor();
builder.Services.AddSingleton<DbCommandCounter>();
builder.Services.AddDbContextPool<BankingDbContext>((sp, o) =>
    o.UseNpgsql(cs).AddInterceptors(sp.GetRequiredService<DbCommandCounter>()), poolSize: 64);

// One typed client via the factory: reuses sockets, recycles DNS. Never `new HttpClient()` per call.
builder.Services.AddHttpClient<PartnerClient>(c => c.Timeout = TimeSpan.FromSeconds(5))
    .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
    {
        PooledConnectionLifetime = TimeSpan.FromMinutes(5),
        MaxConnectionsPerServer = 100,
    });

// ...
app.UseMiddleware<DbCommandCountMiddleware>();

Code explained line by line

  1. GetAwaiter().GetResult() in /bad blocks the current pool thread until the async delay finishes. Each request holds one thread for 500 ms doing nothing. With 200 concurrent requests, the pool runs out of ready threads.
  2. The .NET thread pool adds threads slowly (it measures throughput and injects roughly one or two per second once the minimum is reached) so the queue grows faster than the pool, and latency climbs while CPU stays low. That is starvation.
  3. /good uses await, so the thread returns to the pool during the delay; the same 200 requests are served by a handful of threads.
  4. RuntimeStatsPublisher emits a CloudWatch Embedded Metric Format line: the _aws block says which fields are metrics and which are dimensions. Writing it to stdout is enough on ECS (the log driver ships it) and Lambda. The metric name in the alarm (ThreadPoolQueueLength) matches the field.
  5. ThreadPool.PendingWorkItemCount and ThreadPool.ThreadCount are static properties on ThreadPool (available since .NET Core 3.0). GC.CollectionCount(2) counts gen-2 collections; a steady climb points to memory pressure rather than starvation.
  6. PeriodicTimer replaces Task.Delay loops with a cleaner, cancellation-aware tick.
  7. DbCommandCounter hooks into EF Core's command pipeline, counts reader executions on the HttpContext.Items dictionary and the middleware reads the total after the request finishes. The warning threshold of 15 is arbitrary; tune it to your slowest legitimate endpoint.
  8. In LatestLinesNPlusOneAsync the foreach runs one query per account: 1 + N round trips. The fixed version is a single SQL statement, with AsNoTracking (no change tracking needed for read models) and a projection to a record.
  9. Maximum Pool Size=20 limits each process to 20 connections; Timeout=5 makes callers fail after five seconds waiting for a pooled connection instead of the default queueing behaviour, so exhaustion surfaces fast and loudly.
  10. AddDbContextPool reuses DbContext instances, reducing allocation. It does not change database connection pooling, which is the driver's job.
  11. AddHttpClient<PartnerClient> plus PooledConnectionLifetime keeps connections reusable but periodically refreshed so DNS changes (for example blue/green partner endpoints) are picked up.

Security considerations

⚠ Learning shortcut — not for production

The lab connects to the database with a literal connection string, exposes /bad publicly on a sandbox host and runs diagnostic tools as a privileged user. In production, read credentials from Secrets Manager at start-up (or use IAM database authentication), never ship a deliberately broken endpoint, and gate diagnostic tooling behind break-glass access.

  • Dumps are data. dotnet-dump and dotnet-gcdump files contain process memory: connection strings, tokens, customer fields. Store them encrypted, restrict access, delete after use, and treat them as personal data under UK GDPR.
  • Diagnostic ports. dotnet-counters and dotnet-trace use a local IPC channel. Do not expose it over the network, and avoid adding SYS_PTRACE or shared PID namespaces to production containers casually.
  • Do not log SQL parameters. EF Core can log parameter values when sensitive data logging is on; keep EnableSensitiveDataLogging off outside local development.
  • Alarms need runbooks, not just thresholds. An unowned alarm is a signal nobody acts on.

Banking use case

A summary of the four failures as they occurred on the Harbourline payments service during a rehearsal:

IncidentEvidenceRoot causeFix
Slow payments, CPU 20%, 504sThread count flat, queue length 400+, monitor-lock-contention lowA fraud-check client called .Result on an async SDK callMade the path async; set Timeout on the client; added the queue-length alarm
Timeout expired ... pool errorsRDS connections flat at 100, tasks at 1212 tasks times default pool of 100 exceeded the database's limit; one handler leaked a connectionPool size 20, using for the connection, RDS Proxy for Lambda callers
First request of the morning 6 s on the Lambda pathInit Duration of 4.8 s on cold starts onlyHeavy startup: JSON source generators missing, DI container building every client eagerlyReadyToRun, 1,024 MB, lazy clients, SnapStart on the live alias
Statement endpoint slow for customers with many accounts1 + N SQL statements per request, N up to 40Query inside a loopSingle projected query; command-count warning in logs

The connection arithmetic that should live in every design review:

tasks (or Lambda concurrency) x Max Pool Size  <=  usable database connections - admin reserve
12 tasks x 20                = 240      vs   db.t4g.medium PostgreSQL default max_connections of roughly 400-ish ⚠ VERIFY for your instance class
Lambda at 200 concurrency x 5 = 1,000   ->  needs RDS Proxy or a lower per-function pool

The exact max_connections depends on the engine, parameter group and instance memory; look up the real value with SHOW max_connections; and in the RDS documentation, rather than trusting a rule of thumb. ⚠ VERIFY

Common mistakes

  • Scaling out first. More tasks make connection pressure worse and do nothing for starvation. Diagnose before scaling.
  • .Result, .Wait() and GetAwaiter().GetResult() in request paths. Also Task.Run around async code and async void. Go async end to end.
  • Raising ThreadPool.SetMinThreads as the fix. It can hide the problem and burn memory. Use it as a stopgap while you remove the blocking.
  • Default pool sizes multiplied by many instances. 100 per process looks harmless until the arithmetic is done.
  • Not disposing connections or DbContexts. Leaks show as pool exhaustion that never recovers until restart.
  • new HttpClient() per request. Causes socket exhaustion and stale DNS; use IHttpClientFactory or a shared instance.
  • Lazy-loading proxies in API code. They hide N+1 behind property access.
  • Fixing cold starts with provisioned concurrency before measuring. Measure Init Duration first; most of it is often avoidable initialisation work.
  • Putting I/O in a static constructor of a Lambda function. Errors there fail every invocation of that environment; prefer explicit, logged initialisation.

Troubleshooting

SymptomLikely causeFix
High latency, low CPU, queue length risingThread-pool starvation (sync-over-async, blocking I/O)dotnet-stack report to find blocked frames; make the code async; add timeouts
dotnet-counters shows no dataWrong PID, container without shared diagnostics IPC, or tool version mismatchRun the tool inside the same container (or share /tmp), match tool to runtime
Timeout ... obtaining a connection from the poolPool exhausted by leaks or too few connections for the loadDispose connections, size pool, check slow queries holding connections, add RDS Proxy
too many connections from the databaseSum of pools exceeds server limitReduce pool size or instance count; use RDS Proxy; raise limit only with memory to match
Intermittent socket errors to a partnerPer-request HttpClient or stale connectionsUse the factory, set PooledConnectionLifetime
Slow first request onlyCold start (Lambda INIT, or JIT on a new task)Measure; ReadyToRun, trimming, more memory, SnapStart, warm-up on ECS before registering with the ALB
Init Duration large even with SnapStartSnapshot not applied (using $LATEST), or post-restore work heavyInvoke the published version or alias; move per-invocation work out of init
Many similar SQL statements per requestN+1Count commands, then Include, projection or batch queries
Memory climbs, gen-2 GCs frequentLarge object allocations, caches without boundsdotnet-gcdump, bound caches, check Server vs Workstation GC settings

Cost and cleanup

ItemLabelEstimate
Local reproduction with dotnet-countersfreeRuns on your machine
Lambda function, alias and SnapStartcreditsA few invocations are well within free allowances; SnapStart caching is charged per hour per version for non-Java runtimes ⚠ VERIFY; delete versions after the lab
CloudWatch alarms (2) and EMF metricscreditsPennies for an hour
RDS instance for connection testspaidUse the lab database from Module 6 and stop or delete it; or skip and use local PostgreSQL in Docker

Free alternative: do the whole thread-pool and N+1 reproduction locally. Run PostgreSQL in Docker, set Maximum Pool Size=5, and reproduce exhaustion by holding connections open in a loop. Run dotnet-counters against the local process. Only the SnapStart and alarm parts need an AWS account.

# Remove the stack
cdk destroy PerformanceGuardrailsStack --profile learn-sandbox   # or: terraform destroy

# SnapStart caches snapshots per published version: delete old versions explicitly if you created any by hand.
aws lambda list-versions-by-function --function-name hl-payments-fn --profile learn-sandbox --region us-east-1
aws lambda delete-function --function-name hl-payments-fn --qualifier 1 --profile learn-sandbox --region us-east-1

# Confirm nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --profile learn-sandbox --region us-east-1 --query 'ResourceTagMappingList[].ResourceARN'

Exercises

  • 🟢 Guided (Lab 11.4): Run Starvation/Program.cs. With dotnet-counters attached, send 200 concurrent requests to /bad (use a loop of curl or a small C# load generator), record the queue length and p99 latency, then repeat for /good. Write down the three numbers that differ most.
  • 🟡 Challenge: Add a new endpoint that gets a customer's accounts and their last ten transactions each using the N+1 version, then convert it to a single query. Use DbCommandCounter output to prove the number of commands dropped.
  • 🔴 Live problem: Ticket: "Lambda function hl-payments-fn p99 is 5 seconds on Monday mornings but 120 ms the rest of the week." You have CloudWatch Logs and the code. Write the Logs Insights query, the three hypotheses you will test (cold starts after idle, connection setup, a Monday batch), the evidence for each, and the cheapest fix for each.

Assignment

Write a "performance triage card" for Harbourline on one page: for each of the four failures, the three signals that confirm it, the one command that proves it, the safe first fix and the dangerous first fix to avoid. Add the connection-arithmetic worksheet filled in for your own sandbox. Commit it with the guardrail IaC.

Solutions

Knowledge check

Interview questions

  • How do you tell thread-pool starvation from a slow database? Starvation: low CPU, rising queue length, blocked frames in dotnet-stack, database latency normal. Slow database: queue length normal, traces show long database spans, database metrics and wait events degraded.
  • What would you check if pool-timeout errors appear after a scale-out? The multiplication of instances by pool size against server connections, leaked connections, slow queries holding connections, and whether RDS Proxy is warranted.
  • Name three ways to reduce Lambda .NET cold starts. Increase memory (more CPU), ReadyToRun or Native AOT with trimming, SnapStart on published versions, reduce eager initialisation, provisioned concurrency for strict latency needs.
  • How do you find N+1 in an existing service? Count database commands per request (interceptor or trace span counts), look for repeated statements, and review loops that call the database.

Certification practice

Completion checklist

  • I can explain the four failures and their symptoms in plain language
  • I reproduced thread-pool starvation locally and watched it in dotnet-counters
  • I can do the connection-pool arithmetic for my own deployment
  • I know the SnapStart restrictions and how to read Init Duration
  • I counted database commands per request and fixed an N+1 query
  • I ran the cleanup steps and confirmed nothing is still billing

Key takeaways

  • Most .NET performance failures on AWS are resource-shaped: threads, connections, init time and round trips, not CPU.
  • Starvation looks like high latency with low CPU; async end to end is the fix, thread settings are a stopgap.
  • Pools multiply: instances times pool size must fit under the database limit, or use RDS Proxy.
  • Measure cold starts from Init Duration before choosing a remedy.
  • Count commands per request to catch N+1 before customers do.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page