ASP.NET Core Web API on AWS: the baseline
Prepare an ASP.NET Core API to run behind an Application Load Balancer: Kestrel settings, trusted forwarded headers, TLS terminated at the load balancer, health endpoints, graceful shutdown and the structure of Harbourline's reference Accounts API.
Assumes: .NET 10 LTS (ASP.NET Core 10) · xUnit v3 4.0.3 · Microsoft.AspNetCore.Mvc.Testing 10.0.15 · Application Load Balancer · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Describe the request path from a customer's browser to Kestrel behind an Application Load Balancer (ALB), and which component owns TLS, routing, health checks and client identity at each hop.
- Configure the forwarded headers middleware so the app sees the real client IP and scheme, and only trusts those headers from the load balancer's network.
- Set Kestrel limits and keep-alive values that match the ALB's idle timeout, so you avoid intermittent 502 errors.
- Expose separate liveness and readiness endpoints and explain which one the load balancer should call.
- Make the app shut down gracefully within the orchestrator's stop window, so deployments drop no requests.
- Find your way around Harbourline's reference Accounts API (
labs/m03-accounts-api) and say why each part of its startup pipeline exists.
Prerequisites
- AWS clients and dependency injection
- Public, private and isolated subnets — the ALB lives in public subnets, tasks in private ones
- Certificates, TLS and mTLS is useful background for the TLS section
- Tools: .NET 10 SDK or Docker,
curl
Why this matters in banking
Harbourline's fraud team noticed that every login attempt in the audit log came from the same three IP addresses, all starting with 10.0.. Those were the load balancer's private addresses. For six weeks, the velocity rules that block "too many failed logins from one IP" were blocking the load balancer, and real attackers were invisible. The fix was four lines of configuration. Nobody had added it, because everything "worked" in testing.
The opposite mistake is just as bad. Another team switched forwarded headers on with "trust everything", so anyone on the internet could send X-Forwarded-For: 10.20.30.40 and appear to come from the office network that an internal admin endpoint allowed.
Other baseline problems show up as operational noise: a few 502 errors an hour that nobody can reproduce (keep-alive mismatch), requests dropped at every deployment (no graceful shutdown), and containers restarted in a loop because the health check called the database (liveness and readiness mixed up). Each of these is a small setting. Together they decide whether your API is boring to run, which is the goal.
The idea in plain language
Kestrel is the web server inside every ASP.NET Core app. In AWS you almost never expose Kestrel directly to the internet. A load balancer sits in front of it:
- The ALB accepts HTTPS from customers on port 443, holds the TLS certificate (from AWS Certificate Manager, ACM), and forwards each request over plain HTTP inside the VPC to one of several healthy copies of your app on port 8080.
- Because the ALB opens a new connection to your app, the app sees the ALB as the client. The ALB adds
X-Forwarded-For(the real client IP) andX-Forwarded-Proto(https) headers so the app can recover the original details. - The ALB repeatedly calls a health check URL on each copy and stops sending traffic to copies that fail.
Analogy: the ALB is the bank's reception desk. Customers show ID at reception (TLS), and reception walks them to a free adviser (a healthy target), with a visitor slip that says who they are and how they arrived (forwarded headers). The adviser should only believe slips written by reception, not slips customers bring from home. And before an adviser goes home, reception stops sending them customers, so they can finish the conversations they already have (graceful shutdown).
What problem the baseline solves
Splitting responsibilities between the ALB and the app:
| Concern | Owned by the ALB | Owned by your ASP.NET Core app |
|---|---|---|
| TLS certificate and protocols | ✅ ACM certificate, TLS security policy | Only if you choose end-to-end TLS to targets |
| HTTP → HTTPS redirect | ✅ Listener rule on port 80 | ❌ Do not use UseHttpsRedirection behind the ALB unless headers are trusted |
| Real client IP and scheme | Adds X-Forwarded-For / -Proto / -Port | Applies them via UseForwardedHeaders, trusting only the VPC range |
| Routing to healthy copies | ✅ Target group, health checks | Provides /health/ready |
| Idle connection handling | Idle timeout (default 60 s) | Keep-alive timeout longer than the ALB's |
| Request size and header limits | Some fixed limits | Kestrel MaxRequestBodySize, header timeouts |
| HSTS, security headers, ProblemDetails | ❌ | ✅ |
| Authentication and authorisation | Optional (OIDC on the ALB) | ✅ JWT bearer and policies (later in this module) |
| Draining during deployments | Deregistration delay | HostOptions.ShutdownTimeout, finish in-flight requests |
Limitations to know:
- The ALB does not validate the certificate on an HTTPS target. End-to-end TLS inside the VPC gives encryption in transit, not strong authentication of the target.
- ALB health checks are simple HTTP calls. They cannot tell "the database is slow" from "the app is dead" unless your endpoints make that distinction.
X-Forwarded-Forcan contain values the client made up. Only the rightmost entry, added by your ALB, is trustworthy without other trusted proxies.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| IIS + ARR or Windows NLB in front of web servers | Application Load Balancer + target group | Managed, multi-AZ, scales automatically; billed per hour plus LCUs |
| Azure Application Gateway | Application Load Balancer | Path and host routing, ACM certificates; WAF is a separate attachment (AWS WAF) |
App Service front ends adding X-Forwarded-* | ALB adding X-Forwarded-For, -Proto, -Port | You must enable and scope forwarded-headers processing in the app yourself |
| App Service health check path | Target group health check (path, interval, thresholds) | Health checks decide routing and, in ECS, task replacement |
| App Service managed certificates / Key Vault certificates | AWS Certificate Manager (public certificates free on ALB) | Certificates stay on the ALB; you never install them in the container |
web.config request limits | Kestrel limits in code or configuration | Kestrel runs in the container; there is no IIS layer |
| App Service "Always On" and slot swap draining | Deregistration delay + ECS stop timeout + HostOptions.ShutdownTimeout | Three timers must agree; see the shutdown diagram |
Architecture
Security-boundary view:
- Internet → ALB: TLS terminates at the ALB, with a TLS security policy that allows only TLS 1.2 and 1.3. AWS WAF filters common attacks before the ALB routes anything. Port 80 only redirects.
- ALB → app: inside the VPC. The app's security group allows port 8080 only from the ALB's security group, so nothing else can reach Kestrel or forge forwarded headers. The app trusts
X-Forwarded-*only from the VPC range as a second layer. - App → AWS services: through VPC endpoints or a NAT gateway, signed with the task role's temporary credentials.
What happens during a deployment or scale-in:
Step-by-step implementation
Run the reference API locally
cd labs/m03-accounts-api
dotnet run --project src/Harbourline.Accounts.Api
curl http://localhost:5080/health/live
curl http://localhost:5080/v1/accounts/ACC-0000000001In Development, the reference API uses the in-memory repository and Auth:Mode=Disabled. It refuses to start that way in any other environment unless you explicitly set a learning-shortcut flag. Notice the X-Correlation-ID header on every response and the JSON log lines in the console.
Trust forwarded headers from the VPC only
Configure ForwardedHeadersOptions with ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto, clear the defaults, and add the VPC CIDR as a known network. Call app.UseForwardedHeaders() first in the pipeline.
Set Kestrel limits and keep-alive
Keep KeepAliveTimeout above the ALB idle timeout (60 s by default). Set MaxRequestBodySize to what your API really accepts. Turn off the Server header.
Separate liveness from readiness
/health/live returns 200 if the process is running and has no dependencies. /health/ready checks what the app needs to serve requests. The ALB uses readiness. Container restarts use liveness, or nothing.
Make shutdown graceful
Set HostOptions.ShutdownTimeout a few seconds below the orchestrator's stop timeout. Set the target group's deregistration delay to something realistic for your request durations (30 s for a fast API, not the 300 s default).
Simulate the ALB locally
Run the baseline app with Hosting__TrustedProxyNetworks__0=127.0.0.1/32 and Diagnostics__EnableRequestInfo=true, then:
curl -H "X-Forwarded-For: 203.0.113.50" -H "X-Forwarded-Proto: https" http://localhost:5000/diagnostics/request
# {"scheme":"https","clientIp":"203.0.113.50","host":"localhost:5000"}Remove the trusted network setting and repeat: the scheme becomes http and the client IP becomes 127.0.0.1. The headers are ignored, as they should be from an untrusted source.
AWS Console instructions
You will create a full ALB in Deploying .NET to ECS on Fargate. These are the settings to check on any ALB fronting .NET:
- EC2 → Load Balancers → (your ALB) → Listeners and rules. Port 443 HTTPS with an ACM certificate and a TLS 1.3/1.2 security policy (for example
ELBSecurityPolicy-TLS13-1-2-2021-06; newer policies may exist ⚠ VERIFY the current recommended policy). Port 80 has one default action: Redirect to HTTPS 443, status 301. - (your ALB) → Attributes → Edit.
- Connection idle timeout: 60 s unless you have long downloads.
- Drop invalid header fields: On.
- X-Forwarded-For header: Append (default).
- Desync mitigation mode: Defensive (default) or Strictest.
- EC2 → Target groups → (your TG) → Health checks. Path
/health/ready, success code200, interval 15 s, timeout 5 s, healthy threshold 2, unhealthy threshold 3. - (your TG) → Attributes. Deregistration delay 30 s.
- (your TG) → Targets. Each target should show healthy. If one shows unhealthy, the Health status details column gives the reason, for example
Health checks failed with these codes: [404].
AWS CLI
# Look up your ALB and target group (names from the Fargate lesson).
ALB_ARN=$(aws elbv2 describe-load-balancers --names hl-m03-accounts-alb \
--query "LoadBalancers[0].LoadBalancerArn" --output text --profile learn-sandbox)
TG_ARN=$(aws elbv2 describe-target-groups --names hl-m03-accounts-tg \
--query "TargetGroups[0].TargetGroupArn" --output text --profile learn-sandbox)
# Baseline ALB attributes: idle timeout, drop malformed headers, append client IP to XFF.
aws elbv2 modify-load-balancer-attributes --load-balancer-arn "$ALB_ARN" --profile learn-sandbox --attributes \
Key=idle_timeout.timeout_seconds,Value=60 \
Key=routing.http.drop_invalid_header_fields.enabled,Value=true \
Key=routing.http.xff_header_processing.mode,Value=append
# Health check on the readiness endpoint.
aws elbv2 modify-target-group --target-group-arn "$TG_ARN" --profile learn-sandbox \
--health-check-path /health/ready --health-check-interval-seconds 15 --health-check-timeout-seconds 5 \
--healthy-threshold-count 2 --unhealthy-threshold-count 3 --matcher HttpCode=200
# Drain for 30 s instead of the 300 s default.
aws elbv2 modify-target-group-attributes --target-group-arn "$TG_ARN" --profile learn-sandbox \
--attributes Key=deregistration_delay.timeout_seconds,Value=30
# Why is a target unhealthy?
aws elbv2 describe-target-health --target-group-arn "$TG_ARN" --profile learn-sandbox \
--query "TargetHealthDescriptions[].{Target:Target.Id,State:TargetHealth.State,Reason:TargetHealth.Reason,Detail:TargetHealth.Description}"Infrastructure as Code
The Fargate lesson builds the whole service. Here are just the baseline settings for the ALB, listeners and target group.
// vpc, certificate and the Fargate service are created elsewhere in the stack.
var alb = new ApplicationLoadBalancer(this, "Alb", new ApplicationLoadBalancerProps
{
Vpc = vpc,
InternetFacing = true,
VpcSubnets = new SubnetSelection { SubnetType = SubnetType.PUBLIC },
IdleTimeout = Duration.Seconds(60),
DropInvalidHeaderFields = true,
});
// HTTP only redirects.
alb.AddRedirect(new ApplicationLoadBalancerRedirectConfig { SourcePort = 80, TargetPort = 443 });
var https = alb.AddListener("Https", new BaseApplicationListenerProps
{
Port = 443,
Certificates = [ListenerCertificate.FromCertificateManager(certificate)],
SslPolicy = SslPolicy.RECOMMENDED_TLS,
Open = true,
});
var targets = https.AddTargets("Api", new AddApplicationTargetsProps
{
Port = 8080,
Protocol = ApplicationProtocol.HTTP,
Targets = [fargateService],
DeregistrationDelay = Duration.Seconds(30),
HealthCheck = new Amazon.CDK.AWS.ElasticLoadBalancingV2.HealthCheck
{
Path = "/health/ready",
HealthyHttpCodes = "200",
Interval = Duration.Seconds(15),
Timeout = Duration.Seconds(5),
HealthyThresholdCount = 2,
UnhealthyThresholdCount = 3,
},
});The lab's compiled version of these settings is in labs/m03-accounts-api/infra/cdk/Harbourline.Accounts.Infra/FargateStack.cs. It uses the higher-level ApplicationLoadBalancedFargateService pattern and then adjusts the health check and deregistration delay in the same way.
Complete code
A small, self-contained baseline API that shows every setting in one file. The reference Accounts API in labs/m03-accounts-api applies the same ideas at full size. This project compiles with TreatWarningsAsErrors, and the tests below pass (verified with dotnet test in the .NET 10 SDK container).
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<RootNamespace>Harbourline.Baseline.Api</RootNamespace>
<InvariantGlobalization>true</InvariantGlobalization>
</PropertyGroup>
</Project>{
"Logging": { "LogLevel": { "Default": "Information", "Microsoft.AspNetCore": "Warning" } },
"AllowedHosts": "*",
"Accounts": { "Source": "InMemory" },
"Hosting": { "TrustedProxyNetworks": [ "10.0.0.0/16" ] },
"Diagnostics": { "EnableRequestInfo": false }
}using System.Net;
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.AspNetCore.HttpOverrides;
using Microsoft.Extensions.Diagnostics.HealthChecks;
var builder = WebApplication.CreateBuilder(args);
// 1. Kestrel: HTTP only inside the VPC (TLS ends at the load balancer), small limits, no Server header.
builder.WebHost.ConfigureKestrel(kestrel =>
{
kestrel.AddServerHeader = false;
kestrel.Limits.MaxRequestBodySize = 1 * 1024 * 1024; // 1 MB is plenty for this API
kestrel.Limits.KeepAliveTimeout = TimeSpan.FromSeconds(75); // must be LONGER than the ALB idle timeout (60 s)
kestrel.Limits.RequestHeadersTimeout = TimeSpan.FromSeconds(15);
});
// 2. Graceful shutdown: finish in-flight requests before the orchestrator's SIGKILL (ECS default 30 s).
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(25));
// 3. Trust X-Forwarded-For / X-Forwarded-Proto only from the VPC's address range.
builder.Services.Configure<ForwardedHeadersOptions>(o =>
{
o.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
o.KnownIPNetworks.Clear();
o.KnownProxies.Clear();
foreach (var cidr in builder.Configuration.GetSection("Hosting:TrustedProxyNetworks").Get<string[]>() ?? [])
{
o.KnownIPNetworks.Add(System.Net.IPNetwork.Parse(cidr));
}
});
// 4. Health: liveness has no dependencies; readiness checks what the app needs to serve traffic.
builder.Services.AddHealthChecks()
.AddCheck("self", () => HealthCheckResult.Healthy(), tags: ["live"])
.AddCheck("config", () => string.IsNullOrEmpty(builder.Configuration["Accounts:Source"])
? HealthCheckResult.Unhealthy("Accounts:Source is not configured.")
: HealthCheckResult.Healthy(), tags: ["ready"]);
// 5. Errors as RFC 9457 ProblemDetails; HSTS for browsers that reach us over HTTPS.
builder.Services.AddProblemDetails();
builder.Services.AddHsts(o => { o.MaxAge = TimeSpan.FromDays(365); o.IncludeSubDomains = true; });
var app = builder.Build();
app.UseForwardedHeaders(); // first: everything after it sees the real client IP and scheme
app.UseExceptionHandler();
app.UseStatusCodePages();
if (!app.Environment.IsDevelopment())
{
app.UseHsts(); // only emitted when Request.IsHttps, i.e. X-Forwarded-Proto was trusted
}
app.MapHealthChecks("/health/live", new HealthCheckOptions { Predicate = r => r.Tags.Contains("live") });
app.MapHealthChecks("/health/ready", new HealthCheckOptions { Predicate = r => r.Tags.Contains("ready") });
app.MapGet("/v1/accounts/{accountId}", (string accountId) =>
accountId == "ACC-0000000001"
? Results.Ok(new { accountId, currency = "GBP", product = "current", status = "open", balance = 1250.75m })
: Results.Problem(statusCode: 404, title: "Account not found"));
// Troubleshooting aid for load-balancer set-ups. Off unless explicitly enabled.
if (app.Configuration.GetValue<bool>("Diagnostics:EnableRequestInfo"))
{
app.MapGet("/diagnostics/request", (HttpContext ctx) => new
{
scheme = ctx.Request.Scheme,
clientIp = ctx.Connection.RemoteIpAddress?.ToString(),
host = ctx.Request.Host.Value,
});
}
app.Run();
public partial class Program;The tests prove the security-critical behaviour: headers are honoured from the VPC range and ignored from anywhere else.
using System.Net;
using System.Net.Http.Json;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.Extensions.DependencyInjection;
namespace Harbourline.Baseline.Api.Tests;
public class ForwardedHeaderTests
{
private static CancellationToken Ct => TestContext.Current.CancellationToken;
/// <summary>TestServer has no TCP connection, so pretend the caller is a given address (the ALB or an attacker).</summary>
private sealed class FakeRemoteIp(string ip) : IStartupFilter
{
public Action<IApplicationBuilder> Configure(Action<IApplicationBuilder> next) => app =>
{
app.Use((ctx, nextMiddleware) =>
{
ctx.Connection.RemoteIpAddress = IPAddress.Parse(ip);
return nextMiddleware(ctx);
});
next(app);
};
}
private sealed record RequestInfo(string Scheme, string? ClientIp, string Host);
private static HttpClient ClientFrom(string remoteIp)
{
var factory = new WebApplicationFactory<Program>().WithWebHostBuilder(b =>
{
b.UseSetting("Diagnostics:EnableRequestInfo", "true");
b.ConfigureServices(s => s.AddSingleton<IStartupFilter>(new FakeRemoteIp(remoteIp)));
});
return factory.CreateClient();
}
private static HttpRequestMessage Forwarded() => new(HttpMethod.Get, "/diagnostics/request")
{
Headers = { { "X-Forwarded-For", "203.0.113.50" }, { "X-Forwarded-Proto", "https" } },
};
[Fact]
public async Task Headers_from_the_load_balancer_subnet_are_trusted()
{
var response = await ClientFrom("10.0.1.25").SendAsync(Forwarded(), Ct);
var info = await response.Content.ReadFromJsonAsync<RequestInfo>(Ct);
Assert.Equal("https", info!.Scheme);
Assert.Equal("203.0.113.50", info.ClientIp);
}
[Fact]
public async Task Headers_from_anywhere_else_are_ignored()
{
var response = await ClientFrom("198.51.100.7").SendAsync(Forwarded(), Ct);
var info = await response.Content.ReadFromJsonAsync<RequestInfo>(Ct);
Assert.Equal("http", info!.Scheme);
Assert.Equal("198.51.100.7", info.ClientIp);
}
[Fact]
public async Task Health_and_problem_details_work()
{
using var factory = new WebApplicationFactory<Program>();
var client = factory.CreateClient();
Assert.Equal(HttpStatusCode.OK, (await client.GetAsync("/health/live", Ct)).StatusCode);
Assert.Equal(HttpStatusCode.OK, (await client.GetAsync("/health/ready", Ct)).StatusCode);
var missing = await client.GetAsync("/v1/accounts/ACC-0000000999", Ct);
Assert.Equal(HttpStatusCode.NotFound, missing.StatusCode);
Assert.Equal("application/problem+json", missing.Content.Headers.ContentType?.MediaType);
Assert.Equal(HttpStatusCode.NotFound, (await client.GetAsync("/diagnostics/request", Ct)).StatusCode);
}
}The test project file is the same shape as in the previous lesson: xUnit v3 4.0.3, Microsoft.AspNetCore.Mvc.Testing 10.0.15, OutputType Exe, and a project reference to the API.
Code explained line by line
Kestrel (block 1)
AddServerHeader = falsestops Kestrel from sendingServer: Kestrel. That gives scanners one less fingerprint, and many bank penetration-test reports flag the header.MaxRequestBodySize = 1 MB. The default is about 28.6 MB. An account API never needs that, and smaller limits reduce the cost of abusive requests. Set larger limits per endpoint ([RequestSizeLimit]) where uploads are expected.KeepAliveTimeout = 75 s. AWS recommends that the target's idle timeout is longer than the ALB's (60 s by default). Otherwise Kestrel may close an idle connection at the moment the ALB reuses it, and the customer gets a 502. Kestrel's default (130 s) already satisfies this. Setting it explicitly documents the dependency: if someone raises the ALB idle timeout to 120 s for slow exports, this value must rise too.RequestHeadersTimeout = 15 slimits slow-header attacks. The ALB also protects you here, but defence in depth is cheap.
Shutdown (block 2)
HostOptions.ShutdownTimeout = 25 s. When the container receives SIGTERM, the .NET host stops accepting new connections and waits up to this long for in-flight requests and hosted services. The ECS stop timeout defaults to 30 s, after which the container is killed. 25 s leaves a margin for logging to flush.
Forwarded headers (block 3)
ForwardedHeaders = XForwardedFor | XForwardedProtoprocesses the two headers you need: client IP for audit and fraud rules, scheme for HSTS, redirects and absolute links.X-Forwarded-Hostis deliberately not trusted, because host header injection is a known attack.KnownIPNetworks.Clear(); KnownProxies.Clear();removes the defaults (loopback), so trust is exactly what configuration says. In .NET 10,KnownIPNetworks(based onSystem.Net.IPNetwork) is the property to use; the olderKnownNetworksis obsolete.KnownIPNetworks.Add(IPNetwork.Parse(cidr))trusts the VPC range (10.0.0.0/16). The ALB's nodes have private addresses inside the VPC, so only connections from inside the VPC can set the client IP. Combined with a security group that only admits the ALB, that is two layers.ForwardLimitis left at its default of 1. The middleware reads only the rightmostX-Forwarded-Forentry, the one your ALB appended. If a client sends its ownX-Forwarded-For: 10.20.30.40, the ALB appends the real address after it, and the middleware still picks the real one. RaiseForwardLimitonly if you add another trusted proxy in front, such as CloudFront, and then trust that proxy's ranges too.
Health (block 4)
"self"taggedlivehas no dependencies. If it fails, the process is broken and a restart helps."config"taggedreadystands in for real dependency checks (the reference API checks DynamoDB). If a dependency is down, a restart does not help, so readiness failing should take the instance out of the load balancer, not kill it.
Errors and HSTS (block 5)
AddProblemDetails()+UseExceptionHandler()+UseStatusCodePages()make every error, including a bare 404, an RFC 9457application/problem+jsonbody with no stack trace.UseHsts()addsStrict-Transport-Securityonly to responses for HTTPS requests. Behind the ALB the request is HTTP, so without trustedX-Forwarded-Protothe header would never be sent. This is a good self-test that forwarded headers work.
Pipeline order
UseForwardedHeaders()first. Anything that reads the client IP or scheme (logging, rate limiting, HSTS, authentication redirects) must run after it.- Health endpoints are anonymous and mapped before business endpoints. In the reference API they are explicitly
.AllowAnonymous()because the business endpoints require authorisation by default. /diagnostics/requestexists only when enabled. It is a real troubleshooting aid for load balancer set-ups, but it reveals network details, so it is off by default and must never be enabled in production.
Tests
FakeRemoteIp(anIStartupFilter) setsRemoteIpAddressbefore the app's own middleware, because the in-memoryTestServerhas no TCP connection. It lets one test act as "the ALB" (10.0.1.25) and another as "an attacker on the internet" (198.51.100.7).- The two forwarded-header tests are the regression tests that would have caught both incidents from the introduction.
Tour of the reference Accounts API
Open labs/m03-accounts-api/src/Harbourline.Accounts.Api/Program.cs. Its numbered sections map onto this module:
Step in Program.cs | What it does | Lesson that goes deeper |
|---|---|---|
1. AddAwsConfigurationSources() | appsettings → environment → Parameter Store → AppConfig | Configuration (lesson 10) |
| 2. Serilog to stdout as compact JSON | Every host ships stdout to CloudWatch Logs | Logging and errors (lesson 14) |
3. UsePlatformPort(), AddAWSLambdaHosting(...), AddTrustedForwardedHeaders(...) | Same binary runs on Beanstalk (port from PORT), in containers (8080) and in Lambda | This lesson and Hosting options compared |
4. AddDefaultAWSOptions(...) | SDK defaults from the AWS section | AWS clients and DI |
| 5. Data, pseudonymisation, auth, health checks | IAccountRepository (in-memory or DynamoDB), HMAC pseudonyms for logs, Cognito JWT, live/ready | Secrets, auth and health-check lessons |
6. ProblemDetails + GlobalExceptionHandler | RFC 9457 errors carrying the correlation ID | Logging and errors |
| 7. JSON options | camelCase properties, snake_case enum values (credit_card) | CONVENTIONS.md §2 |
| Middleware order | UseForwardedHeaders → correlation ID → request logging → exception handler → authentication → authorisation | This lesson |
The lessons named by number are in this module's overview.
Security considerations
⚠ Learning shortcut — not for production
The baseline app and the lab's Fargate stack terminate TLS at the ALB and send plain HTTP to port 8080 inside the VPC. Many banks accept this when the private subnets, security groups and VPC flow logs are tightly controlled. If your PCI DSS scope or internal policy requires encryption between the load balancer and targets, use an HTTPS target group with a certificate in the container, remembering that the ALB does not validate target certificates. Discuss the choice with your security and compliance teams; do not assume either answer.
- Never use
ASPNETCORE_FORWARDEDHEADERS_ENABLED=trueon its own behind an internet-facing ALB. It turns on forwarded-header processing with the known-proxy lists cleared, meaning any caller that can reach Kestrel is trusted. It is meant for platforms where only the proxy can reach the app. Use explicitKnownIPNetworksinstead. - Security groups first, forwarded headers second. If Kestrel is reachable only from the ALB's security group, nobody else can forge headers at all.
- Turn on "drop invalid header fields" on the ALB, so malformed header names never reach Kestrel.
- Use a modern TLS policy on the HTTPS listener and redirect port 80. Add AWS WAF in front of internet-facing banking APIs (WAF and Shield).
- No stack traces to clients. ProblemDetails with a correlation ID is enough for support to find the log line.
- Health endpoints must not leak. Return statuses, not exception messages, hostnames or table names. The reference API's
WriteAsyncreturns only status names and a duration. - Log the forwarded client IP, then think about retention. IP addresses are personal data under UK GDPR. Keep them for as long as your fraud and security purposes need, with documented retention, rather than forever.
Banking use case
Harbourline's Open Banking Account Information API serves regulated third-party providers. Its baseline:
- An internet-facing ALB with AWS WAF and a TLS 1.2/1.3 policy. Mutual TLS for third-party certificates is handled in front of the API (see Certificates, TLS and mTLS).
- Kestrel trusts
X-Forwarded-Foronly from the VPC range, and the API's rate limiter partitions by the real client IP and by the provider's client ID. /health/readychecks DynamoDB with a 2-second timeout. During a DynamoDB incident, targets are marked unhealthy and the ALB returns 503, which the providers' clients are built to retry. Containers are not restarted, because restarting would not fix DynamoDB.- Deployments drain over 30 seconds. Monitoring shows zero 5xx responses during the weekly release window, which the operational resilience team uses as evidence for the API's availability target.
Common mistakes
- Not calling
UseForwardedHeaders→ every request appears to come from the ALB → audit logs and fraud rules see one IP → configure and call it first. - Trusting all proxies (
ASPNETCORE_FORWARDEDHEADERS_ENABLED, or clearingKnownIPNetworkswithout adding any) → clients can spoof their IP → trust only the VPC range. UseHttpsRedirectionbehind the ALB without trusted forwarded headers → every request looks like HTTP → infinite redirect loop → redirect at the ALB listener instead.- Keep-alive shorter than the ALB idle timeout → a few random 502s an hour → keep Kestrel's timeout above the ALB's.
- Health check on
/or on an endpoint that needs authentication → 401/404 → targets never become healthy → use an anonymous/health/ready. - Database check on the liveness endpoint → a database blip restarts every container at once → dependencies belong in readiness only.
- Deregistration delay of 300 s with an ECS stop timeout of 30 s → slow deployments, and requests longer than 30 s killed anyway → align the three timers.
- Binding Kestrel to
localhostin the container → the ALB cannot connect → listen on all interfaces (the official images setASPNETCORE_HTTP_PORTS=8080, which does this).
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Logs show 10.0.x.x as the client IP for every request | Forwarded headers not processed, or the ALB subnet is not in KnownIPNetworks | Add the VPC CIDR; call UseForwardedHeaders() first |
Infinite redirect (ERR_TOO_MANY_REDIRECTS) | UseHttpsRedirection sees HTTP because X-Forwarded-Proto is not trusted | Trust the headers or remove app-level redirection |
| Occasional 502 Bad Gateway with healthy targets | App closes idle connections before the ALB does | Kestrel KeepAliveTimeout above the ALB idle timeout |
| 504 Gateway Timeout | A request ran longer than the ALB idle timeout with no data sent | Speed up the request, make it asynchronous (202 + polling) or raise both timeouts |
Targets unhealthy, reason Health checks failed with these codes: [404] | Wrong health check path, or the endpoint requires authorisation | Point at /health/ready and make it anonymous |
Targets unhealthy, reason Request timed out | Security group does not allow the ALB, the app listens on the wrong port, or readiness is too slow | Allow 8080 from the ALB security group; check ASPNETCORE_HTTP_PORTS; time-box the dependency check |
| 5xx spikes during every deployment | No draining or shutdown too short | Deregistration delay plus ShutdownTimeout below the stop timeout |
| 413 Payload Too Large | MaxRequestBodySize too small for that endpoint | Raise it per endpoint with [RequestSizeLimit] |
| HSTS header missing in production | Request.IsHttps is false because X-Forwarded-Proto is not trusted | Same fix as the client-IP problem |
Cost and cleanup
This lesson is free: everything runs locally and in tests.
If you apply the CLI or IaC settings to the ALB from the Fargate lesson, they cost nothing extra. The ALB itself costs USD 0.0225 per hour plus LCU charges in us-east-1 (new accounts get 750 ALB hours per month in the free tier). Follow that lesson's teardown: deleting the stack removes the ALB, listeners and target group.
# Check no load balancer from this module is still running.
aws elbv2 describe-load-balancers --profile learn-sandbox --region us-east-1 \
--query "LoadBalancers[?starts_with(LoadBalancerName, 'hl-m03')].LoadBalancerName"Exercises
- 🟢 Guided: Run the baseline API locally, simulate the ALB with
curlas in the last step, and confirm the scheme and client IP change only when127.0.0.1/32is trusted. Then call/health/readywithAccounts__Source=""and see readiness fail while liveness still passes. - 🟡 Challenge: Add ASP.NET Core rate limiting (
AddRateLimiter) with a fixed window of 100 requests per minute per client IP, and write a test proving that two different forwarded IPs from the ALB range get separate limits, while a spoofedX-Forwarded-Forfrom outside the range does not. - 🔴 Live problem: After moving the Accounts API from an on-premises IIS farm to ECS, the fraud team reports that the "new device from a new country" rule fires for nearly every login, and the operations team sees about ten 502 errors per hour. No code changed during the move. Investigate and write up the fixes.
Assignment
Write an ADR (architecture decision record) for Harbourline titled "TLS termination for internal APIs". Compare TLS terminated at the ALB with HTTP to targets against end-to-end TLS to targets. Cover security, certificate management, PCI DSS scope discussion points, operational effort and cost. State a recommendation and the conditions under which you would change it. Put it in your portfolio repository's docs/adr/ folder.
Solutions
# Trusted: headers honoured.
Hosting__TrustedProxyNetworks__0=127.0.0.1/32 Diagnostics__EnableRequestInfo=true dotnet run
curl -H "X-Forwarded-For: 203.0.113.50" -H "X-Forwarded-Proto: https" http://localhost:5000/diagnostics/request
# {"scheme":"https","clientIp":"203.0.113.50",...}
# Untrusted (default 10.0.0.0/16 only): headers ignored.
Diagnostics__EnableRequestInfo=true dotnet run
# {"scheme":"http","clientIp":"127.0.0.1",...}
# Readiness fails, liveness passes.
Accounts__Source= dotnet run
curl -i http://localhost:5000/health/ready # 503 Unhealthy
curl -i http://localhost:5000/health/live # 200 HealthyIn PowerShell, set variables with $env:Hosting__TrustedProxyNetworks__0 = "127.0.0.1/32" before dotnet run. Why: the double underscore maps to the : section separator, and __0 addresses the first array element. This is exactly how ECS task definitions and Beanstalk environment properties override configuration.
using System.Threading.RateLimiting;
builder.Services.AddRateLimiter(o =>
{
o.RejectionStatusCode = StatusCodes.Status429TooManyRequests;
o.GlobalLimiter = PartitionedRateLimiter.Create<HttpContext, string>(ctx =>
RateLimitPartition.GetFixedWindowLimiter(
ctx.Connection.RemoteIpAddress?.ToString() ?? "unknown", // real client IP after UseForwardedHeaders
_ => new FixedWindowRateLimiterOptions { PermitLimit = 100, Window = TimeSpan.FromMinutes(1) }));
});
// ...
app.UseForwardedHeaders();
app.UseRateLimiter(); // after forwarded headers, so partitions use the real client IPTest idea: use FakeRemoteIp("10.0.1.25") and send 101 requests with X-Forwarded-For: 203.0.113.1, expect the 101st to be 429. Then send one request with X-Forwarded-For: 203.0.113.2 and expect 200. Finally use FakeRemoteIp("198.51.100.7") and show that changing X-Forwarded-For does not reset its limit, because the header is ignored and the partition key stays 198.51.100.7.
Why: a rate limiter keyed on an IP you cannot trust is either useless (everyone shares the ALB's IP) or bypassable (attackers rotate a fake header). Ordering and trust settings make it meaningful. In production, AWS WAF rate-based rules add a second limit at the edge.
- Fraud rule: on IIS, ARR or the hardware load balancer set the client IP that the app read from a server variable. In ECS nothing processes
X-Forwarded-For, so every login appears to come from an ALB node IP. GeoIP maps those to the AWS Region, which is a "new country" for every customer. Fix: configureForwardedHeadersOptionswith the VPC CIDR and callUseForwardedHeaders()first. Add the forwarded-header regression tests from this lesson. - 502s: IIS had its own idle timeouts. Check
HTTPCode_ELB_502_CountagainstTargetConnectionErrorCountin CloudWatch, and the ALB access logs fortarget_status_code = -withelb_status_code = 502. Likely causes: a keep-alive mismatch (Kestrel's timeout lowered in configuration below 60 s), or tasks killed during scale-in without draining. Fix: Kestrel keep-alive above the ALB idle timeout, a 30 s deregistration delay,ShutdownTimeout25 s. - Verify: replay a day of traffic in a test environment. Confirm the 502 count is zero, and that the fraud rule's input IPs match a sample of known customer IPs from the old system.
- Write-up: root cause, why tests did not catch it (no forwarded-header tests; load tests ran directly against tasks, not through the ALB), and the new tests and runbook steps.
Knowledge check
Answer: The ALB opens its own connection to the target, so the TCP peer is an ALB node. The original client IP arrives only in the X-Forwarded-For header, which ASP.NET Core applies only when the forwarded headers middleware is configured and the sender is trusted.
Answer: The client's real IP. The ALB appends the real address after the client's value, and the middleware takes only the rightmost entry from a trusted proxy. The fake value is ignored.
Answer: Readiness (/health/ready). It reflects whether the instance can serve traffic, so the ALB can route around it. Liveness ignores dependencies, so routing on it would send traffic to instances that cannot reach the database; and restarting on readiness would kill healthy processes during a dependency outage.
Answer: The target group's deregistration delay (drain time), the app's HostOptions.ShutdownTimeout (time to finish after SIGTERM) and the container stop timeout (time before SIGKILL, 30 s by default on ECS). Shutdown must finish before the stop timeout.
Answer: If Kestrel closes an idle keep-alive connection before the ALB does, the ALB may send a request on a connection that is being closed and return a 502. Kestrel's keep-alive timeout should be longer than the ALB idle timeout (60 s by default).
Interview questions
- "How do you get the client IP in ASP.NET Core behind an AWS load balancer?" — Configure
ForwardedHeadersOptionsforX-Forwarded-ForandX-Forwarded-Proto, trust only the VPC range or the ALB's subnets, callUseForwardedHeaders()first, and restrict the app's security group to the ALB. Never trust all proxies. - "Where do you terminate TLS for an API on ECS, and what are the trade-offs?" — Usually at the ALB with an ACM certificate (free, automatic renewal, central TLS policy). End-to-end TLS to targets adds encryption inside the VPC for stricter policies, but the ALB does not validate target certificates and you must manage them in the container.
- "What is the difference between liveness and readiness checks?" — Liveness: is the process alive (restart if not). Readiness: can it serve traffic now (route around it if not). Dependencies belong in readiness only.
- "How do you avoid dropped requests during deployments?" — Deregistration delay to drain, a graceful shutdown timeout below the stop timeout, minimum healthy percent at 100% during rolling updates, and a deployment circuit breaker to roll back bad releases.
Certification practice
An ASP.NET Core application runs on EC2 instances behind an Application Load Balancer. The application logs show the same few private IP addresses for all requests, so an IP-based rate limit blocks legitimate users. What should the developer do?
A. Enable proxy protocol v2 on the Application Load Balancer.
B. Read the client IP from the X-Forwarded-For header, using the forwarded headers middleware configured to trust only the load balancer's network.
C. Move the instances to public subnets.
D. Replace the ALB with a Network Load Balancer and keep the code unchanged.
Answer: B. ALBs pass the client IP in X-Forwarded-For. Proxy protocol is an NLB feature, not an ALB one (A). Public subnets do not change what the ALB sends and weaken security (C). D changes the architecture for no reason; an NLB can preserve client IPs, but the ALB's features (path routing, WAF) would be lost.
During each rolling deployment of an ECS service behind an ALB, clients receive a burst of 5xx errors. The target group uses the default settings and the application exits immediately on SIGTERM. Which combination reduces the errors? (Choose TWO.)
A. Configure the application to stop accepting new connections and finish in-flight requests on SIGTERM, within the container stop timeout. B. Disable ALB health checks. C. Set an appropriate deregistration delay on the target group so in-flight requests can complete. D. Increase the number of ALB listeners. E. Use a larger Fargate task size.
Answer: A and C. Draining at the load balancer plus graceful shutdown in the app removes the errors. Disabling health checks (B) would route to broken tasks. More listeners (D) and bigger tasks (E) do not address request draining.
A company runs a web API on ECS Fargate tasks in private subnets behind an internet-facing ALB. The security team requires that the tasks accept traffic only from the ALB and that HTTP requests are upgraded to HTTPS. What should a solutions architect do?
A. Allow port 8080 on the tasks' security group from 0.0.0.0/0 and redirect to HTTPS in the application. B. Allow port 8080 on the tasks' security group only from the ALB's security group, and add an HTTP listener on the ALB with a redirect action to HTTPS. C. Put the tasks in public subnets with a network ACL that blocks port 80. D. Use a Network Load Balancer with TLS passthrough.
Answer: B. Security-group referencing limits the source to the ALB, and the listener redirect upgrades HTTP at the edge. A exposes the tasks to anyone who can route to them. C exposes the tasks publicly. D does not provide HTTP redirects.
Completion checklist
- I can draw the request path and say where TLS ends and who adds which headers
- My forwarded-header tests pass for both trusted and untrusted callers
- I can explain the three shutdown timers and the keep-alive rule
- I separated liveness and readiness and know which one the ALB calls
- I ran the reference Accounts API locally and traced its startup pipeline
Key takeaways
- The ALB terminates TLS and forwards plain HTTP inside the VPC. The app learns the real client and scheme from
X-Forwarded-For/-Proto, which it must trust only from the VPC range. - Security groups that admit only the ALB are the first control. Forwarded-header trust lists are the second.
- Keep Kestrel's keep-alive above the ALB idle timeout to avoid random 502s.
- Readiness for routing, liveness for restarts, and never put dependencies in liveness.
- Align deregistration delay, ShutdownTimeout and the stop timeout for deployments with zero dropped requests.
Further reading
- HTTP headers and Application Load Balancers
- Edit attributes for your Application Load Balancer — idle timeout and the keep-alive recommendation
- Configure ASP.NET Core to work with proxy servers and load balancers
- Health checks in ASP.NET Core
- Previous: AWS clients and dependency injection · Next: Hosting options compared
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Lab 3.1: an SDK v4 console app with SSO and dependency injection
Build a .NET 10 console app that registers AWS SDK v4 clients through dependency injection, signs in with the learn-sandbox SSO profile, calls STS GetCallerIdentity, lists S3 buckets page by page, and is unit tested with a mocked client.
Hosting options for .NET on AWS compared
Compare EC2 (Linux and Windows), Elastic Beanstalk, ECS on Fargate, ECS on EC2 and Managed Instances, EKS, Lambda and ECS Express Mode for .NET workloads on cost, operations, scaling, cold starts and compliance, and know why App Runner is no longer an option for new accounts.