Playbook: Lambda timeouts and cold starts
Tell a Lambda timeout from a cold start from throttling using REPORT lines and Logs Insights, find the slow phase (init, your code or a downstream call), and fix it with deadline-aware code, right-sized memory and the right start-up option for a .NET function.
Assumes: .NET 10 LTS (Lambda dotnet10 runtime) · AWS SDK for .NET v4 · AWS CLI v2 · Amazon.Lambda.Core · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this playbook you can:
- Read a Lambda
REPORTline and say whether an invocation was a cold start, ran close to its timeout, or came close to its memory limit. - Separate the three time buckets of an invocation: init, your code and downstream calls.
- Use CloudWatch Logs Insights to measure cold-start frequency, p99 duration and timeouts over time.
- Diagnose the most common .NET causes: missing network route from a VPC function, unbounded downstream calls, low memory (low CPU), heavy start-up work and a concurrency limit.
- Choose the right remedy for cold starts: memory, ARM64, lazy initialisation, ReadyToRun, SnapStart, Native AOT or provisioned concurrency, knowing which combinations are impossible.
- Write deadline-aware code that fails fast with a clear error before Lambda kills the invocation.
- Line up timeouts across API Gateway, Lambda and SQS so retries do not cause duplicates.
Prerequisites
- Lambda with C#, which covers the programming models and the cold-start options in depth
- CloudWatch Logs Insights queries and Performance bottlenecks in .NET
- Retry strategies and the first playbook, duplicate payment on client retry
- Tools: AWS CLI v2 with the
learn-sandboxprofile; .NET 10 SDK for the helpers and tests (no AWS needed)
Why this matters in banking
Harbourline's GetAccountSummary function backs the mobile app's home screen. After a release, a handful of customers see a blank screen for about ten seconds, then an error. The dashboard shows p50 latency of 60 ms but a p99 of 9 seconds, and a few Task timed out after 10.00 seconds lines. The first reaction is "Lambda is slow", the second is "set the timeout to 15 minutes". The first is vague and the second hides a real fault and raises costs, and a long timeout also makes the retry behaviour of upstream callers worse: API Gateway has already given up, the client retried, and the second request runs while the first is still going.
A timeout is a symptom. The method is to separate init from handler time from downstream time, and then decide whether the fix is code, configuration or a start-up technology.
The idea in plain language
A Lambda invocation has two possible phases:
- Init (cold start): only when Lambda must create a new execution environment: download your code, start the .NET runtime, run your static initialisers and constructor code. This costs time once per environment.
- Invoke: run your handler. This happens for every request, in a warm environment.
Lambda reuses an environment for later requests, but each environment handles one request at a time. When ten requests arrive at once, Lambda needs ten environments, so up to ten cold starts. That is why cold starts show up as the tail (p99) rather than the average, and why they appear after idle periods, deployments and traffic bursts.
A timeout is a separate thing: the handler ran for longer than the configured limit, and Lambda stopped it.
Analogy: A restaurant with one waiter per table. If a table is already seated (warm environment), service is fast. If a party arrives and no table is ready, the staff must lay a table first (cold start). If the kitchen takes forty minutes to cook a dish (a slow downstream call), the guest leaves when patience runs out (the timeout). Adding tables fixes the first, a faster kitchen fixes the second, and telling the guest "we'll wait all night" fixes neither.
The three questions
| Question | Where to look | Healthy sign |
|---|---|---|
| Is it cold-start time? | Init Duration on REPORT lines, X-Ray "Initialization" segment | Few invocations have it, and it is under 1 to 2 seconds |
| Is it handler time? | Duration on REPORT lines, your own stage timers | p99 far below the timeout |
| Is it waiting for a downstream? | X-Ray subsegments, your logs around outbound calls | Each downstream call has its own timeout |
What problem the AWS service solves
Lambda removes servers and scales out by creating execution environments on demand. The trade-offs that matter here:
- Timeout: configurable up to 900 seconds (15 minutes), default 3 seconds. Pick the smallest value that covers the p99.9 of normal work plus a margin.
- Memory and CPU: memory is configurable from 128 MB up to 10,240 MB, and CPU scales with memory. A function with 1,769 MB gets about one vCPU (⚠ VERIFY current figure). A .NET function at 128 MB is usually painfully slow in init and in handler.
- Concurrency: account-level default of 1,000 concurrent executions per Region (new accounts may be lower, ⚠ VERIFY). Reserved concurrency can cap a function, and a low cap causes throttles (HTTP 429 for synchronous callers).
- Scaling speed: Lambda adds environments at a limited rate when traffic bursts (documented per function, ⚠ VERIFY), so a very sharp spike produces queued or throttled requests even below the account limit.
- Init limits and billing: the init phase has a time limit (10 seconds for standard on-demand init, ⚠ VERIFY), and init time is billed for on-demand functions (billing for the init phase was standardised in 2025, ⚠ VERIFY current rules). Do not treat init as free.
- VPC: attaching a function to a VPC no longer adds a large per-cold-start network penalty (⚠ VERIFY), but the function needs a route to everything it calls: NAT gateway or VPC endpoints. No route means calls hang until the timeout.
- Upstream limits: API Gateway has its own integration timeout (29 seconds by default for REST APIs, with an increase available for Regional REST APIs since 2024; 30 seconds maximum for HTTP APIs; ⚠ VERIFY both).
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
Azure Functions functionTimeout in host.json (Consumption default 5 min, max 10 min) | Lambda timeout (default 3 s, max 15 min) | Per function setting, and the default is very short on purpose |
| Consumption plan cold start; Premium plan pre-warmed instances | Cold starts; provisioned concurrency keeps environments initialised | Provisioned concurrency is billed while it is configured |
| Flex Consumption "always ready" instances | Provisioned concurrency | Same idea: pay to avoid cold starts |
| Azure Functions scale controller | Lambda concurrency model: one request per environment | There is no multiple-requests-per-instance setting. More concurrency means more environments |
| Application Insights end-to-end duration | X-Ray trace and REPORT line | REPORT has Duration, Billed Duration, Memory and Init Duration in one line |
CancellationToken from the Functions host | ILambdaContext.RemainingTime | Lambda does not cancel your token for you. You create one from the remaining time |
| ReadyToRun and trimming | ReadyToRun, trimming, Native AOT, SnapStart | SnapStart snapshots an initialised environment (published versions only) |
Architecture
Where time goes in one invocation
Triage flow
Step-by-step implementation
Minutes 0 to 5: scope and stop the bleeding
- Is it all invocations or some? Look at the error rate and p99 by time (CloudWatch metrics:
Errors,Throttles,Duration,ConcurrentExecutions). - Did something change? A deployment, a new dependency, a memory or timeout change, a new VPC or security group, a traffic spike (marketing campaign, payday).
- If a bad release is the cause, roll back through the alias (point
liveto the previous version). That is the fastest, safest move, and it keeps the evidence (the bad version still exists). - Do not raise the timeout to 15 minutes as a first move. It may mask the fault, raise cost and make upstream retries pile up.
Minutes 5 to 20: measure with REPORT lines
Every invocation ends with a REPORT line. Use Logs Insights to get four numbers: how many invocations were cold, p99 duration, timeouts, and memory use against the limit. Queries are in the CLI section. Then decide using the triage flow.
Minutes 20 to 40: if it is a timeout, find what the handler was waiting for
Typical causes, most common first:
| Cause | Evidence | Fix |
|---|---|---|
| No route from a VPC function to a service (DynamoDB, Secrets Manager, partner) | Calls hang, then TaskCanceledException/timeout; same code works outside a VPC | Add a NAT gateway route, or VPC endpoints (gateway endpoint for S3 and DynamoDB; interface endpoints for others) |
| No client timeout, or retries that use the whole budget | Duration equals the function timeout, the downstream is slow, SDK retries repeat | Set per-call timeouts shorter than the function's, limit retries, use a deadline derived from RemainingTime |
| Slow downstream (partner, database under load) | X-Ray shows one subsegment taking most of the time | Fix or protect the downstream; see Partner API timeouts |
| Too little memory (CPU starved) | Max Memory Used near Memory Size, or high duration that falls when memory rises | Raise memory and re-measure. Cost per request often stays flat or drops |
| Batch too big (SQS, Kinesis) | Duration grows with batch size and hits the timeout | Reduce batch size, process in parallel inside the budget, report partial failures |
| Connection storm to a database | Many environments each open connections; DB errors rise with concurrency | Reuse connections across invocations, use RDS Proxy or a data API, cap concurrency |
| Lock or contention | Duration spikes with specific records | Fix the data access pattern |
Minutes 40 to 60: if it is cold starts, choose the lightest effective fix
Climb this ladder only as far as you need. Measure after each rung.
- Memory and architecture: more memory means more CPU, which shortens init. ARM64 is usually cheaper per unit of work.
- Trim init: build the DI container lightly, create AWS SDK clients once as static or singleton objects, do not fetch secrets or call remote services in a static constructor on the hot path (cache lazily on first use, or use the extension with caching), avoid reflection-heavy startup.
- Compile earlier: ReadyToRun reduces JIT time; trimming shrinks the package.
- SnapStart: snapshot after init for published versions; restores quickly. Not compatible with provisioned concurrency or Native AOT, and unique values created during init (random seeds, connection state) must be re-created on restore.
- Native AOT: the fastest start but the most restrictive code (no unrestricted reflection); uses the OS-only runtime.
- Provisioned concurrency: keeps a set number of environments initialised. Costs money while configured. Use it for the peak profile of a customer-facing function, ideally scheduled.
The details and trade-offs are in Lambda with C#, in the section on cold starts.
Minutes 60 onwards: make time a first-class concern
Add deadline-aware code (below), alarms on p99 near the timeout and on throttles, and align timeouts end to end: caller timeout greater than the function timeout, function timeout greater than the sum of its downstream budgets, and for SQS the queue's visibility timeout at least six times the function timeout (AWS recommendation, ⚠ VERIFY).
AWS Console instructions
- Console → Lambda → Functions → (function) → Monitor → Metrics. Check Duration (use p99), Errors, Throttles and Concurrent executions.
- Console → Lambda → Functions → (function) → Monitor → View CloudWatch logs, then Logs Insights and run the queries below.
- Console → Lambda → Functions → (function) → Configuration → General configuration: check Memory, Timeout, and SnapStart. Configuration → Concurrency shows reserved and provisioned concurrency. Configuration → VPC shows subnets and security groups.
- Console → Lambda → Functions → (function) → Aliases to move the
livealias back to the previous version. - Console → CloudWatch → Service map or Traces (X-Ray) to see the Initialization segment and downstream subsegments. Active tracing must be on (Configuration → Monitoring and operations tools).
- Console → VPC → Route tables for the function's subnets, and VPC → Endpoints, if the function is in a VPC and calls hang.
AWS CLI
# 1. Current configuration: memory, timeout, runtime, architecture, VPC, tracing.
aws lambda get-function-configuration --function-name hl-get-account-summary \
--query '{Mem:MemorySize,Timeout:Timeout,Runtime:Runtime,Arch:Architectures,Vpc:VpcConfig.SubnetIds,Tracing:TracingConfig.Mode,SnapStart:SnapStart.ApplyOn}' \
--profile learn-sandbox
# 2. Concurrency settings: a reserved cap that is too low causes throttles.
aws lambda get-function-concurrency --function-name hl-get-account-summary --profile learn-sandbox
aws lambda get-account-settings --query 'AccountLimit.{Concurrent:ConcurrentExecutions,Unreserved:UnreservedConcurrentExecutions}' --profile learn-sandbox
# 3. Roll back: point the live alias to the previous version.
aws lambda update-alias --function-name hl-get-account-summary --name live --function-version 41 --profile learn-sandbox
# 4. Metrics: p99 duration and throttles for the last 3 hours.
aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Duration \
--dimensions Name=FunctionName,Value=hl-get-account-summary \
--start-time $(date -u -d '3 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
--period 300 --extended-statistics p99 --profile learn-sandbox
aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Throttles \
--dimensions Name=FunctionName,Value=hl-get-account-summary \
--start-time $(date -u -d '3 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
--period 300 --statistics Sum --profile learn-sandbox
# 5. Raise memory as an experiment (note the change in your incident record), then re-measure.
aws lambda update-function-configuration --function-name hl-get-account-summary --memory-size 1024 --profile learn-sandboxLogs Insights queries (run on the function's log group, /aws/lambda/hl-get-account-summary):
# A. Cold starts: how many, and how long?
filter @type = "REPORT" and ispresent(@initDuration)
| stats count(*) as coldStarts, avg(@initDuration) as avgInitMs, pct(@initDuration, 95) as p95InitMs, max(@initDuration) as maxInitMs by bin(15m)
| sort @timestamp desc
# B. Handler time: p50, p99 and max by 5-minute bucket
filter @type = "REPORT"
| stats count(*) as invocations, pct(@duration, 50) as p50, pct(@duration, 99) as p99, max(@duration) as maxMs by bin(5m)
| sort @timestamp desc
# C. Timeouts
filter @message like /Task timed out/
| stats count(*) as timeouts by bin(15m)
# D. Memory headroom: peak memory used against the configured size
filter @type = "REPORT"
| stats max(@maxMemoryUsed / 1024 / 1024) as peakMb, max(@memorySize / 1024 / 1024) as sizeMb by bin(1h)
# E. Slowest individual invocations with their request IDs, for X-Ray lookup
filter @type = "REPORT"
| fields @timestamp, @requestId, @duration, @initDuration, @maxMemoryUsed / 1024 / 1024 as memMb
| sort @duration desc
| limit 20Infrastructure as Code
A function that fails fast, starts fast and alarms before it times out.
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.Lambda;
using Constructs;
public class AccountSummaryFunctionStack : Stack
{
public AccountSummaryFunctionStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
{
var fn = new Function(this, "GetAccountSummary", new FunctionProps
{
FunctionName = "hl-get-account-summary",
// dotnet10 may need the explicit Runtime workaround in older CDK versions (⚠ VERIFY)
Runtime = new Runtime("dotnet10", RuntimeFamily.DOTNET_CORE),
Architecture = Architecture.ARM_64,
Handler = "Harbourline.Accounts::Harbourline.Accounts.AccountSummaryFunction::Handler",
Code = Code.FromAsset("publish"),
MemorySize = 1024,
Timeout = Duration.Seconds(10), // below API Gateway's own limit
SnapStart = SnapStartConf.ON_PUBLISHED_VERSIONS, // applies to published versions only
Tracing = Tracing.ACTIVE
});
// SnapStart only takes effect on published versions, so route traffic through an alias.
var alias = new Alias(this, "Live", new AliasProps { AliasName = "live", Version = fn.CurrentVersion });
new Alarm(this, "P99NearTimeout", new AlarmProps
{
AlarmName = "hl-get-account-summary-p99-near-timeout",
AlarmDescription = "p99 duration above 7 s on a 10 s timeout. See the Lambda timeouts playbook.",
Metric = alias.MetricDuration(new MetricOptions { Statistic = "p99", Period = Duration.Minutes(5) }),
Threshold = 7000,
EvaluationPeriods = 3,
ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING
});
new Alarm(this, "Throttled", new AlarmProps
{
AlarmName = "hl-get-account-summary-throttled",
Metric = alias.MetricThrottles(new MetricOptions { Statistic = "Sum", Period = Duration.Minutes(1) }),
Threshold = 0,
EvaluationPeriods = 2,
ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-lambda");
}
}Complete code
Three helpers: a latency budget that turns RemainingTime into cancellation tokens, a REPORT line parser for offline analysis, and a handler that uses the budget and fails fast.
namespace Harbourline.Functions;
public sealed class LatencyBudgetExceededException(string message) : Exception(message);
/// <summary>Turns "time left in this invocation" into per-call cancellation tokens.</summary>
public sealed class LatencyBudget(Func<TimeSpan> remaining, TimeSpan reserve)
{
/// <summary>Time we may still spend on downstream work, keeping a reserve to build the response.</summary>
public TimeSpan Available => remaining() - reserve;
public bool IsExhausted => Available <= TimeSpan.Zero;
/// <summary>A token that cancels at the earlier of the per-call cap and the budget.</summary>
public CancellationTokenSource CreateCancellation(TimeSpan? perCallCap = null)
{
var available = Available;
if (available <= TimeSpan.Zero)
throw new LatencyBudgetExceededException("No time left in this invocation for another downstream call.");
var timeout = perCallCap is { } cap && cap < available ? cap : available;
return new CancellationTokenSource(timeout);
}
}using System.Globalization;
using System.Text.RegularExpressions;
namespace Harbourline.Functions;
public sealed record ReportLine(string RequestId, double DurationMs, double BilledMs, int MemorySizeMb,
int MaxMemoryUsedMb, double? InitDurationMs, double? RestoreDurationMs)
{
public bool IsColdStart => InitDurationMs is not null;
public bool IsSnapStartRestore => RestoreDurationMs is not null;
public string Assess(TimeSpan configuredTimeout)
{
var notes = new List<string>();
if (DurationMs >= configuredTimeout.TotalMilliseconds * 0.9) notes.Add("NEAR TIMEOUT: duration is 90 percent or more of the limit");
if (MaxMemoryUsedMb >= MemorySizeMb * 0.9) notes.Add("NEAR MEMORY LIMIT: raise memory or reduce allocation");
if (IsColdStart && InitDurationMs >= 2000) notes.Add("SLOW COLD START: init over 2 s");
return notes.Count == 0 ? "OK" : string.Join("; ", notes);
}
private static readonly Regex Pattern = new(
@"REPORT RequestId:\s*(?<id>\S+)\s+Duration:\s*(?<dur>[\d.]+) ms\s+Billed Duration:\s*(?<bill>[\d.]+) ms\s+" +
@"Memory Size:\s*(?<mem>\d+) MB\s+Max Memory Used:\s*(?<max>\d+) MB" +
@"(?:\s+Init Duration:\s*(?<init>[\d.]+) ms)?(?:\s+Restore Duration:\s*(?<restore>[\d.]+) ms)?",
RegexOptions.Compiled | RegexOptions.CultureInvariant);
public static ReportLine? TryParse(string line)
{
var m = Pattern.Match(line);
if (!m.Success) return null;
static double D(string s) => double.Parse(s, CultureInfo.InvariantCulture);
return new ReportLine(
m.Groups["id"].Value, D(m.Groups["dur"].Value), D(m.Groups["bill"].Value),
int.Parse(m.Groups["mem"].Value), int.Parse(m.Groups["max"].Value),
m.Groups["init"].Success ? D(m.Groups["init"].Value) : null,
m.Groups["restore"].Success ? D(m.Groups["restore"].Value) : null);
}
}using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.Core;
using System.Diagnostics;
using System.Text.Json;
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]
namespace Harbourline.Functions;
public sealed class AccountSummaryFunction
{
// Created ONCE per execution environment (init phase), reused by every invocation.
private static readonly IAmazonDynamoDB Ddb = new AmazonDynamoDBClient(new AmazonDynamoDBConfig
{
Timeout = TimeSpan.FromSeconds(2), // per-request timeout, below the function timeout
MaxErrorRetry = 1 // keep retries from using the whole budget
});
private static bool _firstInvocation = true;
public async Task<APIGatewayHttpApiV2ProxyResponse> Handler(APIGatewayHttpApiV2ProxyRequest request, ILambdaContext context)
{
var cold = _firstInvocation; // true only for the first request in this environment
_firstInvocation = false;
var sw = Stopwatch.StartNew();
var budget = new LatencyBudget(() => context.RemainingTime, reserve: TimeSpan.FromMilliseconds(500));
try
{
var accountId = request.PathParameters["accountId"];
using var cts = budget.CreateCancellation(perCallCap: TimeSpan.FromSeconds(3));
var item = await Ddb.GetItemAsync(new GetItemRequest
{
TableName = Environment.GetEnvironmentVariable("ACCOUNTS_TABLE"),
Key = new() { ["account_id"] = new AttributeValue { S = accountId } },
ConsistentRead = false
}, cts.Token);
Log(context, "GetAccount", sw, cold, ok: true);
return Json(200, new { accountId, found = item.Item is { Count: > 0 } });
}
catch (LatencyBudgetExceededException)
{
Log(context, "BudgetExceeded", sw, cold, ok: false);
return Problem(503, "Service is busy. Please retry.", retryAfterSeconds: 2);
}
catch (OperationCanceledException)
{
// Our own deadline fired before Lambda's: a clean, loggable failure, not a platform kill.
Log(context, "DownstreamTimeout", sw, cold, ok: false);
return Problem(504, "A dependency did not answer in time.", retryAfterSeconds: 2);
}
}
private static void Log(ILambdaContext ctx, string outcome, Stopwatch sw, bool cold, bool ok) =>
Console.WriteLine(JsonSerializer.Serialize(new
{
requestId = ctx.AwsRequestId, outcome, ok, coldStart = cold,
elapsedMs = sw.ElapsedMilliseconds, remainingMs = (long)ctx.RemainingTime.TotalMilliseconds
}));
private static APIGatewayHttpApiV2ProxyResponse Json(int status, object body) => new()
{
StatusCode = status, Body = JsonSerializer.Serialize(body),
Headers = new Dictionary<string, string> { ["Content-Type"] = "application/json" }
};
private static APIGatewayHttpApiV2ProxyResponse Problem(int status, string detail, int retryAfterSeconds) => new()
{
StatusCode = status, Body = JsonSerializer.Serialize(new { detail }),
Headers = new Dictionary<string, string> { ["Content-Type"] = "application/json", ["Retry-After"] = retryAfterSeconds.ToString() }
};
}using Harbourline.Functions;
using Xunit;
public class LatencyBudgetTests
{
[Fact]
public void Budget_subtracts_the_reserve_and_caps_the_call()
{
var left = TimeSpan.FromSeconds(5);
var budget = new LatencyBudget(() => left, TimeSpan.FromMilliseconds(500));
Assert.Equal(TimeSpan.FromMilliseconds(4500), budget.Available);
using var cts = budget.CreateCancellation(TimeSpan.FromSeconds(2)); // cap wins over budget
Assert.False(cts.IsCancellationRequested);
}
[Fact]
public void Exhausted_budget_throws_instead_of_starting_work()
{
var budget = new LatencyBudget(() => TimeSpan.FromMilliseconds(300), TimeSpan.FromMilliseconds(500));
Assert.True(budget.IsExhausted);
Assert.Throws<LatencyBudgetExceededException>(() => budget.CreateCancellation());
}
}
public class ReportLineTests
{
private const string Cold =
"REPORT RequestId: 7d1b3c3e-0000-4a1b-9c3e-123456789abc Duration: 41.12 ms Billed Duration: 854 ms " +
"Memory Size: 512 MB Max Memory Used: 92 MB Init Duration: 812.40 ms";
private const string Timeout =
"REPORT RequestId: aaaa Duration: 10000.00 ms Billed Duration: 10000 ms Memory Size: 256 MB Max Memory Used: 250 MB";
[Fact]
public void Cold_start_line_is_parsed()
{
var r = ReportLine.TryParse(Cold)!;
Assert.True(r.IsColdStart);
Assert.Equal(812.40, r.InitDurationMs);
Assert.Equal(92, r.MaxMemoryUsedMb);
Assert.Equal("OK", r.Assess(TimeSpan.FromSeconds(10)));
}
[Fact]
public void Near_timeout_and_near_memory_are_flagged()
{
var r = ReportLine.TryParse(Timeout)!;
var assessment = r.Assess(TimeSpan.FromSeconds(10));
Assert.False(r.IsColdStart);
Assert.Contains("NEAR TIMEOUT", assessment);
Assert.Contains("NEAR MEMORY LIMIT", assessment);
}
[Fact]
public void Non_report_lines_return_null() => Assert.Null(ReportLine.TryParse("START RequestId: abc Version: 12"));
}Code explained line by line
LatencyBudgetwraps a delegate for the time remaining, so tests can pass a fixed value. In the handler the delegate is() => context.RemainingTime.Availableis the remaining time minus a reserve (500 ms here) kept for building and returning the response. Without a reserve, you can spend the whole timeout on a downstream call and then be killed before you reply.CreateCancellationtakes the smaller of a per-call cap and the available budget. If nothing is left it throws immediately, so the handler fails fast with a clean error instead of starting work it cannot finish.- The static
Ddbclient is created in the init phase, once per environment, and reused. Creating an SDK client per request wastes time and sockets. ItsTimeoutandMaxErrorRetryare explicit, so retries cannot consume the whole function timeout. _firstInvocationis a static flag, true only for the first request in an environment. It marks cold starts in your own logs, so you can correlate them with latency without parsingREPORTlines.- The handler catches
LatencyBudgetExceededException(503 withRetry-After) andOperationCanceledException(504). Your code decides the failure and logs it, instead of Lambda stopping the invocation withTask timed out. ReportLine.TryParsehandles the optionalInit DurationandRestore Durationfields: a missing init means a warm start; a restore duration means a SnapStart restore.Assessflags 90 percent of the timeout, 90 percent of memory and init over 2 seconds. Those are rules of thumb you can tune.- The tests use realistic fictional lines, including a near-timeout, near-memory case.
Security considerations
⚠ Learning shortcut — not for production
The handler logs a request ID, outcome and timings but takes the account ID straight from the path and does no authorisation. In production, authorise the caller (the API Gateway authorizer supplies the identity), check that the caller owns the account, validate the input format and never log account numbers. A quick fix to "reduce latency" must never remove an authorisation check.
- Do not raise memory or timeout permissions casually.
lambda:UpdateFunctionConfigurationshould belong to the pipeline role. Console edits create drift from the IaC and bypass review. Record any emergency change and put it back into code. - A longer timeout widens the blast radius. A hung function holds database connections and credentials longer. Keep the timeout as short as safe.
- Secrets at init. If you fetch a secret in a static constructor, with SnapStart it can be captured in the snapshot. Fetch it lazily, or after restore, and respect rotation (see Secrets management).
- Unique values and SnapStart. A random seed, a GUID or a connection created during init is duplicated across all restored environments. Create such values after restore.
- VPC functions need least-privilege networking. Do not open
0.0.0.0/0egress to fix a hang; add the specific endpoint or route. - Retries and money. A timeout followed by an upstream retry is how duplicates start. Money-moving functions must be idempotent (see the first playbook).
- Neither this lesson nor its code makes a service compliant with PCI DSS or any regulation; those assessments belong to your compliance function.
Banking use case
For Harbourline's GetAccountSummary, the p99 of 9 seconds turned out to be two effects. First, a release moved the function into a VPC to reach a private database, but the subnet had no route to Secrets Manager, so the first call in each new environment hung until the timeout (Query A showed cold starts with enormous init times, Query C showed the matching timeouts). Adding a Secrets Manager interface endpoint removed the hangs. Second, with the 128 MB default left in place, init was 3.5 seconds; raising memory to 1,024 MB on ARM64 and caching the secret lazily cut init to about 700 ms. The team enabled SnapStart through a live alias for a further reduction, and added the p99 alarm and the 503-with-Retry-After behaviour so overload degrades politely. They did not use provisioned concurrency, which would have cost money all day for a function whose peak is two hours long, and which cannot be combined with SnapStart.
Common mistakes
- Setting the timeout to 15 minutes to "fix" timeouts. It hides faults, raises cost and increases upstream retries.
- Creating SDK clients inside the handler. Slow, and exhausts sockets.
- Fetching secrets in a static constructor without caching or rotation handling.
- No client timeouts. One slow downstream consumes the function timeout.
- Treating cold starts as the only latency problem. Check handler time and downstream time too.
- Low memory. 128 MB .NET functions are slow everywhere. Test 512 to 1,024 MB.
- Expecting SnapStart on an unpublished version, with provisioned concurrency, or with Native AOT. None of those combinations work.
- A function timeout longer than the caller's. API Gateway returns a gateway timeout while Lambda keeps running, and the client's retry starts a second run.
- Visibility timeout shorter than the function timeout on an SQS source: the message is delivered again while the first attempt still runs.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Task timed out after 10.00 seconds, no downstream detail | No client timeouts or a hanging call (often a missing route in a VPC) | Add client timeouts, check VPC routes and endpoints, use X-Ray |
| Timeouts only on the first request after idle | Init is slow or hangs (secret fetch, DB connect in init) | Move remote calls out of init, add endpoints, consider SnapStart |
p99 high, p50 normal, Init Duration on a percentage of requests | Cold starts on scale-out or after idle | Memory, trim init, SnapStart, Native AOT or provisioned concurrency |
Throttles greater than 0, HTTP 429 from the API | Reserved concurrency cap or account limit reached | Raise or remove the cap, request a quota increase, smooth the burst |
| Duration rises with SQS batch size | Batch too large for the timeout | Reduce BatchSize, enable partial batch responses |
| Same SQS message processed twice | Visibility timeout shorter than function duration | Set the visibility timeout to at least six times the function timeout, make the handler idempotent |
Runtime.OutOfMemory or Max Memory Used equals size | Memory too low or an unbounded structure | Raise memory, stream data rather than load it all |
| API returns 504 but Lambda logs success | API integration timeout shorter than the function's duration | Make Lambda finish sooner or fix the longer path, keep Lambda timeout below the API limit |
| SnapStart enabled but no benefit | Calling $LATEST or an unpublished version | Publish a version and invoke it through an alias |
| Database errors rise with traffic | Each environment opens its own connections | Reuse connections, RDS Proxy (⚠ VERIFY suitability), cap concurrency |
Cost and cleanup
Cost label: free. The helpers and tests run locally. Lambda has an always-free monthly allowance (1 million requests and 400,000 GB-seconds, ⚠ VERIFY), so deploying one small function for a short experiment is effectively free. Provisioned concurrency is not free while configured, and SnapStart adds cache and restore charges (⚠ VERIFY current pricing); remove both when finished.
# CDK
cdk destroy AccountSummaryFunctionStack --profile learn-sandbox
# Or by hand
aws lambda delete-alias --function-name hl-get-account-summary --name live --profile learn-sandbox
aws lambda delete-function --function-name hl-get-account-summary --profile learn-sandbox
aws cloudwatch delete-alarms --alarm-names hl-get-account-summary-p99-near-timeout hl-get-account-summary-throttled --profile learn-sandbox
aws logs delete-log-group --log-group-name /aws/lambda/hl-get-account-summary --profile learn-sandbox
# Confirm nothing tagged for this lesson remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-lambda --profile learn-sandboxExercises
- 🟢 Guided: Create a test project, add
LatencyBudget,ReportLineand the tests, and rundotnet test. Then paste three real or inventedREPORTlines (one cold, one warm, one near-timeout) and printAssessfor each. - 🟡 Challenge: Extend
ReportLinehandling into a smallReportSummarythat takes a list of lines and returns: invocation count, cold-start percentage, p95 duration (without a library) and the count of lines flaggedNEAR TIMEOUT. Test it with 20 generated lines. - 🔴 Live problem: "Customers see errors on the home screen for a few seconds right after 08:00 each weekday." The function has a reserved concurrency of 20, memory 512 MB, timeout 10 s and no provisioned concurrency. Logs show
Throttlesspikes at 08:00 andInit Durationof about 1.2 s on the rest. List what you check, what you change first, and what you tell the product owner about cost.
Assignment
Add a Lambda latency triage card to your portfolio: the three questions (init, handler, downstream), the five Logs Insights queries, the cold-start ladder with its incompatibilities, and the timeout alignment rule (caller greater than function greater than downstream budgets; visibility timeout at least six times function timeout). Commit it as playbooks/lambda-timeouts-cold-starts.md with the helper code and tests.
Solutions
The cold line parses with InitDurationMs = 812.40 and assesses OK against a 10-second timeout. A warm line has no init, so IsColdStart is false. A line with Duration: 9500 ms on a 10 s timeout contains NEAR TIMEOUT. Why: the parser makes the optional Init Duration explicit, which is the signal that separates a cold start from slow code.
Parse each line with TryParse, drop nulls, count cold starts and divide by the count. For p95, sort durations and take the element at index ceil(0.95 * n) - 1. Count lines whose Assess contains NEAR TIMEOUT. With 20 generated lines, assert the exact percentage and counts. Why: the summary reproduces what Logs Insights shows, which proves you understand the numbers instead of trusting a tool.
Reading the facts: throttles at 08:00 mean the reserved cap (20) is too low for the burst, and the rest of the problem is cold starts of about 1.2 seconds. Check: peak ConcurrentExecutions at 08:00 versus the cap, the account limit, and the burst pattern. First change: raise or remove the reserved cap (taking care that the downstream database can handle the concurrency), then reduce init (memory 1,024 MB, ARM64, lazy secrets, SnapStart through an alias). Because the spike is predictable, consider scheduled provisioned concurrency for the weekday morning window (note it is incompatible with SnapStart on the same version). Tell the product owner: SnapStart and tuning cost little; scheduled provisioned concurrency costs for the hours it is configured, so you will estimate it from the number of environments and hours before enabling it. Why: throttles and cold starts are different problems with different fixes, and the fix for the first (capacity) can move the bottleneck to the database.
Knowledge check
Answer: A cold start has an Init Duration field. A slow handler has a large Duration without one. Many slow requests with Init Duration point to cold starts; slow Duration on warm invocations points to your code or a downstream call.
Answer: It has no network route to a service it calls in init or the handler (no NAT gateway or VPC endpoint), so the call hangs until the timeout.
Answer: They are alternative ways to avoid init cost: provisioned concurrency keeps environments already initialised, while SnapStart restores from a snapshot of an initialised version. Lambda does not support using both for the same function version.
Answer: The message becomes visible again while the first attempt still runs, so it is delivered again. That causes duplicate processing. Set the visibility timeout to at least six times the function timeout and make the handler idempotent.
Answer: When Lambda kills the invocation you lose control: no controlled response, no cleanup and only a generic log line. A deadline-aware handler logs the cause, returns a clear 503 or 504 with Retry-After, and releases resources.
Interview questions
- A Lambda function has a high p99 but a low average. What do you check? Cold starts (
Init Durationfrequency), concurrency bursts and throttles, and downstream tail latency, using Logs Insights onREPORTlines and X-Ray. - How do you reduce .NET cold starts? Measure first, then raise memory and use ARM64, trim init and create clients once, use ReadyToRun, then SnapStart or Native AOT, and use provisioned concurrency only where the cost is justified.
- How should timeouts be set across API Gateway, Lambda and downstream calls? Caller timeout greater than function timeout, which is greater than the sum of downstream budgets, with each client call having its own shorter timeout and limited retries.
- How do SQS and Lambda timeouts interact? The visibility timeout must exceed the function timeout, with the AWS recommendation of at least six times, or messages are redelivered while still processing.
- When would you not use provisioned concurrency? For spiky, short-peak workloads where scheduled or SnapStart-based options are cheaper, or when SnapStart is already used on that version.
Certification practice
A .NET Lambda function behind API Gateway has acceptable average latency, but the first request after idle periods takes five seconds. Logs show a large Init Duration on those requests. The team wants to reduce this without paying for always-on capacity. Which option is best?
A. Increase the function timeout to 15 minutes. B. Enable Lambda SnapStart for the function and invoke a published version through an alias. C. Configure provisioned concurrency for all hours. D. Move the function into a VPC.
Answer: B. SnapStart restores from a snapshot of an initialised environment without a standing charge for idle capacity. A does not affect init, C costs money continuously, and D does not reduce init time.
A Lambda function shows Throttles greater than zero in CloudWatch during a short daily peak. Total account concurrency is far from its limit. The function has reserved concurrency of 20. What is the most likely cause and fix?
A. The function's memory is too high; reduce it. B. The reserved concurrency acts as a cap of 20; raise it (checking downstream capacity). C. The function timeout is too long; reduce it to 1 second. D. The log retention is too short.
Answer: B. Reserved concurrency limits the function to 20 concurrent executions, so bursts above that are throttled even with account capacity free. A, C and D do not cause throttles.
A Lambda function processes messages from an SQS standard queue and takes up to 45 seconds. Messages are occasionally processed twice. The queue's visibility timeout is the default of 30 seconds. Which combination best resolves this?
A. Increase the queue's visibility timeout to well above the function timeout, and make processing idempotent. B. Decrease the function's memory. C. Convert the queue to FIFO and keep the default visibility timeout. D. Disable the dead-letter queue.
Answer: A. The message reappears when the visibility timeout expires before processing ends. Raising it (AWS recommends at least six times the function timeout) prevents premature redelivery, and idempotency covers the remaining at-least-once cases. B does not change redelivery, C still depends on the visibility timeout, and D removes error handling.
Completion checklist
- I can read a REPORT line and name cold start, near timeout and near memory
- I can run the five Logs Insights queries and say what each shows
- I know which cold-start options cannot be combined
- I ran the helper tests and added one case
- I can explain how to align timeouts across API Gateway, Lambda, downstream calls and SQS
- I ran the cleanup steps and confirmed nothing tagged
m17-playbook-lambdaremains
Key takeaways
- Separate init, handler time and downstream time before changing anything.
- A timeout is a symptom; find what the handler waited for, and give every downstream call its own shorter timeout.
- Climb the cold-start ladder only as far as needed: memory, lean init, ReadyToRun, SnapStart or Native AOT, then provisioned concurrency.
- Throttles are a concurrency problem, not a cold-start problem; check reserved caps and account limits.
- Align timeouts end to end, and make anything that moves money idempotent, because timeouts cause retries.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Playbook: ECS task keeps restarting
Diagnose an ECS service whose tasks start and stop in a loop: read the stopped reason and exit code, work out whether the task never started, crashed, ran out of memory or was killed by a health check, and fix it safely.
Playbook: 502, 503 and 504 from an ALB or API Gateway
Work out which hop returned a gateway error, read the load balancer and API Gateway evidence, and fix target health, keep-alive, idle-timeout and integration-timeout mismatches without guessing.