Skip to Content
Cloud Engineering Academy
17. Daily-Work Playbooks & Career

Playbook: Lambda timeouts and cold starts

Tell a Lambda timeout from a cold start from throttling using REPORT lines and Logs Insights, find the slow phase (init, your code or a downstream call), and fix it with deadline-aware code, right-sized memory and the right start-up option for a .NET function.

intermediate⏱ 1 h 20 min🟢 FreeDVA-C03SOA-C03SAA-C03

Assumes: .NET 10 LTS (Lambda dotnet10 runtime) · AWS SDK for .NET v4 · AWS CLI v2 · Amazon.Lambda.Core · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this playbook you can:

  • Read a Lambda REPORT line and say whether an invocation was a cold start, ran close to its timeout, or came close to its memory limit.
  • Separate the three time buckets of an invocation: init, your code and downstream calls.
  • Use CloudWatch Logs Insights to measure cold-start frequency, p99 duration and timeouts over time.
  • Diagnose the most common .NET causes: missing network route from a VPC function, unbounded downstream calls, low memory (low CPU), heavy start-up work and a concurrency limit.
  • Choose the right remedy for cold starts: memory, ARM64, lazy initialisation, ReadyToRun, SnapStart, Native AOT or provisioned concurrency, knowing which combinations are impossible.
  • Write deadline-aware code that fails fast with a clear error before Lambda kills the invocation.
  • Line up timeouts across API Gateway, Lambda and SQS so retries do not cause duplicates.

Prerequisites

Why this matters in banking

Harbourline's GetAccountSummary function backs the mobile app's home screen. After a release, a handful of customers see a blank screen for about ten seconds, then an error. The dashboard shows p50 latency of 60 ms but a p99 of 9 seconds, and a few Task timed out after 10.00 seconds lines. The first reaction is "Lambda is slow", the second is "set the timeout to 15 minutes". The first is vague and the second hides a real fault and raises costs, and a long timeout also makes the retry behaviour of upstream callers worse: API Gateway has already given up, the client retried, and the second request runs while the first is still going.

A timeout is a symptom. The method is to separate init from handler time from downstream time, and then decide whether the fix is code, configuration or a start-up technology.

The idea in plain language

A Lambda invocation has two possible phases:

  1. Init (cold start): only when Lambda must create a new execution environment: download your code, start the .NET runtime, run your static initialisers and constructor code. This costs time once per environment.
  2. Invoke: run your handler. This happens for every request, in a warm environment.

Lambda reuses an environment for later requests, but each environment handles one request at a time. When ten requests arrive at once, Lambda needs ten environments, so up to ten cold starts. That is why cold starts show up as the tail (p99) rather than the average, and why they appear after idle periods, deployments and traffic bursts.

A timeout is a separate thing: the handler ran for longer than the configured limit, and Lambda stopped it.

Analogy: A restaurant with one waiter per table. If a table is already seated (warm environment), service is fast. If a party arrives and no table is ready, the staff must lay a table first (cold start). If the kitchen takes forty minutes to cook a dish (a slow downstream call), the guest leaves when patience runs out (the timeout). Adding tables fixes the first, a faster kitchen fixes the second, and telling the guest "we'll wait all night" fixes neither.

The three questions

QuestionWhere to lookHealthy sign
Is it cold-start time?Init Duration on REPORT lines, X-Ray "Initialization" segmentFew invocations have it, and it is under 1 to 2 seconds
Is it handler time?Duration on REPORT lines, your own stage timersp99 far below the timeout
Is it waiting for a downstream?X-Ray subsegments, your logs around outbound callsEach downstream call has its own timeout

What problem the AWS service solves

Lambda removes servers and scales out by creating execution environments on demand. The trade-offs that matter here:

  • Timeout: configurable up to 900 seconds (15 minutes), default 3 seconds. Pick the smallest value that covers the p99.9 of normal work plus a margin.
  • Memory and CPU: memory is configurable from 128 MB up to 10,240 MB, and CPU scales with memory. A function with 1,769 MB gets about one vCPU (⚠ VERIFY current figure). A .NET function at 128 MB is usually painfully slow in init and in handler.
  • Concurrency: account-level default of 1,000 concurrent executions per Region (new accounts may be lower, ⚠ VERIFY). Reserved concurrency can cap a function, and a low cap causes throttles (HTTP 429 for synchronous callers).
  • Scaling speed: Lambda adds environments at a limited rate when traffic bursts (documented per function, ⚠ VERIFY), so a very sharp spike produces queued or throttled requests even below the account limit.
  • Init limits and billing: the init phase has a time limit (10 seconds for standard on-demand init, ⚠ VERIFY), and init time is billed for on-demand functions (billing for the init phase was standardised in 2025, ⚠ VERIFY current rules). Do not treat init as free.
  • VPC: attaching a function to a VPC no longer adds a large per-cold-start network penalty (⚠ VERIFY), but the function needs a route to everything it calls: NAT gateway or VPC endpoints. No route means calls hang until the timeout.
  • Upstream limits: API Gateway has its own integration timeout (29 seconds by default for REST APIs, with an increase available for Regional REST APIs since 2024; 30 seconds maximum for HTTP APIs; ⚠ VERIFY both).

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Functions functionTimeout in host.json (Consumption default 5 min, max 10 min)Lambda timeout (default 3 s, max 15 min)Per function setting, and the default is very short on purpose
Consumption plan cold start; Premium plan pre-warmed instancesCold starts; provisioned concurrency keeps environments initialisedProvisioned concurrency is billed while it is configured
Flex Consumption "always ready" instancesProvisioned concurrencySame idea: pay to avoid cold starts
Azure Functions scale controllerLambda concurrency model: one request per environmentThere is no multiple-requests-per-instance setting. More concurrency means more environments
Application Insights end-to-end durationX-Ray trace and REPORT lineREPORT has Duration, Billed Duration, Memory and Init Duration in one line
CancellationToken from the Functions hostILambdaContext.RemainingTimeLambda does not cancel your token for you. You create one from the remaining time
ReadyToRun and trimmingReadyToRun, trimming, Native AOT, SnapStartSnapStart snapshots an initialised environment (published versions only)

Architecture

Where time goes in one invocation

Triage flow

Step-by-step implementation

Minutes 0 to 5: scope and stop the bleeding

  1. Is it all invocations or some? Look at the error rate and p99 by time (CloudWatch metrics: Errors, Throttles, Duration, ConcurrentExecutions).
  2. Did something change? A deployment, a new dependency, a memory or timeout change, a new VPC or security group, a traffic spike (marketing campaign, payday).
  3. If a bad release is the cause, roll back through the alias (point live to the previous version). That is the fastest, safest move, and it keeps the evidence (the bad version still exists).
  4. Do not raise the timeout to 15 minutes as a first move. It may mask the fault, raise cost and make upstream retries pile up.

Minutes 5 to 20: measure with REPORT lines

Every invocation ends with a REPORT line. Use Logs Insights to get four numbers: how many invocations were cold, p99 duration, timeouts, and memory use against the limit. Queries are in the CLI section. Then decide using the triage flow.

Minutes 20 to 40: if it is a timeout, find what the handler was waiting for

Typical causes, most common first:

CauseEvidenceFix
No route from a VPC function to a service (DynamoDB, Secrets Manager, partner)Calls hang, then TaskCanceledException/timeout; same code works outside a VPCAdd a NAT gateway route, or VPC endpoints (gateway endpoint for S3 and DynamoDB; interface endpoints for others)
No client timeout, or retries that use the whole budgetDuration equals the function timeout, the downstream is slow, SDK retries repeatSet per-call timeouts shorter than the function's, limit retries, use a deadline derived from RemainingTime
Slow downstream (partner, database under load)X-Ray shows one subsegment taking most of the timeFix or protect the downstream; see Partner API timeouts
Too little memory (CPU starved)Max Memory Used near Memory Size, or high duration that falls when memory risesRaise memory and re-measure. Cost per request often stays flat or drops
Batch too big (SQS, Kinesis)Duration grows with batch size and hits the timeoutReduce batch size, process in parallel inside the budget, report partial failures
Connection storm to a databaseMany environments each open connections; DB errors rise with concurrencyReuse connections across invocations, use RDS Proxy or a data API, cap concurrency
Lock or contentionDuration spikes with specific recordsFix the data access pattern

Minutes 40 to 60: if it is cold starts, choose the lightest effective fix

Climb this ladder only as far as you need. Measure after each rung.

  1. Memory and architecture: more memory means more CPU, which shortens init. ARM64 is usually cheaper per unit of work.
  2. Trim init: build the DI container lightly, create AWS SDK clients once as static or singleton objects, do not fetch secrets or call remote services in a static constructor on the hot path (cache lazily on first use, or use the extension with caching), avoid reflection-heavy startup.
  3. Compile earlier: ReadyToRun reduces JIT time; trimming shrinks the package.
  4. SnapStart: snapshot after init for published versions; restores quickly. Not compatible with provisioned concurrency or Native AOT, and unique values created during init (random seeds, connection state) must be re-created on restore.
  5. Native AOT: the fastest start but the most restrictive code (no unrestricted reflection); uses the OS-only runtime.
  6. Provisioned concurrency: keeps a set number of environments initialised. Costs money while configured. Use it for the peak profile of a customer-facing function, ideally scheduled.

The details and trade-offs are in Lambda with C#, in the section on cold starts.

Minutes 60 onwards: make time a first-class concern

Add deadline-aware code (below), alarms on p99 near the timeout and on throttles, and align timeouts end to end: caller timeout greater than the function timeout, function timeout greater than the sum of its downstream budgets, and for SQS the queue's visibility timeout at least six times the function timeout (AWS recommendation, ⚠ VERIFY).

AWS Console instructions

  1. Console → Lambda → Functions → (function) → Monitor → Metrics. Check Duration (use p99), Errors, Throttles and Concurrent executions.
  2. Console → Lambda → Functions → (function) → Monitor → View CloudWatch logs, then Logs Insights and run the queries below.
  3. Console → Lambda → Functions → (function) → Configuration → General configuration: check Memory, Timeout, and SnapStart. Configuration → Concurrency shows reserved and provisioned concurrency. Configuration → VPC shows subnets and security groups.
  4. Console → Lambda → Functions → (function) → Aliases to move the live alias back to the previous version.
  5. Console → CloudWatch → Service map or Traces (X-Ray) to see the Initialization segment and downstream subsegments. Active tracing must be on (Configuration → Monitoring and operations tools).
  6. Console → VPC → Route tables for the function's subnets, and VPC → Endpoints, if the function is in a VPC and calls hang.

AWS CLI

# 1. Current configuration: memory, timeout, runtime, architecture, VPC, tracing.
aws lambda get-function-configuration --function-name hl-get-account-summary \
  --query '{Mem:MemorySize,Timeout:Timeout,Runtime:Runtime,Arch:Architectures,Vpc:VpcConfig.SubnetIds,Tracing:TracingConfig.Mode,SnapStart:SnapStart.ApplyOn}' \
  --profile learn-sandbox

# 2. Concurrency settings: a reserved cap that is too low causes throttles.
aws lambda get-function-concurrency --function-name hl-get-account-summary --profile learn-sandbox
aws lambda get-account-settings --query 'AccountLimit.{Concurrent:ConcurrentExecutions,Unreserved:UnreservedConcurrentExecutions}' --profile learn-sandbox

# 3. Roll back: point the live alias to the previous version.
aws lambda update-alias --function-name hl-get-account-summary --name live --function-version 41 --profile learn-sandbox

# 4. Metrics: p99 duration and throttles for the last 3 hours.
aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Duration \
  --dimensions Name=FunctionName,Value=hl-get-account-summary \
  --start-time $(date -u -d '3 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --extended-statistics p99 --profile learn-sandbox

aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Throttles \
  --dimensions Name=FunctionName,Value=hl-get-account-summary \
  --start-time $(date -u -d '3 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --statistics Sum --profile learn-sandbox

# 5. Raise memory as an experiment (note the change in your incident record), then re-measure.
aws lambda update-function-configuration --function-name hl-get-account-summary --memory-size 1024 --profile learn-sandbox

Logs Insights queries (run on the function's log group, /aws/lambda/hl-get-account-summary):

# A. Cold starts: how many, and how long?
filter @type = "REPORT" and ispresent(@initDuration)
| stats count(*) as coldStarts, avg(@initDuration) as avgInitMs, pct(@initDuration, 95) as p95InitMs, max(@initDuration) as maxInitMs by bin(15m)
| sort @timestamp desc

# B. Handler time: p50, p99 and max by 5-minute bucket
filter @type = "REPORT"
| stats count(*) as invocations, pct(@duration, 50) as p50, pct(@duration, 99) as p99, max(@duration) as maxMs by bin(5m)
| sort @timestamp desc

# C. Timeouts
filter @message like /Task timed out/
| stats count(*) as timeouts by bin(15m)

# D. Memory headroom: peak memory used against the configured size
filter @type = "REPORT"
| stats max(@maxMemoryUsed / 1024 / 1024) as peakMb, max(@memorySize / 1024 / 1024) as sizeMb by bin(1h)

# E. Slowest individual invocations with their request IDs, for X-Ray lookup
filter @type = "REPORT"
| fields @timestamp, @requestId, @duration, @initDuration, @maxMemoryUsed / 1024 / 1024 as memMb
| sort @duration desc
| limit 20

Infrastructure as Code

A function that fails fast, starts fast and alarms before it times out.

using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.Lambda;
using Constructs;

public class AccountSummaryFunctionStack : Stack
{
    public AccountSummaryFunctionStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
    {
        var fn = new Function(this, "GetAccountSummary", new FunctionProps
        {
            FunctionName = "hl-get-account-summary",
            // dotnet10 may need the explicit Runtime workaround in older CDK versions (⚠ VERIFY)
            Runtime = new Runtime("dotnet10", RuntimeFamily.DOTNET_CORE),
            Architecture = Architecture.ARM_64,
            Handler = "Harbourline.Accounts::Harbourline.Accounts.AccountSummaryFunction::Handler",
            Code = Code.FromAsset("publish"),
            MemorySize = 1024,
            Timeout = Duration.Seconds(10),                  // below API Gateway's own limit
            SnapStart = SnapStartConf.ON_PUBLISHED_VERSIONS, // applies to published versions only
            Tracing = Tracing.ACTIVE
        });

        // SnapStart only takes effect on published versions, so route traffic through an alias.
        var alias = new Alias(this, "Live", new AliasProps { AliasName = "live", Version = fn.CurrentVersion });

        new Alarm(this, "P99NearTimeout", new AlarmProps
        {
            AlarmName = "hl-get-account-summary-p99-near-timeout",
            AlarmDescription = "p99 duration above 7 s on a 10 s timeout. See the Lambda timeouts playbook.",
            Metric = alias.MetricDuration(new MetricOptions { Statistic = "p99", Period = Duration.Minutes(5) }),
            Threshold = 7000,
            EvaluationPeriods = 3,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
            TreatMissingData = TreatMissingData.NOT_BREACHING
        });

        new Alarm(this, "Throttled", new AlarmProps
        {
            AlarmName = "hl-get-account-summary-throttled",
            Metric = alias.MetricThrottles(new MetricOptions { Statistic = "Sum", Period = Duration.Minutes(1) }),
            Threshold = 0,
            EvaluationPeriods = 2,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
            TreatMissingData = TreatMissingData.NOT_BREACHING
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-lambda");
    }
}

Complete code

Three helpers: a latency budget that turns RemainingTime into cancellation tokens, a REPORT line parser for offline analysis, and a handler that uses the budget and fails fast.

LatencyBudget.cs
namespace Harbourline.Functions;

public sealed class LatencyBudgetExceededException(string message) : Exception(message);

/// <summary>Turns "time left in this invocation" into per-call cancellation tokens.</summary>
public sealed class LatencyBudget(Func<TimeSpan> remaining, TimeSpan reserve)
{
    /// <summary>Time we may still spend on downstream work, keeping a reserve to build the response.</summary>
    public TimeSpan Available => remaining() - reserve;

    public bool IsExhausted => Available <= TimeSpan.Zero;

    /// <summary>A token that cancels at the earlier of the per-call cap and the budget.</summary>
    public CancellationTokenSource CreateCancellation(TimeSpan? perCallCap = null)
    {
        var available = Available;
        if (available <= TimeSpan.Zero)
            throw new LatencyBudgetExceededException("No time left in this invocation for another downstream call.");

        var timeout = perCallCap is { } cap && cap < available ? cap : available;
        return new CancellationTokenSource(timeout);
    }
}
ReportLine.cs
using System.Globalization;
using System.Text.RegularExpressions;

namespace Harbourline.Functions;

public sealed record ReportLine(string RequestId, double DurationMs, double BilledMs, int MemorySizeMb,
                                int MaxMemoryUsedMb, double? InitDurationMs, double? RestoreDurationMs)
{
    public bool IsColdStart => InitDurationMs is not null;
    public bool IsSnapStartRestore => RestoreDurationMs is not null;

    public string Assess(TimeSpan configuredTimeout)
    {
        var notes = new List<string>();
        if (DurationMs >= configuredTimeout.TotalMilliseconds * 0.9) notes.Add("NEAR TIMEOUT: duration is 90 percent or more of the limit");
        if (MaxMemoryUsedMb >= MemorySizeMb * 0.9) notes.Add("NEAR MEMORY LIMIT: raise memory or reduce allocation");
        if (IsColdStart && InitDurationMs >= 2000) notes.Add("SLOW COLD START: init over 2 s");
        return notes.Count == 0 ? "OK" : string.Join("; ", notes);
    }

    private static readonly Regex Pattern = new(
        @"REPORT RequestId:\s*(?<id>\S+)\s+Duration:\s*(?<dur>[\d.]+) ms\s+Billed Duration:\s*(?<bill>[\d.]+) ms\s+" +
        @"Memory Size:\s*(?<mem>\d+) MB\s+Max Memory Used:\s*(?<max>\d+) MB" +
        @"(?:\s+Init Duration:\s*(?<init>[\d.]+) ms)?(?:\s+Restore Duration:\s*(?<restore>[\d.]+) ms)?",
        RegexOptions.Compiled | RegexOptions.CultureInvariant);

    public static ReportLine? TryParse(string line)
    {
        var m = Pattern.Match(line);
        if (!m.Success) return null;

        static double D(string s) => double.Parse(s, CultureInfo.InvariantCulture);
        return new ReportLine(
            m.Groups["id"].Value, D(m.Groups["dur"].Value), D(m.Groups["bill"].Value),
            int.Parse(m.Groups["mem"].Value), int.Parse(m.Groups["max"].Value),
            m.Groups["init"].Success ? D(m.Groups["init"].Value) : null,
            m.Groups["restore"].Success ? D(m.Groups["restore"].Value) : null);
    }
}
AccountSummaryFunction.cs
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.Core;
using System.Diagnostics;
using System.Text.Json;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace Harbourline.Functions;

public sealed class AccountSummaryFunction
{
    // Created ONCE per execution environment (init phase), reused by every invocation.
    private static readonly IAmazonDynamoDB Ddb = new AmazonDynamoDBClient(new AmazonDynamoDBConfig
    {
        Timeout = TimeSpan.FromSeconds(2),   // per-request timeout, below the function timeout
        MaxErrorRetry = 1                    // keep retries from using the whole budget
    });

    private static bool _firstInvocation = true;

    public async Task<APIGatewayHttpApiV2ProxyResponse> Handler(APIGatewayHttpApiV2ProxyRequest request, ILambdaContext context)
    {
        var cold = _firstInvocation;           // true only for the first request in this environment
        _firstInvocation = false;

        var sw = Stopwatch.StartNew();
        var budget = new LatencyBudget(() => context.RemainingTime, reserve: TimeSpan.FromMilliseconds(500));

        try
        {
            var accountId = request.PathParameters["accountId"];

            using var cts = budget.CreateCancellation(perCallCap: TimeSpan.FromSeconds(3));
            var item = await Ddb.GetItemAsync(new GetItemRequest
            {
                TableName = Environment.GetEnvironmentVariable("ACCOUNTS_TABLE"),
                Key = new() { ["account_id"] = new AttributeValue { S = accountId } },
                ConsistentRead = false
            }, cts.Token);

            Log(context, "GetAccount", sw, cold, ok: true);
            return Json(200, new { accountId, found = item.Item is { Count: > 0 } });
        }
        catch (LatencyBudgetExceededException)
        {
            Log(context, "BudgetExceeded", sw, cold, ok: false);
            return Problem(503, "Service is busy. Please retry.", retryAfterSeconds: 2);
        }
        catch (OperationCanceledException)
        {
            // Our own deadline fired before Lambda's: a clean, loggable failure, not a platform kill.
            Log(context, "DownstreamTimeout", sw, cold, ok: false);
            return Problem(504, "A dependency did not answer in time.", retryAfterSeconds: 2);
        }
    }

    private static void Log(ILambdaContext ctx, string outcome, Stopwatch sw, bool cold, bool ok) =>
        Console.WriteLine(JsonSerializer.Serialize(new
        {
            requestId = ctx.AwsRequestId, outcome, ok, coldStart = cold,
            elapsedMs = sw.ElapsedMilliseconds, remainingMs = (long)ctx.RemainingTime.TotalMilliseconds
        }));

    private static APIGatewayHttpApiV2ProxyResponse Json(int status, object body) => new()
    {
        StatusCode = status, Body = JsonSerializer.Serialize(body),
        Headers = new Dictionary<string, string> { ["Content-Type"] = "application/json" }
    };

    private static APIGatewayHttpApiV2ProxyResponse Problem(int status, string detail, int retryAfterSeconds) => new()
    {
        StatusCode = status, Body = JsonSerializer.Serialize(new { detail }),
        Headers = new Dictionary<string, string> { ["Content-Type"] = "application/json", ["Retry-After"] = retryAfterSeconds.ToString() }
    };
}
LambdaHelpersTests.cs
using Harbourline.Functions;
using Xunit;

public class LatencyBudgetTests
{
    [Fact]
    public void Budget_subtracts_the_reserve_and_caps_the_call()
    {
        var left = TimeSpan.FromSeconds(5);
        var budget = new LatencyBudget(() => left, TimeSpan.FromMilliseconds(500));

        Assert.Equal(TimeSpan.FromMilliseconds(4500), budget.Available);
        using var cts = budget.CreateCancellation(TimeSpan.FromSeconds(2));   // cap wins over budget
        Assert.False(cts.IsCancellationRequested);
    }

    [Fact]
    public void Exhausted_budget_throws_instead_of_starting_work()
    {
        var budget = new LatencyBudget(() => TimeSpan.FromMilliseconds(300), TimeSpan.FromMilliseconds(500));

        Assert.True(budget.IsExhausted);
        Assert.Throws<LatencyBudgetExceededException>(() => budget.CreateCancellation());
    }
}

public class ReportLineTests
{
    private const string Cold =
        "REPORT RequestId: 7d1b3c3e-0000-4a1b-9c3e-123456789abc Duration: 41.12 ms Billed Duration: 854 ms " +
        "Memory Size: 512 MB Max Memory Used: 92 MB Init Duration: 812.40 ms";

    private const string Timeout =
        "REPORT RequestId: aaaa Duration: 10000.00 ms Billed Duration: 10000 ms Memory Size: 256 MB Max Memory Used: 250 MB";

    [Fact]
    public void Cold_start_line_is_parsed()
    {
        var r = ReportLine.TryParse(Cold)!;

        Assert.True(r.IsColdStart);
        Assert.Equal(812.40, r.InitDurationMs);
        Assert.Equal(92, r.MaxMemoryUsedMb);
        Assert.Equal("OK", r.Assess(TimeSpan.FromSeconds(10)));
    }

    [Fact]
    public void Near_timeout_and_near_memory_are_flagged()
    {
        var r = ReportLine.TryParse(Timeout)!;
        var assessment = r.Assess(TimeSpan.FromSeconds(10));

        Assert.False(r.IsColdStart);
        Assert.Contains("NEAR TIMEOUT", assessment);
        Assert.Contains("NEAR MEMORY LIMIT", assessment);
    }

    [Fact]
    public void Non_report_lines_return_null() => Assert.Null(ReportLine.TryParse("START RequestId: abc Version: 12"));
}

Code explained line by line

  1. LatencyBudget wraps a delegate for the time remaining, so tests can pass a fixed value. In the handler the delegate is () => context.RemainingTime.
  2. Available is the remaining time minus a reserve (500 ms here) kept for building and returning the response. Without a reserve, you can spend the whole timeout on a downstream call and then be killed before you reply.
  3. CreateCancellation takes the smaller of a per-call cap and the available budget. If nothing is left it throws immediately, so the handler fails fast with a clean error instead of starting work it cannot finish.
  4. The static Ddb client is created in the init phase, once per environment, and reused. Creating an SDK client per request wastes time and sockets. Its Timeout and MaxErrorRetry are explicit, so retries cannot consume the whole function timeout.
  5. _firstInvocation is a static flag, true only for the first request in an environment. It marks cold starts in your own logs, so you can correlate them with latency without parsing REPORT lines.
  6. The handler catches LatencyBudgetExceededException (503 with Retry-After) and OperationCanceledException (504). Your code decides the failure and logs it, instead of Lambda stopping the invocation with Task timed out.
  7. ReportLine.TryParse handles the optional Init Duration and Restore Duration fields: a missing init means a warm start; a restore duration means a SnapStart restore.
  8. Assess flags 90 percent of the timeout, 90 percent of memory and init over 2 seconds. Those are rules of thumb you can tune.
  9. The tests use realistic fictional lines, including a near-timeout, near-memory case.

Security considerations

⚠ Learning shortcut — not for production

The handler logs a request ID, outcome and timings but takes the account ID straight from the path and does no authorisation. In production, authorise the caller (the API Gateway authorizer supplies the identity), check that the caller owns the account, validate the input format and never log account numbers. A quick fix to "reduce latency" must never remove an authorisation check.

  • Do not raise memory or timeout permissions casually. lambda:UpdateFunctionConfiguration should belong to the pipeline role. Console edits create drift from the IaC and bypass review. Record any emergency change and put it back into code.
  • A longer timeout widens the blast radius. A hung function holds database connections and credentials longer. Keep the timeout as short as safe.
  • Secrets at init. If you fetch a secret in a static constructor, with SnapStart it can be captured in the snapshot. Fetch it lazily, or after restore, and respect rotation (see Secrets management).
  • Unique values and SnapStart. A random seed, a GUID or a connection created during init is duplicated across all restored environments. Create such values after restore.
  • VPC functions need least-privilege networking. Do not open 0.0.0.0/0 egress to fix a hang; add the specific endpoint or route.
  • Retries and money. A timeout followed by an upstream retry is how duplicates start. Money-moving functions must be idempotent (see the first playbook).
  • Neither this lesson nor its code makes a service compliant with PCI DSS or any regulation; those assessments belong to your compliance function.

Banking use case

For Harbourline's GetAccountSummary, the p99 of 9 seconds turned out to be two effects. First, a release moved the function into a VPC to reach a private database, but the subnet had no route to Secrets Manager, so the first call in each new environment hung until the timeout (Query A showed cold starts with enormous init times, Query C showed the matching timeouts). Adding a Secrets Manager interface endpoint removed the hangs. Second, with the 128 MB default left in place, init was 3.5 seconds; raising memory to 1,024 MB on ARM64 and caching the secret lazily cut init to about 700 ms. The team enabled SnapStart through a live alias for a further reduction, and added the p99 alarm and the 503-with-Retry-After behaviour so overload degrades politely. They did not use provisioned concurrency, which would have cost money all day for a function whose peak is two hours long, and which cannot be combined with SnapStart.

Common mistakes

  • Setting the timeout to 15 minutes to "fix" timeouts. It hides faults, raises cost and increases upstream retries.
  • Creating SDK clients inside the handler. Slow, and exhausts sockets.
  • Fetching secrets in a static constructor without caching or rotation handling.
  • No client timeouts. One slow downstream consumes the function timeout.
  • Treating cold starts as the only latency problem. Check handler time and downstream time too.
  • Low memory. 128 MB .NET functions are slow everywhere. Test 512 to 1,024 MB.
  • Expecting SnapStart on an unpublished version, with provisioned concurrency, or with Native AOT. None of those combinations work.
  • A function timeout longer than the caller's. API Gateway returns a gateway timeout while Lambda keeps running, and the client's retry starts a second run.
  • Visibility timeout shorter than the function timeout on an SQS source: the message is delivered again while the first attempt still runs.

Troubleshooting

SymptomLikely causeFix
Task timed out after 10.00 seconds, no downstream detailNo client timeouts or a hanging call (often a missing route in a VPC)Add client timeouts, check VPC routes and endpoints, use X-Ray
Timeouts only on the first request after idleInit is slow or hangs (secret fetch, DB connect in init)Move remote calls out of init, add endpoints, consider SnapStart
p99 high, p50 normal, Init Duration on a percentage of requestsCold starts on scale-out or after idleMemory, trim init, SnapStart, Native AOT or provisioned concurrency
Throttles greater than 0, HTTP 429 from the APIReserved concurrency cap or account limit reachedRaise or remove the cap, request a quota increase, smooth the burst
Duration rises with SQS batch sizeBatch too large for the timeoutReduce BatchSize, enable partial batch responses
Same SQS message processed twiceVisibility timeout shorter than function durationSet the visibility timeout to at least six times the function timeout, make the handler idempotent
Runtime.OutOfMemory or Max Memory Used equals sizeMemory too low or an unbounded structureRaise memory, stream data rather than load it all
API returns 504 but Lambda logs successAPI integration timeout shorter than the function's durationMake Lambda finish sooner or fix the longer path, keep Lambda timeout below the API limit
SnapStart enabled but no benefitCalling $LATEST or an unpublished versionPublish a version and invoke it through an alias
Database errors rise with trafficEach environment opens its own connectionsReuse connections, RDS Proxy (⚠ VERIFY suitability), cap concurrency

Cost and cleanup

Cost label: free. The helpers and tests run locally. Lambda has an always-free monthly allowance (1 million requests and 400,000 GB-seconds, ⚠ VERIFY), so deploying one small function for a short experiment is effectively free. Provisioned concurrency is not free while configured, and SnapStart adds cache and restore charges (⚠ VERIFY current pricing); remove both when finished.

# CDK
cdk destroy AccountSummaryFunctionStack --profile learn-sandbox

# Or by hand
aws lambda delete-alias --function-name hl-get-account-summary --name live --profile learn-sandbox
aws lambda delete-function --function-name hl-get-account-summary --profile learn-sandbox
aws cloudwatch delete-alarms --alarm-names hl-get-account-summary-p99-near-timeout hl-get-account-summary-throttled --profile learn-sandbox
aws logs delete-log-group --log-group-name /aws/lambda/hl-get-account-summary --profile learn-sandbox

# Confirm nothing tagged for this lesson remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-lambda --profile learn-sandbox

Exercises

  • 🟢 Guided: Create a test project, add LatencyBudget, ReportLine and the tests, and run dotnet test. Then paste three real or invented REPORT lines (one cold, one warm, one near-timeout) and print Assess for each.
  • 🟡 Challenge: Extend ReportLine handling into a small ReportSummary that takes a list of lines and returns: invocation count, cold-start percentage, p95 duration (without a library) and the count of lines flagged NEAR TIMEOUT. Test it with 20 generated lines.
  • 🔴 Live problem: "Customers see errors on the home screen for a few seconds right after 08:00 each weekday." The function has a reserved concurrency of 20, memory 512 MB, timeout 10 s and no provisioned concurrency. Logs show Throttles spikes at 08:00 and Init Duration of about 1.2 s on the rest. List what you check, what you change first, and what you tell the product owner about cost.

Assignment

Add a Lambda latency triage card to your portfolio: the three questions (init, handler, downstream), the five Logs Insights queries, the cold-start ladder with its incompatibilities, and the timeout alignment rule (caller greater than function greater than downstream budgets; visibility timeout at least six times function timeout). Commit it as playbooks/lambda-timeouts-cold-starts.md with the helper code and tests.

Solutions

Knowledge check

Interview questions

  • A Lambda function has a high p99 but a low average. What do you check? Cold starts (Init Duration frequency), concurrency bursts and throttles, and downstream tail latency, using Logs Insights on REPORT lines and X-Ray.
  • How do you reduce .NET cold starts? Measure first, then raise memory and use ARM64, trim init and create clients once, use ReadyToRun, then SnapStart or Native AOT, and use provisioned concurrency only where the cost is justified.
  • How should timeouts be set across API Gateway, Lambda and downstream calls? Caller timeout greater than function timeout, which is greater than the sum of downstream budgets, with each client call having its own shorter timeout and limited retries.
  • How do SQS and Lambda timeouts interact? The visibility timeout must exceed the function timeout, with the AWS recommendation of at least six times, or messages are redelivered while still processing.
  • When would you not use provisioned concurrency? For spiky, short-peak workloads where scheduled or SnapStart-based options are cheaper, or when SnapStart is already used on that version.

Certification practice

Completion checklist

  • I can read a REPORT line and name cold start, near timeout and near memory
  • I can run the five Logs Insights queries and say what each shows
  • I know which cold-start options cannot be combined
  • I ran the helper tests and added one case
  • I can explain how to align timeouts across API Gateway, Lambda, downstream calls and SQS
  • I ran the cleanup steps and confirmed nothing tagged m17-playbook-lambda remains

Key takeaways

  • Separate init, handler time and downstream time before changing anything.
  • A timeout is a symptom; find what the handler waited for, and give every downstream call its own shorter timeout.
  • Climb the cold-start ladder only as far as needed: memory, lean init, ReadyToRun, SnapStart or Native AOT, then provisioned concurrency.
  • Throttles are a concurrency problem, not a cold-start problem; check reserved caps and account limits.
  • Align timeouts end to end, and make anything that moves money idempotent, because timeouts cause retries.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page