Skip to Content
Cloud Engineering Academy
4. API Development & Integration

Webhooks: send and receive safely with HMAC and replay protection

Receive partner webhooks on AWS with raw-body HMAC-SHA256 verification, timestamp tolerance, replay protection and fast acknowledgement, and send your own signed webhooks with retries, dead-letter queues and SSRF defences.

intermediate⏱ 2 h 🟢 FreeDVA-C03SAA-C03SCS-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SQS / AWSSDK.DynamoDBv2 4.1.x) · Standard Webhooks v1 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-17.

Learning objectives

By the end of this lesson you can:

  • Choose between webhooks, polling and WebSockets for a partner integration and justify the choice.
  • Verify an inbound webhook the Standard Webhooks way: read the raw body, recompute the HMAC-SHA256 over id.timestamp.body, compare in constant time, and enforce a timestamp tolerance.
  • Add replay protection and idempotency with a message-id store (in memory locally, DynamoDB with TTL in production).
  • Rotate webhook secrets without downtime using multiple signatures.
  • Build the receiving architecture API Gateway → Lambda receiver → SQS → processor that acknowledges in milliseconds and processes asynchronously.
  • Send webhooks to customers with signing, exponential backoff, a dead-letter queue, fixed egress IPs and SSRF protection — and explain why EventBridge API destinations alone cannot sign them.

Prerequisites

Why this matters in banking

Harbourline Bank's (fictional) card processor, Paylinq, sends a webhook whenever a card payment is authorised, settled or charged back. An attacker who finds the URL can post a forged payment.settled event for ACC-0000123456 worth GBP 4,999.00; a receiver that trusts any JSON credits money that never existed. And when the receiver is slow, Paylinq's retries deliver the same chargeback three times, so the dispute team opens three cases.

Both incidents come from treating a webhook as a trusted, exactly-once message. It is an unauthenticated HTTP POST from the internet that may arrive late, twice, out of order, or forged.

The idea in plain language

A webhook is a "reverse API call": instead of you asking a partner "anything new?" every minute (polling), the partner calls your HTTPS endpoint when something happens. A WebSocket keeps one long-lived two-way connection open, which suits live dashboards but not server-to-server business events.

PatternWho starts the callGood forWeak at
PollingYou, on a timerPartners without push; simple firewallsLatency vs cost trade-off; wasted calls; rate limits
WebhookPartner, per eventEvent notifications between organisationsYou must expose an endpoint; authenticity; duplicates
WebSocketClient opens, both sendLive UI updates, chat, streaming pricesLong-lived connections; reconnect logic; not for B2B events

Because anyone can POST to your endpoint, the partner signs each message with a shared secret; your receiver recomputes the signature over the exact bytes received. A signed timestamp stops old messages being replayed, and a message id lets you ignore duplicates.

Analogy: A courier brings an envelope to the bank's back door. The tamper-evident seal is the HMAC; the date on the seal is the timestamp (the post room refuses anything sealed over five minutes ago); the reference number is the webhook id (the post room never processes the same envelope twice). The post room signs for the parcel at once and passes it to the back office — the courier never waits for reconciliation.

What problem the AWS services solve

There is no single "webhook service" in AWS; you assemble one from managed parts.

NeedAWS building blockHonest limitations
Public HTTPS endpoint with throttlingAmazon API Gateway (HTTP or REST API)Integration timeout means the receiver must be fast; mapping templates must not alter the body you verify
Verify signature and acknowledgeAWS Lambda (.NET 10, dotnet10)Cold starts add latency; keep the receiver tiny
Decouple acknowledgement from processingAmazon SQS standard queue + dead-letter queueAt-least-once delivery: duplicates are normal, so the processor must be idempotent. Maximum message size is 1 MiB (raised from 256 KiB in August 2025)
Replay / idempotency storeAmazon DynamoDB with TTLTTL deletes expired items "typically within a few days", not instantly — filter on the expiry attribute if you read it
Signing secret storageAWS Secrets ManagerAbout USD 0.40 per secret per month plus USD 0.05 per 10,000 API calls (us-east-1, checked 2026-11-17); cache it, do not fetch per request
Managed outbound deliveryAmazon EventBridge API destinationsSupports Basic, API key and OAuth client-credentials auth only — no per-message HMAC signing and no mTLS; 5-second endpoint timeout

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Function HTTP triggerAPI Gateway + Lambda receiverAPI Gateway is a separate resource with its own throttling and logs
Service Bus queue + queue triggerSQS + Lambda event source mappingNo sessions on standard queues; rely on idempotency
Event Grid webhook deliveryEventBridge API destinationsAuth is Basic/API key/OAuth only; no HMAC signing
Key Vault secretSecrets Manager (KMS-encrypted)Cache it with the caching client
APIM policy checking a signatureCode in the LambdaAPI Gateway has no built-in HMAC check
Cosmos DB TTL for dedupe idsDynamoDB TTL + conditional writesattribute_not_exists gives an atomic "first one wins"
NAT Gateway static IP for allow-listsNAT gateway + Elastic IPAWS also offers a regional NAT gateway mode

Architecture

Security-boundary view:

The important line is the arrow labelled "verified only": nothing reaches the queue unless the signature matched. Everything to the left of that arrow treats the request as hostile.

Step-by-step implementation

Agree the scheme with the partner

Use a published scheme rather than inventing one. This lesson follows Standard Webhooks (standardwebhooks.com), which several webhook providers use:

  • Headers: webhook-id (unique per event, the same on every retry), webhook-timestamp (Unix seconds), webhook-signature.
  • Signature header: one or more space-separated entries of the form v1,<base64 HMAC-SHA256>. v1 means symmetric HMAC-SHA256; v1a is the asymmetric ed25519 variant.
  • Secret: 24–64 random bytes, base64-encoded, prefixed whsec_.
  • Signed content: webhook-id + . + webhook-timestamp + . + the exact body bytes.

If a partner uses its own scheme (a single X-Signature header, hex encoding, a different signed string), write a separate verifier for it rather than loosening one verifier to accept several formats.

Read the raw body before anything parses it

The signature covers bytes. Deserialising and re-serialising changes whitespace, property order and number formats, so read the body into a byte array, verify, and only then parse. In API Gateway use a proxy integration (no mapping template); in Lambda decode the body when IsBase64Encoded is true.

Verify signature, then timestamp

Recompute HMAC-SHA256 for each active secret, compare with CryptographicOperations.FixedTimeEquals, and accept if any candidate matches. Then reject timestamps more than 5 minutes from your clock in either direction (the spec leaves the window to you). A correctly signed message can still be a replay, so you need both checks.

De-duplicate on webhook-id

Replay protection needs ids only for the tolerance window. Idempotency needs them for the sender's whole retry window: Standard Webhooks' example schedule retries for about 75 hours, each retry with the same webhook-id and a fresh timestamp (otherwise later retries would fail the tolerance check). Keep ids for at least 7 days.

Acknowledge fast, process later

Return 2xx as soon as the event is durably enqueued. The spec recommends sender timeouts of 15–30 seconds and EventBridge API destinations time out after 5 seconds, so a receiver that calls core banking synchronously causes retries and duplicates. The processor uses a DynamoDB conditional write as its source of truth for "already processed"; poison messages go to a dead-letter queue.

Rotate secrets with overlap

The partner signs with old and new secrets for a period (two v1, entries in the header); your verifier accepts either. When all traffic matches the new secret, remove the old one on both sides.

AWS Console instructions

Optional — the lab runs fully locally. Add the four mandatory tags to everything.

  1. Secrets Manager → Store a new secret → Other type of secret: hl-m04-webhook-paylinq, value whsec_... (generate it as in the CLI section).
  2. SQS → Create queue → Standard: hl-m04-payment-events-dlq, then hl-m04-payment-events with Dead-letter queue → Enabled, maximum receives 5, SSE-SQS.
  3. DynamoDB → Create table: hl-m04-webhook-ids, partition key pk (String), on-demand; then Time to Live → Turn on with attribute expiresAt.
  4. Lambda → Create function: runtime .NET 10, arm64, from the lab's publish output.
  5. API Gateway → Create API → HTTP API: Lambda integration, route POST /webhooks/payment-events, access logging with 30-day retention.

AWS CLI

# Generate a Standard Webhooks style secret locally: 32 random bytes, base64, whsec_ prefix
echo "whsec_$(openssl rand -base64 32)"

# Store it in Secrets Manager with the mandatory tags (paste the value from above)
aws secretsmanager create-secret --name hl-m04-webhook-paylinq \
  --secret-string "whsec_REPLACE_ME" \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m04-webhook-hmac Key=Environment,Value=learn Key=Owner,Value=your-name \
  --profile learn-sandbox --region us-east-1

# Dead-letter queue, then the main queue with a redrive policy (5 receives)
DLQ_URL=$(aws sqs create-queue --queue-name hl-m04-payment-events-dlq \
  --attributes SqsManagedSseEnabled=true \
  --tags Project=cea-lab,Lab=m04-webhook-hmac,Environment=learn,Owner=your-name \
  --query QueueUrl --output text --profile learn-sandbox --region us-east-1)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" --attribute-names QueueArn \
  --query Attributes.QueueArn --output text --profile learn-sandbox --region us-east-1)
aws sqs create-queue --queue-name hl-m04-payment-events \
  --attributes "{\"SqsManagedSseEnabled\":\"true\",\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"$DLQ_ARN\\\",\\\"maxReceiveCount\\\":\\\"5\\\"}\"}" \
  --tags Project=cea-lab,Lab=m04-webhook-hmac,Environment=learn,Owner=your-name \
  --profile learn-sandbox --region us-east-1

# Idempotency table with TTL on expiresAt
aws dynamodb create-table --table-name hl-m04-webhook-ids \
  --attribute-definitions AttributeName=pk,AttributeType=S \
  --key-schema AttributeName=pk,KeyType=HASH --billing-mode PAY_PER_REQUEST \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m04-webhook-hmac Key=Environment,Value=learn Key=Owner,Value=your-name \
  --profile learn-sandbox --region us-east-1
aws dynamodb update-time-to-live --table-name hl-m04-webhook-ids \
  --time-to-live-specification "Enabled=true,AttributeName=expiresAt" \
  --profile learn-sandbox --region us-east-1

Infrastructure as Code

using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.Lambda.EventSources;
using Amazon.CDK.AWS.SQS;
using Constructs;

namespace Harbourline.Webhooks.Infra;

public sealed class WebhookReceiverStack : Stack
{
    public WebhookReceiverStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        var dlq = new Queue(this, "Dlq", new QueueProps
        {
            QueueName = "hl-m04-payment-events-dlq",
            Encryption = QueueEncryption.SQS_MANAGED,
            RetentionPeriod = Duration.Days(14),
        });

        var queue = new Queue(this, "Events", new QueueProps
        {
            QueueName = "hl-m04-payment-events",
            Encryption = QueueEncryption.SQS_MANAGED,
            VisibilityTimeout = Duration.Seconds(60),
            DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 },
        });

        var ids = new TableV2(this, "WebhookIds", new TablePropsV2
        {
            TableName = "hl-m04-webhook-ids",
            PartitionKey = new Amazon.CDK.AWS.DynamoDB.Attribute { Name = "pk", Type = AttributeType.STRING },
            TimeToLiveAttribute = "expiresAt",
            RemovalPolicy = RemovalPolicy.DESTROY, // learning shortcut: production keeps data
        });

        var processor = new Function(this, "Processor", new FunctionProps
        {
            Runtime = Runtime.DOTNET_10,
            Architecture = Architecture.ARM_64,
            Handler = "Harbourline.Webhooks.Processor",
            Code = Code.FromAsset("../src/Harbourline.Webhooks.Processor/bin/publish"),
            Timeout = Duration.Seconds(30),
        });
        processor.AddEventSource(new SqsEventSource(queue, new SqsEventSourceProps
        {
            BatchSize = 10,
            ReportBatchItemFailures = true,
        }));
        ids.GrantReadWriteData(processor);

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m04-webhook-hmac");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        Amazon.CDK.Tags.Of(this).Add("Owner", "your-name");
    }
}

The receiver function and HTTP API follow the same pattern (HttpApi with a HttpLambdaIntegration); grant it queue.GrantSendMessages, ids.GrantReadData and read access to the one secret.

Complete code

The lab project labs/m04-webhook-hmac/ contains the full solution (WebhookSigner, WebhookVerifier, IReplayCache, InMemoryReplayCache, WebhookSender, the receiver and tests). The listings below are simplified excerpts that compile on their own; the lab may differ in detail.

src/Harbourline.Webhooks/WebhookVerifier.cs
using System.Globalization;
using System.Security.Cryptography;
using System.Text;

namespace Harbourline.Webhooks;

public enum WebhookVerificationFailure { None, MissingHeaders, InvalidTimestamp, TimestampOutOfTolerance, NoMatchingSignature }

public sealed record WebhookVerificationResult(bool IsValid, WebhookVerificationFailure Failure)
{
    public static readonly WebhookVerificationResult Success = new(true, WebhookVerificationFailure.None);
    public static WebhookVerificationResult Fail(WebhookVerificationFailure f) => new(false, f);
}

public sealed class WebhookVerifier
{
    private static readonly TimeSpan Tolerance = TimeSpan.FromMinutes(5);
    private readonly IReadOnlyList<byte[]> _keys;
    private readonly TimeProvider _time;

    public WebhookVerifier(IEnumerable<string> activeSecrets, TimeProvider time)
    {
        _keys = activeSecrets.Select(DecodeSecret).ToArray();
        if (_keys.Count == 0) throw new ArgumentException("At least one secret is required.", nameof(activeSecrets));
        _time = time;
    }

    public static byte[] DecodeSecret(string secret)
    {
        const string prefix = "whsec_";
        var base64 = secret.StartsWith(prefix, StringComparison.Ordinal) ? secret[prefix.Length..] : secret;
        return Convert.FromBase64String(base64);
    }

    public WebhookVerificationResult Verify(string? id, string? timestamp, string? signatureHeader, byte[] rawBody)
    {
        if (string.IsNullOrEmpty(id) || string.IsNullOrEmpty(timestamp) || string.IsNullOrEmpty(signatureHeader) || id.Contains('.'))
            return WebhookVerificationResult.Fail(WebhookVerificationFailure.MissingHeaders);

        if (!long.TryParse(timestamp, NumberStyles.None, CultureInfo.InvariantCulture, out var sentAt))
            return WebhookVerificationResult.Fail(WebhookVerificationFailure.InvalidTimestamp);

        var now = _time.GetUtcNow().ToUnixTimeSeconds();
        if (Math.Abs(now - sentAt) > (long)Tolerance.TotalSeconds)
            return WebhookVerificationResult.Fail(WebhookVerificationFailure.TimestampOutOfTolerance);

        var prefix = Encoding.UTF8.GetBytes($"{id}.{timestamp}.");
        var signedContent = new byte[prefix.Length + rawBody.Length];
        prefix.CopyTo(signedContent, 0);
        rawBody.CopyTo(signedContent, prefix.Length);

        var expected = _keys.Select(k => HMACSHA256.HashData(k, signedContent)).ToArray();

        foreach (var entry in signatureHeader.Split(' ', StringSplitOptions.RemoveEmptyEntries))
        {
            var comma = entry.IndexOf(',');
            if (comma <= 0 || entry[..comma] != "v1") continue;

            var candidate = new byte[32];
            if (!Convert.TryFromBase64String(entry[(comma + 1)..], candidate, out var written) || written != 32) continue;

            foreach (var mac in expected)
            {
                if (CryptographicOperations.FixedTimeEquals(mac, candidate))
                    return WebhookVerificationResult.Success;
            }
        }

        return WebhookVerificationResult.Fail(WebhookVerificationFailure.NoMatchingSignature);
    }
}
src/Harbourline.Webhooks.Receiver/Program.cs
using Harbourline.Webhooks;

var builder = WebApplication.CreateSlimBuilder(args);
builder.Services.AddSingleton(TimeProvider.System);
builder.Services.AddSingleton(sp => new WebhookVerifier(
    builder.Configuration.GetSection("Webhooks:Paylinq:Secrets").Get<string[]>() ?? [],
    sp.GetRequiredService<TimeProvider>()));
builder.Services.AddSingleton<IReplayCache, InMemoryReplayCache>();
builder.Services.AddSingleton<IEventQueue, InMemoryEventQueue>(); // SQS in production

var app = builder.Build();

const int MaxBodyBytes = 256 * 1024;

app.MapPost("/webhooks/payment-events", async (HttpRequest request, WebhookVerifier verifier,
    IReplayCache replay, IEventQueue queue, ILogger<Program> log, CancellationToken ct) =>
{
    if (request.ContentLength is > MaxBodyBytes) return Results.StatusCode(StatusCodes.Status413PayloadTooLarge);

    using var buffer = new MemoryStream();
    await request.Body.CopyToAsync(buffer, ct);
    if (buffer.Length > MaxBodyBytes) return Results.StatusCode(StatusCodes.Status413PayloadTooLarge);
    var rawBody = buffer.ToArray();

    var id = request.Headers["webhook-id"].ToString();
    var result = verifier.Verify(id, request.Headers["webhook-timestamp"].ToString(),
        request.Headers["webhook-signature"].ToString(), rawBody);

    if (!result.IsValid)
    {
        log.LogWarning("Rejected webhook {WebhookId}: {Failure}", id, result.Failure);
        return result.Failure == WebhookVerificationFailure.MissingHeaders ? Results.BadRequest() : Results.Unauthorized();
    }

    if (await replay.IsKnownAsync(id, ct)) return Results.Ok(); // duplicate: acknowledge, do nothing

    await queue.EnqueueAsync(id, rawBody, ct);
    await replay.RememberAsync(id, TimeSpan.FromDays(7), ct);
    return Results.Accepted();
});

app.Run();

public interface IReplayCache
{
    ValueTask<bool> IsKnownAsync(string messageId, CancellationToken ct);
    ValueTask RememberAsync(string messageId, TimeSpan retention, CancellationToken ct);
}

public interface IEventQueue
{
    ValueTask EnqueueAsync(string messageId, byte[] rawBody, CancellationToken ct);
}

public sealed class InMemoryReplayCache(TimeProvider time) : IReplayCache
{
    private readonly System.Collections.Concurrent.ConcurrentDictionary<string, DateTimeOffset> _seen = new();

    public ValueTask<bool> IsKnownAsync(string messageId, CancellationToken ct) =>
        ValueTask.FromResult(_seen.TryGetValue(messageId, out var until) && until > time.GetUtcNow());

    public ValueTask RememberAsync(string messageId, TimeSpan retention, CancellationToken ct)
    {
        _seen[messageId] = time.GetUtcNow() + retention;
        return ValueTask.CompletedTask;
    }
}

public sealed class InMemoryEventQueue : IEventQueue
{
    private readonly System.Threading.Channels.Channel<(string, byte[])> _channel =
        System.Threading.Channels.Channel.CreateUnbounded<(string, byte[])>();

    public ValueTask EnqueueAsync(string messageId, byte[] rawBody, CancellationToken ct) =>
        _channel.Writer.WriteAsync((messageId, rawBody), ct);
}

The production idempotency check in the processor uses an atomic DynamoDB conditional write — the first writer wins, every later duplicate fails the condition:

src/Harbourline.Webhooks.Processor/DynamoDbProcessedEvents.cs
using System.Globalization;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;

namespace Harbourline.Webhooks.Processor;

public sealed class DynamoDbProcessedEvents(IAmazonDynamoDB dynamo, string tableName, TimeProvider time)
{
    /// Returns true if this call claimed the id; false if another delivery already did.
    public async Task<bool> TryClaimAsync(string webhookId, CancellationToken ct)
    {
        var expiresAt = time.GetUtcNow().AddDays(7).ToUnixTimeSeconds();
        try
        {
            await dynamo.PutItemAsync(new PutItemRequest
            {
                TableName = tableName,
                Item = new Dictionary<string, AttributeValue>
                {
                    ["pk"] = new() { S = $"webhook#{webhookId}" },
                    ["expiresAt"] = new() { N = expiresAt.ToString(CultureInfo.InvariantCulture) },
                },
                ConditionExpression = "attribute_not_exists(pk)",
            }, ct);
            return true;
        }
        catch (ConditionalCheckFailedException)
        {
            return false;
        }
    }
}

Code explained line by line

  1. WebhookVerificationFailure — distinct reasons let the receiver choose 400 (malformed) or 401 (bad signature, stale timestamp) and log precisely, without telling the caller which check failed.
  2. Tolerance — the 5-minute replay window: short enough to make captured requests useless, long enough for clock drift.
  3. _keys — a list of active secrets, so old and new keys both work during rotation.
  4. TimeProvider — tests inject FakeTimeProvider to simulate expired and future timestamps without sleeping.
  5. DecodeSecret — strips whsec_ and base64-decodes; the HMAC key is the decoded bytes, not the string.
  6. id.Contains('.') — the dot is the separator in the signed content, so ids must not contain one.
  7. NumberStyles.None and Math.Abs(now - sentAt) — digits only; reject old and future timestamps.
  8. signedContent — id.timestamp. plus the raw body bytes, never a re-serialised object.
  9. HMACSHA256.HashData — the static one-shot API: thread-safe, nothing to dispose.
  10. The Split(' ') loop — checks every signature and skips versions you do not support (such as v1a).
  11. TryFromBase64String with written != 32 — malformed input is "no match", never a 500.
  12. FixedTimeEquals — comparison time does not depend on the first differing byte, so timings leak nothing.
  13. MaxBodyBytes — Content-Length can be missing or false, so the copied length is checked as well.
  14. CopyToAsync before any JSON work — the "raw body first" rule.
  15. Bare BadRequest() / Unauthorized() — reasons go to logs, not to the caller.
  16. IsKnownAsync → Ok() — duplicates are acknowledged so the sender stops retrying.
  17. Enqueue before remember — if enqueueing fails the id is not recorded, the sender retries, and nothing is lost.
  18. attribute_not_exists(pk) — the atomic "process once" guard; two racing invocations cannot both win.
  19. expiresAt in epoch seconds as a Number — the only format DynamoDB TTL accepts.

Run the lab

Follow the webhook lab walkthrough. Its tests cover a valid message, tampered body, wrong secret, expired and future timestamps, replay, and key rotation, plus receiver integration tests with WebApplicationFactory. Run them with dotnet test --solution Harbourline.Webhooks.slnx (or the Docker command in the lab README).

Sending webhooks to customers

Harbourline also sends webhooks — for example to SME customers' accounting systems when a payment settles. Now Harbourline is the producer and must be a good one.

Producer rules (mostly from the Standard Webhooks spec):

  • Sign every attempt: constant webhook-id, fresh webhook-timestamp.
  • Back off exponentially with jitter over days. The spec's example schedule: immediately, 5 s, 5 min, 30 min, 2 h, 5 h, 10 h, 14 h, 20 h, 24 h. SQS delays are capped at 15 minutes (DelaySeconds 0–900), so use stepwise re-enqueueing, EventBridge Scheduler or Step Functions Wait states for longer waits.
  • Interpret status codes: 2xx success; 3xx failure (never follow redirects); 410 Gone disables the endpoint; 429/502/504 mean slow down and honour Retry-After.
  • Dead-letter, alarm and notify the customer after the last attempt.
  • Time out after 15–30 seconds and isolate tenants so one slow customer cannot block others.

EventBridge API destinations as a managed sender

API destinations deliver events to HTTPS endpoints with managed retries, a DLQ and a per-destination rate limit (EventBridge User Guide, checked 2026-11-17):

PropertyValue
Connection auth typesBasic, API key, OAuth (client credentials); credentials stored in a Secrets Manager secret that EventBridge creates (its cost is included in the API destination charge)
Endpoint timeout5 seconds
Retries (event bus)Default maximums 24 hours and 185 attempts; retries on 401, 407, 409, 429 and 5xx; honours Retry-After
Rate limitInvocationRateLimitPerSecond per destination; service quota "Rate of invocations per API destination" defaults to 300 per second (adjustable)
Dead-letter queueSQS DLQ on the rule target
Private endpointsSupported through connections backed by VPC Lattice resource configurations
mTLSNot supported
HMAC signingNot supported — connection headers are static values, so no per-message signature can be computed
PriceUSD 0.20 per million invocations (pricing page example, us-east-1, checked 2026-11-17)

So API destinations fit customers who accept an API key or OAuth token. For customers who require a Standard Webhooks signature, add a signing step (a Lambda target, or a Step Functions state that calls a signing Lambda before an HTTP task) or run your own sender Lambda as above.

Egress IPs and SSRF

Customers often allow-list your source IPs. Run the sender Lambda in private subnets whose route goes through a NAT gateway with an Elastic IP, and publish those IPs. (The newer regional NAT gateway mode spans Availability Zones automatically and uses one Elastic IP per AZ — see Calling partner APIs safely.)

Letting customers register a URL that your servers will call is the textbook server-side request forgery risk (OWASP API7:2023). A malicious customer registers http://169.254.169.254/... or an internal hostname and makes your sender attack your network. Defences, layered:

  1. Validate on registration: https only, port 443, no IP literals, no credentials in the URL, hostname not under your internal domains.
  2. Re-check at connect time, because DNS can change after validation (DNS rebinding). Resolve, reject private/loopback/link-local addresses, and connect to the checked address.
  3. Disable redirects, cap response size, and never return the customer's response body to them (the OWASP example leaked credentials exactly that way).
  4. Run the sender in an isolated subnet whose security group and network ACL allow only outbound 443 to the internet, and enforce IMDSv2 on any EC2 hosts.
src/Harbourline.Webhooks/SafeEgressHandler.cs
using System.Net;
using System.Net.Sockets;

namespace Harbourline.Webhooks;

public static class SafeEgressHandler
{
    public static SocketsHttpHandler Create() => new()
    {
        AllowAutoRedirect = false,
        ConnectTimeout = TimeSpan.FromSeconds(5),
        ConnectCallback = async (context, ct) =>
        {
            var addresses = await Dns.GetHostAddressesAsync(context.DnsEndPoint.Host, ct);
            if (addresses.Length == 0 || addresses.Any(IsBlocked))
                throw new HttpRequestException($"Destination {context.DnsEndPoint.Host} is not allowed.");

            var socket = new Socket(SocketType.Stream, ProtocolType.Tcp) { NoDelay = true };
            try
            {
                await socket.ConnectAsync(new IPEndPoint(addresses[0], context.DnsEndPoint.Port), ct);
                return new NetworkStream(socket, ownsSocket: true);
            }
            catch
            {
                socket.Dispose();
                throw;
            }
        },
    };

    public static bool IsBlocked(IPAddress address)
    {
        if (address.IsIPv4MappedToIPv6) address = address.MapToIPv4();
        if (IPAddress.IsLoopback(address)) return true;

        if (address.AddressFamily == AddressFamily.InterNetwork)
        {
            var b = address.GetAddressBytes();
            return b[0] == 10                                   // 10.0.0.0/8
                || (b[0] == 172 && b[1] >= 16 && b[1] <= 31)    // 172.16.0.0/12
                || (b[0] == 192 && b[1] == 168)                 // 192.168.0.0/16
                || (b[0] == 169 && b[1] == 254)                 // link-local, includes instance metadata
                || (b[0] == 100 && b[1] >= 64 && b[1] <= 127)   // 100.64.0.0/10 carrier-grade NAT
                || b[0] == 0;                                   // 0.0.0.0/8
        }

        var v6 = address.GetAddressBytes();
        return address.IsIPv6LinkLocal || address.IsIPv6SiteLocal || (v6[0] & 0xFE) == 0xFC; // fc00::/7
    }
}

Security considerations

⚠ Learning shortcut — not for production

The lab reads webhook secrets from appsettings.Development.json / user secrets and keeps seen ids in memory. In production store each partner's secrets in Secrets Manager (KMS-encrypted, rotation runbook, least-privilege secretsmanager:GetSecretValue on that one ARN), cache them with the Secrets Manager caching client, and keep ids in DynamoDB with TTL so every Lambda instance shares the same view.

⚠ Learning shortcut — not for production

The local receiver runs without TLS and without throttling. In AWS, API Gateway provides TLS and stage/route throttling; add an AWS WAF web ACL (REST API) with a rate-based rule if the endpoint attracts abuse, and log rejected signatures as security events. If the partner publishes fixed source IPs, add them as an allow-list in addition to the signature, never instead of it.

  • Never log the secret, the signature header or full bodies. Log webhook-id, the result and a body hash. Payment events contain personal data under UK GDPR, so log retention and access need data-protection review.
  • Schema-validate after verification. A signature proves who sent it, not that the content is sensible: validate decimal amounts, currency codes and account id formats.
  • For high-value actions, treat the webhook as a hint and fetch the authoritative state from the partner's API by event id before moving money.
  • Prefer asymmetric v1a (ed25519) signatures when offered: the receiver holds only a public key, so a breached receiver cannot forge partner messages.

Banking use case

Paylinq posts chargeback.created for a card linked to ACC-0000456789 (customer CUS-00012345). The receiver verifies, enqueues and returns 202 in about 30 ms. The processor claims the webhook-id in DynamoDB, confirms the chargeback with Paylinq's API, opens a dispute case and publishes DisputeOpened, which the Notifications Service e-mails to alex.morgan@example.com. A duplicate three hours later fails the conditional write and is acknowledged without side effects. Card-data handling and retention in this flow still need review by Harbourline's compliance and security teams (PCI DSS, UK GDPR).

Common mistakes

  • Verifying a re-serialised body → read raw bytes before any parser touches them.
  • Comparing signatures with == → timing leak → use FixedTimeEquals.
  • Using the whsec_... string as the key → never matches → strip the prefix and base64-decode.
  • Doing business work inside the request → sender timeouts → duplicates. Enqueue and return 2xx.
  • Dedupe window equal to the tolerance → day-old retries processed twice. Keep ids for the retry window.
  • Following redirects in a sender → SSRF to 169.254.169.254. Disable redirects.
  • Detailed error bodies → help attackers. Return a bare 401; log the detail.

Troubleshooting

SymptomLikely causeFix
Every signature fails, even the partner's test eventKey not base64-decoded, or whsec_ prefix left inUse DecodeSecret; compare against the partner's published test vector
Signatures fail only behind API GatewayBody altered: base64-encoded Lambda payload not decoded, or a mapping templateDecode when IsBase64Encoded is true; use proxy integration
Valid events rejected as TimestampOutOfToleranceClock drift on a container host, or partner sends millisecondsCheck NTP; confirm the unit is seconds
Duplicates processedDedupe only in the receiver's memory, or TTL shorter than retry windowConditional write in the processor; 7-day TTL
Messages piling up in the DLQProcessor throws for one bad event and the batch retriesEnable ReportBatchItemFailures and return only the failed ids
Partner reports timeoutsReceiver calls downstream systems synchronously, or cold startsEnqueue only; trim dependencies; consider provisioned concurrency for the receiver
Sender blocked by a customer firewallLambda not in a VPC, so egress IP is not fixedPut it in private subnets behind a NAT gateway with Elastic IPs

Cost and cleanup

The lab is free: it runs locally with in-memory fakes. The optional AWS deployment uses API Gateway HTTP API, Lambda, SQS and on-demand DynamoDB at lab volumes (well inside free allowances or fractions of a cent), plus about USD 0.40/month prorated for one Secrets Manager secret. Do not create a NAT gateway for this lab; the sender section is design-only.

# Tear down the optional deployment
aws secretsmanager delete-secret --secret-id hl-m04-webhook-paylinq --force-delete-without-recovery \
  --profile learn-sandbox --region us-east-1
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m04-payment-events --query QueueUrl --output text --profile learn-sandbox --region us-east-1)" \
  --profile learn-sandbox --region us-east-1
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m04-payment-events-dlq --query QueueUrl --output text --profile learn-sandbox --region us-east-1)" \
  --profile learn-sandbox --region us-east-1
aws dynamodb delete-table --table-name hl-m04-webhook-ids --profile learn-sandbox --region us-east-1
# If you used CDK instead:  cdk destroy --profile learn-sandbox

# Final check: nothing tagged for the course should remain
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --profile learn-sandbox --region us-east-1

Exercises

  • 🟢 Guided: In the lab, add a test that signs a body with secret A, rotates the verifier to [A, B], then to [B], and asserts which deliveries pass at each stage.
  • 🟡 Challenge: Replace InMemoryReplayCache with a DynamoDB implementation and test it against DynamoDB Local in Docker (amazon/dynamodb-local), gated behind an environment variable.
  • 🔴 Live problem: "Since Tuesday, about 2% of Paylinq webhooks fail verification with NoMatchingSignature. Paylinq says nothing changed. The failures are all events with merchant names containing é or ü." Find the cause and fix it.

Assignment

Write a one-page webhook integration standard for Harbourline (Markdown in your portfolio repo): required headers and scheme, secret format and rotation procedure, tolerance and dedupe windows with justification, status codes you return, logging rules (what must never be logged), and the receiver architecture diagram. Add the lab's test results as evidence.

Solutions

Knowledge check

Interview questions

  • How do you secure a webhook endpoint? HMAC over id, timestamp and raw body; constant-time comparison; timestamp tolerance; id-based dedupe; TLS; rate limiting; secrets in a vault with rotation; optional IP allow-list on top.
  • Why FixedTimeEquals? Early-exit comparisons leak, through timing, how many leading bytes matched, enabling incremental forgery.
  • Rotate a secret without downtime? Verifier accepts a list of secrets; sender double-signs during the overlap; then retire the old secret.
  • Design a sender for 10,000 business customers. Queue per event, sign per attempt, backoff over days, per-tenant concurrency limits, DLQ plus notification, fixed egress IPs, SSRF controls, a delivery log customers can query.
  • Webhooks or polling for a sanctions-screening result? Webhooks for latency plus a reconciliation poll as a safety net, so a missed webhook never leaves a payment stuck.

Certification practice

Completion checklist

  • I can explain webhooks vs polling vs WebSockets to a colleague in two sentences
  • I can list the three Standard Webhooks headers and the exact signed content
  • My verifier uses raw bytes, constant-time comparison and a 5-minute tolerance
  • I know why dedupe retention is longer than the tolerance window
  • I ran the lab tests, including rotation and replay
  • I can explain why EventBridge API destinations cannot HMAC-sign
  • I ran the cleanup steps and confirmed nothing tagged Project=cea-lab remains

Key takeaways

  • A webhook is an untrusted POST: verify an HMAC over id.timestamp.raw-body, in constant time, with a timestamp tolerance.
  • Signatures prove origin; ids and conditional writes prevent double-processing; you need both.
  • Acknowledge in milliseconds and process asynchronously through SQS with a DLQ.
  • As a sender: sign each attempt, back off over days, dead-letter, publish fixed egress IPs and treat customer-supplied URLs as an SSRF risk.
  • EventBridge API destinations handle retries and auth, but not HMAC signing or mTLS.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page