Webhooks: send and receive safely with HMAC and replay protection
Receive partner webhooks on AWS with raw-body HMAC-SHA256 verification, timestamp tolerance, replay protection and fast acknowledgement, and send your own signed webhooks with retries, dead-letter queues and SSRF defences.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SQS / AWSSDK.DynamoDBv2 4.1.x) · Standard Webhooks v1 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-17.
Learning objectives
By the end of this lesson you can:
- Choose between webhooks, polling and WebSockets for a partner integration and justify the choice.
- Verify an inbound webhook the Standard Webhooks way: read the raw body, recompute the HMAC-SHA256 over
id.timestamp.body, compare in constant time, and enforce a timestamp tolerance. - Add replay protection and idempotency with a message-id store (in memory locally, DynamoDB with TTL in production).
- Rotate webhook secrets without downtime using multiple signatures.
- Build the receiving architecture API Gateway → Lambda receiver → SQS → processor that acknowledges in milliseconds and processes asynchronously.
- Send webhooks to customers with signing, exponential backoff, a dead-letter queue, fixed egress IPs and SSRF protection — and explain why EventBridge API destinations alone cannot sign them.
Prerequisites
- Idempotency keys — the same "process once" thinking applies here
- API Gateway REST vs HTTP vs WebSocket and Lambda integrations
- Module 3: .NET on AWS — Lambda with .NET 10, SQS basics
- Tools: .NET 10 SDK (or Docker), AWS CLI v2 with the
learn-sandboxSSO profile (only for the optional deploy) - Lab: Webhook HMAC lab walkthrough, code in
labs/m04-webhook-hmac/
Why this matters in banking
Harbourline Bank's (fictional) card processor, Paylinq, sends a webhook whenever a card payment is authorised, settled or charged back. An attacker who finds the URL can post a forged payment.settled event for ACC-0000123456 worth GBP 4,999.00; a receiver that trusts any JSON credits money that never existed. And when the receiver is slow, Paylinq's retries deliver the same chargeback three times, so the dispute team opens three cases.
Both incidents come from treating a webhook as a trusted, exactly-once message. It is an unauthenticated HTTP POST from the internet that may arrive late, twice, out of order, or forged.
The idea in plain language
A webhook is a "reverse API call": instead of you asking a partner "anything new?" every minute (polling), the partner calls your HTTPS endpoint when something happens. A WebSocket keeps one long-lived two-way connection open, which suits live dashboards but not server-to-server business events.
| Pattern | Who starts the call | Good for | Weak at |
|---|---|---|---|
| Polling | You, on a timer | Partners without push; simple firewalls | Latency vs cost trade-off; wasted calls; rate limits |
| Webhook | Partner, per event | Event notifications between organisations | You must expose an endpoint; authenticity; duplicates |
| WebSocket | Client opens, both send | Live UI updates, chat, streaming prices | Long-lived connections; reconnect logic; not for B2B events |
Because anyone can POST to your endpoint, the partner signs each message with a shared secret; your receiver recomputes the signature over the exact bytes received. A signed timestamp stops old messages being replayed, and a message id lets you ignore duplicates.
Analogy: A courier brings an envelope to the bank's back door. The tamper-evident seal is the HMAC; the date on the seal is the timestamp (the post room refuses anything sealed over five minutes ago); the reference number is the webhook id (the post room never processes the same envelope twice). The post room signs for the parcel at once and passes it to the back office — the courier never waits for reconciliation.
What problem the AWS services solve
There is no single "webhook service" in AWS; you assemble one from managed parts.
| Need | AWS building block | Honest limitations |
|---|---|---|
| Public HTTPS endpoint with throttling | Amazon API Gateway (HTTP or REST API) | Integration timeout means the receiver must be fast; mapping templates must not alter the body you verify |
| Verify signature and acknowledge | AWS Lambda (.NET 10, dotnet10) | Cold starts add latency; keep the receiver tiny |
| Decouple acknowledgement from processing | Amazon SQS standard queue + dead-letter queue | At-least-once delivery: duplicates are normal, so the processor must be idempotent. Maximum message size is 1 MiB (raised from 256 KiB in August 2025) |
| Replay / idempotency store | Amazon DynamoDB with TTL | TTL deletes expired items "typically within a few days", not instantly — filter on the expiry attribute if you read it |
| Signing secret storage | AWS Secrets Manager | About USD 0.40 per secret per month plus USD 0.05 per 10,000 API calls (us-east-1, checked 2026-11-17); cache it, do not fetch per request |
| Managed outbound delivery | Amazon EventBridge API destinations | Supports Basic, API key and OAuth client-credentials auth only — no per-message HMAC signing and no mTLS; 5-second endpoint timeout |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Function HTTP trigger | API Gateway + Lambda receiver | API Gateway is a separate resource with its own throttling and logs |
| Service Bus queue + queue trigger | SQS + Lambda event source mapping | No sessions on standard queues; rely on idempotency |
| Event Grid webhook delivery | EventBridge API destinations | Auth is Basic/API key/OAuth only; no HMAC signing |
| Key Vault secret | Secrets Manager (KMS-encrypted) | Cache it with the caching client |
| APIM policy checking a signature | Code in the Lambda | API Gateway has no built-in HMAC check |
| Cosmos DB TTL for dedupe ids | DynamoDB TTL + conditional writes | attribute_not_exists gives an atomic "first one wins" |
| NAT Gateway static IP for allow-lists | NAT gateway + Elastic IP | AWS also offers a regional NAT gateway mode |
Architecture
Security-boundary view:
The important line is the arrow labelled "verified only": nothing reaches the queue unless the signature matched. Everything to the left of that arrow treats the request as hostile.
Step-by-step implementation
Agree the scheme with the partner
Use a published scheme rather than inventing one. This lesson follows Standard Webhooks (standardwebhooks.com), which several webhook providers use:
- Headers:
webhook-id(unique per event, the same on every retry),webhook-timestamp(Unix seconds),webhook-signature. - Signature header: one or more space-separated entries of the form
v1,<base64 HMAC-SHA256>.v1means symmetric HMAC-SHA256;v1ais the asymmetric ed25519 variant. - Secret: 24–64 random bytes, base64-encoded, prefixed
whsec_. - Signed content:
webhook-id+.+webhook-timestamp+.+ the exact body bytes.
If a partner uses its own scheme (a single X-Signature header, hex encoding, a different signed string), write a separate verifier for it rather than loosening one verifier to accept several formats.
Read the raw body before anything parses it
The signature covers bytes. Deserialising and re-serialising changes whitespace, property order and number formats, so read the body into a byte array, verify, and only then parse. In API Gateway use a proxy integration (no mapping template); in Lambda decode the body when IsBase64Encoded is true.
Verify signature, then timestamp
Recompute HMAC-SHA256 for each active secret, compare with CryptographicOperations.FixedTimeEquals, and accept if any candidate matches. Then reject timestamps more than 5 minutes from your clock in either direction (the spec leaves the window to you). A correctly signed message can still be a replay, so you need both checks.
De-duplicate on webhook-id
Replay protection needs ids only for the tolerance window. Idempotency needs them for the sender's whole retry window: Standard Webhooks' example schedule retries for about 75 hours, each retry with the same webhook-id and a fresh timestamp (otherwise later retries would fail the tolerance check). Keep ids for at least 7 days.
Acknowledge fast, process later
Return 2xx as soon as the event is durably enqueued. The spec recommends sender timeouts of 15–30 seconds and EventBridge API destinations time out after 5 seconds, so a receiver that calls core banking synchronously causes retries and duplicates. The processor uses a DynamoDB conditional write as its source of truth for "already processed"; poison messages go to a dead-letter queue.
Rotate secrets with overlap
The partner signs with old and new secrets for a period (two v1, entries in the header); your verifier accepts either. When all traffic matches the new secret, remove the old one on both sides.
AWS Console instructions
Optional — the lab runs fully locally. Add the four mandatory tags to everything.
- Secrets Manager → Store a new secret → Other type of secret:
hl-m04-webhook-paylinq, valuewhsec_...(generate it as in the CLI section). - SQS → Create queue → Standard:
hl-m04-payment-events-dlq, thenhl-m04-payment-eventswith Dead-letter queue → Enabled, maximum receives5, SSE-SQS. - DynamoDB → Create table:
hl-m04-webhook-ids, partition keypk(String), on-demand; then Time to Live → Turn on with attributeexpiresAt. - Lambda → Create function: runtime .NET 10, arm64, from the lab's publish output.
- API Gateway → Create API → HTTP API: Lambda integration, route
POST /webhooks/payment-events, access logging with 30-day retention.
AWS CLI
# Generate a Standard Webhooks style secret locally: 32 random bytes, base64, whsec_ prefix
echo "whsec_$(openssl rand -base64 32)"
# Store it in Secrets Manager with the mandatory tags (paste the value from above)
aws secretsmanager create-secret --name hl-m04-webhook-paylinq \
--secret-string "whsec_REPLACE_ME" \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m04-webhook-hmac Key=Environment,Value=learn Key=Owner,Value=your-name \
--profile learn-sandbox --region us-east-1
# Dead-letter queue, then the main queue with a redrive policy (5 receives)
DLQ_URL=$(aws sqs create-queue --queue-name hl-m04-payment-events-dlq \
--attributes SqsManagedSseEnabled=true \
--tags Project=cea-lab,Lab=m04-webhook-hmac,Environment=learn,Owner=your-name \
--query QueueUrl --output text --profile learn-sandbox --region us-east-1)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" --attribute-names QueueArn \
--query Attributes.QueueArn --output text --profile learn-sandbox --region us-east-1)
aws sqs create-queue --queue-name hl-m04-payment-events \
--attributes "{\"SqsManagedSseEnabled\":\"true\",\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"$DLQ_ARN\\\",\\\"maxReceiveCount\\\":\\\"5\\\"}\"}" \
--tags Project=cea-lab,Lab=m04-webhook-hmac,Environment=learn,Owner=your-name \
--profile learn-sandbox --region us-east-1
# Idempotency table with TTL on expiresAt
aws dynamodb create-table --table-name hl-m04-webhook-ids \
--attribute-definitions AttributeName=pk,AttributeType=S \
--key-schema AttributeName=pk,KeyType=HASH --billing-mode PAY_PER_REQUEST \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m04-webhook-hmac Key=Environment,Value=learn Key=Owner,Value=your-name \
--profile learn-sandbox --region us-east-1
aws dynamodb update-time-to-live --table-name hl-m04-webhook-ids \
--time-to-live-specification "Enabled=true,AttributeName=expiresAt" \
--profile learn-sandbox --region us-east-1Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.Lambda.EventSources;
using Amazon.CDK.AWS.SQS;
using Constructs;
namespace Harbourline.Webhooks.Infra;
public sealed class WebhookReceiverStack : Stack
{
public WebhookReceiverStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
var dlq = new Queue(this, "Dlq", new QueueProps
{
QueueName = "hl-m04-payment-events-dlq",
Encryption = QueueEncryption.SQS_MANAGED,
RetentionPeriod = Duration.Days(14),
});
var queue = new Queue(this, "Events", new QueueProps
{
QueueName = "hl-m04-payment-events",
Encryption = QueueEncryption.SQS_MANAGED,
VisibilityTimeout = Duration.Seconds(60),
DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 },
});
var ids = new TableV2(this, "WebhookIds", new TablePropsV2
{
TableName = "hl-m04-webhook-ids",
PartitionKey = new Amazon.CDK.AWS.DynamoDB.Attribute { Name = "pk", Type = AttributeType.STRING },
TimeToLiveAttribute = "expiresAt",
RemovalPolicy = RemovalPolicy.DESTROY, // learning shortcut: production keeps data
});
var processor = new Function(this, "Processor", new FunctionProps
{
Runtime = Runtime.DOTNET_10,
Architecture = Architecture.ARM_64,
Handler = "Harbourline.Webhooks.Processor",
Code = Code.FromAsset("../src/Harbourline.Webhooks.Processor/bin/publish"),
Timeout = Duration.Seconds(30),
});
processor.AddEventSource(new SqsEventSource(queue, new SqsEventSourceProps
{
BatchSize = 10,
ReportBatchItemFailures = true,
}));
ids.GrantReadWriteData(processor);
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m04-webhook-hmac");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "your-name");
}
}The receiver function and HTTP API follow the same pattern (HttpApi with a HttpLambdaIntegration); grant it queue.GrantSendMessages, ids.GrantReadData and read access to the one secret.
Complete code
The lab project labs/m04-webhook-hmac/ contains the full solution (WebhookSigner, WebhookVerifier, IReplayCache, InMemoryReplayCache, WebhookSender, the receiver and tests). The listings below are simplified excerpts that compile on their own; the lab may differ in detail.
using System.Globalization;
using System.Security.Cryptography;
using System.Text;
namespace Harbourline.Webhooks;
public enum WebhookVerificationFailure { None, MissingHeaders, InvalidTimestamp, TimestampOutOfTolerance, NoMatchingSignature }
public sealed record WebhookVerificationResult(bool IsValid, WebhookVerificationFailure Failure)
{
public static readonly WebhookVerificationResult Success = new(true, WebhookVerificationFailure.None);
public static WebhookVerificationResult Fail(WebhookVerificationFailure f) => new(false, f);
}
public sealed class WebhookVerifier
{
private static readonly TimeSpan Tolerance = TimeSpan.FromMinutes(5);
private readonly IReadOnlyList<byte[]> _keys;
private readonly TimeProvider _time;
public WebhookVerifier(IEnumerable<string> activeSecrets, TimeProvider time)
{
_keys = activeSecrets.Select(DecodeSecret).ToArray();
if (_keys.Count == 0) throw new ArgumentException("At least one secret is required.", nameof(activeSecrets));
_time = time;
}
public static byte[] DecodeSecret(string secret)
{
const string prefix = "whsec_";
var base64 = secret.StartsWith(prefix, StringComparison.Ordinal) ? secret[prefix.Length..] : secret;
return Convert.FromBase64String(base64);
}
public WebhookVerificationResult Verify(string? id, string? timestamp, string? signatureHeader, byte[] rawBody)
{
if (string.IsNullOrEmpty(id) || string.IsNullOrEmpty(timestamp) || string.IsNullOrEmpty(signatureHeader) || id.Contains('.'))
return WebhookVerificationResult.Fail(WebhookVerificationFailure.MissingHeaders);
if (!long.TryParse(timestamp, NumberStyles.None, CultureInfo.InvariantCulture, out var sentAt))
return WebhookVerificationResult.Fail(WebhookVerificationFailure.InvalidTimestamp);
var now = _time.GetUtcNow().ToUnixTimeSeconds();
if (Math.Abs(now - sentAt) > (long)Tolerance.TotalSeconds)
return WebhookVerificationResult.Fail(WebhookVerificationFailure.TimestampOutOfTolerance);
var prefix = Encoding.UTF8.GetBytes($"{id}.{timestamp}.");
var signedContent = new byte[prefix.Length + rawBody.Length];
prefix.CopyTo(signedContent, 0);
rawBody.CopyTo(signedContent, prefix.Length);
var expected = _keys.Select(k => HMACSHA256.HashData(k, signedContent)).ToArray();
foreach (var entry in signatureHeader.Split(' ', StringSplitOptions.RemoveEmptyEntries))
{
var comma = entry.IndexOf(',');
if (comma <= 0 || entry[..comma] != "v1") continue;
var candidate = new byte[32];
if (!Convert.TryFromBase64String(entry[(comma + 1)..], candidate, out var written) || written != 32) continue;
foreach (var mac in expected)
{
if (CryptographicOperations.FixedTimeEquals(mac, candidate))
return WebhookVerificationResult.Success;
}
}
return WebhookVerificationResult.Fail(WebhookVerificationFailure.NoMatchingSignature);
}
}using Harbourline.Webhooks;
var builder = WebApplication.CreateSlimBuilder(args);
builder.Services.AddSingleton(TimeProvider.System);
builder.Services.AddSingleton(sp => new WebhookVerifier(
builder.Configuration.GetSection("Webhooks:Paylinq:Secrets").Get<string[]>() ?? [],
sp.GetRequiredService<TimeProvider>()));
builder.Services.AddSingleton<IReplayCache, InMemoryReplayCache>();
builder.Services.AddSingleton<IEventQueue, InMemoryEventQueue>(); // SQS in production
var app = builder.Build();
const int MaxBodyBytes = 256 * 1024;
app.MapPost("/webhooks/payment-events", async (HttpRequest request, WebhookVerifier verifier,
IReplayCache replay, IEventQueue queue, ILogger<Program> log, CancellationToken ct) =>
{
if (request.ContentLength is > MaxBodyBytes) return Results.StatusCode(StatusCodes.Status413PayloadTooLarge);
using var buffer = new MemoryStream();
await request.Body.CopyToAsync(buffer, ct);
if (buffer.Length > MaxBodyBytes) return Results.StatusCode(StatusCodes.Status413PayloadTooLarge);
var rawBody = buffer.ToArray();
var id = request.Headers["webhook-id"].ToString();
var result = verifier.Verify(id, request.Headers["webhook-timestamp"].ToString(),
request.Headers["webhook-signature"].ToString(), rawBody);
if (!result.IsValid)
{
log.LogWarning("Rejected webhook {WebhookId}: {Failure}", id, result.Failure);
return result.Failure == WebhookVerificationFailure.MissingHeaders ? Results.BadRequest() : Results.Unauthorized();
}
if (await replay.IsKnownAsync(id, ct)) return Results.Ok(); // duplicate: acknowledge, do nothing
await queue.EnqueueAsync(id, rawBody, ct);
await replay.RememberAsync(id, TimeSpan.FromDays(7), ct);
return Results.Accepted();
});
app.Run();
public interface IReplayCache
{
ValueTask<bool> IsKnownAsync(string messageId, CancellationToken ct);
ValueTask RememberAsync(string messageId, TimeSpan retention, CancellationToken ct);
}
public interface IEventQueue
{
ValueTask EnqueueAsync(string messageId, byte[] rawBody, CancellationToken ct);
}
public sealed class InMemoryReplayCache(TimeProvider time) : IReplayCache
{
private readonly System.Collections.Concurrent.ConcurrentDictionary<string, DateTimeOffset> _seen = new();
public ValueTask<bool> IsKnownAsync(string messageId, CancellationToken ct) =>
ValueTask.FromResult(_seen.TryGetValue(messageId, out var until) && until > time.GetUtcNow());
public ValueTask RememberAsync(string messageId, TimeSpan retention, CancellationToken ct)
{
_seen[messageId] = time.GetUtcNow() + retention;
return ValueTask.CompletedTask;
}
}
public sealed class InMemoryEventQueue : IEventQueue
{
private readonly System.Threading.Channels.Channel<(string, byte[])> _channel =
System.Threading.Channels.Channel.CreateUnbounded<(string, byte[])>();
public ValueTask EnqueueAsync(string messageId, byte[] rawBody, CancellationToken ct) =>
_channel.Writer.WriteAsync((messageId, rawBody), ct);
}The production idempotency check in the processor uses an atomic DynamoDB conditional write — the first writer wins, every later duplicate fails the condition:
using System.Globalization;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
namespace Harbourline.Webhooks.Processor;
public sealed class DynamoDbProcessedEvents(IAmazonDynamoDB dynamo, string tableName, TimeProvider time)
{
/// Returns true if this call claimed the id; false if another delivery already did.
public async Task<bool> TryClaimAsync(string webhookId, CancellationToken ct)
{
var expiresAt = time.GetUtcNow().AddDays(7).ToUnixTimeSeconds();
try
{
await dynamo.PutItemAsync(new PutItemRequest
{
TableName = tableName,
Item = new Dictionary<string, AttributeValue>
{
["pk"] = new() { S = $"webhook#{webhookId}" },
["expiresAt"] = new() { N = expiresAt.ToString(CultureInfo.InvariantCulture) },
},
ConditionExpression = "attribute_not_exists(pk)",
}, ct);
return true;
}
catch (ConditionalCheckFailedException)
{
return false;
}
}
}Code explained line by line
WebhookVerificationFailure— distinct reasons let the receiver choose 400 (malformed) or 401 (bad signature, stale timestamp) and log precisely, without telling the caller which check failed.Tolerance— the 5-minute replay window: short enough to make captured requests useless, long enough for clock drift._keys— a list of active secrets, so old and new keys both work during rotation.TimeProvider— tests injectFakeTimeProviderto simulate expired and future timestamps without sleeping.DecodeSecret— stripswhsec_and base64-decodes; the HMAC key is the decoded bytes, not the string.id.Contains('.')— the dot is the separator in the signed content, so ids must not contain one.NumberStyles.NoneandMath.Abs(now - sentAt)— digits only; reject old and future timestamps.signedContent—id.timestamp.plus the raw body bytes, never a re-serialised object.HMACSHA256.HashData— the static one-shot API: thread-safe, nothing to dispose.- The
Split(' ')loop — checks every signature and skips versions you do not support (such asv1a). TryFromBase64Stringwithwritten != 32— malformed input is "no match", never a 500.FixedTimeEquals— comparison time does not depend on the first differing byte, so timings leak nothing.MaxBodyBytes—Content-Lengthcan be missing or false, so the copied length is checked as well.CopyToAsyncbefore any JSON work — the "raw body first" rule.- Bare
BadRequest()/Unauthorized()— reasons go to logs, not to the caller. IsKnownAsync→Ok()— duplicates are acknowledged so the sender stops retrying.- Enqueue before remember — if enqueueing fails the id is not recorded, the sender retries, and nothing is lost.
attribute_not_exists(pk)— the atomic "process once" guard; two racing invocations cannot both win.expiresAtin epoch seconds as a Number — the only format DynamoDB TTL accepts.
Run the lab
Follow the webhook lab walkthrough. Its tests cover a valid message, tampered body, wrong secret, expired and future timestamps, replay, and key rotation, plus receiver integration tests with WebApplicationFactory. Run them with dotnet test --solution Harbourline.Webhooks.slnx (or the Docker command in the lab README).
Sending webhooks to customers
Harbourline also sends webhooks — for example to SME customers' accounting systems when a payment settles. Now Harbourline is the producer and must be a good one.
Producer rules (mostly from the Standard Webhooks spec):
- Sign every attempt: constant
webhook-id, freshwebhook-timestamp. - Back off exponentially with jitter over days. The spec's example schedule: immediately, 5 s, 5 min, 30 min, 2 h, 5 h, 10 h, 14 h, 20 h, 24 h. SQS delays are capped at 15 minutes (
DelaySeconds0–900), so use stepwise re-enqueueing, EventBridge Scheduler or Step FunctionsWaitstates for longer waits. - Interpret status codes: 2xx success; 3xx failure (never follow redirects);
410 Gonedisables the endpoint; 429/502/504 mean slow down and honourRetry-After. - Dead-letter, alarm and notify the customer after the last attempt.
- Time out after 15–30 seconds and isolate tenants so one slow customer cannot block others.
EventBridge API destinations as a managed sender
API destinations deliver events to HTTPS endpoints with managed retries, a DLQ and a per-destination rate limit (EventBridge User Guide, checked 2026-11-17):
| Property | Value |
|---|---|
| Connection auth types | Basic, API key, OAuth (client credentials); credentials stored in a Secrets Manager secret that EventBridge creates (its cost is included in the API destination charge) |
| Endpoint timeout | 5 seconds |
| Retries (event bus) | Default maximums 24 hours and 185 attempts; retries on 401, 407, 409, 429 and 5xx; honours Retry-After |
| Rate limit | InvocationRateLimitPerSecond per destination; service quota "Rate of invocations per API destination" defaults to 300 per second (adjustable) |
| Dead-letter queue | SQS DLQ on the rule target |
| Private endpoints | Supported through connections backed by VPC Lattice resource configurations |
| mTLS | Not supported |
| HMAC signing | Not supported — connection headers are static values, so no per-message signature can be computed |
| Price | USD 0.20 per million invocations (pricing page example, us-east-1, checked 2026-11-17) |
So API destinations fit customers who accept an API key or OAuth token. For customers who require a Standard Webhooks signature, add a signing step (a Lambda target, or a Step Functions state that calls a signing Lambda before an HTTP task) or run your own sender Lambda as above.
Egress IPs and SSRF
Customers often allow-list your source IPs. Run the sender Lambda in private subnets whose route goes through a NAT gateway with an Elastic IP, and publish those IPs. (The newer regional NAT gateway mode spans Availability Zones automatically and uses one Elastic IP per AZ — see Calling partner APIs safely.)
Letting customers register a URL that your servers will call is the textbook server-side request forgery risk (OWASP API7:2023). A malicious customer registers http://169.254.169.254/... or an internal hostname and makes your sender attack your network. Defences, layered:
- Validate on registration:
httpsonly, port 443, no IP literals, no credentials in the URL, hostname not under your internal domains. - Re-check at connect time, because DNS can change after validation (DNS rebinding). Resolve, reject private/loopback/link-local addresses, and connect to the checked address.
- Disable redirects, cap response size, and never return the customer's response body to them (the OWASP example leaked credentials exactly that way).
- Run the sender in an isolated subnet whose security group and network ACL allow only outbound 443 to the internet, and enforce IMDSv2 on any EC2 hosts.
using System.Net;
using System.Net.Sockets;
namespace Harbourline.Webhooks;
public static class SafeEgressHandler
{
public static SocketsHttpHandler Create() => new()
{
AllowAutoRedirect = false,
ConnectTimeout = TimeSpan.FromSeconds(5),
ConnectCallback = async (context, ct) =>
{
var addresses = await Dns.GetHostAddressesAsync(context.DnsEndPoint.Host, ct);
if (addresses.Length == 0 || addresses.Any(IsBlocked))
throw new HttpRequestException($"Destination {context.DnsEndPoint.Host} is not allowed.");
var socket = new Socket(SocketType.Stream, ProtocolType.Tcp) { NoDelay = true };
try
{
await socket.ConnectAsync(new IPEndPoint(addresses[0], context.DnsEndPoint.Port), ct);
return new NetworkStream(socket, ownsSocket: true);
}
catch
{
socket.Dispose();
throw;
}
},
};
public static bool IsBlocked(IPAddress address)
{
if (address.IsIPv4MappedToIPv6) address = address.MapToIPv4();
if (IPAddress.IsLoopback(address)) return true;
if (address.AddressFamily == AddressFamily.InterNetwork)
{
var b = address.GetAddressBytes();
return b[0] == 10 // 10.0.0.0/8
|| (b[0] == 172 && b[1] >= 16 && b[1] <= 31) // 172.16.0.0/12
|| (b[0] == 192 && b[1] == 168) // 192.168.0.0/16
|| (b[0] == 169 && b[1] == 254) // link-local, includes instance metadata
|| (b[0] == 100 && b[1] >= 64 && b[1] <= 127) // 100.64.0.0/10 carrier-grade NAT
|| b[0] == 0; // 0.0.0.0/8
}
var v6 = address.GetAddressBytes();
return address.IsIPv6LinkLocal || address.IsIPv6SiteLocal || (v6[0] & 0xFE) == 0xFC; // fc00::/7
}
}Security considerations
⚠ Learning shortcut — not for production
The lab reads webhook secrets from appsettings.Development.json / user secrets and keeps seen ids in memory. In production store each partner's secrets in Secrets Manager (KMS-encrypted, rotation runbook, least-privilege secretsmanager:GetSecretValue on that one ARN), cache them with the Secrets Manager caching client, and keep ids in DynamoDB with TTL so every Lambda instance shares the same view.
⚠ Learning shortcut — not for production
The local receiver runs without TLS and without throttling. In AWS, API Gateway provides TLS and stage/route throttling; add an AWS WAF web ACL (REST API) with a rate-based rule if the endpoint attracts abuse, and log rejected signatures as security events. If the partner publishes fixed source IPs, add them as an allow-list in addition to the signature, never instead of it.
- Never log the secret, the signature header or full bodies. Log
webhook-id, the result and a body hash. Payment events contain personal data under UK GDPR, so log retention and access need data-protection review. - Schema-validate after verification. A signature proves who sent it, not that the content is sensible: validate
decimalamounts, currency codes and account id formats. - For high-value actions, treat the webhook as a hint and fetch the authoritative state from the partner's API by event id before moving money.
- Prefer asymmetric
v1a(ed25519) signatures when offered: the receiver holds only a public key, so a breached receiver cannot forge partner messages.
Banking use case
Paylinq posts chargeback.created for a card linked to ACC-0000456789 (customer CUS-00012345). The receiver verifies, enqueues and returns 202 in about 30 ms. The processor claims the webhook-id in DynamoDB, confirms the chargeback with Paylinq's API, opens a dispute case and publishes DisputeOpened, which the Notifications Service e-mails to alex.morgan@example.com. A duplicate three hours later fails the conditional write and is acknowledged without side effects. Card-data handling and retention in this flow still need review by Harbourline's compliance and security teams (PCI DSS, UK GDPR).
Common mistakes
- Verifying a re-serialised body → read raw bytes before any parser touches them.
- Comparing signatures with
==→ timing leak → useFixedTimeEquals. - Using the
whsec_...string as the key → never matches → strip the prefix and base64-decode. - Doing business work inside the request → sender timeouts → duplicates. Enqueue and return 2xx.
- Dedupe window equal to the tolerance → day-old retries processed twice. Keep ids for the retry window.
- Following redirects in a sender → SSRF to
169.254.169.254. Disable redirects. - Detailed error bodies → help attackers. Return a bare 401; log the detail.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Every signature fails, even the partner's test event | Key not base64-decoded, or whsec_ prefix left in | Use DecodeSecret; compare against the partner's published test vector |
| Signatures fail only behind API Gateway | Body altered: base64-encoded Lambda payload not decoded, or a mapping template | Decode when IsBase64Encoded is true; use proxy integration |
Valid events rejected as TimestampOutOfTolerance | Clock drift on a container host, or partner sends milliseconds | Check NTP; confirm the unit is seconds |
| Duplicates processed | Dedupe only in the receiver's memory, or TTL shorter than retry window | Conditional write in the processor; 7-day TTL |
| Messages piling up in the DLQ | Processor throws for one bad event and the batch retries | Enable ReportBatchItemFailures and return only the failed ids |
| Partner reports timeouts | Receiver calls downstream systems synchronously, or cold starts | Enqueue only; trim dependencies; consider provisioned concurrency for the receiver |
| Sender blocked by a customer firewall | Lambda not in a VPC, so egress IP is not fixed | Put it in private subnets behind a NAT gateway with Elastic IPs |
Cost and cleanup
The lab is free: it runs locally with in-memory fakes. The optional AWS deployment uses API Gateway HTTP API, Lambda, SQS and on-demand DynamoDB at lab volumes (well inside free allowances or fractions of a cent), plus about USD 0.40/month prorated for one Secrets Manager secret. Do not create a NAT gateway for this lab; the sender section is design-only.
# Tear down the optional deployment
aws secretsmanager delete-secret --secret-id hl-m04-webhook-paylinq --force-delete-without-recovery \
--profile learn-sandbox --region us-east-1
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m04-payment-events --query QueueUrl --output text --profile learn-sandbox --region us-east-1)" \
--profile learn-sandbox --region us-east-1
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m04-payment-events-dlq --query QueueUrl --output text --profile learn-sandbox --region us-east-1)" \
--profile learn-sandbox --region us-east-1
aws dynamodb delete-table --table-name hl-m04-webhook-ids --profile learn-sandbox --region us-east-1
# If you used CDK instead: cdk destroy --profile learn-sandbox
# Final check: nothing tagged for the course should remain
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1Exercises
- 🟢 Guided: In the lab, add a test that signs a body with secret A, rotates the verifier to
[A, B], then to[B], and asserts which deliveries pass at each stage. - 🟡 Challenge: Replace
InMemoryReplayCachewith a DynamoDB implementation and test it against DynamoDB Local in Docker (amazon/dynamodb-local), gated behind an environment variable. - 🔴 Live problem: "Since Tuesday, about 2% of Paylinq webhooks fail verification with
NoMatchingSignature. Paylinq says nothing changed. The failures are all events with merchant names containingéorü." Find the cause and fix it.
Assignment
Write a one-page webhook integration standard for Harbourline (Markdown in your portfolio repo): required headers and scheme, secret format and rotation procedure, tolerance and dedupe windows with justification, status codes you return, logging rules (what must never be logged), and the receiver architecture diagram. Add the lab's test results as evidence.
Solutions
Use WebhookSigner (or HMACSHA256.HashData) with key A to produce v1,<sig>. Create three verifiers sharing a FakeTimeProvider: new WebhookVerifier([secretA], time), [secretA, secretB] and [secretB]. Expected: A-signed passes the first two and fails the third; B-signed fails the first and passes the last two; a header carrying both signatures passes all three. Why: this models the overlap period — the partner double-signs while you accept both, so neither side has to switch at exactly the same second.
Implement IReplayCache with PutItemAsync and ConditionExpression = "attribute_not_exists(pk)" (as in TryClaimAsync) and GetItemAsync with ConsistentRead = true for IsKnownAsync; filter out items whose expiresAt is in the past, because TTL deletion can lag by days. Point AmazonDynamoDBConfig.ServiceURL at http://localhost:8000 for DynamoDB Local and skip the test unless RUN_DYNAMO_TESTS=1. Why: the conditional write is atomic across all Lambda instances; an in-memory dictionary is per instance.
The body is round-tripped through a string with the wrong encoding (for example Encoding.ASCII, or a middleware that normalises Unicode), which changes the bytes for é while ASCII-only events survive. Fix: decode base64 when IsBase64Encoded, otherwise Encoding.UTF8.GetBytes on the untouched value, and add a regression test with Café Ünited Ltd in the body. Why: HMAC is over bytes; any transformation breaks it.
Knowledge check
Answer: The HMAC is computed over the exact bytes the sender transmitted. Parsing and re-serialising can change whitespace, key order, escaping and number formats, so the recomputed HMAC would not match — or, worse, a lenient implementation might verify a different representation from the one it processes.
Answer: No. Reject it (401). A valid signature proves Paylinq created it at some point; the stale timestamp suggests a captured request being replayed. The timestamp is inside the signed content, so an attacker cannot refresh it without the secret.
Answer: Retries of the same event arrive with the same webhook-id and a fresh timestamp, potentially days later. Replay protection needs only the tolerance window; idempotency needs the sender's whole retry window.
Answer: Not by themselves. Connections support Basic, API key and OAuth client-credentials authorization with static parameters; there is no per-message HMAC computation and no mTLS. Add a signing step (Lambda) or use your own sender.
Answer: The sender treats any non-2xx as failure and retries. A duplicate means "already handled", which is success from the sender's point of view; returning an error would create a retry storm.
Interview questions
- How do you secure a webhook endpoint? HMAC over id, timestamp and raw body; constant-time comparison; timestamp tolerance; id-based dedupe; TLS; rate limiting; secrets in a vault with rotation; optional IP allow-list on top.
- Why
FixedTimeEquals? Early-exit comparisons leak, through timing, how many leading bytes matched, enabling incremental forgery. - Rotate a secret without downtime? Verifier accepts a list of secrets; sender double-signs during the overlap; then retire the old secret.
- Design a sender for 10,000 business customers. Queue per event, sign per attempt, backoff over days, per-tenant concurrency limits, DLQ plus notification, fixed egress IPs, SSRF controls, a delivery log customers can query.
- Webhooks or polling for a sanctions-screening result? Webhooks for latency plus a reconciliation poll as a safety net, so a missed webhook never leaves a payment stuck.
Certification practice
A payments partner calls an API Gateway HTTP API backed by a Lambda function. The function verifies an HMAC, then calls an internal ledger API that sometimes takes 25 seconds. The partner times out after 10 seconds and resends, and the ledger shows duplicate postings. What is the MOST effective change?
A. Increase the Lambda timeout to 15 minutes. B. Verify the signature, send the event to an SQS queue, return 202, and process it in a second Lambda that uses a DynamoDB conditional write on the event id. C. Enable API Gateway caching for the route. D. Configure reserved concurrency of 1 on the function.
Answer: B. Fast acknowledgement removes the timeout-driven retries, and the conditional write makes processing idempotent for the duplicates that will still occur. A does not change the partner's 10-second timeout. C caches responses for GET-style reads, not POST side effects. D serialises traffic and makes timeouts worse.
A team stores a webhook HMAC secret as a plaintext Lambda environment variable that every developer can read in the console. Which solution BEST protects the secret while keeping latency low?
A. Base64-encode the environment variable.
B. Store the secret in AWS Secrets Manager, grant secretsmanager:GetSecretValue on that secret only to the function's execution role, and cache it in memory with the Secrets Manager caching client.
C. Put the secret in the API Gateway stage variables.
D. Hard-code the secret in the deployment package.
Answer: B. Secrets Manager encrypts with KMS, centralises rotation and lets IAM restrict access to one role; caching avoids a network call per request. A is encoding, not encryption. C exposes it to anyone who can read the API configuration. D puts it in source control and every artefact.
A bank lets business customers register HTTPS callback URLs that a backend service calls when payments settle. A penetration test shows a callback URL can point at the instance metadata endpoint. Which combination of controls MOST reduces this risk? (Choose TWO.)
A. Resolve the hostname at connection time and refuse private, loopback and link-local addresses; disable HTTP redirects.
B. Increase the timeout for callback requests.
C. Run the sender in an isolated subnet with egress restricted to port 443 and require IMDSv2 on any EC2 hosts.
D. Return the callback response body to the customer for debugging.
E. Allow only http URLs so traffic can be inspected.
Answer: A and C. A blocks SSRF to internal ranges even after DNS changes; C limits blast radius and hardens metadata access. B is irrelevant. D is exactly how SSRF results leak. E removes transport encryption.
Unit tests for a webhook verifier pass locally, but every request through API Gateway fails verification. The Lambda reads request.Body from the proxy event as a string and converts it with Encoding.ASCII.GetBytes. What is the MOST likely cause?
A. API Gateway strips the webhook-signature header.
B. The body may be base64-encoded (IsBase64Encoded) or contain non-ASCII characters, so the bytes differ from those that were signed.
C. HMAC-SHA256 is not supported on the dotnet10 runtime.
D. The Lambda function needs kms:Decrypt.
Answer: B. The verifier must use the original bytes: decode base64 when flagged, otherwise UTF-8. A is false for custom headers. C is false. D would fail when fetching a secret, not on every signature comparison.
Completion checklist
- I can explain webhooks vs polling vs WebSockets to a colleague in two sentences
- I can list the three Standard Webhooks headers and the exact signed content
- My verifier uses raw bytes, constant-time comparison and a 5-minute tolerance
- I know why dedupe retention is longer than the tolerance window
- I ran the lab tests, including rotation and replay
- I can explain why EventBridge API destinations cannot HMAC-sign
- I ran the cleanup steps and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- A webhook is an untrusted POST: verify an HMAC over
id.timestamp.raw-body, in constant time, with a timestamp tolerance. - Signatures prove origin; ids and conditional writes prevent double-processing; you need both.
- Acknowledge in milliseconds and process asynchronously through SQS with a DLQ.
- As a sender: sign each attempt, back off over days, dead-letter, publish fixed egress IPs and treat customer-supplied URLs as an SSRF risk.
- EventBridge API destinations handle retries and auth, but not HMAC signing or mTLS.
Further reading
- Standard Webhooks and the Standard Webhooks specification
- API destinations as targets in Amazon EventBridge
- Connections for API targets in Amazon EventBridge and authorization methods
- Amazon EventBridge quotas
- Using time to live (TTL) in DynamoDB
- Amazon SQS increases maximum message payload size to 1 MiB
- OWASP API7:2023 Server Side Request Forgery
- Related lessons: Idempotency keys, Calling partner APIs safely, API security with WAF and OWASP
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Request validation, models, versioning and CORS
Reject bad input in layers (API Gateway models and validators, ASP.NET Core 10 validation, mass-assignment defences), version APIs on API Gateway, and finally understand CORS preflight on REST APIs, HTTP APIs and ASP.NET Core.
Lab: Webhook receiver with HMAC verification and replay protection (Lab 4.3)
Run and extend a .NET 10 webhook receiver that verifies an HMAC-SHA256 signature over the raw body, rejects stale and future timestamps, acknowledges duplicates without processing them twice, supports key rotation, and a sender that retries safely and refuses to call private addresses.