Lab: Webhook receiver with HMAC verification and replay protection (Lab 4.3)
Run and extend a .NET 10 webhook receiver that verifies an HMAC-SHA256 signature over the raw body, rejects stale and future timestamps, acknowledges duplicates without processing them twice, supports key rotation, and a sender that retries safely and refuses to call private addresses.
Assumes: .NET 10 LTS · ASP.NET Core minimal APIs · xUnit v3 4.0.3 · Standard Webhooks v1 signature format · Region us-east-1 (design only). Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lab you can:
- Verify a Standard Webhooks style signature (
v1,<base64 HMAC-SHA256>overid.timestamp.body) against the raw request bytes, using a constant-time comparison. - Explain why a valid signature alone does not stop replay, and show the two defences the lab uses: a timestamp tolerance and a de-duplication cache.
- Run a receiver that verifies first, de-duplicates second, enqueues third and returns
202quickly, without doing business work inside the request. - Rotate a signing secret with zero downtime by accepting two secrets and signing with both.
- Send webhooks as a well-behaved producer: same id on every retry, fresh timestamp and signature on every attempt, no redirects, and a guard against calls to private or metadata addresses (SSRF).
- Read 44 automated tests and say what each group proves.
This is Lab 4.3 from the module plan and the hands-on half of Webhooks: send and receive safely.
Prerequisites
- Webhooks: send and receive safely with HMAC and replay protection for the theory.
- Idempotency keys helps with the de-duplication idea.
- Docker Desktop. The .NET 10 SDK is optional.
- PowerShell for the curl-style example. No AWS account is needed.
Why this matters in banking
Harbourline's payments team receives payment.settled events from a fictional processor called Paylinq. If the receiver trusts any POST that looks right, anyone who finds the URL can tell the bank that a payment settled, and the ledger will release goods or funds. If the receiver does its work inline and times out, Paylinq retries and the same payment is settled twice. If the sender follows a customer-supplied URL blindly, an attacker registers http://169.254.169.254/ as their "webhook" and reads cloud credentials from the instance metadata service.
The lab shows each control in a few lines of code, with a test that proves it.
The idea in plain language
A webhook is an HTTP call that flows the wrong way round: the processor calls you. You cannot use a login, so the two sides share a secret. The sender computes a message authentication code (HMAC) over the message with the secret and sends it along. The receiver recomputes it. A match proves the message came from someone who knows the secret and was not changed.
That alone is not enough, because an attacker who records a valid message can send it again tomorrow. So the signature also covers the message id and a timestamp. The receiver rejects anything more than five minutes from its own clock, and remembers ids it has already accepted.
Analogy: a branch receives a courier bag with a numbered tamper seal and a delivery slip stamped with the time. The branch checks the seal against the shared codebook (HMAC), rejects a slip stamped last week (timestamp), and checks the register to see whether it already booked in a bag with that number (de-duplication).
What the AWS services do here
The lab runs entirely on your machine, but each in-memory stand-in names the AWS service that replaces it in production:
| Lab component | Production AWS service | Why |
|---|---|---|
Webhooks:Paylinq:Secrets in configuration | AWS Secrets Manager, one secret per partner | Rotation, audit trail and IAM-scoped access |
InMemoryReplayCache | Amazon DynamoDB with a conditional write and a TTL attribute | Shared by every instance, and old ids expire by themselves |
InMemoryEventQueue | Amazon SQS (standard queue plus a dead-letter queue) | Absorbs bursts, retries processing, keeps the HTTP response fast |
| The Kestrel receiver | API Gateway or an ALB in front of ECS/Lambda | TLS, WAF, rate limiting and a body size limit at the edge |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | This lab | What's different |
|---|---|---|
| Event Grid webhook validation handshake | Standard Webhooks signature headers | There is no handshake. Trust is the shared secret plus the signature on every message |
HMACSHA256 for a shared-key check in ASP.NET | Same primitive, with CryptographicOperations.FixedTimeEquals | Constant-time comparison avoids leaking how many bytes matched |
| Azure Service Bus as the work queue | SQS in production; Channel<T> in the lab | The receiver's job is only to hand the event over |
Azure Functions HTTP trigger reading req.Body | Raw HttpRequest.Body copied to bytes first | Model binding would re-serialise and change the bytes the signature covers |
Architecture
The order is deliberate. The id is remembered after the enqueue, so a crash between the two causes a duplicate (which the processing side must tolerate anyway) rather than a lost event.
Step-by-step implementation
The lab folder is labs/m04-webhook-hmac. The solution file is Harbourline.Webhooks.slnx.
src/Harbourline.Webhooks/ WebhookVerifier, WebhookSigner, WebhookSender, SafeEgressHandler,
IReplayCache/InMemoryReplayCache, IEventQueue/InMemoryEventQueue
src/Harbourline.Webhooks.Receiver/ ASP.NET Core minimal API: POST /webhooks/payment-events
tests/Harbourline.Webhooks.Tests/ verifier, rotation, receiver (WebApplicationFactory), sender, SSRF guardPart 1: run the tests (free)
Run them in Docker
MSYS_NO_PATHCONV=1 docker run --rm --memory 3g \
-v "$PWD/labs/m04-webhook-hmac:/src" \
-v cea-nuget-m04:/root/.nuget/packages \
-w /src mcr.microsoft.com/dotnet/sdk:10.0 \
sh -c "dotnet test --solution Harbourline.Webhooks.slnx; find . -type d \( -name bin -o -name obj \) -prune -exec rm -rf {} +"Expected result (verified 2026-11-18 in Docker): total: 44, failed: 0, succeeded: 44, skipped: 0. With the local SDK, dotnet test --solution Harbourline.Webhooks.slnx does the same.
Map the 44 tests to the controls
| Group | Tests | What it proves |
|---|---|---|
| Verifier basics | Valid_message_passes, Signature_matches_an_independent_hmac_computation | The algorithm is exactly HMAC-SHA256 over id.ts.body with the base64-decoded secret as key, checked against a second, hand-written computation |
| Tamper detection | Tampered_body_fails, Tampered_id_fails_because_the_id_is_signed, Wrong_secret_fails | Changing a byte of the body or the id, or using another secret, breaks verification |
| Time | Timestamp_outside_tolerance_fails (-301 s and +301 s), Captured_request_replayed_later_fails_the_timestamp_check | Both stale and future timestamps are rejected: the tolerance is five minutes either way |
| Bad input | Missing_or_malformed_headers_fail (4 cases), Non_numeric_timestamp_fails (3 cases), Malformed_signatures_are_no_match_not_exceptions (4 cases) | Garbage becomes a failure result, never an exception and never a 500 |
| Rotation | Key_rotation_overlap_accepts_old_new_and_double_signed_messages | During overlap, a verifier holding both secrets accepts either, and a message signed with both works for a verifier holding only one |
| Secret format | New_secret_has_the_expected_format_and_strength | New secrets are whsec_ plus base64 of 32 random bytes |
| Receiver | Valid_webhook_is_accepted_and_enqueued_with_its_raw_body, Duplicate_delivery_is_acknowledged_but_not_enqueued_twice, Wrong_secret_is_401_with_no_details, Stale_timestamp_is_401, Missing_headers_is_400, Oversized_body_is_413 | The HTTP contract, using WebApplicationFactory<Program> with a fixed clock and a known secret |
| Sender | Status_codes_map_to_delivery_outcomes (7 cases), Retries_keep_the_id_but_get_a_fresh_timestamp_and_signature_that_the_receiver_accepts | How responses map to retry decisions, and the id-stays, timestamp-changes rule |
| SSRF guard | Egress_guard_blocks_private_and_metadata_addresses (9 cases) | Loopback, 10/8, 172.16/12, 192.168/16, 169.254/16 (instance metadata), 100.64/10, ::1 and fd00::1 are blocked; a public documentation address is not |
Note the "odd spacing on purpose" body in the first receiver test: {"type":"payment.settled", "amount":"10.00"}. The test asserts that the queued body equals the sent text byte for byte. That is the raw-body rule in action.
Part 2: run the receiver and send a signed message (free)
Start the receiver
dotnet run --project src/Harbourline.Webhooks.ReceiverIt listens on http://localhost:5090 and reads the Development secret from appsettings.Development.json. That secret is a public test value in the repository.
Send a signed event from PowerShell
# The secret is the base64 text AFTER the whsec_ prefix, decoded to bytes.
$secret = [Convert]::FromBase64String("bG9jYWwtdGVzdC1vbmx5LXdlYmhvb2stc2VjcmV0LTAxMjM=")
$id = "msg_" + [guid]::NewGuid().ToString("N")
$ts = [DateTimeOffset]::UtcNow.ToUnixTimeSeconds()
$body = '{"type":"payment.settled","data":{"paymentId":"PAY-0001","amount":"125.00","currency":"GBP"}}'
$hmac = [System.Security.Cryptography.HMACSHA256]::new($secret)
$sig = [Convert]::ToBase64String($hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes("$id.$ts.$body")))
Invoke-WebRequest -Method Post -Uri http://localhost:5090/webhooks/payment-events -Body $body -ContentType "application/json" `
-Headers @{ "webhook-id" = $id; "webhook-timestamp" = "$ts"; "webhook-signature" = "v1,$sig" }Expected from the README: 202 the first time, 200 if you send the same $id again, 401 if you change one character of $body after signing. (The automated tests were run in Docker; the live PowerShell session was not, so treat these status codes as ⚠ VERIFY on your machine.)
Break it on purpose
Try each one and write down the status code:
- Change one character of the body after computing the signature.
- Set
$tsto[DateTimeOffset]::UtcNow.AddMinutes(-6).ToUnixTimeSeconds()and re-sign. - Use the
whsec_...string itself as the HMAC key instead of its decoded bytes. - Remove the
webhook-idheader. - Send the same valid message twice.
You should see 401, 401, 401, 400 and 202 then 200.
Part 3: read the code in the order a request flows (free)
WebhookEndpoint.HandleAsync
It checks ContentLength and the real buffered length against MaxBodyBytes (256 KiB) and returns 413 if exceeded. It copies the body into a MemoryStream and calls ToArray() before any parsing. Then it calls verifier.Verify. A failure is logged with the reason (TimestampOutOfTolerance, NoMatchingSignature, and so on) but the caller only sees 401 (or 400 for missing headers). Telling an attacker why a request failed helps them fix the next attempt.
WebhookVerifier.Verify
The order of checks matters: headers present, timestamp numeric, timestamp within tolerance, then the HMAC. Cheap checks first. The signature header may hold several space-separated entries, each v1,<base64>. The loop skips unsupported versions and malformed base64 and compares each candidate with CryptographicOperations.FixedTimeEquals against every active secret. A new byte[32] candidate and the written != 32 check reject truncated values without allocating based on attacker input.
WebhookSender and SafeEgressHandler
SendAsync creates a fresh timestamp and signature every attempt, but the caller passes the same webhookId for each retry. The result type tells the scheduler what to do:
| Response | DeliveryOutcome | Meaning |
|---|---|---|
| 2xx | Delivered | Done |
| 410 Gone | EndpointGone | Disable the subscription |
| 429, 502, 503, 504, 408, other 5xx, network error, blocked destination | RetryLater | Retry with back-off (honour Retry-After when present) |
| 3xx, other 4xx | PermanentFailure | Redirects are never followed; retrying will not help |
SafeEgressHandler sets AllowAutoRedirect = false and resolves the DNS name inside the connect callback, checks every returned address against the blocked ranges, and connects to the address it checked. Resolving and checking in one place closes the DNS-rebinding gap where a name resolves to a public address for the check and a private address for the connection.
AWS Console instructions
None for the local lab. When you move to production, Console → Secrets Manager → Store a new secret → Other type of secret is where the per-partner webhook secret lives, and Console → DynamoDB → Tables → Create table with a TTL attribute is where the replay cache moves. Menu wording changes over time (⚠ VERIFY).
AWS CLI
These commands are the production equivalents you would script; they are shown for orientation and are not required by the lab.
# Create a per-partner secret holding the current and (during rotation) the next secret.
aws secretsmanager create-secret --name hl-lab/webhooks/paylinq \
--secret-string '{"current":"whsec_REPLACE_ME","next":null}' \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m04-webhook-hmac --profile learn-sandbox
# DynamoDB table for seen ids with a TTL attribute named "expiresAt".
aws dynamodb create-table --table-name hl-lab-webhook-seen \
--attribute-definitions AttributeName=messageId,AttributeType=S \
--key-schema AttributeName=messageId,KeyType=HASH --billing-mode PAY_PER_REQUEST --profile learn-sandbox
aws dynamodb update-time-to-live --table-name hl-lab-webhook-seen \
--time-to-live-specification Enabled=true,AttributeName=expiresAt --profile learn-sandboxA secret costs about 0.40 USD per month (prorated); delete both resources the same day if you try them (⚠ VERIFY current prices).
Infrastructure as Code
The lab has no infra/ folder, because the services it models are covered by other labs. The production shape in outline:
// Replay cache: one item per accepted webhook id, deleted by DynamoDB TTL.
var seen = new Table(this, "WebhookSeen", new TableProps
{
PartitionKey = new Attribute { Name = "messageId", Type = AttributeType.STRING },
BillingMode = BillingMode.PAY_PER_REQUEST,
TimeToLiveAttribute = "expiresAt",
PointInTimeRecoverySpecification = new PointInTimeRecoverySpecification { PointInTimeRecoveryEnabled = true },
RemovalPolicy = RemovalPolicy.DESTROY, // learning only: use RETAIN in production
});
// Dead-letter queue for events the worker cannot process.
var dlq = new Queue(this, "WebhookDlq", new QueueProps { RetentionPeriod = Duration.Days(14), Encryption = QueueEncryption.SQS_MANAGED });
var events = new Queue(this, "WebhookEvents", new QueueProps
{
Encryption = QueueEncryption.SQS_MANAGED,
VisibilityTimeout = Duration.Seconds(60),
DeadLetterQueue = new DeadLetterQueue { MaxReceiveCount = 5, Queue = dlq },
});These snippets are illustrative and were not compiled or applied in this lab (⚠ VERIFY property names against your CDK and provider versions).
Complete code
The complete lab is in labs/m04-webhook-hmac/. The signing and verification core:
public string Sign(string id, long unixTimestamp, ReadOnlySpan<byte> rawBody)
{
if (string.IsNullOrEmpty(id) || id.Contains('.'))
throw new ArgumentException("The webhook id must be non-empty and must not contain '.'.", nameof(id));
var prefix = Encoding.UTF8.GetBytes($"{id}.{unixTimestamp.ToString(CultureInfo.InvariantCulture)}.");
var content = new byte[prefix.Length + rawBody.Length];
prefix.CopyTo(content, 0);
rawBody.CopyTo(content.AsSpan(prefix.Length));
return string.Join(' ', _keys.Select(k => "v1," + Convert.ToBase64String(HMACSHA256.HashData(k, content))));
}public WebhookVerificationResult Verify(string? id, string? timestamp, string? signatureHeader, byte[] rawBody)
{
if (string.IsNullOrEmpty(id) || string.IsNullOrEmpty(timestamp) || string.IsNullOrEmpty(signatureHeader) || id.Contains('.'))
return WebhookVerificationResult.Fail(WebhookVerificationFailure.MissingHeaders);
if (!long.TryParse(timestamp, NumberStyles.None, CultureInfo.InvariantCulture, out var sentAt))
return WebhookVerificationResult.Fail(WebhookVerificationFailure.InvalidTimestamp);
var now = _time.GetUtcNow().ToUnixTimeSeconds();
if (Math.Abs(now - sentAt) > (long)Tolerance.TotalSeconds)
return WebhookVerificationResult.Fail(WebhookVerificationFailure.TimestampOutOfTolerance);
var prefix = Encoding.UTF8.GetBytes($"{id}.{timestamp}.");
var signedContent = new byte[prefix.Length + rawBody.Length];
prefix.CopyTo(signedContent, 0);
rawBody.CopyTo(signedContent, prefix.Length);
var expected = _keys.Select(k => HMACSHA256.HashData(k, signedContent)).ToArray();
foreach (var entry in signatureHeader.Split(' ', StringSplitOptions.RemoveEmptyEntries))
{
var comma = entry.IndexOf(',');
if (comma <= 0 || entry[..comma] != "v1") continue;
var candidate = new byte[32];
if (!Convert.TryFromBase64String(entry[(comma + 1)..], candidate, out var written) || written != 32) continue;
foreach (var mac in expected)
{
if (CryptographicOperations.FixedTimeEquals(mac, candidate))
return WebhookVerificationResult.Success;
}
}
return WebhookVerificationResult.Fail(WebhookVerificationFailure.NoMatchingSignature);
}public static async Task<IResult> HandleAsync(HttpRequest request, WebhookVerifier verifier,
IReplayCache replay, IEventQueue queue, ILoggerFactory loggers, CancellationToken ct)
{
var log = loggers.CreateLogger("Harbourline.Webhooks.Receiver");
if (request.ContentLength is > MaxBodyBytes) return Results.StatusCode(StatusCodes.Status413PayloadTooLarge);
using var buffer = new MemoryStream();
await request.Body.CopyToAsync(buffer, ct);
if (buffer.Length > MaxBodyBytes) return Results.StatusCode(StatusCodes.Status413PayloadTooLarge);
var rawBody = buffer.ToArray();
var id = request.Headers[WebhookHeaders.Id].ToString();
var result = verifier.Verify(id, request.Headers[WebhookHeaders.Timestamp].ToString(),
request.Headers[WebhookHeaders.Signature].ToString(), rawBody);
if (!result.IsValid)
{
log.LogWarning("Rejected webhook {WebhookId}: {Failure}", Sanitise(id), result.Failure);
return result.Failure == WebhookVerificationFailure.MissingHeaders ? Results.BadRequest() : Results.Unauthorized();
}
if (await replay.IsKnownAsync(id, ct)) return Results.Ok();
await queue.EnqueueAsync(id, rawBody, ct); // enqueue BEFORE remembering
await replay.RememberAsync(id, DedupeRetention, ct);
return Results.Accepted();
}Code explained line by line
id.Contains('.')is rejected because the signed content isid.timestamp.body. An id containing a dot would let two different (id, timestamp) pairs produce the same signed prefix, a classic ambiguity in delimiter-joined formats.NumberStyles.Noneaccepts digits only, so-1,1.5and12abcall becomeInvalidTimestamp. There is no culture-dependent parsing.Math.Abs(now - sentAt) > 300rejects both old and future timestamps. A future timestamp is not "extra fresh": it is evidence of a wrong clock or an attack.- The signed content is built from raw bytes (
rawBody), not a string. If you decode to text and re-encode, a BOM, different escaping or reordered properties changes the bytes and the signature no longer matches. _keys.Select(...)computes one expected MAC per active secret, enabling rotation. The loop compares each candidate in the header with each expected MAC.CryptographicOperations.FixedTimeEqualstakes the same time whether the first byte or the last differs, so timing cannot be used to guess the signature byte by byte. A normal==on arrays or strings does not give that guarantee.Results.Unauthorized()carries no body. The reason goes to the log viaSanitise(id), which keeps 64 characters of letters, digits,_and-: an attacker-controlled header must not inject log lines.replay.IsKnownAsyncreturning true leads toResults.Ok()(200). Returning success to a duplicate tells the sender to stop retrying; returning an error would make it retry forever.EnqueueAsyncbeforeRememberAsyncis at-least-once delivery: a crash between the two leads to a duplicate on the next retry, never to a lost event. Consumers must still be idempotent (see Idempotency keys).public partial class Program;exposes the entry point soWebApplicationFactory<Program>can host the real receiver in the tests.
Security considerations
⚠ Learning shortcut — not for production
Secrets come from appsettings.Development.json, seen ids live in process memory and events go into an in-memory channel. In production, store one secret per partner in Secrets Manager and cache it with a short TTL, keep seen ids in DynamoDB with TTL so that every instance shares them, send events to SQS with a dead-letter queue, and terminate TLS at API Gateway or an ALB with AWS WAF in front. The Development secret in this repository is public: never reuse it.
- Constant-time comparison and generic errors: the caller never learns which check failed.
- Never log secrets, signature headers or full bodies. Bodies can contain account data. Log the id, the failure reason and the partner name.
- Replay window and dedupe retention are different numbers. The timestamp tolerance (5 minutes) bounds when a captured message can be replayed. The dedupe retention (7 days in the lab) must cover the sender's retry horizon, otherwise a late legitimate retry is processed twice.
- Distributed replay cache. With several receiver instances, an in-memory cache is wrong: instance B has never seen the id instance A accepted. Use a conditional write (
attribute_not_exists(messageId)) so exactly one instance wins. - Rotate secrets. Add the new secret to the verifier, tell the partner to sign with both (or switch), then remove the old one.
- SSRF. Customer-supplied endpoints must be HTTPS, must resolve to public addresses and must not redirect. Block the instance metadata address at the network layer too (IMDSv2 and a hop limit of 1), because code-level guards are one layer, not the only one.
- This lab does not make a system compliant with anything. A webhook that carries cardholder data needs a PCI DSS scoping review, and personal data needs a UK GDPR review.
Banking use case
A card processor sends payment.settled, payment.reversed and chargeback.opened events to the bank. The receiver verifies, de-duplicates and enqueues; a worker updates the ledger and publishes an internal event. When the processor rotates its signing secret, Operations stores the new value as next in the partner's Secrets Manager secret, the receiver accepts both for 48 hours, and the old value is deleted afterwards. A chargeback event processed twice would debit the customer twice, which is why de-duplication and idempotent processing are both required.
Common mistakes
- Parsing JSON before verifying. Verify on the raw bytes; parse afterwards.
- Comparing signatures with
==. Use a fixed-time comparison. - Using the
whsec_...text as the key. The key is the base64-decoded bytes after the prefix. - Doing the business work in the request. A slow handler causes a sender timeout and a retry storm. Acknowledge fast, process from a queue.
- Remembering the id before the work is queued. A crash then loses the event.
- Signing with an old timestamp on a retry. A retry after thirty minutes with the original signature fails the tolerance check. The sender must re-sign.
- Following redirects from a customer-supplied URL. A redirect to a private address defeats the address check.
- Keeping the replay cache per instance.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Always 401 | Using whsec_... as the key, or a different secret | Base64-decode the part after whsec_ |
401 only from PowerShell | The body was re-encoded (BOM, line endings) or differs from what was signed | Sign exactly the bytes you send; avoid Out-File for the body |
401 after a pause | The timestamp is older than five minutes | Sign each attempt with a fresh timestamp |
400 | A webhook-* header is missing, or the id contains . | Send all three headers and use a dot-free id such as msg_<guid> |
413 | Body over 256 KiB | Raise MaxBodyBytes deliberately, or ask the sender to send a reference instead of the payload |
| Port 5090 already in use | Another process | Set a different URL: dotnet run --project ... --urls http://localhost:5091 |
Sender reports RetryLater with no status code | Network error, timeout or SafeEgressHandler blocked the address | Check the destination resolves to a public address; read the inner exception |
| Duplicate processed in production but not in tests | Per-instance replay cache | Use DynamoDB with a conditional write |
Cost and cleanup
Cost: free. The lab runs locally and creates no AWS resources. If you tried the optional CLI commands, delete what you created:
aws secretsmanager delete-secret --secret-id hl-lab/webhooks/paylinq --force-delete-without-recovery --profile learn-sandbox
aws dynamodb delete-table --table-name hl-lab-webhook-seen --profile learn-sandbox
# Remove local build output
find labs/m04-webhook-hmac -type d \( -name bin -o -name obj -o -name TestResults \) -prune -exec rm -rf {} +Stop the receiver with Ctrl+C.
Exercises
- 🟢 Guided: run the tests, start the receiver, send the signed message from PowerShell, then run the five "break it" experiments and record the status codes.
- 🟡 Challenge: add a second endpoint for a different partner with its own secret (
Webhooks:Acme:Secrets). Write tests proving a message signed with Paylinq's secret is rejected at Acme's endpoint. Then add a test for a rotation window where the receiver holds both secrets and receives one message signed with each. - 🔴 Live problem: a ticket says "the receiver processed settlement
msg_77twice yesterday, and only on one of our four instances". Using the code and what you know about the replay cache, explain the cause, design the fix (the table, the key, the conditional write and the TTL) and say what happens to a duplicate when the write fails. Add a test double forIReplayCachethat fails the firstRememberAsyncand show what the receiver should do.
Assignment
Implement DynamoDbReplayCache : IReplayCache using AWSSDK.DynamoDBv2 v4. IsKnownAsync can be a GetItem; RememberAsync must be a conditional PutItem with attribute_not_exists(messageId) and an expiresAt epoch-seconds attribute. Test it against a fake IAmazonDynamoDB that you write (no AWS account). Add a short design note: why the check-then-remember pair is racy across instances, and how a single conditional put (reserve first, then enqueue, release on failure) would fix it. Keep the work in your portfolio repository.
Solutions
| Experiment | Status | Why |
|---|---|---|
| One character changed after signing | 401 | NoMatchingSignature |
| Timestamp six minutes old | 401 | TimestampOutOfTolerance |
whsec_ text used as key | 401 | The MAC is computed with the wrong key bytes |
webhook-id removed | 400 | MissingHeaders |
| Same message twice | 202 then 200 | The second time the id is known, and the receiver acknowledges without enqueuing |
Register a second WebhookVerifier with a keyed or named registration (or construct verifiers from a dictionary of partner name to secrets) and map POST /webhooks/acme-events. The endpoint handler selects its verifier by route. In the test, sign with the Paylinq secret and post to the Acme route: expect 401 with NoMatchingSignature in the log. For rotation, create new WebhookVerifier([old, next], time) and assert that messages signed with each verify, as Key_rotation_overlap_accepts_old_new_and_double_signed_messages already shows at the unit level.
Cause: each instance has its own InMemoryReplayCache. A retry hit a different instance that had not seen msg_77, so it verified, enqueued and remembered the id locally. Fix: one DynamoDB table keyed by messageId, a conditional PutItem with attribute_not_exists(messageId) and expiresAt set to the end of the retention window, with TTL enabled on that attribute. If the conditional write fails because the item exists, treat the message as a duplicate and return 200. If the write fails for another reason (throttling, outage), return 503 so the sender retries; do not return success, because the event has not been safely recorded. For the test, a replay-cache double that throws on its first RememberAsync should cause a 5xx and, on the retry, a 202. Note that DynamoDB TTL deletion is asynchronous, so the receiver must also compare expiresAt with the current time rather than rely on the item being gone (⚠ VERIFY the TTL delay in the DynamoDB documentation).
Knowledge check
Answer: The signature covers exactly the bytes sent. Deserialising and re-serialising can change whitespace, property order or escaping, so the recomputed MAC would not match. Verify first, parse afterwards.
Answer: It returns 401 because the timestamp is outside the five-minute tolerance. This is what stops an attacker replaying a captured message. A legitimate sender retrying later must sign with a fresh timestamp.
Answer: The first is accepted (202) and enqueued; the second is acknowledged (200) and ignored. The timestamp check cannot catch this, because both are fresh, which is why the replay cache exists.
Answer: The id identifies the event, so the receiver can de-duplicate. The timestamp and signature identify the attempt, so each attempt falls within the receiver's tolerance window.
Answer: If you resolve and check first and then let the handler resolve again, an attacker's DNS can return a public address the first time and a private one the second (DNS rebinding). Checking and connecting to the same resolved address closes the gap.
Interview questions
- "How do you secure a webhook endpoint?" Verify an HMAC over the raw body with a constant-time comparison; bind the signature to an id and a timestamp; enforce a tolerance; de-duplicate by id with a shared store; limit body size; return generic errors; acknowledge fast and process asynchronously.
- "The partner is rotating its secret next week. What is your plan?" Support multiple active secrets in the verifier, ask the partner to sign with both during overlap, watch the logs for which secret matched, then remove the old one.
- "What is SSRF and how does it apply to sending webhooks?" The server fetches a URL chosen by a user, so an attacker points it at an internal address such as the metadata service. Defend with HTTPS-only endpoints, a resolved-address blocklist, no redirects, short timeouts and network-level controls.
- "Why return 200 for a duplicate?" To stop the sender retrying. The event was already handled.
Certification practice
A partner sends HMAC-signed webhooks to an API running on several containers. After a retry storm, some events are processed twice even though every signature is valid and every timestamp is fresh. What is the MOST likely cause and fix?
A. The HMAC algorithm is weak; switch to MD5. B. Each container keeps its own record of seen message ids; store them in a shared DynamoDB table with a conditional write and TTL. C. The timestamp tolerance is too long; reduce it to one second. D. The containers need larger memory.
Answer: B. Duplicates with valid signatures and fresh timestamps point to missing or per-instance de-duplication. A is wrong because MD5 is weaker, not stronger. C breaks legitimate clock drift and does not address duplicates that are both fresh. D has no relation to duplicate processing.
A feature lets customers register a URL that the bank's service will call with signed event data. A penetration tester registers http://169.254.169.254/latest/meta-data/ and receives credentials in the response log. Which combination BEST mitigates this?
A. Require HTTPS, resolve the host and block private and link-local addresses at connect time, disable redirects, and require IMDSv2 with a hop limit of 1. B. Add the customer's URL to the WAF allow list. C. Rotate the webhook signing secret monthly. D. Increase the webhook timeout.
Answer: A. It blocks the destination at several layers. B and D do not address the destination. C protects the signature, not where the request goes.
Completion checklist
- I ran the tests and saw
total: 44, succeeded: 44 - I sent a signed message and saw
202, then200for the repeat - I broke the signature, the timestamp and the key on purpose and saw
401 - I can explain the difference between the timestamp tolerance and the dedupe retention
- I deleted
bin/andobj/and removed any optional AWS resources
Key takeaways
- A webhook is authenticated by a shared secret and a signature over id, timestamp and raw body.
- Signature, timestamp tolerance and a shared replay cache are three separate defences; each stops a different attack.
- Acknowledge quickly and process from a queue; enqueue before remembering the id.
- Senders keep the id, refresh the timestamp and signature, never follow redirects and never call private addresses.
- In-memory stand-ins in the lab name the AWS services that replace them: Secrets Manager, DynamoDB with TTL, SQS with a dead-letter queue.
Further reading
- Webhooks: send and receive safely with HMAC and replay protection, the lesson this lab belongs to
- Idempotency keys, for idempotent processing after the queue
- Module 4 overview
- Standard Webhooks specification ⚠ VERIFY the URL before relying on it
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Webhooks: send and receive safely with HMAC and replay protection
Receive partner webhooks on AWS with raw-body HMAC-SHA256 verification, timestamp tolerance, replay protection and fast acknowledgement, and send your own signed webhooks with retries, dead-letter queues and SSRF defences.
Calling partner APIs safely: egress, secrets, timeouts, retries and circuit breakers
Call external partners such as a sanctions-screening or credit-reference API from AWS with controlled egress, cached secrets, outbound mTLS and a tested Microsoft.Extensions.Http.Resilience pipeline that respects partner rate limits.