Lab 7.6: near-real-time fraud signals with Kinesis, a .NET Lambda and SNS
Run the velocity and high-amount rules locally first, then deploy a one-shard Kinesis stream, a .NET 10 Lambda and an SNS topic, send a burst of transactions, receive exactly one velocity signal, and destroy everything inside the hour.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (Kinesis 4.1.1, DynamoDB 4.1.0.2, SNS 4.1.0.2) · AWS CDK v2 (C#) · Lambda dotnet10 on ARM64 · Region us-east-1 · stack HlM07-Streaming. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lab you can:
- Explain the two rules (HIGH_AMOUNT and VELOCITY), why they use a fixed ten-minute window and why each window raises at most one signal.
- Run the rule engine locally with
hl-pipeline produce --dry-runand predict its output. - Build and deploy a Kinesis → Lambda → SNS stack, subscribe to the alerts and trigger a signal from a burst of eight payments.
- Explain
PutRecordspartial failure, the partial-batch response, a "seen" marker in DynamoDB and why Kinesis delivery needs de-duplication. - Show what happens to a poison record and to a repeated burst in the same window.
- Delete every resource, confirm the stream is gone and state what the lab cost.
Prerequisites
- Streaming ingestion (shards,
PutRecords, retries, Firehose) and Lambda transforms. - Lab 7.3 Part A tools: AWS CLI v2 with
learn-sandbox, .NET 10 SDK, Node.js and the CDK CLI, an account bootstrapped for CDK. Docker for the publish step. - Read Fraud detection integration later for how signals feed a case system.
- A mailbox you control for the SNS e-mail subscription (or skip it and read the Lambda log).
Cost label
🟡 Credits. A provisioned Kinesis shard costs about USD 0.015 per shard-hour plus PUT payload units, so a one-hour session is a couple of cents. Lambda, DynamoDB on-demand, SNS and SQS are pennies at this volume (free tier may apply). Because HlM07-Streaming depends on HlM07-DataLake, deploying it also creates the data lake stack if it is not already deployed (the key, buckets and the secret) — about USD 1.40 per month for the key and secret, prorated. Total for a one-hour session: well under USD 0.10. A stream left running costs about USD 11 a month, so the teardown is part of the lab.
Free alternative: do Part A only. The dry run executes the same rule engine in memory and needs no AWS account.
Scenario
Fraud analysts want to know within seconds when an account does something suspicious. Two simple rules catch a lot: a single debit of GBP 5,000 or more, and more than five card, transfer or ATM debits on one account inside ten minutes. Amazon Fraud Detector is closed to new customers, so the rules run in code on a stream. The output is a signal: not a decision to block a payment, but a message that a person or a case system reviews.
Architecture
Security-boundary view. The producer needs only kinesis:PutRecords on this one stream. The Lambda role is built by CDK grants: read the stream, read/write one table, publish to one topic, use one key. Every store (stream, table, topic, queue, log group) is encrypted with the data key, and the topic and queue enforce TLS.
The files
| File | Role in the lab |
|---|---|
src/Harbourline.DataPipeline.Core/Fraud/FraudRuleEngine.cs | The two rules, the window maths, the deterministic signal id |
src/Harbourline.DataPipeline.Core/Fraud/DynamoDbVelocityStore.cs | Counters in DynamoDB |
src/Harbourline.DataPipeline.Functions/FraudSignalFunction.cs | The Lambda handler |
src/Harbourline.DataPipeline.Cli/KinesisProducer.cs and Program.cs | hl-pipeline produce |
infra/cdk/.../StreamingStack.cs | The stack HlM07-Streaming |
tests/.../FraudTests.cs | The six xUnit tests for all of the above |
Part A: the rules, locally and free
Read the rules
- HIGH_AMOUNT: a debit (negative amount) whose absolute value is at least
5000.00, in either currency. - VELOCITY: more than 5 debits of type
card_payment,transfer_outoratm_withdrawalon one account in one fixed 10-minute window (10:00 to 10:10, 10:10 to 10:20 and so on). The signal fires exactly when the counter reaches 6, so a window with 20 payments still raises one signal. - Signal ids are
SHA-256("<transactionId>|<RULE>")truncated to 32 hex characters, the same in C# and Python, so a retry or a re-run produces the same id and consumers can de-duplicate.
A fixed window is simple and cheap (one counter per account and window) but has a known weakness: six payments split 3 and 3 across a window boundary are never flagged. A sliding window fixes it at the cost of more state. This is a deliberate trade-off to discuss, not an oversight.
Run the dry run
cd labs/m07-transaction-pipeline
MSYS_NO_PATHCONV=1 docker run --rm --memory 3g -v "$PWD:/src" -v cea-nuget-m07:/root/.nuget/packages -w /src \
mcr.microsoft.com/dotnet/sdk:10.0 sh -c \
"dotnet run --project src/Harbourline.DataPipeline.Cli -c Release -- produce data/samples/transactions.csv --dry-run --burst ACC-1000000001 --limit 300; \
find . -type d \( -name bin -o -name obj \) -prune -exec rm -rf {} +"The author's run printed:
VELOCITY ACC-1000000001 -9.99 GBP more than 5 debits in the 10-minute window starting 09:00Z
Dry run: 308 transactions evaluated locally, nothing sent.308 = the first 300 clean rows plus the 8-payment burst. The 300 historic rows trigger nothing (no single debit of 5,000, no six debits in ten minutes), and the burst raises one VELOCITY signal, at the sixth payment. The window start (09:00Z here) is the start of the current ten-minute window when you run it, so your time will differ.
Why the burst is aligned to a window
The burst helper creates eight payments of GBP 9.99, 30 seconds apart. It starts at the beginning of the current ten-minute window, not "now". An earlier version started at the current time, so a run near a window edge would split the eight payments 5 plus 3 across two windows and raise no signal: the lab would pass or fail depending on the clock. The fix is in Program.cs: FraudRuleEngine.WindowStart(DateTime.UtcNow, TimeSpan.FromMinutes(10)). The timestamps are slightly in the past, which is fine because the engine uses the record's own CreatedAt, not the time of arrival.
Run the unit tests
The six tests in FraudTests.cs cover: one velocity signal on the sixth debit, high-amount plus duplicate delivery, the window and signal-id twins of the Python code, the Lambda's partial-batch behaviour, the DynamoDB conditional put and atomic counter, and the producer retrying only the failed records. They were part of the 27 .NET tests, which all passed in the author's run:
Test run summary: Passed!
total: 27
failed: 0
succeeded: 27Part B: deploy and run
Publish the Lambda package (Linux ARM64, built in Docker)
MSYS_NO_PATHCONV=1 docker run --rm -v "$PWD:/src" -v cea-nuget-m07:/root/.nuget/packages -w /src mcr.microsoft.com/dotnet/sdk:10.0 \
sh -c "dotnet publish src/Harbourline.DataPipeline.Functions -c Release -r linux-arm64 --self-contained false -o publish/functions"This creates publish/functions/. Its presence is what tells the CDK app to define the ingest API and HlM07-Streaming. The folder is generated; do not commit it.
Deploy
cd infra/cdk
aws sso login --profile learn-sandbox
cdk deploy HlM07-Streaming --profile learn-sandbox -c owner=<your-name>CDK deploys HlM07-DataLake first if needed, because the stream uses its key. Read the change summary: it should list a Kinesis stream, a DynamoDB table, an SNS topic, an SQS queue, a Lambda function and role, an event source mapping and encrypted log groups. The Lambda runtime is dotnet10; the stack builds it with new Runtime("dotnet10", RuntimeFamily.DOTNET_CORE) because CDK may not yet have a named constant for it ⚠ VERIFY that dotnet10 is accepted in your Region. Note the outputs StreamName and SignalsTopicArn.
Subscribe to the alerts
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1 MSYS_NO_PATHCONV=1
TOPIC=$(aws cloudformation describe-stacks --stack-name HlM07-Streaming --query "Stacks[0].Outputs[?OutputKey=='SignalsTopicArn'].OutputValue" --output text)
aws sns subscribe --topic-arn "$TOPIC" --protocol email --notification-endpoint you@example.comReplace the address with your own and confirm the e-mail AWS sends. The mapping starts reading at LATEST, so wait about a minute after deploy before sending: records sent before the event source mapping is active are not read.
Send the transactions and the burst
cd ../..
export AWS_PROFILE=learn-sandbox
dotnet run --project src/Harbourline.DataPipeline.Cli -- produce data/samples/transactions.csv \
--stream hl-m07-transactions --limit 200 --burst ACC-1000000001(If you have no .NET SDK locally, run the same command in the Docker image, adding -v ~/.aws:/root/.aws -e AWS_PROFILE=learn-sandbox.) Expected: Sent 208 records to hl-m07-transactions. The producer sends in PutRecords batches of up to 500, uses the account id as the partition key (so one account's records stay in order on one shard) and re-sends only the entries that failed, with exponential back-off.
See the signal
Within a few seconds you should receive an e-mail with the subject Harbourline fraud signal VELOCITY and a JSON body like:
{"signalId":"...","rule":"VELOCITY","transactionId":"...","accountId":"ACC-1000000001",
"amount":-9.99,"currency":"GBP","createdAt":"...","detail":"more than 5 debits in the 10-minute window starting 09:00Z"}Check the Lambda's own summary line:
aws logs tail /aws/lambda/hl-m07-fraud-rules --since 10mOne log line per batch: {"records":208,"signals":1,"failures":0} (the batch may be split in two, so you may see two lines whose records add up to 208). The velocity table now holds markers and counters:
aws dynamodb scan --table-name hl-m07-velocity --select COUNTExpect on the order of the number of processed transactions plus a counter item per account and window, all with a TTL of one day.
Experiment 1: send the burst again
Run the same produce command again within the same ten-minute window. No second e-mail arrives. The burst uses new transaction ids, so the "seen" check passes, but the counter for the window is already 8, and the rule fires only when it equals 6. The new payments take it to 16 without crossing 6 again. This is "one signal per window" in action.
Experiment 2: a poison record
aws kinesis put-record --stream-name hl-m07-transactions --partition-key poison --cli-binary-format raw-in-base64-out --data "not json"
aws logs tail /aws/lambda/hl-m07-fraud-rules --since 2m | grep -i skippingThe Lambda logs Skipping unparseable record <sequence number> and carries on: it does not fail the batch. A record that can never parse would otherwise block the shard forever (Kinesis delivers in order, so one stuck record stops everything behind it until it expires). The handler logs ids only, never the payload. A production design also sends such records to a dead-letter destination for inspection.
Part C: read the code
if (!await store.TryMarkSeenAsync(tx.TransactionId, cancellationToken))
{
return []; // already evaluated: do not count it twice or alert twice
}
var signals = new List<FraudSignal>();
if (tx.Amount < 0 && -tx.Amount >= _options.HighAmountThreshold)
{
signals.Add(Create(HighAmount, tx, ...));
}
if (tx.Amount < 0 && _options.VelocityTypes.Contains(tx.Type))
{
var window = WindowStart(tx.CreatedAt, _options.VelocityWindow);
var count = await store.IncrementAsync(tx.AccountId, window, cancellationToken);
if (count == _options.VelocityMaxDebits + 1)
{
signals.Add(Create(Velocity, tx, ...)); // exactly at the crossing point
}
}
return signals;TryMarkSeenAsync— Kinesis delivers at least once. A Lambda retry, a stream replay or a duplicate put can present the same transaction twice. DynamoDB doesPutItemwithattribute_not_exists(pk); the second attempt throwsConditionalCheckFailedException, which the store turns intofalse. The engine returns no signals for a duplicate, so counts and alerts are not doubled.- High amount — a debit is negative; the rule uses its absolute value and
>=. A credit never triggers it. - Velocity types — only card payments, outgoing transfers and ATM withdrawals count. A direct debit or a fee does not.
WindowStart— floors the record's own timestamp to the window start by integer arithmetic on ticks (ticks - ticks % window.Ticks). Using the record time, not the arrival time, makes a replay produce the same windows.IncrementAsync— an atomicADD debit_count :oneon the itemVEL#<account>#<window start>, returning the new value. Atomicity matters because Lambda may process several batches at once; two concurrent increments both get distinct values and exactly one of them sees6.count == Max + 1— equality, not>=, so each window alerts once.
catch (JsonException ex)
{
context.Logger.LogError($"Skipping unparseable record {record.Kinesis.SequenceNumber}: {ex.Message}");
}
catch (Exception ex) when (ex is AmazonSimpleNotificationServiceException or AmazonDynamoDBException)
{
if (tx is not null) { await _engine.ReleaseAsync(tx.TransactionId); }
response.BatchItemFailures.Add(new StreamsEventResponse.BatchItemFailure { ItemIdentifier = record.Kinesis.SequenceNumber });
break;
}- Poison versus transient. A
JsonExceptionwill never succeed, so the handler skips it. An SNS or DynamoDB failure is transient, so the handler asks for a retry. ReleaseAsync— deletes the "seen" marker, otherwise the retry would be dropped as a duplicate and the signal would be lost. The cost: the retry may count the transaction twice in its velocity window. Over-alerting is safer than losing an alert.BatchItemFailureswith the failing sequence number, thenbreak— withReportBatchItemFailuresenabled, Lambda retries from that record. Records before it are not re-sent, and stopping at the first failure preserves per-account order.- In the stack:
RetryAttempts = 5,MaxRecordAge = 1 hour,OnFailure = SQS— after five attempts or an hour, the metadata of the failed batch goes tohl-m07-fraud-rules-failures(not the payload, which stays in Kinesis for its retention).
Security considerations
⚠ Learning shortcut — not for production
The stream has one shard, no consumer isolation and a 24-hour retention; the alert goes to an e-mail address; the rules are two constants; velocity is a fixed window; and a failed record is described, not preserved. A bank sizes shards (or uses on-demand mode) from peak traffic, sends signals to a case-management system with an audit trail, tunes thresholds with the fraud team, uses sliding windows or a stateful stream processor, and keeps failed payloads in an encrypted dead-letter store.
- Signals are not decisions. A rule match creates a review task. Blocking a customer's payment automatically is a conduct, legal and customer-harm question for fraud, compliance and legal teams, not a Lambda.
- Do not log payloads. The handler logs counts and sequence numbers. A transaction contains account ids, which are personal data in context.
- Encrypt every hop: stream, table, topic, queue and logs use the customer managed key; TLS is enforced on the topic and queue.
- Least privilege: the producer role needs only
kinesis:PutRecordson this stream. Do not run the producer with administrator credentials outside the sandbox. - Alert fatigue is a risk too. A rule that fires too often is ignored. Thresholds need measuring against real false-positive rates.
Compliance review required
Transaction monitoring for fraud and financial crime has legal obligations (for example suspicious-activity reporting) that this learning lab does not meet. Rules, thresholds, retention and who may see alerts are decisions for your fraud, compliance and data-protection teams.
Common mistakes
- Sending immediately after deploy → the mapping starts at
LATEST, so early records are not read. - Expecting a second alert for the same window → by design there is one.
- Using the transaction id as the partition key → one account's records scatter over shards and the velocity count loses order guarantees. Use the account id.
- Treating
PutRecordsas all-or-nothing → checkFailedRecordCountand retry only the failed entries (the producer does). - Forgetting to delete the stream → about USD 11 a month for nothing.
- Retrying a poison record forever → skip and log it, then dead-letter it.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Sent 0 records or an access error | Wrong profile or region, or no permission | export AWS_PROFILE=learn-sandbox; check aws sts get-caller-identity |
ProvisionedThroughputExceededException in the producer | One shard accepts 1 MB/s or 1,000 records/s of writes | The producer retries with back-off; lower --limit |
| No e-mail arrives | Subscription not confirmed, or sent before the mapping was active, or mail in spam | Confirm the subscription; wait a minute; resend; read the Lambda log |
Lambda log: Environment variable VELOCITY_TABLE is not set | Stack changed outside CDK | Redeploy |
Lambda errors AccessDeniedException (KMS) | Function role lost key access | Redeploy; the stack calls GrantEncryptDecrypt for the function |
| Deploy fails on the runtime | dotnet10 not available in your Region or CDK version | Update CDK and check the Lambda runtimes page for your Region ⚠ VERIFY |
| Signals duplicated | Retry after a transient failure (by design) | Consumers de-duplicate on signalId |
cdk deploy says the key is in use | Another stack still references the lake's key | Destroy streaming first, then the lake |
Cost and cleanup
Destroy the stream stack the same session:
cd infra/cdk
cdk destroy HlM07-Streaming --profile learn-sandbox
aws sns list-subscriptions-by-topic --topic-arn "$TOPIC" 2>/dev/null # should fail: the topic is gone
aws kinesis list-streams --query "StreamNames[?contains(@, 'hl-m07')]" # should be []
aws dynamodb list-tables --query "TableNames[?contains(@, 'hl-m07')]" # should be []If you continue to the project, keep HlM07-DataLake; otherwise cdk destroy HlM07-DataLake too, and delete publish/ and the unsubscribed e-mail confirmation. Then run the tag check:
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--query "ResourceTagMappingList[?contains(ResourceARN, 'hl-m07')].ResourceARN"A stream left running costs about USD 11 a month (0.015 × 24 × 30) plus data. If the destroy is stuck, delete the stream by hand: aws kinesis delete-stream --stream-name hl-m07-transactions --enforce-consumer-deletion.
Exercises
- 🟢 Guided: Run the dry run with and without
--burst, then changeVelocityMaxDebitsto 3 inFraudRuleOptions, re-run and explain the new signal's transaction. Do not edit the deployed Lambda. - 🟡 Challenge: Implement a sliding window locally: keep the timestamps of the last debits per account in the in-memory store and raise the signal when more than five fall within any 10-minute span. Write a test where six payments straddle a fixed-window boundary and show the fixed window misses it and yours catches it.
- 🔴 Live problem: After a deploy, fraud analysts report two alerts for one burst, and one missing alert for a large transfer. Using what you know about at-least-once delivery,
ReleaseAsync,MaxRecordAgeand partial-batch responses, list the likely causes for each symptom and how you would investigate them with the Lambda log, the DynamoDB table and the SQS failure queue.
Assignment
Write a docs/streaming-fraud-design.md for your portfolio: the two rules and their thresholds, the window trade-off, the at-least-once handling, the failure paths (poison record, transient failure, record too old) and what you would change before production. Include the dry-run output and the cost of your session.
Solutions
Without --burst, the 300 historic rows raise nothing. With the burst, one VELOCITY signal for the sixth payment. Lowering the threshold to 3 raises the signal at the fourth debit in a window, and the historic rows may now trigger additional signals if any account has four card, transfer or ATM debits in one ten-minute span (rare in this data). The lesson is that thresholds are data: change them in options, not in code paths.
Keep, per account, a sorted queue of debit timestamps. For each new debit, drop timestamps older than t - 10 minutes, add the new one, and raise the signal when the queue length first exceeds five (and remember the account until it drops below six again, to avoid repeated alerts). Test: payments at 09:08:00, 09:08:30, 09:09:00, 09:09:30 and 09:10:00, 09:10:30 straddle the 09:10 boundary. A fixed window sees four and then two; the sliding window sees six within 2.5 minutes and fires. The price is state per account and more complex de-duplication across retries.
Duplicate alert: a transient SNS or DynamoDB failure after the publish succeeded but before the batch completed makes the batch retry; ReleaseAsync removed the marker, so the retry evaluates the record again and re-publishes. Consumers should de-duplicate on signalId. Missing alert: (a) the record was older than MaxRecordAge (one hour) when it was finally processed and was dropped to the failure queue; (b) a JsonException made the handler skip it; (c) the transfer was not a velocity type and below 5,000 for HIGH_AMOUNT; (d) it arrived before the mapping started (LATEST). Investigate with the Lambda log (skips and failure lines), the SQS failure queue (batch metadata), the DynamoDB TX#<id> marker for the transaction (present means evaluated) and the Kinesis iterator age metric.
Knowledge check
Answer: Equality fires once when the counter crosses the threshold, so a window with many more debits still produces one signal instead of one per extra transaction.
Answer: Kinesis preserves order within a shard, and records with the same partition key go to the same shard, so one account's transactions stay in order and the per-account counter sees them in sequence.
Answer: The handler returns the sequence number of the first failed record. Lambda retries from that record instead of the whole batch, so earlier records are not processed twice.
Answer: Otherwise the retry would be treated as a duplicate and its signal would never be published. The price is possible double counting in the velocity window, which is safer than a missed alert.
Answer: About USD 11 for one provisioned shard (0.015 per shard-hour × 24 × 30), plus PUT payload units, even with no data.
Interview questions
- "How do you detect velocity fraud in a stream?" — Key by account, keep a counter per account and time window in a fast store with atomic increments and a TTL, and emit a signal when the threshold is crossed. Mention fixed versus sliding windows and de-duplication.
- "Kinesis delivers at least once. How do you cope?" — Make consumers idempotent: a conditional write marks processed ids, signal ids are deterministic and downstream systems de-duplicate.
- "What is a partial batch response?" — A Lambda return value listing the failed item so only it and later records are retried.
- "Why a signal and not a block?" — Automated blocking has customer and legal consequences, so rules create reviewable signals for analysts or a case system.
Certification practice
A stream processes card transactions. A consumer must see each account's transactions in the order they occurred. What should the producer use as the partition key?
A. A random UUID per record. B. The account id. C. The current timestamp. D. A constant string.
Answer: B. Records with the same key go to the same shard, in order. A scatters an account's records, C does likewise, D puts everything on one shard and limits throughput.
A Lambda function processes a Kinesis batch of 100 records. Record 40 fails every time because of malformed data, and the shard stops progressing. Which change keeps the stream moving without losing the other records?
A. Increase the Lambda timeout. B. Handle the bad record (skip and log or send to a dead-letter destination) and report failures only for transient errors using partial batch responses, with a bounded retry and maximum record age. C. Delete the stream. D. Increase the shard count.
Answer: B. A poison record must not block the shard. A, D do not fix a deterministic failure; C destroys data.
A team keeps a provisioned one-shard Kinesis stream running for a rarely used test. What is the cheapest correct action when the test ends?
A. Do nothing. B. Delete the stream and other lab resources, verifying with a tag query. C. Reduce the retention to 24 hours. D. Add a second shard.
Answer: B. A provisioned shard bills per hour whether or not it carries data. Retention changes do not remove the shard-hour charge.
Completion checklist
- I ran the dry run and can explain why the output has one VELOCITY line
- I deployed, subscribed, sent the burst and received one signal
- I tried the repeat burst and the poison record and explained both results
- I can explain at-least-once delivery, the seen marker and
ReleaseAsync - I destroyed the stack, confirmed no stream or table remains and ran the tag query
Key takeaways
- Two simple rules on a stream give near-real-time signals without an ML service.
- Fixed ten-minute windows with an atomic counter and equality at the threshold give exactly one signal per window.
- At-least-once delivery needs idempotent consumers: conditional writes and deterministic signal ids.
- Separate poison records (skip) from transient failures (retry from the failing record).
- A running stream bills by the hour; deploy, test and destroy within the hour.
Further reading
infra/cdk/Harbourline.DataPipeline.Infra/StreamingStack.csandtests/.../FraudTests.csin the lab- Fraud detection integration (Module 14) for case management
- Previous: Lab 7.5 · Next: Module project
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Lab 7.5: an analyst who sees only GBP rows and no free-text columns
Register the curated bucket with Lake Formation, switch the transactions table from IAM-only access to Lake Formation permissions, create a data cells filter, and prove from the analyst's own session that the row and column restrictions hold.
Module project: banking transaction processing pipeline
Deploy, run, break and audit a complete batch pipeline: raw files in an encrypted S3 zone, an event-driven Step Functions and Glue workflow that validates, tokenises and reconciles, quarantine with reason codes, partitioned Parquet, Athena reports and one audit record per run, with 27 .NET and 18 Python tests that need no AWS account.