Skip to Content
Cloud Engineering Academy
7. Data Engineering & ETL Pipelines

Lab 7.6: near-real-time fraud signals with Kinesis, a .NET Lambda and SNS

Run the velocity and high-amount rules locally first, then deploy a one-shard Kinesis stream, a .NET 10 Lambda and an SNS topic, send a burst of transactions, receive exactly one velocity signal, and destroy everything inside the hour.

advanced⏱ 1 h 30 min🟡 Low costDEA-C01SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (Kinesis 4.1.1, DynamoDB 4.1.0.2, SNS 4.1.0.2) · AWS CDK v2 (C#) · Lambda dotnet10 on ARM64 · Region us-east-1 · stack HlM07-Streaming. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lab you can:

  • Explain the two rules (HIGH_AMOUNT and VELOCITY), why they use a fixed ten-minute window and why each window raises at most one signal.
  • Run the rule engine locally with hl-pipeline produce --dry-run and predict its output.
  • Build and deploy a Kinesis → Lambda → SNS stack, subscribe to the alerts and trigger a signal from a burst of eight payments.
  • Explain PutRecords partial failure, the partial-batch response, a "seen" marker in DynamoDB and why Kinesis delivery needs de-duplication.
  • Show what happens to a poison record and to a repeated burst in the same window.
  • Delete every resource, confirm the stream is gone and state what the lab cost.

Prerequisites

  • Streaming ingestion (shards, PutRecords, retries, Firehose) and Lambda transforms.
  • Lab 7.3 Part A tools: AWS CLI v2 with learn-sandbox, .NET 10 SDK, Node.js and the CDK CLI, an account bootstrapped for CDK. Docker for the publish step.
  • Read Fraud detection integration later for how signals feed a case system.
  • A mailbox you control for the SNS e-mail subscription (or skip it and read the Lambda log).

Cost label

🟡 Credits. A provisioned Kinesis shard costs about USD 0.015 per shard-hour plus PUT payload units, so a one-hour session is a couple of cents. Lambda, DynamoDB on-demand, SNS and SQS are pennies at this volume (free tier may apply). Because HlM07-Streaming depends on HlM07-DataLake, deploying it also creates the data lake stack if it is not already deployed (the key, buckets and the secret) — about USD 1.40 per month for the key and secret, prorated. Total for a one-hour session: well under USD 0.10. A stream left running costs about USD 11 a month, so the teardown is part of the lab.

Free alternative: do Part A only. The dry run executes the same rule engine in memory and needs no AWS account.

Scenario

Fraud analysts want to know within seconds when an account does something suspicious. Two simple rules catch a lot: a single debit of GBP 5,000 or more, and more than five card, transfer or ATM debits on one account inside ten minutes. Amazon Fraud Detector is closed to new customers, so the rules run in code on a stream. The output is a signal: not a decision to block a payment, but a message that a person or a case system reviews.

Architecture

Security-boundary view. The producer needs only kinesis:PutRecords on this one stream. The Lambda role is built by CDK grants: read the stream, read/write one table, publish to one topic, use one key. Every store (stream, table, topic, queue, log group) is encrypted with the data key, and the topic and queue enforce TLS.

The files

FileRole in the lab
src/Harbourline.DataPipeline.Core/Fraud/FraudRuleEngine.csThe two rules, the window maths, the deterministic signal id
src/Harbourline.DataPipeline.Core/Fraud/DynamoDbVelocityStore.csCounters in DynamoDB
src/Harbourline.DataPipeline.Functions/FraudSignalFunction.csThe Lambda handler
src/Harbourline.DataPipeline.Cli/KinesisProducer.cs and Program.cshl-pipeline produce
infra/cdk/.../StreamingStack.csThe stack HlM07-Streaming
tests/.../FraudTests.csThe six xUnit tests for all of the above

Part A: the rules, locally and free

Read the rules

  • HIGH_AMOUNT: a debit (negative amount) whose absolute value is at least 5000.00, in either currency.
  • VELOCITY: more than 5 debits of type card_payment, transfer_out or atm_withdrawal on one account in one fixed 10-minute window (10:00 to 10:10, 10:10 to 10:20 and so on). The signal fires exactly when the counter reaches 6, so a window with 20 payments still raises one signal.
  • Signal ids are SHA-256("<transactionId>|<RULE>") truncated to 32 hex characters, the same in C# and Python, so a retry or a re-run produces the same id and consumers can de-duplicate.

A fixed window is simple and cheap (one counter per account and window) but has a known weakness: six payments split 3 and 3 across a window boundary are never flagged. A sliding window fixes it at the cost of more state. This is a deliberate trade-off to discuss, not an oversight.

Run the dry run

cd labs/m07-transaction-pipeline
MSYS_NO_PATHCONV=1 docker run --rm --memory 3g -v "$PWD:/src" -v cea-nuget-m07:/root/.nuget/packages -w /src \
  mcr.microsoft.com/dotnet/sdk:10.0 sh -c \
  "dotnet run --project src/Harbourline.DataPipeline.Cli -c Release -- produce data/samples/transactions.csv --dry-run --burst ACC-1000000001 --limit 300; \
   find . -type d \( -name bin -o -name obj \) -prune -exec rm -rf {} +"

The author's run printed:

VELOCITY     ACC-1000000001      -9.99 GBP more than 5 debits in the 10-minute window starting 09:00Z
Dry run: 308 transactions evaluated locally, nothing sent.

308 = the first 300 clean rows plus the 8-payment burst. The 300 historic rows trigger nothing (no single debit of 5,000, no six debits in ten minutes), and the burst raises one VELOCITY signal, at the sixth payment. The window start (09:00Z here) is the start of the current ten-minute window when you run it, so your time will differ.

Why the burst is aligned to a window

The burst helper creates eight payments of GBP 9.99, 30 seconds apart. It starts at the beginning of the current ten-minute window, not "now". An earlier version started at the current time, so a run near a window edge would split the eight payments 5 plus 3 across two windows and raise no signal: the lab would pass or fail depending on the clock. The fix is in Program.cs: FraudRuleEngine.WindowStart(DateTime.UtcNow, TimeSpan.FromMinutes(10)). The timestamps are slightly in the past, which is fine because the engine uses the record's own CreatedAt, not the time of arrival.

Run the unit tests

The six tests in FraudTests.cs cover: one velocity signal on the sixth debit, high-amount plus duplicate delivery, the window and signal-id twins of the Python code, the Lambda's partial-batch behaviour, the DynamoDB conditional put and atomic counter, and the producer retrying only the failed records. They were part of the 27 .NET tests, which all passed in the author's run:

Test run summary: Passed!
  total: 27
  failed: 0
  succeeded: 27

Part B: deploy and run

Publish the Lambda package (Linux ARM64, built in Docker)

MSYS_NO_PATHCONV=1 docker run --rm -v "$PWD:/src" -v cea-nuget-m07:/root/.nuget/packages -w /src mcr.microsoft.com/dotnet/sdk:10.0 \
  sh -c "dotnet publish src/Harbourline.DataPipeline.Functions -c Release -r linux-arm64 --self-contained false -o publish/functions"

This creates publish/functions/. Its presence is what tells the CDK app to define the ingest API and HlM07-Streaming. The folder is generated; do not commit it.

Deploy

cd infra/cdk
aws sso login --profile learn-sandbox
cdk deploy HlM07-Streaming --profile learn-sandbox -c owner=<your-name>

CDK deploys HlM07-DataLake first if needed, because the stream uses its key. Read the change summary: it should list a Kinesis stream, a DynamoDB table, an SNS topic, an SQS queue, a Lambda function and role, an event source mapping and encrypted log groups. The Lambda runtime is dotnet10; the stack builds it with new Runtime("dotnet10", RuntimeFamily.DOTNET_CORE) because CDK may not yet have a named constant for it ⚠ VERIFY that dotnet10 is accepted in your Region. Note the outputs StreamName and SignalsTopicArn.

Subscribe to the alerts

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1 MSYS_NO_PATHCONV=1
TOPIC=$(aws cloudformation describe-stacks --stack-name HlM07-Streaming --query "Stacks[0].Outputs[?OutputKey=='SignalsTopicArn'].OutputValue" --output text)
aws sns subscribe --topic-arn "$TOPIC" --protocol email --notification-endpoint you@example.com

Replace the address with your own and confirm the e-mail AWS sends. The mapping starts reading at LATEST, so wait about a minute after deploy before sending: records sent before the event source mapping is active are not read.

Send the transactions and the burst

cd ../..
export AWS_PROFILE=learn-sandbox
dotnet run --project src/Harbourline.DataPipeline.Cli -- produce data/samples/transactions.csv \
  --stream hl-m07-transactions --limit 200 --burst ACC-1000000001

(If you have no .NET SDK locally, run the same command in the Docker image, adding -v ~/.aws:/root/.aws -e AWS_PROFILE=learn-sandbox.) Expected: Sent 208 records to hl-m07-transactions. The producer sends in PutRecords batches of up to 500, uses the account id as the partition key (so one account's records stay in order on one shard) and re-sends only the entries that failed, with exponential back-off.

See the signal

Within a few seconds you should receive an e-mail with the subject Harbourline fraud signal VELOCITY and a JSON body like:

{"signalId":"...","rule":"VELOCITY","transactionId":"...","accountId":"ACC-1000000001",
 "amount":-9.99,"currency":"GBP","createdAt":"...","detail":"more than 5 debits in the 10-minute window starting 09:00Z"}

Check the Lambda's own summary line:

aws logs tail /aws/lambda/hl-m07-fraud-rules --since 10m

One log line per batch: {"records":208,"signals":1,"failures":0} (the batch may be split in two, so you may see two lines whose records add up to 208). The velocity table now holds markers and counters:

aws dynamodb scan --table-name hl-m07-velocity --select COUNT

Expect on the order of the number of processed transactions plus a counter item per account and window, all with a TTL of one day.

Experiment 1: send the burst again

Run the same produce command again within the same ten-minute window. No second e-mail arrives. The burst uses new transaction ids, so the "seen" check passes, but the counter for the window is already 8, and the rule fires only when it equals 6. The new payments take it to 16 without crossing 6 again. This is "one signal per window" in action.

Experiment 2: a poison record

aws kinesis put-record --stream-name hl-m07-transactions --partition-key poison --cli-binary-format raw-in-base64-out --data "not json"
aws logs tail /aws/lambda/hl-m07-fraud-rules --since 2m | grep -i skipping

The Lambda logs Skipping unparseable record <sequence number> and carries on: it does not fail the batch. A record that can never parse would otherwise block the shard forever (Kinesis delivers in order, so one stuck record stops everything behind it until it expires). The handler logs ids only, never the payload. A production design also sends such records to a dead-letter destination for inspection.

Part C: read the code

FraudRuleEngine.EvaluateAsync (core of the rules)
if (!await store.TryMarkSeenAsync(tx.TransactionId, cancellationToken))
{
    return []; // already evaluated: do not count it twice or alert twice
}

var signals = new List<FraudSignal>();
if (tx.Amount < 0 && -tx.Amount >= _options.HighAmountThreshold)
{
    signals.Add(Create(HighAmount, tx, ...));
}

if (tx.Amount < 0 && _options.VelocityTypes.Contains(tx.Type))
{
    var window = WindowStart(tx.CreatedAt, _options.VelocityWindow);
    var count = await store.IncrementAsync(tx.AccountId, window, cancellationToken);
    if (count == _options.VelocityMaxDebits + 1)
    {
        signals.Add(Create(Velocity, tx, ...)); // exactly at the crossing point
    }
}
return signals;
  1. TryMarkSeenAsync — Kinesis delivers at least once. A Lambda retry, a stream replay or a duplicate put can present the same transaction twice. DynamoDB does PutItem with attribute_not_exists(pk); the second attempt throws ConditionalCheckFailedException, which the store turns into false. The engine returns no signals for a duplicate, so counts and alerts are not doubled.
  2. High amount — a debit is negative; the rule uses its absolute value and >=. A credit never triggers it.
  3. Velocity types — only card payments, outgoing transfers and ATM withdrawals count. A direct debit or a fee does not.
  4. WindowStart — floors the record's own timestamp to the window start by integer arithmetic on ticks (ticks - ticks % window.Ticks). Using the record time, not the arrival time, makes a replay produce the same windows.
  5. IncrementAsync — an atomic ADD debit_count :one on the item VEL#<account>#<window start>, returning the new value. Atomicity matters because Lambda may process several batches at once; two concurrent increments both get distinct values and exactly one of them sees 6.
  6. count == Max + 1 — equality, not >=, so each window alerts once.
FraudSignalFunction.HandleAsync (error handling)
catch (JsonException ex)
{
    context.Logger.LogError($"Skipping unparseable record {record.Kinesis.SequenceNumber}: {ex.Message}");
}
catch (Exception ex) when (ex is AmazonSimpleNotificationServiceException or AmazonDynamoDBException)
{
    if (tx is not null) { await _engine.ReleaseAsync(tx.TransactionId); }
    response.BatchItemFailures.Add(new StreamsEventResponse.BatchItemFailure { ItemIdentifier = record.Kinesis.SequenceNumber });
    break;
}
  1. Poison versus transient. A JsonException will never succeed, so the handler skips it. An SNS or DynamoDB failure is transient, so the handler asks for a retry.
  2. ReleaseAsync — deletes the "seen" marker, otherwise the retry would be dropped as a duplicate and the signal would be lost. The cost: the retry may count the transaction twice in its velocity window. Over-alerting is safer than losing an alert.
  3. BatchItemFailures with the failing sequence number, then break — with ReportBatchItemFailures enabled, Lambda retries from that record. Records before it are not re-sent, and stopping at the first failure preserves per-account order.
  4. In the stack: RetryAttempts = 5, MaxRecordAge = 1 hour, OnFailure = SQS — after five attempts or an hour, the metadata of the failed batch goes to hl-m07-fraud-rules-failures (not the payload, which stays in Kinesis for its retention).

Security considerations

⚠ Learning shortcut — not for production

The stream has one shard, no consumer isolation and a 24-hour retention; the alert goes to an e-mail address; the rules are two constants; velocity is a fixed window; and a failed record is described, not preserved. A bank sizes shards (or uses on-demand mode) from peak traffic, sends signals to a case-management system with an audit trail, tunes thresholds with the fraud team, uses sliding windows or a stateful stream processor, and keeps failed payloads in an encrypted dead-letter store.

  • Signals are not decisions. A rule match creates a review task. Blocking a customer's payment automatically is a conduct, legal and customer-harm question for fraud, compliance and legal teams, not a Lambda.
  • Do not log payloads. The handler logs counts and sequence numbers. A transaction contains account ids, which are personal data in context.
  • Encrypt every hop: stream, table, topic, queue and logs use the customer managed key; TLS is enforced on the topic and queue.
  • Least privilege: the producer role needs only kinesis:PutRecords on this stream. Do not run the producer with administrator credentials outside the sandbox.
  • Alert fatigue is a risk too. A rule that fires too often is ignored. Thresholds need measuring against real false-positive rates.

Compliance review required

Transaction monitoring for fraud and financial crime has legal obligations (for example suspicious-activity reporting) that this learning lab does not meet. Rules, thresholds, retention and who may see alerts are decisions for your fraud, compliance and data-protection teams.

Common mistakes

  • Sending immediately after deploy → the mapping starts at LATEST, so early records are not read.
  • Expecting a second alert for the same window → by design there is one.
  • Using the transaction id as the partition key → one account's records scatter over shards and the velocity count loses order guarantees. Use the account id.
  • Treating PutRecords as all-or-nothing → check FailedRecordCount and retry only the failed entries (the producer does).
  • Forgetting to delete the stream → about USD 11 a month for nothing.
  • Retrying a poison record forever → skip and log it, then dead-letter it.

Troubleshooting

SymptomLikely causeFix
Sent 0 records or an access errorWrong profile or region, or no permissionexport AWS_PROFILE=learn-sandbox; check aws sts get-caller-identity
ProvisionedThroughputExceededException in the producerOne shard accepts 1 MB/s or 1,000 records/s of writesThe producer retries with back-off; lower --limit
No e-mail arrivesSubscription not confirmed, or sent before the mapping was active, or mail in spamConfirm the subscription; wait a minute; resend; read the Lambda log
Lambda log: Environment variable VELOCITY_TABLE is not setStack changed outside CDKRedeploy
Lambda errors AccessDeniedException (KMS)Function role lost key accessRedeploy; the stack calls GrantEncryptDecrypt for the function
Deploy fails on the runtimedotnet10 not available in your Region or CDK versionUpdate CDK and check the Lambda runtimes page for your Region ⚠ VERIFY
Signals duplicatedRetry after a transient failure (by design)Consumers de-duplicate on signalId
cdk deploy says the key is in useAnother stack still references the lake's keyDestroy streaming first, then the lake

Cost and cleanup

Destroy the stream stack the same session:

cd infra/cdk
cdk destroy HlM07-Streaming --profile learn-sandbox
aws sns list-subscriptions-by-topic --topic-arn "$TOPIC" 2>/dev/null   # should fail: the topic is gone
aws kinesis list-streams --query "StreamNames[?contains(@, 'hl-m07')]"   # should be []
aws dynamodb list-tables --query "TableNames[?contains(@, 'hl-m07')]"    # should be []

If you continue to the project, keep HlM07-DataLake; otherwise cdk destroy HlM07-DataLake too, and delete publish/ and the unsubscribed e-mail confirmation. Then run the tag check:

aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --query "ResourceTagMappingList[?contains(ResourceARN, 'hl-m07')].ResourceARN"

A stream left running costs about USD 11 a month (0.015 × 24 × 30) plus data. If the destroy is stuck, delete the stream by hand: aws kinesis delete-stream --stream-name hl-m07-transactions --enforce-consumer-deletion.

Exercises

  • 🟢 Guided: Run the dry run with and without --burst, then change VelocityMaxDebits to 3 in FraudRuleOptions, re-run and explain the new signal's transaction. Do not edit the deployed Lambda.
  • 🟡 Challenge: Implement a sliding window locally: keep the timestamps of the last debits per account in the in-memory store and raise the signal when more than five fall within any 10-minute span. Write a test where six payments straddle a fixed-window boundary and show the fixed window misses it and yours catches it.
  • 🔴 Live problem: After a deploy, fraud analysts report two alerts for one burst, and one missing alert for a large transfer. Using what you know about at-least-once delivery, ReleaseAsync, MaxRecordAge and partial-batch responses, list the likely causes for each symptom and how you would investigate them with the Lambda log, the DynamoDB table and the SQS failure queue.

Assignment

Write a docs/streaming-fraud-design.md for your portfolio: the two rules and their thresholds, the window trade-off, the at-least-once handling, the failure paths (poison record, transient failure, record too old) and what you would change before production. Include the dry-run output and the cost of your session.

Solutions

Knowledge check

Interview questions

  • "How do you detect velocity fraud in a stream?" — Key by account, keep a counter per account and time window in a fast store with atomic increments and a TTL, and emit a signal when the threshold is crossed. Mention fixed versus sliding windows and de-duplication.
  • "Kinesis delivers at least once. How do you cope?" — Make consumers idempotent: a conditional write marks processed ids, signal ids are deterministic and downstream systems de-duplicate.
  • "What is a partial batch response?" — A Lambda return value listing the failed item so only it and later records are retried.
  • "Why a signal and not a block?" — Automated blocking has customer and legal consequences, so rules create reviewable signals for analysts or a case system.

Certification practice

Completion checklist

  • I ran the dry run and can explain why the output has one VELOCITY line
  • I deployed, subscribed, sent the burst and received one signal
  • I tried the repeat burst and the poison record and explained both results
  • I can explain at-least-once delivery, the seen marker and ReleaseAsync
  • I destroyed the stack, confirmed no stream or table remains and ran the tag query

Key takeaways

  • Two simple rules on a stream give near-real-time signals without an ML service.
  • Fixed ten-minute windows with an atomic counter and equality at the threshold give exactly one signal per window.
  • At-least-once delivery needs idempotent consumers: conditional writes and deterministic signal ids.
  • Separate poison records (skip) from transient failures (retry from the failing record).
  • A running stream bills by the hour; deploy, test and destroy within the hour.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page