Fraud-detection integration: rules, ML scoring and human review
Integrate fraud signals into banking flows without relying on services closed to new customers: layered rules on Kinesis and Lambda, optional ML scoring through SageMaker AI, Bedrock for analyst summaries (never for the decision), a human review queue built from SQS and DynamoDB, and a feedback loop.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SageMakerRuntime, AWSSDK.BedrockRuntime, AWSSDK.SQS 4.x) · CDK v2 (Amazon.CDK.Lib 2.277.0) · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Separate a fraud signal (evidence) from a fraud decision (action), and design the policy that turns one into the other.
- Place fraud checks in the payment flow: a fast synchronous check before money moves, and slower streaming analysis afterwards.
- Build layered detection: rules (velocity, new payee plus large amount, impossible travel), model scores and case review.
- Integrate an ML scoring endpoint with a timeout and a safe fallback, using Amazon SageMaker AI, or a local C# scorer when you do not want any cloud cost.
- Explain what Amazon Bedrock should and should not do in a fraud workflow.
- Build a human review queue from SQS, DynamoDB and Step Functions, now that Amazon Augmented AI (A2I) is closed to new customers.
- Design the feedback loop (analyst outcomes become labels) and the metrics that show whether the system works.
Prerequisites
- Payment processing and real-time transaction processing
- Streaming ingestion and the Kinesis fraud-signals lab for the rule engine mechanics
- Dead-letter queues and redrive
- Tools: .NET 10 SDK or Docker. AWS access is optional; this lesson is design-first
Why this matters in banking
Fraud is an adversarial problem: the other side changes tactics when you block them. Three tensions shape every design:
- Catch more, annoy fewer. Every false positive is a legitimate customer blocked at the worst moment. Every missed fraud is a loss and often a regulatory conversation. There is no setting with zero of both, so the design must make the trade-off explicit and adjustable.
- Speed versus depth. A decision before payment must take milliseconds. Network analysis across thousands of accounts takes longer. You need both, at different points.
- Explainability. A customer, an ombudsman or a regulator may ask why a payment was stopped. "The model said so" is not an acceptable answer. Rules, reason codes and analyst notes are.
Educational content, not model-risk or legal advice
Fraud models are subject to your bank's model-risk management, fairness and data-protection requirements, including rules on automated decision-making under UK GDPR ⚠ VERIFY. Treat every threshold and rule in this lesson as a fictional teaching value. Fraud, model-risk, data-protection and compliance teams must review a real design before use.
Service availability — verified 2026-11-18
Amazon Fraud Detector has been closed to new customers since 2025-11-07, so this lesson does not use it. SageMaker AI Clarify, Model Monitor and Ground Truth, and Amazon Augmented AI (A2I), are closed to new customers since 2026-07-30; the review queue is therefore built from general-purpose services, and monitoring uses open-source tools and CloudWatch. These may still appear as exam vocabulary.
The idea in plain language
Fraud detection is a pipeline of increasingly expensive checks, each able to stop, challenge, escalate or allow:
- Hard rules: certain things are never allowed (payment from a frozen account, sanctioned country list handled elsewhere).
- Velocity and behaviour rules: "ten payments in two minutes", "new payee plus 90% of balance", "login from a new device and a new country within an hour".
- Model score: a machine-learning model that has learned patterns from past fraud, returning a probability-like number.
- Human review: an analyst looks at ambiguous cases with the evidence collected.
- Learning: analyst outcomes and customer disputes become labels that improve rules and models.
The output of automated checks is a signal: a score, a list of triggered rules and reason codes. A separate policy turns signals into an action.
Analogy: airport security. The metal detector (rule) beeps for belts, the body scanner (model) flags unusual shapes, and an officer (human) decides what to do about the ambiguous ones. The policy, "who gets a second check", is set by the airport, not by the machine. Over time the officers' decisions teach the system what really matters.
Signals versus decisions
| Signal | Decision | |
|---|---|---|
| Produced by | Rules, models, device and network intelligence | A policy component owned by the fraud team |
| Content | Score 0.0 to 1.0, reason codes, features used, model version | Allow, Challenge (step-up), Hold for review, Block |
| Changes how often | Model retrains, rules updated | Thresholds tuned by fraud strategy |
| Stored for audit | Yes, with the version that produced it | Yes, with who or what decided |
Keeping the two separate means you can change a threshold in minutes without redeploying a model, and you can explain every decision.
What problem the AWS services solve
| Service | Role | Limits and sharp edges |
|---|---|---|
| Amazon Kinesis Data Streams | Carries events to streaming rule engines | Ordered per key; hot keys limit throughput |
| AWS Lambda / Amazon Managed Service for Apache Flink | Streaming rules and aggregates (velocity counters, sliding windows) | Lambda suits simple rules; Flink suits stateful windows over large streams ⚠ VERIFY service naming and runtime support |
| Amazon DynamoDB / ElastiCache | Velocity counters, device and payee history | Counters need atomic increments and expiry; cache is not authoritative |
| Amazon SageMaker AI real-time or serverless inference | Hosts a trained model behind an endpoint | Real-time endpoints bill hourly while running; serverless inference bills per use but has cold starts and concurrency limits ⚠ VERIFY |
| SageMaker AI training, MLflow, Pipelines | Train, track and deploy models | Operating ML needs data science skills and governance |
| Amazon Bedrock | Language models for analyst assistance (summaries, drafting), with Guardrails | Not a fraud classifier; outputs can be wrong; never send data you are not allowed to process there ⚠ VERIFY data handling terms |
| Amazon SQS + DynamoDB + Step Functions | Review queue, case state, SLA timers and callbacks | You build the analyst tool or integrate a case-management product |
| Amazon S3 + Athena | Labelled history for training and for metrics | Needs a governed data lake (lake governance) |
| Amazon Fraud Detector | Closed to new customers since 2025-11-07 | Use the components above |
| A2I, Ground Truth | Closed to new customers since 2026-07-30 | Build the review queue yourself |
Honest limitation: a bank's fraud capability usually includes commercial specialist products (consortium data, device intelligence, behavioural biometrics). AWS gives you the streaming, hosting and workflow platform around them, not the fraud intelligence itself.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Stream Analytics windows | Managed Service for Apache Flink, or Lambda with DynamoDB counters | Flink is code-first (Java/Python/SQL); Stream Analytics is SQL-first |
| Azure Machine Learning managed online endpoint | SageMaker AI real-time or serverless endpoint | Similar. Both bill compute while the endpoint exists (unless serverless) |
| Azure AI Content Safety and Azure OpenAI with filters | Bedrock with Guardrails | Guardrails apply content and sensitive-information filters around model calls |
| Power Automate approvals, Logic Apps human tasks | Step Functions callbacks plus SQS and a review UI | You assemble the analyst tooling |
| Azure Sentinel fraud analytics | Security Hub, GuardDuty (security), plus custom fraud pipeline | Security tools detect cloud threats, not customer fraud: do not confuse them |
| ML.NET models in the app | SageMaker endpoint or an ONNX model loaded in-process with ONNX Runtime | An in-process model avoids a network call and its latency |
Architecture
Where fraud checks sit
Security-boundary view
| Concern | Control |
|---|---|
| Analyst access to personal data | SSO with MFA; role per queue; purpose-limited views; every view logged |
| Model and training data | Training data in a restricted S3 bucket with a dedicated KMS key; features minimised; no direct PAN or full account numbers |
| Rule changes | Rules and thresholds in version control, reviewed by two people, deployed through a pipeline; emergency changes logged |
| Bedrock calls | Only case summaries with masked fields; Guardrails for sensitive information; model invocation logging to a restricted store ⚠ VERIFY settings |
| Endpoint | Private subnets and VPC endpoints; IAM role scoped to sagemaker:InvokeEndpoint on one endpoint |
| Evasion | Do not expose rule details or reason codes to the customer; reveal only a generic message |
Step-by-step implementation
Define what you are optimising (sections 1 to 3)
| Measure | Fictional target |
|---|---|
| Fraud detected (by value) | At least 85% of fraud value caught before or at payment |
| False-positive ratio | No more than 8 genuine payments challenged per fraud caught (operational target) |
| Decision latency added to payment | p99 under 40 ms |
| Review SLA | High-risk cases reviewed within 15 minutes; standard within 4 hours |
| Availability | If fraud checks are unavailable, behaviour is defined by policy (below), never "allow everything" |
| Explainability | Every Hold or Block stores reason codes and the model version |
Design the policy (section 7 and 11)
Thresholds map a combined risk to an action. Example (fictional):
| Combined risk | Action | What the customer sees |
|---|---|---|
| Below 0.30 | Allow | Normal payment |
| 0.30 to 0.60 | Challenge: step-up authentication | "Confirm it's you" |
| 0.60 to 0.85 | Hold for review | "We're checking this payment" |
| Above 0.85 or a hard rule | Block | A generic message and a phone number |
Each threshold is a configuration value with an owner, a change history and a back-test result.
Choose the scorer and its fallback (section 5 and 11)
Behind an interface, supply: a rules scorer (always available, in-process); a model scorer (SageMaker endpoint or in-process ONNX model). The decision service calls the model with a short timeout. On timeout or error, fall back to rules only and tag the decision modelUnavailable. Alarm when the share of fallbacks rises. Decide with the fraud team whether lower-confidence decisions should be stricter in fallback mode (for example, challenge more often).
Add streaming rules for what a single payment cannot show (section 10)
A single payment cannot show ten payees added in an hour, or ten accounts paying one new payee. Consume the event stream, maintain windowed counters, and emit signals to a topic. Signals create cases or tighten policy for a customer ("elevated risk for 24 hours").
Build the review queue (section 10)
When the action is Hold, the saga writes a case item in DynamoDB (status = OPEN, SLA deadline) and sends a message to SQS with the case id. The payment saga pauses with a task token (see the onboarding lesson). The analyst tool reads the queue, shows evidence, records the decision and reason, and calls SendTaskSuccess. Messages that fail repeatedly go to a dead-letter queue with an alarm. Add escalation timers so no case waits silently.
Use Bedrock only to help the analyst (section 5)
Give the analyst a short generated summary of the case ("new payee created 3 minutes ago, amount is 92% of balance, device not seen before"). The analyst remains the decision-maker. Feed the model only masked fields, use Guardrails, store the prompt and answer with the case, and show the underlying facts beside the summary so the analyst can detect mistakes. Do not use a language model to choose Allow or Block.
Close the feedback loop (section 14)
Every analyst outcome and every customer dispute becomes a labelled example in the lake with the features present at decision time (not features recomputed later, which leaks the future). Retrain on a schedule, evaluate against a held-out period, compare to the current model on the same data, and deploy by shadow mode first (score but do not act), then canary.
AWS Console instructions
- Console → SQS → Create queue. Standard queue
hl-m14-fraud-review, with a dead-letter queuehl-m14-fraud-review-dlq, redrive after 5 receives, SSE enabled, tagsProject=cea-lab,Lab=m14-l7. - Console → DynamoDB → Create table
hl-m14-fraud-cases, partition keycaseId(String), on-demand. - Console → SageMaker AI → Inference → Endpoints: look at the pricing note on creating an endpoint, but do not create one unless you will delete it in the same hour. The local C# scorer below replaces it for the lab.
- Console → Bedrock → Model access / Playground: try a fictional case summary prompt to see output quality. Do not paste real data. Model availability and pricing vary ⚠ VERIFY.
AWS CLI
# Create the review queue and its dead-letter queue.
aws sqs create-queue --queue-name hl-m14-fraud-review-dlq --profile learn-sandbox --region us-east-1
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$(aws sqs get-queue-url --queue-name hl-m14-fraud-review-dlq --query QueueUrl --output text --profile learn-sandbox --region us-east-1)" \
--attribute-names QueueArn --query Attributes.QueueArn --output text --profile learn-sandbox --region us-east-1)
aws sqs create-queue --queue-name hl-m14-fraud-review \
--attributes "{\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"$DLQ_ARN\\\",\\\"maxReceiveCount\\\":\\\"5\\\"}\",\"SqsManagedSseEnabled\":\"true\"}" \
--tags Project=cea-lab,Lab=m14-l7,Environment=learn,Owner=learner \
--profile learn-sandbox --region us-east-1
# Send a fictional case message.
aws sqs send-message --queue-url "$(aws sqs get-queue-url --queue-name hl-m14-fraud-review --query QueueUrl --output text --profile learn-sandbox --region us-east-1)" \
--message-body '{"caseId":"CASE-0001","paymentId":"PAY-TEST-001","riskScore":0.72,"reasons":["NEW_PAYEE","HIGH_PERCENT_OF_BALANCE"]}' \
--profile learn-sandbox --region us-east-1The RedrivePolicy is a JSON string inside JSON, so the escaping looks heavy. In practice put the attributes in a file and use --attributes file://attributes.json.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.SQS;
using Constructs;
using DdbAttribute = Amazon.CDK.AWS.DynamoDB.Attribute;
public sealed class FraudReviewStack : Stack
{
public FraudReviewStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
{
var dlq = new Queue(this, "ReviewDlq", new QueueProps
{
QueueName = "hl-m14-fraud-review-dlq",
RetentionPeriod = Duration.Days(14),
Encryption = QueueEncryption.SQS_MANAGED,
});
_ = new Queue(this, "Review", new QueueProps
{
QueueName = "hl-m14-fraud-review",
VisibilityTimeout = Duration.Minutes(5), // analyst working time before the message reappears
Encryption = QueueEncryption.SQS_MANAGED,
DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 },
});
_ = new Table(this, "Cases", new TableProps
{
TableName = "hl-m14-fraud-cases",
PartitionKey = new DdbAttribute { Name = "caseId", Type = AttributeType.STRING },
BillingMode = BillingMode.PAY_PER_REQUEST,
Encryption = TableEncryption.AWS_MANAGED,
PointInTimeRecoverySpecification = new PointInTimeRecoverySpecification { PointInTimeRecoveryEnabled = true },
RemovalPolicy = RemovalPolicy.DESTROY, // lab only
});
}
}Complete code
A decision service with layered scoring, a model call with a timeout and fallback, and a pure policy function. All of it runs locally with no AWS account.
namespace Harbourline.Fraud;
public sealed record PaymentSignals(
string CustomerId,
decimal Amount,
decimal AvailableBalance,
bool PayeeIsNew,
int PaymentsLastTenMinutes,
bool DeviceIsNew,
bool CountryChangedLastHour);
public sealed record Score(decimal Value, IReadOnlyList<string> Reasons, string Source);
public enum FraudAction { Allow, Challenge, Hold, Block }
public sealed record FraudDecision(FraudAction Action, decimal Risk, IReadOnlyList<string> Reasons, bool ModelUsed);
public interface IFraudScorer
{
Task<Score> ScoreAsync(PaymentSignals signals, CancellationToken ct);
}
/// <summary>Always-available scoring in plain C#. Each rule adds to a capped score and records its reason code.</summary>
public sealed class RulesScorer : IFraudScorer
{
public Task<Score> ScoreAsync(PaymentSignals s, CancellationToken ct)
{
decimal score = 0m;
var reasons = new List<string>();
if (s.PayeeIsNew && s.AvailableBalance > 0m && s.Amount / s.AvailableBalance >= 0.9m)
{
score += 0.45m;
reasons.Add("NEW_PAYEE_HIGH_PERCENT_OF_BALANCE");
}
if (s.PaymentsLastTenMinutes >= 5)
{
score += 0.30m;
reasons.Add("VELOCITY_5_IN_10_MIN");
}
if (s.DeviceIsNew && s.CountryChangedLastHour)
{
score += 0.35m;
reasons.Add("NEW_DEVICE_AND_COUNTRY_CHANGE");
}
return Task.FromResult(new Score(Math.Min(score, 1m), reasons, "rules"));
}
}
/// <summary>Calls a model scorer within a time slice; on timeout or error returns null so the caller can fall back.</summary>
public sealed class GuardedModelScorer(IFraudScorer model, TimeSpan timeout)
{
public async Task<Score?> TryScoreAsync(PaymentSignals signals, CancellationToken ct)
{
using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
cts.CancelAfter(timeout);
try
{
return await model.ScoreAsync(signals, cts.Token);
}
catch (OperationCanceledException) when (!ct.IsCancellationRequested)
{
return null; // our timeout fired, not the caller's cancellation
}
catch (HttpRequestException)
{
return null;
}
}
}
public sealed record FraudPolicy(decimal ChallengeAt, decimal HoldAt, decimal BlockAt)
{
public static FraudPolicy Default { get; } = new(0.30m, 0.60m, 0.85m);
public FraudAction ActionFor(decimal risk) =>
risk >= BlockAt ? FraudAction.Block
: risk >= HoldAt ? FraudAction.Hold
: risk >= ChallengeAt ? FraudAction.Challenge
: FraudAction.Allow;
}
public sealed class FraudDecider(RulesScorer rules, GuardedModelScorer model, FraudPolicy policy)
{
public async Task<FraudDecision> DecideAsync(PaymentSignals signals, CancellationToken ct)
{
var ruleScore = await rules.ScoreAsync(signals, ct);
var modelScore = await model.TryScoreAsync(signals, ct);
// The higher of the two wins: a model can raise risk but cannot hide a rule hit.
var risk = Math.Max(ruleScore.Value, modelScore?.Value ?? 0m);
var reasons = ruleScore.Reasons.Concat(modelScore?.Reasons ?? []).Distinct().ToList();
if (modelScore is null)
{
reasons.Add("MODEL_UNAVAILABLE_RULES_ONLY");
}
return new FraudDecision(policy.ActionFor(risk), risk, reasons, ModelUsed: modelScore is not null);
}
}using System.Text;
using System.Text.Json;
using Amazon.SageMakerRuntime;
using Amazon.SageMakerRuntime.Model;
namespace Harbourline.Fraud;
/// <summary>Calls a SageMaker AI real-time endpoint. Only used when you choose to deploy one; the lab uses RulesScorer.</summary>
public sealed class SageMakerScorer(IAmazonSageMakerRuntime runtime, string endpointName) : IFraudScorer
{
public async Task<Score> ScoreAsync(PaymentSignals s, CancellationToken ct)
{
// Send engineered features only: no names, no account numbers.
var features = new
{
amount = s.Amount,
percent_of_balance = s.AvailableBalance > 0 ? s.Amount / s.AvailableBalance : 1m,
payee_is_new = s.PayeeIsNew ? 1 : 0,
payments_10m = s.PaymentsLastTenMinutes,
device_is_new = s.DeviceIsNew ? 1 : 0,
country_changed_1h = s.CountryChangedLastHour ? 1 : 0,
};
var response = await runtime.InvokeEndpointAsync(new InvokeEndpointRequest
{
EndpointName = endpointName,
ContentType = "application/json",
Accept = "application/json",
Body = new MemoryStream(Encoding.UTF8.GetBytes(JsonSerializer.Serialize(features))),
}, ct);
using var doc = await JsonDocument.ParseAsync(response.Body, cancellationToken: ct);
var value = doc.RootElement.GetProperty("score").GetDecimal();
return new Score(Math.Clamp(value, 0m, 1m), ["MODEL_SCORE"], $"sagemaker:{endpointName}");
}
}Code explained line by line
PaymentSignalsholds only the features the decision needs. It contains no names, addresses or account numbers, which limits personal data exposure to the model and to logs.RulesScoreradds to a capped score for each rule and records a reason code. Reason codes become the explanation shown to analysts and stored for audit.GuardedModelScorerlinks the caller's cancellation token with its own timeout. If the timeout fires and the caller did not cancel, the exception filterwhen (!ct.IsCancellationRequested)returnsnull. A real caller cancellation still propagates.FraudPolicy.ActionForis a pure function from risk to action, with the thresholds in a record so they can be loaded from reviewed configuration.FraudDecidertakes the maximum of the rule score and the model score. The model can raise risk, but a model that has drifted or is silent cannot hide a hard rule hit.- When the model is unavailable the decision carries
MODEL_UNAVAILABLE_RULES_ONLYandModelUsed = false, so dashboards can show the fallback rate. SageMakerScorerserialises features to JSON, calls the endpoint throughInvokeEndpointAsyncand clamps the returned score to the range 0 to 1 so a bad model output cannot produce an out-of-range value.IAmazonSageMakerRuntimeis injected, so a unit test can fake the endpoint with no AWS access.
Optional: an analyst summary with Bedrock
using Amazon.BedrockRuntime;
using Amazon.BedrockRuntime.Model;
public sealed class CaseSummariser(IAmazonBedrockRuntime bedrock, string modelId, string guardrailId, string guardrailVersion)
{
public async Task<string> SummariseAsync(string maskedCaseFacts, CancellationToken ct)
{
var response = await bedrock.ConverseAsync(new ConverseRequest
{
ModelId = modelId,
Messages =
[
new Message
{
Role = ConversationRole.User,
Content = [new ContentBlock { Text =
"Summarise these fraud case facts in three short bullet points for an analyst. " +
"Use only the facts given. If something is unknown, say so.\n\n" + maskedCaseFacts }],
},
],
InferenceConfig = new InferenceConfiguration { MaxTokens = 300, Temperature = 0.2f },
GuardrailConfig = new GuardrailConfiguration { GuardrailIdentifier = guardrailId, GuardrailVersion = guardrailVersion },
}, ct);
return response.Output.Message.Content[0].Text;
}
}Property names for the Converse API and for guardrails should be checked against the current AWSSDK.BedrockRuntime package ⚠ VERIFY. The model id is a placeholder you supply from the Bedrock console for a model enabled in your account.
Security considerations
⚠ Learning shortcut — not for production
The lab combines rules and a score with fixed numbers, and the thresholds are constants. Production needs: back-testing of every threshold change on historical data before release, a champion/challenger or shadow period for new models, governance and approval for model changes, protection of the rule set from insiders, bias and fairness testing, and a documented human-in-the-loop for automated decisions with significant effect ⚠ VERIFY under UK GDPR with your data protection officer.
- Do not leak detection logic. Customer-facing messages are generic; reason codes stay internal.
- Insider risk: limit who can change rules; use two-person review and log all changes.
- Data minimisation: features for the model and the LLM are the smallest set that works; mask account numbers and names.
- Prompt injection: free-text fields from customers (payment references) can contain instructions aimed at a language model. Treat them as untrusted data, exclude them from prompts when possible, and never let model output trigger actions.
- Adversarial adaptation: attackers probe thresholds. Rate-limit and randomise challenges where policy allows ⚠ VERIFY with the fraud team.
- Auditability: store the model version, features, thresholds and reason codes with every decision.
- Fallback is a control: the behaviour when the model is down is a risk decision, approved and documented.
Banking use case
Harbourline's fraud team (fictional) sees authorised push payment scams: customers are tricked into paying a new payee.
- A payment of GBP 4,800 from an account with GBP 5,000 to a payee created 2 minutes ago, from a new device, scores 0.80 through rules alone (new payee and high percentage plus new device); the policy puts it on Hold.
- The saga creates a case and pauses. An analyst sees the facts and a Bedrock-generated summary, calls the customer on a known number, and confirms the customer is being coached by a scammer. The analyst releases a Block with a reason code
APP_SCAM_CONFIRMED. - Outcome and features go to the lake as a label. Next month's retraining sees this pattern again.
- During a model-endpoint outage, the fallback rate rises to 100%. The decision service keeps working on rules; policy tightens the challenge threshold for new payees during the fallback ⚠ VERIFY with the fraud team. The on-call engineer is paged on the fallback rate alarm.
Common mistakes
- Letting the model decide alone → no explanation, no override → rules and policy sit around the model.
- No timeout on the model call → one slow endpoint stalls every payment → timeout plus fallback.
- Training on features computed after the fact → the model "sees the future" and fails live → store features as they were at decision time.
- Treating the fallback as free → silent risk increase → alarm on fallback rate, decide stricter fallback policy.
- Review queue without SLA timers → cases sit for days → escalation timers and dashboards.
- Using an LLM to classify fraud → unstable and unauditable → LLM for summaries only.
- Sending raw personal data to a model → data-protection exposure → engineered, minimised features.
- Measuring only detection rate → customers drown in false positives → track both, with cost per case.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Fallback rate is 100% | Endpoint down, wrong name, or IAM missing sagemaker:InvokeEndpoint | Check endpoint status, role policy and the 25 ms timeout vs real latency |
| Latency spikes on decisions | Model cold start (serverless) or large payload | Keep payload tiny, use provisioned concurrency or an in-process model, raise timeout carefully |
| Many false positives after a rule change | Threshold or rule too broad | Roll back; back-test against labelled history before change |
| Review queue messages end up in the DLQ | Analyst tool failing to acknowledge, or poison message | Inspect DLQ, fix the consumer, redrive (DLQ lesson) |
| Bedrock call returns an access error | Model access not enabled or role lacks permission | Enable the model in the console; grant bedrock:InvokeModel for that model |
| Velocity counters wrong after scale-out | Counters kept in local memory | Use DynamoDB atomic counters or ElastiCache with expiry |
| Model performs worse each month | Data drift, new fraud patterns | Monitor feature distributions and outcomes with CloudWatch and open-source tooling; retrain |
SchemaException from the endpoint | Feature names or order changed | Version the feature contract with the model |
Cost and cleanup
Label: 🟡 credits. The C# rules, policy and tests are free and run locally. A SageMaker real-time endpoint bills per hour while it exists, so for a learning lab prefer the local scorer. SQS and DynamoDB at lab volumes cost cents or fall within free allowances ⚠ VERIFY. Bedrock charges per token ⚠ VERIFY; ten fictional prompts cost very little. Free alternative: do the design and run the local code only.
# Delete the review queues and the case table.
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m14-fraud-review --query QueueUrl --output text --profile learn-sandbox --region us-east-1)" --profile learn-sandbox --region us-east-1
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m14-fraud-review-dlq --query QueueUrl --output text --profile learn-sandbox --region us-east-1)" --profile learn-sandbox --region us-east-1
aws dynamodb delete-table --table-name hl-m14-fraud-cases --profile learn-sandbox --region us-east-1
# If you created a SageMaker endpoint, delete it AND its config and model.
aws sagemaker delete-endpoint --endpoint-name <name> --profile learn-sandbox --region us-east-1
# Confirm nothing tagged for the course remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1Exercises
- 🟢 Guided: write tests for
RulesScorerandFraudPolicy: a payment that triggers no rule isAllow; "new payee plus 95% of balance" scores 0.45 and isChallenge; the same plus a new device in a new country (total 0.80) isHold; risk 0.85 exactly isBlock. - 🟡 Challenge: write a fake
IFraudScorerthat delays 200 ms and a test showingGuardedModelScorerwith a 25 ms timeout returnsnull, and thatFraudDeciderthen reportsModelUsed = falsewith theMODEL_UNAVAILABLE_RULES_ONLYreason. - 🔴 Live problem: the fraud team asks to lower the Hold threshold from 0.60 to 0.50 before the holiday season. Design the safe change process: what data you back-test on, what metrics you compare, the shadow period, the rollback, the capacity impact on the review queue and who must approve.
Assignment
Write the 15-section fraud integration architecture in architecture/fraud-integration/: the layered diagram, the policy table with owners, the decision-service contract with reason codes, the review-queue design (state, SLA, escalation), the fallback ADR, the feedback loop and monitoring plan, and the compliance touchpoints (model risk, data protection and automated decisions, fraud operations, third-party risk for any vendor). Include the local C# scorer and its tests.
Solutions
using Harbourline.Fraud;
using Xunit;
public class FraudTests
{
private static PaymentSignals Signals(bool newPayee = false, decimal amount = 100m, int recent = 0, bool newDevice = false, bool country = false) =>
new("CUS-00000001", amount, 5000m, newPayee, recent, newDevice, country);
[Fact]
public async Task No_rule_means_allow()
{
var score = await new RulesScorer().ScoreAsync(Signals(), default);
Assert.Equal(0m, score.Value);
Assert.Equal(FraudAction.Allow, FraudPolicy.Default.ActionFor(score.Value));
}
[Fact]
public async Task New_payee_high_percent_is_challenge()
{
var score = await new RulesScorer().ScoreAsync(Signals(newPayee: true, amount: 4750m), default);
Assert.Equal(0.45m, score.Value);
Assert.Equal(FraudAction.Challenge, FraudPolicy.Default.ActionFor(score.Value));
}
[Fact]
public async Task Combined_rules_reach_hold()
{
var score = await new RulesScorer().ScoreAsync(Signals(newPayee: true, amount: 4750m, newDevice: true, country: true), default);
Assert.Equal(0.80m, score.Value);
Assert.Equal(FraudAction.Hold, FraudPolicy.Default.ActionFor(score.Value));
}
[Fact]
public void Block_threshold_is_inclusive() =>
Assert.Equal(FraudAction.Block, FraudPolicy.Default.ActionFor(0.85m));
}Why: the numbers are chosen so each branch of the policy is hit once, and the inclusive boundary at 0.85 is pinned because boundaries are where policy changes cause surprises.
internal sealed class SlowScorer(TimeSpan delay) : IFraudScorer
{
public async Task<Score> ScoreAsync(PaymentSignals s, CancellationToken ct)
{
await Task.Delay(delay, ct);
return new Score(0.9m, ["MODEL_SCORE"], "slow");
}
}
[Fact]
public async Task Slow_model_falls_back_to_rules()
{
var decider = new FraudDecider(
new RulesScorer(),
new GuardedModelScorer(new SlowScorer(TimeSpan.FromMilliseconds(200)), TimeSpan.FromMilliseconds(25)),
FraudPolicy.Default);
var decision = await decider.DecideAsync(
new PaymentSignals("CUS-00000001", 100m, 5000m, false, 0, false, false), default);
Assert.False(decision.ModelUsed);
Assert.Contains("MODEL_UNAVAILABLE_RULES_ONLY", decision.Reasons);
Assert.Equal(FraudAction.Allow, decision.Action);
}Why: the timeout (25 ms) is shorter than the model's delay (200 ms), so Task.Delay is cancelled and the guard returns null. The decision still completes from rules alone, which is the whole point of the fallback.
(1) Re-run the new threshold over at least several months of labelled history and compare detection (value caught) and false-positive ratio against the current threshold, including seasonal periods. (2) Estimate extra cases per hour and check against review capacity and SLA. (3) Run shadow mode for a week: apply the new threshold virtually and compare. (4) Canary for a slice of traffic with a kill switch. (5) Pre-agree rollback triggers (queue age, complaint rate). (6) Approvals: fraud strategy owner, operations (capacity), model-risk or governance as policy requires, and compliance for customer-impact aspects. (7) Record the change as versioned configuration with the evidence attached.
Knowledge check
Answer: A signal is evidence (score, reason codes, rule hits). A decision is the action chosen by a separate policy (allow, challenge, hold, block). Separating them lets the fraud team tune thresholds quickly and explain every action.
Answer: The model can raise risk but cannot suppress a rule hit, so a drifting or failing model cannot silently let a known pattern through. The result is also easier to explain.
Answer: Rules on Kinesis and Lambda (or Flink), optional model scoring through SageMaker AI, human review through an SQS-based queue, and Bedrock only for analyst summaries.
Answer: Recomputing features later can include information that was not available at the time (for example, later transactions), so the model performs unrealistically well in testing and poorly in production.
Answer: No. Its output is not stable, explainable or auditable enough for a control decision. Use it to help the analyst (summaries), with masked inputs, guardrails and human decisions.
Interview questions
- How would you add fraud scoring to a payment flow? A decision service called before the hold step with a tight budget, layered rules plus an optional model with timeout and fallback, a policy mapping risk to allow, challenge, hold or block, reason codes stored for audit, and a review queue for holds.
- What if the ML endpoint is down? Fall back to rules with a documented, possibly stricter, policy; alarm on fallback rate; keep payments flowing within risk appetite.
- How do you avoid alert fatigue? Measure false positives, tune thresholds by back-testing, route by risk tier, and give analysts good evidence and summaries.
- How do you keep a model trustworthy over time? Monitor feature and outcome drift, retrain on labelled outcomes, use shadow and canary deployment, and keep model-risk governance and approvals.
- Where would you use generative AI in fraud operations? For summarising cases and drafting customer contact notes for an analyst, with masked data, guardrails, logging and a human decision. Not for the allow or block decision.
Certification practice
A bank wants a fraud scoring call added to its payment path without risking the payment SLA if the model endpoint is slow. Which design is best?
A. Call the endpoint synchronously with a long timeout so scores are always available. B. Call the endpoint with a short timeout and fall back to rule-based scoring, flagging the fallback and alarming on its rate. C. Skip fraud checks when the endpoint is slow. D. Move all fraud checks to a nightly batch.
Answer: B. A short timeout with a safe fallback protects the SLA and preserves control. A risks timeouts, C disables the control, D cannot stop payments before they leave.
Which use of a generative AI model in a fraud-review workflow is most appropriate?
A. Automatically blocking customer accounts based on the model's text output B. Summarising masked case facts for an analyst who makes the decision, with guardrails and logging C. Training the model on unmasked customer data without controls D. Sending full card numbers in prompts for convenience
Answer: B. Human-in-the-loop assistance with masked data and guardrails is appropriate. A removes human oversight, and C and D mishandle personal data.
A fraud-scoring service sends features to a SageMaker endpoint. Which practice best limits personal data exposure?
A. Send full customer records so the model has maximum context. B. Send only minimised, engineered features with no direct identifiers, over a private connection, with a least-privilege role. C. Log every request body for debugging. D. Use one shared administrator role for all services.
Answer: B. Data minimisation, private connectivity and least privilege reduce exposure. The others increase it.
A fraud review queue must not lose cases when analyst tooling fails repeatedly on a message. Which feature helps?
A. Reduce the SQS retention to one minute B. An SQS dead-letter queue with a redrive policy and an alarm on queue depth C. Disable encryption D. Use a single consumer only
Answer: B. A dead-letter queue holds messages that keep failing for inspection and redrive. The other options increase loss risk or add none.
Completion checklist
- I can explain signal versus decision and why they are separate
- I wrote the policy table with thresholds and owners
- I ran or wrote tests for the rules, policy and fallback
- I can explain how the review queue replaces A2I
- I limited Bedrock to analyst assistance with masked inputs
- I deleted anything I created and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- Fraud detection is layers: rules, scores, humans and a learning loop, with policy deciding the action.
- Always give the model call a short timeout and a defined fallback; make the fallback a monitored, approved control.
- Closed services (Fraud Detector, A2I, Ground Truth) are replaced by Kinesis, Lambda, SageMaker AI, SQS and Step Functions.
- Language models help analysts; they do not make control decisions.
- Store reason codes, versions and features for every decision, and feed outcomes back as labels.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Real-time transaction processing: latency budgets, back-pressure and ordering
Design a real-time authorisation path that decides in milliseconds on a small hot data set, then streams events to everything else: latency budgets with deadline propagation, DynamoDB balance reservations, Kinesis ordering per account, back-pressure and load shedding, and graceful degradation.
Customer notification systems: fan-out, preferences, retries and duplicate suppression
Design an event-driven notification platform for a bank: planning notifications from domain events, respecting preferences and quiet hours, fanning out to e-mail, SMS and push through per-channel queues, retrying safely, suppressing duplicates and recording delivery evidence.