Skip to Content
Cloud Engineering Academy
14. Advanced Banking Architecture

Fraud-detection integration: rules, ML scoring and human review

Integrate fraud signals into banking flows without relying on services closed to new customers: layered rules on Kinesis and Lambda, optional ML scoring through SageMaker AI, Bedrock for analyst summaries (never for the decision), a human review queue built from SQS and DynamoDB, and a feedback loop.

advanced⏱ 2 h 30 min🟡 Low costSAP-C03SCS-C03AIF-C01SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SageMakerRuntime, AWSSDK.BedrockRuntime, AWSSDK.SQS 4.x) · CDK v2 (Amazon.CDK.Lib 2.277.0) · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Separate a fraud signal (evidence) from a fraud decision (action), and design the policy that turns one into the other.
  • Place fraud checks in the payment flow: a fast synchronous check before money moves, and slower streaming analysis afterwards.
  • Build layered detection: rules (velocity, new payee plus large amount, impossible travel), model scores and case review.
  • Integrate an ML scoring endpoint with a timeout and a safe fallback, using Amazon SageMaker AI, or a local C# scorer when you do not want any cloud cost.
  • Explain what Amazon Bedrock should and should not do in a fraud workflow.
  • Build a human review queue from SQS, DynamoDB and Step Functions, now that Amazon Augmented AI (A2I) is closed to new customers.
  • Design the feedback loop (analyst outcomes become labels) and the metrics that show whether the system works.

Prerequisites

Why this matters in banking

Fraud is an adversarial problem: the other side changes tactics when you block them. Three tensions shape every design:

  • Catch more, annoy fewer. Every false positive is a legitimate customer blocked at the worst moment. Every missed fraud is a loss and often a regulatory conversation. There is no setting with zero of both, so the design must make the trade-off explicit and adjustable.
  • Speed versus depth. A decision before payment must take milliseconds. Network analysis across thousands of accounts takes longer. You need both, at different points.
  • Explainability. A customer, an ombudsman or a regulator may ask why a payment was stopped. "The model said so" is not an acceptable answer. Rules, reason codes and analyst notes are.

Educational content, not model-risk or legal advice

Fraud models are subject to your bank's model-risk management, fairness and data-protection requirements, including rules on automated decision-making under UK GDPR ⚠ VERIFY. Treat every threshold and rule in this lesson as a fictional teaching value. Fraud, model-risk, data-protection and compliance teams must review a real design before use.

Service availability — verified 2026-11-18

Amazon Fraud Detector has been closed to new customers since 2025-11-07, so this lesson does not use it. SageMaker AI Clarify, Model Monitor and Ground Truth, and Amazon Augmented AI (A2I), are closed to new customers since 2026-07-30; the review queue is therefore built from general-purpose services, and monitoring uses open-source tools and CloudWatch. These may still appear as exam vocabulary.

The idea in plain language

Fraud detection is a pipeline of increasingly expensive checks, each able to stop, challenge, escalate or allow:

  1. Hard rules: certain things are never allowed (payment from a frozen account, sanctioned country list handled elsewhere).
  2. Velocity and behaviour rules: "ten payments in two minutes", "new payee plus 90% of balance", "login from a new device and a new country within an hour".
  3. Model score: a machine-learning model that has learned patterns from past fraud, returning a probability-like number.
  4. Human review: an analyst looks at ambiguous cases with the evidence collected.
  5. Learning: analyst outcomes and customer disputes become labels that improve rules and models.

The output of automated checks is a signal: a score, a list of triggered rules and reason codes. A separate policy turns signals into an action.

Analogy: airport security. The metal detector (rule) beeps for belts, the body scanner (model) flags unusual shapes, and an officer (human) decides what to do about the ambiguous ones. The policy, "who gets a second check", is set by the airport, not by the machine. Over time the officers' decisions teach the system what really matters.

Signals versus decisions

SignalDecision
Produced byRules, models, device and network intelligenceA policy component owned by the fraud team
ContentScore 0.0 to 1.0, reason codes, features used, model versionAllow, Challenge (step-up), Hold for review, Block
Changes how oftenModel retrains, rules updatedThresholds tuned by fraud strategy
Stored for auditYes, with the version that produced itYes, with who or what decided

Keeping the two separate means you can change a threshold in minutes without redeploying a model, and you can explain every decision.

What problem the AWS services solve

ServiceRoleLimits and sharp edges
Amazon Kinesis Data StreamsCarries events to streaming rule enginesOrdered per key; hot keys limit throughput
AWS Lambda / Amazon Managed Service for Apache FlinkStreaming rules and aggregates (velocity counters, sliding windows)Lambda suits simple rules; Flink suits stateful windows over large streams ⚠ VERIFY service naming and runtime support
Amazon DynamoDB / ElastiCacheVelocity counters, device and payee historyCounters need atomic increments and expiry; cache is not authoritative
Amazon SageMaker AI real-time or serverless inferenceHosts a trained model behind an endpointReal-time endpoints bill hourly while running; serverless inference bills per use but has cold starts and concurrency limits ⚠ VERIFY
SageMaker AI training, MLflow, PipelinesTrain, track and deploy modelsOperating ML needs data science skills and governance
Amazon BedrockLanguage models for analyst assistance (summaries, drafting), with GuardrailsNot a fraud classifier; outputs can be wrong; never send data you are not allowed to process there ⚠ VERIFY data handling terms
Amazon SQS + DynamoDB + Step FunctionsReview queue, case state, SLA timers and callbacksYou build the analyst tool or integrate a case-management product
Amazon S3 + AthenaLabelled history for training and for metricsNeeds a governed data lake (lake governance)
Amazon Fraud DetectorClosed to new customers since 2025-11-07Use the components above
A2I, Ground TruthClosed to new customers since 2026-07-30Build the review queue yourself

Honest limitation: a bank's fraud capability usually includes commercial specialist products (consortium data, device intelligence, behavioural biometrics). AWS gives you the streaming, hosting and workflow platform around them, not the fraud intelligence itself.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Stream Analytics windowsManaged Service for Apache Flink, or Lambda with DynamoDB countersFlink is code-first (Java/Python/SQL); Stream Analytics is SQL-first
Azure Machine Learning managed online endpointSageMaker AI real-time or serverless endpointSimilar. Both bill compute while the endpoint exists (unless serverless)
Azure AI Content Safety and Azure OpenAI with filtersBedrock with GuardrailsGuardrails apply content and sensitive-information filters around model calls
Power Automate approvals, Logic Apps human tasksStep Functions callbacks plus SQS and a review UIYou assemble the analyst tooling
Azure Sentinel fraud analyticsSecurity Hub, GuardDuty (security), plus custom fraud pipelineSecurity tools detect cloud threats, not customer fraud: do not confuse them
ML.NET models in the appSageMaker endpoint or an ONNX model loaded in-process with ONNX RuntimeAn in-process model avoids a network call and its latency

Architecture

Where fraud checks sit

Security-boundary view

ConcernControl
Analyst access to personal dataSSO with MFA; role per queue; purpose-limited views; every view logged
Model and training dataTraining data in a restricted S3 bucket with a dedicated KMS key; features minimised; no direct PAN or full account numbers
Rule changesRules and thresholds in version control, reviewed by two people, deployed through a pipeline; emergency changes logged
Bedrock callsOnly case summaries with masked fields; Guardrails for sensitive information; model invocation logging to a restricted store ⚠ VERIFY settings
EndpointPrivate subnets and VPC endpoints; IAM role scoped to sagemaker:InvokeEndpoint on one endpoint
EvasionDo not expose rule details or reason codes to the customer; reveal only a generic message

Step-by-step implementation

Define what you are optimising (sections 1 to 3)

MeasureFictional target
Fraud detected (by value)At least 85% of fraud value caught before or at payment
False-positive ratioNo more than 8 genuine payments challenged per fraud caught (operational target)
Decision latency added to paymentp99 under 40 ms
Review SLAHigh-risk cases reviewed within 15 minutes; standard within 4 hours
AvailabilityIf fraud checks are unavailable, behaviour is defined by policy (below), never "allow everything"
ExplainabilityEvery Hold or Block stores reason codes and the model version

Design the policy (section 7 and 11)

Thresholds map a combined risk to an action. Example (fictional):

Combined riskActionWhat the customer sees
Below 0.30AllowNormal payment
0.30 to 0.60Challenge: step-up authentication"Confirm it's you"
0.60 to 0.85Hold for review"We're checking this payment"
Above 0.85 or a hard ruleBlockA generic message and a phone number

Each threshold is a configuration value with an owner, a change history and a back-test result.

Choose the scorer and its fallback (section 5 and 11)

Behind an interface, supply: a rules scorer (always available, in-process); a model scorer (SageMaker endpoint or in-process ONNX model). The decision service calls the model with a short timeout. On timeout or error, fall back to rules only and tag the decision modelUnavailable. Alarm when the share of fallbacks rises. Decide with the fraud team whether lower-confidence decisions should be stricter in fallback mode (for example, challenge more often).

Add streaming rules for what a single payment cannot show (section 10)

A single payment cannot show ten payees added in an hour, or ten accounts paying one new payee. Consume the event stream, maintain windowed counters, and emit signals to a topic. Signals create cases or tighten policy for a customer ("elevated risk for 24 hours").

Build the review queue (section 10)

When the action is Hold, the saga writes a case item in DynamoDB (status = OPEN, SLA deadline) and sends a message to SQS with the case id. The payment saga pauses with a task token (see the onboarding lesson). The analyst tool reads the queue, shows evidence, records the decision and reason, and calls SendTaskSuccess. Messages that fail repeatedly go to a dead-letter queue with an alarm. Add escalation timers so no case waits silently.

Use Bedrock only to help the analyst (section 5)

Give the analyst a short generated summary of the case ("new payee created 3 minutes ago, amount is 92% of balance, device not seen before"). The analyst remains the decision-maker. Feed the model only masked fields, use Guardrails, store the prompt and answer with the case, and show the underlying facts beside the summary so the analyst can detect mistakes. Do not use a language model to choose Allow or Block.

Close the feedback loop (section 14)

Every analyst outcome and every customer dispute becomes a labelled example in the lake with the features present at decision time (not features recomputed later, which leaks the future). Retrain on a schedule, evaluate against a held-out period, compare to the current model on the same data, and deploy by shadow mode first (score but do not act), then canary.

AWS Console instructions

  1. Console → SQS → Create queue. Standard queue hl-m14-fraud-review, with a dead-letter queue hl-m14-fraud-review-dlq, redrive after 5 receives, SSE enabled, tags Project=cea-lab, Lab=m14-l7.
  2. Console → DynamoDB → Create table hl-m14-fraud-cases, partition key caseId (String), on-demand.
  3. Console → SageMaker AI → Inference → Endpoints: look at the pricing note on creating an endpoint, but do not create one unless you will delete it in the same hour. The local C# scorer below replaces it for the lab.
  4. Console → Bedrock → Model access / Playground: try a fictional case summary prompt to see output quality. Do not paste real data. Model availability and pricing vary ⚠ VERIFY.

AWS CLI

# Create the review queue and its dead-letter queue.
aws sqs create-queue --queue-name hl-m14-fraud-review-dlq --profile learn-sandbox --region us-east-1
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$(aws sqs get-queue-url --queue-name hl-m14-fraud-review-dlq --query QueueUrl --output text --profile learn-sandbox --region us-east-1)" \
  --attribute-names QueueArn --query Attributes.QueueArn --output text --profile learn-sandbox --region us-east-1)

aws sqs create-queue --queue-name hl-m14-fraud-review \
  --attributes "{\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"$DLQ_ARN\\\",\\\"maxReceiveCount\\\":\\\"5\\\"}\",\"SqsManagedSseEnabled\":\"true\"}" \
  --tags Project=cea-lab,Lab=m14-l7,Environment=learn,Owner=learner \
  --profile learn-sandbox --region us-east-1

# Send a fictional case message.
aws sqs send-message --queue-url "$(aws sqs get-queue-url --queue-name hl-m14-fraud-review --query QueueUrl --output text --profile learn-sandbox --region us-east-1)" \
  --message-body '{"caseId":"CASE-0001","paymentId":"PAY-TEST-001","riskScore":0.72,"reasons":["NEW_PAYEE","HIGH_PERCENT_OF_BALANCE"]}' \
  --profile learn-sandbox --region us-east-1

The RedrivePolicy is a JSON string inside JSON, so the escaping looks heavy. In practice put the attributes in a file and use --attributes file://attributes.json.

Infrastructure as Code

using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.SQS;
using Constructs;
using DdbAttribute = Amazon.CDK.AWS.DynamoDB.Attribute;

public sealed class FraudReviewStack : Stack
{
    public FraudReviewStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
    {
        var dlq = new Queue(this, "ReviewDlq", new QueueProps
        {
            QueueName = "hl-m14-fraud-review-dlq",
            RetentionPeriod = Duration.Days(14),
            Encryption = QueueEncryption.SQS_MANAGED,
        });

        _ = new Queue(this, "Review", new QueueProps
        {
            QueueName = "hl-m14-fraud-review",
            VisibilityTimeout = Duration.Minutes(5),    // analyst working time before the message reappears
            Encryption = QueueEncryption.SQS_MANAGED,
            DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 },
        });

        _ = new Table(this, "Cases", new TableProps
        {
            TableName = "hl-m14-fraud-cases",
            PartitionKey = new DdbAttribute { Name = "caseId", Type = AttributeType.STRING },
            BillingMode = BillingMode.PAY_PER_REQUEST,
            Encryption = TableEncryption.AWS_MANAGED,
            PointInTimeRecoverySpecification = new PointInTimeRecoverySpecification { PointInTimeRecoveryEnabled = true },
            RemovalPolicy = RemovalPolicy.DESTROY, // lab only
        });
    }
}

Complete code

A decision service with layered scoring, a model call with a timeout and fallback, and a pure policy function. All of it runs locally with no AWS account.

FraudDecision.cs
namespace Harbourline.Fraud;

public sealed record PaymentSignals(
    string CustomerId,
    decimal Amount,
    decimal AvailableBalance,
    bool PayeeIsNew,
    int PaymentsLastTenMinutes,
    bool DeviceIsNew,
    bool CountryChangedLastHour);

public sealed record Score(decimal Value, IReadOnlyList<string> Reasons, string Source);

public enum FraudAction { Allow, Challenge, Hold, Block }

public sealed record FraudDecision(FraudAction Action, decimal Risk, IReadOnlyList<string> Reasons, bool ModelUsed);

public interface IFraudScorer
{
    Task<Score> ScoreAsync(PaymentSignals signals, CancellationToken ct);
}

/// <summary>Always-available scoring in plain C#. Each rule adds to a capped score and records its reason code.</summary>
public sealed class RulesScorer : IFraudScorer
{
    public Task<Score> ScoreAsync(PaymentSignals s, CancellationToken ct)
    {
        decimal score = 0m;
        var reasons = new List<string>();

        if (s.PayeeIsNew && s.AvailableBalance > 0m && s.Amount / s.AvailableBalance >= 0.9m)
        {
            score += 0.45m;
            reasons.Add("NEW_PAYEE_HIGH_PERCENT_OF_BALANCE");
        }

        if (s.PaymentsLastTenMinutes >= 5)
        {
            score += 0.30m;
            reasons.Add("VELOCITY_5_IN_10_MIN");
        }

        if (s.DeviceIsNew && s.CountryChangedLastHour)
        {
            score += 0.35m;
            reasons.Add("NEW_DEVICE_AND_COUNTRY_CHANGE");
        }

        return Task.FromResult(new Score(Math.Min(score, 1m), reasons, "rules"));
    }
}

/// <summary>Calls a model scorer within a time slice; on timeout or error returns null so the caller can fall back.</summary>
public sealed class GuardedModelScorer(IFraudScorer model, TimeSpan timeout)
{
    public async Task<Score?> TryScoreAsync(PaymentSignals signals, CancellationToken ct)
    {
        using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
        cts.CancelAfter(timeout);
        try
        {
            return await model.ScoreAsync(signals, cts.Token);
        }
        catch (OperationCanceledException) when (!ct.IsCancellationRequested)
        {
            return null; // our timeout fired, not the caller's cancellation
        }
        catch (HttpRequestException)
        {
            return null;
        }
    }
}

public sealed record FraudPolicy(decimal ChallengeAt, decimal HoldAt, decimal BlockAt)
{
    public static FraudPolicy Default { get; } = new(0.30m, 0.60m, 0.85m);

    public FraudAction ActionFor(decimal risk) =>
        risk >= BlockAt ? FraudAction.Block
        : risk >= HoldAt ? FraudAction.Hold
        : risk >= ChallengeAt ? FraudAction.Challenge
        : FraudAction.Allow;
}

public sealed class FraudDecider(RulesScorer rules, GuardedModelScorer model, FraudPolicy policy)
{
    public async Task<FraudDecision> DecideAsync(PaymentSignals signals, CancellationToken ct)
    {
        var ruleScore = await rules.ScoreAsync(signals, ct);
        var modelScore = await model.TryScoreAsync(signals, ct);

        // The higher of the two wins: a model can raise risk but cannot hide a rule hit.
        var risk = Math.Max(ruleScore.Value, modelScore?.Value ?? 0m);
        var reasons = ruleScore.Reasons.Concat(modelScore?.Reasons ?? []).Distinct().ToList();
        if (modelScore is null)
        {
            reasons.Add("MODEL_UNAVAILABLE_RULES_ONLY");
        }

        return new FraudDecision(policy.ActionFor(risk), risk, reasons, ModelUsed: modelScore is not null);
    }
}
SageMakerScorer.cs
using System.Text;
using System.Text.Json;
using Amazon.SageMakerRuntime;
using Amazon.SageMakerRuntime.Model;

namespace Harbourline.Fraud;

/// <summary>Calls a SageMaker AI real-time endpoint. Only used when you choose to deploy one; the lab uses RulesScorer.</summary>
public sealed class SageMakerScorer(IAmazonSageMakerRuntime runtime, string endpointName) : IFraudScorer
{
    public async Task<Score> ScoreAsync(PaymentSignals s, CancellationToken ct)
    {
        // Send engineered features only: no names, no account numbers.
        var features = new
        {
            amount = s.Amount,
            percent_of_balance = s.AvailableBalance > 0 ? s.Amount / s.AvailableBalance : 1m,
            payee_is_new = s.PayeeIsNew ? 1 : 0,
            payments_10m = s.PaymentsLastTenMinutes,
            device_is_new = s.DeviceIsNew ? 1 : 0,
            country_changed_1h = s.CountryChangedLastHour ? 1 : 0,
        };

        var response = await runtime.InvokeEndpointAsync(new InvokeEndpointRequest
        {
            EndpointName = endpointName,
            ContentType = "application/json",
            Accept = "application/json",
            Body = new MemoryStream(Encoding.UTF8.GetBytes(JsonSerializer.Serialize(features))),
        }, ct);

        using var doc = await JsonDocument.ParseAsync(response.Body, cancellationToken: ct);
        var value = doc.RootElement.GetProperty("score").GetDecimal();
        return new Score(Math.Clamp(value, 0m, 1m), ["MODEL_SCORE"], $"sagemaker:{endpointName}");
    }
}

Code explained line by line

  1. PaymentSignals holds only the features the decision needs. It contains no names, addresses or account numbers, which limits personal data exposure to the model and to logs.
  2. RulesScorer adds to a capped score for each rule and records a reason code. Reason codes become the explanation shown to analysts and stored for audit.
  3. GuardedModelScorer links the caller's cancellation token with its own timeout. If the timeout fires and the caller did not cancel, the exception filter when (!ct.IsCancellationRequested) returns null. A real caller cancellation still propagates.
  4. FraudPolicy.ActionFor is a pure function from risk to action, with the thresholds in a record so they can be loaded from reviewed configuration.
  5. FraudDecider takes the maximum of the rule score and the model score. The model can raise risk, but a model that has drifted or is silent cannot hide a hard rule hit.
  6. When the model is unavailable the decision carries MODEL_UNAVAILABLE_RULES_ONLY and ModelUsed = false, so dashboards can show the fallback rate.
  7. SageMakerScorer serialises features to JSON, calls the endpoint through InvokeEndpointAsync and clamps the returned score to the range 0 to 1 so a bad model output cannot produce an out-of-range value.
  8. IAmazonSageMakerRuntime is injected, so a unit test can fake the endpoint with no AWS access.

Optional: an analyst summary with Bedrock

using Amazon.BedrockRuntime;
using Amazon.BedrockRuntime.Model;

public sealed class CaseSummariser(IAmazonBedrockRuntime bedrock, string modelId, string guardrailId, string guardrailVersion)
{
    public async Task<string> SummariseAsync(string maskedCaseFacts, CancellationToken ct)
    {
        var response = await bedrock.ConverseAsync(new ConverseRequest
        {
            ModelId = modelId,
            Messages =
            [
                new Message
                {
                    Role = ConversationRole.User,
                    Content = [new ContentBlock { Text =
                        "Summarise these fraud case facts in three short bullet points for an analyst. " +
                        "Use only the facts given. If something is unknown, say so.\n\n" + maskedCaseFacts }],
                },
            ],
            InferenceConfig = new InferenceConfiguration { MaxTokens = 300, Temperature = 0.2f },
            GuardrailConfig = new GuardrailConfiguration { GuardrailIdentifier = guardrailId, GuardrailVersion = guardrailVersion },
        }, ct);

        return response.Output.Message.Content[0].Text;
    }
}

Property names for the Converse API and for guardrails should be checked against the current AWSSDK.BedrockRuntime package ⚠ VERIFY. The model id is a placeholder you supply from the Bedrock console for a model enabled in your account.

Security considerations

⚠ Learning shortcut — not for production

The lab combines rules and a score with fixed numbers, and the thresholds are constants. Production needs: back-testing of every threshold change on historical data before release, a champion/challenger or shadow period for new models, governance and approval for model changes, protection of the rule set from insiders, bias and fairness testing, and a documented human-in-the-loop for automated decisions with significant effect ⚠ VERIFY under UK GDPR with your data protection officer.

  • Do not leak detection logic. Customer-facing messages are generic; reason codes stay internal.
  • Insider risk: limit who can change rules; use two-person review and log all changes.
  • Data minimisation: features for the model and the LLM are the smallest set that works; mask account numbers and names.
  • Prompt injection: free-text fields from customers (payment references) can contain instructions aimed at a language model. Treat them as untrusted data, exclude them from prompts when possible, and never let model output trigger actions.
  • Adversarial adaptation: attackers probe thresholds. Rate-limit and randomise challenges where policy allows ⚠ VERIFY with the fraud team.
  • Auditability: store the model version, features, thresholds and reason codes with every decision.
  • Fallback is a control: the behaviour when the model is down is a risk decision, approved and documented.

Banking use case

Harbourline's fraud team (fictional) sees authorised push payment scams: customers are tricked into paying a new payee.

  • A payment of GBP 4,800 from an account with GBP 5,000 to a payee created 2 minutes ago, from a new device, scores 0.80 through rules alone (new payee and high percentage plus new device); the policy puts it on Hold.
  • The saga creates a case and pauses. An analyst sees the facts and a Bedrock-generated summary, calls the customer on a known number, and confirms the customer is being coached by a scammer. The analyst releases a Block with a reason code APP_SCAM_CONFIRMED.
  • Outcome and features go to the lake as a label. Next month's retraining sees this pattern again.
  • During a model-endpoint outage, the fallback rate rises to 100%. The decision service keeps working on rules; policy tightens the challenge threshold for new payees during the fallback ⚠ VERIFY with the fraud team. The on-call engineer is paged on the fallback rate alarm.

Common mistakes

  • Letting the model decide alone → no explanation, no override → rules and policy sit around the model.
  • No timeout on the model call → one slow endpoint stalls every payment → timeout plus fallback.
  • Training on features computed after the fact → the model "sees the future" and fails live → store features as they were at decision time.
  • Treating the fallback as free → silent risk increase → alarm on fallback rate, decide stricter fallback policy.
  • Review queue without SLA timers → cases sit for days → escalation timers and dashboards.
  • Using an LLM to classify fraud → unstable and unauditable → LLM for summaries only.
  • Sending raw personal data to a model → data-protection exposure → engineered, minimised features.
  • Measuring only detection rate → customers drown in false positives → track both, with cost per case.

Troubleshooting

SymptomLikely causeFix
Fallback rate is 100%Endpoint down, wrong name, or IAM missing sagemaker:InvokeEndpointCheck endpoint status, role policy and the 25 ms timeout vs real latency
Latency spikes on decisionsModel cold start (serverless) or large payloadKeep payload tiny, use provisioned concurrency or an in-process model, raise timeout carefully
Many false positives after a rule changeThreshold or rule too broadRoll back; back-test against labelled history before change
Review queue messages end up in the DLQAnalyst tool failing to acknowledge, or poison messageInspect DLQ, fix the consumer, redrive (DLQ lesson)
Bedrock call returns an access errorModel access not enabled or role lacks permissionEnable the model in the console; grant bedrock:InvokeModel for that model
Velocity counters wrong after scale-outCounters kept in local memoryUse DynamoDB atomic counters or ElastiCache with expiry
Model performs worse each monthData drift, new fraud patternsMonitor feature distributions and outcomes with CloudWatch and open-source tooling; retrain
SchemaException from the endpointFeature names or order changedVersion the feature contract with the model

Cost and cleanup

Label: 🟡 credits. The C# rules, policy and tests are free and run locally. A SageMaker real-time endpoint bills per hour while it exists, so for a learning lab prefer the local scorer. SQS and DynamoDB at lab volumes cost cents or fall within free allowances ⚠ VERIFY. Bedrock charges per token ⚠ VERIFY; ten fictional prompts cost very little. Free alternative: do the design and run the local code only.

# Delete the review queues and the case table.
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m14-fraud-review --query QueueUrl --output text --profile learn-sandbox --region us-east-1)" --profile learn-sandbox --region us-east-1
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m14-fraud-review-dlq --query QueueUrl --output text --profile learn-sandbox --region us-east-1)" --profile learn-sandbox --region us-east-1
aws dynamodb delete-table --table-name hl-m14-fraud-cases --profile learn-sandbox --region us-east-1

# If you created a SageMaker endpoint, delete it AND its config and model.
aws sagemaker delete-endpoint --endpoint-name <name> --profile learn-sandbox --region us-east-1

# Confirm nothing tagged for the course remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --profile learn-sandbox --region us-east-1

Exercises

  • 🟢 Guided: write tests for RulesScorer and FraudPolicy: a payment that triggers no rule is Allow; "new payee plus 95% of balance" scores 0.45 and is Challenge; the same plus a new device in a new country (total 0.80) is Hold; risk 0.85 exactly is Block.
  • 🟡 Challenge: write a fake IFraudScorer that delays 200 ms and a test showing GuardedModelScorer with a 25 ms timeout returns null, and that FraudDecider then reports ModelUsed = false with the MODEL_UNAVAILABLE_RULES_ONLY reason.
  • 🔴 Live problem: the fraud team asks to lower the Hold threshold from 0.60 to 0.50 before the holiday season. Design the safe change process: what data you back-test on, what metrics you compare, the shadow period, the rollback, the capacity impact on the review queue and who must approve.

Assignment

Write the 15-section fraud integration architecture in architecture/fraud-integration/: the layered diagram, the policy table with owners, the decision-service contract with reason codes, the review-queue design (state, SLA, escalation), the fallback ADR, the feedback loop and monitoring plan, and the compliance touchpoints (model risk, data protection and automated decisions, fraud operations, third-party risk for any vendor). Include the local C# scorer and its tests.

Solutions

Knowledge check

Interview questions

  • How would you add fraud scoring to a payment flow? A decision service called before the hold step with a tight budget, layered rules plus an optional model with timeout and fallback, a policy mapping risk to allow, challenge, hold or block, reason codes stored for audit, and a review queue for holds.
  • What if the ML endpoint is down? Fall back to rules with a documented, possibly stricter, policy; alarm on fallback rate; keep payments flowing within risk appetite.
  • How do you avoid alert fatigue? Measure false positives, tune thresholds by back-testing, route by risk tier, and give analysts good evidence and summaries.
  • How do you keep a model trustworthy over time? Monitor feature and outcome drift, retrain on labelled outcomes, use shadow and canary deployment, and keep model-risk governance and approvals.
  • Where would you use generative AI in fraud operations? For summarising cases and drafting customer contact notes for an analyst, with masked data, guardrails, logging and a human decision. Not for the allow or block decision.

Certification practice

Completion checklist

  • I can explain signal versus decision and why they are separate
  • I wrote the policy table with thresholds and owners
  • I ran or wrote tests for the rules, policy and fallback
  • I can explain how the review queue replaces A2I
  • I limited Bedrock to analyst assistance with masked inputs
  • I deleted anything I created and confirmed nothing tagged Project=cea-lab remains

Key takeaways

  • Fraud detection is layers: rules, scores, humans and a learning loop, with policy deciding the action.
  • Always give the model call a short timeout and a defined fallback; make the fallback a monitored, approved control.
  • Closed services (Fraud Detector, A2I, Ground Truth) are replaced by Kinesis, Lambda, SageMaker AI, SQS and Step Functions.
  • Language models help analysts; they do not make control decisions.
  • Store reason codes, versions and features for every decision, and feed outcomes back as labels.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page