Skip to Content
Cloud Engineering Academy
16. Generative AI on AWS for Banking

GenAI in plain language

Explain foundation models, tokens, context windows, embeddings, RAG and agents in plain words, and say why a model's answer can be confidently wrong.

beginnerโฑ 1 h ๐ŸŸข FreeAIF-C01AIP-C01

Assumes: .NET 10 LTS ยท C# only, no AWS calls in this lesson ยท model facts checked against the Amazon Bedrock model cards on 2026-11-18. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain what a foundation model, a token, a context window and a temperature setting are, without jargon.
  • Describe what an embedding is and compare two pieces of text by cosine similarity in C#.
  • Draw the difference between a plain prompt, retrieval-augmented generation (RAG) and an agent.
  • List at least five reasons a model's answer can be wrong, and say which ones better prompting cannot fix.
  • Sort a handful of banking use cases into good fit, controlled fit and not without strong controls.

Prerequisites

  • Module 16 overview: why a bank must answer hard questions before it adds AI.
  • Comfort reading C#. This lesson makes no AWS calls, so you need no AWS account, SDK or credentials.
  • Tools: .NET 10 SDK (or Docker with the mcr.microsoft.com/dotnet/sdk:10.0 image).

Why this matters in banking

A product manager at Harbourline Bank asks you to "put a chatbot on the policy wiki". Two weeks later a support agent pastes a customer's full IBAN and email address into it, asks whether a disputed payment can be refunded, and acts on the confident answer. The answer was fluent, polite and wrong: the model had never seen the current dispute policy and filled the gap with something plausible.

None of that is an AWS problem. It is a misunderstanding of what a generative model is. If you know what the machine is actually doing, you can predict where it will fail, and you can design the controls (redaction, retrieval, guardrails, human review) before an incident, not after. That is the whole point of this module.

The idea in plain language

A large language model (LLM) is a program that has read a very large amount of text and learned to predict what text is likely to come next. You give it some text (the prompt); it writes a continuation, one small piece at a time. When the prompt looks like a question and the model has seen many question-and-answer pairs, the continuation looks like an answer.

Three consequences follow, and everything else in this module hangs off them:

  1. It is fluent, not factual. It optimises for text that sounds right. Whether the text is true depends on whether the truth was common in its training data and present in your prompt.
  2. It has no memory of your bank. It does not know your policies, your customers or yesterday's incident unless you put that text in the prompt.
  3. It does not run the same way twice. By default there is randomness in how it picks each next piece.

Analogy: a foundation model is a very fast, very well-read new graduate. Brilliant at drafting and summarising, and happy to bluff when it does not know. You would never let a new graduate approve a loan or move money unsupervised, and you would give them the policy manual before asking a policy question. Retrieval is the manual; human review is the supervisor.

The vocabulary you need

TermPlain meaningWhy you care
Foundation model (FM)A large model trained on broad data that can be reused for many tasksYou rent one through a service such as Amazon Bedrock rather than train your own
TokenA chunk of text, roughly three-quarters of an English word. Roughly four characters on average for EnglishModels read and write tokens, and you are billed per token
PromptEverything you send: instructions, retrieved documents, the user's questionIt is the only way to give the model facts and rules
System promptStanding instructions ("you are a policy assistant, answer only from the documents")Your first, weak, line of defence
Context windowThe most tokens the model can consider at once (prompt plus answer)The two Claude 5.5 models on Amazon Bedrock list a 1 million token window on their model cards, and many models are far smaller
TemperatureA knob for randomness. Low means more repeatable, high means more variedUse low for policy answers and classification
EmbeddingA list of numbers that captures the meaning of a piece of textIt lets a computer find text with similar meaning
HallucinationA confident statement that is not supported by factsThe main reason outputs need checking
InferenceRunning the trained model to get an answerWhat Bedrock charges you for

Tokens, in numbers you can feel

You do not need an exact tokenizer to plan. A useful rule of thumb for English is about four characters per token, so a 12-word question is around 13 to 16 tokens and a one-page policy is around 500 to 700 tokens. Numbers, code, IBANs and non-English text use more tokens per character. The rule is only for estimating; the real count comes back in the API response (you will use it in Calling Bedrock from C#).

The context window matters for two reasons. First, cost and latency grow with prompt size: sending a 200-page document with every question is slow and expensive. Second, a model can lose track of details in the middle of a very long prompt, so "just put everything in" is not a retrieval strategy.

Embeddings and similarity

An embedding model turns text into a list of numbers (a vector), typically a few hundred to a thousand numbers. The training makes texts with similar meaning land close together. To find the most relevant policy paragraph for a question, you embed the question, embed every paragraph once, and pick the paragraphs whose vectors point in the most similar direction. The usual measure is cosine similarity: 1 means the same direction, 0 means unrelated.

Here is the idea in runnable C#. It uses a toy "embedding" (word counts) so you can see the mechanics without any service. Real embeddings come from a neural network and understand synonyms; the comparison maths is identical.

Program.cs
using System.Text.RegularExpressions;

string[] policies =
[
    "A customer may raise a card dispute within 120 days of the booking date.",
    "A current account can be closed once the balance is zero and no payments are pending.",
    "Branches open from 09:00 to 17:00 on weekdays.",
];

const string question = "How many days do I have to dispute a card payment?";

// 1. A rough token estimate: English text averages about four characters per token.
static int EstimateTokens(string text) => (text.Length + 3) / 4;
Console.WriteLine($"The question is roughly {EstimateTokens(question)} tokens.");

// 2. A toy "embedding": how often each word appears. Real embeddings are 256 to 1,024 numbers produced by a
//    neural network, but comparing them uses exactly the same maths as below.
static Dictionary<string, double> Embed(string text) =>
    Regex.Matches(text.ToLowerInvariant(), "[a-z0-9]+")
        .Select(m => m.Value)
        .Where(word => word.Length > 3)
        .GroupBy(word => word)
        .ToDictionary(g => g.Key, g => (double)g.Count());

static double Cosine(Dictionary<string, double> a, Dictionary<string, double> b)
{
    var dot = a.Where(kv => b.ContainsKey(kv.Key)).Sum(kv => kv.Value * b[kv.Key]);
    var lengthA = Math.Sqrt(a.Values.Sum(v => v * v));
    var lengthB = Math.Sqrt(b.Values.Sum(v => v * v));
    return lengthA == 0 || lengthB == 0 ? 0 : dot / (lengthA * lengthB);
}

var query = Embed(question);
foreach (var (text, score) in policies
             .Select(p => (p, Cosine(query, Embed(p))))
             .OrderByDescending(x => x.Item2))
{
    Console.WriteLine($"{score:0.00}  {text}");
}

Expected output:

The question is roughly 13 tokens.
0.43  A customer may raise a card dispute within 120 days of the booking date.
0.00  A current account can be closed once the balance is zero and no payments are pending.
0.00  Branches open from 09:00 to 17:00 on weekdays.

The toy finds the right policy because the question and the policy share the words "dispute", "days", "card" and "payment". Ask it "How long do I have to raise a chargeback?" and it scores 0.00 for everything, because "chargeback" shares no word with the policy. A real embedding model would still rank the dispute policy first. That gap between matching words and matching meaning is why retrieval uses embeddings.

What problem the AWS service solves

This lesson is about concepts, but you should know where each concept lands on AWS so the rest of the module has a map. Amazon Bedrock is the managed service that gives you foundation models behind one API, without you running GPUs. The pieces:

  • Plain prompt: you call a model with text and get text back (the Converse API).
  • Retrieval-augmented generation (RAG): before calling the model, you search your own approved documents and paste the best passages into the prompt. The model answers from them and can cite them.
  • Agent: a loop where the model decides which tool to call (look up an account, search a wiki), your code runs the tool, and the result goes back to the model until the task is done.

The more autonomy you move to the right, the more you gain and the more can go wrong. A plain prompt can only say something wrong. A RAG assistant can say something wrong about a real document. An agent can do something wrong. Each step needs stronger controls, which is why agents get tight limits in a bank.

Prompting, RAG or fine-tuning?

People often ask whether to "train the model on our data". For most bank assistants the answer is no.

ApproachWhat it changesUse whenBanking caution
Better promptingThe instructions and examples in the promptFormat, tone, simple rulesCheap and reversible: try first
RAGAdds fresh, approved facts to the prompt at question timeAnswers must reflect current policy, with citationsYou must control who may see which source document
Fine-tuningAdjusts the model's weights with your examplesA stable style or narrow task, lots of good examplesBakes data into the model; hard to delete under UK GDPR erasure
Training from scratchEverythingAlmost neverOut of scope for a bank's first projects

Why answers go wrong

CauseWhat it looks likeDoes a better prompt fix it?
HallucinationA made-up policy number or regulation, stated confidentlyPartly: "answer only from the documents" helps, retrieval helps more
Stale knowledgeDescribes a rule that changed after the training cutoff (the two Claude 5.5 models list a knowledge cutoff of June 2026)No: add current documents through RAG
Missing contextThe answer needs a fact you never suppliedNo: retrieve it or ask the user
Retrieval failureThe right paragraph was not found, so the model improvisesNo: fix chunking, search and evaluation
Ambiguity"Refund" means three things in three productsPartly
Non-determinismTwo runs give different answersLower temperature narrows it; it never reaches zero
Prompt injectionText in a document or email says "ignore your instructions" and the model compliesNever fully; needs layered defences
BiasDifferent quality or tone for different groups of customersNo: it needs testing and governance
Over-trust by humansThe reader believes it because it sounds sureNo: it is a process and training problem

Notice how few of these are cured by writing a cleverer prompt. Most need a control around the model, which is the engineering this module teaches.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure OpenAI Service / Azure AI FoundryAmazon BedrockMany providers behind one set of APIs; you choose the model per call
Azure.AI.OpenAI chat completionsBedrock Converse APIOne request shape across providers, so swapping a model is a string change
Azure AI Search with vector fieldsBedrock Knowledge BasesManaged ingestion, chunking and retrieval (next lessons)
Azure AI Content SafetyBedrock GuardrailsFilters on both the prompt and the response, plus PII and grounding checks
Microsoft.Extensions.AI (IChatClient)Same package, with a Bedrock adapterKeep your own code provider-neutral
GitHub CopilotKiro and the Amazon Q Developer console featuresCovered in a later lesson

Architecture

Where does a generative feature sit in a bank's estate? Always behind your own API, never called directly from a browser, and always with a person or a rule between the model and anything that affects a customer.

Security-boundary view: the user, your API and the retrieval store sit inside your trust boundary. The model call crosses into a managed service, so what you send is a data-sharing decision. You will practise every control in that diagram in the lessons and the lab.

Step-by-step implementation

This lesson has one hands-on exercise: run the cosine-similarity program and extend it.

Create the console app

dotnet new console -n TinyEmbeddings -f net10.0
cd TinyEmbeddings

Replace Program.cs with the code above.

Run it

dotnet run

You should see the output shown earlier. If you use Docker instead of a local SDK, mount the folder and run dotnet run inside mcr.microsoft.com/dotnet/sdk:10.0.

Break it on purpose

Change the question to use a synonym (for example "chargeback") and watch every score drop to 0.00. Write down in one sentence why a real embedding model would not have that problem.

AWS Console instructions

None needed. You will meet the Bedrock console in Amazon Bedrock basics.

AWS CLI

None in this lesson. A first command you can try later, once you have a profile, lists the foundation models your Region offers:

# Lists models by provider; needs the learn-sandbox SSO profile and bedrock:ListFoundationModels
aws bedrock list-foundation-models --by-provider anthropic --profile learn-sandbox --region us-east-1

Infrastructure as Code

Not applicable: this lesson creates no AWS resources.

Complete code

The full program is the listing above; it needs only the .NET 10 SDK. The lab for this module, Lab 16.1, is the complete, tested codebase.

Code explained line by line

  1. static int EstimateTokens(string text) => (text.Length + 3) / 4; rounds characters divided by four up. It is an estimate for planning only.
  2. Embed lower-cases the text, splits it into words with a regular expression, drops words of three letters or fewer, and counts the rest. The result is a sparse vector: a dictionary of word to count.
  3. Cosine computes the dot product over the words the two vectors share, divided by the product of their lengths. A score of 1 means identical direction.
  4. The final loop scores every policy against the question and prints them best first. This is the retrieval step of RAG in miniature: embed, compare, sort.

Security considerations

โš  Learning shortcut โ€” not for production

The toy embedding in this lesson matches words, not meaning, and it runs on hard-coded sample text. A production system uses a real embedding model, retrieves only from approved documents the user is allowed to see, and removes personal data before anything is sent to a model.

Three habits to start now:

  • Treat every prompt as an outbound data transfer. Anything you paste into it leaves your process. Decide what may leave before you write the code, not after.
  • Treat model output as untrusted input. Never execute it, never put it in a SQL string, never render it as HTML without encoding.
  • Never put secrets, full card numbers or customer identifiers in a prompt unless a data-protection assessment says you may. Your compliance and legal teams own that decision; this course does not claim any approach is compliant.

Banking use case

Sort these fictional Harbourline Bank requests. The categories are the same ones you will use in the last lesson of the module.

RequestFitWhy
Search and summarise the approved card-dispute policy for a call-centre agentGood fitInternal, read-only, cites sources, a person decides
Draft a first version of an internal release note from a change logGood fitLow impact, easy to review
Summarise a customer complaint email for a case handlerControlled fitContains personal data: redact first, log, review
Suggest a reply to a customer for an agent to edit and sendControlled fitA human approves every message
Decide whether to approve an overdraftNot without strong controlsA decision about a customer; needs governance, explainability and a qualified human
Let the assistant move money when a customer asks in chatNot without strong controlsAn autonomous action with financial impact

Common mistakes

  • "The model knows our policies." It does not. Why it happens: the answer sounds authoritative. Avoid it by retrieving the document and citing it.
  • "Temperature zero means deterministic." It narrows variation, but you can still see differences. Test for stability rather than assume it.
  • "A longer system prompt is a security control." A system prompt is an instruction, not an enforcement point. Treat it as one layer.
  • "We will fine-tune on customer data to make it smarter." That bakes personal data into model weights. Start with RAG over approved documents.
  • Counting characters as tokens. Cost and limits are in tokens; estimate with the four-characters rule and read the real counts from responses.

Troubleshooting

SymptomLikely causeFix
dotnet: command not found.NET SDK not installed locallyUse the Docker image mcr.microsoft.com/dotnet/sdk:10.0
Every score is 0.00The question shares no words longer than three letters with the policiesUse words from the policy, or move to real embeddings in the next lessons
error CS8803 about top-level statementsAnother file in the folder also has top-level codeKeep one Program.cs with top-level statements per project
Output order differs from this pageYou changed a policy or the questionExpected; rerun and read the scores

Cost and cleanup

This lesson costs nothing: it runs locally and calls no AWS service. There is nothing to tear down. Delete the TinyEmbeddings folder if you wish.

Exercises

  • ๐ŸŸข Guided: run the program, then add a fourth policy sentence about cancelling a standing order. Confirm that a question about standing orders now ranks it first.
  • ๐ŸŸก Challenge: replace the word-count vectors with word-presence vectors (1 or 0) and add simple stemming (strip a trailing "s"). Does "disputes" now match "dispute"? What new mistakes does stemming introduce?
  • ๐Ÿ”ด Live problem: a colleague says, "Let's fine-tune a model on two years of call transcripts so it answers like our best agents." Write five questions you would ask before agreeing, covering personal data, erasure requests, consent, evaluation and who owns the model risk.

Assignment

Write a one-page "GenAI glossary for my team" in your portfolio repository. For each of the nine terms in the vocabulary table, give a one-sentence definition and one banking example. Add a final section titled "Five ways this goes wrong" using the table above, with one fictional Harbourline example per row.

Solutions

Knowledge check

Interview questions

  • What is the difference between RAG and fine-tuning, and which would you start with for a policy assistant? RAG supplies current approved text at question time and can cite it; fine-tuning changes weights. Start with RAG because it is easier to update, audit and delete.
  • Why can a model sound certain and be wrong? It is trained to produce likely text, not verified text. Confidence of tone carries no information about truth. You add grounding, citations and review.
  • How would you explain a context window to a product manager? It is the model's working desk. Only what fits on the desk can be considered, and a bigger desk costs more and is slower to search.
  • What would make you refuse a GenAI use case at a bank? An autonomous decision or money movement without a qualified human, personal data leaving without a lawful basis, or no way to evaluate and monitor quality.

Certification practice

Completion checklist

  • I can explain tokens, context windows and embeddings to a colleague in two sentences each
  • I ran the cosine-similarity program and broke it with a synonym
  • I can name five ways an answer goes wrong and which ones prompts cannot fix
  • I sorted the banking use cases into good, controlled and not-without-controls
  • I understand that this lesson created nothing in AWS, so there is nothing to clean up

Key takeaways

  • A language model predicts likely text; fluent is not the same as factual.
  • It knows nothing about your bank unless you put it in the prompt: that is what retrieval is for.
  • Most failure causes are cured by controls around the model, not by cleverer wording.
  • Autonomy raises both value and risk: plain prompt, then RAG, then agent, each needing stronger controls.
  • Start every design with the question "what leaves my process, and who checks the answer?"

Further reading

Independent learning resource โ€” not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page