Event-driven banking architecture
Design domain events, per-domain EventBridge buses, schema governance and safe replay for a bank, keep personal data out of events, and check event contracts and duplicate delivery with a runnable C# validator.
Assumes: .NET 10 LTS (file-based apps) · AWS SDK for .NET v4 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Distinguish a domain event, a command and an integration event, and say which belongs on a shared bus.
- Design a bus topology (one bus per domain, a central audit bus, cross-account access) and justify it.
- Define an event envelope and a schema-governance process with versioning and compatibility rules.
- Keep personal data out of events, and explain how that supports erasure, retention and replay.
- Choose between EventBridge, SNS, SQS, Kinesis Data Streams and Amazon MSK for a given requirement.
- Make consumers safe under duplicates, reordering and replay, and run a C# envelope validator with a duplicate filter.
Prerequisites
- Asynchronous event-driven architecture and Delivery guarantees and ordering
- Outbox pattern with EF Core and Dead-letter queues and redrive
- API orchestration for the orchestration versus choreography decision
- UK GDPR and data residency
- Tools: .NET 10 SDK or Docker for the envelope validator
Why this matters in banking
A bank is a network of domains: customers, accounts, payments, cards, fraud, notifications, reporting. When a payment settles, many parties care: the ledger, the customer's app, the fraud team's analytics, the data lake, the regulatory-reporting pipeline. If the payments service calls each one directly, every new consumer means changing payments, and every consumer outage slows payments down.
Events invert that. Payments publishes "payment settled" once. Interested domains subscribe. But events also create new risks that a bank must design out:
- A careless event carries a customer's email address, and now personal data is copied into six systems and an archive that cannot be erased.
- Producers change a field name and break a consumer that nobody knew existed.
- A replay re-sends 10,000 old events and customers receive 10,000 duplicate texts.
- Two events arrive in the wrong order and an account appears closed before it was opened.
An event-driven architecture in banking is therefore as much about contracts and governance as it is about services.
Educational content, not legal advice
How personal data in events interacts with erasure rights, retention duties and record-keeping obligations is a legal and data-protection question ⚠ VERIFY with your data-protection officer and legal team. This lesson shows engineering patterns that reduce the problem; it does not claim that any design satisfies UK GDPR or any other regime.
The idea in plain language
An event is a fact about something that already happened, named in the past tense: payment.settled, account.frozen. It is immutable. A command is a request: settle this payment. Commands go to one owner; events go to anyone who is interested.
| Kind | Example | Audience | Where it travels |
|---|---|---|---|
| Command | ReserveFunds | One service | Direct call, queue |
| Domain event | PaymentSettled | Inside the owning domain | Internal bus or stream |
| Integration event | payments.payment.settled v1 | Other domains | Shared bus, with a published schema |
Analogy: a branch notice board versus a phone call. A command is a phone call to a named person. An event is a dated notice pinned to the board: "Account ACC-... was frozen at 10:02". Anyone may read it, nobody can edit it, and the board has rules about what may be written on a public notice (never a customer's home address).
Notification events versus state-carrying events
| Style | Payload | Pros | Cons |
|---|---|---|---|
| Notification (thin) | Identifiers and type, such as paymentId, customerId | Small, little personal data, always current when the consumer looks up | Consumer must call back, so the producer sees extra load |
| Event-carried state (fat) | Many fields | Consumers independent of the producer | More copied data, harder to retire fields, higher privacy risk |
Banking default: thin events plus a small set of non-personal facts (amount, currency, status, timestamps). Consumers that need more fetch it through an authorised API.
What problem the AWS services solve
| Need | Service | Honest limits |
|---|---|---|
| Route events by content to many targets | Amazon EventBridge event bus and rules | At-least-once delivery, no ordering guarantee; payload size limits (about 256 KB per entry historically; some limits were raised recently ⚠ VERIFY) |
| Cross-account and cross-Region routing | EventBridge bus targets and bus resource policies | Policy design matters; one misconfigured policy opens a bus |
| Contract definition | EventBridge Schema Registry (JSON Schema or OpenAPI) | A registry stores schemas; it does not enforce them at publish time unless you add a check |
| Replay | EventBridge archives and replays | Replays re-trigger every matching rule, including side-effect consumers |
| Fan-out to queues and topics | SQS, SNS | SNS adds pub/sub with filtering; SQS gives a durable buffer per consumer |
| Ordered, high-throughput streams | Kinesis Data Streams, Amazon MSK | Shard or partition management; ordering within a shard or partition only |
| Point-to-point source to target | EventBridge Pipes | Good for filtering and enrichment between two services |
Choosing the transport
| Requirement | Likely choice |
|---|---|
| Many independent consumers, content routing, SaaS and AWS integrations | EventBridge bus |
| One consumer, work queue, back-pressure, retries | SQS |
| Simple topic fan-out with filtering and mobile or email delivery | SNS |
| Ordered per key, replay from an offset, very high volume | Kinesis Data Streams or MSK |
| Kafka-compatible tooling already in use | Amazon MSK |
Many banks use EventBridge for business events and Kinesis or MSK for high-volume streams such as card authorisation telemetry (real-time processing).
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Event Grid | EventBridge | Both route events by filters; EventBridge adds archives, replay and a schema registry |
| Azure Service Bus topics and subscriptions | SNS with SQS subscriptions | Pub/sub with durable per-consumer queues |
| Azure Event Hubs | Kinesis Data Streams, MSK | Partitioned, ordered per partition, replay by offset |
| Azure Schema Registry | EventBridge Schema Registry, AWS Glue Schema Registry | Glue Schema Registry targets Kafka/Kinesis serialisation formats |
| MassTransit or NServiceBus message contracts | Shared contract library plus schema registry | Same discipline; AWS gives you a hosted catalogue |
| CloudEvents spec | Same spec idea; EventBridge has its own envelope | Map between them deliberately |
Architecture
Security-boundary view
A producer can publish only to its own bus and only with its own source. Consumers get only the events their rules select, delivered into queues they own.
The 15-section design
Sections 1 to 3: requirements
| Section | Content |
|---|---|
| 1 Business | Let domains react to each other without direct coupling; give the data and reporting teams one governed feed; make onboarding a new consumer a matter of days, not a change request to the producer |
| 2 Functional | F1 Publish integration events from each domain reliably (outbox). F2 Route by type to consumers. F3 Version and publish schemas. F4 Archive and replay for recovery. F5 Reject events that break the contract in CI. F6 Provide an audit copy of all events |
| 3 Non-functional | End-to-end p99 delivery under 2 seconds for business events. 99.95% publish availability. At-least-once delivery with consumer idempotency. Throughput 500 events per second sustained, 3,000 burst (⚠ check quotas). Archive 30 days. Audit copy retained 7 years ⚠ VERIFY. No personal data in event payloads |
Section 5: service selection
| Choice | Why | Rejected alternative | Trade-off |
|---|---|---|---|
| One EventBridge bus per domain | Isolated policies, clear ownership, per-domain quotas and archives | One bus for the bank | More buses to manage; far less blast radius |
| Central audit bus in a log-archive account | One tamper-resistant copy | Each domain keeps its own history only | Another hop and bus to run |
| SQS per consumer behind rules | Buffering, DLQs, back-pressure | Lambda targets directly | Slightly more resources; much better failure behaviour |
| Outbox for publishing | Atomic with the business transaction | Publish after commit in code | Relay component to run (outbox pattern) |
| Kinesis for high-volume telemetry | Ordered per key, replay by position | EventBridge for everything | Capacity planning, separate from the business bus |
Sections 6 and 7: network and security
- Network. EventBridge, SQS and SNS are regional service endpoints. Private subnets reach them through interface VPC endpoints (VPC endpoints).
- Bus policies. Allow
events:PutEventsonly for principals in your organisation (aws:PrincipalOrgID) and, where possible, restrict thesourcefield with theevents:sourcecondition key ⚠ VERIFY the key and syntax. - Encryption. Customer-managed KMS keys for buses and archives where supported ⚠ VERIFY; SQS with KMS or SQS-managed encryption.
- No personal data in events. Use
customerId,accountIdandpaymentIdas opaque IDs; amounts and currencies are fine. A consumer needing a name or address calls an authorised API. - Schema rules in CI. Block a pull request if an event schema change is breaking.
- Audit. CloudTrail records bus management calls; the audit bus keeps the event stream itself (audit and compliance architecture).
Sections 8 to 10: API, data and flow
Envelope. Every integration event uses the same envelope:
| Field | Meaning |
|---|---|
eventId | Globally unique (UUID v7); the deduplication key |
type | domain.entity.event, for example payments.payment.settled |
schemaVersion | Integer; consumers declare which versions they handle |
source | Producing domain, for example harbourline.payments |
occurredAt | When the fact happened (UTC) |
correlationId | Ties the event to the originating request |
data | Non-personal facts and opaque IDs |
Schema governance.
| Change | Allowed without a new version? |
|---|---|
| Add an optional field | Yes (backward compatible) |
| Add a new enum value | Only if consumers are told to tolerate unknown values; otherwise treat as breaking |
| Remove or rename a field | No: new schemaVersion; run old and new in parallel until consumers move |
| Change a field's type or meaning | No: new version |
| Add a required field | No: new version |
Flow. (1) Service commits state and an outbox row. (2) Relay publishes to the domain bus. (3) Rules deliver to consumer queues. (4) Consumers validate, deduplicate by eventId, process, and delete the message. (5) The archive and audit bus keep copies.
Section 11: failure scenarios
| Failure | Detection | Automatic response | Impact | Manual step |
|---|---|---|---|---|
| Duplicate delivery | eventId already processed | Skip | None | None |
| Out-of-order events | occurredAt or version older than stored state | Ignore stale or hold briefly | None | None |
| Consumer down | Queue age alarm | Events buffer in SQS | Delayed reaction | Fix consumer |
| Poison event | Repeated failure | Move to DLQ | One event unprocessed | Fix and redrive |
| Breaking schema change | CI contract test fails | Block merge | None in production | Version the schema |
| Replay causes side effects | Replay planned | Consumers are idempotent; side-effect consumers filter on replay-name or a flag | Avoided | Plan replay with owners |
| Rule target throttled or unreachable | Failed invocation metric | Retries per target retry policy, then target DLQ | Delay | Investigate |
| Wrong bus policy | Access Analyzer finding | None; detection | Possible exposure | Fix policy |
Sections 12 to 15
- Scalability. Add consumers without touching producers. Watch EventBridge
PutEventsand invocation quotas in your Region ⚠ VERIFY, and use Kinesis or MSK for sustained very high rates. - Cost. EventBridge bills per event published and per event replayed or archived ⚠ VERIFY; SQS per request; archive storage per GB. Estimate events per day by type, number of rules and targets. Thin events lower cost as well as risk.
- Implementation. Phase 1 envelope and schema repo. Phase 2 one domain bus with outbox relay. Phase 3 two consumers behind SQS. Phase 4 archive, replay rehearsal, audit bus. Phase 5 contract tests in every producer's pipeline.
- Compliance touchpoints.
| Area | Where it touches | Reviewing team | Evidence |
|---|---|---|---|
| Data protection and erasure | What fields events may carry; archive retention | Data protection officer | Field catalogue, archive policy ⚠ VERIFY |
| Record keeping | Audit bus retention | Compliance, legal | Retention schedule |
| Operational resilience | Dependency of important services on the bus | Operational resilience lead | Scenario tests |
| Third-party risk | Any SaaS event sources or targets | Third-party risk | Assessment |
| Access governance | Bus policies, who may publish | Security | Policy review |
Step-by-step implementation
Define the envelope and the forbidden-fields list
Agree the seven envelope fields and a list of banned payload names (email, phone, names, dates of birth, IBAN). Put both in a shared repository that producers and consumers reference.
Create one bus per domain with an archive
Name them hl-payments, hl-accounts and so on. Enable an archive with a retention you can justify. Set the bus policy for your organisation.
Publish through an outbox
Write the event to an outbox table in the same transaction as the state change. A relay reads and calls PutEvents, and marks rows sent. Never publish directly after commit and hope.
Put an SQS queue between each rule and its consumer
Give every queue a DLQ and an age alarm. The consumer validates the envelope, checks eventId against a dedupe table, processes, then deletes the message.
Add schema checks to CI
Validate sample events against the registry schemas and fail on breaking changes. Keep the schema repository the source of truth, and publish to the registry from the pipeline.
Rehearse a replay
Replay one hour of archive events into a test consumer and confirm that idempotent consumers do nothing new and side-effect consumers are protected.
AWS Console instructions
- Console → Amazon EventBridge → Buses → Event buses → Create event bus. Name it, set the resource policy, optionally enable Archive.
- EventBridge → Rules → Create rule on that bus. Use an event pattern with
sourceanddetail-type, choose an SQS target, and configure Dead-letter queue and Retry policy under additional settings. - EventBridge → Schema registry → Registries / Discovered schemas to browse schemas, view versions and download code bindings where available.
- EventBridge → Archives and Replays to start a replay for a time range.
AWS CLI
# Create a domain bus.
aws events create-event-bus --name hl-m14-payments \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m14-l14 Key=Environment,Value=learn Key=Owner,Value=learner \
--profile learn-sandbox --region us-east-1
# Archive everything from the payments source for 30 days.
aws events create-archive --archive-name hl-m14-payments-archive \
--event-source-arn arn:aws:events:us-east-1:111122223333:event-bus/hl-m14-payments \
--retention-days 30 \
--event-pattern '{"source":["harbourline.payments"]}' \
--profile learn-sandbox --region us-east-1
# Publish one event (note: Detail is a JSON string; no personal data).
aws events put-events --entries '[{
"EventBusName":"hl-m14-payments",
"Source":"harbourline.payments",
"DetailType":"payments.payment.settled",
"Detail":"{\"eventId\":\"0198a1b2-0000-7000-8000-000000000001\",\"schemaVersion\":1,\"correlationId\":\"corr-123\",\"data\":{\"paymentId\":\"PAY-0198a1b2\",\"amount\":\"25.00\",\"currency\":\"GBP\"}}"
}]' --profile learn-sandbox --region us-east-1
# Replay a time range from the archive back to the bus.
aws events start-replay --replay-name hl-m14-replay-1 \
--event-source-arn arn:aws:events:us-east-1:111122223333:archive/hl-m14-payments-archive \
--destination Arn=arn:aws:events:us-east-1:111122223333:event-bus/hl-m14-payments \
--event-start-time 2026-11-18T09:00:00Z --event-end-time 2026-11-18T10:00:00Z \
--profile learn-sandbox --region us-east-1Account IDs are fictional. In a real event, eventId, type, source and occurredAt sit in the envelope your producers add; EventBridge itself supplies source, detail-type and time, so decide which fields you duplicate in detail ⚠ VERIFY your convention.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.EventSchemas;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.Events.Targets;
using Amazon.CDK.AWS.SQS;
using Constructs;
using EventBus = Amazon.CDK.AWS.Events.EventBus;
using EventBusProps = Amazon.CDK.AWS.Events.EventBusProps;
namespace Harbourline.Architecture.Events;
public class DomainEventBusStack : Stack
{
public DomainEventBusStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
var bus = new EventBus(this, "PaymentsBus", new EventBusProps { EventBusName = "hl-m14-payments" });
bus.Archive("PaymentsArchive", new BaseArchiveProps
{
ArchiveName = "hl-m14-payments-archive",
Description = "30-day archive for replay",
EventPattern = new EventPattern { Source = ["harbourline.payments"] },
Retention = Duration.Days(30),
});
var dlq = new Queue(this, "LedgerConsumerDlq", new QueueProps
{
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
RetentionPeriod = Duration.Days(14),
});
var ledgerQueue = new Queue(this, "LedgerConsumerQueue", new QueueProps
{
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
VisibilityTimeout = Duration.Seconds(60),
DeadLetterQueue = new DeadLetterQueue { MaxReceiveCount = 5, Queue = dlq },
});
var targetDlq = new Queue(this, "RuleTargetDlq", new QueueProps
{
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
RetentionPeriod = Duration.Days(14),
});
new Rule(this, "PaymentSettledToLedger", new RuleProps
{
EventBus = bus,
EventPattern = new EventPattern
{
Source = ["harbourline.payments"],
DetailType = ["payments.payment.settled"],
},
Targets =
[
new SqsQueue(ledgerQueue, new SqsQueueProps
{
DeadLetterQueue = targetDlq,
RetryAttempts = 10,
MaxEventAge = Duration.Hours(2),
}),
],
});
// Contract: the schema registry holds the published shape of the event.
var registry = new CfnRegistry(this, "PaymentsRegistry", new CfnRegistryProps
{
RegistryName = "hl-m14-payments",
Description = "Integration event contracts for the payments domain",
});
new CfnSchema(this, "PaymentSettledV1", new CfnSchemaProps
{
RegistryName = registry.RegistryName!,
SchemaName = "payments.payment.settled",
Type = "JSONSchemaDraft4",
Content = """
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"required": ["eventId", "schemaVersion", "correlationId", "data"],
"properties": {
"eventId": { "type": "string" },
"schemaVersion": { "type": "integer", "enum": [1] },
"correlationId": { "type": "string" },
"data": {
"type": "object",
"required": ["paymentId", "amount", "currency"],
"properties": {
"paymentId": { "type": "string" },
"amount": { "type": "string" },
"currency": { "type": "string" }
},
"additionalProperties": false
}
}
}
""",
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m14-l14");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "learner");
}
}The two using aliases resolve a name clash: EventBus exists in both Amazon.CDK.AWS.Events and Amazon.CDK.AWS.Events.Targets. additionalProperties: false on data is how the contract keeps surprise fields (such as an email) out. The schema registry itself does not reject publishes; enforce the schema in CI and in consumers.
Complete code
The validator checks the producer contract (required fields, naming pattern, supported version, no personal-data fields) and the consumer discipline (skip duplicates). Run it before relying on any consumer to be correct.
// Run with: dotnet run EventEnvelope.cs
// A producer-side contract check and a consumer-side duplicate filter for domain events.
using System.Text.Json;
using System.Text.RegularExpressions;
var validator = new EnvelopeValidator(supportedVersions: [1, 2]);
var dedupe = new Deduplicator();
string good = """
{"eventId":"0198a1b2-0000-7000-8000-000000000001","type":"payments.payment.settled","schemaVersion":1,
"source":"harbourline.payments","occurredAt":"2026-11-18T10:00:00Z","correlationId":"corr-123",
"data":{"paymentId":"PAY-0198a1b2","customerId":"CUS-00012345","amount":"25.00","currency":"GBP"}}
""";
string leaksPii = good.Replace("\"customerId\":\"CUS-00012345\"", "\"email\":\"jo@example.com\"")
.Replace("0000000001", "0000000002");
string futureVersion = good.Replace("\"schemaVersion\":1", "\"schemaVersion\":9").Replace("0000000001", "0000000003");
foreach (var (label, json) in new[] { ("good", good), ("leaks PII", leaksPii), ("future version", futureVersion), ("good again", good) })
{
var problems = validator.Validate(json, out var id);
var verdict = problems.Count > 0 ? "REJECT " + string.Join("; ", problems)
: dedupe.FirstTime(id!) ? "PROCESS" : "DUPLICATE (skip)";
Console.WriteLine($"{label,-15} -> {verdict}");
}
return 0;
public sealed class Deduplicator
{
// Production: DynamoDB conditional put on eventId with a TTL longer than the maximum redelivery window.
private readonly HashSet<string> _seen = [];
public bool FirstTime(string eventId) => _seen.Add(eventId);
}
public sealed partial class EnvelopeValidator(int[] supportedVersions)
{
private static readonly string[] Required =
["eventId", "type", "schemaVersion", "source", "occurredAt", "correlationId", "data"];
private static readonly HashSet<string> ForbiddenInPayload = new(StringComparer.OrdinalIgnoreCase)
{ "email", "phone", "first_name", "firstName", "last_name", "lastName", "date_of_birth", "dateOfBirth", "iban", "pan" };
[GeneratedRegex(@"^[a-z]+(\.[a-z]+){2,}$")]
private static partial Regex TypePattern();
public List<string> Validate(string json, out string? eventId)
{
var problems = new List<string>();
eventId = null;
if (json.Length > 200_000) { problems.Add("envelope too large: store a reference, not the payload"); return problems; }
using var doc = JsonDocument.Parse(json);
var root = doc.RootElement;
foreach (var name in Required)
if (!root.TryGetProperty(name, out _)) problems.Add($"missing {name}");
if (problems.Count > 0) return problems;
eventId = root.GetProperty("eventId").GetString();
if (!TypePattern().IsMatch(root.GetProperty("type").GetString() ?? ""))
problems.Add("type must look like domain.entity.event");
if (!supportedVersions.Contains(root.GetProperty("schemaVersion").GetInt32()))
problems.Add("unsupported schemaVersion");
if (root.GetProperty("source").GetString()?.StartsWith("harbourline.", StringComparison.Ordinal) != true)
problems.Add("source must start with harbourline.");
foreach (var bad in FindForbidden(root.GetProperty("data")))
problems.Add($"personal-data field '{bad}' must not travel in events (use customerId)");
return problems;
}
private static IEnumerable<string> FindForbidden(JsonElement element)
{
if (element.ValueKind == JsonValueKind.Object)
foreach (var p in element.EnumerateObject())
{
if (ForbiddenInPayload.Contains(p.Name)) yield return p.Name;
foreach (var inner in FindForbidden(p.Value)) yield return inner;
}
else if (element.ValueKind == JsonValueKind.Array)
foreach (var item in element.EnumerateArray())
foreach (var inner in FindForbidden(item)) yield return inner;
}
}Expected output:
good -> PROCESS
leaks PII -> REJECT personal-data field 'email' must not travel in events (use customerId)
future version -> REJECT unsupported schemaVersion
good again -> DUPLICATE (skip)Code explained line by line
- The sample events are C# raw string literals (
"""..."""), so JSON needs no escaping.Replacecreates the two bad variants; the digit replacement gives each a fresheventIdso duplicate detection does not hide the contract errors. EnvelopeValidator(int[] supportedVersions)takes the versions the consumer can handle. A consumer that has not been upgraded for version 3 rejects it explicitly instead of misreading it.Requiredlists the seven envelope fields. Missing fields are reported all at once, and the method returns early because later checks assume they exist.- The
[GeneratedRegex]attribute compiles the pattern at build time.^[a-z]+(\.[a-z]+){2,}$demands at least three lower-case dot-separated words, such aspayments.payment.settled. ForbiddenInPayloadis a case-insensitive set of names that signal personal data.FindForbiddenwalks the JSON recursively withyield return, so nested objects and arrays are covered.Deduplicator.FirstTimerelies onHashSet.Addreturning false for an existing item. In production this is a DynamoDB conditional write with a TTL that outlasts the longest redelivery or replay window.- The final loop shows the order of decisions: contract first (reject), then duplicates (skip), then processing. Rejected events go to the DLQ for a human; duplicates are silently dropped.
- The size check (200,000 characters) guards the parser and encodes the principle that events carry references, not documents.
Security considerations
⚠ Learning shortcut — not for production
The lab uses SQS-managed encryption, an open-to-account bus policy and a 30-day archive. Production uses customer-managed keys where supported, organisation-scoped bus policies with source conditions, an archive retention agreed with data protection and legal, and a schema-governance process owned by a named team. Never add personal data to events "just for now": the archive and every consumer's copy will outlive the shortcut.
- Least privilege. Producers may put events only to their own bus; consumers read only their own queue.
- Do not trust event content. An event is input. Validate it, even from an internal producer.
- Protect the archive. It is a copy of everything on the bus. Encrypt it, restrict replays, and log who starts one.
- Guard replays. A replay is a privileged operation: require approval and a runbook.
- Detect exposure. Use IAM Access Analyzer to find buses or queues accessible from outside the organisation (Access Analyzer).
Banking use case
Harbourline's payments domain publishes payments.payment.settled v1 with paymentId, customerId, amount and currency. The ledger consumer posts the entry. The notifications consumer looks up the customer's contact preferences through an API and sends an SMS. The data lake consumer writes the event to Parquet for reporting. When the fraud team asks for a new feed, they add a rule and a queue in their own account without changing payments. A month later payments wants to add scheme: it is an optional field, so it ships in v1. When payments needs to split amount into amount and fee, it publishes v2 alongside v1 for 90 days, tracks which consumers still read v1, and then retires it.
Common mistakes
- Treating events as an API with no owner. Every schema needs a named owning team and a deprecation policy.
- Putting personal data in the payload. It spreads into archives, queues and lakes beyond erasure's reach.
- Assuming ordering. EventBridge does not guarantee order; design consumers around versions or timestamps, or use Kinesis for ordered streams.
- No consumer idempotency. At-least-once delivery means duplicates are normal.
- Replaying into side-effect consumers. Customers get old notifications again.
- One shared bus for the whole bank. One policy mistake exposes everything.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Rule matches nothing | Pattern case or field mismatch (detail-type vs DetailType) | Test the pattern with the console test tool |
| Events arrive but SQS is empty | Queue policy missing for events.amazonaws.com | Add the policy with aws:SourceArn condition |
PutEvents returns FailedEntryCount greater than zero | Entry too large, bad bus name or throttling | Inspect each entry's ErrorCode; reduce size |
| Replay does nothing | Time range outside archive or pattern filter excludes events | Check archive state and range |
| Consumer processes the same event twice | No dedupe store, or TTL shorter than redelivery window | Add conditional-put dedupe with sufficient TTL |
| CI contract test fails after a producer change | Breaking schema change | Add an optional field instead, or publish a new version |
Cost and cleanup
The C# validator is free. The CDK or Terraform stack creates an event bus, an archive, three queues and a registry: effectively free at lab volumes (a few events). Archives bill by stored GB and replays by event ⚠ VERIFY; keep the lab to a handful of events and delete everything afterwards.
# Delete the replay and archive before the bus, then the bus and queues, or use the IaC destroy.
aws events delete-archive --archive-name hl-m14-payments-archive --profile learn-sandbox --region us-east-1
cdk destroy --profile learn-sandbox
terraform destroy
# Confirm nothing tagged for the course remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1Exercises
- 🟢 Guided: run the validator and add a fifth event with
typeset toPaymentSettled. Predict the verdict. - 🟡 Challenge: write a compatibility checker that takes two JSON Schemas for
payments.payment.settled(v1 and v1.1) and reports whether the change is backward compatible according to the table in this lesson. - 🔴 Live problem: after an incident, operations replays six hours of the payments archive. Three hundred customers receive duplicate "payment sent" texts. Explain why it happened, what the notifications consumer should have done, and the controls you would add around replay.
Assignment
Add architecture/event-backbone/ to your portfolio: the 15-section document, an event catalogue for three domains with owners and schema versions, the envelope validator with unit tests, and an ADR "one bus per domain with an outbox and SQS per consumer".
Solutions
PaymentSettled fails the type pattern because it has upper-case letters and no dots, so the verdict is REJECT type must look like domain.entity.event. If the event also reused an existing eventId, the contract check would still reject it first, because validation happens before the duplicate check. Why: a malformed event must never be treated as "processed".
Parse both schemas, collect required and properties for data. Backward compatible if: no property removed, no type changed, required set unchanged or smaller, and any added property is not in required. Report each violation with the field name. Why: a consumer written against v1 must still work when it receives a v1.1 event, so only optional additions are safe.
The replay re-sent events to every matching rule, including the notifications queue, whose consumer had no dedupe because notifications "never repeat". It should have recorded eventId (or a business key like paymentId + notification type) in a dedupe table with a TTL beyond the archive window, and skipped repeats. Controls: replays need approval and a named runbook; replayed events are tagged (EventBridge adds a replay-name field) so side-effect consumers can ignore them; replay into a separate bus or filtered rule set first.
Knowledge check
Answer: An event states a fact that has already happened. It cannot be refused or edited, unlike a command.
Answer: Events are copied into queues, archives and lakes, so personal data spreads beyond control. Opaque IDs let consumers fetch data through an authorised API.
Answer: Removing or renaming a field, changing a type or meaning, or adding a required field. Adding an optional field does not.
Answer: No. Delivery is at-least-once and ordering is not guaranteed, so consumers must be idempotent and tolerate reordering.
Answer: It buffers during outages, gives retries and a DLQ, and applies back-pressure, instead of dropping events when the consumer is slow or down.
Interview questions
- How do you stop an event system becoming a tangle? One bus per domain, published schemas with owners and versions, contract tests in CI, and a catalogue of who consumes what.
- How do you handle schema evolution? Optional additions are compatible; anything else is a new version run in parallel until consumers move.
- How do you make a consumer safe? Validate, deduplicate by
eventId, tolerate reordering, and send failures to a DLQ. - EventBridge or Kinesis? EventBridge for routed business events; Kinesis or MSK for ordered, high-volume streams with offset replay.
- How do you handle replay safely? Idempotent consumers, flagged replays, approval and a runbook, and a test replay into a non-production consumer.
Certification practice
A consumer must not lose events when it is down for an hour, and must scale independently. Which design fits?
A. EventBridge rule with a Lambda target and no retry policy. B. EventBridge rule targeting an SQS queue consumed by the service, with a DLQ. C. Synchronous HTTP calls from the producer. D. An SNS topic with email subscription.
Answer: B. SQS buffers events while the consumer is down and decouples scaling. A can drop events after retries expire, C couples availability, and D is not a processing channel.
Many teams publish events and a breaking field rename caused an outage. Which control best prevents repeats?
A. Ask teams to be careful. B. A shared schema repository with compatibility checks in each producer's CI pipeline and a versioning policy. C. Disable the archive. D. Use one bus for all events.
Answer: B. Automated contract checks catch breaking changes before release. A is not a control, C is unrelated, and D increases blast radius.
An SQS consumer sometimes processes the same event twice. What is the correct fix?
A. Increase the visibility timeout to infinity. B. Make processing idempotent using the event ID as a deduplication key. C. Switch to a standard queue only. D. Delete the queue and recreate it.
Answer: B. At-least-once delivery means duplicates happen; idempotent processing is the reliable fix. A hides failures, C does not help (standard queues can duplicate), D is destructive.
Completion checklist
- I can explain event versus command and thin versus fat events
- I ran the validator and understand each verdict
- I wrote the schema compatibility rules for my domain
- My design has a replay procedure that protects side-effect consumers
- I ran cleanup and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- Integration events are public contracts: versioned, owned and free of personal data.
- One bus per domain limits blast radius; an SQS queue per consumer provides buffering and DLQs.
- Delivery is at-least-once with no ordering guarantee, so every consumer is idempotent.
- Replay is powerful and dangerous; plan it, flag it and approve it.
- Erasure, retention and audit questions need data-protection and legal review.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
API orchestration: Step Functions or in-service
Decide when a multi-step banking operation should be orchestrated by AWS Step Functions and when by code inside the service, design timeouts, retries and compensation, and run a C# saga runner and an Amazon States Language definition.
Audit and compliance architecture
Design a tamper-evident audit trail with CloudTrail, a locked log archive, application audit records and evidence collection, answer who-accessed-what questions with Athena, and verify a hash-chained audit log in C#.