Asynchronous communication and event-driven architecture
Tell queues, topics, event buses, streams and workflows apart, decide which AWS service fits a Harbourline integration with a written decision tree, and publish your first well-formed domain event from .NET 10.
Assumes: .NET 10 LTS ยท AWS SDK for .NET v4 (AWSSDK.EventBridge 4.1.0.2, AWSSDK.SQS 4.1.0.2) ยท AWS CLI v2 ยท Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain the difference between synchronous and asynchronous communication, and name the three kinds of coupling (time, location, format) that asynchronous messaging removes.
- Tell apart a command, an event and a document message, and decide which one a given Harbourline interaction should be.
- Describe the five shapes of asynchronous integration on AWS โ queue, topic, event bus, stream and workflow โ and map each to its AWS service (SQS, SNS, EventBridge, Kinesis/MSK, Step Functions), plus Amazon MQ for legacy protocols.
- Walk the "which service when?" decision tree for a new integration and defend the answer in a design review.
- Design a domain event envelope that keeps personal data out of the payload, and publish it to an EventBridge bus from .NET 10 with the AWS SDK v4.
Prerequisites
- Module 3: .NET on AWS โ Lambda, Worker Services, registering SDK clients with
AddAWSService<T>() - Module 6: Databases โ transactions and consistency vocabulary
- Module 8 overview โ the module map and the notification-system project you are working towards
- Tools: .NET 10 SDK, AWS CLI v2 with the
learn-sandboxSSO profile (only for the optional AWS steps)
Why this matters in banking
It is the last Friday of the month at Harbourline Bank. Salaries land between 00:30 and 02:00, and the mobile app sends a push notification for every credit. Today the Payments Service calls the Notifications Service over HTTP for each posted transaction, waits for a 200, and only then marks the posting as complete.
At 01:10 the SMS provider behind the Notifications Service slows down. Calls that took 80 ms now take eight seconds. Payments threads pile up waiting for replies, the connection pool runs dry, and the posting engine โ the part that actually moves money โ starts timing out. A cosmetic feature (a "you've been paid" text) has taken down a critical one (crediting salaries). The incident review records one root cause: temporal coupling. Payments could only finish its work if Notifications was healthy at the same moment.
Asynchronous messaging is how you break that dependency. Payments records the posting, emits a TransactionPosted event and moves on. Notifications reads the event when it can. If the SMS provider is down for an hour, events wait safely in a queue and drain afterwards. Nobody's salary is late.
But asynchronous design brings its own failure modes, and they matter more in a bank than almost anywhere else: a message delivered twice can mean a double debit, a message processed out of order can show a customer a balance that never existed, and a malformed message can block a whole queue. This module teaches the patterns that make asynchronous systems safe. This first lesson gives you the vocabulary and the decision framework for everything that follows.
The idea in plain language
Synchronous communication is a phone call. You dial, the other person must pick up right now, and you wait on the line until they answer. If they are busy, you are stuck.
Asynchronous communication is a letter or a voicemail. You leave the message somewhere durable and get on with your day. The recipient deals with it when they can. You do not know exactly when, and you may not get a reply at all โ but you are not blocked.
In software, "somewhere durable" is a piece of middleware: a queue, a topic, an event bus or a stream. The sender (producer) hands the message to the middleware, which stores it redundantly and delivers it to one or more receivers (consumers).
Asynchronous messaging removes three kinds of coupling โ dependencies that make two systems fail together:
| Coupling | Meaning | How messaging removes it |
|---|---|---|
| Time | Both systems must be up at the same moment | The message waits in durable storage until the consumer is ready |
| Location | The sender must know the receiver's address | The sender only knows the queue, topic or bus; consumers can move, scale or be replaced |
| Rate | The sender can only go as fast as the receiver | The queue absorbs bursts; consumers process at their own pace (load levelling) |
What messaging does not remove is format coupling: both sides must still agree on what a message looks like. That is why event schemas and versioning matter (you meet the EventBridge schema registry in the EventBridge lesson).
Analogy: Harbourline's branch post room. Tellers drop internal mail into labelled trays (queues). A circular for "all branch managers" goes on the noticeboard (a topic) and every manager reads their own copy. The head-office sorting clerk reads every envelope and forwards it by rules โ "anything about mortgages goes to Leeds" โ (an event bus). The general ledger is a bound book where every posting is written in order and auditors can reread any page (a stream). And a mortgage application follows a checklist that a case officer walks step by step, chasing each department (a workflow).
Commands, events and documents
Not every message means the same thing. Getting the type right shapes the whole design.
| Kind | Meaning | Tense | Who decides what happens? | Harbourline example |
|---|---|---|---|---|
| Command | "Please do this" | Imperative | The sender expects a specific receiver to act | SendSmsNotification, SubmitPaymentToScheme |
| Event | "This happened" | Past | Each subscriber decides for itself; the sender does not care who listens | TransactionPosted, PaymentSettled, CardFrozen |
| Document | "Here is some data" | Neutral | The receiver decides | A daily exchange-rate file; a statement batch |
A command usually has one intended handler, so it belongs on a queue. An event can have many listeners โ Notifications, Fraud, the Data Platform โ so it belongs on a topic, bus or stream. A common mistake is to publish "events" that are really commands in disguise (SendWelcomeEmailEvent), which couples the producer back to a specific consumer.
Delivery semantics in one paragraph
Every AWS messaging service in this module delivers at least once: under failure, retries or network glitches, a consumer can see the same message twice. Some services (SQS FIFO, SNS FIFO) deduplicate within a window, and some (SQS FIFO, Kinesis per shard, MSK per partition) preserve order within a key. None of them can make your side effects happen exactly once โ that is your job, using idempotent consumers. You build those properly in the idempotency lesson (see the module overview, lesson 11) and in the module project.
What problem these AWS services solve
AWS offers several managed services because the five integration shapes behave very differently. Here is each one in plain words, with its honest limitations.
| Shape | AWS service | What it does | Delivery | Ordering | Retention / replay | Sharp edges |
|---|---|---|---|---|---|---|
| Queue (point-to-point, pull) | Amazon SQS | Stores messages until one consumer receives and deletes each | At least once (standard); deduplicated within 5 minutes (FIFO) | None (standard); per message group (FIFO) | 60 s to 14 days, default 4 days; no replay after delete | Consumers must delete; visibility timeout must exceed processing time |
| Topic (pub/sub, push) | Amazon SNS | Copies each message to every subscriber (SQS, Lambda, HTTPS, email, SMS) | At least once, with delivery retries | None (standard); per group (FIFO topics) | No storage โ if a subscriber is down after retries, the message goes to a subscription DLQ or is lost | Subscribers must be reachable; use SNS โ SQS for durability |
| Event bus (content-based routing, push) | Amazon EventBridge | Matches events against rules and sends to up to 5 targets per rule | At least once; retries for up to 24 hours by default | None | Optional archive and replay | Higher latency than SNS (typically around half a second) โ VERIFY; no ordering |
| Stream (append-only log, pull) | Kinesis Data Streams, Amazon MSK (Kafka) | Stores an ordered log per shard/partition; many consumers read at their own position | At least once | Per shard / partition key | Kinesis: 24 hours default, up to 365 days; MSK: configurable | You pay per shard-hour or broker-hour even when idle (except on-demand/serverless options) |
| Workflow (orchestration) | AWS Step Functions | Runs a state machine that calls services in order, with retries, branches and compensation | Standard: exactly-once step execution; Express: at least once | Defined by the state machine | Standard runs up to 1 year; Express up to 5 minutes | Not a transport โ it coordinates calls; state size limits |
| Legacy broker | Amazon MQ (ActiveMQ, RabbitMQ) | Managed broker speaking JMS, AMQP, MQTT, STOMP, OpenWire | Broker semantics | Per queue | Broker storage | You pay per broker-hour; you still size and patch-window a broker |
Two things in this table deserve emphasis for a bank:
- Only streams and EventBridge archives let you replay history. SQS deletes messages once processed. If the Data Platform team asks "send me every
TransactionPostedfrom last Tuesday again", a queue cannot help; a Kinesis stream (within retention) or an EventBridge archive can. - Only FIFO queues/topics, Kinesis and MSK preserve order โ and only per key. Global ordering across a whole bank does not exist at scale. You will design for "ordered per account" instead (lesson 10 in the overview).
Payload sizes in 2026
Payload limits rose recently, and older blog posts and practice exams still quote 256 KB everywhere.
| Service | Maximum message / event size (checked 2026-11-18) |
|---|---|
| SQS | 1 MiB (1,048,576 bytes), billed in 64 KB chunks |
EventBridge PutEvents | 1 MB per request (raised from 256 KB in January 2026) |
| Lambda asynchronous invocation | 1 MB (raised from 256 KB in October 2025) |
| SNS | 256 KB โ VERIFY โ check the SNS quotas page; a 1 MiB option has been reported but not confirmed in the documentation at time of writing |
Bigger is not better. A notification event should carry identifiers, not documents. If you need a large payload, store it in S3 and send a pointer (the "claim check" pattern).
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
Synchronous HttpClient call between services | API Gateway / ALB โ service | Same idea; still couples time and rate |
| MSMQ private queue | Amazon SQS standard queue | No server; unlimited backlog; pull over HTTPS |
| Azure Service Bus queue (sessions) | SQS FIFO queue (message groups) | MessageGroupId plays the role of SessionId; visibility timeout replaces peek-lock |
| Azure Service Bus topic + subscriptions with SQL filters | SNS topic + filter policies (JSON) | SNS pushes; it does not store. Fan out to SQS for durability |
| Azure Event Grid | Amazon EventBridge | EventBridge rules use JSON event patterns; built-in archive/replay, Pipes and Scheduler |
| Azure Event Hubs | Kinesis Data Streams / Amazon MSK | Shards (Kinesis) or partitions (MSK) instead of Event Hubs partitions; MSK is real Apache Kafka |
| Azure Durable Functions / Logic Apps | AWS Step Functions | State machines defined in Amazon States Language (JSON) or the visual Workflow Studio |
| IBM MQ / on-premises ActiveMQ | Amazon MQ | Managed ActiveMQ or RabbitMQ only โ not IBM MQ |
| MassTransit / NServiceBus | AWS.Messaging for .NET (lesson 15) | AWS's own lightweight framework; MassTransit also has an SQS/SNS transport |
IHostedService background worker polling a Service Bus queue | .NET Worker Service polling SQS, or Lambda with an SQS event source mapping | Lambda polling is managed for you; a Worker gives you more control |
The biggest mental shift from Service Bus: Service Bus is one product that does queues and topics with rich broker features; AWS splits those jobs across SQS (store), SNS (fan-out) and EventBridge (route) and you compose them. The classic AWS combination "SNS topic โ several SQS queues" is the equivalent of a Service Bus topic with subscriptions.
Architecture
Before: a synchronous chain
Every box in the chain must be healthy at the same time, and the slowest one sets the response time.
After: event-driven
Payments now depends on exactly one thing besides its ledger: the EventBridge API. Fraud and Notifications evolve and fail independently. This is the shape of the module project in the notification system walkthrough.
Security-boundary view
Every hop is an authenticated AWS API call: there are no shared connection strings. Each queue's resource policy says who may send to it (for example, "only this SNS topic"), and each consumer's IAM role says which queue it may read. Encryption at rest applies to SQS (SSE-SQS by default on new queues, or SSE-KMS), SNS (KMS) and EventBridge (AWS owned key by default; customer managed keys are supported for buses โ VERIFY scope). Payload content is visible to anyone who can read the queue, so keep personal data out.
The "which service when?" decision tree
This is the heart of the lesson. Walk it top to bottom for any new integration. It is deliberately opinionated: it gives the default answer a design review would accept, not the only possible one.
How to read the leaves:
- Synchronous API. Balance enquiry from the app, confirmation of payee, card authorisation โ the user is waiting. Keep it synchronous, but protect it with timeouts and circuit breakers (lesson 9).
- Step Functions. "Validate โ sanctions screen โ reserve funds โ submit to scheme โ wait for settlement โ notify, and undo the reservation if anything fails." That is a workflow, not a message. The Step Functions lesson covers it.
- Amazon MQ. A card processor that only speaks JMS over OpenWire. The Amazon MQ lesson shows how to keep it and bridge to native services.
- Kinesis / MSK. Card-authorisation telemetry at thousands of records per second, read by fraud scoring, the data lake and a real-time dashboard, each at its own pace, with replay after a model change. See Kinesis and MSK.
- SQS standard / FIFO. "Generate this statement PDF", "send this SMS". One kind of worker does the job. FIFO when order per customer matters or you want SQS to drop duplicate sends. See SQS in depth.
- EventBridge.
TransactionPostedis interesting to Notifications, Fraud, Rewards and the Data Platform, each wanting a different subset. Rules on content keep producers ignorant of consumers. See EventBridge. - SNS. Very high-volume, low-latency fan-out of the same message to known subscriber types, especially "one event โ N SQS queues". See SNS.
The last box matters: EventBridge and SNS push. If a consumer is slow or briefly unavailable, push delivery retries but eventually gives up. Putting an SQS queue between the topic or bus and the consumer turns push into durable pull, gives you a dead-letter queue, and lets the consumer control its own concurrency. In banking, nearly every consumer gets its own queue.
Tie-breakers when two answers look right
| Dilemma | Lean towards | Because |
|---|---|---|
| SNS vs EventBridge for fan-out | EventBridge by default; SNS when you need very high throughput, SMS/email/mobile push endpoints or FIFO fan-out | EventBridge's content filtering, archive/replay and schema registry pay off in a bank; SNS has lower latency and FIFO topics |
| SQS vs Kinesis for a high-volume feed | SQS when each record is an independent job; Kinesis when consumers need order per key or replay | SQS scales consumers per message; Kinesis scales per shard |
| Kinesis vs MSK | Kinesis unless you need Kafka APIs, Kafka Connect, long retention economics or existing Kafka skills | Kinesis is simpler and fully serverless; MSK is a Kafka cluster (or MSK Serverless) |
| Step Functions vs choreography with events | Step Functions when one team owns the end-to-end process and needs visibility; choreography when domains are independent | Lesson 12 compares them in depth |
| Amazon MQ vs SQS for a new system | SQS/SNS | Amazon MQ exists for protocol compatibility, not as the default for new builds |
Designing a good event
A well-designed event is the contract between teams. Harbourline's integration standard uses this envelope (the same shape EventBridge itself uses for detail):
{
"eventId": "01937a4e-5c1b-7c4e-9a51-2f0f6d2c8b11",
"eventType": "TransactionPosted",
"eventVersion": "1.0",
"occurredAt": "2026-11-27T01:12:44.512Z",
"correlationId": "c0b9f4a2-1d7e-4c37-8a7f-6a0e1b2c3d4e",
"producer": "harbourline.core-banking",
"data": {
"transactionId": "01937a4e-5b90-7f21-8c3d-9e8f7a6b5c4d",
"accountId": "ACC-0000123456",
"customerId": "CUS-00012345",
"amount": 2450.00,
"currency": "GBP",
"type": "transfer_in",
"channel": "batch",
"bookingDate": "2026-11-27"
}
}Rules behind it:
- Past tense, business language.
TransactionPosted, notTxnTableInsert. - A unique
eventId(UUID v7, time-ordered) so every consumer can deduplicate. occurredAtfrom the producer's clock, not the broker's, so consumers can detect late or out-of-order events.eventVersion, so you can add fields (minor version) or make breaking changes (new major version, run both in parallel).- Identifiers, not personal data. No name, email, phone or full IBAN. The Notifications Service looks up contact details from the customer service using
customerId. This keeps personal data out of queues, archives and logs, where retention and access are harder to control โ and it matters because SNS Message Data Protection is closed to new customers (see the module overview). - Money as a decimal with a currency code. Serialise with
System.Text.Jsonand adecimal; neverdouble. - A
correlationIdcarried from the original API request through every hop, so one customer complaint can be traced end to end.
Thin vs fat events. A thin event ("transaction X posted, go and look it up") minimises data exposure but makes every consumer call back to the producer. A fat event carries everything a consumer needs. Harbourline's compromise, above, is "fat on business facts, thin on personal data": amount, currency and type are in; who the customer is stays behind an API.
Step-by-step implementation
In this lesson you will classify Harbourline's integrations with the decision tree, then publish one real event to EventBridge and read it back through a queue. The full notification system comes later, in the project walkthrough.
Classify the integrations on paper
Take these six Harbourline flows and run each through the decision tree. Write the answer and the one-line reason. (Model answers are in the Solutions section.)
- Mobile app shows the current balance.
- A posted transaction triggers an SMS and an email.
- A SEPA Instant payment: validate, screen, reserve funds, submit, await confirmation, compensate on failure.
- Card-authorisation telemetry at 3,000 records per second, read by fraud scoring and the data lake.
- The legacy treasury system sends FX deals over JMS and cannot be changed this year.
- A nightly job asks for 40,000 PDF statements to be generated.
Create a bus, a queue and a rule (optional, free tier)
Use the CLI commands in the AWS CLI section. You create an EventBridge custom bus hl-m08-events, an SQS queue hl-m08-l1-tap with a policy that lets only that rule send to it, and a rule that matches TransactionPosted events. This is a "tap" queue that lets you see events arriving.
Publish an event from .NET
Run the console app in Complete code. It builds the envelope above and calls PutEvents. It prints the event ID that EventBridge assigned and fails loudly if the entry was rejected.
Read it from the queue
Receive from the tap queue with the CLI. You will see EventBridge's own envelope (version, id, detail-type, source, account, time, region, resources, detail) wrapped around your JSON. Notice that your data.amount arrives intact as a JSON number.
Tear down
Run the cleanup commands. Nothing in this lesson bills by the hour, but leaving resources behind is a bad habit and an audit finding.
AWS Console instructions
You can do the optional AWS steps in the console instead of the CLI:
- Console โ Amazon EventBridge โ Buses โ Event buses โ Create event bus. Name
hl-m08-events. Leave archive off for now. Add tagsProject=cea-lab,Lab=m08-l1,Environment=learn,Owner=<you>. Create. - Console โ Amazon SQS โ Create queue. Type Standard, name
hl-m08-l1-tap. Under Encryption, keep server-side encryption enabled (SSE-SQS). Add the same tags. Create. - Console โ Amazon EventBridge โ Buses โ Rules โ Create rule. Event bus
hl-m08-events, namehl-m08-l1-tap-rule, rule type Rule with an event pattern. Choose Custom pattern (JSON editor) and paste the pattern from the CLI section. Target: SQS queue โhl-m08-l1-tap. The console offers to update the queue's access policy for you; accept it. Create. - To send a test event without code: EventBridge โ Event buses โ hl-m08-events โ Send events. Source
harbourline.core-banking, detail typeTransactionPosted, and paste thedata-bearing JSON as the detail. - SQS โ hl-m08-l1-tap โ Send and receive messages โ Poll for messages to see it arrive.
Console layouts change; if a label differs, look for the same idea under a nearby heading.
AWS CLI
# Variables used by every command below
export AWS_PROFILE=learn-sandbox
export AWS_REGION=us-east-1
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
TAGS='Key=Project,Value=cea-lab Key=Lab,Value=m08-l1 Key=Environment,Value=learn Key=Owner,Value=you'
# 1. Custom event bus. Keep your events off the account's default bus.
aws events create-event-bus --name hl-m08-events \
--tags $TAGS
# 2. A standard queue to "tap" the bus. SSE-SQS is on by default for new queues;
# we set it explicitly so the intent is visible in code review.
QUEUE_URL=$(aws sqs create-queue --queue-name hl-m08-l1-tap \
--attributes SqsManagedSseEnabled=true,MessageRetentionPeriod=3600 \
--tags Project=cea-lab,Lab=m08-l1,Environment=learn,Owner=you \
--query QueueUrl --output text)
QUEUE_ARN=$(aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
--attribute-names QueueArn --query Attributes.QueueArn --output text)
# 3. A rule that matches TransactionPosted events from core banking
aws events put-rule --name hl-m08-l1-tap-rule \
--event-bus-name hl-m08-events \
--event-pattern '{"source":["harbourline.core-banking"],"detail-type":["TransactionPosted"]}' \
--tags $TAGS
RULE_ARN=$(aws events describe-rule --name hl-m08-l1-tap-rule \
--event-bus-name hl-m08-events --query Arn --output text)
# 4. Queue policy: only this rule (via the EventBridge service) may send
cat > tap-policy.json <<EOF
{
"Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"AllowRuleOnly\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"events.amazonaws.com\"},\"Action\":\"sqs:SendMessage\",\"Resource\":\"$QUEUE_ARN\",\"Condition\":{\"ArnEquals\":{\"aws:SourceArn\":\"$RULE_ARN\"}}}]}"
}
EOF
aws sqs set-queue-attributes --queue-url "$QUEUE_URL" --attributes file://tap-policy.json
# 5. Point the rule at the queue
aws events put-targets --rule hl-m08-l1-tap-rule --event-bus-name hl-m08-events \
--targets "Id=tap,Arn=$QUEUE_ARN"
# 6. Send a test event from the CLI (the .NET app does the same thing)
aws events put-events --entries '[{"EventBusName":"hl-m08-events","Source":"harbourline.core-banking","DetailType":"TransactionPosted","Detail":"{\"eventId\":\"cli-test-1\",\"eventVersion\":\"1.0\",\"data\":{\"accountId\":\"ACC-0000123456\",\"amount\":12.50,\"currency\":\"GBP\"}}"}]'
# 7. Read it back with long polling (waits up to 10 s)
aws sqs receive-message --queue-url "$QUEUE_URL" --wait-time-seconds 10 \
--max-number-of-messages 10Things to notice:
put-eventsreturnsFailedEntryCount. A 200 response does not mean every entry was accepted โ always check that count. The .NET code below does.- The queue policy uses
aws:SourceArnso that no other EventBridge rule โ in your account or anyone else's โ can write into the queue. This is the "confused deputy" protection you met in Module 5.
Infrastructure as Code
The same three resources in CDK (C#) and Terraform. The module project's infra/cdk builds the full version.
using Amazon.CDK;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.Events.Targets;
using Amazon.CDK.AWS.SQS;
using Constructs;
namespace Harbourline.Notifications.Infra;
public sealed class TapStack : Stack
{
public TapStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
// Custom bus: producers publish here, never to the default bus.
var bus = new EventBus(this, "Bus", new EventBusProps { EventBusName = "hl-m08-events" });
// Tap queue, encrypted with SQS-managed keys, short retention for a lab.
var tap = new Queue(this, "Tap", new QueueProps
{
QueueName = "hl-m08-l1-tap",
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
RetentionPeriod = Duration.Hours(1),
});
// Rule matching TransactionPosted from core banking.
var rule = new Rule(this, "TapRule", new RuleProps
{
RuleName = "hl-m08-l1-tap-rule",
EventBus = bus,
EventPattern = new EventPattern
{
Source = ["harbourline.core-banking"],
DetailType = ["TransactionPosted"],
},
});
// SqsQueue target also writes a queue policy scoped to this rule's ARN.
rule.AddTarget(new SqsQueue(tap));
foreach (var (k, v) in new[] { ("Project", "cea-lab"), ("Lab", "m08-l1"), ("Environment", "learn"), ("Owner", "you") })
Amazon.CDK.Tags.Of(this).Add(k, v);
}
}Terraform still calls EventBridge resources aws_cloudwatch_event_* because EventBridge grew out of CloudWatch Events.
Complete code
A .NET 10 console app that publishes a TransactionPosted event. It runs against AWS with your SSO profile; there is no local EventBridge emulator in this course, so for offline work use the in-memory simulator in the module project (labs/m08-notification-system/).
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<RootNamespace>Harbourline.Events.Publisher</RootNamespace>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AWSSDK.EventBridge" Version="4.1.0.2" />
</ItemGroup>
</Project>using System.Text.Json;
using System.Text.Json.Serialization;
using Amazon;
using Amazon.EventBridge;
using Amazon.EventBridge.Model;
// --- 1. Build the domain event (identifiers only, no personal data) -------------
var evt = new EventEnvelope<TransactionPostedData>(
EventId: Guid.CreateVersion7(),
EventType: "TransactionPosted",
EventVersion: "1.0",
OccurredAt: DateTimeOffset.UtcNow,
CorrelationId: Guid.NewGuid(),
Producer: "harbourline.core-banking",
Data: new TransactionPostedData(
TransactionId: Guid.CreateVersion7(),
AccountId: "ACC-0000123456",
CustomerId: "CUS-00012345",
Amount: 2450.00m,
Currency: "GBP",
Type: "transfer_in",
Channel: "batch",
BookingDate: DateOnly.FromDateTime(DateTime.UtcNow)));
var json = JsonSerializer.Serialize(evt, EventJson.Options);
// --- 2. Create the client (credentials from the learn-sandbox SSO profile) ------
var busName = Environment.GetEnvironmentVariable("HL_EVENT_BUS") ?? "hl-m08-events";
using var client = new AmazonEventBridgeClient(RegionEndpoint.USEast1);
// --- 3. Publish ---------------------------------------------------------------
var response = await client.PutEventsAsync(new PutEventsRequest
{
Entries =
[
new PutEventsRequestEntry
{
EventBusName = busName,
Source = evt.Producer,
DetailType = evt.EventType,
Detail = json,
Time = evt.OccurredAt.UtcDateTime,
},
],
});
// --- 4. Check per-entry results: HTTP 200 can still contain failures ------------
var entry = response.Entries?.FirstOrDefault();
if ((response.FailedEntryCount ?? 0) > 0 || entry is null || entry.ErrorCode is not null)
{
Console.Error.WriteLine($"Rejected: {entry?.ErrorCode} {entry?.ErrorMessage}");
return 1;
}
Console.WriteLine($"Published {evt.EventType} {evt.EventId} as EventBridge id {entry.EventId}");
Console.WriteLine($"Correlation id: {evt.CorrelationId}");
return 0;
// --- Types --------------------------------------------------------------------
public sealed record EventEnvelope<T>(
Guid EventId,
string EventType,
string EventVersion,
DateTimeOffset OccurredAt,
Guid CorrelationId,
string Producer,
T Data);
public sealed record TransactionPostedData(
Guid TransactionId,
string AccountId,
string CustomerId,
decimal Amount,
string Currency,
string Type,
string Channel,
DateOnly BookingDate);
public static class EventJson
{
public static readonly JsonSerializerOptions Options = new(JsonSerializerDefaults.Web)
{
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
};
}Run it:
aws sso login --profile learn-sandbox
AWS_PROFILE=learn-sandbox dotnet runExpected output (IDs differ):
Published TransactionPosted 01937a4e-5c1b-7c4e-9a51-2f0f6d2c8b11 as EventBridge id 5f6c0b2e-0d4a-8a1c-3e6b-2a9c7d0e4f11
Correlation id: c0b9f4a2-1d7e-4c37-8a7f-6a0e1b2c3d4eCode explained line by line
EventEnvelope<TransactionPostedData>โ a generic record so every Harbourline event shares the same outer fields. Consumers can readeventTypeandeventVersionbefore deciding how to parsedata.Guid.CreateVersion7()โ new in .NET 9, a time-ordered UUID v7. It matches thetransaction_idconvention inCONVENTIONS.mdand makes IDs sort roughly by creation time, which helps when debugging.OccurredAt: DateTimeOffset.UtcNowโ the producer's clock. Always UTC.Amount: 2450.00mโ adecimalliteral.System.Text.Jsonwrites it as a JSON number without binary floating-point error.JsonSerializerDefaults.Webโ camelCase property names, matching the JSON convention inCONVENTIONS.md.HL_EVENT_BUSenvironment variable โ the bus name is configuration, not code, so the same binary can publish to a test bus.new AmazonEventBridgeClient(RegionEndpoint.USEast1)โ the SDK finds credentials through the default chain;AWS_PROFILE=learn-sandboxmakes it use your SSO session. In an ASP.NET Core app you would registerIAmazonEventBridgewithAddAWSService<IAmazonEventBridge>()instead.Entries = [ ... ]โ a C# 12 collection expression.PutEventsaccepts several entries per call (batching saves money and latency).Source,DetailType,Detailโ the three fields rules match on.Sourcemust not start withaws.; that prefix is reserved for AWS services.Timeโ sets the event'stimefield to the business time instead of the moment EventBridge received it.response.FailedEntryCount ?? 0โ in SDK v4, value-type properties are nullable (int?) and collections default tonull, so the code guards both. Forgetting this is the most common v3 โ v4 upgrade bug.entry.ErrorCodeโ per-entry failures (throttling, an internal error, a malformed entry) appear here even when the HTTP call succeeded. A production publisher retries only the failed entries, with backoff.return 1โ a non-zero exit code makes failures visible to scripts and CI.
Security considerations
โ Learning shortcut โ not for production
The console app publishes with your own SSO identity, which can do far more than publish one event. In production the Payments Service runs under a task or function role whose only messaging permission is events:PutEvents on arn:aws:events:us-east-1:<account>:event-bus/hl-m08-events, ideally with a condition on events:source so it can only publish as harbourline.core-banking โ VERIFY the condition key name for your use.
โ Learning shortcut โ not for production
The tap queue uses SSE-SQS (keys managed by SQS). For queues that may contain customer-related data, banks usually require SSE-KMS with a customer managed key so key use is logged in CloudTrail and access can be revoked by key policy. Note that queue metadata and message attributes are not encrypted by SSE โ only the body.
Further points:
- Least privilege on both sides. Producers get publish rights to one bus or topic; consumers get receive/delete on one queue. Resource policies on queues and topics restrict who may send.
- No personal data in events. Send identifiers and let consumers fetch details through an authorised API. This keeps queues, archives, DLQs and logs out of scope for most data-subject requests and reduces breach impact. Your data-protection officer still needs to review what is in the payload โ account IDs and amounts are personal data under GDPR when linked to a person.
- Retention is a data-protection decision. SQS keeps messages up to 14 days, EventBridge archives can keep events indefinitely, Kinesis up to a year. Set retention deliberately and record why.
- TLS everywhere. Add
aws:SecureTransportconditions to queue and topic policies (CDK'sEnforceSSL = truedoes this). - Audit. CloudTrail records management calls (create queue, put rule) by default; data-plane calls like
SendMessageandPutEventsare data events you opt into โ VERIFY which messaging data events CloudTrail supports.
Banking use case
Harbourline's architecture board adopted these rules for its event-driven estate (fictional, but typical):
| Flow | Pattern | Service | Why |
|---|---|---|---|
| Card authorisation | Synchronous | API (ALB โ ECS) | Merchant terminal waits for approve/decline in under a second |
TransactionPosted | Event, many consumers | EventBridge โ per-consumer SQS | Notifications, Fraud, Rewards and Data Platform each filter differently; archive for replay |
| Customer notifications | Command per channel | SNS โ SQS โ Lambda | Fan-out by channel with filter policies; DLQ per channel |
| Outbound payment | Workflow with compensation | Step Functions Standard | Must survive hours waiting for scheme confirmation; auditable history |
| Card telemetry for fraud models | Stream | Kinesis Data Streams | Ordered per card; ML team replays after retraining |
| FX deals from treasury | Legacy JMS | Amazon MQ (ActiveMQ) bridged to EventBridge | Treasury system cannot change this year |
| Statement generation | Work queue | SQS standard + Fargate workers | 40,000 independent jobs; scale workers on queue depth |
Note how many flows end in SQS. Queues are the shock absorbers of the estate.
Common mistakes
- Making everything asynchronous. A balance check that goes through a queue and back is slower, harder to debug and no more reliable. โ Use the first question of the decision tree honestly.
- Using SNS or EventBridge to deliver straight to a fragile consumer. Push delivery retries, then gives up. โ Put an SQS queue (with a DLQ) in front of each consumer.
- Assuming exactly-once. Every service here can deliver twice. โ Make every consumer idempotent from day one.
- Assuming order on a standard queue or on EventBridge. Neither preserves order. โ Use FIFO, Kinesis or MSK with a sensible key, or design consumers to tolerate disorder (use
occurredAtand version numbers). - Putting personal data in events "for convenience". It then lives in queues, DLQs, archives and logs. โ Identifiers only.
- Events that are commands in disguise.
SendSmsEventon a bus couples the producer to one consumer. โ Publish facts (TransactionPosted); let Notifications decide to send an SMS. - Ignoring
FailedEntryCountonPutEvents(orFailedon SQS batch sends). โ Check partial failures and retry only those entries. - One giant shared queue for every message type. A poison message of one type blocks or delays all others; IAM cannot separate them. โ One queue per consumer and purpose.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
PutEvents returns 200 but the tap queue stays empty | Rule pattern does not match (detail-type vs DetailType, case, or a typo in source) | Compare the pattern with the event; use EventBridge โ Rules โ Edit โ Test pattern with a sample event |
Rule matches (metric MatchedEvents rises) but queue still empty; FailedInvocations rises | Queue policy does not allow events.amazonaws.com from this rule | Fix the policy's aws:SourceArn; check the rule ARN includes the bus name |
FailedInvocations with a KMS-encrypted queue | EventBridge cannot use the queue's customer managed key | Allow events.amazonaws.com kms:GenerateDataKey and kms:Decrypt in the key policy |
AccessDeniedException on PutEvents | Your role lacks events:PutEvents on the bus ARN | Grant it on the specific bus |
ExpiredTokenException / SSO session expired | SSO login timed out | aws sso login --profile learn-sandbox |
NullReferenceException reading response.Entries | SDK v4 collections default to null | Use ?. and ?? [], as in the sample |
| Events arrive minutes late | Target throttling and retries, or a consumer backlog | Check ThrottledRules, the queue's ApproximateAgeOfOldestMessage, and consumer concurrency |
Cost and cleanup
Cost: free at lab volumes. Custom-bus events published with PutEvents cost about $1.00 per million (each 64 KB chunk counts as one event) โ VERIFY current price; SQS includes 1 million requests per month free, permanently. A few dozen test events cost nothing measurable. None of these resources bills by the hour.
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
# Remove the target before the rule, and the rule before the bus
aws events remove-targets --rule hl-m08-l1-tap-rule --event-bus-name hl-m08-events --ids tap
aws events delete-rule --name hl-m08-l1-tap-rule --event-bus-name hl-m08-events
aws events delete-event-bus --name hl-m08-events
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m08-l1-tap --query QueueUrl --output text)"
rm -f tap-policy.json
# Confirm nothing tagged for this lab remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m08-l1 \
--query 'ResourceTagMappingList[].ResourceARN'The last command should print []. (The tagging API can lag by a few minutes after deletion.) If you plan to go straight on to the module project, you may keep hl-m08-events; the project's CDK stack creates its own resources and expects the name to be free, so delete it before deploying the project.
Exercises
- ๐ข Guided: Complete the six-flow classification in Step 1. For each, write the leaf of the decision tree you reached and the deciding question.
- ๐ก Challenge: Extend the console app to publish a batch of 10
TransactionPostedevents in onePutEventscall, deliberately make one entry invalid (for example, an emptyDetail), and retry only the failed entries with exponential backoff, at most three times. - ๐ด Live problem: A ticket from the Rewards team: "We subscribed a Lambda function directly to the
TransactionPostedSNS topic. During last night's deployment the function was failing for 40 minutes and we lost about 6,000 events. Fraud, who use a queue, lost none. Can messaging be made reliable?" Write the reply: explain why the two teams saw different outcomes, what you would change, and how to recover the lost 6,000 events given that EventBridge has an archive on the bus.
Assignment
Write a one-page integration decision record (ADR) for Harbourline's "account frozen" event: who produces it, who consumes it (at least Notifications, Cards and Fraud), which service carries it and why (cite the decision tree), the event envelope with a sample payload containing no personal data, retention, and the IAM boundaries. Commit it to your portfolio repository as docs/adr/0001-account-frozen-event.md.
Solutions
- Balance shown in the app โ synchronous API. The user waits for the answer (question 1 = yes).
- SMS and email after posting โ EventBridge (or SNS) fan-out to one SQS queue per channel. Many consumers, no strict ordering, nobody waits. EventBridge if you also want content routing and archive; SNS with filter policies is equally defensible for pure notification fan-out โ the module project uses both (EventBridge rule โ SNS topic โ filtered SQS queues).
- SEPA Instant with compensation โ Step Functions (question 2 = yes). It has branching, waiting and undo steps. Use Express only if the whole flow fits inside five minutes and you accept at-least-once step execution; SEPA Instant's ten-second target fits, but many banks still choose Standard for the exactly-once step semantics and execution history.
- Card telemetry at 3,000 records/s, two consumers, replay โ Kinesis Data Streams (question 4 = yes, no Kafka requirement). Partition key = card token for per-card order.
- Treasury JMS feed โ Amazon MQ (question 3 = yes), with a bridge into EventBridge for new consumers.
- 40,000 PDF statements โ SQS standard queue. One kind of worker, independent jobs, no order needed; scale workers on
ApproximateNumberOfMessagesVisible.
Why: the first "yes" in the tree wins. Most wrong answers come from skipping question 1 ("does anyone need the answer now?").
using Amazon.EventBridge;
using Amazon.EventBridge.Model;
public static class BatchPublisher
{
public static async Task<int> PublishAsync(IAmazonEventBridge client, List<PutEventsRequestEntry> entries, CancellationToken ct)
{
var pending = entries;
for (var attempt = 0; attempt < 4 && pending.Count > 0; attempt++)
{
if (attempt > 0)
{
// Full jitter: wait a random time between 0 and 200 ms * 2^attempt
var cap = 200 * (1 << attempt);
await Task.Delay(Random.Shared.Next(0, cap), ct);
}
var response = await client.PutEventsAsync(new PutEventsRequest { Entries = pending }, ct);
var results = response.Entries ?? [];
// Results come back in the same order as the request entries.
pending = pending
.Where((_, i) => i < results.Count && results[i].ErrorCode is not null)
.ToList();
}
return pending.Count; // entries still failing after all retries
}
}Why: PutEvents is partially successful by design. Retrying the whole batch would publish the good entries twice; consumers would cope (they are idempotent), but you would pay twice and muddy metrics. The response entries are positional, so the index links each result to its request entry. A permanently invalid entry (empty Detail) will still fail after retries โ log it with its correlation ID and raise an alarm rather than retrying forever.
Why the outcomes differed. SNS pushes to Lambda asynchronously. When the function kept failing, the asynchronous invocation retries ran out and, with no on-failure destination or DLQ configured, the events were discarded. Fraud's SQS queue simply held the messages until its consumer recovered โ the queue decouples availability.
What to change.
- Subscribe an SQS queue (
hl-m08-rewards) to the topic and let Rewards' Lambda consume from the queue with an event source mapping. - Give that queue a DLQ with
maxReceiveCountof at least 5 and an alarm on DLQ depth. - Use partial batch responses (
ReportBatchItemFailures) so one bad message does not retry the whole batch. - Make the handler idempotent on
eventId, because the replay below will resend events Rewards may already have processed.
How to recover. Start an EventBridge replay from the bus archive for the 40-minute window, targeting only the Rewards rule (replays can be limited to specific rules). Rewards' idempotency check skips anything it already handled. Record the incident and the replay in the change log.
Knowledge check
Answer: Format coupling. Producer and consumer must still agree on the message structure. Queues remove time coupling (the consumer can be down), location coupling (the producer only knows the queue) and rate coupling (bursts are buffered).
Answer: An EventBridge archive (replay) or a Kinesis/MSK stream within its retention. An SQS queue cannot: messages are deleted after successful processing, and there is no replay.
Answer: A command โ imperative, with one intended handler. It belongs on a queue owned by the Notifications Service. Publishing it to an event bus suggests other systems might react to "please send an SMS", which is rarely what you mean.
Answer: To make delivery durable and controllable. The queue stores messages while the consumer is failing or throttled, gives you a dead-letter queue and redrive, and lets you set batch size and maximum concurrency. Direct SNS โ Lambda uses asynchronous invocation, which retries for a limited time and then drops the event unless a DLQ or destination is configured.
Answer: 1 MiB (1,048,576 bytes). Each 64 KB chunk is billed as one request, so a 1 MiB message costs 16 requests. Practice material that says 256 KB is out of date.
Interview questions
- "When would you not use messaging between two services?" โ When the caller needs the result to continue (balance check, authorisation), when latency budgets are tight and the dependency is reliable, or when the added operational complexity outweighs the decoupling. I'd keep it synchronous but add timeouts, retries with jitter and a circuit breaker.
- "Explain the difference between SNS and EventBridge." โ Both are push-based pub/sub. SNS is a high-throughput fan-out of the same message to subscribers, with attribute or payload filter policies and FIFO topics. EventBridge routes events by matching rules on content, integrates SaaS and AWS service events, and adds a schema registry, archive/replay, Pipes and cross-account buses. I default to EventBridge for domain events and use SNS for very high-volume fan-out or SMS/email/mobile endpoints.
- "How do you stop a slow downstream system from taking down your API?" โ Remove temporal coupling: record the fact, publish an event, return. The downstream reads from its own queue at its own pace. Then bound what remains synchronous with timeouts and bulkheads.
- "What would you put in an event payload in a bank?" โ A unique event ID, type, version, occurrence time, correlation ID and business facts needed for routing and processing. Identifiers instead of personal data; consumers fetch details through authorised APIs. Money as decimal plus currency.
- "Can you get exactly-once processing on AWS?" โ Not end to end from the transport alone. Services give at-least-once delivery, some with deduplication windows. Exactly-once effects come from idempotent consumers, typically a conditional write on an idempotency key in DynamoDB.
Certification practice
A bank's payment API calls an SMS gateway synchronously after each payment. When the gateway slows down, the payment API times out. Notifications may be delayed by up to 30 minutes but must not be lost. Which change is MOST resilient?
A. Increase the payment API's HTTP client timeout to 60 seconds. B. Publish a message to an Amazon SQS queue after the payment is stored, and have a separate consumer send SMS messages. C. Invoke an AWS Lambda function synchronously to send the SMS. D. Add an Application Load Balancer in front of the SMS gateway.
Answer: B. The queue removes temporal coupling and stores messages durably while the gateway is slow. A makes threads wait longer and worsens exhaustion. C is still synchronous. D balances load but does not decouple or buffer.
A developer must deliver each AccountFrozen event to three teams. One team only wants events where reason is fraud. The security team also wants to replay last week's events after deploying a new rule. Which approach requires the LEAST custom code?
A. Amazon SQS standard queue polled by all three teams. B. Amazon EventBridge custom bus with one rule per team and an archive on the bus. C. Amazon Kinesis Data Streams with three consumers that filter in code. D. Amazon SNS standard topic with three email subscriptions.
Answer: B. Rules filter on detail.reason, each team gets its own target, and archive/replay is built in. A delivers each message to only one consumer. C needs filtering and checkpointing code and would need sufficient retention. D has no replay, and email is not a system integration.
Card-authorisation telemetry arrives at 5,000 records per second. A fraud model and a data-lake loader must both read every record in order per card, and the fraud team must reprocess the last 3 days after retraining. Which service fits BEST?
A. Amazon SQS FIFO queue with the card token as MessageGroupId.
B. Amazon Kinesis Data Streams with the card token as the partition key and extended retention.
C. Amazon SNS FIFO topic.
D. AWS Step Functions Express workflows.
Answer: B. Kinesis supports multiple independent consumers, per-key ordering and replay within retention. A delivers each message to one consumer and deletes it, so there is no reprocessing. C fans out but does not retain history. D orchestrates calls; it is not a log.
A data engineer ingests transaction events from an Amazon EventBridge bus into an S3 data lake through a rule and Amazon Data Firehose. A bug in a transformation corrupted 6 hours of data. What should the engineer do to re-ingest those events with the least effort?
A. Ask every producer to resend events for the 6-hour window. B. Start an EventBridge replay from an archive on the bus for that window, targeting the ingestion rule. C. Read the events again from the rule's SQS dead-letter queue. D. Restore the S3 bucket from versioning.
Answer: B โ provided an archive was enabled before the incident, which is why banks turn archives on from day one. A is slow and error-prone. C only contains events that failed delivery. D restores the corrupted output, not the source events.
Completion checklist
- I can explain time, location, rate and format coupling to a colleague
- I can say whether a message is a command, an event or a document
- I can walk the decision tree for a new integration and justify the leaf
- I designed an event envelope with no personal data
- (Optional) I published an event with .NET and saw it in the tap queue
- I ran the cleanup commands and the tag check returned
[]
Key takeaways
- Asynchronous messaging removes time, location and rate coupling; it does not remove format coupling or the need for idempotency.
- Queue (SQS), topic (SNS), bus (EventBridge), stream (Kinesis/MSK) and workflow (Step Functions) are different tools; Amazon MQ exists for protocol compatibility.
- Walk the decision tree: synchronous need first, then workflow, legacy protocol, replay/order, and finally one consumer vs many.
- In banking, put an SQS queue with a DLQ in front of almost every consumer, and keep personal data out of events.
- Limits have moved: SQS and EventBridge now accept payloads up to 1 MiB / 1 MB; check before trusting old material.
Further reading
- Choosing between Amazon SQS, Amazon SNS and Amazon EventBridge (AWS decision guide)
- What is Amazon EventBridge?
- What is Amazon SNS?
- Amazon SQS message quotas
- What is AWS Step Functions?
- What is Amazon Kinesis Data Streams?
- Event-driven architecture (AWS overview)
- More room to build: serverless services now support payloads up to 1 MB (AWS Compute Blog)
- Migrating to version 4 of the AWS SDK for .NET
- Next: Amazon SQS in depth
Independent learning resource โ not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Module 8: Messaging, Events & Microservices
Learn how AWS services pass messages and events between systems reliably: SQS, SNS, EventBridge, Step Functions, Kinesis, MSK and Amazon MQ. You apply retries, dead-letter queues, idempotency, sagas and the outbox pattern in .NET, and build a banking notification system that survives duplicates and failures.
Amazon SQS in depth
Configure Amazon SQS standard and FIFO queues for banking workloads โ visibility timeout, long polling, batching, message groups, deduplication and encryption โ and build a production-shaped .NET 10 consumer loop with batch deletes and a visibility heartbeat.