Module 8: Messaging, Events & Microservices
Learn how AWS services pass messages and events between systems reliably: SQS, SNS, EventBridge, Step Functions, Kinesis, MSK and Amazon MQ. You apply retries, dead-letter queues, idempotency, sagas and the outbox pattern in .NET, and build a banking notification system that survives duplicates and failures.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS.Messaging 1.6.0 · Powertools for AWS Lambda (.NET) 3.2.x · Region us-east-1. Verified against official sources on 2026-11-18.
Learn how AWS services pass messages and events between systems reliably: SQS, SNS, EventBridge, Step Functions, Kinesis, MSK and Amazon MQ. You apply retries, dead-letter queues, idempotency, sagas and the outbox pattern in .NET, and build a banking notification system that survives duplicates and failures.
| Estimated time | About 22 hours (lessons 13 h · hands-on exercises 5 h · project 4 h) |
| Priority for you | High — priority 2 (system integration) |
| Level | Intermediate → advanced |
| Certifications | DVA-C03 (primary) · SAA-C03 · DEA-C01 |
| Cost profile | 🟢 SQS, SNS, EventBridge, Lambda, Step Functions at lab volumes · 🟡 Kinesis · 🔴 Amazon MQ, MSK |
Why this matters in banking
Payments, card authorisations and customer notifications cross many systems, and every hop can fail, time out or deliver twice. A queue consumer that is not idempotent posts the same debit twice; a missing dead-letter queue lets one malformed message block a whole payment run; an unordered event stream shows a customer a balance that never existed. Asynchronous design is how banks absorb peaks (salary day, Black Friday) and isolate failures, but only if duplicates, ordering and compensation are designed in from the start. This module gives you those patterns with .NET code you can reuse.
From .NET/Azure to AWS
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| MSMQ / Service Bus queues | Amazon SQS (standard or FIFO) | No broker to run; FIFO gives ordering per message group; visibility timeout replaces peek-lock |
| Service Bus topics + subscriptions | Amazon SNS with subscription filter policies | SNS pushes to SQS, Lambda, HTTP, email; typical pattern is SNS → SQS fan-out |
| Azure Event Grid | Amazon EventBridge | Rules match on event content; archive/replay and Pipes built in |
| Durable Functions | AWS Step Functions | JSON/visual state machines; Standard (long-running) vs Express (high-volume) |
| Event Hubs | Kinesis Data Streams / Amazon MSK | Shards vs Kafka partitions; replay by position |
| IBM MQ / ActiveMQ on-premises | Amazon MQ | Managed ActiveMQ or RabbitMQ for JMS/AMQP legacy integration |
| MassTransit / NServiceBus | AWS.Messaging framework | AWS's .NET library for publishing and handling SQS/SNS/EventBridge messages |
| Service Bus duplicate detection | SQS FIFO deduplication + your own idempotency store | Window-limited dedup; durable idempotency lives in DynamoDB |
Learning objectives
By the end of this module you can:
- Choose between SQS, SNS, EventBridge, Kinesis, MSK and Amazon MQ for a given integration and justify the choice.
- Configure SQS standard and FIFO queues with suitable visibility timeouts, long polling, batching, dead-letter queues and redrive.
- Build SNS fan-out with filter policies and EventBridge rules with schemas, archive/replay and Pipes.
- Implement retries with exponential backoff and jitter, and make every consumer idempotent with a DynamoDB idempotency table or Powertools.
- Explain at-least-once vs exactly-once delivery and event ordering, and what each means for a ledger.
- Model a payment as a saga with compensation in Step Functions, and publish events reliably with the outbox pattern in EF Core.
- Design microservice boundaries (bounded contexts, domain events) and service discovery with Cloud Map or ECS Service Connect.
Prerequisites
- Module 3: .NET on AWS — Lambda, ECS, Worker Services and DI
- Module 6: Databases — DynamoDB conditional writes and EF Core transactions
- Recommended: Module 4: APIs for idempotency keys at the API edge
How the pieces fit
Lessons
The order follows the dependencies between lessons: the building blocks first (queues, topics, buses), then the reliability toolkit (retries, delivery guarantees, idempotency), then workflows and streaming, then the architecture patterns that tie them together.
- Asynchronous communication and event-driven architecture — queues vs pub/sub vs event buses vs streams, with a "which service when?" decision tree.
- Amazon SQS in depth — standard vs FIFO, visibility timeout, long polling, batching, message groups and the .NET consumer loop.
- Dead-letter queues and redrive — poison messages,
maxReceiveCount, partial batch responses, redrive to source and alarms on DLQ depth. - Amazon SNS: fan-out and message filtering — topics, fan-out to SQS and Lambda, filter policies, delivery retries and FIFO topics.
- Amazon EventBridge: buses, rules, schemas and Pipes — buses, rules, the schema registry, archive and replay, Pipes and the enhanced custom event bus.
- Retry strategies — exponential backoff with jitter, SDK retry modes and avoiding retry storms.
- Delivery guarantees and ordering — at-least-once vs exactly-once, FIFO groups, sequence numbers and out-of-order events.
- Idempotency in .NET with DynamoDB and Powertools — idempotency keys, conditional writes, TTL and the Powertools for AWS Lambda (.NET) idempotency utility.
- Distributed transactions and the saga pattern — orchestration vs choreography, compensation and a Step Functions payment saga.
- AWS Step Functions: workflows, retries and callbacks — Standard vs Express workflows, retries and catch, task-token callbacks, Map and Distributed Map.
- The outbox pattern with EF Core — writing state and events atomically, relaying to EventBridge and handling relay failures.
- Kinesis Data Streams and Amazon MSK — streams vs queues, shards and partitions, ordering and replay.
- Amazon MQ: bridging legacy JMS and AMQP systems — when MQ-style systems must stay, and bridging them to native services.
- Microservices communication and service discovery — bounded contexts, queries vs commands vs events, safe synchronous calls and discovery with Cloud Map and ECS Service Connect.
- AWS.Messaging for .NET — publishers, handlers, the message envelope and hosting handlers in a Worker Service or Lambda.
Hands-on code and module project
Module 8 has no separate lab walkthrough pages. The hands-on work sits inside the lessons, backed by two code folders in the repository. Both run their tests without an AWS account.
| Folder | What it holds | Used by |
|---|---|---|
labs/m08-notification-system/ | The module project: EventBridge → SNS → SQS → Lambda notification system with idempotent handlers, backoff, a DLQ, redrive, failure-injection scripts and RUNBOOK.md (41 tests) | Module project, plus the SQS, DLQ and EventBridge lessons |
labs/m08-messaging-services/ | Runnable, tested code for the service lessons: SNS publisher and filter-policy checks, EventBridge publisher, Step Functions definition and client, Kinesis shard routing and producers, Amazon MQ translator and bridge (30 tests) | SNS, EventBridge, Step Functions, Kinesis and MSK, Amazon MQ |
Hands-on topics, each inside the named lesson with its own cost label:
- SQS standard vs FIFO from .NET: send, receive, batch, change visibility, see deduplication — Amazon SQS in depth.
- Poison-message drill: a handler that rejects bad input, a DLQ that catches it, a redrive after the fix — Dead-letter queues and redrive.
- SNS → SQS fan-out with filter policies, then an EventBridge rule with archive and replay — SNS and EventBridge.
- Payment saga with compensation in Step Functions — Saga pattern and Step Functions.
- Outbox pattern: EF Core writes the payment and the event in one transaction and a relay publishes it — Outbox pattern (tests run on SQLite).
- Kinesis ordering and replay (🟡 deploy, test and delete within an hour) — Kinesis and MSK.
Module project — banking transaction notification system. Using .NET, SQS, SNS, EventBridge and Lambda, notify fictional customers when a transaction posts. Demonstrate reliable processing, duplicate prevention (idempotency table), retries with backoff, a poison message landing in a DLQ, redrive after a fix, and recovery after a downstream outage. It includes a failure-injection script and a short runbook.
When you finish, keep the Module 8 cheat sheet handy for revision.
Cost profile
| Part | Label | Free alternative |
|---|---|---|
| SQS, SNS, EventBridge, Lambda, Step Functions at lab volumes | 🟢 free | Within the always-free allowances for small test volumes ⚠ VERIFY current allowances; AWS Emulator or SAM local for offline runs |
| DynamoDB idempotency table | 🟢 free | DynamoDB Local or the always-free table allowance |
| Kinesis Data Streams | 🟡 credits | Shard-hours bill while the stream exists — deploy-and-destroy within an hour |
| Amazon MQ broker | 🔴 paid | Design-only, or ActiveMQ/RabbitMQ in Docker locally |
| Amazon MSK | 🔴 paid | Design-only, or Kafka in Docker locally |
| ECS Service Connect / Cloud Map demo on Fargate | 🟡 credits | Deploy-and-destroy; or design-only with the Module 3 Fargate stack |
Certification alignment
| Exam | Domains (weight) | What this module covers |
|---|---|---|
| DVA-C03 | Development (30%) · Troubleshoot (22%) | SQS/SNS/EventBridge APIs, Lambda event sources, Step Functions, DLQs and retries |
| SAA-C03 | Resilient (26%) | Decoupling with queues and events, loosely coupled architectures |
| DEA-C01 | Ingestion (34%) | Kinesis and MSK streaming ingestion concepts |
The domain labels are short forms; see each exam page for the official wording.
Service availability notes
Services that changed — verified 2026-11-17
Amazon SNS Message Data Protection is in maintenance (announced 2026-03-31) and closed to new customers. Do not rely on it to detect or mask personal data in messages; keep PII out of event payloads (send identifiers and let consumers fetch details) and mask in your own code.
SQS, SNS, EventBridge, Step Functions, Kinesis, MSK and Amazon MQ are all fully available.
Design note for banks
Notification content (balances, partial account numbers) is still personal data. Which channels may carry which data, and for how long messages may be retained in queues and archives, needs review by your data-protection and security teams.
Key AWS services
Amazon SQS · Amazon SNS · Amazon EventBridge (rules, schema registry, archive/replay, Pipes, Scheduler) · AWS Step Functions · Amazon Kinesis Data Streams · Amazon MSK · Amazon MQ · AWS Lambda · Amazon DynamoDB · AWS Cloud Map · Amazon ECS Service Connect · AWS.Messaging for .NET · Powertools for AWS Lambda (.NET)
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Module 7 cheat sheet
One-page revision summary for Data Engineering & ETL Pipelines: services, Azure mapping, CLI commands, limits, decision rules, exam traps and cost traps.
Asynchronous communication and event-driven architecture
Tell queues, topics, event buses, streams and workflows apart, decide which AWS service fits a Harbourline integration with a written decision tree, and publish your first well-formed domain event from .NET 10.