Skip to Content
Cloud Engineering Academy
Projects

Capstone 2: Event-driven transaction processing

A payment saga with compensation, a transactional outbox, EventBridge, SNS and SQS fan-out, and idempotent Lambda consumers with dead-letter queues, built on the tested Module 8 and Module 14 labs plus 13 end-to-end tests that prove money moves once and the customer is told once.

advancedโฑ 8 h ๐ŸŸข FreeDVA-C03SAA-C03SAP-C02

Assumes: .NET 10 LTS ยท AWS SDK for .NET v4 ยท xUnit v3 4.0.3 ยท Amazon.CDK.Lib 2.277.0 ยท EventBridge, SNS, SQS, Lambda dotnet10, Step Functions, DynamoDB ยท Region us-east-1. Verified against official sources on 2026-11-18.

What you are building

A payment is requested, screened for sanctions, funded, sent to a payment scheme and settled by a saga with compensation. When, and only when, it settles, a TransactionPosted event is announced reliably through a transactional outbox, routed through EventBridge, SNS and SQS, and turned into at most one SMS and one e-mail per customer by idempotent consumers that survive duplicates, poison messages and provider outages.

The point of the capstone is behaviour under failure. Networks time out, messages arrive twice, a partner goes down, and a "no answer" from the payment scheme is not a "no". The project shows how to build a system where those things happen and money still moves exactly once.

Educational project

Sanctions screening, scheme connectivity and customer consent are simulated. A real payments system needs contracts, vetted providers, regulated-process controls and legal review that this project does not contain. All data is fictional.

Learning objectives

By the end you can:

  • Explain the saga's pivot: compensate before the scheme accepts, move forward after, and treat a timeout as unknown rather than failed.
  • Show why a transactional outbox removes the dual-write problem, and what a relay crash does (a duplicate publish, absorbed by idempotent consumers).
  • Layer idempotency: the payment command (key plus body hash) and the notification consumer (business key, not message ID).
  • Wire EventBridge, SNS filter policies and SQS with dead-letter queues, alarms and a redrive runbook.
  • Run failure drills and read their output, offline and in a sandbox.

The architecture

The repository adds saga-state and sequence diagrams (happy path, duplicate and relay crash, provider outage) and the security boundaries in projects/p2-event-driven-transactions/docs/architecture.md.

Prerequisites and the lessons it builds on

Start from the Module 8 overview if you have not completed that module.

What is in the repository

WhereWhat
labs/m14-payment-saga-failuresThe saga in plain C# (PaymentSaga), in-memory ledger, idempotency store, scheme and sanctions interfaces, a pacs.008 writer, 22 tests, an optional Step Functions proof of concept (CDK)
labs/m08-notification-systemEventBridge, SNS, SQS, Lambda handlers, DynamoDB idempotency, DLQs, alarms, simulator, failure-injection scripts, runbook, 41 tests
labs/m08-messaging-servicesSNS, EventBridge, Step Functions and ASL linter building blocks, 30 tests
projects/p2-event-driven-transactionsThe capstone: src/Harbourline.Capstone.Transactions (outbox, relay, settlement recorder), 13 end-to-end tests, docs/ (architecture, five ADRs, security controls, testing, monitoring, cost and teardown), checklist.md

The glue code, in one table

ClassJob
PaymentEventRecorderRuns the saga, and only for a Settled payment writes a TransactionPosted event to the outbox. Also resolves unknown outcomes and recovers missing events
InMemoryOutboxThe outbox table. The event ID is derived from the payment, so writing the same fact twice keeps one row
OutboxRelayPublishes pending rows and marks them published only after the publish succeeded: at least once
SimulatorPublisherHands the event to the m08 notification simulator (EventBridge, SNS, SQS and the real processors)

Build path

Run the three suites (free). m14 (22 tests), m08 notification (41 tests, one skipped) and the capstone (13). Read EndToEndTests.cs first: each test is a sentence about money.

Run the offline drill. dotnet run --project labs/m08-notification-system/src/Harbourline.Notifications.Cli -- simulate narrates duplicates, poison messages and an outage with recovery.

Break it on purpose. Remove the Settled check in PaymentEventRecorder and see which tests fail. Remove the duplicate guard in the outbox and watch the duplicate-retry test.

Deploy the consumer side (optional, cents). cdk deploy HlM08-Notifications and run inject-failures.ps1 for duplicates, poison, outage on and off, then redrive.ps1.

Deploy the saga proof of concept (optional). The m14 CDK stack runs the saga on Step Functions Express with DynamoDB. Compare each execution (clear, hit, down, reject, repeated key) with the matching unit test.

Design the production outbox. Write the DynamoDB transaction, the Stream and the EventBridge Pipe, as ADR 2 describes, and add it to your repository.

Tear down and check nothing tagged Project=cea-lab remains.

What was verified when this page was written

CheckResult
Capstone end-to-end tests (dotnet test in Docker, .NET SDK 10)13 passed, 0 failed
Lab m14 and m08 suites22 and 41 tests per their READMEs (the capstone's project references build both libraries)
Not runA deployment of the saga or the consumers for this capstone, and the production outbox (DynamoDB Streams and Pipes), which is design only

The failure properties it proves

TestProperty
Settled payment notifies each opted-in channel exactly onceOne SMS and one e-mail, minimal text, money moved once
A duplicate client retryOne payment, one outbox row, one set of notifications
Sanctions hit, provider down, insufficient funds, scheme rejectionNo event, no net money movement
A relay that dies after publishingA duplicate publish, but the customer is notified once
A notification outageSettlement unaffected; the backlog drains after recovery; DLQs stay empty
An unknown outcomeAnnounced only after reconciliation resolves it
A settled payment whose event was never writtenRecovered by the reconciliation job, once

Security controls

The repository's docs/security-controls.md lists controls for money integrity, messaging integrity, data protection (events carry identifiers and the minimum text, never full account numbers), access control and detection. Honest gaps: the offline outbox, ledger and idempotency stores are in memory; the lab encrypts queues with SQS-managed keys and the topic is not encrypted with a customer managed key (an acknowledged shortcut with the production alternative); a handler that dies after the provider accepted a message can send it twice, a documented and accepted window.

Cost and teardown

Offline: free. Deployed at lab volume: cents (SQS free tier, EventBridge about USD 1 per million events, SNS about 0.50 per million publishes, Step Functions Standard about USD 25 per million transitions with 4,000 free a month, alarms about 0.10 each per month; all โš  VERIFY on the pricing pages). At production scale the orchestrator usually dominates the bill, so the sizing question is how many states each payment uses. Run scripts/teardown.ps1 in the m08 lab, cdk destroy the m14 stack, and check the tag query; also look for EventBridge archives and log groups by hand.

Acceptance criteria

  • The three test suites pass on a clean machine and you can explain the one skipped test.
  • You can narrate each failure drill: duplicate, poison, outage, redrive, relay crash, unknown outcome.
  • Your README states the pivot rule and why a timeout is not a failure.
  • You wrote the production outbox design (or implemented it) and its test.
  • Alarms and a runbook exist for the DLQ and for the OutcomeUnknown state.
  • Everything is torn down.

Portfolio-ready checklist

The full list is projects/p2-event-driven-transactions/checklist.md. The essentials:

  • Five ADRs: orchestration, outbox, two-layer idempotency, fan-out, pivot and fail-closed.
  • Diagrams: big picture, saga states, three sequences, security boundaries.
  • One failure test of your own, for a partner behaviour you invented.
  • Runbooks for DLQ, stuck payments and replay.
  • Extensions: DynamoDB TransactWriteItems plus Streams and a Pipe, a manual-review Standard workflow with a task token, a second consumer.

Exercises

๐ŸŸข Guided. Add a test where the customer has opted into no channels. What should the outbox hold and what should the consumers do?

๐ŸŸก Challenge. Implement IEventPublisher for EventBridge PutEvents with the AWS SDK for .NET v4. Check FailedEntryCount in the response and treat a failed entry as a failed publish. Test it with a fake client.

๐Ÿ”ด Live problem. On salary day a customer says they received four SMS messages for one payment and the DLQ is empty. List what you would check first (publisher retries, the idempotency key, TTL, a changed event ID, two handlers with different keys) and how each would show in the logs.

Interview questions

  1. What is the dual-write problem, and how does an outbox solve it?
  2. A scheme call times out. What do you do, and why must you not release the hold?
  3. Why is the notification idempotency key the transaction ID and channel rather than the SQS message ID?
  4. What guarantees does EventBridge to SNS to SQS give, and what must the consumer provide?
  5. When would you choose Express over Standard workflows, and what does each cost?

Key takeaways

  • Money correctness comes from idempotency, not from hoping messages arrive once.
  • Compensation has a pivot. After it, only move forward.
  • Announce a fact only when it is true: write the event in the same transaction as the fact.
  • Monitor age and depth, not just errors: failure in event systems is quiet.

Independent learning resource โ€” not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page