The outbox pattern with EF Core
Solve the dual-write problem: write business state and its events in one EF Core transaction, relay them to EventBridge with a leased, batch-aware BackgroundService, and handle relay crashes, partial PutEvents failures, poison rows and cleanup — with tests that run on SQLite.
Assumes: .NET 10 LTS · EF Core 10.0.15 · Npgsql.EntityFrameworkCore.PostgreSQL 10.0.3 · AWS SDK for .NET v4 (AWSSDK.EventBridge 4.1.0.2) · xunit.v3 4.0.3 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain the dual-write problem and why "save, then publish" (or "publish, then save") loses or invents events.
- Implement a transactional outbox in EF Core 10:
- domain events collected by a
SaveChangesInterceptor; - events stored in an
OutboxMessagestable in the same transaction as the business change.
- domain events collected by a
- Build an outbox relay as a .NET
BackgroundServicethat:- leases rows safely across several instances;
- publishes to Amazon EventBridge with
PutEvents; - handles partial batch failures;
- retries with backoff and parks poison rows.
- Explain why the relay is at-least-once, and how consumers stay correct (event IDs and idempotency).
- Compare the outbox with change data capture alternatives: DynamoDB Streams with EventBridge Pipes, and database log-based CDC.
Prerequisites
- The EventBridge lesson earlier in this module (see the module overview)
- Delivery guarantees and ordering and Idempotency in .NET
- Module 6: Databases — EF Core on Aurora PostgreSQL / RDS
- Tools: .NET 10 SDK; Docker (optional, for PostgreSQL); AWS CLI v2 with
learn-sandboxfor the optional EventBridge step
Why this matters in banking
Harbourline's Payments Service saves a new payment to Aurora PostgreSQL and then publishes PaymentReceived to EventBridge, so screening, fraud scoring and notifications can start. The code looked fine:
await db.SaveChangesAsync(ct); // 1. commit the payment
await eventBridge.PutEventsAsync(evt, ct); // 2. publish the eventDuring a routine deployment, ECS stopped tasks while requests were in flight. Forty-one payments were committed in step 1 but never published in step 2. Nothing downstream knew they existed. They sat in received status until a customer complained two days later that their rent hadn't arrived.
The team tried swapping the order (publish first, then save). On the next database failover, events were published for payments that were then rolled back. Fraud scoring flagged transactions that never existed, and a customer received a "payment sent" SMS for a payment that wasn't.
This is the dual-write problem. You can't atomically write to two systems (a database and a message bus) without a shared transaction, and AWS gives you no transaction that spans Aurora and EventBridge. The transactional outbox turns two writes into one, then delivers the event reliably afterwards.
The idea in plain language
Instead of publishing the event directly, you write it into a table in the same database, in the same transaction as the business change. Either both the payment and its event are committed, or neither is.
A separate process, the relay, reads that table and publishes the events to EventBridge. When EventBridge confirms, the relay marks the row as published. If the relay crashes, the rows are still there, and it carries on after a restart.
Analogy: A bank branch doesn't phone head office every time it processes a cheque, because the call might fail halfway. Instead, the clerk writes each cheque into the day book at the same moment they stamp it. A courier collects the day book and delivers its entries to head office. If the courier's van breaks down, the entries are still in the book and go with the next courier. Head office might receive an entry twice if the courier isn't sure it was delivered, so head office checks the entry numbers. The day book is the outbox; the courier is the relay; the entry numbers are event IDs.
Three properties follow:
- No lost events. If the payment committed, its event is in the outbox.
- No phantom events. If the payment rolled back, so did its event.
- At-least-once publishing. The relay may publish an event, then crash before marking it published. After restart it publishes it again. Consumers must deduplicate by event ID.
What problem the AWS services solve (and don't)
| Piece | Role | What it doesn't do |
|---|---|---|
| Aurora PostgreSQL / RDS (or SQL Server on RDS) | Holds business tables and the outbox table; one local ACID transaction covers both | Publish anything by itself |
| EF Core 10 | SaveChangesAsync wraps all tracked changes in one transaction by default | Know about your message bus |
Amazon EventBridge PutEvents | Receives events and routes them to rules/subscribers; returns a per-entry result | Join your database transaction; deduplicate on a Classic bus |
| ECS Fargate service / Worker Service | Hosts the relay continuously (low latency) | Guarantee a single instance; your lease logic does that |
| EventBridge Scheduler + Lambda | Alternative relay host that runs every minute | Sub-minute latency |
| DynamoDB Streams + EventBridge Pipes | Built-in CDC for DynamoDB: every item change flows to a target without your own relay | Help when your data is in a relational database |
Limits to design around:
- Classic bus:
PutEventsaccepts up to 10 entries per request ⚠ VERIFY current limit on your bus type. The newer EventBridge Custom Event Bus accepts up to 100. - Each entry has a size limit (check the current EventBridge quotas; keep events small and reference IDs).
PutEventscan partly succeed. The response'sFailedEntryCountand per-entryErrorCodetell you which entries failed. A request that "didn't throw" isn't a request where everything succeeded.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| NServiceBus / MassTransit transactional outbox | Your own EF Core outbox + relay (this lesson) | AWS.Messaging for .NET doesn't include an outbox at the time of writing ⚠ VERIFY; you own the table and relay |
TransactionScope spanning SQL Server and MSMQ (MSDTC) | Not available across Aurora and EventBridge | The outbox replaces the distributed transaction |
| SQL Server Change Data Capture / Debezium → Event Hubs | AWS DMS CDC, Debezium on MSK Connect, DynamoDB Streams + EventBridge Pipes | CDC publishes table changes, not intentional domain events, unless you capture an outbox table |
| Cosmos DB change feed → Azure Functions | DynamoDB Streams → Lambda / EventBridge Pipes | Same idea: the database's change log is the outbox |
| Hangfire / Quartz recurring job polling a table | BackgroundService on ECS, or EventBridge Scheduler → Lambda | Same polling idea; use leases so instances don't collide |
Architecture
Sequence, including the crash that causes a duplicate
Step-by-step implementation
Model domain events on the aggregate
Payment gets a private list of domain events. Business methods (Create, MarkScreened) add events. The aggregate knows nothing about EventBridge.
Add the outbox table
OutboxMessage has:
Id(UUID v7, time-ordered);Type,Source,AggregateId,Payload(JSON);OccurredAt;- delivery bookkeeping:
PublishedAt,Attempts,NextAttemptAt,LeaseToken,LeasedUntilandLastError.
Add an index on (PublishedAt, NextAttemptAt).
Convert domain events to outbox rows inside SaveChanges
A SaveChangesInterceptor runs just before EF Core saves. It collects events from tracked aggregates and adds OutboxMessage entities to the same context, so they join the same transaction automatically.
Write the relay
A BackgroundService loop:
- Lease a batch with a single atomic
UPDATE. - Read the rows it leased.
- Publish them with
PutEvents. - Mark the successful entries published, and schedule the failed ones with backoff.
- Sleep briefly when idle.
Test without AWS
Run the tests in Complete code on SQLite in-memory with a fake publisher that fails selected entries. They prove atomicity, partial-failure handling and lease safety.
Optional: publish to a real bus
Create the hl-m08-payments bus and a rule that sends everything to a CloudWatch Logs group. Run the service locally with the learn-sandbox profile and watch events arrive.
AWS Console instructions
The database side is code. The AWS side is small:
- Console → Amazon EventBridge → Event buses → Create event bus. Name
hl-m08-payments. Add the lab tags. (In the console this bus type is listed as Custom Event Bus - Classic if you choose the rules-and-targets model.) - Rules → Create rule on that bus: name
hl-m08-all-payments-to-logs, event pattern{"source":["harbourline.payments"]}. Target CloudWatch log group/aws/events/hl-m08-payments(create it, and set retention to 1 week). - Monitoring: on the bus's Monitoring tab, watch
PutEventsRequestSizeand the rule'sMatchedEvents. In CloudWatch, alarm onPutEventsFailedEntriesif your Region publishes it ⚠ VERIFY metric name. Also publish your own relay metrics (see Code explained). - IAM: the ECS task role needs only
events:PutEventson the bus ARN.
AWS CLI
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
aws events create-event-bus --name hl-m08-payments \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m08-outbox Key=Environment,Value=learn Key=Owner,Value=alex
aws logs create-log-group --log-group-name /aws/events/hl-m08-payments
aws logs put-retention-policy --log-group-name /aws/events/hl-m08-payments --retention-in-days 7
aws events put-rule --name hl-m08-all-payments-to-logs --event-bus-name hl-m08-payments \
--event-pattern '{"source":["harbourline.payments"]}'
aws events put-targets --rule hl-m08-all-payments-to-logs --event-bus-name hl-m08-payments \
--targets "Id=logs,Arn=arn:aws:logs:us-east-1:$(aws sts get-caller-identity --query Account --output text):log-group:/aws/events/hl-m08-payments"
# Note: EventBridge needs a CloudWatch Logs resource policy to write to the group; the console adds it for you.
# Publish one event by hand to see the per-entry result shape the relay must handle.
aws events put-events --entries '[{
"EventBusName":"hl-m08-payments","Source":"harbourline.payments","DetailType":"PaymentReceived",
"Detail":"{\"eventId\":\"0192d1c2-0000-7000-8000-0000000000e1\",\"paymentId\":\"PAY-0192d1c2-0000-7000-8000-000000000001\",\"amount\":\"1250.00\",\"currency\":\"GBP\"}"
}]'
# → {"FailedEntryCount":0,"Entries":[{"EventId":"..."}]}
# Read what arrived
aws logs tail /aws/events/hl-m08-payments --since 10mInfrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.Events.Targets;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.Logs;
using Constructs;
namespace Harbourline.Infra;
public sealed class OutboxStack : Stack
{
public OutboxStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
var bus = new EventBus(this, "PaymentsBus", new EventBusProps { EventBusName = "hl-m08-payments" });
// Debug sink: every payments event into a short-retention log group.
var logGroup = new LogGroup(this, "PaymentsEvents", new LogGroupProps
{
LogGroupName = "/aws/events/hl-m08-payments",
Retention = RetentionDays.ONE_WEEK,
RemovalPolicy = RemovalPolicy.DESTROY
});
new Rule(this, "AllPaymentsToLogs", new RuleProps
{
RuleName = "hl-m08-all-payments-to-logs",
EventBus = bus,
EventPattern = new EventPattern { Source = ["harbourline.payments"] },
Targets = [new CloudWatchLogGroup(logGroup)]
});
// Policy the Payments Service task role attaches: PutEvents on this bus only.
new ManagedPolicy(this, "RelayPublish", new ManagedPolicyProps
{
ManagedPolicyName = "hl-m08-outbox-relay-publish",
Statements =
[
new PolicyStatement(new PolicyStatementProps
{
Actions = ["events:PutEvents"],
Resources = [bus.EventBusArn]
})
]
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m08-outbox");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "alex");
}
}Complete code
1. Domain: aggregate with domain events
namespace Harbourline.Payments.Domain;
public interface IDomainEvent
{
Guid EventId { get; }
DateTime OccurredAtUtc { get; }
}
public abstract class AggregateRoot
{
private readonly List<IDomainEvent> _events = [];
public IReadOnlyList<IDomainEvent> DomainEvents => _events;
protected void Raise(IDomainEvent e) => _events.Add(e);
public void ClearDomainEvents() => _events.Clear();
}
public sealed record PaymentReceived(
Guid EventId, DateTime OccurredAtUtc, string PaymentId, string DebtorAccountId,
decimal Amount, string Currency, string Scheme) : IDomainEvent;
public sealed class Payment : AggregateRoot
{
private Payment() { } // EF Core
public string PaymentId { get; private set; } = default!;
public string DebtorAccountId { get; private set; } = default!;
public string CreditorName { get; private set; } = default!;
public string CreditorIban { get; private set; } = default!;
public decimal Amount { get; private set; }
public string Currency { get; private set; } = default!;
public string Scheme { get; private set; } = default!;
public string Status { get; private set; } = default!;
public static Payment Create(string debtorAccountId, string creditorName, string creditorIban,
decimal amount, string currency, string scheme, TimeProvider clock)
{
if (amount <= 0) throw new ArgumentOutOfRangeException(nameof(amount), "Amount must be positive");
var p = new Payment
{
PaymentId = $"PAY-{Guid.CreateVersion7()}",
DebtorAccountId = debtorAccountId,
CreditorName = creditorName,
CreditorIban = creditorIban,
Amount = amount,
Currency = currency,
Scheme = scheme,
Status = "received"
};
p.Raise(new PaymentReceived(Guid.CreateVersion7(), clock.GetUtcNow().UtcDateTime,
p.PaymentId, debtorAccountId, amount, currency, scheme));
return p;
}
}2. Persistence: outbox entity, DbContext and interceptor
namespace Harbourline.Payments.Persistence;
public sealed class OutboxMessage
{
public Guid Id { get; set; } // = domain EventId (UUID v7, time-ordered)
public string Type { get; set; } = default!; // EventBridge DetailType, e.g. "PaymentReceived"
public string Source { get; set; } = "harbourline.payments";
public string AggregateId { get; set; } = default!;
public string Payload { get; set; } = default!; // JSON detail
public DateTime OccurredAtUtc { get; set; }
// Delivery bookkeeping
public DateTime? PublishedAtUtc { get; set; }
public int Attempts { get; set; }
public DateTime NextAttemptAtUtc { get; set; }
public Guid? LeaseToken { get; set; }
public DateTime? LeasedUntilUtc { get; set; }
public string? LastError { get; set; }
public bool Parked { get; set; } // poison: stop retrying, alert a human
}using Harbourline.Payments.Domain;
using Microsoft.EntityFrameworkCore;
namespace Harbourline.Payments.Persistence;
public sealed class PaymentsDbContext(DbContextOptions<PaymentsDbContext> options) : DbContext(options)
{
public DbSet<Payment> Payments => Set<Payment>();
public DbSet<OutboxMessage> Outbox => Set<OutboxMessage>();
protected override void OnModelCreating(ModelBuilder b)
{
b.Entity<Payment>(e =>
{
e.ToTable("payments");
e.HasKey(p => p.PaymentId);
e.Property(p => p.Amount).HasPrecision(18, 2); // money: decimal(18,2), never double
e.Ignore(p => p.DomainEvents);
});
b.Entity<OutboxMessage>(e =>
{
e.ToTable("outbox_messages");
e.HasKey(m => m.Id);
e.Property(m => m.Type).HasMaxLength(128);
e.Property(m => m.AggregateId).HasMaxLength(64);
// The relay's query: unpublished, not parked, due now.
e.HasIndex(m => new { m.PublishedAtUtc, m.Parked, m.NextAttemptAtUtc });
});
}
}using System.Text.Json;
using Harbourline.Payments.Domain;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Diagnostics;
namespace Harbourline.Payments.Persistence;
/// <summary>Turns domain events into outbox rows in the SAME SaveChanges (and so the same transaction).</summary>
public sealed class DomainEventsToOutboxInterceptor : SaveChangesInterceptor
{
private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
public override ValueTask<InterceptionResult<int>> SavingChangesAsync(
DbContextEventData eventData, InterceptionResult<int> result, CancellationToken ct = default)
{
if (eventData.Context is not DbContext db) return ValueTask.FromResult(result);
var aggregates = db.ChangeTracker.Entries<AggregateRoot>()
.Select(e => e.Entity)
.Where(a => a.DomainEvents.Count > 0)
.ToList();
foreach (var aggregate in aggregates)
{
foreach (var e in aggregate.DomainEvents)
{
db.Set<OutboxMessage>().Add(new OutboxMessage
{
Id = e.EventId,
Type = e.GetType().Name,
AggregateId = aggregate is Payment p ? p.PaymentId : "unknown",
Payload = JsonSerializer.Serialize(e, e.GetType(), Json),
OccurredAtUtc = e.OccurredAtUtc,
NextAttemptAtUtc = e.OccurredAtUtc
});
}
aggregate.ClearDomainEvents();
}
return ValueTask.FromResult(result);
}
}3. The relay
using Amazon.EventBridge;
using Amazon.EventBridge.Model;
using Harbourline.Payments.Persistence;
namespace Harbourline.Payments.Outbox;
/// <summary>Result per input message, in the same order. Null error = published.</summary>
public interface IEventPublisher
{
Task<IReadOnlyList<string?>> PublishAsync(IReadOnlyList<OutboxMessage> batch, CancellationToken ct);
}
public sealed class EventBridgePublisher(IAmazonEventBridge eventBridge, string busName) : IEventPublisher
{
public async Task<IReadOnlyList<string?>> PublishAsync(IReadOnlyList<OutboxMessage> batch, CancellationToken ct)
{
var response = await eventBridge.PutEventsAsync(new PutEventsRequest
{
Entries = batch.Select(m => new PutEventsRequestEntry
{
EventBusName = busName,
Source = m.Source,
DetailType = m.Type,
Detail = m.Payload, // includes eventId for consumer deduplication
Time = m.OccurredAtUtc, // event time, not publish time
Resources = [m.AggregateId]
}).ToList()
}, ct);
// Entries come back in request order. An entry with an ErrorCode failed; others succeeded.
var entries = response.Entries ?? [];
return batch.Select((_, i) =>
i < entries.Count && string.IsNullOrEmpty(entries[i].ErrorCode)
? null
: $"{(i < entries.Count ? entries[i].ErrorCode : "MissingResult")}: {(i < entries.Count ? entries[i].ErrorMessage : "")}")
.ToList();
}
}using Harbourline.Payments.Persistence;
using Microsoft.EntityFrameworkCore;
namespace Harbourline.Payments.Outbox;
public sealed class OutboxRelayOptions
{
public int BatchSize { get; set; } = 10; // Classic bus PutEvents limit
public TimeSpan Lease { get; set; } = TimeSpan.FromSeconds(30);
public TimeSpan IdleDelay { get; set; } = TimeSpan.FromMilliseconds(500);
public int MaxAttempts { get; set; } = 12;
}
public sealed class OutboxProcessor(PaymentsDbContext db, IEventPublisher publisher, TimeProvider clock,
OutboxRelayOptions options)
{
/// <summary>Processes one batch. Returns how many rows it handled (0 = nothing due).</summary>
public async Task<int> ProcessBatchAsync(CancellationToken ct)
{
var now = clock.GetUtcNow().UtcDateTime;
var token = Guid.NewGuid();
var leasedUntil = now + options.Lease;
// 1. Pick candidate IDs (oldest first).
var candidates = await db.Outbox
.Where(m => m.PublishedAtUtc == null && !m.Parked && m.NextAttemptAtUtc <= now
&& (m.LeasedUntilUtc == null || m.LeasedUntilUtc < now))
.OrderBy(m => m.Id)
.Select(m => m.Id)
.Take(options.BatchSize)
.ToListAsync(ct);
if (candidates.Count == 0) return 0;
// 2. Lease them in ONE atomic UPDATE. A competing relay's UPDATE can only win rows we didn't.
await db.Outbox
.Where(m => candidates.Contains(m.Id) && (m.LeasedUntilUtc == null || m.LeasedUntilUtc < now))
.ExecuteUpdateAsync(s => s
.SetProperty(m => m.LeaseToken, token)
.SetProperty(m => m.LeasedUntilUtc, leasedUntil), ct);
// 3. Read back exactly the rows WE leased.
var batch = await db.Outbox.AsNoTracking()
.Where(m => m.LeaseToken == token)
.OrderBy(m => m.Id)
.ToListAsync(ct);
if (batch.Count == 0) return 0;
// 4. Publish. A thrown exception (network, throttling after SDK retries) fails the whole batch.
IReadOnlyList<string?> results;
try
{
results = await publisher.PublishAsync(batch, ct);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
results = batch.Select(_ => (string?)$"{ex.GetType().Name}: {ex.Message}").ToList();
}
// 5. Record outcomes. Only rows still under OUR lease are touched.
for (var i = 0; i < batch.Count; i++)
{
var id = batch[i].Id;
if (results[i] is null)
{
await db.Outbox.Where(m => m.Id == id && m.LeaseToken == token)
.ExecuteUpdateAsync(s => s
.SetProperty(m => m.PublishedAtUtc, now)
.SetProperty(m => m.LeaseToken, (Guid?)null)
.SetProperty(m => m.LeasedUntilUtc, (DateTime?)null), ct);
}
else
{
var attempts = batch[i].Attempts + 1;
var park = attempts >= options.MaxAttempts;
var delay = Backoff(attempts);
var error = results[i]!.Length > 1000 ? results[i]![..1000] : results[i]!;
await db.Outbox.Where(m => m.Id == id && m.LeaseToken == token)
.ExecuteUpdateAsync(s => s
.SetProperty(m => m.Attempts, attempts)
.SetProperty(m => m.NextAttemptAtUtc, now + delay)
.SetProperty(m => m.LastError, error)
.SetProperty(m => m.Parked, park)
.SetProperty(m => m.LeaseToken, (Guid?)null)
.SetProperty(m => m.LeasedUntilUtc, (DateTime?)null), ct);
}
}
return batch.Count;
}
/// <summary>Exponential backoff with full jitter: up to 1 s, 2 s, 4 s … capped at 5 minutes.</summary>
private static TimeSpan Backoff(int attempts) =>
TimeSpan.FromMilliseconds(Random.Shared.NextDouble() * Math.Min(300_000, 1000 * Math.Pow(2, attempts - 1)));
}
public sealed class OutboxRelay(IServiceScopeFactory scopes, OutboxRelayOptions options, ILogger<OutboxRelay> log)
: BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
try
{
await using var scope = scopes.CreateAsyncScope();
var processor = scope.ServiceProvider.GetRequiredService<OutboxProcessor>();
var handled = await processor.ProcessBatchAsync(stoppingToken);
if (handled == 0) await Task.Delay(options.IdleDelay, stoppingToken);
}
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
{
break;
}
catch (Exception ex)
{
// Database unavailable etc. Never let the loop die.
log.LogError(ex, "Outbox relay iteration failed");
await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
}
}
}
}4. Wiring it up
using Amazon.EventBridge;
using Harbourline.Payments.Domain;
using Harbourline.Payments.Outbox;
using Harbourline.Payments.Persistence;
using Microsoft.EntityFrameworkCore;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddSingleton(TimeProvider.System);
builder.Services.AddSingleton<DomainEventsToOutboxInterceptor>();
builder.Services.AddDbContext<PaymentsDbContext>((sp, o) => o
.UseNpgsql(builder.Configuration.GetConnectionString("Payments")) // from Secrets Manager / IAM auth in prod
.AddInterceptors(sp.GetRequiredService<DomainEventsToOutboxInterceptor>()));
builder.Services.AddSingleton<IAmazonEventBridge>(_ => new AmazonEventBridgeClient(new AmazonEventBridgeConfig
{
RegionEndpoint = Amazon.RegionEndpoint.USEast1,
RetryMode = Amazon.Runtime.RequestRetryMode.Standard
}));
builder.Services.AddSingleton<IEventPublisher>(sp =>
new EventBridgePublisher(sp.GetRequiredService<IAmazonEventBridge>(), "hl-m08-payments"));
builder.Services.AddSingleton(new OutboxRelayOptions());
builder.Services.AddScoped<OutboxProcessor>();
builder.Services.AddHostedService<OutboxRelay>();
var app = builder.Build();
app.MapPost("/payments", async (CreatePayment req, PaymentsDbContext db, TimeProvider clock, CancellationToken ct) =>
{
var payment = Payment.Create(req.DebtorAccountId, req.CreditorName, req.CreditorIban,
req.Amount, req.Currency, req.Scheme, clock);
db.Payments.Add(payment);
await db.SaveChangesAsync(ct); // payment row + outbox row, one transaction
return Results.Accepted($"/payments/{payment.PaymentId}", new { payment.PaymentId, payment.Status });
});
app.Run();
public sealed record CreatePayment(string DebtorAccountId, string CreditorName, string CreditorIban,
decimal Amount, string Currency, string Scheme);5. Tests on SQLite (no AWS, no Docker)
using Harbourline.Payments.Domain;
using Harbourline.Payments.Outbox;
using Harbourline.Payments.Persistence;
using Microsoft.Data.Sqlite;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Time.Testing;
using Xunit;
public sealed class OutboxTests : IAsyncLifetime
{
private readonly SqliteConnection _conn = new("DataSource=:memory:");
private readonly FakeTimeProvider _clock = new(new DateTimeOffset(2026, 11, 18, 9, 0, 0, TimeSpan.Zero));
public async ValueTask InitializeAsync()
{
await _conn.OpenAsync();
await using var db = NewDb();
await db.Database.EnsureCreatedAsync();
}
public async ValueTask DisposeAsync() => await _conn.DisposeAsync();
private PaymentsDbContext NewDb() => new(new DbContextOptionsBuilder<PaymentsDbContext>()
.UseSqlite(_conn).AddInterceptors(new DomainEventsToOutboxInterceptor()).Options);
private async Task AddPaymentsAsync(int count)
{
await using var db = NewDb();
for (var i = 0; i < count; i++)
db.Payments.Add(Payment.Create("ACC-0000012345", "Example Supplies Ltd", "GB33BUKB20201555555555",
100.00m + i, "GBP", "FPS", _clock));
await db.SaveChangesAsync(TestContext.Current.CancellationToken);
}
private sealed class FakePublisher(Func<int, string?> resultFor) : IEventPublisher
{
public List<Guid> Published { get; } = [];
public Task<IReadOnlyList<string?>> PublishAsync(IReadOnlyList<OutboxMessage> batch, CancellationToken ct)
{
var results = batch.Select((m, i) => resultFor(i)).ToList();
Published.AddRange(batch.Where((m, i) => results[i] is null).Select(m => m.Id));
return Task.FromResult<IReadOnlyList<string?>>(results);
}
}
[Fact]
public async Task Saving_a_payment_writes_its_event_in_the_same_transaction()
{
await AddPaymentsAsync(1);
await using var db = NewDb();
Assert.Equal(1, await db.Payments.CountAsync(TestContext.Current.CancellationToken));
var msg = await db.Outbox.SingleAsync(TestContext.Current.CancellationToken);
Assert.Equal("PaymentReceived", msg.Type);
Assert.Null(msg.PublishedAtUtc);
}
[Fact]
public async Task Failed_save_writes_neither_payment_nor_event()
{
await using var db = NewDb();
db.Payments.Add(Payment.Create("ACC-0000012345", "A", "GB33BUKB20201555555555", 1m, "GBP", "FPS", _clock));
db.Payments.Add(Payment.Create("ACC-0000012345", "B", "GB33BUKB20201555555555", 2m, "GBP", "FPS", _clock));
// Force a failure inside the same SaveChanges: duplicate primary key on the outbox.
var dup = Guid.CreateVersion7();
db.Outbox.Add(new OutboxMessage { Id = dup, Type = "X", AggregateId = "x", Payload = "{}" });
db.Outbox.Add(new OutboxMessage { Id = dup, Type = "X", AggregateId = "x", Payload = "{}" });
await Assert.ThrowsAnyAsync<Exception>(() => db.SaveChangesAsync(TestContext.Current.CancellationToken));
await using var check = NewDb();
Assert.Equal(0, await check.Payments.CountAsync(TestContext.Current.CancellationToken));
Assert.Equal(0, await check.Outbox.CountAsync(TestContext.Current.CancellationToken));
}
[Fact]
public async Task Partial_PutEvents_failure_marks_only_successful_entries()
{
await AddPaymentsAsync(5);
var publisher = new FakePublisher(i => i == 2 ? "ThrottlingException: Rate exceeded" : null);
await using var db = NewDb();
var processor = new OutboxProcessor(db, publisher, _clock, new OutboxRelayOptions());
Assert.Equal(5, await processor.ProcessBatchAsync(TestContext.Current.CancellationToken));
await using var check = NewDb();
var rows = await check.Outbox.OrderBy(m => m.Id).ToListAsync(TestContext.Current.CancellationToken);
Assert.Equal(4, rows.Count(r => r.PublishedAtUtc is not null));
var failed = rows.Single(r => r.PublishedAtUtc is null);
Assert.Equal(1, failed.Attempts);
Assert.StartsWith("ThrottlingException", failed.LastError);
Assert.Null(failed.LeaseToken);
}
[Fact]
public async Task Two_relays_never_publish_the_same_row_in_one_round()
{
await AddPaymentsAsync(10);
var p1 = new FakePublisher(_ => null);
var p2 = new FakePublisher(_ => null);
await using var db1 = NewDb();
await using var db2 = NewDb();
var opts = new OutboxRelayOptions { BatchSize = 10 };
await new OutboxProcessor(db1, p1, _clock, opts).ProcessBatchAsync(TestContext.Current.CancellationToken);
await new OutboxProcessor(db2, p2, _clock, opts).ProcessBatchAsync(TestContext.Current.CancellationToken);
Assert.Empty(p1.Published.Intersect(p2.Published));
Assert.Equal(10, p1.Published.Count + p2.Published.Count);
}
}Test project packages: xunit.v3 4.0.3, Microsoft.EntityFrameworkCore.Sqlite 10.0.15 and Microsoft.Extensions.TimeProvider.Testing 10.11.0, with <OutputType>Exe</OutputType> and the Microsoft.Testing.Platform runner in global.json. The "two relays" test runs the processors one after the other on purpose. It proves the lease excludes rows another relay holds. True concurrency is better tested against PostgreSQL with Testcontainers.
Code explained line by line
AggregateRoot.Raise— business code records what happened (PaymentReceived) without any messaging dependency. The domain stays testable and transport-agnostic.e.Ignore(p => p.DomainEvents)— the event list is in-memory only. It's never mapped as a column.HasPrecision(18, 2)— money asdecimal(18,2), as in the course's domain model.DomainEventsToOutboxInterceptor.SavingChangesAsync— runs insideSaveChangesAsync, before EF Core opens its transaction and writes. Rows added here are part of the same unit of work, so payments and outbox rows commit or roll back together.Id = e.EventId— the outbox row's primary key is the event ID. The same value travels insidePayloadaseventId, and consumers deduplicate on it.aggregate.ClearDomainEvents()— prevents the same events being written again if the context is saved a second time.PutEventsRequestEntry.Time = m.OccurredAtUtc— consumers see when the payment happened, not when the relay got round to publishing it.response.Entriesin request order — EventBridge returns one result per entry, in order. An entry with anErrorCodefailed.FailedEntryCounttells you how many. Never assume "no exception" means "all published".- Candidate query
.OrderBy(m => m.Id)— UUID v7 IDs are time-ordered, so the relay publishes roughly in creation order. EventBridge Classic doesn't preserve order anyway; consumers that care use sequence numbers (see Delivery guarantees and ordering). - The lease
ExecuteUpdateAsync— a single SQLUPDATE ... WHERE id IN (...) AND (leased_until IS NULL OR leased_until < now). Databases apply eachUPDATEatomically, so two relays can't both lease the same row. This works on PostgreSQL, SQL Server and SQLite. On PostgreSQL you can insteadSELECT ... FOR UPDATE SKIP LOCKEDinside a transaction, which avoids the read-then-update round trip. Where(m => m.LeaseToken == token)— the relay reads back only rows carrying its random lease token, never a row another instance grabbed in between.- Lease expiry (
Lease = 30 s) — if the relay dies holding a lease, another instance takes the rows after 30 seconds. The lease must be longer than one publish call plus the marking step, or two relays will publish the same rows. That is harmless thanks to deduplication, but wasteful. - Catch-all around
PublishAsync— a thrown exception (network failure, or throttling after the SDK's own retries) is recorded against every row in the batch, with backoff. The loop never dies. - Mark with
m.LeaseToken == token— if our lease expired and another relay took over, we don't overwrite its bookkeeping. Attempts >= MaxAttempts→Parked = true— a row that keeps failing (for example, aDetailthat's not valid JSON) is parked, not retried forever. Alert on parked rows; a person fixes and un-parks them.Backoff— exponential with full jitter, capped at five minutes, as in Retry strategies.OutboxRelaycreates a scope per iteration —DbContextis scoped and not thread-safe. A fresh scope per batch keeps the change tracker small and avoids stale state.
What to measure
Publish three custom metrics from the relay: outbox lag (age of the oldest unpublished row), publish failures per minute, and parked rows. Lag is the one to alarm on: if it grows past, say, 60 seconds, downstream systems are working from stale information even though nothing is "down".
Security considerations
- The outbox holds personal data. Event payloads contain account IDs and amounts, sometimes names. Keep payloads minimal (IDs, amounts, statuses) and let consumers fetch details through APIs with their own authorisation. That is also the guidance for SNS now that SNS Message Data Protection is closed to new customers.
- Retention: delete published rows after a short period (for example 7 days) with a scheduled job. The outbox is a delivery mechanism, not an audit log. Use proper audit storage for audit.
- Least privilege: the service's task role needs
events:PutEventson one bus ARN. Restrict it further with theevents:sourcecondition key so the service can only publishharbourline.paymentsevents ⚠ VERIFY condition key support for your bus type. - Database credentials: use IAM database authentication or Secrets Manager with rotation for the Aurora connection string, never a password in
appsettings.json. - Integrity of event sources: downstream rules should match on
sourceand the bus. A bus resource policy should allowPutEventsonly from the accounts and roles you expect.
⚠ Learning shortcut — not for production
The relay polls every 500 ms when idle. That's fine for one service, but at scale each instance adds constant database queries. Production options: a longer idle delay with PostgreSQL LISTEN/NOTIFY to wake the relay on commit; or log-based CDC (Debezium on MSK Connect, or AWS DMS) reading the outbox table, which needs no polling. The example also has no cleanup job and no un-park tooling; both are mandatory in production. Finally, run the relay in the same service (and database) that owns the data. A shared, central relay reading many services' databases breaks service boundaries.
Banking use case
Harbourline adopted the outbox as the only sanctioned way for services with relational databases to publish domain events. The rollout produced these rules:
- One outbox per service database, in the service's own schema, owned by the service team.
- Event contracts are versioned (
PaymentReceived.v1) and published to the EventBridge schema registry. The outbox stores the exact JSON that will be sent, so the contract at commit time is the contract delivered. - Lag SLO: 99% of events published within 5 seconds of commit. The lag alarm pages the service team, not the platform team.
- Consumers deduplicate on
detail.eventId. The relay is at-least-once by design, and every consumer review checks for an idempotency key (see Idempotency in .NET). - Services on DynamoDB don't need a relay. The Accounts projection service writes items and uses DynamoDB Streams → EventBridge Pipes → EventBridge. When an intentional event must accompany an item change, it writes both in one
TransactWriteItems: the business item plus an "event" item that the pipe filters on. - Reconciliation: a nightly job compares the count of payments created per hour with the count of
PaymentReceivedevents received by the data platform, and raises a ticket on any difference.
Common mistakes
- Publishing inside the transaction ("open transaction, save, call EventBridge, commit") → the call can succeed and the commit fail → phantom events; plus the transaction is held open during a network call → only write the outbox row in the transaction.
- Using a separate
DbContextor connection for the outbox row → not the same transaction → same context, sameSaveChangesAsync. - Ignoring partial
PutEventsfailures → some events silently never published → check each entry'sErrorCodeandFailedEntryCount. - No lease, several relay instances → every instance publishes every row → atomic lease or
FOR UPDATE SKIP LOCKED. - Lease shorter than the publish time → overlapping relays and duplicates → size the lease well above the p99 of publish plus mark.
- Retrying a poison row forever → it blocks nothing but burns money and hides the problem → cap attempts, park and alert.
- Never deleting published rows → the table grows without limit and the relay query slows → scheduled cleanup and a good index.
- Expecting exactly-once delivery → consumers without idempotency double-process after a relay crash → consumers dedupe on
eventId. - Putting full customer records in events → personal data copied to every subscriber and archive → IDs plus minimal facts.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Events appear twice downstream | Relay crashed or lease expired after PutEvents but before marking | Expected occasionally; consumers dedupe on eventId; check that the lease is long enough |
| Outbox lag grows steadily | Relay not running, throttled, or each batch too slow | Check relay logs and the PutEvents throttle quota; run more instances (the lease makes it safe); increase batch size on buses that allow it |
ValidationException for an entry | Invalid Detail JSON, empty Source/DetailType, or entry too large | Fix the serialiser; reduce payload; the row will be parked after MaxAttempts |
Some entries fail with ThrottlingException | Above the account's PutEvents rate | Backoff handles it; request a quota increase if sustained |
AccessDeniedException on PutEvents | Task role lacks events:PutEvents on this bus ARN, or bus policy denies | Grant on the exact bus ARN; check bus resource policy |
| Nothing in the outbox after saving | Interceptor not registered on this DbContext, or events cleared before saving | Register with AddInterceptors; check the aggregate raises events |
| SQLite tests fail comparing dates | SQLite provider limitations with DateTimeOffset | Store UTC DateTime (as in this lesson), or test on PostgreSQL with Testcontainers |
Cost and cleanup
| Item | Cost at lab volume |
|---|---|
| EventBridge custom events | Billed per million events published ⚠ VERIFY current price; lab volume is negligible |
| CloudWatch Logs target | Ingestion and storage of a few KB; 7-day retention |
| Database | Local SQLite or PostgreSQL in Docker: free. Aurora is not needed for this lesson |
| ECS Fargate | Not needed; run the service locally |
The free alternative is the whole lesson minus the optional EventBridge step. The tests run on SQLite with a fake publisher.
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
aws events remove-targets --rule hl-m08-all-payments-to-logs --event-bus-name hl-m08-payments --ids logs
aws events delete-rule --name hl-m08-all-payments-to-logs --event-bus-name hl-m08-payments
aws events delete-event-bus --name hl-m08-payments
aws logs delete-log-group --log-group-name /aws/events/hl-m08-payments
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m08-outbox \
--query "ResourceTagMappingList[].ResourceARN"If you started PostgreSQL in Docker, stop and remove the container and its volume.
Exercises
- 🟢 Guided: Run the four tests. Then add a test that proves a row is parked after
MaxAttemptsfailures. AdvanceFakeTimeProviderpast each backoff between calls. - 🟡 Challenge: Add an inbox to a consumer. Write a
PaymentReceivedHandler(EF Core, its own database) that storeseventIdin aninbox_messagestable with a unique key, in the same transaction as its own business change. Show that a duplicate delivery makes no second change. - 🔴 Live problem: The Payments Service runs 6 ECS tasks. After a database failover, outbox lag jumped to 40 minutes and stayed there although the database is healthy again. Logs show the relays running and publishing about 10 events every 2 seconds each. There are 300,000 unpublished rows. Diagnose and fix.
Assignment
Package the outbox as a small library, Harbourline.Outbox, in your portfolio repository. Include:
OutboxMessage;- the interceptor;
OutboxProcessorandOutboxRelay;- the publisher abstraction with an EventBridge implementation;
- a cleanup
BackgroundService; - the tests on SQLite plus one Testcontainers PostgreSQL test (gated behind
RUN_DOCKER_TESTS=1).
Document the at-least-once contract in the README.
Solutions
[Fact]
public async Task Row_is_parked_after_max_attempts()
{
await AddPaymentsAsync(1);
var opts = new OutboxRelayOptions { MaxAttempts = 3 };
var publisher = new FakePublisher(_ => "ValidationException: Detail is malformed");
for (var i = 0; i < 3; i++)
{
await using var db = NewDb();
await new OutboxProcessor(db, publisher, _clock, opts).ProcessBatchAsync(TestContext.Current.CancellationToken);
_clock.Advance(TimeSpan.FromMinutes(6)); // past the maximum backoff
}
await using var check = NewDb();
var row = await check.Outbox.SingleAsync(TestContext.Current.CancellationToken);
Assert.True(row.Parked);
Assert.Equal(3, row.Attempts);
Assert.Null(row.PublishedAtUtc);
}Why: Parking stops a permanently failing row from consuming capacity and retry budget, and turns it into an alertable state.
public sealed class InboxMessage { public Guid EventId { get; set; } public DateTime ProcessedAtUtc { get; set; } }
// OnModelCreating: b.Entity<InboxMessage>().HasKey(m => m.EventId);
public async Task HandleAsync(PaymentReceived e, ScreeningDbContext db, CancellationToken ct)
{
db.Inbox.Add(new InboxMessage { EventId = e.EventId, ProcessedAtUtc = DateTime.UtcNow });
db.ScreeningCases.Add(ScreeningCase.Open(e.PaymentId, e.Amount, e.Currency));
try
{
await db.SaveChangesAsync(ct); // inbox row + business change, one transaction
}
catch (DbUpdateException ex) when (IsUniqueViolation(ex))
{
// Already processed this eventId: the duplicate is a no-op. Acknowledge the message.
}
}IsUniqueViolation checks the provider's error code: PostgreSQL 23505 through Npgsql.PostgresException.SqlState, or SQLite error 19 (constraint).
Why: The inbox is the outbox's mirror image. The primary key on EventId makes "have I seen this?" and "record that I've seen it" one atomic step, in the same transaction as the business change.
Diagnosis:
- 6 relays × 10 events every 2 seconds = 30 events/s, about 108,000/hour. A 300,000-row backlog takes nearly 3 hours to drain at that rate.
- Each batch takes 2 seconds, far longer than a
PutEventscall should. The likely culprit is the candidate query: with 300,000 rows, a missing or unused index makesWHERE published_at IS NULL AND NOT parked AND next_attempt_at <= now ORDER BY idscan the table on every iteration. - During the failover, every failing batch also pushed
next_attempt_atforward with backoff. Some rows are simply not due yet.
Fix:
- Check the plan (
EXPLAIN ANALYZE). Add a partial index on PostgreSQL:CREATE INDEX ... ON outbox_messages (id) WHERE published_at IS NULL AND NOT parked. Batch time should drop to tens of milliseconds. - Temporarily reset
next_attempt_at = now()for rows that failed only during the failover window, so they're due immediately. - Increase throughput safely: more relay iterations per second per task (no idle delay while there's backlog), and parallel
PutEventscalls per relay within the account's throttle limit. - Add the lag alarm. Forty minutes of silent lag means the alarm was missing. Add a cleanup job so the table doesn't grow to this size in normal operation.
- Tell downstream owners (notifications, fraud) to expect a burst, and check their concurrency limits.
Why: Outbox throughput is usually limited by the database query, not EventBridge. The backoff that protects a broken bus becomes a delay once it's fixed, so plan a recovery procedure.
Knowledge check
Answer: Writing to two systems (for example, a database and EventBridge) without a shared transaction. A crash between the writes either loses the event (save then publish) or publishes an event for a change that rolled back (publish then save).
Answer: The relay can publish an event and then fail before marking the row published; after restart or lease expiry it publishes again. Consumers must deduplicate, typically by the event ID carried in the payload.
Answer: No. PutEvents can partly fail. Check FailedEntryCount and each result entry's ErrorCode; mark only the successful entries and retry the others.
Answer: Each relay claims rows with one atomic UPDATE that only matches rows with no valid lease, setting its own random lease token. It then reads back only rows with its token. A second relay's UPDATE can't match rows already leased.
Answer: When the service's data is in DynamoDB. The stream records every item change exactly once, ordered per item, and Pipes delivers it to EventBridge without your own relay code. To publish an explicit domain event, write an event item in the same TransactWriteItems and filter on it in the pipe.
Interview questions
- "How do you reliably publish events from a service that uses a relational database?" — Transactional outbox: write the event row in the same transaction as the state change. A relay publishes with leases, batching, partial-failure handling, backoff and parking. Consumers are idempotent on event ID. Monitor lag.
- "Outbox vs CDC?" — An outbox publishes intentional domain events with stable contracts. Raw CDC publishes table changes, which couples consumers to your schema. A good hybrid is CDC that reads only the outbox table: no polling, and intentional events.
- "What's the inbox pattern?" — The consumer-side mirror: record each processed event ID in the consumer's database, in the same transaction as its own change, using a unique key to make duplicates no-ops.
- "How do you handle ordering with an outbox?" — Publish in creation order per aggregate where the transport allows (SQS FIFO with
MessageGroupId, or an EventBridge FIFO subscriber withEventGroupId). Always carry a per-aggregate sequence so consumers can reject stale events.
Certification practice
A developer's application calls PutEvents with 10 entries. The call returns successfully with FailedEntryCount equal to 2. What should the application do?
A. Nothing; the request succeeded.
B. Resend all 10 entries.
C. Identify the entries with an ErrorCode in the response and retry only those, with backoff.
D. Switch to PutRule for each entry.
Answer: C. PutEvents reports per-entry results in request order. Retry only failed entries with backoff. A loses 2 events. B duplicates 8. D is unrelated, because PutRule creates rules.
An order service stores orders in Amazon Aurora PostgreSQL and must publish an event to Amazon EventBridge for every committed order. Events must not be lost if the service crashes, and no event may be published for an order that was rolled back. What should a solutions architect recommend?
A. Publish the event first, then write the order.
B. Write the order, then publish the event in a finally block.
C. Write the order and an event row to an outbox table in the same database transaction; a separate process publishes outbox rows to EventBridge and marks them as sent.
D. Use an SQS FIFO queue with content-based deduplication between the service and EventBridge.
Answer: C. The single local transaction guarantees both or neither. The relay delivers at least once. A creates phantom events on rollback. B still loses events if the process dies before or during publishing. D doesn't make the database write and the send atomic.
After introducing an outbox relay, a consumer occasionally processes the same OrderPlaced event twice, minutes apart. The relay logs show a few restarts during deployments. What is the BEST fix?
A. Make the consumer idempotent using the event ID included in the event detail. B. Disable relay restarts during deployments. C. Increase the EventBridge rule's retry attempts. D. Use a larger PutEvents batch size.
Answer: A. The relay is at-least-once by design: a restart between publishing and marking causes a re-send. Consumers must deduplicate. B is impractical and doesn't cover crashes. C would add deliveries. D doesn't affect duplicates.
Completion checklist
- I can explain the dual-write problem with a banking example
- My EF Core interceptor writes outbox rows in the same
SaveChangesAsyncas the business change - My relay leases rows atomically, handles partial
PutEventsfailures, backs off and parks poison rows - All outbox tests pass on SQLite without AWS
- I can say when DynamoDB Streams + EventBridge Pipes replaces a polled outbox
- I deleted the bus, rule, log group, and any local database containers
Key takeaways
- Never dual-write. Write the event into an outbox table in the same transaction as the state change.
- The relay is at-least-once: lease rows atomically, publish in batches, check per-entry
PutEventsresults, back off, and park poison rows. - Consumers deduplicate on event ID (inbox pattern or idempotency store).
- Monitor outbox lag, clean up published rows, and keep payloads minimal, because the outbox holds personal data.
- On DynamoDB, use Streams + EventBridge Pipes (with transactional event items) instead of a polled outbox.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
AWS Step Functions: workflows, retries and callbacks
Orchestrate multi-step banking processes with AWS Step Functions: Standard versus Express workflows, the state types, JSONata, service integrations, Retry and Catch, human-approval callbacks with task tokens, Map and Distributed Map, redrive, testing with TestState, and a .NET 10 client, CDK and Terraform.
Kinesis Data Streams and Amazon MSK for event streaming
Use an ordered, replayable log instead of a queue: Kinesis Data Streams shards, partition keys, capacity modes, consumers and poison records, then Amazon MSK (Kafka) with IAM authentication and a .NET producer, and a clear rule for choosing between streams, queues and buses.