Skip to Content
Cloud Engineering Academy
8. Messaging, Events & Microservices

The outbox pattern with EF Core

Solve the dual-write problem: write business state and its events in one EF Core transaction, relay them to EventBridge with a leased, batch-aware BackgroundService, and handle relay crashes, partial PutEvents failures, poison rows and cleanup — with tests that run on SQLite.

advanced⏱ 2 h 30 min🟢 FreeDVA-C03SAA-C03

Assumes: .NET 10 LTS · EF Core 10.0.15 · Npgsql.EntityFrameworkCore.PostgreSQL 10.0.3 · AWS SDK for .NET v4 (AWSSDK.EventBridge 4.1.0.2) · xunit.v3 4.0.3 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain the dual-write problem and why "save, then publish" (or "publish, then save") loses or invents events.
  • Implement a transactional outbox in EF Core 10:
    • domain events collected by a SaveChangesInterceptor;
    • events stored in an OutboxMessages table in the same transaction as the business change.
  • Build an outbox relay as a .NET BackgroundService that:
    • leases rows safely across several instances;
    • publishes to Amazon EventBridge with PutEvents;
    • handles partial batch failures;
    • retries with backoff and parks poison rows.
  • Explain why the relay is at-least-once, and how consumers stay correct (event IDs and idempotency).
  • Compare the outbox with change data capture alternatives: DynamoDB Streams with EventBridge Pipes, and database log-based CDC.

Prerequisites

Why this matters in banking

Harbourline's Payments Service saves a new payment to Aurora PostgreSQL and then publishes PaymentReceived to EventBridge, so screening, fraud scoring and notifications can start. The code looked fine:

await db.SaveChangesAsync(ct);                  // 1. commit the payment
await eventBridge.PutEventsAsync(evt, ct);      // 2. publish the event

During a routine deployment, ECS stopped tasks while requests were in flight. Forty-one payments were committed in step 1 but never published in step 2. Nothing downstream knew they existed. They sat in received status until a customer complained two days later that their rent hadn't arrived.

The team tried swapping the order (publish first, then save). On the next database failover, events were published for payments that were then rolled back. Fraud scoring flagged transactions that never existed, and a customer received a "payment sent" SMS for a payment that wasn't.

This is the dual-write problem. You can't atomically write to two systems (a database and a message bus) without a shared transaction, and AWS gives you no transaction that spans Aurora and EventBridge. The transactional outbox turns two writes into one, then delivers the event reliably afterwards.

The idea in plain language

Instead of publishing the event directly, you write it into a table in the same database, in the same transaction as the business change. Either both the payment and its event are committed, or neither is.

A separate process, the relay, reads that table and publishes the events to EventBridge. When EventBridge confirms, the relay marks the row as published. If the relay crashes, the rows are still there, and it carries on after a restart.

Analogy: A bank branch doesn't phone head office every time it processes a cheque, because the call might fail halfway. Instead, the clerk writes each cheque into the day book at the same moment they stamp it. A courier collects the day book and delivers its entries to head office. If the courier's van breaks down, the entries are still in the book and go with the next courier. Head office might receive an entry twice if the courier isn't sure it was delivered, so head office checks the entry numbers. The day book is the outbox; the courier is the relay; the entry numbers are event IDs.

Three properties follow:

  1. No lost events. If the payment committed, its event is in the outbox.
  2. No phantom events. If the payment rolled back, so did its event.
  3. At-least-once publishing. The relay may publish an event, then crash before marking it published. After restart it publishes it again. Consumers must deduplicate by event ID.

What problem the AWS services solve (and don't)

PieceRoleWhat it doesn't do
Aurora PostgreSQL / RDS (or SQL Server on RDS)Holds business tables and the outbox table; one local ACID transaction covers bothPublish anything by itself
EF Core 10SaveChangesAsync wraps all tracked changes in one transaction by defaultKnow about your message bus
Amazon EventBridge PutEventsReceives events and routes them to rules/subscribers; returns a per-entry resultJoin your database transaction; deduplicate on a Classic bus
ECS Fargate service / Worker ServiceHosts the relay continuously (low latency)Guarantee a single instance; your lease logic does that
EventBridge Scheduler + LambdaAlternative relay host that runs every minuteSub-minute latency
DynamoDB Streams + EventBridge PipesBuilt-in CDC for DynamoDB: every item change flows to a target without your own relayHelp when your data is in a relational database

Limits to design around:

  • Classic bus: PutEvents accepts up to 10 entries per request ⚠ VERIFY current limit on your bus type. The newer EventBridge Custom Event Bus accepts up to 100.
  • Each entry has a size limit (check the current EventBridge quotas; keep events small and reference IDs).
  • PutEvents can partly succeed. The response's FailedEntryCount and per-entry ErrorCode tell you which entries failed. A request that "didn't throw" isn't a request where everything succeeded.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
NServiceBus / MassTransit transactional outboxYour own EF Core outbox + relay (this lesson)AWS.Messaging for .NET doesn't include an outbox at the time of writing ⚠ VERIFY; you own the table and relay
TransactionScope spanning SQL Server and MSMQ (MSDTC)Not available across Aurora and EventBridgeThe outbox replaces the distributed transaction
SQL Server Change Data Capture / Debezium → Event HubsAWS DMS CDC, Debezium on MSK Connect, DynamoDB Streams + EventBridge PipesCDC publishes table changes, not intentional domain events, unless you capture an outbox table
Cosmos DB change feed → Azure FunctionsDynamoDB Streams → Lambda / EventBridge PipesSame idea: the database's change log is the outbox
Hangfire / Quartz recurring job polling a tableBackgroundService on ECS, or EventBridge Scheduler → LambdaSame polling idea; use leases so instances don't collide

Architecture

Sequence, including the crash that causes a duplicate

Step-by-step implementation

Model domain events on the aggregate

Payment gets a private list of domain events. Business methods (Create, MarkScreened) add events. The aggregate knows nothing about EventBridge.

Add the outbox table

OutboxMessage has:

  • Id (UUID v7, time-ordered);
  • Type, Source, AggregateId, Payload (JSON);
  • OccurredAt;
  • delivery bookkeeping: PublishedAt, Attempts, NextAttemptAt, LeaseToken, LeasedUntil and LastError.

Add an index on (PublishedAt, NextAttemptAt).

Convert domain events to outbox rows inside SaveChanges

A SaveChangesInterceptor runs just before EF Core saves. It collects events from tracked aggregates and adds OutboxMessage entities to the same context, so they join the same transaction automatically.

Write the relay

A BackgroundService loop:

  1. Lease a batch with a single atomic UPDATE.
  2. Read the rows it leased.
  3. Publish them with PutEvents.
  4. Mark the successful entries published, and schedule the failed ones with backoff.
  5. Sleep briefly when idle.

Test without AWS

Run the tests in Complete code on SQLite in-memory with a fake publisher that fails selected entries. They prove atomicity, partial-failure handling and lease safety.

Optional: publish to a real bus

Create the hl-m08-payments bus and a rule that sends everything to a CloudWatch Logs group. Run the service locally with the learn-sandbox profile and watch events arrive.

AWS Console instructions

The database side is code. The AWS side is small:

  1. Console → Amazon EventBridge → Event buses → Create event bus. Name hl-m08-payments. Add the lab tags. (In the console this bus type is listed as Custom Event Bus - Classic if you choose the rules-and-targets model.)
  2. Rules → Create rule on that bus: name hl-m08-all-payments-to-logs, event pattern {"source":["harbourline.payments"]}. Target CloudWatch log group /aws/events/hl-m08-payments (create it, and set retention to 1 week).
  3. Monitoring: on the bus's Monitoring tab, watch PutEventsRequestSize and the rule's MatchedEvents. In CloudWatch, alarm on PutEventsFailedEntries if your Region publishes it ⚠ VERIFY metric name. Also publish your own relay metrics (see Code explained).
  4. IAM: the ECS task role needs only events:PutEvents on the bus ARN.

AWS CLI

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1

aws events create-event-bus --name hl-m08-payments \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m08-outbox Key=Environment,Value=learn Key=Owner,Value=alex

aws logs create-log-group --log-group-name /aws/events/hl-m08-payments
aws logs put-retention-policy --log-group-name /aws/events/hl-m08-payments --retention-in-days 7

aws events put-rule --name hl-m08-all-payments-to-logs --event-bus-name hl-m08-payments \
  --event-pattern '{"source":["harbourline.payments"]}'
aws events put-targets --rule hl-m08-all-payments-to-logs --event-bus-name hl-m08-payments \
  --targets "Id=logs,Arn=arn:aws:logs:us-east-1:$(aws sts get-caller-identity --query Account --output text):log-group:/aws/events/hl-m08-payments"
# Note: EventBridge needs a CloudWatch Logs resource policy to write to the group; the console adds it for you.

# Publish one event by hand to see the per-entry result shape the relay must handle.
aws events put-events --entries '[{
  "EventBusName":"hl-m08-payments","Source":"harbourline.payments","DetailType":"PaymentReceived",
  "Detail":"{\"eventId\":\"0192d1c2-0000-7000-8000-0000000000e1\",\"paymentId\":\"PAY-0192d1c2-0000-7000-8000-000000000001\",\"amount\":\"1250.00\",\"currency\":\"GBP\"}"
}]'
# → {"FailedEntryCount":0,"Entries":[{"EventId":"..."}]}

# Read what arrived
aws logs tail /aws/events/hl-m08-payments --since 10m

Infrastructure as Code

OutboxStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.Events.Targets;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.Logs;
using Constructs;

namespace Harbourline.Infra;

public sealed class OutboxStack : Stack
{
    public OutboxStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        var bus = new EventBus(this, "PaymentsBus", new EventBusProps { EventBusName = "hl-m08-payments" });

        // Debug sink: every payments event into a short-retention log group.
        var logGroup = new LogGroup(this, "PaymentsEvents", new LogGroupProps
        {
            LogGroupName = "/aws/events/hl-m08-payments",
            Retention = RetentionDays.ONE_WEEK,
            RemovalPolicy = RemovalPolicy.DESTROY
        });
        new Rule(this, "AllPaymentsToLogs", new RuleProps
        {
            RuleName = "hl-m08-all-payments-to-logs",
            EventBus = bus,
            EventPattern = new EventPattern { Source = ["harbourline.payments"] },
            Targets = [new CloudWatchLogGroup(logGroup)]
        });

        // Policy the Payments Service task role attaches: PutEvents on this bus only.
        new ManagedPolicy(this, "RelayPublish", new ManagedPolicyProps
        {
            ManagedPolicyName = "hl-m08-outbox-relay-publish",
            Statements =
            [
                new PolicyStatement(new PolicyStatementProps
                {
                    Actions = ["events:PutEvents"],
                    Resources = [bus.EventBusArn]
                })
            ]
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m08-outbox");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        Amazon.CDK.Tags.Of(this).Add("Owner", "alex");
    }
}

Complete code

1. Domain: aggregate with domain events

Domain/Payment.cs
namespace Harbourline.Payments.Domain;

public interface IDomainEvent
{
    Guid EventId { get; }
    DateTime OccurredAtUtc { get; }
}

public abstract class AggregateRoot
{
    private readonly List<IDomainEvent> _events = [];
    public IReadOnlyList<IDomainEvent> DomainEvents => _events;
    protected void Raise(IDomainEvent e) => _events.Add(e);
    public void ClearDomainEvents() => _events.Clear();
}

public sealed record PaymentReceived(
    Guid EventId, DateTime OccurredAtUtc, string PaymentId, string DebtorAccountId,
    decimal Amount, string Currency, string Scheme) : IDomainEvent;

public sealed class Payment : AggregateRoot
{
    private Payment() { } // EF Core

    public string PaymentId { get; private set; } = default!;
    public string DebtorAccountId { get; private set; } = default!;
    public string CreditorName { get; private set; } = default!;
    public string CreditorIban { get; private set; } = default!;
    public decimal Amount { get; private set; }
    public string Currency { get; private set; } = default!;
    public string Scheme { get; private set; } = default!;
    public string Status { get; private set; } = default!;

    public static Payment Create(string debtorAccountId, string creditorName, string creditorIban,
        decimal amount, string currency, string scheme, TimeProvider clock)
    {
        if (amount <= 0) throw new ArgumentOutOfRangeException(nameof(amount), "Amount must be positive");
        var p = new Payment
        {
            PaymentId = $"PAY-{Guid.CreateVersion7()}",
            DebtorAccountId = debtorAccountId,
            CreditorName = creditorName,
            CreditorIban = creditorIban,
            Amount = amount,
            Currency = currency,
            Scheme = scheme,
            Status = "received"
        };
        p.Raise(new PaymentReceived(Guid.CreateVersion7(), clock.GetUtcNow().UtcDateTime,
            p.PaymentId, debtorAccountId, amount, currency, scheme));
        return p;
    }
}

2. Persistence: outbox entity, DbContext and interceptor

Persistence/OutboxMessage.cs
namespace Harbourline.Payments.Persistence;

public sealed class OutboxMessage
{
    public Guid Id { get; set; }                 // = domain EventId (UUID v7, time-ordered)
    public string Type { get; set; } = default!; // EventBridge DetailType, e.g. "PaymentReceived"
    public string Source { get; set; } = "harbourline.payments";
    public string AggregateId { get; set; } = default!;
    public string Payload { get; set; } = default!;  // JSON detail
    public DateTime OccurredAtUtc { get; set; }

    // Delivery bookkeeping
    public DateTime? PublishedAtUtc { get; set; }
    public int Attempts { get; set; }
    public DateTime NextAttemptAtUtc { get; set; }
    public Guid? LeaseToken { get; set; }
    public DateTime? LeasedUntilUtc { get; set; }
    public string? LastError { get; set; }
    public bool Parked { get; set; }             // poison: stop retrying, alert a human
}
Persistence/PaymentsDbContext.cs
using Harbourline.Payments.Domain;
using Microsoft.EntityFrameworkCore;

namespace Harbourline.Payments.Persistence;

public sealed class PaymentsDbContext(DbContextOptions<PaymentsDbContext> options) : DbContext(options)
{
    public DbSet<Payment> Payments => Set<Payment>();
    public DbSet<OutboxMessage> Outbox => Set<OutboxMessage>();

    protected override void OnModelCreating(ModelBuilder b)
    {
        b.Entity<Payment>(e =>
        {
            e.ToTable("payments");
            e.HasKey(p => p.PaymentId);
            e.Property(p => p.Amount).HasPrecision(18, 2);   // money: decimal(18,2), never double
            e.Ignore(p => p.DomainEvents);
        });

        b.Entity<OutboxMessage>(e =>
        {
            e.ToTable("outbox_messages");
            e.HasKey(m => m.Id);
            e.Property(m => m.Type).HasMaxLength(128);
            e.Property(m => m.AggregateId).HasMaxLength(64);
            // The relay's query: unpublished, not parked, due now.
            e.HasIndex(m => new { m.PublishedAtUtc, m.Parked, m.NextAttemptAtUtc });
        });
    }
}
Persistence/DomainEventsToOutboxInterceptor.cs
using System.Text.Json;
using Harbourline.Payments.Domain;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Diagnostics;

namespace Harbourline.Payments.Persistence;

/// <summary>Turns domain events into outbox rows in the SAME SaveChanges (and so the same transaction).</summary>
public sealed class DomainEventsToOutboxInterceptor : SaveChangesInterceptor
{
    private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);

    public override ValueTask<InterceptionResult<int>> SavingChangesAsync(
        DbContextEventData eventData, InterceptionResult<int> result, CancellationToken ct = default)
    {
        if (eventData.Context is not DbContext db) return ValueTask.FromResult(result);

        var aggregates = db.ChangeTracker.Entries<AggregateRoot>()
            .Select(e => e.Entity)
            .Where(a => a.DomainEvents.Count > 0)
            .ToList();

        foreach (var aggregate in aggregates)
        {
            foreach (var e in aggregate.DomainEvents)
            {
                db.Set<OutboxMessage>().Add(new OutboxMessage
                {
                    Id = e.EventId,
                    Type = e.GetType().Name,
                    AggregateId = aggregate is Payment p ? p.PaymentId : "unknown",
                    Payload = JsonSerializer.Serialize(e, e.GetType(), Json),
                    OccurredAtUtc = e.OccurredAtUtc,
                    NextAttemptAtUtc = e.OccurredAtUtc
                });
            }
            aggregate.ClearDomainEvents();
        }
        return ValueTask.FromResult(result);
    }
}

3. The relay

Outbox/IEventPublisher.cs
using Amazon.EventBridge;
using Amazon.EventBridge.Model;
using Harbourline.Payments.Persistence;

namespace Harbourline.Payments.Outbox;

/// <summary>Result per input message, in the same order. Null error = published.</summary>
public interface IEventPublisher
{
    Task<IReadOnlyList<string?>> PublishAsync(IReadOnlyList<OutboxMessage> batch, CancellationToken ct);
}

public sealed class EventBridgePublisher(IAmazonEventBridge eventBridge, string busName) : IEventPublisher
{
    public async Task<IReadOnlyList<string?>> PublishAsync(IReadOnlyList<OutboxMessage> batch, CancellationToken ct)
    {
        var response = await eventBridge.PutEventsAsync(new PutEventsRequest
        {
            Entries = batch.Select(m => new PutEventsRequestEntry
            {
                EventBusName = busName,
                Source = m.Source,
                DetailType = m.Type,
                Detail = m.Payload,          // includes eventId for consumer deduplication
                Time = m.OccurredAtUtc,      // event time, not publish time
                Resources = [m.AggregateId]
            }).ToList()
        }, ct);

        // Entries come back in request order. An entry with an ErrorCode failed; others succeeded.
        var entries = response.Entries ?? [];
        return batch.Select((_, i) =>
                i < entries.Count && string.IsNullOrEmpty(entries[i].ErrorCode)
                    ? null
                    : $"{(i < entries.Count ? entries[i].ErrorCode : "MissingResult")}: {(i < entries.Count ? entries[i].ErrorMessage : "")}")
            .ToList();
    }
}
Outbox/OutboxRelay.cs
using Harbourline.Payments.Persistence;
using Microsoft.EntityFrameworkCore;

namespace Harbourline.Payments.Outbox;

public sealed class OutboxRelayOptions
{
    public int BatchSize { get; set; } = 10;                     // Classic bus PutEvents limit
    public TimeSpan Lease { get; set; } = TimeSpan.FromSeconds(30);
    public TimeSpan IdleDelay { get; set; } = TimeSpan.FromMilliseconds(500);
    public int MaxAttempts { get; set; } = 12;
}

public sealed class OutboxProcessor(PaymentsDbContext db, IEventPublisher publisher, TimeProvider clock,
    OutboxRelayOptions options)
{
    /// <summary>Processes one batch. Returns how many rows it handled (0 = nothing due).</summary>
    public async Task<int> ProcessBatchAsync(CancellationToken ct)
    {
        var now = clock.GetUtcNow().UtcDateTime;
        var token = Guid.NewGuid();
        var leasedUntil = now + options.Lease;

        // 1. Pick candidate IDs (oldest first).
        var candidates = await db.Outbox
            .Where(m => m.PublishedAtUtc == null && !m.Parked && m.NextAttemptAtUtc <= now
                        && (m.LeasedUntilUtc == null || m.LeasedUntilUtc < now))
            .OrderBy(m => m.Id)
            .Select(m => m.Id)
            .Take(options.BatchSize)
            .ToListAsync(ct);
        if (candidates.Count == 0) return 0;

        // 2. Lease them in ONE atomic UPDATE. A competing relay's UPDATE can only win rows we didn't.
        await db.Outbox
            .Where(m => candidates.Contains(m.Id) && (m.LeasedUntilUtc == null || m.LeasedUntilUtc < now))
            .ExecuteUpdateAsync(s => s
                .SetProperty(m => m.LeaseToken, token)
                .SetProperty(m => m.LeasedUntilUtc, leasedUntil), ct);

        // 3. Read back exactly the rows WE leased.
        var batch = await db.Outbox.AsNoTracking()
            .Where(m => m.LeaseToken == token)
            .OrderBy(m => m.Id)
            .ToListAsync(ct);
        if (batch.Count == 0) return 0;

        // 4. Publish. A thrown exception (network, throttling after SDK retries) fails the whole batch.
        IReadOnlyList<string?> results;
        try
        {
            results = await publisher.PublishAsync(batch, ct);
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            results = batch.Select(_ => (string?)$"{ex.GetType().Name}: {ex.Message}").ToList();
        }

        // 5. Record outcomes. Only rows still under OUR lease are touched.
        for (var i = 0; i < batch.Count; i++)
        {
            var id = batch[i].Id;
            if (results[i] is null)
            {
                await db.Outbox.Where(m => m.Id == id && m.LeaseToken == token)
                    .ExecuteUpdateAsync(s => s
                        .SetProperty(m => m.PublishedAtUtc, now)
                        .SetProperty(m => m.LeaseToken, (Guid?)null)
                        .SetProperty(m => m.LeasedUntilUtc, (DateTime?)null), ct);
            }
            else
            {
                var attempts = batch[i].Attempts + 1;
                var park = attempts >= options.MaxAttempts;
                var delay = Backoff(attempts);
                var error = results[i]!.Length > 1000 ? results[i]![..1000] : results[i]!;
                await db.Outbox.Where(m => m.Id == id && m.LeaseToken == token)
                    .ExecuteUpdateAsync(s => s
                        .SetProperty(m => m.Attempts, attempts)
                        .SetProperty(m => m.NextAttemptAtUtc, now + delay)
                        .SetProperty(m => m.LastError, error)
                        .SetProperty(m => m.Parked, park)
                        .SetProperty(m => m.LeaseToken, (Guid?)null)
                        .SetProperty(m => m.LeasedUntilUtc, (DateTime?)null), ct);
            }
        }
        return batch.Count;
    }

    /// <summary>Exponential backoff with full jitter: up to 1 s, 2 s, 4 s … capped at 5 minutes.</summary>
    private static TimeSpan Backoff(int attempts) =>
        TimeSpan.FromMilliseconds(Random.Shared.NextDouble() * Math.Min(300_000, 1000 * Math.Pow(2, attempts - 1)));
}

public sealed class OutboxRelay(IServiceScopeFactory scopes, OutboxRelayOptions options, ILogger<OutboxRelay> log)
    : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await using var scope = scopes.CreateAsyncScope();
                var processor = scope.ServiceProvider.GetRequiredService<OutboxProcessor>();
                var handled = await processor.ProcessBatchAsync(stoppingToken);
                if (handled == 0) await Task.Delay(options.IdleDelay, stoppingToken);
            }
            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
            {
                break;
            }
            catch (Exception ex)
            {
                // Database unavailable etc. Never let the loop die.
                log.LogError(ex, "Outbox relay iteration failed");
                await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
            }
        }
    }
}

4. Wiring it up

Program.cs (Payments Service, excerpt)
using Amazon.EventBridge;
using Harbourline.Payments.Domain;
using Harbourline.Payments.Outbox;
using Harbourline.Payments.Persistence;
using Microsoft.EntityFrameworkCore;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddSingleton(TimeProvider.System);
builder.Services.AddSingleton<DomainEventsToOutboxInterceptor>();
builder.Services.AddDbContext<PaymentsDbContext>((sp, o) => o
    .UseNpgsql(builder.Configuration.GetConnectionString("Payments")) // from Secrets Manager / IAM auth in prod
    .AddInterceptors(sp.GetRequiredService<DomainEventsToOutboxInterceptor>()));

builder.Services.AddSingleton<IAmazonEventBridge>(_ => new AmazonEventBridgeClient(new AmazonEventBridgeConfig
{
    RegionEndpoint = Amazon.RegionEndpoint.USEast1,
    RetryMode = Amazon.Runtime.RequestRetryMode.Standard
}));
builder.Services.AddSingleton<IEventPublisher>(sp =>
    new EventBridgePublisher(sp.GetRequiredService<IAmazonEventBridge>(), "hl-m08-payments"));
builder.Services.AddSingleton(new OutboxRelayOptions());
builder.Services.AddScoped<OutboxProcessor>();
builder.Services.AddHostedService<OutboxRelay>();

var app = builder.Build();

app.MapPost("/payments", async (CreatePayment req, PaymentsDbContext db, TimeProvider clock, CancellationToken ct) =>
{
    var payment = Payment.Create(req.DebtorAccountId, req.CreditorName, req.CreditorIban,
        req.Amount, req.Currency, req.Scheme, clock);
    db.Payments.Add(payment);
    await db.SaveChangesAsync(ct);   // payment row + outbox row, one transaction
    return Results.Accepted($"/payments/{payment.PaymentId}", new { payment.PaymentId, payment.Status });
});

app.Run();

public sealed record CreatePayment(string DebtorAccountId, string CreditorName, string CreditorIban,
    decimal Amount, string Currency, string Scheme);

5. Tests on SQLite (no AWS, no Docker)

tests/OutboxTests.cs
using Harbourline.Payments.Domain;
using Harbourline.Payments.Outbox;
using Harbourline.Payments.Persistence;
using Microsoft.Data.Sqlite;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Time.Testing;
using Xunit;

public sealed class OutboxTests : IAsyncLifetime
{
    private readonly SqliteConnection _conn = new("DataSource=:memory:");
    private readonly FakeTimeProvider _clock = new(new DateTimeOffset(2026, 11, 18, 9, 0, 0, TimeSpan.Zero));

    public async ValueTask InitializeAsync()
    {
        await _conn.OpenAsync();
        await using var db = NewDb();
        await db.Database.EnsureCreatedAsync();
    }

    public async ValueTask DisposeAsync() => await _conn.DisposeAsync();

    private PaymentsDbContext NewDb() => new(new DbContextOptionsBuilder<PaymentsDbContext>()
        .UseSqlite(_conn).AddInterceptors(new DomainEventsToOutboxInterceptor()).Options);

    private async Task AddPaymentsAsync(int count)
    {
        await using var db = NewDb();
        for (var i = 0; i < count; i++)
            db.Payments.Add(Payment.Create("ACC-0000012345", "Example Supplies Ltd", "GB33BUKB20201555555555",
                100.00m + i, "GBP", "FPS", _clock));
        await db.SaveChangesAsync(TestContext.Current.CancellationToken);
    }

    private sealed class FakePublisher(Func<int, string?> resultFor) : IEventPublisher
    {
        public List<Guid> Published { get; } = [];
        public Task<IReadOnlyList<string?>> PublishAsync(IReadOnlyList<OutboxMessage> batch, CancellationToken ct)
        {
            var results = batch.Select((m, i) => resultFor(i)).ToList();
            Published.AddRange(batch.Where((m, i) => results[i] is null).Select(m => m.Id));
            return Task.FromResult<IReadOnlyList<string?>>(results);
        }
    }

    [Fact]
    public async Task Saving_a_payment_writes_its_event_in_the_same_transaction()
    {
        await AddPaymentsAsync(1);
        await using var db = NewDb();
        Assert.Equal(1, await db.Payments.CountAsync(TestContext.Current.CancellationToken));
        var msg = await db.Outbox.SingleAsync(TestContext.Current.CancellationToken);
        Assert.Equal("PaymentReceived", msg.Type);
        Assert.Null(msg.PublishedAtUtc);
    }

    [Fact]
    public async Task Failed_save_writes_neither_payment_nor_event()
    {
        await using var db = NewDb();
        db.Payments.Add(Payment.Create("ACC-0000012345", "A", "GB33BUKB20201555555555", 1m, "GBP", "FPS", _clock));
        db.Payments.Add(Payment.Create("ACC-0000012345", "B", "GB33BUKB20201555555555", 2m, "GBP", "FPS", _clock));
        // Force a failure inside the same SaveChanges: duplicate primary key on the outbox.
        var dup = Guid.CreateVersion7();
        db.Outbox.Add(new OutboxMessage { Id = dup, Type = "X", AggregateId = "x", Payload = "{}" });
        db.Outbox.Add(new OutboxMessage { Id = dup, Type = "X", AggregateId = "x", Payload = "{}" });
        await Assert.ThrowsAnyAsync<Exception>(() => db.SaveChangesAsync(TestContext.Current.CancellationToken));

        await using var check = NewDb();
        Assert.Equal(0, await check.Payments.CountAsync(TestContext.Current.CancellationToken));
        Assert.Equal(0, await check.Outbox.CountAsync(TestContext.Current.CancellationToken));
    }

    [Fact]
    public async Task Partial_PutEvents_failure_marks_only_successful_entries()
    {
        await AddPaymentsAsync(5);
        var publisher = new FakePublisher(i => i == 2 ? "ThrottlingException: Rate exceeded" : null);
        await using var db = NewDb();
        var processor = new OutboxProcessor(db, publisher, _clock, new OutboxRelayOptions());

        Assert.Equal(5, await processor.ProcessBatchAsync(TestContext.Current.CancellationToken));

        await using var check = NewDb();
        var rows = await check.Outbox.OrderBy(m => m.Id).ToListAsync(TestContext.Current.CancellationToken);
        Assert.Equal(4, rows.Count(r => r.PublishedAtUtc is not null));
        var failed = rows.Single(r => r.PublishedAtUtc is null);
        Assert.Equal(1, failed.Attempts);
        Assert.StartsWith("ThrottlingException", failed.LastError);
        Assert.Null(failed.LeaseToken);
    }

    [Fact]
    public async Task Two_relays_never_publish_the_same_row_in_one_round()
    {
        await AddPaymentsAsync(10);
        var p1 = new FakePublisher(_ => null);
        var p2 = new FakePublisher(_ => null);
        await using var db1 = NewDb();
        await using var db2 = NewDb();
        var opts = new OutboxRelayOptions { BatchSize = 10 };

        await new OutboxProcessor(db1, p1, _clock, opts).ProcessBatchAsync(TestContext.Current.CancellationToken);
        await new OutboxProcessor(db2, p2, _clock, opts).ProcessBatchAsync(TestContext.Current.CancellationToken);

        Assert.Empty(p1.Published.Intersect(p2.Published));
        Assert.Equal(10, p1.Published.Count + p2.Published.Count);
    }
}

Test project packages: xunit.v3 4.0.3, Microsoft.EntityFrameworkCore.Sqlite 10.0.15 and Microsoft.Extensions.TimeProvider.Testing 10.11.0, with <OutputType>Exe</OutputType> and the Microsoft.Testing.Platform runner in global.json. The "two relays" test runs the processors one after the other on purpose. It proves the lease excludes rows another relay holds. True concurrency is better tested against PostgreSQL with Testcontainers.

Code explained line by line

  1. AggregateRoot.Raise — business code records what happened (PaymentReceived) without any messaging dependency. The domain stays testable and transport-agnostic.
  2. e.Ignore(p => p.DomainEvents) — the event list is in-memory only. It's never mapped as a column.
  3. HasPrecision(18, 2) — money as decimal(18,2), as in the course's domain model.
  4. DomainEventsToOutboxInterceptor.SavingChangesAsync — runs inside SaveChangesAsync, before EF Core opens its transaction and writes. Rows added here are part of the same unit of work, so payments and outbox rows commit or roll back together.
  5. Id = e.EventId — the outbox row's primary key is the event ID. The same value travels inside Payload as eventId, and consumers deduplicate on it.
  6. aggregate.ClearDomainEvents() — prevents the same events being written again if the context is saved a second time.
  7. PutEventsRequestEntry.Time = m.OccurredAtUtc — consumers see when the payment happened, not when the relay got round to publishing it.
  8. response.Entries in request order — EventBridge returns one result per entry, in order. An entry with an ErrorCode failed. FailedEntryCount tells you how many. Never assume "no exception" means "all published".
  9. Candidate query .OrderBy(m => m.Id) — UUID v7 IDs are time-ordered, so the relay publishes roughly in creation order. EventBridge Classic doesn't preserve order anyway; consumers that care use sequence numbers (see Delivery guarantees and ordering).
  10. The lease ExecuteUpdateAsync — a single SQL UPDATE ... WHERE id IN (...) AND (leased_until IS NULL OR leased_until < now). Databases apply each UPDATE atomically, so two relays can't both lease the same row. This works on PostgreSQL, SQL Server and SQLite. On PostgreSQL you can instead SELECT ... FOR UPDATE SKIP LOCKED inside a transaction, which avoids the read-then-update round trip.
  11. Where(m => m.LeaseToken == token) — the relay reads back only rows carrying its random lease token, never a row another instance grabbed in between.
  12. Lease expiry (Lease = 30 s) — if the relay dies holding a lease, another instance takes the rows after 30 seconds. The lease must be longer than one publish call plus the marking step, or two relays will publish the same rows. That is harmless thanks to deduplication, but wasteful.
  13. Catch-all around PublishAsync — a thrown exception (network failure, or throttling after the SDK's own retries) is recorded against every row in the batch, with backoff. The loop never dies.
  14. Mark with m.LeaseToken == token — if our lease expired and another relay took over, we don't overwrite its bookkeeping.
  15. Attempts >= MaxAttempts → Parked = true — a row that keeps failing (for example, a Detail that's not valid JSON) is parked, not retried forever. Alert on parked rows; a person fixes and un-parks them.
  16. Backoff — exponential with full jitter, capped at five minutes, as in Retry strategies.
  17. OutboxRelay creates a scope per iteration — DbContext is scoped and not thread-safe. A fresh scope per batch keeps the change tracker small and avoids stale state.

What to measure

Publish three custom metrics from the relay: outbox lag (age of the oldest unpublished row), publish failures per minute, and parked rows. Lag is the one to alarm on: if it grows past, say, 60 seconds, downstream systems are working from stale information even though nothing is "down".

Security considerations

  • The outbox holds personal data. Event payloads contain account IDs and amounts, sometimes names. Keep payloads minimal (IDs, amounts, statuses) and let consumers fetch details through APIs with their own authorisation. That is also the guidance for SNS now that SNS Message Data Protection is closed to new customers.
  • Retention: delete published rows after a short period (for example 7 days) with a scheduled job. The outbox is a delivery mechanism, not an audit log. Use proper audit storage for audit.
  • Least privilege: the service's task role needs events:PutEvents on one bus ARN. Restrict it further with the events:source condition key so the service can only publish harbourline.payments events ⚠ VERIFY condition key support for your bus type.
  • Database credentials: use IAM database authentication or Secrets Manager with rotation for the Aurora connection string, never a password in appsettings.json.
  • Integrity of event sources: downstream rules should match on source and the bus. A bus resource policy should allow PutEvents only from the accounts and roles you expect.

⚠ Learning shortcut — not for production

The relay polls every 500 ms when idle. That's fine for one service, but at scale each instance adds constant database queries. Production options: a longer idle delay with PostgreSQL LISTEN/NOTIFY to wake the relay on commit; or log-based CDC (Debezium on MSK Connect, or AWS DMS) reading the outbox table, which needs no polling. The example also has no cleanup job and no un-park tooling; both are mandatory in production. Finally, run the relay in the same service (and database) that owns the data. A shared, central relay reading many services' databases breaks service boundaries.

Banking use case

Harbourline adopted the outbox as the only sanctioned way for services with relational databases to publish domain events. The rollout produced these rules:

  1. One outbox per service database, in the service's own schema, owned by the service team.
  2. Event contracts are versioned (PaymentReceived.v1) and published to the EventBridge schema registry. The outbox stores the exact JSON that will be sent, so the contract at commit time is the contract delivered.
  3. Lag SLO: 99% of events published within 5 seconds of commit. The lag alarm pages the service team, not the platform team.
  4. Consumers deduplicate on detail.eventId. The relay is at-least-once by design, and every consumer review checks for an idempotency key (see Idempotency in .NET).
  5. Services on DynamoDB don't need a relay. The Accounts projection service writes items and uses DynamoDB Streams → EventBridge Pipes → EventBridge. When an intentional event must accompany an item change, it writes both in one TransactWriteItems: the business item plus an "event" item that the pipe filters on.
  6. Reconciliation: a nightly job compares the count of payments created per hour with the count of PaymentReceived events received by the data platform, and raises a ticket on any difference.

Common mistakes

  • Publishing inside the transaction ("open transaction, save, call EventBridge, commit") → the call can succeed and the commit fail → phantom events; plus the transaction is held open during a network call → only write the outbox row in the transaction.
  • Using a separate DbContext or connection for the outbox row → not the same transaction → same context, same SaveChangesAsync.
  • Ignoring partial PutEvents failures → some events silently never published → check each entry's ErrorCode and FailedEntryCount.
  • No lease, several relay instances → every instance publishes every row → atomic lease or FOR UPDATE SKIP LOCKED.
  • Lease shorter than the publish time → overlapping relays and duplicates → size the lease well above the p99 of publish plus mark.
  • Retrying a poison row forever → it blocks nothing but burns money and hides the problem → cap attempts, park and alert.
  • Never deleting published rows → the table grows without limit and the relay query slows → scheduled cleanup and a good index.
  • Expecting exactly-once delivery → consumers without idempotency double-process after a relay crash → consumers dedupe on eventId.
  • Putting full customer records in events → personal data copied to every subscriber and archive → IDs plus minimal facts.

Troubleshooting

SymptomLikely causeFix
Events appear twice downstreamRelay crashed or lease expired after PutEvents but before markingExpected occasionally; consumers dedupe on eventId; check that the lease is long enough
Outbox lag grows steadilyRelay not running, throttled, or each batch too slowCheck relay logs and the PutEvents throttle quota; run more instances (the lease makes it safe); increase batch size on buses that allow it
ValidationException for an entryInvalid Detail JSON, empty Source/DetailType, or entry too largeFix the serialiser; reduce payload; the row will be parked after MaxAttempts
Some entries fail with ThrottlingExceptionAbove the account's PutEvents rateBackoff handles it; request a quota increase if sustained
AccessDeniedException on PutEventsTask role lacks events:PutEvents on this bus ARN, or bus policy deniesGrant on the exact bus ARN; check bus resource policy
Nothing in the outbox after savingInterceptor not registered on this DbContext, or events cleared before savingRegister with AddInterceptors; check the aggregate raises events
SQLite tests fail comparing datesSQLite provider limitations with DateTimeOffsetStore UTC DateTime (as in this lesson), or test on PostgreSQL with Testcontainers

Cost and cleanup

ItemCost at lab volume
EventBridge custom eventsBilled per million events published ⚠ VERIFY current price; lab volume is negligible
CloudWatch Logs targetIngestion and storage of a few KB; 7-day retention
DatabaseLocal SQLite or PostgreSQL in Docker: free. Aurora is not needed for this lesson
ECS FargateNot needed; run the service locally

The free alternative is the whole lesson minus the optional EventBridge step. The tests run on SQLite with a fake publisher.

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
aws events remove-targets --rule hl-m08-all-payments-to-logs --event-bus-name hl-m08-payments --ids logs
aws events delete-rule --name hl-m08-all-payments-to-logs --event-bus-name hl-m08-payments
aws events delete-event-bus --name hl-m08-payments
aws logs delete-log-group --log-group-name /aws/events/hl-m08-payments

aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m08-outbox \
  --query "ResourceTagMappingList[].ResourceARN"

If you started PostgreSQL in Docker, stop and remove the container and its volume.

Exercises

  • 🟢 Guided: Run the four tests. Then add a test that proves a row is parked after MaxAttempts failures. Advance FakeTimeProvider past each backoff between calls.
  • 🟡 Challenge: Add an inbox to a consumer. Write a PaymentReceivedHandler (EF Core, its own database) that stores eventId in an inbox_messages table with a unique key, in the same transaction as its own business change. Show that a duplicate delivery makes no second change.
  • 🔴 Live problem: The Payments Service runs 6 ECS tasks. After a database failover, outbox lag jumped to 40 minutes and stayed there although the database is healthy again. Logs show the relays running and publishing about 10 events every 2 seconds each. There are 300,000 unpublished rows. Diagnose and fix.

Assignment

Package the outbox as a small library, Harbourline.Outbox, in your portfolio repository. Include:

  • OutboxMessage;
  • the interceptor;
  • OutboxProcessor and OutboxRelay;
  • the publisher abstraction with an EventBridge implementation;
  • a cleanup BackgroundService;
  • the tests on SQLite plus one Testcontainers PostgreSQL test (gated behind RUN_DOCKER_TESTS=1).

Document the at-least-once contract in the README.

Solutions

Knowledge check

Interview questions

  • "How do you reliably publish events from a service that uses a relational database?" — Transactional outbox: write the event row in the same transaction as the state change. A relay publishes with leases, batching, partial-failure handling, backoff and parking. Consumers are idempotent on event ID. Monitor lag.
  • "Outbox vs CDC?" — An outbox publishes intentional domain events with stable contracts. Raw CDC publishes table changes, which couples consumers to your schema. A good hybrid is CDC that reads only the outbox table: no polling, and intentional events.
  • "What's the inbox pattern?" — The consumer-side mirror: record each processed event ID in the consumer's database, in the same transaction as its own change, using a unique key to make duplicates no-ops.
  • "How do you handle ordering with an outbox?" — Publish in creation order per aggregate where the transport allows (SQS FIFO with MessageGroupId, or an EventBridge FIFO subscriber with EventGroupId). Always carry a per-aggregate sequence so consumers can reject stale events.

Certification practice

Completion checklist

  • I can explain the dual-write problem with a banking example
  • My EF Core interceptor writes outbox rows in the same SaveChangesAsync as the business change
  • My relay leases rows atomically, handles partial PutEvents failures, backs off and parks poison rows
  • All outbox tests pass on SQLite without AWS
  • I can say when DynamoDB Streams + EventBridge Pipes replaces a polled outbox
  • I deleted the bus, rule, log group, and any local database containers

Key takeaways

  • Never dual-write. Write the event into an outbox table in the same transaction as the state change.
  • The relay is at-least-once: lease rows atomically, publish in batches, check per-entry PutEvents results, back off, and park poison rows.
  • Consumers deduplicate on event ID (inbox pattern or idempotency store).
  • Monitor outbox lag, clean up published rows, and keep payloads minimal, because the outbox holds personal data.
  • On DynamoDB, use Streams + EventBridge Pipes (with transactional event items) instead of a polled outbox.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page