Skip to Content
Cloud Engineering Academy
8. Messaging, Events & Microservices

Amazon SNS: fan-out and message filtering

Publish one message to an Amazon SNS topic and deliver it to many consumers: SNS to SQS fan-out, subscription filter policies on attributes and on the message body, delivery retries, subscription dead-letter queues and FIFO topics, with .NET 10 publishers, CDK and Terraform.

intermediate⏱ 2 h 30 min🟢 FreeDVA-C03SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SimpleNotificationService 4.1.0.2) · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain what an SNS topic, subscription and publish are, and why the standard banking pattern is SNS to SQS fan-out rather than SNS straight to a service.
  • Write subscription filter policies on message attributes and on the message body, predict which subscriptions receive a message, and respect the documented limits (five keys, 150 combinations).
  • Describe SNS delivery retries per endpoint type, and attach a dead-letter queue to a subscription for the failures SNS cannot fix itself.
  • Choose between standard and FIFO topics, and between raw and enveloped delivery.
  • Publish single and batched messages from .NET 10 with the AWS SDK v4, and check a batch for partial failure.
  • Deploy a filtered fan-out with CDK (C#) or Terraform, and tear it down.

Prerequisites

Why this matters in banking

A posted transaction is interesting to many parts of Harbourline: Notifications (text and e-mail), Fraud (only large or foreign payments), Rewards (only card spend), the Data Platform (everything). The first design had the core banking adapter call each of them in turn. Two things went wrong in the first month:

  1. The Rewards service was down for forty minutes. Because the adapter called it synchronously, postings backed up behind it and the adapter's own queue overflowed.
  2. A team added a new consumer, "Collections", and had to ask the adapter team to add one more call, test it and deploy. The change took three weeks for what should have been a configuration change.

A topic fixes both. The adapter publishes once and does not know who listens. Each consumer subscribes on its own, with its own queue and its own filter. If Rewards is down, its queue grows and nothing else notices. A new team subscribes without a ticket to the adapter team.

The sharp edges are the ones this lesson spends time on: a filter that silently matches nothing, a message published without the attribute the filter needs, a subscription whose queue policy was forgotten, and personal data in a message that now fans out to twelve places.

The idea in plain language

Amazon Simple Notification Service (SNS) is a managed publish/subscribe service. You create a topic. Producers publish messages to the topic. Anything subscribed to the topic receives its own copy. SNS pushes: it calls the subscriber, instead of the subscriber asking for work.

A subscriber, called an endpoint, can be an SQS queue, a Lambda function, an HTTPS URL, an e-mail address, an SMS number, a mobile push endpoint or a Firehose delivery stream.

A filter policy on a subscription says "only send me messages that look like this". Without one, a subscription gets everything.

Analogy: A bank's internal circular. The compliance team writes one circular and puts it in the post room's "all staff" distribution list (the topic). The post room copies it into each subscribed department's tray (the SQS queues). A department can say "only send me circulars about card fraud" (a filter policy). The author never learns who is subscribed, and a department that is on holiday just finds a taller pile when it returns.

The key property is the one in the analogy: SNS does not store messages for you. It delivers to the subscribers that exist at the moment of publishing, retries for a while, and then drops the message unless you attached a dead-letter queue. The durable part of the design is the SQS queue behind each subscription.

What problem Amazon SNS solves

SNS removes the producer's knowledge of its consumers. It gives you:

  • Fan-out: one publish, many deliveries, in parallel.
  • Protocol bridging: the same message can go to a queue, a function and a webhook.
  • Filtering at the subscription: consumers choose; producers do not change.
  • Very high publish throughput with no capacity planning.

Limits and defaults (checked 2026-11-18)

SettingValue
Maximum message size256 KiB by default; raise the topic's MaximumMessageSize attribute up to 1 MiB. Larger payloads: store in S3 and publish a pointer (the SNS Extended Client Libraries do this up to 2 GB)
Topics per account100,000 standard; 1,000 FIFO
Subscriptions per topic12,500,000 standard; 100 FIFO
Filter policies200 per topic; 10,000 per account
PublishBatchUp to 10 entries per request
Publish rate, us-east-130,000 messages per second per account, Publish and PublishBatch combined (soft quota)
FIFO throughputUp to 3,000 messages per second per topic (or 20 MB per second) in the default Topic scope; 300 messages per second per message group
Message attributes to an SQS subscription with raw deliveryMaximum 10
Filter policy changesCan take up to 15 minutes to take full effect
BillingEach 64 KB chunk of a publish is one request. SNS does not charge a per-message delivery fee for SQS or Lambda endpoints

Honest limitations

  • No storage, no replay on standard topics. A message published when a subscriber is down is retried, then dropped. Put a queue behind every subscriber that matters. (FIFO topics can archive and replay; see below.)
  • No ordering on standard topics. Messages may arrive out of order and more than once.
  • Filtering is not routing logic. You cannot transform a message, join two fields or do arithmetic. If you need that, use EventBridge.
  • A filtered-out message is not an error. It simply never arrives. Debugging a wrong filter means reading metrics, not exceptions.
  • Personal data multiplies. One publish puts a copy of the message in every subscriber's queue, each with its own retention.

Service availability — verified 2026-11-18

Amazon SNS Message Data Protection (the feature that inspects and masks sensitive data in topics) is in maintenance and closed to new customers. Do not design around it. Keep personal data out of messages: publish identifiers and facts, and let authorised consumers fetch the rest. SNS itself, SQS, Lambda and the rest of the features in this lesson are fully available.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Service Bus topic with subscriptionsSNS topic with subscriptionsA Service Bus subscription is itself a durable queue. An SNS subscription is only a delivery rule: the durable queue is a separate SQS queue you attach
Service Bus subscription rules (SQL filter, correlation filter)SNS filter policyFilter policies are JSON, not SQL. They match on equality, prefix, suffix, numeric ranges and existence, not on expressions
Service Bus ApplicationPropertiesSNS message attributesAttributes are the default filter scope. In Service Bus the broker filters on properties by default too
Event Grid topic and subscriptionSNS topic, or an EventBridge busEvent Grid is closer to EventBridge. SNS is closer to Service Bus topics, without the storage
Service Bus sessions and duplicate detectionSNS FIFO topic with MessageGroupId and deduplicationFIFO topics only deliver to FIFO-compatible subscribers, and have a 100-subscription limit
Service Bus dead-letter sub-queueSNS subscription redrive policy to an SQS queueSNS only dead-letters delivery failures. A message the consumer cannot process is the consumer queue's DLQ problem
Azure.Messaging.ServiceBus ServiceBusSenderAmazonSimpleNotificationServiceClient.PublishAsyncNo connection to manage: each call is an HTTPS request signed with SigV4

Delivery semantics and retries

SNS defines a delivery policy per protocol. The numbers below are from the AWS documentation (checked 2026-11-18).

Endpoint typeRetry behaviour on server-side errorsCan you change it?
SQS, Lambda (AWS managed)3 immediate retries, 2 at one second, 10 with exponential backoff from 1 to 20 seconds, then up to 100,000 retries 20 seconds apart: 100,015 attempts over about 23 daysNo
SMTP, SMS, mobile push2 retries 10 seconds apart, 10 with backoff from 10 seconds to 10 minutes, 38 at 10 minutes: 50 attempts over about 6 hoursNo
HTTP/SA four-phase policy (immediate, pre-backoff, backoff, post-backoff) that you can customise. The documented example makes 50 attempts. The total retry time cannot exceed 3,600 secondsYes, at topic or subscription level

Two rules decide what happens next:

  • Client-side errors are not retried. If the endpoint no longer exists, or a policy now denies SNS, the delivery fails immediately. Examples: a deleted queue, a missing queue policy, KMS denying SNS access to an encrypted queue.
  • Server-side errors are retried with jitter. For SQS and Lambda endpoints that means "for weeks", which is long enough that retries are rarely your problem. A wrong IAM policy is.

When the policy is exhausted, or on a client-side error, SNS discards the message unless the subscription has a dead-letter queue. The DLQ is an ordinary SQS queue in the same account and Region, and its type must match the topic: a standard topic uses a standard queue, a FIFO topic a FIFO queue.

Two different dead-letter queues

The subscription DLQ catches messages SNS could not put into the consumer's queue. The consumer queue's DLQ (the SQS maxReceiveCount one from the previous lesson) catches messages the consumer received and could not process. You usually want both, and an alarm on each.

Message filtering

Two scopes

Set the subscription attribute FilterPolicyScope:

ScopeThe policy is matched againstWho can publish for it
MessageAttributes (the default)The message attributes you pass with PublishOnly publishers that set attributes: your own code. EventBridge and S3 do not set your attributes
MessageBodyThe JSON message body. The body must be well-formed JSONAnyone, including EventBridge rules that target the topic

The lab project of this module uses MessageBody because EventBridge is the publisher. When you publish from .NET yourself, attributes are cheaper to reason about and keep routing metadata out of the payload.

The rules

  • Keys are ANDed. Every key in the policy must match.
  • Values in a key's array are ORed. "currency": ["GBP", "EUR"] means GBP or EUR.
  • Properties not named in the policy are ignored.
  • An array in the message matches if any element matches. A message attribute of type String.Array with ["sms","email"] matches the policy {"channel":["email"]}.
  • A key that is missing from the message does not match, unless the policy uses {"exists": false} for it.
  • String matching is case-sensitive.

Operators

OperatorExampleMatches
Exact string"channel": ["sms"]the string sms
Exact number"amount": [100]the number 100
prefix"type": [{"prefix": "CARD_"}]CARD_PURCHASE, CARD_REFUND
suffix"file": [{"suffix": ".pdf"}]strings ending .pdf
equals-ignore-case"ccy": [{"equals-ignore-case": "gbp"}]GBP, Gbp
anything-but"channel": [{"anything-but": ["push"]}]everything except push (also works with prefix inside)
numeric"amount": [{"numeric": [">=", 1000, "<", 50000]}]1000 up to but not including 50000
exists"promo": [{"exists": false}]messages without a promo key
cidr"ip": [{"cidr": "10.0.0.0/24"}]IP addresses in the range
wildcard"name": [{"wildcard": "inv-*.pdf"}]glob-style match (complexity capped at 100 points)

Numbers must be between -1 billion and 1 billion with up to five digits after the decimal point.

Limits you will actually hit

  • At most five keys per policy. In a nested (body) policy only the leaf keys count.
  • At most 150 combinations: multiply the number of values in each key. Three values times one times two is 6. A key with 12 values and another with 13 is 156 and is rejected.
  • A policy can be up to 256 KB, but you will never get close to the other limits first.
  • Changes take up to 15 minutes to take full effect. Do not conclude "my filter is wrong" thirty seconds after saving it.

The FilterPolicyComplexity helper below checks the key and combination limits in your CI pipeline, so the mistake does not first appear at deploy time.

Seeing filtering in CloudWatch

Filtered messages are counted, not logged. The SNS topic metrics include NumberOfMessagesPublished, NumberOfNotificationsDelivered, NumberOfNotificationsFailed and NumberOfNotificationsFilteredOut (with variants that say why: no message attributes, invalid attributes, or message body ⚠ VERIFY the exact suffixes in the current metrics page). If Published rises, Delivered stays flat and FilteredOut rises, your filter excludes everything, which is almost always a scope mismatch.

Raw delivery

By default SNS wraps each message for SQS and HTTP endpoints in a JSON envelope with Type, MessageId, TopicArn, Message (the body, as a string), Timestamp, a signature and an unsubscribe URL. Your consumer then has to unwrap twice: once for the envelope, once for your own JSON.

With raw message delivery on a subscription, SNS strips its own metadata and delivers the body as it was published. For SQS subscriptions, up to 10 message attributes are also passed on. Beyond 10 the message is discarded as a client error, so keep attributes few.

Use raw delivery for your own services: the consumer code is the same whether the message came from SNS, EventBridge or a direct SendMessage. Keep the envelope when you need MessageId and Timestamp from SNS, or when an HTTPS endpoint must verify the SNS signature.

FIFO topics

A FIFO topic keeps strict order per MessageGroupId and removes duplicate publishes (same MessageDeduplicationId within five minutes, or content-based deduplication). Its name ends in .fifo.

Standard topicFIFO topic
OrderNonePer message group
DuplicatesPossibleRemoved at publish within the 5-minute window
SubscribersAll protocolsSQS FIFO queues for sure; ⚠ VERIFY the current list of other supported protocols (standard SQS, Lambda, HTTP/S, Firehose) before you rely on one
Subscriptions per topic12.5 million100
Throughput30,000 per second (us-east-1 account quota)3,000 per second per topic, 300 per group
Archive and replayNot availableYes: an archive policy keeps messages, a replay policy redelivers them

Use FIFO when order matters downstream (a ledger projection that applies postings per account) and the subscriber count is small. Use standard and an idempotent consumer when it does not. FIFO's order is per group, so choose the group key as you would for an SQS FIFO queue: account ID, not a constant.

Architecture

The module project (the Harbourline notification system) puts SNS between EventBridge and the per-channel queues. The pattern generalises to any fan-out.

Security-boundary view

Three authorisation checks are in play, and each one can break a fan-out: the publisher's IAM policy on the topic, the queue policy that lets sns.amazonaws.com send to the queue, and (for encrypted queues) the KMS key policy that lets SNS use the key.

Step-by-step implementation

Create the topic, the queues and their DLQs

Create a standard topic hl-m08gap-transaction-notices, a queue per consumer (hl-m08gap-sms, hl-m08gap-highvalue), a DLQ for each queue with maxReceiveCount 5, and one shared subscription DLQ. Enable server-side encryption on the queues and enforce TLS. The Infrastructure as Code section does all of this in one deploy.

Allow SNS to send to each queue

An SQS queue accepts messages from SNS only if its queue policy allows sqs:SendMessage to the principal sns.amazonaws.com with a condition ArnEquals on aws:SourceArn set to your topic. Without the condition, any SNS topic in any account that knows the queue ARN could write to it. CDK and the Terraform in this lesson add the condition for you; the CLI route below shows it explicitly.

Subscribe each queue with a filter and raw delivery

Create the subscription with the attributes RawMessageDelivery=true, a FilterPolicy and (when matching on the body) FilterPolicyScope=MessageBody. Add a RedrivePolicy pointing at the subscription DLQ.

Publish with attributes

Use the .NET publisher below. It sets channel as String.Array, currency as String and amount as Number, so the three policies in this lesson work.

Prove the routing

Publish three notices: an SMS for £25, an e-mail for £5,000 and an SMS for €1,200. Receive from each queue and confirm the matrix in the next section. Then look at the topic's NumberOfNotificationsFilteredOut metric.

Expected routing for the three test messages with the two filters from the CDK stack:

Messagenotify-sms (channel is sms)high-value (GBP or EUR, amount at least 1000)
SMS, GBP 25DeliveredFiltered out (25 is under 1000)
E-mail, GBP 5,000Filtered out (channel is not sms)Delivered
SMS, EUR 1,200DeliveredDelivered (one publish, two copies)

AWS Console instructions

Console → SNS → Topics → Create topic. Choose Standard, name it, leave encryption on if offered. Then Subscriptions → Create subscription: pick the topic, protocol Amazon SQS, endpoint your queue ARN. Open Subscription filter policy, switch on the policy, choose the scope (Message attributes or Message body) and paste the JSON. Tick Enable raw message delivery. Under Redrive policy (dead-letter queue) pick the subscription DLQ.

Add the queue policy on the SQS side: Console → SQS → your queue → Access policy → Edit, and paste the statement from the CLI section. The SQS console can also create the subscription and update the policy for you (SNS subscriptions → Subscribe to Amazon SNS topic) ⚠ VERIFY the current menu name. Whichever way you do it, check that the resulting policy contains the aws:SourceArn condition.

To see filtering in action, Topic → Monitoring shows the delivered and filtered-out graphs.

AWS CLI

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1

# 1. Topic and queues.
TOPIC_ARN=$(aws sns create-topic --name hl-m08gap-transaction-notices \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m08-messaging-services \
  --query TopicArn --output text)
QUEUE_URL=$(aws sqs create-queue --queue-name hl-m08gap-sms \
  --attributes SqsManagedSseEnabled=true --query QueueUrl --output text)
QUEUE_ARN=$(aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)

# 2. Queue policy: only THIS topic may send. The SourceArn condition is the important line.
cat > queue-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "sns.amazonaws.com" },
    "Action": "sqs:SendMessage",
    "Resource": "$QUEUE_ARN",
    "Condition": { "ArnEquals": { "aws:SourceArn": "$TOPIC_ARN" } }
  }]
}
EOF
aws sqs set-queue-attributes --queue-url "$QUEUE_URL" \
  --attributes "{\"Policy\": $(jq -Rs . < queue-policy.json)}"

# 3. Subscribe with raw delivery and an attribute-scoped filter policy.
SUB_ARN=$(aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol sqs \
  --notification-endpoint "$QUEUE_ARN" --return-subscription-arn \
  --attributes '{"RawMessageDelivery":"true","FilterPolicyScope":"MessageAttributes","FilterPolicy":"{\"channel\":[\"sms\"]}"}' \
  --query SubscriptionArn --output text)

# 4. Publish with message attributes. String.Array lets one attribute hold several values.
aws sns publish --topic-arn "$TOPIC_ARN" \
  --message '{"transactionId":"T-0001","customerId":"CUST-1","amount":25.00,"currency":"GBP","channels":["sms"]}' \
  --message-attributes '{"channel":{"DataType":"String.Array","StringValue":"[\"sms\"]"},"currency":{"DataType":"String","StringValue":"GBP"},"amount":{"DataType":"Number","StringValue":"25.00"}}'

# 5. Receive (long poll), then look at what filtering did.
aws sqs receive-message --queue-url "$QUEUE_URL" --wait-time-seconds 10 --max-number-of-messages 10

# Inspect and change a filter later (takes up to 15 minutes to be fully effective).
aws sns get-subscription-attributes --subscription-arn "$SUB_ARN" \
  --query 'Attributes.{scope:FilterPolicyScope,policy:FilterPolicy,raw:RawMessageDelivery}'
aws sns set-subscription-attributes --subscription-arn "$SUB_ARN" \
  --attribute-name FilterPolicy --attribute-value '{"channel":["sms","email"]}'

Infrastructure as Code

The stack creates the topic with TLS enforced, a consumer queue and DLQ per subscriber, a shared subscription DLQ, two filtered subscriptions with raw delivery, and a FIFO topic with a FIFO queue. The SqsSubscription construct writes the queue policy with the aws:SourceArn condition for you.

FanOutStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.SNS;
using Amazon.CDK.AWS.SNS.Subscriptions;
using Amazon.CDK.AWS.SQS;
using Constructs;

namespace Harbourline.Messaging.Infra;

/// <summary>
/// One SNS topic fanning out to a queue per consumer. Every queue has its own DLQ, and every subscription has a
/// redrive DLQ for the case where SNS itself cannot deliver (a deleted queue, a broken policy, KMS denied).
/// Filters live on the subscription, so a consumer changes what it receives without touching the publisher.
/// </summary>
public sealed class FanOutStack : Stack
{
    public Topic Topic { get; }

    public FanOutStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        Topic = new Topic(this, "NoticeTopic", new TopicProps
        {
            TopicName = "hl-m08gap-transaction-notices",
            EnforceSSL = true, // topic policy that denies non-TLS publishes
        });

        // Subscription DLQ: SNS-level failures (not consumer failures) land here. Standard topic -> standard queue.
        var subscriptionDlq = new Queue(this, "SubscriptionDlq", new QueueProps
        {
            QueueName = "hl-m08gap-subscription-dlq",
            RetentionPeriod = Duration.Days(14),
            Encryption = QueueEncryption.SQS_MANAGED,
            EnforceSSL = true,
        });

        AddConsumer("Sms", subscriptionDlq, new Dictionary<string, SubscriptionFilter>
        {
            // Attribute scope (the default): the publisher sets these as message attributes.
            ["channel"] = SubscriptionFilter.StringFilter(new StringConditions { Allowlist = ["sms"] }),
        });

        AddConsumer("HighValue", subscriptionDlq, new Dictionary<string, SubscriptionFilter>
        {
            // AND across keys, OR inside a key: GBP or EUR, AND amount of 1000 or more.
            ["currency"] = SubscriptionFilter.StringFilter(new StringConditions { Allowlist = ["GBP", "EUR"] }),
            ["amount"] = SubscriptionFilter.NumericFilter(new NumericConditions { GreaterThanOrEqualTo = 1000 }),
        });

        // A FIFO topic keeps order per MessageGroupId and drops duplicate publishes (5-minute window).
        // Its subscribers must be FIFO queues, so the ledger projection sees one account's events in order.
        var ledgerTopic = new Topic(this, "LedgerTopic", new TopicProps
        {
            TopicName = "hl-m08gap-ledger.fifo",
            Fifo = true,
            ContentBasedDeduplication = true,
            EnforceSSL = true,
        });
        var ledgerQueue = new Queue(this, "LedgerQueue", new QueueProps
        {
            QueueName = "hl-m08gap-ledger.fifo",
            Fifo = true,
            Encryption = QueueEncryption.SQS_MANAGED,
            EnforceSSL = true,
        });
        ledgerTopic.AddSubscription(new SqsSubscription(ledgerQueue, new SqsSubscriptionProps { RawMessageDelivery = true }));
    }

    private void AddConsumer(string name, Queue subscriptionDlq, IDictionary<string, SubscriptionFilter> filter)
    {
        var dlq = new Queue(this, $"{name}Dlq", new QueueProps
        {
            QueueName = $"hl-m08gap-{name.ToLowerInvariant()}-dlq",
            RetentionPeriod = Duration.Days(14),
            Encryption = QueueEncryption.SQS_MANAGED,
            EnforceSSL = true,
        });
        var queue = new Queue(this, $"{name}Queue", new QueueProps
        {
            QueueName = $"hl-m08gap-{name.ToLowerInvariant()}",
            VisibilityTimeout = Duration.Seconds(180),
            Encryption = QueueEncryption.SQS_MANAGED,
            EnforceSSL = true,
            DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 },
        });

        Topic.AddSubscription(new SqsSubscription(queue, new SqsSubscriptionProps
        {
            RawMessageDelivery = true,
            FilterPolicy = filter,
            DeadLetterQueue = subscriptionDlq,
        }));
    }
}

Both examples use SQS_MANAGED (SSE-SQS) encryption, which needs no key policy. If you switch the queues to a customer managed KMS key, the key policy must allow sns.amazonaws.com to use kms:GenerateDataKey and kms:Decrypt, or every delivery will fail with a client-side error.

Complete code

Two small classes carry the lesson. They were compiled and unit-tested with xUnit v3 against AWS SDK for .NET v4 and live in labs/m08-messaging-services/. To use them in your own project:

dotnet new classlib -n Harbourline.Messaging.Samples
cd Harbourline.Messaging.Samples
dotnet add package AWSSDK.SimpleNotificationService --version 4.1.0.2
TransactionTopicPublisher.cs
using System.Globalization;
using System.Text.Json;
using Amazon.SimpleNotificationService;
using Amazon.SimpleNotificationService.Model;

namespace Harbourline.Messaging.Samples.Sns;

/// <summary>The minimum a notification needs: identifiers and facts, never names, full IBANs or balances.</summary>
public sealed record TransactionNotice(
    string TransactionId,
    string CustomerId,
    decimal Amount,
    string Currency,
    IReadOnlyList<string> Channels);

/// <summary>
/// Publishes notices to an SNS topic with message attributes, so subscribers can use the default
/// attribute-scoped filter policies. This only works when YOU publish: EventBridge and S3 do not set your attributes.
/// </summary>
public sealed class TransactionTopicPublisher(IAmazonSimpleNotificationService sns, string topicArn)
{
    // PublishBatch accepts at most 10 entries per call.
    public const int MaxBatchSize = 10;

    public async Task<string> PublishAsync(TransactionNotice notice, CancellationToken ct = default)
    {
        var response = await sns.PublishAsync(ToRequest(notice), ct);
        return response.MessageId;
    }

    /// <summary>Publishes in batches of 10 and returns the transaction ids SNS rejected, so the caller retries only those.</summary>
    public async Task<IReadOnlyList<string>> PublishManyAsync(
        IReadOnlyList<TransactionNotice> notices, CancellationToken ct = default)
    {
        var failed = new List<string>();
        foreach (var chunk in notices.Chunk(MaxBatchSize))
        {
            var request = new PublishBatchRequest
            {
                TopicArn = topicArn,
                PublishBatchRequestEntries = chunk.Select(n =>
                {
                    var single = ToRequest(n);
                    return new PublishBatchRequestEntry
                    {
                        Id = n.TransactionId, // unique inside the batch; echoed back in Failed[]
                        Message = single.Message,
                        MessageAttributes = single.MessageAttributes,
                    };
                }).ToList(),
            };

            var response = await sns.PublishBatchAsync(request, ct);

            // A batch call can succeed (HTTP 200) while individual entries fail. Always read Failed.
            failed.AddRange(response.Failed.Select(f => f.Id));
        }
        return failed;
    }

    private PublishRequest ToRequest(TransactionNotice notice) => new()
    {
        TopicArn = topicArn,
        Message = JsonSerializer.Serialize(notice, JsonSerializerOptions.Web),
        MessageAttributes = new Dictionary<string, MessageAttributeValue>
        {
            // "String.Array" lets one attribute hold several channels; the policy {"channel":["sms"]} matches if ANY element matches.
            ["channel"] = new() { DataType = "String.Array", StringValue = JsonSerializer.Serialize(notice.Channels) },
            ["currency"] = new() { DataType = "String", StringValue = notice.Currency },
            // DataType "Number" lets a policy use {"numeric": [">=", 1000]}.
            ["amount"] = new() { DataType = "Number", StringValue = notice.Amount.ToString(CultureInfo.InvariantCulture) },
        },
    };
}
FilterPolicyComplexity.cs
using System.Text.Json.Nodes;

namespace Harbourline.Messaging.Samples.Sns;

/// <summary>
/// Pre-flight check of an SNS subscription filter policy against the documented limits: at most 5 keys and at most
/// 150 combinations. Combinations = the product of the match values in each key; in a nested (MessageBody scope)
/// policy only LEAF keys count, and each leaf's value count is multiplied by its nesting depth.
/// SNS stays the source of truth: this catches the mistake in CI instead of at deployment.
/// </summary>
public static class FilterPolicyComplexity
{
    public const int MaxKeys = 5;
    public const int MaxCombinations = 150;

    public sealed record Result(int LeafKeys, int Combinations, bool WithinLimits);

    public static Result Measure(string policyJson)
    {
        var root = JsonNode.Parse(policyJson) as JsonObject
            ?? throw new ArgumentException("A filter policy must be a JSON object.", nameof(policyJson));

        var leaves = new List<(int Values, int Depth)>();
        Walk(root, depth: 1, leaves);

        // long, not int: a careless policy can overflow int before we get the chance to report it.
        long combinations = 1;
        foreach (var (values, depth) in leaves)
        {
            combinations *= (long)values * depth;
            if (combinations > int.MaxValue) { combinations = int.MaxValue; break; }
        }

        var total = (int)combinations;
        return new Result(leaves.Count, total, leaves.Count <= MaxKeys && total <= MaxCombinations);
    }

    private static void Walk(JsonObject node, int depth, List<(int Values, int Depth)> leaves)
    {
        foreach (var (_, value) in node)
        {
            switch (value)
            {
                case JsonObject child:
                    Walk(child, depth + 1, leaves);
                    break;
                case JsonArray values:
                    leaves.Add((Math.Max(values.Count, 1), depth));
                    break;
                default:
                    throw new ArgumentException("Policy values must be objects (nesting) or arrays (match values).");
            }
        }
    }
}

The tests (excerpt) pin the documented examples: the flat policy is 6 combinations, the nested policy is 72, and a policy with 156 combinations is rejected.

SnsAndEventBridgeTests.cs (excerpt)
[Fact]
public void Nested_policy_counts_leaf_keys_and_multiplies_by_depth()
{
    // (4 x 3) x (3 x 2) = 72, two leaf keys.
    var result = FilterPolicyComplexity.Measure("""
        {
          "key_a": { "key_b": { "key_c": ["1","2","3","4"] } },
          "key_d": { "key_e": ["1","2","3"] }
        }
        """);

    Assert.Equal(2, result.LeafKeys);
    Assert.Equal(72, result.Combinations);
    Assert.True(result.WithinLimits);
}

The module project also carries an offline teaching matcher for filter policies, FilterPolicy in labs/m08-notification-system/src/Harbourline.Notifications.Core/Routing/. It matches a message body against a policy so you can unit-test routing without AWS. Treat it as a learning aid: SNS is the source of truth.

Code explained line by line

  1. TransactionNotice carries identifiers and facts only: no name, no full IBAN, no balance. Every subscriber's queue gets a copy, so the minimum is the safest content.
  2. MaxBatchSize = 10 mirrors the PublishBatch limit. PublishManyAsync slices the input with Chunk(10).
  3. Each batch entry has an Id (the transaction ID). SNS echoes it in the Failed list, which is how the caller learns exactly which notices to retry.
  4. response.Failed is read after every batch call. A PublishBatch request can return HTTP 200 with some entries failed; ignoring Failed silently drops those notifications.
  5. ToRequest builds the attributes. channel uses DataType = "String.Array" with a JSON array as the value; that is how one attribute carries several channels. amount uses DataType = "Number" so a numeric filter works. CultureInfo.InvariantCulture keeps the decimal separator a dot on every machine.
  6. JsonSerializerOptions.Web produces camelCase JSON, matching the event contract in the rest of the module.
  7. In FilterPolicyComplexity.Measure, Walk collects one (values, depth) pair per leaf array. The product of values × depth follows the documented formula. It uses long so that a careless policy overflows nothing before it is reported.
  8. WithinLimits is the CI gate: at most five leaf keys and at most 150 combinations.

Security considerations

⚠ Learning shortcut — not for production

The lab uses SSE-SQS on the queues and no encryption on the topic, because the data is fictional and identifiers only. For anything linked to a customer, turn on SNS server-side encryption with a customer managed KMS key, and give the key policy statements for the publishers (kms:GenerateDataKey, kms:Decrypt) and, when EventBridge or CloudWatch alarms publish to the topic, for events.amazonaws.com or cloudwatch.amazonaws.com. Budget about USD 1 per key per month and check key-use logging in CloudTrail.

⚠ Learning shortcut — not for production

You publish with your SSO identity. In production the publisher's role gets only sns:Publish on the one topic ARN, each consumer role gets only sqs:ReceiveMessage, sqs:DeleteMessage and sqs:GetQueueAttributes on its own queue, and nobody except the pipeline can call sns:Subscribe or sns:SetSubscriptionAttributes, because a subscription is a data-exfiltration path.

  • Topic policy. Deny aws:SecureTransport = false. For cross-account publishers or subscribers, allow named principals only, and use aws:SourceAccount or aws:SourceArn conditions for service principals.
  • Queue policy. Always include the aws:SourceArn condition for the topic. A queue that allows sns.amazonaws.com without it can be written to by any topic.
  • Subscriptions are access. Anyone allowed to subscribe an endpoint can read every unfiltered message. Control sns:Subscribe and alarm on new subscriptions with a CloudTrail rule.
  • HTTPS endpoints. Verify the SNS message signature and confirm the subscription yourself. Never treat an inbound POST to /sns as trusted because of its URL.
  • Personal data. Fan-out copies a message to every subscriber's queue, each with its own retention. Data minimisation (identifiers only) is the strongest control. Agree retention for topic-adjacent queues and DLQs with your data-protection team. This lesson does not make any system compliant with GDPR or PCI DSS: legal and security review decide that.
  • Filters are not authorisation. A filter keeps messages out of a queue for efficiency. It does not stop a principal with sqs:ReceiveMessage on another queue from reading it. Authorisation lives in IAM.

Banking use case

Harbourline's TransactionPosted fan-out, designed with this lesson:

SubscriberFilterWhy
Notifications SMSchannel contains smsCustomer opted in
Notifications e-mailchannel contains emailCustomer opted in
Fraud scoringcurrency not GBP, or amount at least 5,000Only the risky tail
Rewardstype has prefix CARD_Only card spend
Data PlatformnoneEverything; it is the system of record for analytics

Each subscriber has its own queue with its own DLQ and its own alarm. The adapter publishes once. When Collections later wants all overdraft-fee events, it subscribes with {"type":["OVERDRAFT_FEE"]} and nobody else deploys anything.

Common mistakes

  • Filtering on attributes when EventBridge publishes. EventBridge does not set your message attributes, so every message fails an attribute filter. → Use FilterPolicyScope=MessageBody.
  • Forgetting the queue policy. The subscription is confirmed but nothing arrives, and the failure is a client-side error that is not retried. → Add the sqs:SendMessage statement with the aws:SourceArn condition and attach a subscription DLQ so you can see failures.
  • Publishing a number as a string. "amount": "1200" as a String attribute fails a numeric filter. → Use DataType = "Number".
  • Expecting standard topics to keep messages. A subscriber added later does not receive history. → Use FIFO archive and replay, or an EventBridge archive, or give the consumer its own queue from day one.
  • Trusting HTTP 200 from PublishBatch. Individual entries can fail. → Read Failed.
  • Wrapped payloads in consumers. Parsing Message out of the SNS envelope in some consumers and not others. → Turn on raw delivery everywhere you own the consumer.
  • Over-complex policies. More than five keys, or more than 150 combinations. → Split into two subscriptions, or move the logic to an EventBridge rule or the consumer.
  • A constant MessageGroupId on a FIFO topic. All messages in one group serialise at 300 per second. → Group by the entity whose order matters.

Troubleshooting

SymptomLikely causeFix
Published, but nothing in the queueMissing queue policy, or the filter excludes the messageCheck the subscription DLQ and NumberOfNotificationsFailed; compare NumberOfNotificationsFilteredOut; read the effective filter with get-subscription-attributes
Messages arrive in some queues but never in oneThat subscription's filter is too narrow, or has the wrong scopeTest the policy against a real message; for EventBridge-published messages use MessageBody
Filter change has no effectEventual consistency, up to 15 minutesWait, then retest with a fresh publish
Consumer sees a JSON wrapper with Type and MessageRaw message delivery is offset-subscription-attributes --attribute-name RawMessageDelivery --attribute-value true
Messages silently discardedMore than 10 message attributes to a raw-delivery SQS subscriptionReduce attributes to 10 or fewer
InvalidParameter: Invalid parameter: FilterPolicyMore than five keys, more than 150 combinations, or invalid JSONRun the policy through FilterPolicyComplexity and a JSON linter
KMS.AccessDeniedException-style delivery failuresEncrypted queue or DLQ with a key policy that does not trust SNSAdd sns.amazonaws.com to the key policy; use a customer managed key (the AWS managed key cannot be shared)
AuthorizationError on publishPublisher role lacks sns:Publish, or the topic policy denies itCheck the IAM policy and the topic policy with the policy simulator
PublishBatch returns Failed entriesThrottling or an invalid entryRetry the failed IDs with backoff; log the Code
Order of notifications is wrongStandard topic; no ordering guaranteeUse a FIFO topic with an account-level group, or put sequence information in the payload

Cost and cleanup

Cost: free at lab volumes. SNS charges per publish (each 64 KB chunk is one request) and per delivery to some endpoint types. Delivery to SQS and Lambda carries no per-message delivery fee. SQS requests and Lambda invocations are billed by their own services and are covered by their free tiers at these volumes. Public pricing is about USD 0.50 per million publishes for standard topics ⚠ VERIFY on the SNS pricing page, and the monthly free allowances are listed there too. SMS and e-mail deliveries are priced separately, which is why this lesson does not use them. This lesson sends a few dozen messages.

Free alternative: run the offline matcher and the unit tests, which need no AWS account. For a local broker-style topic, LocalStack or an SNS-compatible emulator can be used for demos, but verify filter behaviour against real SNS before you rely on it.

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1

# Remove subscriptions first, then the topic, then the queues.
for sub in $(aws sns list-subscriptions-by-topic --topic-arn "$TOPIC_ARN" \
    --query 'Subscriptions[].SubscriptionArn' --output text); do
  aws sns unsubscribe --subscription-arn "$sub"
done
aws sns delete-topic --topic-arn "$TOPIC_ARN"

for q in hl-m08gap-sms hl-m08gap-sms-dlq hl-m08gap-highvalue hl-m08gap-highvalue-dlq hl-m08gap-subscription-dlq hl-m08gap-ledger.fifo; do
  url=$(aws sqs get-queue-url --queue-name "$q" --query QueueUrl --output text 2>/dev/null) \
    && aws sqs delete-queue --queue-url "$url" && echo "deleted $q"
done
rm -f queue-policy.json

# If you deployed with IaC, destroy instead:
# cdk destroy HlM08Gap-FanOut      |      terraform destroy

# Confirm nothing tagged for this lab remains (can lag a few minutes).
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m08-messaging-services \
  --query 'ResourceTagMappingList[].ResourceARN'

After deleting a queue, wait 60 seconds before re-creating one with the same name.

Exercises

  • 🟢 Guided: Deploy the fan-out, publish the three messages from the routing table with the .NET publisher (or the CLI), and read each queue. Then change notify-sms to {"channel":["sms","email"]} and republish after the propagation delay. Record which messages changed route and when.
  • 🟡 Challenge: Add a subscription for "Fraud scoring" with a policy meaning "currency is anything but GBP, or amount at least 5,000". Think before you write: keys are ANDed, so this is two subscriptions (or one body policy using $or, if you verify it in the current docs). Prove each case with a published message and the FilteredOut metric.
  • 🔴 Live problem: Ticket from the Collections team: "We subscribed collections-queue to the transactions topic with {"type":["OVERDRAFT_FEE"]} an hour ago. Fees are being posted (we see them in the ledger) and the queue is empty. The topic shows deliveries to other queues. The adapter team says nothing changed." Work out the likely causes in order and how you would check each without redeploying anything.

Assignment

Add sns-fanout-design.md to your portfolio repository for one Harbourline flow of your choice. Include: the topic (standard or FIFO, and why), the message contract (attributes and body, identifiers only), a table of subscribers with their filter policies and the FilterPolicyComplexity result for each, the queue and DLQ settings behind each subscriber, both alarms you would create (subscription DLQ, consumer DLQ), the IAM policies for publisher and consumers, the KMS decision, and a monthly cost estimate at 20 million publishes with five subscribers.

Solutions

Knowledge check

Interview questions

  • "Why SNS to SQS rather than SNS to Lambda?" — A queue adds durability beyond SNS's retry window, controls concurrency (so a backlog after an outage does not stampede a downstream), and gives a DLQ and metrics. SNS to Lambda directly is fine for fire-and-forget, idempotent work.
  • "How do you stop every consumer receiving every message?" — Subscription filter policies on attributes or body, on the subscription not the publisher. Keys AND, values OR. Remember the limits and the 15-minute propagation.
  • "What happens if a consumer is down for a day?" — The consumer's own SQS queue holds the messages (up to 14 days retention). SNS has already delivered to the queue, so SNS retry is not the safety net here; the queue is.
  • "Standard or FIFO topic?" — Standard by default, with idempotent consumers. FIFO when downstream order per key matters and the fan-out is small and bounded, remembering the 100-subscription and throughput limits.
  • "How do you keep personal data out of a fan-out?" — Publish identifiers and facts only, never names, balances or full account numbers. Consumers that need more look it up with their own permissions. SNS Message Data Protection is closed to new customers, so do not rely on it.

Certification practice

Completion checklist

  • I can explain why a queue sits behind every SNS subscription
  • I can write attribute-scope and body-scope filter policies and say which publishers need which
  • I can compute a policy's combination count and know the five-key limit
  • I deployed the fan-out and saw one publish reach two queues and be filtered from a third
  • I can name the three authorisation checks (topic IAM, queue policy, KMS key policy) that can break a fan-out
  • I ran the cleanup and the tag check returned []

Key takeaways

  • SNS is push with no storage: put a queue and a DLQ behind every subscriber that matters, and a subscription DLQ for delivery failures.
  • Filter policies live on the subscription. Keys are ANDed, values ORed, at most five keys and 150 combinations, and changes can take 15 minutes.
  • Attribute scope needs a publisher that sets attributes. EventBridge-published messages need MessageBody scope.
  • The queue policy with an aws:SourceArn condition is as important as the subscription: without it nothing arrives.
  • Publish identifiers, not personal data, and check Failed in every batch response.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page