Amazon SNS: fan-out and message filtering
Publish one message to an Amazon SNS topic and deliver it to many consumers: SNS to SQS fan-out, subscription filter policies on attributes and on the message body, delivery retries, subscription dead-letter queues and FIFO topics, with .NET 10 publishers, CDK and Terraform.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SimpleNotificationService 4.1.0.2) · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain what an SNS topic, subscription and publish are, and why the standard banking pattern is SNS to SQS fan-out rather than SNS straight to a service.
- Write subscription filter policies on message attributes and on the message body, predict which subscriptions receive a message, and respect the documented limits (five keys, 150 combinations).
- Describe SNS delivery retries per endpoint type, and attach a dead-letter queue to a subscription for the failures SNS cannot fix itself.
- Choose between standard and FIFO topics, and between raw and enveloped delivery.
- Publish single and batched messages from .NET 10 with the AWS SDK v4, and check a batch for partial failure.
- Deploy a filtered fan-out with CDK (C#) or Terraform, and tear it down.
Prerequisites
- Asynchronous communication and event-driven architecture: queue, topic, bus and stream compared
- Amazon SQS in depth and Dead-letter queues and redrive
- Module 5: Security: IAM resource policies and KMS key policies (you will write both)
- Tools: .NET 10 SDK or Docker, AWS CLI v2 with the
learn-sandboxSSO profile, optionally the CDK CLI or Terraform 1.14
Why this matters in banking
A posted transaction is interesting to many parts of Harbourline: Notifications (text and e-mail), Fraud (only large or foreign payments), Rewards (only card spend), the Data Platform (everything). The first design had the core banking adapter call each of them in turn. Two things went wrong in the first month:
- The Rewards service was down for forty minutes. Because the adapter called it synchronously, postings backed up behind it and the adapter's own queue overflowed.
- A team added a new consumer, "Collections", and had to ask the adapter team to add one more call, test it and deploy. The change took three weeks for what should have been a configuration change.
A topic fixes both. The adapter publishes once and does not know who listens. Each consumer subscribes on its own, with its own queue and its own filter. If Rewards is down, its queue grows and nothing else notices. A new team subscribes without a ticket to the adapter team.
The sharp edges are the ones this lesson spends time on: a filter that silently matches nothing, a message published without the attribute the filter needs, a subscription whose queue policy was forgotten, and personal data in a message that now fans out to twelve places.
The idea in plain language
Amazon Simple Notification Service (SNS) is a managed publish/subscribe service. You create a topic. Producers publish messages to the topic. Anything subscribed to the topic receives its own copy. SNS pushes: it calls the subscriber, instead of the subscriber asking for work.
A subscriber, called an endpoint, can be an SQS queue, a Lambda function, an HTTPS URL, an e-mail address, an SMS number, a mobile push endpoint or a Firehose delivery stream.
A filter policy on a subscription says "only send me messages that look like this". Without one, a subscription gets everything.
Analogy: A bank's internal circular. The compliance team writes one circular and puts it in the post room's "all staff" distribution list (the topic). The post room copies it into each subscribed department's tray (the SQS queues). A department can say "only send me circulars about card fraud" (a filter policy). The author never learns who is subscribed, and a department that is on holiday just finds a taller pile when it returns.
The key property is the one in the analogy: SNS does not store messages for you. It delivers to the subscribers that exist at the moment of publishing, retries for a while, and then drops the message unless you attached a dead-letter queue. The durable part of the design is the SQS queue behind each subscription.
What problem Amazon SNS solves
SNS removes the producer's knowledge of its consumers. It gives you:
- Fan-out: one publish, many deliveries, in parallel.
- Protocol bridging: the same message can go to a queue, a function and a webhook.
- Filtering at the subscription: consumers choose; producers do not change.
- Very high publish throughput with no capacity planning.
Limits and defaults (checked 2026-11-18)
| Setting | Value |
|---|---|
| Maximum message size | 256 KiB by default; raise the topic's MaximumMessageSize attribute up to 1 MiB. Larger payloads: store in S3 and publish a pointer (the SNS Extended Client Libraries do this up to 2 GB) |
| Topics per account | 100,000 standard; 1,000 FIFO |
| Subscriptions per topic | 12,500,000 standard; 100 FIFO |
| Filter policies | 200 per topic; 10,000 per account |
PublishBatch | Up to 10 entries per request |
| Publish rate, us-east-1 | 30,000 messages per second per account, Publish and PublishBatch combined (soft quota) |
| FIFO throughput | Up to 3,000 messages per second per topic (or 20 MB per second) in the default Topic scope; 300 messages per second per message group |
| Message attributes to an SQS subscription with raw delivery | Maximum 10 |
| Filter policy changes | Can take up to 15 minutes to take full effect |
| Billing | Each 64 KB chunk of a publish is one request. SNS does not charge a per-message delivery fee for SQS or Lambda endpoints |
Honest limitations
- No storage, no replay on standard topics. A message published when a subscriber is down is retried, then dropped. Put a queue behind every subscriber that matters. (FIFO topics can archive and replay; see below.)
- No ordering on standard topics. Messages may arrive out of order and more than once.
- Filtering is not routing logic. You cannot transform a message, join two fields or do arithmetic. If you need that, use EventBridge.
- A filtered-out message is not an error. It simply never arrives. Debugging a wrong filter means reading metrics, not exceptions.
- Personal data multiplies. One publish puts a copy of the message in every subscriber's queue, each with its own retention.
Service availability — verified 2026-11-18
Amazon SNS Message Data Protection (the feature that inspects and masks sensitive data in topics) is in maintenance and closed to new customers. Do not design around it. Keep personal data out of messages: publish identifiers and facts, and let authorised consumers fetch the rest. SNS itself, SQS, Lambda and the rest of the features in this lesson are fully available.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Service Bus topic with subscriptions | SNS topic with subscriptions | A Service Bus subscription is itself a durable queue. An SNS subscription is only a delivery rule: the durable queue is a separate SQS queue you attach |
| Service Bus subscription rules (SQL filter, correlation filter) | SNS filter policy | Filter policies are JSON, not SQL. They match on equality, prefix, suffix, numeric ranges and existence, not on expressions |
Service Bus ApplicationProperties | SNS message attributes | Attributes are the default filter scope. In Service Bus the broker filters on properties by default too |
| Event Grid topic and subscription | SNS topic, or an EventBridge bus | Event Grid is closer to EventBridge. SNS is closer to Service Bus topics, without the storage |
| Service Bus sessions and duplicate detection | SNS FIFO topic with MessageGroupId and deduplication | FIFO topics only deliver to FIFO-compatible subscribers, and have a 100-subscription limit |
| Service Bus dead-letter sub-queue | SNS subscription redrive policy to an SQS queue | SNS only dead-letters delivery failures. A message the consumer cannot process is the consumer queue's DLQ problem |
Azure.Messaging.ServiceBus ServiceBusSender | AmazonSimpleNotificationServiceClient.PublishAsync | No connection to manage: each call is an HTTPS request signed with SigV4 |
Delivery semantics and retries
SNS defines a delivery policy per protocol. The numbers below are from the AWS documentation (checked 2026-11-18).
| Endpoint type | Retry behaviour on server-side errors | Can you change it? |
|---|---|---|
| SQS, Lambda (AWS managed) | 3 immediate retries, 2 at one second, 10 with exponential backoff from 1 to 20 seconds, then up to 100,000 retries 20 seconds apart: 100,015 attempts over about 23 days | No |
| SMTP, SMS, mobile push | 2 retries 10 seconds apart, 10 with backoff from 10 seconds to 10 minutes, 38 at 10 minutes: 50 attempts over about 6 hours | No |
| HTTP/S | A four-phase policy (immediate, pre-backoff, backoff, post-backoff) that you can customise. The documented example makes 50 attempts. The total retry time cannot exceed 3,600 seconds | Yes, at topic or subscription level |
Two rules decide what happens next:
- Client-side errors are not retried. If the endpoint no longer exists, or a policy now denies SNS, the delivery fails immediately. Examples: a deleted queue, a missing queue policy, KMS denying SNS access to an encrypted queue.
- Server-side errors are retried with jitter. For SQS and Lambda endpoints that means "for weeks", which is long enough that retries are rarely your problem. A wrong IAM policy is.
When the policy is exhausted, or on a client-side error, SNS discards the message unless the subscription has a dead-letter queue. The DLQ is an ordinary SQS queue in the same account and Region, and its type must match the topic: a standard topic uses a standard queue, a FIFO topic a FIFO queue.
Two different dead-letter queues
The subscription DLQ catches messages SNS could not put into the consumer's queue. The consumer queue's DLQ (the SQS maxReceiveCount one from the previous lesson) catches messages the consumer received and could not process. You usually want both, and an alarm on each.
Message filtering
Two scopes
Set the subscription attribute FilterPolicyScope:
| Scope | The policy is matched against | Who can publish for it |
|---|---|---|
MessageAttributes (the default) | The message attributes you pass with Publish | Only publishers that set attributes: your own code. EventBridge and S3 do not set your attributes |
MessageBody | The JSON message body. The body must be well-formed JSON | Anyone, including EventBridge rules that target the topic |
The lab project of this module uses MessageBody because EventBridge is the publisher. When you publish from .NET yourself, attributes are cheaper to reason about and keep routing metadata out of the payload.
The rules
- Keys are ANDed. Every key in the policy must match.
- Values in a key's array are ORed.
"currency": ["GBP", "EUR"]means GBP or EUR. - Properties not named in the policy are ignored.
- An array in the message matches if any element matches. A message attribute of type
String.Arraywith["sms","email"]matches the policy{"channel":["email"]}. - A key that is missing from the message does not match, unless the policy uses
{"exists": false}for it. - String matching is case-sensitive.
Operators
| Operator | Example | Matches |
|---|---|---|
| Exact string | "channel": ["sms"] | the string sms |
| Exact number | "amount": [100] | the number 100 |
prefix | "type": [{"prefix": "CARD_"}] | CARD_PURCHASE, CARD_REFUND |
suffix | "file": [{"suffix": ".pdf"}] | strings ending .pdf |
equals-ignore-case | "ccy": [{"equals-ignore-case": "gbp"}] | GBP, Gbp |
anything-but | "channel": [{"anything-but": ["push"]}] | everything except push (also works with prefix inside) |
numeric | "amount": [{"numeric": [">=", 1000, "<", 50000]}] | 1000 up to but not including 50000 |
exists | "promo": [{"exists": false}] | messages without a promo key |
cidr | "ip": [{"cidr": "10.0.0.0/24"}] | IP addresses in the range |
wildcard | "name": [{"wildcard": "inv-*.pdf"}] | glob-style match (complexity capped at 100 points) |
Numbers must be between -1 billion and 1 billion with up to five digits after the decimal point.
Limits you will actually hit
- At most five keys per policy. In a nested (body) policy only the leaf keys count.
- At most 150 combinations: multiply the number of values in each key. Three values times one times two is 6. A key with 12 values and another with 13 is 156 and is rejected.
- A policy can be up to 256 KB, but you will never get close to the other limits first.
- Changes take up to 15 minutes to take full effect. Do not conclude "my filter is wrong" thirty seconds after saving it.
The FilterPolicyComplexity helper below checks the key and combination limits in your CI pipeline, so the mistake does not first appear at deploy time.
Seeing filtering in CloudWatch
Filtered messages are counted, not logged. The SNS topic metrics include NumberOfMessagesPublished, NumberOfNotificationsDelivered, NumberOfNotificationsFailed and NumberOfNotificationsFilteredOut (with variants that say why: no message attributes, invalid attributes, or message body ⚠ VERIFY the exact suffixes in the current metrics page). If Published rises, Delivered stays flat and FilteredOut rises, your filter excludes everything, which is almost always a scope mismatch.
Raw delivery
By default SNS wraps each message for SQS and HTTP endpoints in a JSON envelope with Type, MessageId, TopicArn, Message (the body, as a string), Timestamp, a signature and an unsubscribe URL. Your consumer then has to unwrap twice: once for the envelope, once for your own JSON.
With raw message delivery on a subscription, SNS strips its own metadata and delivers the body as it was published. For SQS subscriptions, up to 10 message attributes are also passed on. Beyond 10 the message is discarded as a client error, so keep attributes few.
Use raw delivery for your own services: the consumer code is the same whether the message came from SNS, EventBridge or a direct SendMessage. Keep the envelope when you need MessageId and Timestamp from SNS, or when an HTTPS endpoint must verify the SNS signature.
FIFO topics
A FIFO topic keeps strict order per MessageGroupId and removes duplicate publishes (same MessageDeduplicationId within five minutes, or content-based deduplication). Its name ends in .fifo.
| Standard topic | FIFO topic | |
|---|---|---|
| Order | None | Per message group |
| Duplicates | Possible | Removed at publish within the 5-minute window |
| Subscribers | All protocols | SQS FIFO queues for sure; ⚠ VERIFY the current list of other supported protocols (standard SQS, Lambda, HTTP/S, Firehose) before you rely on one |
| Subscriptions per topic | 12.5 million | 100 |
| Throughput | 30,000 per second (us-east-1 account quota) | 3,000 per second per topic, 300 per group |
| Archive and replay | Not available | Yes: an archive policy keeps messages, a replay policy redelivers them |
Use FIFO when order matters downstream (a ledger projection that applies postings per account) and the subscriber count is small. Use standard and an idempotent consumer when it does not. FIFO's order is per group, so choose the group key as you would for an SQS FIFO queue: account ID, not a constant.
Architecture
The module project (the Harbourline notification system) puts SNS between EventBridge and the per-channel queues. The pattern generalises to any fan-out.
Security-boundary view
Three authorisation checks are in play, and each one can break a fan-out: the publisher's IAM policy on the topic, the queue policy that lets sns.amazonaws.com send to the queue, and (for encrypted queues) the KMS key policy that lets SNS use the key.
Step-by-step implementation
Create the topic, the queues and their DLQs
Create a standard topic hl-m08gap-transaction-notices, a queue per consumer (hl-m08gap-sms, hl-m08gap-highvalue), a DLQ for each queue with maxReceiveCount 5, and one shared subscription DLQ. Enable server-side encryption on the queues and enforce TLS. The Infrastructure as Code section does all of this in one deploy.
Allow SNS to send to each queue
An SQS queue accepts messages from SNS only if its queue policy allows sqs:SendMessage to the principal sns.amazonaws.com with a condition ArnEquals on aws:SourceArn set to your topic. Without the condition, any SNS topic in any account that knows the queue ARN could write to it. CDK and the Terraform in this lesson add the condition for you; the CLI route below shows it explicitly.
Subscribe each queue with a filter and raw delivery
Create the subscription with the attributes RawMessageDelivery=true, a FilterPolicy and (when matching on the body) FilterPolicyScope=MessageBody. Add a RedrivePolicy pointing at the subscription DLQ.
Publish with attributes
Use the .NET publisher below. It sets channel as String.Array, currency as String and amount as Number, so the three policies in this lesson work.
Prove the routing
Publish three notices: an SMS for £25, an e-mail for £5,000 and an SMS for €1,200. Receive from each queue and confirm the matrix in the next section. Then look at the topic's NumberOfNotificationsFilteredOut metric.
Expected routing for the three test messages with the two filters from the CDK stack:
| Message | notify-sms (channel is sms) | high-value (GBP or EUR, amount at least 1000) |
|---|---|---|
| SMS, GBP 25 | Delivered | Filtered out (25 is under 1000) |
| E-mail, GBP 5,000 | Filtered out (channel is not sms) | Delivered |
| SMS, EUR 1,200 | Delivered | Delivered (one publish, two copies) |
AWS Console instructions
Console → SNS → Topics → Create topic. Choose Standard, name it, leave encryption on if offered. Then Subscriptions → Create subscription: pick the topic, protocol Amazon SQS, endpoint your queue ARN. Open Subscription filter policy, switch on the policy, choose the scope (Message attributes or Message body) and paste the JSON. Tick Enable raw message delivery. Under Redrive policy (dead-letter queue) pick the subscription DLQ.
Add the queue policy on the SQS side: Console → SQS → your queue → Access policy → Edit, and paste the statement from the CLI section. The SQS console can also create the subscription and update the policy for you (SNS subscriptions → Subscribe to Amazon SNS topic) ⚠ VERIFY the current menu name. Whichever way you do it, check that the resulting policy contains the aws:SourceArn condition.
To see filtering in action, Topic → Monitoring shows the delivered and filtered-out graphs.
AWS CLI
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
# 1. Topic and queues.
TOPIC_ARN=$(aws sns create-topic --name hl-m08gap-transaction-notices \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m08-messaging-services \
--query TopicArn --output text)
QUEUE_URL=$(aws sqs create-queue --queue-name hl-m08gap-sms \
--attributes SqsManagedSseEnabled=true --query QueueUrl --output text)
QUEUE_ARN=$(aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
--attribute-names QueueArn --query Attributes.QueueArn --output text)
# 2. Queue policy: only THIS topic may send. The SourceArn condition is the important line.
cat > queue-policy.json <<EOF
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": { "Service": "sns.amazonaws.com" },
"Action": "sqs:SendMessage",
"Resource": "$QUEUE_ARN",
"Condition": { "ArnEquals": { "aws:SourceArn": "$TOPIC_ARN" } }
}]
}
EOF
aws sqs set-queue-attributes --queue-url "$QUEUE_URL" \
--attributes "{\"Policy\": $(jq -Rs . < queue-policy.json)}"
# 3. Subscribe with raw delivery and an attribute-scoped filter policy.
SUB_ARN=$(aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol sqs \
--notification-endpoint "$QUEUE_ARN" --return-subscription-arn \
--attributes '{"RawMessageDelivery":"true","FilterPolicyScope":"MessageAttributes","FilterPolicy":"{\"channel\":[\"sms\"]}"}' \
--query SubscriptionArn --output text)
# 4. Publish with message attributes. String.Array lets one attribute hold several values.
aws sns publish --topic-arn "$TOPIC_ARN" \
--message '{"transactionId":"T-0001","customerId":"CUST-1","amount":25.00,"currency":"GBP","channels":["sms"]}' \
--message-attributes '{"channel":{"DataType":"String.Array","StringValue":"[\"sms\"]"},"currency":{"DataType":"String","StringValue":"GBP"},"amount":{"DataType":"Number","StringValue":"25.00"}}'
# 5. Receive (long poll), then look at what filtering did.
aws sqs receive-message --queue-url "$QUEUE_URL" --wait-time-seconds 10 --max-number-of-messages 10
# Inspect and change a filter later (takes up to 15 minutes to be fully effective).
aws sns get-subscription-attributes --subscription-arn "$SUB_ARN" \
--query 'Attributes.{scope:FilterPolicyScope,policy:FilterPolicy,raw:RawMessageDelivery}'
aws sns set-subscription-attributes --subscription-arn "$SUB_ARN" \
--attribute-name FilterPolicy --attribute-value '{"channel":["sms","email"]}'Infrastructure as Code
The stack creates the topic with TLS enforced, a consumer queue and DLQ per subscriber, a shared subscription DLQ, two filtered subscriptions with raw delivery, and a FIFO topic with a FIFO queue. The SqsSubscription construct writes the queue policy with the aws:SourceArn condition for you.
using Amazon.CDK;
using Amazon.CDK.AWS.SNS;
using Amazon.CDK.AWS.SNS.Subscriptions;
using Amazon.CDK.AWS.SQS;
using Constructs;
namespace Harbourline.Messaging.Infra;
/// <summary>
/// One SNS topic fanning out to a queue per consumer. Every queue has its own DLQ, and every subscription has a
/// redrive DLQ for the case where SNS itself cannot deliver (a deleted queue, a broken policy, KMS denied).
/// Filters live on the subscription, so a consumer changes what it receives without touching the publisher.
/// </summary>
public sealed class FanOutStack : Stack
{
public Topic Topic { get; }
public FanOutStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
Topic = new Topic(this, "NoticeTopic", new TopicProps
{
TopicName = "hl-m08gap-transaction-notices",
EnforceSSL = true, // topic policy that denies non-TLS publishes
});
// Subscription DLQ: SNS-level failures (not consumer failures) land here. Standard topic -> standard queue.
var subscriptionDlq = new Queue(this, "SubscriptionDlq", new QueueProps
{
QueueName = "hl-m08gap-subscription-dlq",
RetentionPeriod = Duration.Days(14),
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
});
AddConsumer("Sms", subscriptionDlq, new Dictionary<string, SubscriptionFilter>
{
// Attribute scope (the default): the publisher sets these as message attributes.
["channel"] = SubscriptionFilter.StringFilter(new StringConditions { Allowlist = ["sms"] }),
});
AddConsumer("HighValue", subscriptionDlq, new Dictionary<string, SubscriptionFilter>
{
// AND across keys, OR inside a key: GBP or EUR, AND amount of 1000 or more.
["currency"] = SubscriptionFilter.StringFilter(new StringConditions { Allowlist = ["GBP", "EUR"] }),
["amount"] = SubscriptionFilter.NumericFilter(new NumericConditions { GreaterThanOrEqualTo = 1000 }),
});
// A FIFO topic keeps order per MessageGroupId and drops duplicate publishes (5-minute window).
// Its subscribers must be FIFO queues, so the ledger projection sees one account's events in order.
var ledgerTopic = new Topic(this, "LedgerTopic", new TopicProps
{
TopicName = "hl-m08gap-ledger.fifo",
Fifo = true,
ContentBasedDeduplication = true,
EnforceSSL = true,
});
var ledgerQueue = new Queue(this, "LedgerQueue", new QueueProps
{
QueueName = "hl-m08gap-ledger.fifo",
Fifo = true,
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
});
ledgerTopic.AddSubscription(new SqsSubscription(ledgerQueue, new SqsSubscriptionProps { RawMessageDelivery = true }));
}
private void AddConsumer(string name, Queue subscriptionDlq, IDictionary<string, SubscriptionFilter> filter)
{
var dlq = new Queue(this, $"{name}Dlq", new QueueProps
{
QueueName = $"hl-m08gap-{name.ToLowerInvariant()}-dlq",
RetentionPeriod = Duration.Days(14),
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
});
var queue = new Queue(this, $"{name}Queue", new QueueProps
{
QueueName = $"hl-m08gap-{name.ToLowerInvariant()}",
VisibilityTimeout = Duration.Seconds(180),
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 },
});
Topic.AddSubscription(new SqsSubscription(queue, new SqsSubscriptionProps
{
RawMessageDelivery = true,
FilterPolicy = filter,
DeadLetterQueue = subscriptionDlq,
}));
}
}Both examples use SQS_MANAGED (SSE-SQS) encryption, which needs no key policy. If you switch the queues to a customer managed KMS key, the key policy must allow sns.amazonaws.com to use kms:GenerateDataKey and kms:Decrypt, or every delivery will fail with a client-side error.
Complete code
Two small classes carry the lesson. They were compiled and unit-tested with xUnit v3 against AWS SDK for .NET v4 and live in labs/m08-messaging-services/. To use them in your own project:
dotnet new classlib -n Harbourline.Messaging.Samples
cd Harbourline.Messaging.Samples
dotnet add package AWSSDK.SimpleNotificationService --version 4.1.0.2using System.Globalization;
using System.Text.Json;
using Amazon.SimpleNotificationService;
using Amazon.SimpleNotificationService.Model;
namespace Harbourline.Messaging.Samples.Sns;
/// <summary>The minimum a notification needs: identifiers and facts, never names, full IBANs or balances.</summary>
public sealed record TransactionNotice(
string TransactionId,
string CustomerId,
decimal Amount,
string Currency,
IReadOnlyList<string> Channels);
/// <summary>
/// Publishes notices to an SNS topic with message attributes, so subscribers can use the default
/// attribute-scoped filter policies. This only works when YOU publish: EventBridge and S3 do not set your attributes.
/// </summary>
public sealed class TransactionTopicPublisher(IAmazonSimpleNotificationService sns, string topicArn)
{
// PublishBatch accepts at most 10 entries per call.
public const int MaxBatchSize = 10;
public async Task<string> PublishAsync(TransactionNotice notice, CancellationToken ct = default)
{
var response = await sns.PublishAsync(ToRequest(notice), ct);
return response.MessageId;
}
/// <summary>Publishes in batches of 10 and returns the transaction ids SNS rejected, so the caller retries only those.</summary>
public async Task<IReadOnlyList<string>> PublishManyAsync(
IReadOnlyList<TransactionNotice> notices, CancellationToken ct = default)
{
var failed = new List<string>();
foreach (var chunk in notices.Chunk(MaxBatchSize))
{
var request = new PublishBatchRequest
{
TopicArn = topicArn,
PublishBatchRequestEntries = chunk.Select(n =>
{
var single = ToRequest(n);
return new PublishBatchRequestEntry
{
Id = n.TransactionId, // unique inside the batch; echoed back in Failed[]
Message = single.Message,
MessageAttributes = single.MessageAttributes,
};
}).ToList(),
};
var response = await sns.PublishBatchAsync(request, ct);
// A batch call can succeed (HTTP 200) while individual entries fail. Always read Failed.
failed.AddRange(response.Failed.Select(f => f.Id));
}
return failed;
}
private PublishRequest ToRequest(TransactionNotice notice) => new()
{
TopicArn = topicArn,
Message = JsonSerializer.Serialize(notice, JsonSerializerOptions.Web),
MessageAttributes = new Dictionary<string, MessageAttributeValue>
{
// "String.Array" lets one attribute hold several channels; the policy {"channel":["sms"]} matches if ANY element matches.
["channel"] = new() { DataType = "String.Array", StringValue = JsonSerializer.Serialize(notice.Channels) },
["currency"] = new() { DataType = "String", StringValue = notice.Currency },
// DataType "Number" lets a policy use {"numeric": [">=", 1000]}.
["amount"] = new() { DataType = "Number", StringValue = notice.Amount.ToString(CultureInfo.InvariantCulture) },
},
};
}using System.Text.Json.Nodes;
namespace Harbourline.Messaging.Samples.Sns;
/// <summary>
/// Pre-flight check of an SNS subscription filter policy against the documented limits: at most 5 keys and at most
/// 150 combinations. Combinations = the product of the match values in each key; in a nested (MessageBody scope)
/// policy only LEAF keys count, and each leaf's value count is multiplied by its nesting depth.
/// SNS stays the source of truth: this catches the mistake in CI instead of at deployment.
/// </summary>
public static class FilterPolicyComplexity
{
public const int MaxKeys = 5;
public const int MaxCombinations = 150;
public sealed record Result(int LeafKeys, int Combinations, bool WithinLimits);
public static Result Measure(string policyJson)
{
var root = JsonNode.Parse(policyJson) as JsonObject
?? throw new ArgumentException("A filter policy must be a JSON object.", nameof(policyJson));
var leaves = new List<(int Values, int Depth)>();
Walk(root, depth: 1, leaves);
// long, not int: a careless policy can overflow int before we get the chance to report it.
long combinations = 1;
foreach (var (values, depth) in leaves)
{
combinations *= (long)values * depth;
if (combinations > int.MaxValue) { combinations = int.MaxValue; break; }
}
var total = (int)combinations;
return new Result(leaves.Count, total, leaves.Count <= MaxKeys && total <= MaxCombinations);
}
private static void Walk(JsonObject node, int depth, List<(int Values, int Depth)> leaves)
{
foreach (var (_, value) in node)
{
switch (value)
{
case JsonObject child:
Walk(child, depth + 1, leaves);
break;
case JsonArray values:
leaves.Add((Math.Max(values.Count, 1), depth));
break;
default:
throw new ArgumentException("Policy values must be objects (nesting) or arrays (match values).");
}
}
}
}The tests (excerpt) pin the documented examples: the flat policy is 6 combinations, the nested policy is 72, and a policy with 156 combinations is rejected.
[Fact]
public void Nested_policy_counts_leaf_keys_and_multiplies_by_depth()
{
// (4 x 3) x (3 x 2) = 72, two leaf keys.
var result = FilterPolicyComplexity.Measure("""
{
"key_a": { "key_b": { "key_c": ["1","2","3","4"] } },
"key_d": { "key_e": ["1","2","3"] }
}
""");
Assert.Equal(2, result.LeafKeys);
Assert.Equal(72, result.Combinations);
Assert.True(result.WithinLimits);
}The module project also carries an offline teaching matcher for filter policies, FilterPolicy in labs/m08-notification-system/src/Harbourline.Notifications.Core/Routing/. It matches a message body against a policy so you can unit-test routing without AWS. Treat it as a learning aid: SNS is the source of truth.
Code explained line by line
TransactionNoticecarries identifiers and facts only: no name, no full IBAN, no balance. Every subscriber's queue gets a copy, so the minimum is the safest content.MaxBatchSize = 10mirrors thePublishBatchlimit.PublishManyAsyncslices the input withChunk(10).- Each batch entry has an
Id(the transaction ID). SNS echoes it in theFailedlist, which is how the caller learns exactly which notices to retry. response.Failedis read after every batch call. APublishBatchrequest can return HTTP 200 with some entries failed; ignoringFailedsilently drops those notifications.ToRequestbuilds the attributes.channelusesDataType = "String.Array"with a JSON array as the value; that is how one attribute carries several channels.amountusesDataType = "Number"so a numeric filter works.CultureInfo.InvariantCulturekeeps the decimal separator a dot on every machine.JsonSerializerOptions.Webproduces camelCase JSON, matching the event contract in the rest of the module.- In
FilterPolicyComplexity.Measure,Walkcollects one(values, depth)pair per leaf array. The product ofvalues × depthfollows the documented formula. It useslongso that a careless policy overflows nothing before it is reported. WithinLimitsis the CI gate: at most five leaf keys and at most 150 combinations.
Security considerations
⚠ Learning shortcut — not for production
The lab uses SSE-SQS on the queues and no encryption on the topic, because the data is fictional and identifiers only. For anything linked to a customer, turn on SNS server-side encryption with a customer managed KMS key, and give the key policy statements for the publishers (kms:GenerateDataKey, kms:Decrypt) and, when EventBridge or CloudWatch alarms publish to the topic, for events.amazonaws.com or cloudwatch.amazonaws.com. Budget about USD 1 per key per month and check key-use logging in CloudTrail.
⚠ Learning shortcut — not for production
You publish with your SSO identity. In production the publisher's role gets only sns:Publish on the one topic ARN, each consumer role gets only sqs:ReceiveMessage, sqs:DeleteMessage and sqs:GetQueueAttributes on its own queue, and nobody except the pipeline can call sns:Subscribe or sns:SetSubscriptionAttributes, because a subscription is a data-exfiltration path.
- Topic policy. Deny
aws:SecureTransport = false. For cross-account publishers or subscribers, allow named principals only, and useaws:SourceAccountoraws:SourceArnconditions for service principals. - Queue policy. Always include the
aws:SourceArncondition for the topic. A queue that allowssns.amazonaws.comwithout it can be written to by any topic. - Subscriptions are access. Anyone allowed to subscribe an endpoint can read every unfiltered message. Control
sns:Subscribeand alarm on new subscriptions with a CloudTrail rule. - HTTPS endpoints. Verify the SNS message signature and confirm the subscription yourself. Never treat an inbound POST to
/snsas trusted because of its URL. - Personal data. Fan-out copies a message to every subscriber's queue, each with its own retention. Data minimisation (identifiers only) is the strongest control. Agree retention for topic-adjacent queues and DLQs with your data-protection team. This lesson does not make any system compliant with GDPR or PCI DSS: legal and security review decide that.
- Filters are not authorisation. A filter keeps messages out of a queue for efficiency. It does not stop a principal with
sqs:ReceiveMessageon another queue from reading it. Authorisation lives in IAM.
Banking use case
Harbourline's TransactionPosted fan-out, designed with this lesson:
| Subscriber | Filter | Why |
|---|---|---|
| Notifications SMS | channel contains sms | Customer opted in |
| Notifications e-mail | channel contains email | Customer opted in |
| Fraud scoring | currency not GBP, or amount at least 5,000 | Only the risky tail |
| Rewards | type has prefix CARD_ | Only card spend |
| Data Platform | none | Everything; it is the system of record for analytics |
Each subscriber has its own queue with its own DLQ and its own alarm. The adapter publishes once. When Collections later wants all overdraft-fee events, it subscribes with {"type":["OVERDRAFT_FEE"]} and nobody else deploys anything.
Common mistakes
- Filtering on attributes when EventBridge publishes. EventBridge does not set your message attributes, so every message fails an attribute filter. → Use
FilterPolicyScope=MessageBody. - Forgetting the queue policy. The subscription is confirmed but nothing arrives, and the failure is a client-side error that is not retried. → Add the
sqs:SendMessagestatement with theaws:SourceArncondition and attach a subscription DLQ so you can see failures. - Publishing a number as a string.
"amount": "1200"as aStringattribute fails a numeric filter. → UseDataType = "Number". - Expecting standard topics to keep messages. A subscriber added later does not receive history. → Use FIFO archive and replay, or an EventBridge archive, or give the consumer its own queue from day one.
- Trusting HTTP 200 from
PublishBatch. Individual entries can fail. → ReadFailed. - Wrapped payloads in consumers. Parsing
Messageout of the SNS envelope in some consumers and not others. → Turn on raw delivery everywhere you own the consumer. - Over-complex policies. More than five keys, or more than 150 combinations. → Split into two subscriptions, or move the logic to an EventBridge rule or the consumer.
- A constant
MessageGroupIdon a FIFO topic. All messages in one group serialise at 300 per second. → Group by the entity whose order matters.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Published, but nothing in the queue | Missing queue policy, or the filter excludes the message | Check the subscription DLQ and NumberOfNotificationsFailed; compare NumberOfNotificationsFilteredOut; read the effective filter with get-subscription-attributes |
| Messages arrive in some queues but never in one | That subscription's filter is too narrow, or has the wrong scope | Test the policy against a real message; for EventBridge-published messages use MessageBody |
| Filter change has no effect | Eventual consistency, up to 15 minutes | Wait, then retest with a fresh publish |
Consumer sees a JSON wrapper with Type and Message | Raw message delivery is off | set-subscription-attributes --attribute-name RawMessageDelivery --attribute-value true |
| Messages silently discarded | More than 10 message attributes to a raw-delivery SQS subscription | Reduce attributes to 10 or fewer |
InvalidParameter: Invalid parameter: FilterPolicy | More than five keys, more than 150 combinations, or invalid JSON | Run the policy through FilterPolicyComplexity and a JSON linter |
KMS.AccessDeniedException-style delivery failures | Encrypted queue or DLQ with a key policy that does not trust SNS | Add sns.amazonaws.com to the key policy; use a customer managed key (the AWS managed key cannot be shared) |
AuthorizationError on publish | Publisher role lacks sns:Publish, or the topic policy denies it | Check the IAM policy and the topic policy with the policy simulator |
PublishBatch returns Failed entries | Throttling or an invalid entry | Retry the failed IDs with backoff; log the Code |
| Order of notifications is wrong | Standard topic; no ordering guarantee | Use a FIFO topic with an account-level group, or put sequence information in the payload |
Cost and cleanup
Cost: free at lab volumes. SNS charges per publish (each 64 KB chunk is one request) and per delivery to some endpoint types. Delivery to SQS and Lambda carries no per-message delivery fee. SQS requests and Lambda invocations are billed by their own services and are covered by their free tiers at these volumes. Public pricing is about USD 0.50 per million publishes for standard topics ⚠ VERIFY on the SNS pricing page, and the monthly free allowances are listed there too. SMS and e-mail deliveries are priced separately, which is why this lesson does not use them. This lesson sends a few dozen messages.
Free alternative: run the offline matcher and the unit tests, which need no AWS account. For a local broker-style topic, LocalStack or an SNS-compatible emulator can be used for demos, but verify filter behaviour against real SNS before you rely on it.
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
# Remove subscriptions first, then the topic, then the queues.
for sub in $(aws sns list-subscriptions-by-topic --topic-arn "$TOPIC_ARN" \
--query 'Subscriptions[].SubscriptionArn' --output text); do
aws sns unsubscribe --subscription-arn "$sub"
done
aws sns delete-topic --topic-arn "$TOPIC_ARN"
for q in hl-m08gap-sms hl-m08gap-sms-dlq hl-m08gap-highvalue hl-m08gap-highvalue-dlq hl-m08gap-subscription-dlq hl-m08gap-ledger.fifo; do
url=$(aws sqs get-queue-url --queue-name "$q" --query QueueUrl --output text 2>/dev/null) \
&& aws sqs delete-queue --queue-url "$url" && echo "deleted $q"
done
rm -f queue-policy.json
# If you deployed with IaC, destroy instead:
# cdk destroy HlM08Gap-FanOut | terraform destroy
# Confirm nothing tagged for this lab remains (can lag a few minutes).
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m08-messaging-services \
--query 'ResourceTagMappingList[].ResourceARN'After deleting a queue, wait 60 seconds before re-creating one with the same name.
Exercises
- 🟢 Guided: Deploy the fan-out, publish the three messages from the routing table with the .NET publisher (or the CLI), and read each queue. Then change
notify-smsto{"channel":["sms","email"]}and republish after the propagation delay. Record which messages changed route and when. - 🟡 Challenge: Add a subscription for "Fraud scoring" with a policy meaning "currency is anything but GBP, or amount at least 5,000". Think before you write: keys are ANDed, so this is two subscriptions (or one body policy using
$or, if you verify it in the current docs). Prove each case with a published message and theFilteredOutmetric. - 🔴 Live problem: Ticket from the Collections team: "We subscribed
collections-queueto the transactions topic with{"type":["OVERDRAFT_FEE"]}an hour ago. Fees are being posted (we see them in the ledger) and the queue is empty. The topic shows deliveries to other queues. The adapter team says nothing changed." Work out the likely causes in order and how you would check each without redeploying anything.
Assignment
Add sns-fanout-design.md to your portfolio repository for one Harbourline flow of your choice. Include: the topic (standard or FIFO, and why), the message contract (attributes and body, identifiers only), a table of subscribers with their filter policies and the FilterPolicyComplexity result for each, the queue and DLQ settings behind each subscriber, both alarms you would create (subscription DLQ, consumer DLQ), the IAM policies for publisher and consumers, the KMS decision, and a monthly cost estimate at 20 million publishes with five subscribers.
Solutions
Expected routing before the change: notify-sms receives the first and third messages (channel sms); high-value receives the second and third (amount at least 1000 in GBP or EUR). After you widen the SMS filter to include email, the e-mail message also reaches notify-sms, but only for messages published after the change has propagated, which can take up to 15 minutes.
Why: filter policies are evaluated by SNS at publish time, per subscription, against the policy stored at that moment. Messages already delivered or filtered out are not re-evaluated. The one-publish, two-copies behaviour of the EUR 1,200 SMS is fan-out working as designed.
A single flat policy cannot say "currency is not GBP or amount at least 5,000", because keys are ANDed. Two clean solutions:
- Two subscriptions to the same queue, one with
{"currency":[{"anything-but":["GBP"]}]}and one with{"amount":[{"numeric":[">=",5000]}]}. A message matching both is delivered twice, so the fraud consumer must be idempotent (it should be anyway). - Body scope with
$or: SNS supports an OR operator for body policies. ⚠ VERIFY the syntax in the current docs and test it with a real publish.
Prove each case by publishing GBP 25 (neither: filtered out), EUR 25 (first), GBP 9,000 (second) and EUR 9,000 (both, so two copies), then check NumberOfNotificationsFilteredOut. Why two copies are acceptable: SNS is at-least-once anyway, and an idempotency key (the transaction ID) turns the duplicate into a no-op.
Work from cheapest check to costliest.
- Is the filter on the right scope? If the adapter publishes through EventBridge to SNS, there are no message attributes, so an attribute-scoped policy matches nothing.
get-subscription-attributesshowsFilterPolicyScope. Switch toMessageBodyand rewrite the policy to follow the body ({"detail":{"type":[...]}}) if the payload is an EventBridge envelope. - Does the policy match the real payload? Fetch one real message from a working queue and compare field names and values, including case (
OVERDRAFT_FEEversusoverdraft_fee) and nesting. - Is it just propagation? An hour is longer than the 15-minute window, so this is unlikely, but check the subscription's last modification time.
- Is delivery failing rather than filtering? Compare
NumberOfNotificationsFilteredOutwithNumberOfNotificationsFailedfor the topic. If failures rise, the queue policy is missing the topic'saws:SourceArnstatement, or the queue is KMS-encrypted with a key SNS cannot use. There is no subscription DLQ, so add one to see the evidence. - Is the event type actually present on that topic? The ledger may be fed by a different path (a batch job posting fees through a different topic).
Lasting fix: subscriptions created from IaC with a subscription DLQ and a policy test in CI, and an alarm on the subscription DLQ so "empty queue for an hour" cannot happen silently.
Knowledge check
Answer: SNS is push and stores nothing. A queue gives the consumer durability, its own pace and concurrency, a dead-letter queue and a backlog to drain after an outage. Direct HTTPS or Lambda subscriptions work for low-stakes cases, but in banking nearly every consumer gets a queue.
Answer: 4 × 5 × 8 = 160, which is over the 150-combination limit, so it is rejected. The five-key limit is satisfied.
Answer: Nothing is delivered. EventBridge does not set your message attributes, so the policy never matches. Use FilterPolicyScope=MessageBody and filter on the event's JSON.
Answer: That is a client-side error: SNS does not retry. It discards the message unless the subscription has a dead-letter queue, in which case the message goes there.
Answer: A FIFO topic with the account ID as MessageGroupId. It is limited to 100 subscriptions per topic, 3,000 messages per second per topic, 300 per second per group, and subscribers that can take FIFO delivery.
Answer: No. The response has Successful and Failed lists. Check Failed and retry those entries.
Interview questions
- "Why SNS to SQS rather than SNS to Lambda?" — A queue adds durability beyond SNS's retry window, controls concurrency (so a backlog after an outage does not stampede a downstream), and gives a DLQ and metrics. SNS to Lambda directly is fine for fire-and-forget, idempotent work.
- "How do you stop every consumer receiving every message?" — Subscription filter policies on attributes or body, on the subscription not the publisher. Keys AND, values OR. Remember the limits and the 15-minute propagation.
- "What happens if a consumer is down for a day?" — The consumer's own SQS queue holds the messages (up to 14 days retention). SNS has already delivered to the queue, so SNS retry is not the safety net here; the queue is.
- "Standard or FIFO topic?" — Standard by default, with idempotent consumers. FIFO when downstream order per key matters and the fan-out is small and bounded, remembering the 100-subscription and throughput limits.
- "How do you keep personal data out of a fan-out?" — Publish identifiers and facts only, never names, balances or full account numbers. Consumers that need more look it up with their own permissions. SNS Message Data Protection is closed to new customers, so do not rely on it.
Certification practice
A payments application publishes events to an Amazon SNS topic. Three consuming services each need only a subset of events: card events, high-value events over 10,000, and everything. The developer wants to avoid filtering code in the consumers. What should the developer do?
A. Create three topics and make the publisher decide which to use. B. Add a filter policy to each subscription and publish events with message attributes. C. Use one SQS queue and have each consumer skip messages it does not need. D. Enable raw message delivery on each subscription.
Answer: B. Filter policies are evaluated by SNS per subscription, so consumers receive only matching messages. A couples the publisher to the consumers again. C pushes the filtering into code and wastes receives. D only changes the message format.
A developer subscribes an SQS queue to an SNS topic. The subscription shows as confirmed, publishes succeed, but the queue stays empty. There is no filter policy. What is the MOST likely cause?
A. The queue's visibility timeout is too short.
B. The queue policy does not allow sns.amazonaws.com to send messages from the topic.
C. The topic is a standard topic.
D. Raw message delivery is enabled.
Answer: B. SNS needs sqs:SendMessage permission on the queue. A affects redelivery, not arrival. C is the normal case. D changes format only.
An order system must notify an inventory service, a billing service and an analytics service when an order is placed. Any service may be down for several hours and must not lose notifications, and new services must be addable without changing the order service. Which design meets these requirements?
A. The order service calls each service's REST API. B. The order service publishes to an SNS topic; each service has its own SQS queue subscribed to the topic. C. The order service publishes to an SNS topic; each service subscribes with an HTTPS endpoint. D. The order service writes to one SQS queue read by all three services.
Answer: B. The topic decouples publisher from subscribers, and per-service queues keep messages through outages. A couples them. C relies on SNS retries, which are finite for HTTPS (at most an hour with a custom policy). D makes services compete for messages instead of each getting a copy.
A ledger service must apply account postings in order per account, and two downstream services must each receive every posting. Which solution preserves order per account?
A. A standard SNS topic with two standard SQS queues. B. A FIFO SNS topic with the account ID as the message group ID and two SQS FIFO queue subscriptions. C. A standard SNS topic with a Lambda function that sorts messages. D. An SQS FIFO queue read by both services.
Answer: B. FIFO topics keep per-group order and deliver to FIFO queues. A gives no ordering. C cannot see messages it has not received. D makes the services share messages rather than each receive a copy.
Completion checklist
- I can explain why a queue sits behind every SNS subscription
- I can write attribute-scope and body-scope filter policies and say which publishers need which
- I can compute a policy's combination count and know the five-key limit
- I deployed the fan-out and saw one publish reach two queues and be filtered from a third
- I can name the three authorisation checks (topic IAM, queue policy, KMS key policy) that can break a fan-out
- I ran the cleanup and the tag check returned
[]
Key takeaways
- SNS is push with no storage: put a queue and a DLQ behind every subscriber that matters, and a subscription DLQ for delivery failures.
- Filter policies live on the subscription. Keys are ANDed, values ORed, at most five keys and 150 combinations, and changes can take 15 minutes.
- Attribute scope needs a publisher that sets attributes. EventBridge-published messages need
MessageBodyscope. - The queue policy with an
aws:SourceArncondition is as important as the subscription: without it nothing arrives. - Publish identifiers, not personal data, and check
Failedin every batch response.
Further reading
- Amazon SNS message filtering
- Amazon SNS subscription filter policies and operators
- Filter policy constraints
- Amazon SNS message delivery retries
- Amazon SNS dead-letter queues
- Amazon SNS raw message delivery
- Message ordering and deduplication with SNS FIFO topics
- Amazon SNS endpoints and quotas
- Amazon SNS pricing
- Next: Amazon EventBridge: buses, rules and Pipes
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Dead-letter queues and redrive
Stop one bad message from blocking a payment run: configure SQS dead-letter queues with a sensible maxReceiveCount, use partial batch responses in .NET Lambda handlers, alarm on DLQ depth, and redrive messages safely after a fix.
Amazon EventBridge: buses, rules, schemas and Pipes
Route business events by content with Amazon EventBridge: event buses and rules, patterns and input transformers, target retries and dead-letter queues, archive and replay, the schema registry (and its .NET gap), EventBridge Pipes, and the new enhanced custom event bus, with .NET 10, CDK and Terraform.