Kinesis Data Streams and Amazon MSK for event streaming
Use an ordered, replayable log instead of a queue: Kinesis Data Streams shards, partition keys, capacity modes, consumers and poison records, then Amazon MSK (Kafka) with IAM authentication and a .NET producer, and a clear rule for choosing between streams, queues and buses.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.Kinesis 4.1.1) · Amazon.Lambda.KinesisEvents 4.0.0 · Confluent.Kafka 2.16.0 · AWS.MSK.Auth 1.1.3 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain how a stream (an ordered, replayable log) differs from a queue, and say which banking problems each one fits.
- Describe Kinesis shards, partition keys, sequence numbers, retention and iterators, and design a partition key that keeps one account in order without creating a hot shard.
- Predict which shard a key lands on, using the MD5 hash rule, and test a key design with a sample of real keys.
- Choose between on-demand and provisioned capacity, and between shared and enhanced fan-out consumers.
- Write a producer that handles partial
PutRecordsfailure and a Lambda consumer that preserves order, reports the failed record and survives a poison record. - Describe Amazon MSK (Provisioned Standard, Provisioned Express, Serverless), connect a .NET producer with IAM authentication, and say when MSK is the right answer.
- Run the module's Kinesis ordering and replay demo and delete it within an hour.
Prerequisites
- Asynchronous communication and event-driven architecture: where streams sit in the decision tree
- Delivery guarantees and ordering and Idempotency in .NET
- Dead-letter queues and redrive: you will meet the stream version of a poison message
- Tools: .NET 10 SDK or Docker, AWS CLI v2 with the
learn-sandboxSSO profile, optionally the CDK CLI or Terraform 1.14
Why this matters in banking
Every card authorisation at Harbourline produces a small event: account, amount, merchant category, time. At peak that is thousands per second. Three teams want it, each at their own pace:
- Fraud scoring needs each account's events in order within a second or two of the authorisation.
- The data lake loads everything every few minutes.
- A live operations dashboard needs the last few minutes only.
Then the fraud team ships a better model and wants to re-score the last seven days of events with it. A queue cannot do this: once a consumer deletes a message, it is gone, and a second consumer would have to compete for the same messages. A stream keeps the events for a retention period, lets every consumer read at its own position, and lets you rewind that position to a time in the past.
Streams also come with the hardest operational problems in this module: a "hot" key that overloads one shard, a poison record that blocks a shard for a day, and consumers that fall behind without any error. This lesson is about recognising those before production does.
The idea in plain language
A queue is a to-do list: take an item, finish it, cross it off. A stream is a ledger: events are appended in order and never crossed off. Readers keep a bookmark (a position) and read forward. Two readers have two bookmarks and do not interfere. Deleting happens only when the retention period expires.
Analogy: The branch's CCTV recorder. Cameras write continuously to a loop of tape that keeps the last 24 hours (or longer, if you pay for more). Security, the fraud team and the police each watch from their own position and can rewind. Nobody removes footage by watching it. A busy till with ten cameras on it can overload one recorder channel while others are idle: that is a hot shard.
Queue or stream?
| Queue (SQS) | Stream (Kinesis, Kafka) | |
|---|---|---|
| Consumption | A message is processed by one consumer, then deleted | Every consumer reads every record at its own position |
| Replay | No | Yes, within retention (rewind to a timestamp or offset) |
| Ordering | Best effort (standard) or per group (FIFO) | Per shard (Kinesis) or partition (Kafka), strictly |
| Scaling unit | Automatic | Shards or partitions: you design the key, and the key decides the load |
| Consumer scaling | Add workers freely | At most one active reader per shard per consumer application |
| Failure of one record | That message retries on its own | A failing record blocks the rest of its shard unless you skip or park it |
| Typical latency | Milliseconds to seconds | Tens of milliseconds to a second |
| Fit | "Do this job" | "Here is what happened, in order; many things care" |
If you only need fan-out and filtering, SNS or EventBridge are simpler. If you need work distribution, SQS is simpler. Choose a stream when ordering, replay or many independent readers at high volume are real requirements.
What problem a stream solves
A durable, ordered, partitioned log lets many consumers process the same high-volume events independently, rewind, and catch up after downtime. In AWS you have two main choices:
- Amazon Kinesis Data Streams: a fully managed AWS-native stream. No brokers. You choose shards (or on-demand) and pay for capacity or throughput.
- Amazon MSK (Managed Streaming for Apache Kafka): managed Apache Kafka. You get the Kafka protocol, ecosystem and semantics (consumer groups, transactions, compaction), and you manage more of it.
Limits and defaults, Kinesis Data Streams (checked 2026-11-18)
| Setting | Value |
|---|---|
| Write per shard (provisioned) | Up to 1 MB/s or 1,000 records/s |
| Read per shard (shared throughput) | Up to 2 MB/s, 5 GetRecords calls per second |
| Record size | Up to 10 MiB before base64 (designed for intermittent large records; most are tiny) |
PutRecords | Up to 500 records and 10 MiB per request |
| Retention | 24 hours by default, extendable to 365 days |
| Registered (enhanced fan-out) consumers | 20 per stream (50 with On-demand Advantage) |
| On-demand stream throughput | New streams start at 4 MB/s write and 8 MB/s read, and scale to 10 GB/s write in us-east-1 (200 MB/s in some other Regions) |
| On-demand streams per account | 50 by default (a soft limit) |
| Provisioned shards per account | 20,000 in us-east-1, us-west-2 and eu-west-1 by default |
| Capacity-mode switches | Twice per 24 hours per stream |
| Shard iterator lifetime | 5 minutes |
| Encryption | Server-side with AWS KMS |
Limits and defaults, Amazon MSK (checked 2026-11-18)
| Setting | Value |
|---|---|
| Brokers per cluster (Standard or Express) | 30 (ZooKeeper-based) or 60 (KRaft) |
| Express broker throughput per partition | Max 15 MB/s; per broker, sustained ingress from 15.6 MB/s (express.m7g.large) to 667 MB/s (express.m8g.24xlarge) |
| Express partitions per broker | Recommended 1,000 on express.m7g.large, hard maximum 1,500 |
| Serverless, per cluster | 200 MB/s in, 400 MB/s out, 2,400 partitions (120 for compacted topics), 3,000 connections, 8 MiB maximum message |
| Serverless, per partition | 5 MB/s in, 10 MB/s out |
| IAM-authenticated connections per broker | 3,000, with a creation rate of 100 per second (4 on t3) |
Honest limitations
- You own the key design. The partition key decides both ordering and load. A bad key is a production incident that no amount of money fixes.
- One failing record can stall a shard. The default Lambda retry for Kinesis is unlimited until the record expires.
- At-least-once. A retried batch re-reads records. Consumers must be idempotent.
- Provisioned capacity is a floor you pay for, busy or not. On-demand also bills per stream-hour.
- Kinesis needs the Java-centric KCL for the richest consumer features. The Kinesis Client Library is a Java library. Other languages, including .NET, use it through a Java process called the MultiLangDaemon. For .NET the usual consumer is Lambda, a Pipe, or Firehose.
- MSK is real Kafka. It brings Kafka's operational model: partitions, replication factor, consumer-group rebalances, topic configuration, version upgrades.
- Per-record costs. Each Kinesis
PutRecordspayload is billed in 25 KB units in provisioned mode; many tiny records can cost more than their bytes suggest.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Event Hubs (Standard/Premium) | Kinesis Data Streams | Event Hubs partitions ≈ Kinesis shards. Event Hubs throughput units ≈ provisioned shards; Event Hubs auto-inflate ≈ on-demand mode. Event Hubs Capture to storage ≈ Kinesis Data Firehose to S3 |
| Event Hubs with the Kafka endpoint | Amazon MSK | MSK is actual Kafka brokers; Event Hubs speaks the Kafka protocol on top of its own service |
| Event Hubs consumer groups | Kinesis enhanced fan-out consumers, or KCL application names | A Kinesis consumer application tracks its position in a DynamoDB table (KCL) or in the Lambda event source mapping |
| EventProcessorClient (checkpoints in Blob Storage) | KCL (checkpoints in DynamoDB) | KCL is Java; .NET goes through MultiLangDaemon or uses Lambda |
| Event Hubs partition key | Kinesis partition key | Kinesis hashes the key with MD5 into a 128-bit space; shards own ranges |
| Azure Stream Analytics | Managed Service for Apache Flink, or Lambda | Not covered here |
| Event Hubs Capture | Firehose delivery to S3, or Kinesis streaming tables to S3 Tables (August 2026 ⚠ VERIFY) | |
| Azure Schema Registry | AWS Glue Schema Registry (MSK and Kinesis) | Out of scope for this lesson |
Kinesis Data Streams in detail
The vocabulary
| Term | Meaning |
|---|---|
| Stream | A named log with a retention period |
| Shard | A unit of capacity and ordering: a slice of the 128-bit hash space and a sequence of records |
| Record | A blob (data), a partition key and a sequence number assigned on write |
| Partition key | A string you choose. Its MD5 hash (read as a 128-bit unsigned integer) picks the shard whose hash range contains it |
| Sequence number | An increasing identifier within a shard. Order is guaranteed within a shard only |
| Shard iterator | A cursor for reading: TRIM_HORIZON (oldest), LATEST, AT_TIMESTAMP, AT_SEQUENCE_NUMBER, AFTER_SEQUENCE_NUMBER. Expires after 5 minutes |
| Retention | How long records stay readable: 24 hours to 365 days |
| Consumer | An application reading the stream. Shared consumers split the shard's 2 MB/s; enhanced fan-out consumers each get their own 2 MB/s per shard, pushed over HTTP/2 |
Capacity modes
| On-demand | Provisioned | |
|---|---|---|
| You decide | Nothing: no shard count | The shard count (UpdateShardCount, split, merge) |
| Scaling | Automatic, up to 10 GB/s in us-east-1 | Yours |
| Billing (us-east-1, from the pricing page's worked examples ⚠ VERIFY) | About USD 0.040 per stream-hour, USD 0.08 per GB written and USD 0.04 per GB read | About USD 0.015 per shard-hour plus about USD 0.014 per million 25 KB PUT payload units |
| Extended retention | Extra per-GB charges | About USD 0.020 per shard-hour for retention up to 7 days; long-term retention bills storage per GB-month |
| Enhanced fan-out | Extra per-GB retrieval charge | About USD 0.015 per consumer-shard-hour plus about USD 0.013 per GB |
| Best when | Spiky or unknown traffic; you do not want to size | Steady, predictable traffic where shard-hours are cheaper |
On-demand Advantage (announced November 2025) is a pricing mode for accounts with large sustained volumes: lower per-GB prices, no per-stream-hour charge, enhanced fan-out consumers at no extra charge, and a minimum charge of 25 MB/s of ingest and 25 MB/s of retrieval at the account level ⚠ VERIFY. It is the wrong choice for a learning account.
In July 2026 AWS added warm throughput scale-down, and in August 2026 streaming tables that deliver a stream to Apache Iceberg tables in S3 Tables ⚠ VERIFY both in the current documentation if you need them.
Partition keys, ordering and hot shards
Two rules decide everything:
- Records with the same partition key go to the same shard, and are read in the order written.
- A shard can absorb only 1 MB/s or 1,000 records/s of writes. A key that carries more than that overloads its shard no matter how many shards exist.
So you want a key that is stable for the entity whose order you care about (the account) and diverse across the population (millions of accounts). For card authorisations the account ID is right. A merchant ID is wrong: one very large merchant sends a large share of traffic on one key, saturates one shard and delays every account that happens to share it, while the other shards idle. That is the classic hot shard.
You can test a key design before you build anything, because the mapping is deterministic:
var shards = ShardRouter.EvenShards(4);
var spread = ShardRouter.Distribution(sampleAccountIds, shards); // about a quarter each
var hot = ShardRouter.Distribution(Enumerable.Repeat("MERCHANT-BIG", 4000), shards); // all on one shardThe ShardRouter class in this lesson implements the documented rule (MD5, unsigned big-endian, inclusive hash ranges). A stream freshly created with N shards splits the space evenly, as the helper's EvenShards does. After splits and merges the ranges differ: read the real ones from ListShards.
If one key really is hot: add a bounded salt (MERCHANT-BIG#0 to #7) so that key spreads over up to eight shards, and accept that order is now only per salt. Whether that is acceptable is a business question about what order the consumer needs.
Resharding and order. When a shard splits or merges, child shards are read only after their parent is fully read and checkpointed, which keeps per-key order. KCL and the Lambda event source mapping both handle this for you. A hand-written reader must handle it.
Producing: partial success is normal
PutRecords takes up to 500 records and is partial-success: the HTTP call succeeds, and individual records may be rejected (typically ProvisionedThroughputExceededException for a busy shard). The response has FailedRecordCount and one result per record in the same order. The producer must resend only the failed records, with backoff and jitter, and return what it still cannot send.
Choose the partition key per record: AccountId. The producer in this lesson does that and nothing else clever. For very high volumes, the Kinesis Producer Library (Java) aggregates records to save cost; from .NET, batch with PutRecords and keep records small.
Consuming and the poison-record problem
| Consumer | Notes |
|---|---|
| Lambda event source mapping | The default choice for .NET. Reads each shard, batches, retries, scales with shards. Settings below |
| Enhanced fan-out via Lambda | Register a consumer ARN for a dedicated 2 MB/s per shard; useful when several readers share a stream |
| Firehose | Delivers to S3, Redshift, OpenSearch or HTTP endpoints with buffering; no code |
| EventBridge Pipe | Kinesis source, filter, enrichment, target: point-to-point with no code |
| KCL (Java, MultiLangDaemon for other languages) | Leases shards across workers using a DynamoDB table, checkpoints, handles resharding. Powerful; more to run |
The Lambda event source mapping, setting by setting
| Setting | What it does | Recommended for ordered banking events |
|---|---|---|
StartingPosition | TRIM_HORIZON, LATEST or AT_TIMESTAMP | TRIM_HORIZON for a new consumer that must see history; AT_TIMESTAMP to replay |
BatchSize | Records per invocation (default 100, up to 10,000) | 100 |
MaximumBatchingWindowInSeconds | Wait up to this long to fill a batch | 1 |
ParallelizationFactor | Concurrent batches per shard (1 to 10) | 1 keeps per-key order. A factor above 1 still keeps order per partition key but is subtle; test it before use |
ReportBatchItemFailures | The function returns the sequence number to resume from | On |
BisectBatchOnFunctionError | Split a failing batch in two to isolate the bad record | On |
MaximumRetryAttempts | Default -1: retry until the record expires | Set it (for example 5) |
MaximumRecordAgeInSeconds | Skip records older than this | Set it (for example 6 hours) |
DestinationConfig.OnFailure | Where a description of a discarded batch is sent | An SQS queue (it records shard and sequence range, not the data) |
The trap: with the defaults, one record that always throws is retried forever, and because a shard is read in order, every record behind it in that shard waits, for up to the retention period. The iterator age grows and nothing else looks wrong. The settings above (bounded retries, record age, bisect, a failure destination) turn that into "skip it and tell someone".
Ordered semantics with partial batches. With SQS you report every failed message. With a stream you report the first failed record's sequence number and stop processing the batch; Lambda retries from that record. If you carried on past the failure, later records of the same key would be applied before the failed one. The FraudScoringFunction does exactly this.
Watch the lag. The metric that matters is GetRecords.IteratorAgeMilliseconds: how old the newest record read is. Alarm when it exceeds your latency budget. Also watch WriteProvisionedThroughputExceeded (hot shard or too few shards) and, for provisioned streams, IncomingBytes against capacity.
Replay. Create a second mapping with AT_TIMESTAMP (or reset a KCL application's lease table) to re-read last week. As with EventBridge replay, a replay is a deliberate duplicate: consumers must be idempotent, and side effects need the idempotency key.
Amazon MSK
Amazon MSK runs Apache Kafka for you: brokers, storage, patching and replication across Availability Zones. Your applications use standard Kafka clients, so existing Kafka code, tools and connectors work.
Cluster types
| Type | What you manage | Billing (from the pricing page's worked examples, us-east-1 ⚠ VERIFY) | Notes |
|---|---|---|---|
| Provisioned: Standard brokers | Broker instance type and count, storage per broker, configuration | Hourly per broker (for example about USD 0.204 for kafka.m7g.large), plus storage about USD 0.10 per GB-month | The classic MSK; tiered storage available |
| Provisioned: Express brokers | Broker size and count; storage is elastic and managed | Hourly per broker (for example about USD 0.408 for express.m7g.large), plus about USD 0.01 per GB ingested and storage per GB-month | Higher throughput per broker, faster scaling and recovery, fewer settings to tune |
| Serverless | Nothing: capacity scales automatically | About USD 0.75 per cluster-hour, USD 0.0015 per partition-hour, USD 0.10 per GB in, USD 0.05 per GB out, plus storage | Hard quotas (see above); IAM authentication only ⚠ VERIFY |
All three are expensive to leave running: the cluster-hour or broker-hours accrue 24 hours a day. The pricing page's own five-topic Serverless example comes to about USD 1,300 for a month. For learning, design only, or run Kafka in Docker.
Authentication and networking
MSK clusters live in your VPC; clients must reach them over private networking. Authentication options include IAM access control (port 9098, SigV4-based, works with IAM roles for ECS, EC2 and Lambda), SASL/SCRAM with credentials in Secrets Manager, mutual TLS and, if you allow it, unauthenticated access ⚠ VERIFY which options each cluster type supports. In a bank you choose IAM or mTLS, never unauthenticated, and encrypt in transit and at rest (a customer managed KMS key where policy demands).
Kafka in one table
| Kafka term | Kinesis equivalent |
|---|---|
| Topic | Stream |
| Partition | Shard |
| Message key | Partition key |
| Offset | Sequence number |
| Consumer group | A consumer application (KCL) |
| Replication factor | Managed for you in Kinesis (3 AZs) |
| Compacted topic | No equivalent |
| Transactions and idempotent producer | No equivalent (idempotency is yours) |
| Retention by size or time, unlimited with tiered storage | 24 hours to 365 days |
From .NET
Use Confluent.Kafka. The IAM mechanism is OAUTHBEARER with a token that the AWS.MSK.Auth package builds from your AWS credentials. The producer below also turns on EnableIdempotence and Acks.All, which gives ordered, duplicate-free writes per partition even when the producer retries.
For consuming from Lambda, an MSK event source mapping works like the Kinesis one (batches per partition, retry on error). Pipes also read MSK and self-managed Kafka.
Kinesis, MSK, SQS, SNS, EventBridge: choosing
| If you need | Choose |
|---|---|
| Hand work to workers, retry and dead-letter each job | SQS |
| Push one message to a few known subscribers quickly | SNS (with a queue behind each) |
| Route business events between teams by content, with replay of modest volumes | EventBridge |
| Ordered, replayable, high-volume events read by several independent consumers, AWS-native, least operations | Kinesis Data Streams |
| The Kafka protocol, ecosystem (Kafka Connect, Streams, existing code), compaction, transactions, very long retention, or portability | Amazon MSK |
| A legacy JMS/AMQP system must keep working | Amazon MQ (next lesson) |
The honest default for a .NET team on AWS: Kinesis unless a concrete requirement (an existing Kafka estate, Kafka Connect, compaction, transactions) points at MSK.
Architecture
Security-boundary view
Step-by-step implementation (Kinesis ordering and replay demo)
This is Lab 8.6 of the module. It deploys one on-demand stream, shows ordering per key, replays from a timestamp, and deletes everything. Keep it under an hour.
Create the stream and note the time
Create hl-m08gap-authorisations in on-demand mode with 24-hour retention. The stream costs about USD 0.04 per hour while it exists.
Write ordered events for three accounts
Send ten numbered events per account with the .NET producer or the CLI. Use the account ID as the partition key, and put a seq field in each payload.
Read one shard back and check the order
For each shard, get an iterator at TRIM_HORIZON and read the records. Within each shard, the events of an account appear in seq order, even though accounts interleave. Events of different accounts on the same shard are ordered relative to each other too, but nothing orders events on different shards.
Predict the shards
Run ShardRouter for the three account IDs against the real shard ranges from list-shards. Confirm the prediction matches where the records landed.
Replay
Note the time before step 3, write ten more events, then read again with AT_TIMESTAMP set to that time. You will see only the later events, in order: that is a rewound consumer.
Break it and watch the lag
Deploy the Lambda consumer (optional), make the handler throw on one record, and watch GetRecords.IteratorAgeMilliseconds rise while the retry settings run. Then see the bounded retry skip it and the failure destination receive a description.
Delete everything
Run the cleanup below and the tag check. A forgotten on-demand stream bills every hour.
AWS Console instructions
Console → Kinesis → Data streams → Create data stream. Name it, choose On-demand under Capacity mode (or Provisioned and a shard count), and create. On the stream, Configuration shows the retention period (edit it up to 365 days), Server-side encryption, and Enhanced fan-out. Monitoring shows IncomingRecords, IteratorAge and throttling. Data viewer reads records from a chosen shard and position without code, which is the quickest way to see ordering.
Console → MSK → Clusters → Create cluster. Choose Quick create or Custom create, then the cluster type (Provisioned with Standard or Express brokers, or Serverless). Check the Estimated cost before you press Create. Choose IAM access control, three private subnets and a security group that allows port 9098 only from your application subnets. Client information → View client information shows the bootstrap servers. Delete the cluster when finished.
AWS CLI
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
STREAM=hl-m08gap-authorisations
# On-demand stream with the default 24-hour retention and KMS encryption.
aws kinesis create-stream --stream-name "$STREAM" --stream-mode-details StreamMode=ON_DEMAND
aws kinesis wait stream-exists --stream-name "$STREAM"
aws kinesis start-stream-encryption --stream-name "$STREAM" --encryption-type KMS --key-id alias/aws/kinesis
aws kinesis add-tags-to-stream --stream-name "$STREAM" --tags Project=cea-lab,Lab=m08-messaging-services
# Note the time, then write ordered events. The partition key is the account id.
START=$(date -u +%Y-%m-%dT%H:%M:%SZ)
for acc in ACC-001 ACC-002 ACC-003; do
for seq in 1 2 3 4 5; do
aws kinesis put-record --stream-name "$STREAM" --partition-key "$acc" \
--cli-binary-format raw-in-base64-out \
--data "{\"accountId\":\"$acc\",\"seq\":$seq,\"amount\":$((seq * 10))}" \
--query '{shard:ShardId,seq:SequenceNumber}' --output text
done
done
# Where are the shards and which hash ranges do they own? (compare with ShardRouter)
aws kinesis list-shards --stream-name "$STREAM" \
--query 'Shards[].{id:ShardId,from:HashKeyRange.StartingHashKey,to:HashKeyRange.EndingHashKey}'
# Read one shard from the start. Data comes back base64-encoded.
SHARD=$(aws kinesis list-shards --stream-name "$STREAM" --query 'Shards[0].ShardId' --output text)
ITER=$(aws kinesis get-shard-iterator --stream-name "$STREAM" --shard-id "$SHARD" \
--shard-iterator-type TRIM_HORIZON --query ShardIterator --output text)
aws kinesis get-records --shard-iterator "$ITER" --limit 100 \
--query 'Records[].[PartitionKey,SequenceNumber,Data]' --output text \
| while read -r key seq data; do echo "$key $(echo "$data" | base64 -d)"; done
# Replay: read again from a point in time.
ITER=$(aws kinesis get-shard-iterator --stream-name "$STREAM" --shard-id "$SHARD" \
--shard-iterator-type AT_TIMESTAMP --timestamp "$START" --query ShardIterator --output text)
# Retention: raise to 7 days (costs extra) and back.
aws kinesis increase-stream-retention-period --stream-name "$STREAM" --retention-period-hours 168
aws kinesis decrease-stream-retention-period --stream-name "$STREAM" --retention-period-hours 24
# MSK (design only unless you accept the cost): list what exists and the Kafka versions available.
aws kafka list-clusters-v2 --query 'ClusterInfoList[].{name:ClusterName,type:ClusterType,state:State}'
aws kafka list-kafka-versions --query 'KafkaVersions[].Version'Infrastructure as Code
The stack creates an on-demand, KMS-encrypted stream with 24-hour retention, and (when you give it a published function directory) a Lambda consumer with the safe event-source settings: ordered, one batch per shard, bounded retries, record age limit, bisect, partial-batch reporting and a failure destination.
using Amazon.CDK;
using Amazon.CDK.AWS.Kinesis;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.Lambda.EventSources;
using Amazon.CDK.AWS.SQS;
using Constructs;
namespace Harbourline.Messaging.Infra;
public sealed class StreamStackProps : StackProps
{
/// <summary>Output of dotnet publish for the consumer function. Null = create the stream only.</summary>
public string? FunctionsPublishDirectory { get; init; }
}
/// <summary>
/// An on-demand Kinesis stream and an ordered, retry-safe Lambda consumer. On-demand mode means no shard maths and
/// no UpdateShardCount, but you pay per stream-hour and per GB: delete it after the lab.
/// </summary>
public sealed class StreamStack : Stack
{
public StreamStack(Construct scope, string id, StreamStackProps props) : base(scope, id, props)
{
var stream = new Amazon.CDK.AWS.Kinesis.Stream(this, "Authorisations", new StreamProps
{
StreamName = "hl-m08gap-authorisations",
StreamMode = StreamMode.ON_DEMAND,
RetentionPeriod = Duration.Hours(24), // the default; raise it only for a real replay requirement
Encryption = StreamEncryption.MANAGED, // KMS with the AWS managed key aws/kinesis
});
// Failed batches that exhaust their retries are described here (shard, sequence range), not copied: read the stream to recover.
var failedBatches = new Queue(this, "FailedBatches", new QueueProps
{
QueueName = "hl-m08gap-failed-batches",
RetentionPeriod = Duration.Days(14),
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
});
_ = new CfnOutput(this, "StreamName", new CfnOutputProps { Value = stream.StreamName });
if (props.FunctionsPublishDirectory is not { } publishDir) return;
var scorer = new Function(this, "FraudScorer", new FunctionProps
{
FunctionName = "hl-m08gap-fraud-scorer",
Runtime = new Runtime("dotnet10", RuntimeFamily.DOTNET_CORE),
Handler = "Harbourline.Messaging.Samples::Harbourline.Messaging.Samples.Kinesis.FraudScoringFunction::HandleAsync",
Code = Code.FromAsset(publishDir),
Architecture = Architecture.ARM_64,
MemorySize = 512,
Timeout = Duration.Seconds(30),
});
scorer.AddEventSource(new KinesisEventSource(stream, new KinesisEventSourceProps
{
StartingPosition = StartingPosition.TRIM_HORIZON,
BatchSize = 100,
MaxBatchingWindow = Duration.Seconds(1),
ParallelizationFactor = 1, // 1 concurrent batch per shard keeps per-key order; raise to 10 only if order across keys is irrelevant
ReportBatchItemFailures = true, // the handler returns the sequence number to resume from
BisectBatchOnError = true, // split a failing batch to isolate the bad record
RetryAttempts = 5, // default is -1 (retry until the record expires: one poison record can block a shard for a day)
MaxRecordAge = Duration.Hours(6),
OnFailure = new SqsDlq(failedBatches),
}));
}
}Complete code
All four classes were compiled and unit-tested with xUnit v3 (see labs/m08-messaging-services/). Packages:
dotnet add package AWSSDK.Kinesis --version 4.1.1
dotnet add package Amazon.Lambda.Core --version 3.4.0
dotnet add package Amazon.Lambda.KinesisEvents --version 4.0.0
dotnet add package Confluent.Kafka --version 2.16.0
dotnet add package AWS.MSK.Auth --version 1.1.3using System.Numerics;
using System.Security.Cryptography;
using System.Text;
namespace Harbourline.Messaging.Samples.Kinesis;
/// <summary>A shard owns an inclusive range of the 128-bit hash space.</summary>
public sealed record ShardRange(string ShardId, BigInteger StartingHashKey, BigInteger EndingHashKey);
/// <summary>
/// Reproduces how Kinesis Data Streams maps a partition key to a shard: MD5(partitionKey) read as an unsigned
/// 128-bit big-endian integer, then the shard whose [StartingHashKey, EndingHashKey] range contains it.
/// Use it to PREDICT hot shards from a sample of real partition keys before choosing a key design.
/// </summary>
public static class ShardRouter
{
public static readonly BigInteger MaxHashKey = (BigInteger.One << 128) - 1;
public static BigInteger HashKey(string partitionKey)
{
var md5 = MD5.HashData(Encoding.UTF8.GetBytes(partitionKey));
return new BigInteger(md5, isUnsigned: true, isBigEndian: true);
}
/// <summary>The ranges of a stream freshly created with shardCount shards (an even split of the hash space).</summary>
public static IReadOnlyList<ShardRange> EvenShards(int shardCount)
{
ArgumentOutOfRangeException.ThrowIfLessThan(shardCount, 1);
var width = (MaxHashKey + 1) / shardCount;
var shards = new List<ShardRange>(shardCount);
for (var i = 0; i < shardCount; i++)
{
var start = width * i;
var end = i == shardCount - 1 ? MaxHashKey : start + width - 1;
shards.Add(new ShardRange($"shardId-{i:D12}", start, end));
}
return shards;
}
public static ShardRange Route(string partitionKey, IReadOnlyList<ShardRange> shards)
{
var hash = HashKey(partitionKey);
return shards.Single(s => hash >= s.StartingHashKey && hash <= s.EndingHashKey);
}
/// <summary>Counts records per shard for a sample of partition keys. One shard far above 1/N of the traffic is a hot shard.</summary>
public static IReadOnlyDictionary<string, int> Distribution(IEnumerable<string> partitionKeys, IReadOnlyList<ShardRange> shards)
{
var counts = shards.ToDictionary(s => s.ShardId, _ => 0);
foreach (var key in partitionKeys) counts[Route(key, shards).ShardId]++;
return counts;
}
}using System.Text.Json;
using Amazon.Kinesis;
using Amazon.Kinesis.Model;
namespace Harbourline.Messaging.Samples.Kinesis;
public sealed record CardAuthorisation(string AuthorisationId, string AccountId, decimal Amount, string Currency, string MerchantCategory, DateTimeOffset At);
/// <summary>
/// Writes card-authorisation events to a Kinesis stream. The partition key is the ACCOUNT id, so every event for
/// one account lands on the same shard and is read in order, while different accounts spread across shards.
/// PutRecords is a partial-success API: HTTP 200 can carry per-record errors (usually a throttled shard),
/// so the loop resends only the failed records.
/// </summary>
public sealed class AuthorisationProducer(IAmazonKinesis kinesis, string streamName, Func<int, TimeSpan>? backoff = null)
{
public const int MaxRecordsPerCall = 500;
private const int MaxAttempts = 5;
private readonly Func<int, TimeSpan> _backoff = backoff
?? (attempt => TimeSpan.FromMilliseconds(Random.Shared.Next(0, 50 * (1 << attempt))));
/// <summary>Returns the authorisations that could not be written after all attempts.</summary>
public async Task<IReadOnlyList<CardAuthorisation>> PutAsync(
IReadOnlyList<CardAuthorisation> batch, CancellationToken ct = default)
{
var pending = batch.ToList();
for (var attempt = 1; attempt <= MaxAttempts && pending.Count > 0; attempt++)
{
var stillFailing = new List<CardAuthorisation>();
foreach (var chunk in pending.Chunk(MaxRecordsPerCall))
{
var response = await kinesis.PutRecordsAsync(new PutRecordsRequest
{
StreamName = streamName,
Records = chunk.Select(a => new PutRecordsRequestEntry
{
PartitionKey = a.AccountId,
Data = new MemoryStream(JsonSerializer.SerializeToUtf8Bytes(a, JsonSerializerOptions.Web)),
}).ToList(),
}, ct);
if (response.FailedRecordCount is null or 0) continue;
// Records[i] answers request record i; an ErrorCode marks the failures.
for (var i = 0; i < chunk.Length; i++)
{
if (!string.IsNullOrEmpty(response.Records[i].ErrorCode)) stillFailing.Add(chunk[i]);
}
}
pending = stillFailing;
if (pending.Count > 0 && attempt < MaxAttempts) await Task.Delay(_backoff(attempt), ct);
}
return pending;
}
}using System.Text.Json;
using Amazon.Lambda.Core;
using Amazon.Lambda.KinesisEvents;
namespace Harbourline.Messaging.Samples.Kinesis;
public interface IFraudScorer
{
/// <summary>Must be idempotent: Kinesis delivers at least once, and a retried batch re-reads the same records.</summary>
Task ScoreAsync(CardAuthorisation authorisation, CancellationToken ct);
}
/// <summary>
/// Lambda handler for a Kinesis event source mapping with ReportBatchItemFailures enabled.
/// Records of one shard arrive in order. On the first failure we report that record's sequence number and STOP:
/// Lambda then retries from that record onward, so nothing is skipped and per-account order is preserved.
/// (With SQS you report every failed item; with an ordered stream, continuing past a failure would reorder events.)
/// </summary>
public sealed class FraudScoringFunction(IFraudScorer scorer)
{
public FraudScoringFunction() : this(new LoggingFraudScorer()) { }
public async Task<StreamsEventResponse> HandleAsync(KinesisEvent kinesisEvent, ILambdaContext context)
{
var failures = new List<StreamsEventResponse.BatchItemFailure>();
foreach (var record in kinesisEvent.Records)
{
try
{
var authorisation = JsonSerializer.Deserialize<CardAuthorisation>(record.Kinesis.Data, JsonSerializerOptions.Web)
?? throw new JsonException("Empty authorisation record.");
await scorer.ScoreAsync(authorisation, CancellationToken.None);
}
catch (Exception ex)
{
context.Logger.LogError($"Failed at sequence {record.Kinesis.SequenceNumber}: {ex.GetType().Name}");
failures.Add(new StreamsEventResponse.BatchItemFailure { ItemIdentifier = record.Kinesis.SequenceNumber });
break; // do not process later records of this batch out of order
}
}
return new StreamsEventResponse { BatchItemFailures = failures };
}
private sealed class LoggingFraudScorer : IFraudScorer
{
public Task ScoreAsync(CardAuthorisation authorisation, CancellationToken ct)
{
Console.WriteLine($"scored {authorisation.AuthorisationId}");
return Task.CompletedTask;
}
}
}using Amazon;
using AWS.MSK.Auth;
using Confluent.Kafka;
namespace Harbourline.Messaging.Samples.Kafka;
/// <summary>
/// A Kafka producer for Amazon MSK with IAM access control. Confluent.Kafka speaks OAUTHBEARER; the AWS.MSK.Auth
/// package turns your AWS credentials into the short-lived token (a signed URL) that MSK accepts.
/// EnableIdempotence + Acks.All give per-partition ordering without duplicates caused by producer retries.
/// </summary>
public static class MskIamProducer
{
public static IProducer<string, string> Create(string bootstrapServers, RegionEndpoint region)
{
var config = new ProducerConfig
{
BootstrapServers = bootstrapServers, // the "IAM" endpoint of the cluster, port 9098
SecurityProtocol = SecurityProtocol.SaslSsl,
SaslMechanism = SaslMechanism.OAuthBearer,
Acks = Acks.All,
EnableIdempotence = true, // retries cannot create duplicates or reorder within a partition
MessageTimeoutMs = 30_000,
};
return new ProducerBuilder<string, string>(config)
.SetOAuthBearerTokenRefreshHandler((client, _) =>
{
// Called when the token is about to expire. Credentials come from the normal AWS credential chain.
var (token, expiryMs) = new AWSMSKAuthTokenGenerator().GenerateAuthTokenAsync(region).GetAwaiter().GetResult();
client.OAuthBearerSetToken(token, expiryMs, "msk-iam");
})
.Build();
}
/// <summary>Key = account id: all events of one account go to one partition, so they stay ordered.</summary>
public static async Task SendAsync(IProducer<string, string> producer, string topic, string accountId, string json, CancellationToken ct)
{
var result = await producer.ProduceAsync(topic, new Message<string, string> { Key = accountId, Value = json }, ct);
Console.WriteLine($"partition {result.Partition.Value} offset {result.Offset.Value}");
}
}Two tests carry the lesson: a good key spreads, a hot key does not; and a failing record stops the batch and reports its sequence number.
[Fact]
public void Many_accounts_spread_evenly_but_one_hot_key_lands_on_one_shard()
{
var shards = ShardRouter.EvenShards(4);
var spread = ShardRouter.Distribution(Enumerable.Range(0, 4000).Select(i => $"ACC-{i}"), shards);
Assert.All(spread.Values, count => Assert.InRange(count, 800, 1200)); // roughly a quarter each
var hot = ShardRouter.Distribution(Enumerable.Repeat("ACC-ONE-BUSY-MERCHANT", 4000), shards);
Assert.Equal(4000, hot.Values.Max()); // all traffic on one shard
Assert.Equal(3, hot.Values.Count(c => c == 0));
}
[Fact]
public async Task First_failure_is_reported_and_later_records_are_not_processed()
{
// A-1 succeeds, A-2 throws, A-3 must NOT be processed out of order.
var response = await new FraudScoringFunction(scorer).HandleAsync(
new KinesisEvent { Records = [Record("100", "A-1"), Record("101", "A-2"), Record("102", "A-3")] }, context);
Assert.Equal("101", Assert.Single(response.BatchItemFailures).ItemIdentifier);
await scorer.DidNotReceive().ScoreAsync(Arg.Is<CardAuthorisation>(a => a.AuthorisationId == "A-3"), Arg.Any<CancellationToken>());
}Code explained line by line
ShardRouter.HashKeycomputesMD5(UTF-8 bytes of the key)and reads the 16 bytes as an unsigned, big-endian 128-bit integer, the rule Kinesis documents for partition keys.isUnsigned: true, isBigEndian: trueonBigIntegeris what makes the result match.EvenShardssplits[0, 2^128 - 1]into equal inclusive ranges, giving the last shard the remainder.Routefinds the one range that contains the hash.Distributioncounts keys per shard so you can see skew before you build.AuthorisationProducer.PutAsyncloops up to five attempts. Each attempt slices pending records into chunks of 500 (Chunk), and sends them with the account ID as partition key.- After each call it reads
FailedRecordCountand pairschunk[i]withresponse.Records[i]: a non-emptyErrorCodemarks a failure. Only failures go intostillFailing, so successful records are never rewritten by the retry (which would create duplicates). - The backoff delegate is full jitter and injectable. After the last attempt the method returns the leftovers instead of throwing: the caller decides whether to park them.
FraudScoringFunction.HandleAsynciterateskinesisEvent.Recordsin order. On the first exception it logs only the sequence number, adds it as theBatchItemFailure, andbreaks. Lambda retries from that record, so order is preserved. The error log never prints the record body.IFraudScorerhas a comment that matters: implementations must be idempotent. A retried batch and a replay both re-read the same records.MskIamProducer.CreatesetsSecurityProtocol.SaslSslandSaslMechanism.OAuthBearer. The token refresh handler callsAWSMSKAuthTokenGenerator.GenerateAuthTokenAsyncwith the Region, and hands the token and its expiry (in milliseconds) to the client.EnableIdempotenceplusAcks.Allkeep order and avoid duplicates from producer retries, per partition.SendAsyncuses the account ID as the Kafka message key, the equivalent of the Kinesis partition key.
Security considerations
⚠ Learning shortcut — not for production
The lab stream uses the AWS managed key alias/aws/kinesis, and your SSO identity writes and reads it. Production uses a customer managed KMS key per data classification (key use is logged in CloudTrail, and the key policy is a second authorisation layer), a producer role limited to kinesis:PutRecord and kinesis:PutRecords on one stream, and a consumer role limited to the read actions on that stream. Anyone who can read a stream can read every record in it, so streams that carry anything linked to a customer need access reviews.
⚠ Learning shortcut — not for production
The MSK Terraform uses IAM authentication but default VPC placeholders. A real cluster sits in private subnets, its security group allows port 9098 only from the application subnets, access is granted with kafka-cluster:* IAM actions scoped to the cluster, topic and group ARNs, and the cluster is encrypted with a customer managed key. Never enable unauthenticated access on a cluster that holds payment data.
- Retention is a data-protection setting. A 365-day stream keeps a year of card events. Choose retention against the replay requirement, not "because it is cheap", and agree it with data protection. Deleting a customer's data does not delete it from a stream until the records expire: design for that (identifiers, not personal data, in records).
- Minimise what is in a record. Account ID, amount, currency, merchant category and time are enough for fraud scoring. No names, no full card numbers, no CVV. Never put PCI-scoped data on a stream without a scoped-down design reviewed by security.
- Encrypt and isolate. Server-side KMS encryption for Kinesis, interface VPC endpoints for private access, and a bucket and key policy for Firehose targets.
- Consumers are privileged. The Lambda consumer's role reads everything on the stream. Give each consumer its own role, and register enhanced fan-out consumers by name so you can see who reads what.
- Audit. Use CloudTrail for control-plane actions (key use, consumer registration, retention changes). For the data plane, treat consumer logs as the record of who read what.
- Kafka specifics. Use IAM or mTLS, enforce TLS, turn on broker logs to CloudWatch or S3, restrict
kafka-cluster:AlterGroupandDeleteTopic, and keep topic ACLs (if you use mTLS) in code. - Compliance. This lesson does not make any system compliant with GDPR, UK GDPR or PCI DSS. Retention, minimisation and access need legal, data-protection and security review.
Banking use case
Harbourline's card-authorisation stream, designed with this lesson:
| Decision | Choice | Reason |
|---|---|---|
| Service | Kinesis Data Streams, on-demand | AWS-native, no Kafka estate to preserve, spiky salary-day traffic |
| Partition key | accountId | Order per account for fraud rules; millions of keys spread evenly |
| Retention | 7 days | Fraud team's re-scoring window; reviewed with data protection |
| Consumers | Fraud (Lambda, ordered, idempotent), Firehose to S3, dashboard (enhanced fan-out) | Independent positions; the dashboard must not slow the fraud consumer |
| Record content | Identifiers and facts; no PAN or name | Minimisation; PCI scope kept off the stream |
| Alarms | Iterator age, write throttling, failed-batches queue depth | Lag and a poison record are silent otherwise |
| MSK | Not used. A planned migration from an on-premises Kafka cluster would use MSK Express to keep the Kafka protocol | Portability is the only reason to pay for Kafka |
Common mistakes
- A hot partition key (merchant, country, a constant). → Choose the entity whose order matters and that is diverse; test with
ShardRouterand real keys; salt only if order per key can be relaxed. - Default Lambda retry on a stream. One poison record blocks a shard for a day. → Bounded retries, record age, bisect,
ReportBatchItemFailuresand a failure destination. - Reporting every failed record in an ordered stream. Later records run before earlier ones. → Report the first failure and stop.
- Ignoring
FailedRecordCount. Records silently lost during throttling. → Retry only the failed records with jitter. - Reading with many consumers on shared throughput. They split 2 MB/s per shard and throttle each other. → Enhanced fan-out for each independent reader, or fewer readers.
- Treating the stream as a database. Retention ends. → Land the data in S3 or a table.
- Non-idempotent consumers. Retries and replays double-apply. → Idempotency key from the record's business ID.
- Provisioned shards left at a guess. Throttling or waste. → On-demand until the traffic is understood.
- Leaving MSK or an on-demand stream running. Hourly charges. → Tags, budget alarm and the cleanup script.
- Choosing MSK because "streams are Kafka". → Kinesis is simpler unless you need Kafka itself.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
ProvisionedThroughputExceededException on put | Hot shard, or provisioned shards too few (1 MB/s or 1,000 records/s each) | Check WriteProvisionedThroughputExceeded; fix the key; switch to on-demand or add shards; retry with jitter |
| Iterator age keeps growing | Consumer slower than producers, a poison record, or Lambda throttled | Check consumer errors and duration; bound retries and bisect; raise the batch size or concurrency; add shards |
| Same records processed repeatedly | Retried batch, no partial-batch reporting, or a replay | Idempotency; ReportBatchItemFailures; check for an AT_TIMESTAMP mapping |
| Records out of order across keys | Ordering is per shard only | Put the ordering requirement into the partition key |
ReadProvisionedThroughputExceeded | Too many shared consumers on one shard | Enhanced fan-out consumers; fewer readers |
ExpiredIteratorException | An iterator is older than 5 minutes | Get a fresh iterator; checkpoint progress |
| No records from a new stream consumer | StartingPosition is LATEST | Use TRIM_HORIZON or AT_TIMESTAMP |
KMSAccessDeniedException | Consumer or producer role lacks kms:Decrypt or kms:GenerateDataKey | Update the key policy and the role |
| MSK client cannot connect | Wrong bootstrap endpoint, security group, or the IAM port (9098) not open | Use the IAM bootstrap string from the console; allow 9098 from the client subnets |
Kafka SaslAuthenticationFailed | Token expired or wrong Region in AWS.MSK.Auth, or the role lacks kafka-cluster:Connect | Check the Region passed to the token generator and the role's IAM policy |
| Lambda MSK mapping not invoking | Function not in a VPC with a route to the cluster, or missing permissions | Place the function in the VPC and grant the MSK describe permissions |
Cost and cleanup
Cost: credits for Kinesis (well under USD 1 if deleted within the hour); paid and design-only for MSK. An on-demand stream costs about USD 0.04 per stream-hour plus a few cents per GB. This demo moves kilobytes, so it costs roughly four cents an hour ⚠ VERIFY on the Kinesis pricing page. There is no free tier for Kinesis Data Streams, so delete the stream when you finish. Enhanced fan-out, extended retention and the Lambda consumer add a little. MSK has no free tier (⚠ VERIFY), and the cheapest Serverless or Provisioned cluster costs on the order of USD 0.20 to 0.90 per hour; do not create one unless you have a budget alarm and will delete it the same hour.
Free alternatives: run the ShardRouter, producer and consumer unit tests (no AWS), or run Kafka locally with Docker (an Apache Kafka container on localhost:9092) and point Confluent.Kafka at it without the IAM settings.
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
# Event source mappings first (if you deployed the consumer), then the stream.
for uuid in $(aws lambda list-event-source-mappings --function-name hl-m08gap-fraud-scorer \
--query 'EventSourceMappings[].UUID' --output text 2>/dev/null); do
aws lambda delete-event-source-mapping --uuid "$uuid"
done
aws kinesis delete-stream --stream-name hl-m08gap-authorisations --enforce-consumer-deletion
url=$(aws sqs get-queue-url --queue-name hl-m08gap-failed-batches --query QueueUrl --output text 2>/dev/null) \
&& aws sqs delete-queue --queue-url "$url"
# MSK, if you created a cluster (this is the important one for your bill).
for arn in $(aws kafka list-clusters-v2 --query "ClusterInfoList[?contains(ClusterName,'hl-m08gap')].ClusterArn" --output text); do
aws kafka delete-cluster --cluster-arn "$arn"
done
# If you deployed with IaC: cdk destroy HlM08Gap-Stream | terraform destroy
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m08-messaging-services \
--query 'ResourceTagMappingList[].ResourceARN'
aws kinesis list-streams --query 'StreamNames'
aws kafka list-clusters-v2 --query 'ClusterInfoList[].ClusterName'Exercises
- 🟢 Guided: Run the demo: create the stream, write ten numbered events for each of three accounts, read every shard, and check that within each account the
seqvalues are in order. RunShardRouterwith the real shard ranges fromlist-shardsand confirm its prediction matches where each account's records landed. Delete the stream. - 🟡 Challenge: Take 5,000 real-looking account IDs plus one merchant ID that accounts for 40 percent of traffic. Use
ShardRouterwith 8 shards to compare two partition keys,accountIdandmerchantId, and report the busiest shard's share of traffic for each. Then design a salted key for the merchant and state what ordering you give up. - 🔴 Live problem: Ticket at 09:40: "Fraud alerts are 35 minutes late. Lambda shows no errors and no throttling.
GetRecords.IteratorAgeMillisecondsis climbing on one shard and flat on the other nine. Traffic is normal." Work out the likely causes, how to confirm each, a safe immediate mitigation, and what you would change so it cannot recur.
Assignment
Add streaming-design.md to your portfolio repository for one Harbourline stream. Include: the choice of Kinesis or MSK with the reasons, the record schema (identifiers only), the partition key with a ShardRouter distribution table from a realistic sample, capacity mode and the monthly cost estimate at 2,000 events per second of 1 KB, retention and who approved it, each consumer with its read method and settings, the poison-record policy (retries, record age, failure destination), idempotency keys, alarms (iterator age, throttling, failure queue), and the IAM and KMS design.
Solutions
Within one account's records, seq is strictly increasing because all of an account's records share a partition key and therefore a shard, and Kinesis orders records within a shard. Different accounts on the same shard interleave but each account's subsequence stays in order. Across shards there is no ordering at all, so reading shard by shard shows each account's ten events contiguous in seq order.
The ShardRouter prediction matches because the algorithm is deterministic: MD5 of the key, as an unsigned big-endian integer, inside a shard's [StartingHashKey, EndingHashKey]. If they differ, you used the even-split helper on a stream whose shards were split or merged: read the real ranges from list-shards.
With the account key, 5,000 accounts spread close to 1/8 each, so the busiest shard carries roughly 12.5 to 14 percent of traffic. With merchantId as the key, the dominant merchant's 40 percent lands on one shard, so that shard carries at least 40 percent, about three times its fair share, and it is the one that throttles first.
The salted key is merchantId#n where n = hash(accountId) % 8 (derive the salt from a stable attribute, not random, so retries land on the same shard). The one merchant's traffic now spreads over up to eight shards. What you give up: a single global order for that merchant. You keep order per (merchant, salt), which means per group of accounts. If the consumer only needs per-account order, use the account as the key in the first place and the problem disappears.
Read the signals. One shard behind and nine fine means the problem is local to that shard, not the consumer's general capacity.
- A poison record on that shard. The Lambda mapping retries the same batch (default retry is unlimited), so nothing behind it is read. Check the function's logs for the same sequence number failing repeatedly, and the mapping's
MaximumRetryAttempts(-1?). - A hot key on that shard. If one account or merchant sends a burst, the shard has more records per second than a single batch stream can process. Check
IncomingRecordsandIncomingBytesby shard (enhanced monitoring) and the keys in the shard. - A slow downstream for that shard's keys (the model endpoint is slow only for certain customers), so each batch takes longer.
- A recent reshard leaving a parent shard unfinished while its children wait.
Confirm: read that shard with the CLI from the consumer's position and find the failing record or the dominant key. Mitigate now: if it is a poison record, set bounded retries and record age (or temporarily skip past it by moving the starting position past its sequence number, after capturing it for investigation), and ensure the failure destination is set. If it is a hot key, add capacity or reduce the batch size and raise parallelism for that shard.
Prevent: bounded retries, bisect and a failure destination on every stream mapping, an alarm on iterator age per stream, a hot-key check using ShardRouter and production samples before choosing a key, and a runbook for "one shard behind".
Knowledge check
Answer: The partition key. All records with the same key go to the same shard and are read in the order written. There is no ordering across shards.
Answer: Lambda retries the batch until the record expires (the default is unlimited), and because a shard is read in order, every record behind it in that shard is delayed. Set MaximumRetryAttempts, MaximumRecordAgeInSeconds, bisect on error and a failure destination.
Answer: Register them as enhanced fan-out consumers, each with a dedicated 2 MB/s per shard, or reduce the number of readers. Shared consumers split 2 MB/s per shard.
Answer: Find the three entries with an ErrorCode (in the same order as the request) and resend only those, with backoff and jitter. Do not resend the whole batch.
Answer: The Kafka protocol and ecosystem (existing Kafka code, Kafka Connect, Streams), and features Kinesis lacks such as compacted topics, Kafka transactions and very long retention; also portability. Otherwise Kinesis is simpler to run.
Answer: The default is 24 hours and the maximum is 365 days (8,760 hours). Longer retention costs more.
Interview questions
- "Queue or stream?" — A queue distributes jobs to one consumer each and deletes them; a stream keeps ordered records that every consumer reads at its own position and can replay. If you need ordering, replay or several independent readers at high volume, use a stream; otherwise a queue or an event bus is simpler.
- "How do you pick a partition key?" — The entity whose order matters (the account), which is also diverse (millions of values) so load spreads. Test it with a sample of real keys against the hash-range rule, avoid keys that dominate traffic (a large merchant), and salt only if order per key can be relaxed.
- "What is a hot shard and how do you find it?" — One shard receiving more than 1 MB/s or 1,000 records/s because of a dominant key. You see
WriteProvisionedThroughputExceededand uneven per-shard metrics, and you can reproduce it offline by hashing sampled keys. - "How do you stop a poison record blocking a stream?" — Bound retries and record age, bisect the batch, report the failing sequence number, send a description of the discarded batch to a queue, alarm on iterator age and recover from the stream itself.
- "When would you use MSK?" — When Kafka itself is a requirement: an existing Kafka estate or tooling, Kafka Connect, compaction or transactions, or portability. For everything else on AWS, Kinesis is the lower-effort default.
Certification practice
A bank ingests card events and must process the events of each account in order while many accounts are processed in parallel. Which design meets this?
A. An Amazon SQS standard queue. B. A Kinesis data stream with the account ID as the partition key. C. An Amazon SNS topic with one subscriber per account. D. A Kinesis data stream with a random partition key.
Answer: B. Records with the same partition key share a shard and keep their order, and different accounts spread across shards. A gives no ordering. C does not scale to accounts. D scatters an account's events across shards and destroys order.
A Lambda function consumes a Kinesis stream. One record repeatedly causes an exception. Records behind it in the same shard are delayed for hours. What should the developer configure to limit the impact?
A. Increase the Lambda memory. B. Set a maximum retry attempts value and maximum record age, enable bisect on error, and add an on-failure destination. C. Switch the stream to provisioned mode. D. Add more shards.
Answer: B. By default Lambda retries until the record expires, blocking the shard. Bounded retries, record age, bisecting to isolate the record and a failure destination let processing move on. A, C and D do not change the retry behaviour.
Several applications each need to read all records of the same Kinesis data stream with low latency, and they are throttling each other. Which feature solves this?
A. Increase the stream's retention period. B. Enhanced fan-out with a registered consumer per application. C. Convert the stream to a queue. D. Enable server-side encryption.
Answer: B. Enhanced fan-out gives each registered consumer a dedicated 2 MB/s of read throughput per shard. A changes retention, C is not an option and D changes security only.
A company has applications that use the Apache Kafka protocol and Kafka Connect and wants a managed service with minimal code changes. Which service should the company use?
A. Amazon Kinesis Data Streams. B. Amazon SQS FIFO. C. Amazon MSK. D. Amazon EventBridge.
Answer: C. MSK is managed Apache Kafka, so Kafka clients and connectors work unchanged. A, B and D use different APIs and would need code changes.
Completion checklist
- I can explain a stream versus a queue in two sentences
- I can predict a record's shard with
ShardRouterand spot a hot key - I wrote a producer that retries only failed
PutRecordsentries - I can list the Lambda event-source settings that stop a poison record blocking a shard
- I ran the Kinesis demo, saw ordering per account and replayed from a timestamp
- I know when MSK is justified, and what it costs to leave running
- I deleted the stream (and any cluster) and the checks returned nothing
Key takeaways
- A stream keeps ordered, replayable records for many independent readers. Order exists per shard (Kinesis) or partition (Kafka) and is decided by your partition key.
- A shard absorbs about 1 MB/s or 1,000 records/s of writes: one dominant key creates a hot shard. Test key designs offline with the MD5 hash rule.
PutRecordsis partial-success. Resend only the failed records, with jitter.- Defaults are dangerous for consumers: unlimited retry lets one poison record block a shard. Bound retries and record age, bisect, report the first failed sequence number and stop.
- Kinesis is the low-effort AWS-native default. MSK is real Kafka and costs more to run and to leave running. Both are at-least-once: consumers must be idempotent.
Further reading
- Amazon Kinesis Data Streams quotas and limits
- Use the Kinesis Client Library
- Amazon Kinesis Data Streams pricing
- Amazon MSK quotas
- Amazon MSK pricing
- Using Lambda with Amazon Kinesis
- Amazon EventBridge Pipes sources
- Kinesis Data Streams launches On-demand Advantage mode
- Kinesis Data Streams scale down with warm throughput
- Next: Amazon MQ for legacy integration
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
The outbox pattern with EF Core
Solve the dual-write problem: write business state and its events in one EF Core transaction, relay them to EventBridge with a leased, batch-aware BackgroundService, and handle relay crashes, partial PutEvents failures, poison rows and cleanup — with tests that run on SQLite.
Amazon MQ: bridging legacy JMS and AMQP systems
Know when a managed ActiveMQ or RabbitMQ broker is the right tool: keep a JMS/AMQP system running on Amazon MQ, bridge it to SQS, SNS and EventBridge with a Lambda event source mapping and a translator, handle poison messages, and plan the migration, with .NET 10 clients, CDK and Terraform.