Skip to Content
Cloud Engineering Academy
8. Messaging, Events & Microservices

Kinesis Data Streams and Amazon MSK for event streaming

Use an ordered, replayable log instead of a queue: Kinesis Data Streams shards, partition keys, capacity modes, consumers and poison records, then Amazon MSK (Kafka) with IAM authentication and a .NET producer, and a clear rule for choosing between streams, queues and buses.

advanced⏱ 3 h 🟡 Low costDEA-C01SAA-C03DVA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.Kinesis 4.1.1) · Amazon.Lambda.KinesisEvents 4.0.0 · Confluent.Kafka 2.16.0 · AWS.MSK.Auth 1.1.3 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain how a stream (an ordered, replayable log) differs from a queue, and say which banking problems each one fits.
  • Describe Kinesis shards, partition keys, sequence numbers, retention and iterators, and design a partition key that keeps one account in order without creating a hot shard.
  • Predict which shard a key lands on, using the MD5 hash rule, and test a key design with a sample of real keys.
  • Choose between on-demand and provisioned capacity, and between shared and enhanced fan-out consumers.
  • Write a producer that handles partial PutRecords failure and a Lambda consumer that preserves order, reports the failed record and survives a poison record.
  • Describe Amazon MSK (Provisioned Standard, Provisioned Express, Serverless), connect a .NET producer with IAM authentication, and say when MSK is the right answer.
  • Run the module's Kinesis ordering and replay demo and delete it within an hour.

Prerequisites

Why this matters in banking

Every card authorisation at Harbourline produces a small event: account, amount, merchant category, time. At peak that is thousands per second. Three teams want it, each at their own pace:

  • Fraud scoring needs each account's events in order within a second or two of the authorisation.
  • The data lake loads everything every few minutes.
  • A live operations dashboard needs the last few minutes only.

Then the fraud team ships a better model and wants to re-score the last seven days of events with it. A queue cannot do this: once a consumer deletes a message, it is gone, and a second consumer would have to compete for the same messages. A stream keeps the events for a retention period, lets every consumer read at its own position, and lets you rewind that position to a time in the past.

Streams also come with the hardest operational problems in this module: a "hot" key that overloads one shard, a poison record that blocks a shard for a day, and consumers that fall behind without any error. This lesson is about recognising those before production does.

The idea in plain language

A queue is a to-do list: take an item, finish it, cross it off. A stream is a ledger: events are appended in order and never crossed off. Readers keep a bookmark (a position) and read forward. Two readers have two bookmarks and do not interfere. Deleting happens only when the retention period expires.

Analogy: The branch's CCTV recorder. Cameras write continuously to a loop of tape that keeps the last 24 hours (or longer, if you pay for more). Security, the fraud team and the police each watch from their own position and can rewind. Nobody removes footage by watching it. A busy till with ten cameras on it can overload one recorder channel while others are idle: that is a hot shard.

Queue or stream?

Queue (SQS)Stream (Kinesis, Kafka)
ConsumptionA message is processed by one consumer, then deletedEvery consumer reads every record at its own position
ReplayNoYes, within retention (rewind to a timestamp or offset)
OrderingBest effort (standard) or per group (FIFO)Per shard (Kinesis) or partition (Kafka), strictly
Scaling unitAutomaticShards or partitions: you design the key, and the key decides the load
Consumer scalingAdd workers freelyAt most one active reader per shard per consumer application
Failure of one recordThat message retries on its ownA failing record blocks the rest of its shard unless you skip or park it
Typical latencyMilliseconds to secondsTens of milliseconds to a second
Fit"Do this job""Here is what happened, in order; many things care"

If you only need fan-out and filtering, SNS or EventBridge are simpler. If you need work distribution, SQS is simpler. Choose a stream when ordering, replay or many independent readers at high volume are real requirements.

What problem a stream solves

A durable, ordered, partitioned log lets many consumers process the same high-volume events independently, rewind, and catch up after downtime. In AWS you have two main choices:

  • Amazon Kinesis Data Streams: a fully managed AWS-native stream. No brokers. You choose shards (or on-demand) and pay for capacity or throughput.
  • Amazon MSK (Managed Streaming for Apache Kafka): managed Apache Kafka. You get the Kafka protocol, ecosystem and semantics (consumer groups, transactions, compaction), and you manage more of it.

Limits and defaults, Kinesis Data Streams (checked 2026-11-18)

SettingValue
Write per shard (provisioned)Up to 1 MB/s or 1,000 records/s
Read per shard (shared throughput)Up to 2 MB/s, 5 GetRecords calls per second
Record sizeUp to 10 MiB before base64 (designed for intermittent large records; most are tiny)
PutRecordsUp to 500 records and 10 MiB per request
Retention24 hours by default, extendable to 365 days
Registered (enhanced fan-out) consumers20 per stream (50 with On-demand Advantage)
On-demand stream throughputNew streams start at 4 MB/s write and 8 MB/s read, and scale to 10 GB/s write in us-east-1 (200 MB/s in some other Regions)
On-demand streams per account50 by default (a soft limit)
Provisioned shards per account20,000 in us-east-1, us-west-2 and eu-west-1 by default
Capacity-mode switchesTwice per 24 hours per stream
Shard iterator lifetime5 minutes
EncryptionServer-side with AWS KMS

Limits and defaults, Amazon MSK (checked 2026-11-18)

SettingValue
Brokers per cluster (Standard or Express)30 (ZooKeeper-based) or 60 (KRaft)
Express broker throughput per partitionMax 15 MB/s; per broker, sustained ingress from 15.6 MB/s (express.m7g.large) to 667 MB/s (express.m8g.24xlarge)
Express partitions per brokerRecommended 1,000 on express.m7g.large, hard maximum 1,500
Serverless, per cluster200 MB/s in, 400 MB/s out, 2,400 partitions (120 for compacted topics), 3,000 connections, 8 MiB maximum message
Serverless, per partition5 MB/s in, 10 MB/s out
IAM-authenticated connections per broker3,000, with a creation rate of 100 per second (4 on t3)

Honest limitations

  • You own the key design. The partition key decides both ordering and load. A bad key is a production incident that no amount of money fixes.
  • One failing record can stall a shard. The default Lambda retry for Kinesis is unlimited until the record expires.
  • At-least-once. A retried batch re-reads records. Consumers must be idempotent.
  • Provisioned capacity is a floor you pay for, busy or not. On-demand also bills per stream-hour.
  • Kinesis needs the Java-centric KCL for the richest consumer features. The Kinesis Client Library is a Java library. Other languages, including .NET, use it through a Java process called the MultiLangDaemon. For .NET the usual consumer is Lambda, a Pipe, or Firehose.
  • MSK is real Kafka. It brings Kafka's operational model: partitions, replication factor, consumer-group rebalances, topic configuration, version upgrades.
  • Per-record costs. Each Kinesis PutRecords payload is billed in 25 KB units in provisioned mode; many tiny records can cost more than their bytes suggest.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Event Hubs (Standard/Premium)Kinesis Data StreamsEvent Hubs partitions ≈ Kinesis shards. Event Hubs throughput units ≈ provisioned shards; Event Hubs auto-inflate ≈ on-demand mode. Event Hubs Capture to storage ≈ Kinesis Data Firehose to S3
Event Hubs with the Kafka endpointAmazon MSKMSK is actual Kafka brokers; Event Hubs speaks the Kafka protocol on top of its own service
Event Hubs consumer groupsKinesis enhanced fan-out consumers, or KCL application namesA Kinesis consumer application tracks its position in a DynamoDB table (KCL) or in the Lambda event source mapping
EventProcessorClient (checkpoints in Blob Storage)KCL (checkpoints in DynamoDB)KCL is Java; .NET goes through MultiLangDaemon or uses Lambda
Event Hubs partition keyKinesis partition keyKinesis hashes the key with MD5 into a 128-bit space; shards own ranges
Azure Stream AnalyticsManaged Service for Apache Flink, or LambdaNot covered here
Event Hubs CaptureFirehose delivery to S3, or Kinesis streaming tables to S3 Tables (August 2026 ⚠ VERIFY)
Azure Schema RegistryAWS Glue Schema Registry (MSK and Kinesis)Out of scope for this lesson

Kinesis Data Streams in detail

The vocabulary

TermMeaning
StreamA named log with a retention period
ShardA unit of capacity and ordering: a slice of the 128-bit hash space and a sequence of records
RecordA blob (data), a partition key and a sequence number assigned on write
Partition keyA string you choose. Its MD5 hash (read as a 128-bit unsigned integer) picks the shard whose hash range contains it
Sequence numberAn increasing identifier within a shard. Order is guaranteed within a shard only
Shard iteratorA cursor for reading: TRIM_HORIZON (oldest), LATEST, AT_TIMESTAMP, AT_SEQUENCE_NUMBER, AFTER_SEQUENCE_NUMBER. Expires after 5 minutes
RetentionHow long records stay readable: 24 hours to 365 days
ConsumerAn application reading the stream. Shared consumers split the shard's 2 MB/s; enhanced fan-out consumers each get their own 2 MB/s per shard, pushed over HTTP/2

Capacity modes

On-demandProvisioned
You decideNothing: no shard countThe shard count (UpdateShardCount, split, merge)
ScalingAutomatic, up to 10 GB/s in us-east-1Yours
Billing (us-east-1, from the pricing page's worked examples ⚠ VERIFY)About USD 0.040 per stream-hour, USD 0.08 per GB written and USD 0.04 per GB readAbout USD 0.015 per shard-hour plus about USD 0.014 per million 25 KB PUT payload units
Extended retentionExtra per-GB chargesAbout USD 0.020 per shard-hour for retention up to 7 days; long-term retention bills storage per GB-month
Enhanced fan-outExtra per-GB retrieval chargeAbout USD 0.015 per consumer-shard-hour plus about USD 0.013 per GB
Best whenSpiky or unknown traffic; you do not want to sizeSteady, predictable traffic where shard-hours are cheaper

On-demand Advantage (announced November 2025) is a pricing mode for accounts with large sustained volumes: lower per-GB prices, no per-stream-hour charge, enhanced fan-out consumers at no extra charge, and a minimum charge of 25 MB/s of ingest and 25 MB/s of retrieval at the account level ⚠ VERIFY. It is the wrong choice for a learning account.

In July 2026 AWS added warm throughput scale-down, and in August 2026 streaming tables that deliver a stream to Apache Iceberg tables in S3 Tables ⚠ VERIFY both in the current documentation if you need them.

Partition keys, ordering and hot shards

Two rules decide everything:

  1. Records with the same partition key go to the same shard, and are read in the order written.
  2. A shard can absorb only 1 MB/s or 1,000 records/s of writes. A key that carries more than that overloads its shard no matter how many shards exist.

So you want a key that is stable for the entity whose order you care about (the account) and diverse across the population (millions of accounts). For card authorisations the account ID is right. A merchant ID is wrong: one very large merchant sends a large share of traffic on one key, saturates one shard and delays every account that happens to share it, while the other shards idle. That is the classic hot shard.

You can test a key design before you build anything, because the mapping is deterministic:

var shards = ShardRouter.EvenShards(4);
var spread = ShardRouter.Distribution(sampleAccountIds, shards);        // about a quarter each
var hot = ShardRouter.Distribution(Enumerable.Repeat("MERCHANT-BIG", 4000), shards); // all on one shard

The ShardRouter class in this lesson implements the documented rule (MD5, unsigned big-endian, inclusive hash ranges). A stream freshly created with N shards splits the space evenly, as the helper's EvenShards does. After splits and merges the ranges differ: read the real ones from ListShards.

If one key really is hot: add a bounded salt (MERCHANT-BIG#0 to #7) so that key spreads over up to eight shards, and accept that order is now only per salt. Whether that is acceptable is a business question about what order the consumer needs.

Resharding and order. When a shard splits or merges, child shards are read only after their parent is fully read and checkpointed, which keeps per-key order. KCL and the Lambda event source mapping both handle this for you. A hand-written reader must handle it.

Producing: partial success is normal

PutRecords takes up to 500 records and is partial-success: the HTTP call succeeds, and individual records may be rejected (typically ProvisionedThroughputExceededException for a busy shard). The response has FailedRecordCount and one result per record in the same order. The producer must resend only the failed records, with backoff and jitter, and return what it still cannot send.

Choose the partition key per record: AccountId. The producer in this lesson does that and nothing else clever. For very high volumes, the Kinesis Producer Library (Java) aggregates records to save cost; from .NET, batch with PutRecords and keep records small.

Consuming and the poison-record problem

ConsumerNotes
Lambda event source mappingThe default choice for .NET. Reads each shard, batches, retries, scales with shards. Settings below
Enhanced fan-out via LambdaRegister a consumer ARN for a dedicated 2 MB/s per shard; useful when several readers share a stream
FirehoseDelivers to S3, Redshift, OpenSearch or HTTP endpoints with buffering; no code
EventBridge PipeKinesis source, filter, enrichment, target: point-to-point with no code
KCL (Java, MultiLangDaemon for other languages)Leases shards across workers using a DynamoDB table, checkpoints, handles resharding. Powerful; more to run

The Lambda event source mapping, setting by setting

SettingWhat it doesRecommended for ordered banking events
StartingPositionTRIM_HORIZON, LATEST or AT_TIMESTAMPTRIM_HORIZON for a new consumer that must see history; AT_TIMESTAMP to replay
BatchSizeRecords per invocation (default 100, up to 10,000)100
MaximumBatchingWindowInSecondsWait up to this long to fill a batch1
ParallelizationFactorConcurrent batches per shard (1 to 10)1 keeps per-key order. A factor above 1 still keeps order per partition key but is subtle; test it before use
ReportBatchItemFailuresThe function returns the sequence number to resume fromOn
BisectBatchOnFunctionErrorSplit a failing batch in two to isolate the bad recordOn
MaximumRetryAttemptsDefault -1: retry until the record expiresSet it (for example 5)
MaximumRecordAgeInSecondsSkip records older than thisSet it (for example 6 hours)
DestinationConfig.OnFailureWhere a description of a discarded batch is sentAn SQS queue (it records shard and sequence range, not the data)

The trap: with the defaults, one record that always throws is retried forever, and because a shard is read in order, every record behind it in that shard waits, for up to the retention period. The iterator age grows and nothing else looks wrong. The settings above (bounded retries, record age, bisect, a failure destination) turn that into "skip it and tell someone".

Ordered semantics with partial batches. With SQS you report every failed message. With a stream you report the first failed record's sequence number and stop processing the batch; Lambda retries from that record. If you carried on past the failure, later records of the same key would be applied before the failed one. The FraudScoringFunction does exactly this.

Watch the lag. The metric that matters is GetRecords.IteratorAgeMilliseconds: how old the newest record read is. Alarm when it exceeds your latency budget. Also watch WriteProvisionedThroughputExceeded (hot shard or too few shards) and, for provisioned streams, IncomingBytes against capacity.

Replay. Create a second mapping with AT_TIMESTAMP (or reset a KCL application's lease table) to re-read last week. As with EventBridge replay, a replay is a deliberate duplicate: consumers must be idempotent, and side effects need the idempotency key.

Amazon MSK

Amazon MSK runs Apache Kafka for you: brokers, storage, patching and replication across Availability Zones. Your applications use standard Kafka clients, so existing Kafka code, tools and connectors work.

Cluster types

TypeWhat you manageBilling (from the pricing page's worked examples, us-east-1 ⚠ VERIFY)Notes
Provisioned: Standard brokersBroker instance type and count, storage per broker, configurationHourly per broker (for example about USD 0.204 for kafka.m7g.large), plus storage about USD 0.10 per GB-monthThe classic MSK; tiered storage available
Provisioned: Express brokersBroker size and count; storage is elastic and managedHourly per broker (for example about USD 0.408 for express.m7g.large), plus about USD 0.01 per GB ingested and storage per GB-monthHigher throughput per broker, faster scaling and recovery, fewer settings to tune
ServerlessNothing: capacity scales automaticallyAbout USD 0.75 per cluster-hour, USD 0.0015 per partition-hour, USD 0.10 per GB in, USD 0.05 per GB out, plus storageHard quotas (see above); IAM authentication only ⚠ VERIFY

All three are expensive to leave running: the cluster-hour or broker-hours accrue 24 hours a day. The pricing page's own five-topic Serverless example comes to about USD 1,300 for a month. For learning, design only, or run Kafka in Docker.

Authentication and networking

MSK clusters live in your VPC; clients must reach them over private networking. Authentication options include IAM access control (port 9098, SigV4-based, works with IAM roles for ECS, EC2 and Lambda), SASL/SCRAM with credentials in Secrets Manager, mutual TLS and, if you allow it, unauthenticated access ⚠ VERIFY which options each cluster type supports. In a bank you choose IAM or mTLS, never unauthenticated, and encrypt in transit and at rest (a customer managed KMS key where policy demands).

Kafka in one table

Kafka termKinesis equivalent
TopicStream
PartitionShard
Message keyPartition key
OffsetSequence number
Consumer groupA consumer application (KCL)
Replication factorManaged for you in Kinesis (3 AZs)
Compacted topicNo equivalent
Transactions and idempotent producerNo equivalent (idempotency is yours)
Retention by size or time, unlimited with tiered storage24 hours to 365 days

From .NET

Use Confluent.Kafka. The IAM mechanism is OAUTHBEARER with a token that the AWS.MSK.Auth package builds from your AWS credentials. The producer below also turns on EnableIdempotence and Acks.All, which gives ordered, duplicate-free writes per partition even when the producer retries.

For consuming from Lambda, an MSK event source mapping works like the Kinesis one (batches per partition, retry on error). Pipes also read MSK and self-managed Kafka.

Kinesis, MSK, SQS, SNS, EventBridge: choosing

If you needChoose
Hand work to workers, retry and dead-letter each jobSQS
Push one message to a few known subscribers quicklySNS (with a queue behind each)
Route business events between teams by content, with replay of modest volumesEventBridge
Ordered, replayable, high-volume events read by several independent consumers, AWS-native, least operationsKinesis Data Streams
The Kafka protocol, ecosystem (Kafka Connect, Streams, existing code), compaction, transactions, very long retention, or portabilityAmazon MSK
A legacy JMS/AMQP system must keep workingAmazon MQ (next lesson)

The honest default for a .NET team on AWS: Kinesis unless a concrete requirement (an existing Kafka estate, Kafka Connect, compaction, transactions) points at MSK.

Architecture

Security-boundary view

Step-by-step implementation (Kinesis ordering and replay demo)

This is Lab 8.6 of the module. It deploys one on-demand stream, shows ordering per key, replays from a timestamp, and deletes everything. Keep it under an hour.

Create the stream and note the time

Create hl-m08gap-authorisations in on-demand mode with 24-hour retention. The stream costs about USD 0.04 per hour while it exists.

Write ordered events for three accounts

Send ten numbered events per account with the .NET producer or the CLI. Use the account ID as the partition key, and put a seq field in each payload.

Read one shard back and check the order

For each shard, get an iterator at TRIM_HORIZON and read the records. Within each shard, the events of an account appear in seq order, even though accounts interleave. Events of different accounts on the same shard are ordered relative to each other too, but nothing orders events on different shards.

Predict the shards

Run ShardRouter for the three account IDs against the real shard ranges from list-shards. Confirm the prediction matches where the records landed.

Replay

Note the time before step 3, write ten more events, then read again with AT_TIMESTAMP set to that time. You will see only the later events, in order: that is a rewound consumer.

Break it and watch the lag

Deploy the Lambda consumer (optional), make the handler throw on one record, and watch GetRecords.IteratorAgeMilliseconds rise while the retry settings run. Then see the bounded retry skip it and the failure destination receive a description.

Delete everything

Run the cleanup below and the tag check. A forgotten on-demand stream bills every hour.

AWS Console instructions

Console → Kinesis → Data streams → Create data stream. Name it, choose On-demand under Capacity mode (or Provisioned and a shard count), and create. On the stream, Configuration shows the retention period (edit it up to 365 days), Server-side encryption, and Enhanced fan-out. Monitoring shows IncomingRecords, IteratorAge and throttling. Data viewer reads records from a chosen shard and position without code, which is the quickest way to see ordering.

Console → MSK → Clusters → Create cluster. Choose Quick create or Custom create, then the cluster type (Provisioned with Standard or Express brokers, or Serverless). Check the Estimated cost before you press Create. Choose IAM access control, three private subnets and a security group that allows port 9098 only from your application subnets. Client information → View client information shows the bootstrap servers. Delete the cluster when finished.

AWS CLI

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
STREAM=hl-m08gap-authorisations

# On-demand stream with the default 24-hour retention and KMS encryption.
aws kinesis create-stream --stream-name "$STREAM" --stream-mode-details StreamMode=ON_DEMAND
aws kinesis wait stream-exists --stream-name "$STREAM"
aws kinesis start-stream-encryption --stream-name "$STREAM" --encryption-type KMS --key-id alias/aws/kinesis
aws kinesis add-tags-to-stream --stream-name "$STREAM" --tags Project=cea-lab,Lab=m08-messaging-services

# Note the time, then write ordered events. The partition key is the account id.
START=$(date -u +%Y-%m-%dT%H:%M:%SZ)
for acc in ACC-001 ACC-002 ACC-003; do
  for seq in 1 2 3 4 5; do
    aws kinesis put-record --stream-name "$STREAM" --partition-key "$acc" \
      --cli-binary-format raw-in-base64-out \
      --data "{\"accountId\":\"$acc\",\"seq\":$seq,\"amount\":$((seq * 10))}" \
      --query '{shard:ShardId,seq:SequenceNumber}' --output text
  done
done

# Where are the shards and which hash ranges do they own? (compare with ShardRouter)
aws kinesis list-shards --stream-name "$STREAM" \
  --query 'Shards[].{id:ShardId,from:HashKeyRange.StartingHashKey,to:HashKeyRange.EndingHashKey}'

# Read one shard from the start. Data comes back base64-encoded.
SHARD=$(aws kinesis list-shards --stream-name "$STREAM" --query 'Shards[0].ShardId' --output text)
ITER=$(aws kinesis get-shard-iterator --stream-name "$STREAM" --shard-id "$SHARD" \
  --shard-iterator-type TRIM_HORIZON --query ShardIterator --output text)
aws kinesis get-records --shard-iterator "$ITER" --limit 100 \
  --query 'Records[].[PartitionKey,SequenceNumber,Data]' --output text \
  | while read -r key seq data; do echo "$key $(echo "$data" | base64 -d)"; done

# Replay: read again from a point in time.
ITER=$(aws kinesis get-shard-iterator --stream-name "$STREAM" --shard-id "$SHARD" \
  --shard-iterator-type AT_TIMESTAMP --timestamp "$START" --query ShardIterator --output text)

# Retention: raise to 7 days (costs extra) and back.
aws kinesis increase-stream-retention-period --stream-name "$STREAM" --retention-period-hours 168
aws kinesis decrease-stream-retention-period --stream-name "$STREAM" --retention-period-hours 24

# MSK (design only unless you accept the cost): list what exists and the Kafka versions available.
aws kafka list-clusters-v2 --query 'ClusterInfoList[].{name:ClusterName,type:ClusterType,state:State}'
aws kafka list-kafka-versions --query 'KafkaVersions[].Version'

Infrastructure as Code

The stack creates an on-demand, KMS-encrypted stream with 24-hour retention, and (when you give it a published function directory) a Lambda consumer with the safe event-source settings: ordered, one batch per shard, bounded retries, record age limit, bisect, partial-batch reporting and a failure destination.

StreamStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.Kinesis;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.Lambda.EventSources;
using Amazon.CDK.AWS.SQS;
using Constructs;

namespace Harbourline.Messaging.Infra;

public sealed class StreamStackProps : StackProps
{
    /// <summary>Output of dotnet publish for the consumer function. Null = create the stream only.</summary>
    public string? FunctionsPublishDirectory { get; init; }
}

/// <summary>
/// An on-demand Kinesis stream and an ordered, retry-safe Lambda consumer. On-demand mode means no shard maths and
/// no UpdateShardCount, but you pay per stream-hour and per GB: delete it after the lab.
/// </summary>
public sealed class StreamStack : Stack
{
    public StreamStack(Construct scope, string id, StreamStackProps props) : base(scope, id, props)
    {
        var stream = new Amazon.CDK.AWS.Kinesis.Stream(this, "Authorisations", new StreamProps
        {
            StreamName = "hl-m08gap-authorisations",
            StreamMode = StreamMode.ON_DEMAND,
            RetentionPeriod = Duration.Hours(24),   // the default; raise it only for a real replay requirement
            Encryption = StreamEncryption.MANAGED,  // KMS with the AWS managed key aws/kinesis
        });

        // Failed batches that exhaust their retries are described here (shard, sequence range), not copied: read the stream to recover.
        var failedBatches = new Queue(this, "FailedBatches", new QueueProps
        {
            QueueName = "hl-m08gap-failed-batches",
            RetentionPeriod = Duration.Days(14),
            Encryption = QueueEncryption.SQS_MANAGED,
            EnforceSSL = true,
        });

        _ = new CfnOutput(this, "StreamName", new CfnOutputProps { Value = stream.StreamName });

        if (props.FunctionsPublishDirectory is not { } publishDir) return;

        var scorer = new Function(this, "FraudScorer", new FunctionProps
        {
            FunctionName = "hl-m08gap-fraud-scorer",
            Runtime = new Runtime("dotnet10", RuntimeFamily.DOTNET_CORE),
            Handler = "Harbourline.Messaging.Samples::Harbourline.Messaging.Samples.Kinesis.FraudScoringFunction::HandleAsync",
            Code = Code.FromAsset(publishDir),
            Architecture = Architecture.ARM_64,
            MemorySize = 512,
            Timeout = Duration.Seconds(30),
        });

        scorer.AddEventSource(new KinesisEventSource(stream, new KinesisEventSourceProps
        {
            StartingPosition = StartingPosition.TRIM_HORIZON,
            BatchSize = 100,
            MaxBatchingWindow = Duration.Seconds(1),
            ParallelizationFactor = 1,         // 1 concurrent batch per shard keeps per-key order; raise to 10 only if order across keys is irrelevant
            ReportBatchItemFailures = true,    // the handler returns the sequence number to resume from
            BisectBatchOnError = true,         // split a failing batch to isolate the bad record
            RetryAttempts = 5,                 // default is -1 (retry until the record expires: one poison record can block a shard for a day)
            MaxRecordAge = Duration.Hours(6),
            OnFailure = new SqsDlq(failedBatches),
        }));
    }
}

Complete code

All four classes were compiled and unit-tested with xUnit v3 (see labs/m08-messaging-services/). Packages:

dotnet add package AWSSDK.Kinesis --version 4.1.1
dotnet add package Amazon.Lambda.Core --version 3.4.0
dotnet add package Amazon.Lambda.KinesisEvents --version 4.0.0
dotnet add package Confluent.Kafka --version 2.16.0
dotnet add package AWS.MSK.Auth --version 1.1.3
ShardRouter.cs
using System.Numerics;
using System.Security.Cryptography;
using System.Text;

namespace Harbourline.Messaging.Samples.Kinesis;

/// <summary>A shard owns an inclusive range of the 128-bit hash space.</summary>
public sealed record ShardRange(string ShardId, BigInteger StartingHashKey, BigInteger EndingHashKey);

/// <summary>
/// Reproduces how Kinesis Data Streams maps a partition key to a shard: MD5(partitionKey) read as an unsigned
/// 128-bit big-endian integer, then the shard whose [StartingHashKey, EndingHashKey] range contains it.
/// Use it to PREDICT hot shards from a sample of real partition keys before choosing a key design.
/// </summary>
public static class ShardRouter
{
    public static readonly BigInteger MaxHashKey = (BigInteger.One << 128) - 1;

    public static BigInteger HashKey(string partitionKey)
    {
        var md5 = MD5.HashData(Encoding.UTF8.GetBytes(partitionKey));
        return new BigInteger(md5, isUnsigned: true, isBigEndian: true);
    }

    /// <summary>The ranges of a stream freshly created with shardCount shards (an even split of the hash space).</summary>
    public static IReadOnlyList<ShardRange> EvenShards(int shardCount)
    {
        ArgumentOutOfRangeException.ThrowIfLessThan(shardCount, 1);
        var width = (MaxHashKey + 1) / shardCount;
        var shards = new List<ShardRange>(shardCount);
        for (var i = 0; i < shardCount; i++)
        {
            var start = width * i;
            var end = i == shardCount - 1 ? MaxHashKey : start + width - 1;
            shards.Add(new ShardRange($"shardId-{i:D12}", start, end));
        }
        return shards;
    }

    public static ShardRange Route(string partitionKey, IReadOnlyList<ShardRange> shards)
    {
        var hash = HashKey(partitionKey);
        return shards.Single(s => hash >= s.StartingHashKey && hash <= s.EndingHashKey);
    }

    /// <summary>Counts records per shard for a sample of partition keys. One shard far above 1/N of the traffic is a hot shard.</summary>
    public static IReadOnlyDictionary<string, int> Distribution(IEnumerable<string> partitionKeys, IReadOnlyList<ShardRange> shards)
    {
        var counts = shards.ToDictionary(s => s.ShardId, _ => 0);
        foreach (var key in partitionKeys) counts[Route(key, shards).ShardId]++;
        return counts;
    }
}
AuthorisationProducer.cs
using System.Text.Json;
using Amazon.Kinesis;
using Amazon.Kinesis.Model;

namespace Harbourline.Messaging.Samples.Kinesis;

public sealed record CardAuthorisation(string AuthorisationId, string AccountId, decimal Amount, string Currency, string MerchantCategory, DateTimeOffset At);

/// <summary>
/// Writes card-authorisation events to a Kinesis stream. The partition key is the ACCOUNT id, so every event for
/// one account lands on the same shard and is read in order, while different accounts spread across shards.
/// PutRecords is a partial-success API: HTTP 200 can carry per-record errors (usually a throttled shard),
/// so the loop resends only the failed records.
/// </summary>
public sealed class AuthorisationProducer(IAmazonKinesis kinesis, string streamName, Func<int, TimeSpan>? backoff = null)
{
    public const int MaxRecordsPerCall = 500;
    private const int MaxAttempts = 5;

    private readonly Func<int, TimeSpan> _backoff = backoff
        ?? (attempt => TimeSpan.FromMilliseconds(Random.Shared.Next(0, 50 * (1 << attempt))));

    /// <summary>Returns the authorisations that could not be written after all attempts.</summary>
    public async Task<IReadOnlyList<CardAuthorisation>> PutAsync(
        IReadOnlyList<CardAuthorisation> batch, CancellationToken ct = default)
    {
        var pending = batch.ToList();

        for (var attempt = 1; attempt <= MaxAttempts && pending.Count > 0; attempt++)
        {
            var stillFailing = new List<CardAuthorisation>();

            foreach (var chunk in pending.Chunk(MaxRecordsPerCall))
            {
                var response = await kinesis.PutRecordsAsync(new PutRecordsRequest
                {
                    StreamName = streamName,
                    Records = chunk.Select(a => new PutRecordsRequestEntry
                    {
                        PartitionKey = a.AccountId,
                        Data = new MemoryStream(JsonSerializer.SerializeToUtf8Bytes(a, JsonSerializerOptions.Web)),
                    }).ToList(),
                }, ct);

                if (response.FailedRecordCount is null or 0) continue;

                // Records[i] answers request record i; an ErrorCode marks the failures.
                for (var i = 0; i < chunk.Length; i++)
                {
                    if (!string.IsNullOrEmpty(response.Records[i].ErrorCode)) stillFailing.Add(chunk[i]);
                }
            }

            pending = stillFailing;
            if (pending.Count > 0 && attempt < MaxAttempts) await Task.Delay(_backoff(attempt), ct);
        }

        return pending;
    }
}
FraudScoringFunction.cs
using System.Text.Json;
using Amazon.Lambda.Core;
using Amazon.Lambda.KinesisEvents;

namespace Harbourline.Messaging.Samples.Kinesis;

public interface IFraudScorer
{
    /// <summary>Must be idempotent: Kinesis delivers at least once, and a retried batch re-reads the same records.</summary>
    Task ScoreAsync(CardAuthorisation authorisation, CancellationToken ct);
}

/// <summary>
/// Lambda handler for a Kinesis event source mapping with ReportBatchItemFailures enabled.
/// Records of one shard arrive in order. On the first failure we report that record's sequence number and STOP:
/// Lambda then retries from that record onward, so nothing is skipped and per-account order is preserved.
/// (With SQS you report every failed item; with an ordered stream, continuing past a failure would reorder events.)
/// </summary>
public sealed class FraudScoringFunction(IFraudScorer scorer)
{
    public FraudScoringFunction() : this(new LoggingFraudScorer()) { }

    public async Task<StreamsEventResponse> HandleAsync(KinesisEvent kinesisEvent, ILambdaContext context)
    {
        var failures = new List<StreamsEventResponse.BatchItemFailure>();

        foreach (var record in kinesisEvent.Records)
        {
            try
            {
                var authorisation = JsonSerializer.Deserialize<CardAuthorisation>(record.Kinesis.Data, JsonSerializerOptions.Web)
                    ?? throw new JsonException("Empty authorisation record.");
                await scorer.ScoreAsync(authorisation, CancellationToken.None);
            }
            catch (Exception ex)
            {
                context.Logger.LogError($"Failed at sequence {record.Kinesis.SequenceNumber}: {ex.GetType().Name}");
                failures.Add(new StreamsEventResponse.BatchItemFailure { ItemIdentifier = record.Kinesis.SequenceNumber });
                break; // do not process later records of this batch out of order
            }
        }

        return new StreamsEventResponse { BatchItemFailures = failures };
    }

    private sealed class LoggingFraudScorer : IFraudScorer
    {
        public Task ScoreAsync(CardAuthorisation authorisation, CancellationToken ct)
        {
            Console.WriteLine($"scored {authorisation.AuthorisationId}");
            return Task.CompletedTask;
        }
    }
}
MskIamProducer.cs
using Amazon;
using AWS.MSK.Auth;
using Confluent.Kafka;

namespace Harbourline.Messaging.Samples.Kafka;

/// <summary>
/// A Kafka producer for Amazon MSK with IAM access control. Confluent.Kafka speaks OAUTHBEARER; the AWS.MSK.Auth
/// package turns your AWS credentials into the short-lived token (a signed URL) that MSK accepts.
/// EnableIdempotence + Acks.All give per-partition ordering without duplicates caused by producer retries.
/// </summary>
public static class MskIamProducer
{
    public static IProducer<string, string> Create(string bootstrapServers, RegionEndpoint region)
    {
        var config = new ProducerConfig
        {
            BootstrapServers = bootstrapServers,       // the "IAM" endpoint of the cluster, port 9098
            SecurityProtocol = SecurityProtocol.SaslSsl,
            SaslMechanism = SaslMechanism.OAuthBearer,
            Acks = Acks.All,
            EnableIdempotence = true,                  // retries cannot create duplicates or reorder within a partition
            MessageTimeoutMs = 30_000,
        };

        return new ProducerBuilder<string, string>(config)
            .SetOAuthBearerTokenRefreshHandler((client, _) =>
            {
                // Called when the token is about to expire. Credentials come from the normal AWS credential chain.
                var (token, expiryMs) = new AWSMSKAuthTokenGenerator().GenerateAuthTokenAsync(region).GetAwaiter().GetResult();
                client.OAuthBearerSetToken(token, expiryMs, "msk-iam");
            })
            .Build();
    }

    /// <summary>Key = account id: all events of one account go to one partition, so they stay ordered.</summary>
    public static async Task SendAsync(IProducer<string, string> producer, string topic, string accountId, string json, CancellationToken ct)
    {
        var result = await producer.ProduceAsync(topic, new Message<string, string> { Key = accountId, Value = json }, ct);
        Console.WriteLine($"partition {result.Partition.Value} offset {result.Offset.Value}");
    }
}

Two tests carry the lesson: a good key spreads, a hot key does not; and a failing record stops the batch and reports its sequence number.

StreamingAndMqTests.cs (excerpt)
[Fact]
public void Many_accounts_spread_evenly_but_one_hot_key_lands_on_one_shard()
{
    var shards = ShardRouter.EvenShards(4);

    var spread = ShardRouter.Distribution(Enumerable.Range(0, 4000).Select(i => $"ACC-{i}"), shards);
    Assert.All(spread.Values, count => Assert.InRange(count, 800, 1200)); // roughly a quarter each

    var hot = ShardRouter.Distribution(Enumerable.Repeat("ACC-ONE-BUSY-MERCHANT", 4000), shards);
    Assert.Equal(4000, hot.Values.Max());                                  // all traffic on one shard
    Assert.Equal(3, hot.Values.Count(c => c == 0));
}

[Fact]
public async Task First_failure_is_reported_and_later_records_are_not_processed()
{
    // A-1 succeeds, A-2 throws, A-3 must NOT be processed out of order.
    var response = await new FraudScoringFunction(scorer).HandleAsync(
        new KinesisEvent { Records = [Record("100", "A-1"), Record("101", "A-2"), Record("102", "A-3")] }, context);

    Assert.Equal("101", Assert.Single(response.BatchItemFailures).ItemIdentifier);
    await scorer.DidNotReceive().ScoreAsync(Arg.Is<CardAuthorisation>(a => a.AuthorisationId == "A-3"), Arg.Any<CancellationToken>());
}

Code explained line by line

  1. ShardRouter.HashKey computes MD5(UTF-8 bytes of the key) and reads the 16 bytes as an unsigned, big-endian 128-bit integer, the rule Kinesis documents for partition keys. isUnsigned: true, isBigEndian: true on BigInteger is what makes the result match.
  2. EvenShards splits [0, 2^128 - 1] into equal inclusive ranges, giving the last shard the remainder. Route finds the one range that contains the hash. Distribution counts keys per shard so you can see skew before you build.
  3. AuthorisationProducer.PutAsync loops up to five attempts. Each attempt slices pending records into chunks of 500 (Chunk), and sends them with the account ID as partition key.
  4. After each call it reads FailedRecordCount and pairs chunk[i] with response.Records[i]: a non-empty ErrorCode marks a failure. Only failures go into stillFailing, so successful records are never rewritten by the retry (which would create duplicates).
  5. The backoff delegate is full jitter and injectable. After the last attempt the method returns the leftovers instead of throwing: the caller decides whether to park them.
  6. FraudScoringFunction.HandleAsync iterates kinesisEvent.Records in order. On the first exception it logs only the sequence number, adds it as the BatchItemFailure, and breaks. Lambda retries from that record, so order is preserved. The error log never prints the record body.
  7. IFraudScorer has a comment that matters: implementations must be idempotent. A retried batch and a replay both re-read the same records.
  8. MskIamProducer.Create sets SecurityProtocol.SaslSsl and SaslMechanism.OAuthBearer. The token refresh handler calls AWSMSKAuthTokenGenerator.GenerateAuthTokenAsync with the Region, and hands the token and its expiry (in milliseconds) to the client. EnableIdempotence plus Acks.All keep order and avoid duplicates from producer retries, per partition. SendAsync uses the account ID as the Kafka message key, the equivalent of the Kinesis partition key.

Security considerations

⚠ Learning shortcut — not for production

The lab stream uses the AWS managed key alias/aws/kinesis, and your SSO identity writes and reads it. Production uses a customer managed KMS key per data classification (key use is logged in CloudTrail, and the key policy is a second authorisation layer), a producer role limited to kinesis:PutRecord and kinesis:PutRecords on one stream, and a consumer role limited to the read actions on that stream. Anyone who can read a stream can read every record in it, so streams that carry anything linked to a customer need access reviews.

⚠ Learning shortcut — not for production

The MSK Terraform uses IAM authentication but default VPC placeholders. A real cluster sits in private subnets, its security group allows port 9098 only from the application subnets, access is granted with kafka-cluster:* IAM actions scoped to the cluster, topic and group ARNs, and the cluster is encrypted with a customer managed key. Never enable unauthenticated access on a cluster that holds payment data.

  • Retention is a data-protection setting. A 365-day stream keeps a year of card events. Choose retention against the replay requirement, not "because it is cheap", and agree it with data protection. Deleting a customer's data does not delete it from a stream until the records expire: design for that (identifiers, not personal data, in records).
  • Minimise what is in a record. Account ID, amount, currency, merchant category and time are enough for fraud scoring. No names, no full card numbers, no CVV. Never put PCI-scoped data on a stream without a scoped-down design reviewed by security.
  • Encrypt and isolate. Server-side KMS encryption for Kinesis, interface VPC endpoints for private access, and a bucket and key policy for Firehose targets.
  • Consumers are privileged. The Lambda consumer's role reads everything on the stream. Give each consumer its own role, and register enhanced fan-out consumers by name so you can see who reads what.
  • Audit. Use CloudTrail for control-plane actions (key use, consumer registration, retention changes). For the data plane, treat consumer logs as the record of who read what.
  • Kafka specifics. Use IAM or mTLS, enforce TLS, turn on broker logs to CloudWatch or S3, restrict kafka-cluster:AlterGroup and DeleteTopic, and keep topic ACLs (if you use mTLS) in code.
  • Compliance. This lesson does not make any system compliant with GDPR, UK GDPR or PCI DSS. Retention, minimisation and access need legal, data-protection and security review.

Banking use case

Harbourline's card-authorisation stream, designed with this lesson:

DecisionChoiceReason
ServiceKinesis Data Streams, on-demandAWS-native, no Kafka estate to preserve, spiky salary-day traffic
Partition keyaccountIdOrder per account for fraud rules; millions of keys spread evenly
Retention7 daysFraud team's re-scoring window; reviewed with data protection
ConsumersFraud (Lambda, ordered, idempotent), Firehose to S3, dashboard (enhanced fan-out)Independent positions; the dashboard must not slow the fraud consumer
Record contentIdentifiers and facts; no PAN or nameMinimisation; PCI scope kept off the stream
AlarmsIterator age, write throttling, failed-batches queue depthLag and a poison record are silent otherwise
MSKNot used. A planned migration from an on-premises Kafka cluster would use MSK Express to keep the Kafka protocolPortability is the only reason to pay for Kafka

Common mistakes

  • A hot partition key (merchant, country, a constant). → Choose the entity whose order matters and that is diverse; test with ShardRouter and real keys; salt only if order per key can be relaxed.
  • Default Lambda retry on a stream. One poison record blocks a shard for a day. → Bounded retries, record age, bisect, ReportBatchItemFailures and a failure destination.
  • Reporting every failed record in an ordered stream. Later records run before earlier ones. → Report the first failure and stop.
  • Ignoring FailedRecordCount. Records silently lost during throttling. → Retry only the failed records with jitter.
  • Reading with many consumers on shared throughput. They split 2 MB/s per shard and throttle each other. → Enhanced fan-out for each independent reader, or fewer readers.
  • Treating the stream as a database. Retention ends. → Land the data in S3 or a table.
  • Non-idempotent consumers. Retries and replays double-apply. → Idempotency key from the record's business ID.
  • Provisioned shards left at a guess. Throttling or waste. → On-demand until the traffic is understood.
  • Leaving MSK or an on-demand stream running. Hourly charges. → Tags, budget alarm and the cleanup script.
  • Choosing MSK because "streams are Kafka". → Kinesis is simpler unless you need Kafka itself.

Troubleshooting

SymptomLikely causeFix
ProvisionedThroughputExceededException on putHot shard, or provisioned shards too few (1 MB/s or 1,000 records/s each)Check WriteProvisionedThroughputExceeded; fix the key; switch to on-demand or add shards; retry with jitter
Iterator age keeps growingConsumer slower than producers, a poison record, or Lambda throttledCheck consumer errors and duration; bound retries and bisect; raise the batch size or concurrency; add shards
Same records processed repeatedlyRetried batch, no partial-batch reporting, or a replayIdempotency; ReportBatchItemFailures; check for an AT_TIMESTAMP mapping
Records out of order across keysOrdering is per shard onlyPut the ordering requirement into the partition key
ReadProvisionedThroughputExceededToo many shared consumers on one shardEnhanced fan-out consumers; fewer readers
ExpiredIteratorExceptionAn iterator is older than 5 minutesGet a fresh iterator; checkpoint progress
No records from a new stream consumerStartingPosition is LATESTUse TRIM_HORIZON or AT_TIMESTAMP
KMSAccessDeniedExceptionConsumer or producer role lacks kms:Decrypt or kms:GenerateDataKeyUpdate the key policy and the role
MSK client cannot connectWrong bootstrap endpoint, security group, or the IAM port (9098) not openUse the IAM bootstrap string from the console; allow 9098 from the client subnets
Kafka SaslAuthenticationFailedToken expired or wrong Region in AWS.MSK.Auth, or the role lacks kafka-cluster:ConnectCheck the Region passed to the token generator and the role's IAM policy
Lambda MSK mapping not invokingFunction not in a VPC with a route to the cluster, or missing permissionsPlace the function in the VPC and grant the MSK describe permissions

Cost and cleanup

Cost: credits for Kinesis (well under USD 1 if deleted within the hour); paid and design-only for MSK. An on-demand stream costs about USD 0.04 per stream-hour plus a few cents per GB. This demo moves kilobytes, so it costs roughly four cents an hour ⚠ VERIFY on the Kinesis pricing page. There is no free tier for Kinesis Data Streams, so delete the stream when you finish. Enhanced fan-out, extended retention and the Lambda consumer add a little. MSK has no free tier (⚠ VERIFY), and the cheapest Serverless or Provisioned cluster costs on the order of USD 0.20 to 0.90 per hour; do not create one unless you have a budget alarm and will delete it the same hour.

Free alternatives: run the ShardRouter, producer and consumer unit tests (no AWS), or run Kafka locally with Docker (an Apache Kafka container on localhost:9092) and point Confluent.Kafka at it without the IAM settings.

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1

# Event source mappings first (if you deployed the consumer), then the stream.
for uuid in $(aws lambda list-event-source-mappings --function-name hl-m08gap-fraud-scorer \
    --query 'EventSourceMappings[].UUID' --output text 2>/dev/null); do
  aws lambda delete-event-source-mapping --uuid "$uuid"
done
aws kinesis delete-stream --stream-name hl-m08gap-authorisations --enforce-consumer-deletion

url=$(aws sqs get-queue-url --queue-name hl-m08gap-failed-batches --query QueueUrl --output text 2>/dev/null) \
  && aws sqs delete-queue --queue-url "$url"

# MSK, if you created a cluster (this is the important one for your bill).
for arn in $(aws kafka list-clusters-v2 --query "ClusterInfoList[?contains(ClusterName,'hl-m08gap')].ClusterArn" --output text); do
  aws kafka delete-cluster --cluster-arn "$arn"
done

# If you deployed with IaC: cdk destroy HlM08Gap-Stream   |   terraform destroy

aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m08-messaging-services \
  --query 'ResourceTagMappingList[].ResourceARN'
aws kinesis list-streams --query 'StreamNames'
aws kafka list-clusters-v2 --query 'ClusterInfoList[].ClusterName'

Exercises

  • 🟢 Guided: Run the demo: create the stream, write ten numbered events for each of three accounts, read every shard, and check that within each account the seq values are in order. Run ShardRouter with the real shard ranges from list-shards and confirm its prediction matches where each account's records landed. Delete the stream.
  • 🟡 Challenge: Take 5,000 real-looking account IDs plus one merchant ID that accounts for 40 percent of traffic. Use ShardRouter with 8 shards to compare two partition keys, accountId and merchantId, and report the busiest shard's share of traffic for each. Then design a salted key for the merchant and state what ordering you give up.
  • 🔴 Live problem: Ticket at 09:40: "Fraud alerts are 35 minutes late. Lambda shows no errors and no throttling. GetRecords.IteratorAgeMilliseconds is climbing on one shard and flat on the other nine. Traffic is normal." Work out the likely causes, how to confirm each, a safe immediate mitigation, and what you would change so it cannot recur.

Assignment

Add streaming-design.md to your portfolio repository for one Harbourline stream. Include: the choice of Kinesis or MSK with the reasons, the record schema (identifiers only), the partition key with a ShardRouter distribution table from a realistic sample, capacity mode and the monthly cost estimate at 2,000 events per second of 1 KB, retention and who approved it, each consumer with its read method and settings, the poison-record policy (retries, record age, failure destination), idempotency keys, alarms (iterator age, throttling, failure queue), and the IAM and KMS design.

Solutions

Knowledge check

Interview questions

  • "Queue or stream?" — A queue distributes jobs to one consumer each and deletes them; a stream keeps ordered records that every consumer reads at its own position and can replay. If you need ordering, replay or several independent readers at high volume, use a stream; otherwise a queue or an event bus is simpler.
  • "How do you pick a partition key?" — The entity whose order matters (the account), which is also diverse (millions of values) so load spreads. Test it with a sample of real keys against the hash-range rule, avoid keys that dominate traffic (a large merchant), and salt only if order per key can be relaxed.
  • "What is a hot shard and how do you find it?" — One shard receiving more than 1 MB/s or 1,000 records/s because of a dominant key. You see WriteProvisionedThroughputExceeded and uneven per-shard metrics, and you can reproduce it offline by hashing sampled keys.
  • "How do you stop a poison record blocking a stream?" — Bound retries and record age, bisect the batch, report the failing sequence number, send a description of the discarded batch to a queue, alarm on iterator age and recover from the stream itself.
  • "When would you use MSK?" — When Kafka itself is a requirement: an existing Kafka estate or tooling, Kafka Connect, compaction or transactions, or portability. For everything else on AWS, Kinesis is the lower-effort default.

Certification practice

Completion checklist

  • I can explain a stream versus a queue in two sentences
  • I can predict a record's shard with ShardRouter and spot a hot key
  • I wrote a producer that retries only failed PutRecords entries
  • I can list the Lambda event-source settings that stop a poison record blocking a shard
  • I ran the Kinesis demo, saw ordering per account and replayed from a timestamp
  • I know when MSK is justified, and what it costs to leave running
  • I deleted the stream (and any cluster) and the checks returned nothing

Key takeaways

  • A stream keeps ordered, replayable records for many independent readers. Order exists per shard (Kinesis) or partition (Kafka) and is decided by your partition key.
  • A shard absorbs about 1 MB/s or 1,000 records/s of writes: one dominant key creates a hot shard. Test key designs offline with the MD5 hash rule.
  • PutRecords is partial-success. Resend only the failed records, with jitter.
  • Defaults are dangerous for consumers: unlimited retry lets one poison record block a shard. Bound retries and record age, bisect, report the first failed sequence number and stop.
  • Kinesis is the low-effort AWS-native default. MSK is real Kafka and costs more to run and to leave running. Both are at-least-once: consumers must be idempotent.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page