Skip to Content
Cloud Engineering Academy
7. Data Engineering & ETL Pipelines

Streaming ingestion: Kinesis Data Streams, Data Firehose and MSK

Move Harbourline transactions from batch files to a near-real-time stream: produce to Kinesis from C#, consume with a .NET Lambda that reports partial batch failures, deliver to S3 with Data Firehose, and know when Amazon MSK is the better fit.

intermediate⏱ 1 h 40 min🟡 Low costDEA-C01SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.Kinesis 4.1.1) · Amazon.Lambda.KinesisEvents 4.0.0 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain streams, shards, partition keys, sequence numbers and retention, and calculate how many shards a workload needs.
  • Choose between provisioned and on-demand Kinesis Data Streams capacity, and between a stream, Amazon Data Firehose and Amazon MSK.
  • Write a C# producer that batches with PutRecords and retries only the records that failed.
  • Write a .NET Lambda consumer that is idempotent and returns partial batch failures so one bad record does not block a shard.
  • Deliver the same stream to S3 as partitioned Parquet with Data Firehose.

Prerequisites

Why this matters in banking

Card fraud happens in minutes. A stolen card is typically tested with a few small payments, then used for as many purchases as possible before the customer notices. If Harbourline only checks transactions in the nightly batch, the fraud team learns about it the next morning — after the money has gone. The same is true for velocity limits on Faster Payments, ATM bursts and suspicious log-in patterns.

Streaming does not replace the batch pipeline. The nightly job still produces the reconciled, audited curated data. Streaming adds a second path that sees each transaction within seconds, applies simple rules and raises a signal — not a decision — for the fraud team or a case system. That second path must never lose events, must cope with duplicates, and must stay in order per account.

The idea in plain language

A stream is an append-only log that many readers can read independently, at their own pace, for a retention period (24 hours by default on Kinesis). Unlike a queue, reading a record does not delete it.

  • A shard is a lane of the log. Each shard accepts up to 1 MB/s or 1,000 records/s of writes and serves up to 2 MB/s of reads.
  • A partition key (for example the account id) decides which shard a record goes to. All records with the same key go to the same shard, in order.
  • A sequence number identifies each record's position in its shard. Consumers remember the last sequence number they processed (a checkpoint).

Analogy: a stream is the CCTV recording of the bank lobby, not a ticket machine. Several people (fraud, analytics, audit) can watch the same recording, rewind it within the retention window, and each knows where they paused. Lanes (shards) are separate cameras; each customer (partition key) is always filmed by the same camera.

What problem the AWS service solves

ServiceWhat it isUse it whenLimits and sharp edges
Kinesis Data StreamsManaged, sharded logYou need multiple consumers, replay, ordering per key, custom processing (Lambda, KCL, Flink)Hot partition keys throttle one shard; consumers must checkpoint; at-least-once delivery means duplicates
Amazon Data FirehoseManaged delivery service (no consumers to write)You want data landed in S3, Redshift, OpenSearch or others, optionally converted to Parquet, with bufferingNear-real-time (buffered by size/time), not for per-event logic; no replay of delivered data
Amazon MSK (and MSK Serverless)Managed Apache KafkaThe bank already runs Kafka, needs Kafka APIs/ecosystem (Connect, Streams), or long retention with compactionMore concepts to operate (brokers, partitions, consumer groups); higher minimum cost
Amazon SQSQueueWork items that one consumer processes and deletesNo replay, no multiple independent readers (that is SNS fan-out)

Capacity modes for Kinesis Data Streams:

  • Provisioned: you choose the number of shards and pay per shard-hour plus PUT payload units. Cheapest for small, steady, predictable traffic — like a one-shard lab.
  • On-demand: Kinesis scales shards for you. Billed per stream-hour plus per GB in and out (on-demand Standard), or — with the newer on-demand Advantage mode — per GB only with an account-level minimum throughput commitment. Good when traffic is spiky or unknown.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Event HubsKinesis Data StreamsEvent Hubs partitions ≈ shards; throughput units ≈ provisioned shards / on-demand
Event Hubs consumer groups + checkpoint storeKinesis consumers with KCL (DynamoDB lease table) or Lambda event source mappingLambda manages checkpoints for you; KCL uses DynamoDB
Event Hubs CaptureAmazon Data Firehose to S3Firehose also converts to Parquet and partitions dynamically
Event Hubs for Apache Kafka / Confluent on AzureAmazon MSK / MSK ServerlessMSK is real Apache Kafka, so existing Kafka clients and Connect work unchanged
Azure Functions Event Hubs triggerLambda event source mapping for KinesisBatch size, batching window, bisect on error, partial batch response are settings on the mapping
Stream AnalyticsAmazon Managed Service for Apache FlinkSQL/Java streaming analytics; beyond this lesson

Architecture

Security-boundary view. The stream, table, topic and queue are encrypted with alias/hl-m07-data. The producer's IAM role may only kinesis:PutRecord(s) on this one stream. The Lambda's execution role may read the stream, write its table, publish to its topic and nothing else. Firehose has its own role that can only write under stream/ in the raw bucket. Data never leaves AWS over the internet; producers inside a VPC use a Kinesis interface endpoint.

Sizing shards

Harbourline peaks at 600 card transactions per second; each JSON record is about 600 bytes.

  • Writes by records: 600 / 1,000 → 1 shard.
  • Writes by bytes: 600 × 600 B = 360 KB/s / 1 MB/s → 1 shard.
  • Add headroom (×2 for peaks and retries) → 2 shards.
  • Readers: Lambda and Firehose both read with shared throughput (2 MB/s per shard, shared). 360 KB/s × 2 readers = 720 KB/s, fine. With more consumers, use enhanced fan-out, which gives each consumer its own 2 MB/s per shard.

The real risk is a hot key: if one merchant account receives 30% of traffic and you partition by merchant, one shard saturates. Partition by the entity whose order matters (the cardholder's account) and whose distribution is wide.

Step-by-step implementation

Create the stream

One provisioned shard, 24-hour retention, server-side encryption with the data key. In the lab this comes from the HlM07-Streaming CDK stack.

Write a producer

Batch up to 500 records per PutRecords call. Check FailedRecordCount and retry only the failed entries with exponential back-off — PutRecords is not all-or-nothing.

Write the consumer

A Lambda with an event source mapping: batch size 100, batching window 2 seconds, ReportBatchItemFailures on, maximum record age 1 hour, retry attempts 5, an on-failure destination (SQS). The handler returns the sequence number of the first record it could not process.

Make processing idempotent

Kinesis and Lambda deliver at least once. Record "seen" transaction ids with a conditional write (DynamoDB attribute_not_exists) and derive signal ids deterministically, so a retry does not create a second alert.

Land the stream in S3 with Firehose

A Firehose stream with the Kinesis stream as source, record format conversion to Parquet using the Glue table schema, and dynamic partitioning on booking_date. This gives analysts the streaming data in the lake within minutes, without a consumer of your own.

AWS Console instructions

  1. Console → Amazon Kinesis → Data streams → Create data stream. Name hl-m07-transactions, capacity mode Provisioned, 1 shard. After creation: Configuration → Encryption → Edit → enable server-side encryption with alias/hl-m07-data.
  2. Lambda → Functions → hl-m07-fraud-rules → Add trigger → Kinesis → select the stream; batch size 100; starting position Latest; under Additional settings tick Report batch item failures, set retry attempts 5 and maximum age of record 3600 seconds, and an on-failure destination (SQS queue).
  3. Amazon Data Firehose → Create Firehose stream → source Amazon Kinesis Data Streams, destination Amazon S3. Under Transform and convert records, enable record format conversion to Apache Parquet using the Glue table hl_m07_curated.transactions_stream ⚠ VERIFY the exact console labels in your Region. Enable Dynamic partitioning with inline JSON parsing, key booking_date. S3 prefix: stream/year=!{partitionKeyFromQuery:year}/month=!{partitionKeyFromQuery:month}/day=!{partitionKeyFromQuery:day}/. Error prefix: stream-errors/!{firehose:error-output-type}/.
  4. Monitoring tab on the stream: watch IncomingRecords, WriteProvisionedThroughputExceeded and, on the Lambda, IteratorAge.

AWS CLI

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1

# Create a one-shard provisioned stream and encrypt it with the module's key.
aws kinesis create-stream --stream-name hl-m07-transactions --shard-count 1 --stream-mode-details StreamMode=PROVISIONED
aws kinesis wait stream-exists --stream-name hl-m07-transactions
aws kinesis start-stream-encryption --stream-name hl-m07-transactions --encryption-type KMS --key-id alias/hl-m07-data
aws kinesis add-tags-to-stream --stream-name hl-m07-transactions \
  --tags Project=cea-lab,Lab=m07-l6,Environment=learn,Owner=learner

# Put one test record (the CLI v2 expects base64 for blobs unless you pass --cli-binary-format).
aws kinesis put-record --stream-name hl-m07-transactions --partition-key ACC-1000000001 \
  --cli-binary-format raw-in-base64-out \
  --data '{"transactionId":"01a1-test","accountId":"ACC-1000000001","amount":-7500.00,"currency":"GBP","type":"card_payment","channel":"card","createdAt":"2026-11-18T10:00:00Z"}'

# Read it back from the start of the shard (for debugging; real consumers checkpoint).
SHARD=$(aws kinesis list-shards --stream-name hl-m07-transactions --query 'Shards[0].ShardId' --output text)
IT=$(aws kinesis get-shard-iterator --stream-name hl-m07-transactions --shard-id "$SHARD" --shard-iterator-type TRIM_HORIZON --query ShardIterator --output text)
aws kinesis get-records --shard-iterator "$IT" --limit 5

# Connect the Lambda with partial batch responses and a failure destination.
aws lambda create-event-source-mapping --function-name hl-m07-fraud-rules \
  --event-source-arn "arn:aws:kinesis:us-east-1:111122223333:stream/hl-m07-transactions" \
  --starting-position LATEST --batch-size 100 --maximum-batching-window-in-seconds 2 \
  --function-response-types ReportBatchItemFailures --maximum-retry-attempts 5 \
  --maximum-record-age-in-seconds 3600 \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:us-east-1:111122223333:hl-m07-fraud-rules-failures"}}'

Infrastructure as Code

The full stack is labs/m07-transaction-pipeline/infra/cdk/Harbourline.DataPipeline.Infra/StreamingStack.cs. The essential parts:

var stream = new Stream(this, "Transactions", new StreamProps
{
    StreamName = "hl-m07-transactions",
    StreamMode = StreamMode.PROVISIONED,
    ShardCount = 1,
    RetentionPeriod = Duration.Hours(24),
    Encryption = StreamEncryption.KMS,
    EncryptionKey = dataKey,
});

function.AddEventSource(new KinesisEventSource(stream, new KinesisEventSourceProps
{
    StartingPosition = StartingPosition.LATEST,
    BatchSize = 100,
    MaxBatchingWindow = Duration.Seconds(2),
    ReportBatchItemFailures = true,
    RetryAttempts = 5,
    MaxRecordAge = Duration.Hours(1),
    OnFailure = new SqsDlq(failures),
}));

Complete code

The producer and consumer below are the ones used in Lab 7.6; they are in labs/m07-transaction-pipeline/src/ with unit tests.

KinesisProducer.cs
using System.Text;
using System.Text.Json;
using Amazon.Kinesis;
using Amazon.Kinesis.Model;
using Harbourline.DataPipeline.Core.Fraud;

namespace Harbourline.DataPipeline.Cli;

public sealed class KinesisProducer(IAmazonKinesis kinesis, string streamName, int maxAttempts = 5)
{
    public const int MaxBatchSize = 500;
    private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web);

    public async Task<int> SendAsync(IEnumerable<StreamTransaction> transactions, CancellationToken cancellationToken = default)
    {
        var sent = 0;
        foreach (var batch in transactions.Chunk(MaxBatchSize))
        {
            var pending = batch.Select(tx => new PutRecordsRequestEntry
            {
                PartitionKey = tx.AccountId,
                Data = new MemoryStream(Encoding.UTF8.GetBytes(JsonSerializer.Serialize(tx, JsonOptions))),
            }).ToList();

            for (var attempt = 1; pending.Count > 0; attempt++)
            {
                if (attempt > maxAttempts)
                {
                    throw new InvalidOperationException($"{pending.Count} records still failing after {maxAttempts} attempts");
                }

                var response = await kinesis.PutRecordsAsync(new PutRecordsRequest { StreamName = streamName, Records = pending }, cancellationToken);
                var failed = new List<PutRecordsRequestEntry>();
                for (var i = 0; i < response.Records.Count; i++)
                {
                    if (response.Records[i].ErrorCode is null)
                    {
                        sent++;
                    }
                    else
                    {
                        pending[i].Data.Position = 0;
                        failed.Add(pending[i]);
                    }
                }

                pending = failed;
                if (pending.Count > 0)
                {
                    await Task.Delay(TimeSpan.FromMilliseconds(100 * Math.Pow(2, attempt)), cancellationToken);
                }
            }
        }

        return sent;
    }
}
FraudSignalFunction.cs (handler loop)
public async Task<StreamsEventResponse> HandleAsync(KinesisEvent kinesisEvent, ILambdaContext context)
{
    var response = new StreamsEventResponse { BatchItemFailures = [] };
    foreach (var record in kinesisEvent.Records)
    {
        StreamTransaction? tx = null;
        try
        {
            tx = JsonSerializer.Deserialize<StreamTransaction>(record.Kinesis.Data, JsonOptions)
                 ?? throw new JsonException("empty record");
            foreach (var signal in await _engine.EvaluateAsync(tx))
            {
                await _sns.PublishAsync(new PublishRequest
                {
                    TopicArn = _topicArn,
                    Message = JsonSerializer.Serialize(signal, JsonOptions),
                    MessageAttributes = new Dictionary<string, MessageAttributeValue>
                    {
                        ["rule"] = new() { DataType = "String", StringValue = signal.Rule },
                    },
                });
            }
        }
        catch (JsonException ex)
        {
            context.Logger.LogError($"Skipping unparseable record {record.Kinesis.SequenceNumber}: {ex.Message}");
        }
        catch (Exception ex) when (ex is AmazonSimpleNotificationServiceException or AmazonDynamoDBException)
        {
            if (tx is not null)
            {
                await _engine.ReleaseAsync(tx.TransactionId);
            }

            response.BatchItemFailures.Add(new StreamsEventResponse.BatchItemFailure { ItemIdentifier = record.Kinesis.SequenceNumber });
            break;
        }
    }

    return response;
}

Code explained line by line

Producer

  1. transactions.Chunk(MaxBatchSize) — PutRecords accepts at most 500 records (and 5 MB) per call, so the input is split into batches of 500.
  2. PartitionKey = tx.AccountId — all transactions for one account land on one shard, in order. The velocity rule depends on that order.
  3. JsonSerializerDefaults.Web — camelCase JSON, the API convention in CONVENTIONS.md.
  4. The inner for loop — each attempt sends only what is still pending.
  5. response.Records[i].ErrorCode — the response array is in the same order as the request; a non-null error code (usually ProvisionedThroughputExceededException) marks a failed entry.
  6. pending[i].Data.Position = 0 — the SDK has read the stream once; rewind it before resending.
  7. Task.Delay(100 * 2^attempt) — exponential back-off: 200 ms, 400 ms, 800 ms… gives the shard time to recover.

Consumer

  1. StreamsEventResponse { BatchItemFailures = [] } — an empty list tells Lambda the whole batch succeeded.
  2. record.Kinesis.Data — the record payload as a MemoryStream; Lambda has already base64-decoded it.
  3. _engine.EvaluateAsync(tx) — first marks the transaction as seen (conditional put) so duplicates are ignored, then applies HIGH_AMOUNT and VELOCITY.
  4. MessageAttributes["rule"] — subscribers (e-mail for HIGH_AMOUNT, the case system for everything) can filter without parsing JSON.
  5. catch (JsonException) — a record that can never be parsed is a poison pill. Retrying it would block the shard for the whole retry period, so it is logged by sequence number (no payload, no PII) and skipped.
  6. catch (... AWS exceptions) — a transient failure: release the "seen" marker so the retry re-evaluates, report this record's sequence number and stop. Lambda retries from this record; earlier records are not resent. Stopping keeps per-shard order.

Security considerations

⚠ Learning shortcut — not for production

The lab stream uses one shard, a 24-hour retention period and LATEST as the starting position, and the producer runs from your laptop with your SSO role. In production: producers use their own IAM roles limited to kinesis:PutRecord(s) on one stream ARN; consumers start from TRIM_HORIZON after a deployment so nothing is skipped; retention is sized to your recovery objective (up to 7 days with extended retention, or longer with long-term retention); and the failure destination is alarmed and drained by a runbook.

  • Encrypt streams, Firehose destinations, DynamoDB and SNS with a customer managed key; the producer and consumer roles need kms:GenerateDataKey / kms:Decrypt on it.
  • Minimise the payload. The stream carries account id, amount, currency, type, channel and time — enough for the rules. Do not stream names, full card numbers or free-text references. Card data on a stream would bring the stream into PCI DSS scope.
  • Private connectivity. Producers in a VPC use the Kinesis interface endpoint with an endpoint policy restricted to your stream.
  • Logs without PII. Log sequence numbers and transaction ids, not payloads.

Compliance review required

Automated fraud signals feed decisions that affect customers. Under UK GDPR, decisions based solely on automated processing with significant effects have specific rules. Keep a human in the loop for actions such as blocking a card, document the rules, and have the compliance team review them.

Banking use case

Harbourline's card authorisation service publishes every authorised card transaction to hl-m07-transactions, partitioned by account. The fraud-rules Lambda raises HIGH_AMOUNT for single debits of £5,000 or more and VELOCITY when an account makes more than five debits in a 10-minute window. Signals go to SNS; the fraud case system subscribes and queues them for analysts. Firehose lands the same stream in the raw zone every few minutes, so the nightly batch can reconcile streaming totals against the core banking extract — two independent paths that must agree.

Common mistakes

  • Partitioning by a low-cardinality key (currency, channel) → two or three hot shards. Use the account id.
  • Ignoring FailedRecordCount → silent data loss on throttling. Always retry failed entries.
  • Throwing from the handler on one bad record → the whole batch retries until the record expires, and the shard stalls (IteratorAge climbs). Use partial batch responses and skip poison pills deliberately.
  • Assuming exactly-once → duplicate alerts and double-counted velocity. Make processing idempotent.
  • Using Firehose where per-event logic is needed → Firehose buffers (by size and time) and is for delivery, not decisions.
  • Leaving the stream running → shard-hours bill every hour, used or not.

Troubleshooting

SymptomLikely causeFix
ProvisionedThroughputExceededException on putOver 1 MB/s or 1,000 records/s on a shard, or a hot keyRetry with back-off; add shards; check key distribution
Lambda IteratorAge keeps growingConsumer slower than producers, or a batch failing repeatedlyCheck errors; increase batch size or parallelization factor; fix poison records
Same alert received twiceAt-least-once delivery without idempotencyConditional write on transaction id; deterministic signal ids
AccessDeniedException with KMS in the messageRole lacks kms:GenerateDataKey (producer) or kms:Decrypt (consumer)Grant on the stream's key
Firehose writes to the error prefixSchema mismatch in format conversion, or dynamic partition key missingRead the error objects; align the Glue table; ensure the JSON field exists
No records reach Lambda after deployStarting position LATEST and nothing sent sinceSend new records, or use TRIM_HORIZON for testing

Cost and cleanup

us-east-1 prices checked 2026-11-18 (see Kinesis Data Streams pricing):

ItemPriceLab estimate (1 hour)
Provisioned shardUSD 0.015 per shard-hourUSD 0.015
PUT payload units (25 KB each)USD 0.014 per millionunder USD 0.01
On-demand Standard (if chosen)USD 0.040 per stream-hour + USD 0.08/GB in + USD 0.04/GB outabout USD 0.04
Lambda, DynamoDB on-demand, SNSper requestcents or free tier
Data Firehoseper GB ingested, plus format conversion and dynamic partitioning charges ⚠ VERIFY on the Firehose pricing pagecents for MBs
MSK Serverlessper cluster-hour plus partition-hours and data ⚠ VERIFYdesign-only in this module

Deploy, test and destroy within one hour: the whole streaming lab costs a few cents. Free alternative: run hl-pipeline produce data/samples/transactions.csv --dry-run --burst ACC-1000000001, which evaluates the same rules locally with an in-memory store.

aws lambda list-event-source-mappings --function-name hl-m07-fraud-rules --query 'EventSourceMappings[].UUID' --output text \
  | xargs -n1 aws lambda delete-event-source-mapping --uuid
aws firehose delete-delivery-stream --delivery-stream-name hl-m07-transactions-to-raw || true
aws kinesis delete-stream --stream-name hl-m07-transactions --enforce-consumer-deletion
# or, for the lab stack: cd infra/cdk && cdk destroy HlM07-Streaming

Exercises

  • 🟢 Guided: Calculate the shard count for 2,500 records/s at 1.2 KB each, with two standard consumers and 50% headroom.
  • 🟡 Challenge: Add a third rule, ATM_BURST: three or more ATM withdrawals on one account within 60 minutes. Write it in FraudRuleEngine, add a unit test, and explain what state it needs.
  • 🔴 Live problem: At 09:05 the fraud team says "no alerts since 08:40". CloudWatch shows IteratorAge at 25 minutes and rising on one shard only. Write down what you check, in order, and how you restore service without losing or duplicating alerts.

Assignment

Deploy the streaming stack, send 200 clean transactions plus a burst, capture the SNS signal (e-mail subscription), screenshot the Lambda metrics, destroy the stack, and write a short runbook: "Fraud signal pipeline is lagging".

Solutions

Knowledge check

Interview questions

  • "Queue or stream?" — Queue for work items consumed once (SQS); stream for an ordered, replayable log read by several consumers (Kinesis/Kafka).
  • "How do you avoid duplicates in a streaming consumer?" — Accept at-least-once delivery and make the consumer idempotent: conditional writes keyed on a business id, deterministic output ids, and upserts rather than inserts.
  • "A shard is throttled but the others are idle. Why?" — Hot partition key. Choose a key with wide, even distribution, or add a suffix and aggregate downstream if ordering is not required.
  • "What is enhanced fan-out?" — A dedicated 2 MB/s per shard per consumer, pushed over HTTP/2, instead of sharing 2 MB/s across all consumers; useful with many consumers or low-latency needs, at extra cost.

Certification practice

Completion checklist

  • I can size a stream from records per second, record size and consumer count
  • I can explain at-least-once delivery and how my consumer stays idempotent
  • I understand partial batch responses and poison-pill handling
  • I know when to choose Firehose or MSK instead of a stream
  • I deleted the stream, event source mapping and Firehose stream

Key takeaways

  • A stream is an ordered, replayable log; shards set throughput and the partition key sets ordering.
  • PutRecords can partially fail: retry only failed entries with back-off.
  • Lambda consumers must be idempotent and should use partial batch responses with bounded retries and a failure destination.
  • Firehose is the low-code path to Parquet in S3; MSK is for Kafka ecosystems.
  • Shard-hours bill continuously: deploy, test, destroy.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page