Streaming ingestion: Kinesis Data Streams, Data Firehose and MSK
Move Harbourline transactions from batch files to a near-real-time stream: produce to Kinesis from C#, consume with a .NET Lambda that reports partial batch failures, deliver to S3 with Data Firehose, and know when Amazon MSK is the better fit.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.Kinesis 4.1.1) · Amazon.Lambda.KinesisEvents 4.0.0 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain streams, shards, partition keys, sequence numbers and retention, and calculate how many shards a workload needs.
- Choose between provisioned and on-demand Kinesis Data Streams capacity, and between a stream, Amazon Data Firehose and Amazon MSK.
- Write a C# producer that batches with
PutRecordsand retries only the records that failed. - Write a .NET Lambda consumer that is idempotent and returns partial batch failures so one bad record does not block a shard.
- Deliver the same stream to S3 as partitioned Parquet with Data Firehose.
Prerequisites
- Module 7 overview and S3 data lake design
- Lightweight transforms with Lambda for Lambda basics, and Module 8: Messaging for queues vs streams
- Tools: AWS CLI v2 (
learn-sandbox), Docker for building .NET
Why this matters in banking
Card fraud happens in minutes. A stolen card is typically tested with a few small payments, then used for as many purchases as possible before the customer notices. If Harbourline only checks transactions in the nightly batch, the fraud team learns about it the next morning — after the money has gone. The same is true for velocity limits on Faster Payments, ATM bursts and suspicious log-in patterns.
Streaming does not replace the batch pipeline. The nightly job still produces the reconciled, audited curated data. Streaming adds a second path that sees each transaction within seconds, applies simple rules and raises a signal — not a decision — for the fraud team or a case system. That second path must never lose events, must cope with duplicates, and must stay in order per account.
The idea in plain language
A stream is an append-only log that many readers can read independently, at their own pace, for a retention period (24 hours by default on Kinesis). Unlike a queue, reading a record does not delete it.
- A shard is a lane of the log. Each shard accepts up to 1 MB/s or 1,000 records/s of writes and serves up to 2 MB/s of reads.
- A partition key (for example the account id) decides which shard a record goes to. All records with the same key go to the same shard, in order.
- A sequence number identifies each record's position in its shard. Consumers remember the last sequence number they processed (a checkpoint).
Analogy: a stream is the CCTV recording of the bank lobby, not a ticket machine. Several people (fraud, analytics, audit) can watch the same recording, rewind it within the retention window, and each knows where they paused. Lanes (shards) are separate cameras; each customer (partition key) is always filmed by the same camera.
What problem the AWS service solves
| Service | What it is | Use it when | Limits and sharp edges |
|---|---|---|---|
| Kinesis Data Streams | Managed, sharded log | You need multiple consumers, replay, ordering per key, custom processing (Lambda, KCL, Flink) | Hot partition keys throttle one shard; consumers must checkpoint; at-least-once delivery means duplicates |
| Amazon Data Firehose | Managed delivery service (no consumers to write) | You want data landed in S3, Redshift, OpenSearch or others, optionally converted to Parquet, with buffering | Near-real-time (buffered by size/time), not for per-event logic; no replay of delivered data |
| Amazon MSK (and MSK Serverless) | Managed Apache Kafka | The bank already runs Kafka, needs Kafka APIs/ecosystem (Connect, Streams), or long retention with compaction | More concepts to operate (brokers, partitions, consumer groups); higher minimum cost |
| Amazon SQS | Queue | Work items that one consumer processes and deletes | No replay, no multiple independent readers (that is SNS fan-out) |
Capacity modes for Kinesis Data Streams:
- Provisioned: you choose the number of shards and pay per shard-hour plus PUT payload units. Cheapest for small, steady, predictable traffic — like a one-shard lab.
- On-demand: Kinesis scales shards for you. Billed per stream-hour plus per GB in and out (on-demand Standard), or — with the newer on-demand Advantage mode — per GB only with an account-level minimum throughput commitment. Good when traffic is spiky or unknown.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Event Hubs | Kinesis Data Streams | Event Hubs partitions ≈ shards; throughput units ≈ provisioned shards / on-demand |
| Event Hubs consumer groups + checkpoint store | Kinesis consumers with KCL (DynamoDB lease table) or Lambda event source mapping | Lambda manages checkpoints for you; KCL uses DynamoDB |
| Event Hubs Capture | Amazon Data Firehose to S3 | Firehose also converts to Parquet and partitions dynamically |
| Event Hubs for Apache Kafka / Confluent on Azure | Amazon MSK / MSK Serverless | MSK is real Apache Kafka, so existing Kafka clients and Connect work unchanged |
| Azure Functions Event Hubs trigger | Lambda event source mapping for Kinesis | Batch size, batching window, bisect on error, partial batch response are settings on the mapping |
| Stream Analytics | Amazon Managed Service for Apache Flink | SQL/Java streaming analytics; beyond this lesson |
Architecture
Security-boundary view. The stream, table, topic and queue are encrypted with alias/hl-m07-data. The producer's IAM role may only kinesis:PutRecord(s) on this one stream. The Lambda's execution role may read the stream, write its table, publish to its topic and nothing else. Firehose has its own role that can only write under stream/ in the raw bucket. Data never leaves AWS over the internet; producers inside a VPC use a Kinesis interface endpoint.
Sizing shards
Harbourline peaks at 600 card transactions per second; each JSON record is about 600 bytes.
- Writes by records: 600 / 1,000 → 1 shard.
- Writes by bytes: 600 × 600 B = 360 KB/s / 1 MB/s → 1 shard.
- Add headroom (×2 for peaks and retries) → 2 shards.
- Readers: Lambda and Firehose both read with shared throughput (2 MB/s per shard, shared). 360 KB/s × 2 readers = 720 KB/s, fine. With more consumers, use enhanced fan-out, which gives each consumer its own 2 MB/s per shard.
The real risk is a hot key: if one merchant account receives 30% of traffic and you partition by merchant, one shard saturates. Partition by the entity whose order matters (the cardholder's account) and whose distribution is wide.
Step-by-step implementation
Create the stream
One provisioned shard, 24-hour retention, server-side encryption with the data key. In the lab this comes from the HlM07-Streaming CDK stack.
Write a producer
Batch up to 500 records per PutRecords call. Check FailedRecordCount and retry only the failed entries with exponential back-off — PutRecords is not all-or-nothing.
Write the consumer
A Lambda with an event source mapping: batch size 100, batching window 2 seconds, ReportBatchItemFailures on, maximum record age 1 hour, retry attempts 5, an on-failure destination (SQS). The handler returns the sequence number of the first record it could not process.
Make processing idempotent
Kinesis and Lambda deliver at least once. Record "seen" transaction ids with a conditional write (DynamoDB attribute_not_exists) and derive signal ids deterministically, so a retry does not create a second alert.
Land the stream in S3 with Firehose
A Firehose stream with the Kinesis stream as source, record format conversion to Parquet using the Glue table schema, and dynamic partitioning on booking_date. This gives analysts the streaming data in the lake within minutes, without a consumer of your own.
AWS Console instructions
- Console → Amazon Kinesis → Data streams → Create data stream. Name
hl-m07-transactions, capacity mode Provisioned, 1 shard. After creation: Configuration → Encryption → Edit → enable server-side encryption withalias/hl-m07-data. - Lambda → Functions → hl-m07-fraud-rules → Add trigger → Kinesis → select the stream; batch size 100; starting position Latest; under Additional settings tick Report batch item failures, set retry attempts 5 and maximum age of record 3600 seconds, and an on-failure destination (SQS queue).
- Amazon Data Firehose → Create Firehose stream → source Amazon Kinesis Data Streams, destination Amazon S3. Under Transform and convert records, enable record format conversion to Apache Parquet using the Glue table
hl_m07_curated.transactions_stream⚠ VERIFY the exact console labels in your Region. Enable Dynamic partitioning with inline JSON parsing, keybooking_date. S3 prefix:stream/year=!{partitionKeyFromQuery:year}/month=!{partitionKeyFromQuery:month}/day=!{partitionKeyFromQuery:day}/. Error prefix:stream-errors/!{firehose:error-output-type}/. - Monitoring tab on the stream: watch
IncomingRecords,WriteProvisionedThroughputExceededand, on the Lambda,IteratorAge.
AWS CLI
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
# Create a one-shard provisioned stream and encrypt it with the module's key.
aws kinesis create-stream --stream-name hl-m07-transactions --shard-count 1 --stream-mode-details StreamMode=PROVISIONED
aws kinesis wait stream-exists --stream-name hl-m07-transactions
aws kinesis start-stream-encryption --stream-name hl-m07-transactions --encryption-type KMS --key-id alias/hl-m07-data
aws kinesis add-tags-to-stream --stream-name hl-m07-transactions \
--tags Project=cea-lab,Lab=m07-l6,Environment=learn,Owner=learner
# Put one test record (the CLI v2 expects base64 for blobs unless you pass --cli-binary-format).
aws kinesis put-record --stream-name hl-m07-transactions --partition-key ACC-1000000001 \
--cli-binary-format raw-in-base64-out \
--data '{"transactionId":"01a1-test","accountId":"ACC-1000000001","amount":-7500.00,"currency":"GBP","type":"card_payment","channel":"card","createdAt":"2026-11-18T10:00:00Z"}'
# Read it back from the start of the shard (for debugging; real consumers checkpoint).
SHARD=$(aws kinesis list-shards --stream-name hl-m07-transactions --query 'Shards[0].ShardId' --output text)
IT=$(aws kinesis get-shard-iterator --stream-name hl-m07-transactions --shard-id "$SHARD" --shard-iterator-type TRIM_HORIZON --query ShardIterator --output text)
aws kinesis get-records --shard-iterator "$IT" --limit 5
# Connect the Lambda with partial batch responses and a failure destination.
aws lambda create-event-source-mapping --function-name hl-m07-fraud-rules \
--event-source-arn "arn:aws:kinesis:us-east-1:111122223333:stream/hl-m07-transactions" \
--starting-position LATEST --batch-size 100 --maximum-batching-window-in-seconds 2 \
--function-response-types ReportBatchItemFailures --maximum-retry-attempts 5 \
--maximum-record-age-in-seconds 3600 \
--destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:us-east-1:111122223333:hl-m07-fraud-rules-failures"}}'Infrastructure as Code
The full stack is labs/m07-transaction-pipeline/infra/cdk/Harbourline.DataPipeline.Infra/StreamingStack.cs. The essential parts:
var stream = new Stream(this, "Transactions", new StreamProps
{
StreamName = "hl-m07-transactions",
StreamMode = StreamMode.PROVISIONED,
ShardCount = 1,
RetentionPeriod = Duration.Hours(24),
Encryption = StreamEncryption.KMS,
EncryptionKey = dataKey,
});
function.AddEventSource(new KinesisEventSource(stream, new KinesisEventSourceProps
{
StartingPosition = StartingPosition.LATEST,
BatchSize = 100,
MaxBatchingWindow = Duration.Seconds(2),
ReportBatchItemFailures = true,
RetryAttempts = 5,
MaxRecordAge = Duration.Hours(1),
OnFailure = new SqsDlq(failures),
}));Complete code
The producer and consumer below are the ones used in Lab 7.6; they are in labs/m07-transaction-pipeline/src/ with unit tests.
using System.Text;
using System.Text.Json;
using Amazon.Kinesis;
using Amazon.Kinesis.Model;
using Harbourline.DataPipeline.Core.Fraud;
namespace Harbourline.DataPipeline.Cli;
public sealed class KinesisProducer(IAmazonKinesis kinesis, string streamName, int maxAttempts = 5)
{
public const int MaxBatchSize = 500;
private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web);
public async Task<int> SendAsync(IEnumerable<StreamTransaction> transactions, CancellationToken cancellationToken = default)
{
var sent = 0;
foreach (var batch in transactions.Chunk(MaxBatchSize))
{
var pending = batch.Select(tx => new PutRecordsRequestEntry
{
PartitionKey = tx.AccountId,
Data = new MemoryStream(Encoding.UTF8.GetBytes(JsonSerializer.Serialize(tx, JsonOptions))),
}).ToList();
for (var attempt = 1; pending.Count > 0; attempt++)
{
if (attempt > maxAttempts)
{
throw new InvalidOperationException($"{pending.Count} records still failing after {maxAttempts} attempts");
}
var response = await kinesis.PutRecordsAsync(new PutRecordsRequest { StreamName = streamName, Records = pending }, cancellationToken);
var failed = new List<PutRecordsRequestEntry>();
for (var i = 0; i < response.Records.Count; i++)
{
if (response.Records[i].ErrorCode is null)
{
sent++;
}
else
{
pending[i].Data.Position = 0;
failed.Add(pending[i]);
}
}
pending = failed;
if (pending.Count > 0)
{
await Task.Delay(TimeSpan.FromMilliseconds(100 * Math.Pow(2, attempt)), cancellationToken);
}
}
}
return sent;
}
}public async Task<StreamsEventResponse> HandleAsync(KinesisEvent kinesisEvent, ILambdaContext context)
{
var response = new StreamsEventResponse { BatchItemFailures = [] };
foreach (var record in kinesisEvent.Records)
{
StreamTransaction? tx = null;
try
{
tx = JsonSerializer.Deserialize<StreamTransaction>(record.Kinesis.Data, JsonOptions)
?? throw new JsonException("empty record");
foreach (var signal in await _engine.EvaluateAsync(tx))
{
await _sns.PublishAsync(new PublishRequest
{
TopicArn = _topicArn,
Message = JsonSerializer.Serialize(signal, JsonOptions),
MessageAttributes = new Dictionary<string, MessageAttributeValue>
{
["rule"] = new() { DataType = "String", StringValue = signal.Rule },
},
});
}
}
catch (JsonException ex)
{
context.Logger.LogError($"Skipping unparseable record {record.Kinesis.SequenceNumber}: {ex.Message}");
}
catch (Exception ex) when (ex is AmazonSimpleNotificationServiceException or AmazonDynamoDBException)
{
if (tx is not null)
{
await _engine.ReleaseAsync(tx.TransactionId);
}
response.BatchItemFailures.Add(new StreamsEventResponse.BatchItemFailure { ItemIdentifier = record.Kinesis.SequenceNumber });
break;
}
}
return response;
}Code explained line by line
Producer
transactions.Chunk(MaxBatchSize)—PutRecordsaccepts at most 500 records (and 5 MB) per call, so the input is split into batches of 500.PartitionKey = tx.AccountId— all transactions for one account land on one shard, in order. The velocity rule depends on that order.JsonSerializerDefaults.Web— camelCase JSON, the API convention inCONVENTIONS.md.- The inner
forloop — each attempt sends only what is still pending. response.Records[i].ErrorCode— the response array is in the same order as the request; a non-null error code (usuallyProvisionedThroughputExceededException) marks a failed entry.pending[i].Data.Position = 0— the SDK has read the stream once; rewind it before resending.Task.Delay(100 * 2^attempt)— exponential back-off: 200 ms, 400 ms, 800 ms… gives the shard time to recover.
Consumer
StreamsEventResponse { BatchItemFailures = [] }— an empty list tells Lambda the whole batch succeeded.record.Kinesis.Data— the record payload as aMemoryStream; Lambda has already base64-decoded it._engine.EvaluateAsync(tx)— first marks the transaction as seen (conditional put) so duplicates are ignored, then appliesHIGH_AMOUNTandVELOCITY.MessageAttributes["rule"]— subscribers (e-mail forHIGH_AMOUNT, the case system for everything) can filter without parsing JSON.catch (JsonException)— a record that can never be parsed is a poison pill. Retrying it would block the shard for the whole retry period, so it is logged by sequence number (no payload, no PII) and skipped.catch (... AWS exceptions)— a transient failure: release the "seen" marker so the retry re-evaluates, report this record's sequence number and stop. Lambda retries from this record; earlier records are not resent. Stopping keeps per-shard order.
Security considerations
⚠ Learning shortcut — not for production
The lab stream uses one shard, a 24-hour retention period and LATEST as the starting position, and the producer runs from your laptop with your SSO role. In production: producers use their own IAM roles limited to kinesis:PutRecord(s) on one stream ARN; consumers start from TRIM_HORIZON after a deployment so nothing is skipped; retention is sized to your recovery objective (up to 7 days with extended retention, or longer with long-term retention); and the failure destination is alarmed and drained by a runbook.
- Encrypt streams, Firehose destinations, DynamoDB and SNS with a customer managed key; the producer and consumer roles need
kms:GenerateDataKey/kms:Decrypton it. - Minimise the payload. The stream carries account id, amount, currency, type, channel and time — enough for the rules. Do not stream names, full card numbers or free-text references. Card data on a stream would bring the stream into PCI DSS scope.
- Private connectivity. Producers in a VPC use the Kinesis interface endpoint with an endpoint policy restricted to your stream.
- Logs without PII. Log sequence numbers and transaction ids, not payloads.
Compliance review required
Automated fraud signals feed decisions that affect customers. Under UK GDPR, decisions based solely on automated processing with significant effects have specific rules. Keep a human in the loop for actions such as blocking a card, document the rules, and have the compliance team review them.
Banking use case
Harbourline's card authorisation service publishes every authorised card transaction to hl-m07-transactions, partitioned by account. The fraud-rules Lambda raises HIGH_AMOUNT for single debits of £5,000 or more and VELOCITY when an account makes more than five debits in a 10-minute window. Signals go to SNS; the fraud case system subscribes and queues them for analysts. Firehose lands the same stream in the raw zone every few minutes, so the nightly batch can reconcile streaming totals against the core banking extract — two independent paths that must agree.
Common mistakes
- Partitioning by a low-cardinality key (currency, channel) → two or three hot shards. Use the account id.
- Ignoring
FailedRecordCount→ silent data loss on throttling. Always retry failed entries. - Throwing from the handler on one bad record → the whole batch retries until the record expires, and the shard stalls (
IteratorAgeclimbs). Use partial batch responses and skip poison pills deliberately. - Assuming exactly-once → duplicate alerts and double-counted velocity. Make processing idempotent.
- Using Firehose where per-event logic is needed → Firehose buffers (by size and time) and is for delivery, not decisions.
- Leaving the stream running → shard-hours bill every hour, used or not.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
ProvisionedThroughputExceededException on put | Over 1 MB/s or 1,000 records/s on a shard, or a hot key | Retry with back-off; add shards; check key distribution |
Lambda IteratorAge keeps growing | Consumer slower than producers, or a batch failing repeatedly | Check errors; increase batch size or parallelization factor; fix poison records |
| Same alert received twice | At-least-once delivery without idempotency | Conditional write on transaction id; deterministic signal ids |
AccessDeniedException with KMS in the message | Role lacks kms:GenerateDataKey (producer) or kms:Decrypt (consumer) | Grant on the stream's key |
| Firehose writes to the error prefix | Schema mismatch in format conversion, or dynamic partition key missing | Read the error objects; align the Glue table; ensure the JSON field exists |
| No records reach Lambda after deploy | Starting position LATEST and nothing sent since | Send new records, or use TRIM_HORIZON for testing |
Cost and cleanup
us-east-1 prices checked 2026-11-18 (see Kinesis Data Streams pricing):
| Item | Price | Lab estimate (1 hour) |
|---|---|---|
| Provisioned shard | USD 0.015 per shard-hour | USD 0.015 |
| PUT payload units (25 KB each) | USD 0.014 per million | under USD 0.01 |
| On-demand Standard (if chosen) | USD 0.040 per stream-hour + USD 0.08/GB in + USD 0.04/GB out | about USD 0.04 |
| Lambda, DynamoDB on-demand, SNS | per request | cents or free tier |
| Data Firehose | per GB ingested, plus format conversion and dynamic partitioning charges ⚠ VERIFY on the Firehose pricing page | cents for MBs |
| MSK Serverless | per cluster-hour plus partition-hours and data ⚠ VERIFY | design-only in this module |
Deploy, test and destroy within one hour: the whole streaming lab costs a few cents. Free alternative: run hl-pipeline produce data/samples/transactions.csv --dry-run --burst ACC-1000000001, which evaluates the same rules locally with an in-memory store.
aws lambda list-event-source-mappings --function-name hl-m07-fraud-rules --query 'EventSourceMappings[].UUID' --output text \
| xargs -n1 aws lambda delete-event-source-mapping --uuid
aws firehose delete-delivery-stream --delivery-stream-name hl-m07-transactions-to-raw || true
aws kinesis delete-stream --stream-name hl-m07-transactions --enforce-consumer-deletion
# or, for the lab stack: cd infra/cdk && cdk destroy HlM07-StreamingExercises
- 🟢 Guided: Calculate the shard count for 2,500 records/s at 1.2 KB each, with two standard consumers and 50% headroom.
- 🟡 Challenge: Add a third rule,
ATM_BURST: three or more ATM withdrawals on one account within 60 minutes. Write it inFraudRuleEngine, add a unit test, and explain what state it needs. - 🔴 Live problem: At 09:05 the fraud team says "no alerts since 08:40". CloudWatch shows
IteratorAgeat 25 minutes and rising on one shard only. Write down what you check, in order, and how you restore service without losing or duplicating alerts.
Assignment
Deploy the streaming stack, send 200 clean transactions plus a burst, capture the SNS signal (e-mail subscription), screenshot the Lambda metrics, destroy the stack, and write a short runbook: "Fraud signal pipeline is lagging".
Solutions
Write bytes: 2,500 × 1.2 KB = 3,000 KB/s ≈ 2.93 MB/s → 3 shards by bytes (1 MB/s each). Write records: 2,500 / 1,000 → 3 shards. Reads: two shared consumers each need 2.93 MB/s = 5.86 MB/s total, at 2 MB/s per shard → 3 shards. With 50% headroom: 4.4 → 5 shards. Alternatively, on-demand mode removes the calculation; compare its per-GB price against 5 shard-hours.
Add "ATM_BURST" with a 60-minute fixed window counter keyed ATM#<account>#<window>, incremented only for type == "atm_withdrawal" and signalled when the count reaches exactly 3. It reuses the same store pattern (atomic ADD, TTL), so state is a counter per account per hour. Test: three withdrawals 10 minutes apart → one signal on the third; a fourth → no new signal. A sliding window would be more accurate but needs a list of timestamps per account; fixed windows are simpler and explainable to auditors.
- Lambda errors and logs for that shard's invocations: a repeated
BatchItemFailureat the same sequence number means a record keeps failing. 2. Look for throttling on DynamoDB or SNS (transient AWS errors) versus a code bug. 3. If a dependency is throttled, fix capacity; retries will drain the backlog in order. 4. If a specific record is the problem, the maximum record age (1 hour) will eventually send it to the failure queue — or deploy a fix that treats it as a poison pill. 5. Because "seen" markers and signal ids are idempotent, replays do not duplicate alerts. 6. After recovery, compare signals since 08:40 with the batchfraud_signalstable to confirm nothing was lost.
Knowledge check
Answer: Ordering is guaranteed only within a shard, and the velocity rule needs all of one account's transactions in order. Transaction ids would spread one account across shards.
Answer: The function can return the sequence number of the first failed record; Lambda then retries from that record instead of retrying the whole batch. Without it, any exception retries the entire batch.
Answer: Firehose. It buffers, converts to Parquet with a Glue schema, partitions dynamically and needs no consumer code. Use a stream when you need custom processing, replay or multiple consumers (Firehose can read from the stream too).
Answer: When the organisation already uses Kafka clients, Kafka Connect or Kafka Streams, or needs Kafka features such as log compaction. For a .NET team starting fresh on AWS, Kinesis is simpler.
Interview questions
- "Queue or stream?" — Queue for work items consumed once (SQS); stream for an ordered, replayable log read by several consumers (Kinesis/Kafka).
- "How do you avoid duplicates in a streaming consumer?" — Accept at-least-once delivery and make the consumer idempotent: conditional writes keyed on a business id, deterministic output ids, and upserts rather than inserts.
- "A shard is throttled but the others are idle. Why?" — Hot partition key. Choose a key with wide, even distribution, or add a suffix and aggregate downstream if ordering is not required.
- "What is enhanced fan-out?" — A dedicated 2 MB/s per shard per consumer, pushed over HTTP/2, instead of sharing 2 MB/s across all consumers; useful with many consumers or low-latency needs, at extra cost.
Certification practice
A Lambda function consumes a Kinesis data stream. Occasionally a malformed record causes the function to fail, and the iterator age grows for hours until the record expires. The team must keep processing valid records in order and isolate failures. What should they do?
A. Increase the batch size to 10,000.
B. Enable ReportBatchItemFailures, return the failed record's sequence number, set a maximum record age and retry attempts, and configure an on-failure destination.
C. Switch the stream to on-demand capacity mode.
D. Use SQS standard queues instead of Kinesis.
Answer: B. Partial batch responses with bounded retries and a failure destination isolate the bad record. A makes failures larger. C changes capacity, not error handling. D loses ordering and replay.
A bank wants clickstream events from a Kinesis data stream stored in S3 as Parquet, partitioned by date, queryable in Athena, with minimal code. Which solution meets this?
A. A custom KCL application on EC2 that writes Parquet files. B. Amazon Data Firehose reading from the stream, with record format conversion using a Glue table schema and dynamic partitioning. C. Kinesis Data Streams enhanced fan-out directly to S3. D. A Lambda function that writes each event as a separate JSON object.
Answer: B. Managed conversion and partitioning. A is code to run and maintain. C does not exist (fan-out is a read mechanism). D creates millions of tiny JSON files, expensive to query.
An application must process each customer's account events in the order they occurred, with several independent consumer applications. Which service and design fit best?
A. SQS standard queue with multiple consumers. B. Kinesis Data Streams with the customer account id as the partition key. C. SNS topic with e-mail subscriptions. D. S3 event notifications.
Answer: B. Per-key ordering on a shard and independent consumers. A has no ordering guarantee. C and D are not ordered logs.
Completion checklist
- I can size a stream from records per second, record size and consumer count
- I can explain at-least-once delivery and how my consumer stays idempotent
- I understand partial batch responses and poison-pill handling
- I know when to choose Firehose or MSK instead of a stream
- I deleted the stream, event source mapping and Firehose stream
Key takeaways
- A stream is an ordered, replayable log; shards set throughput and the partition key sets ordering.
PutRecordscan partially fail: retry only failed entries with back-off.- Lambda consumers must be idempotent and should use partial batch responses with bounded retries and a failure destination.
- Firehose is the low-code path to Parquet in S3; MSK is for Kafka ecosystems.
- Shard-hours bill continuously: deploy, test, destroy.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Amazon Redshift and Redshift Serverless
Decide when Harbourline Bank needs a warehouse rather than Athena, load curated Parquet into Redshift with COPY, choose distribution and sort keys, query the lake with Spectrum, and predict the bill.
Orchestration: Step Functions, EventBridge and MWAA
Run Harbourline's transaction pipeline as an event-driven Step Functions workflow — triggered by S3 through EventBridge, waiting on a Glue job, retrying safely and alerting on failure — and know when EventBridge Scheduler, Glue workflows or Apache Airflow (MWAA) fit better.