Real-time transaction processing: latency budgets, back-pressure and ordering
Design a real-time authorisation path that decides in milliseconds on a small hot data set, then streams events to everything else: latency budgets with deadline propagation, DynamoDB balance reservations, Kinesis ordering per account, back-pressure and load shedding, and graceful degradation.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.DynamoDBv2, AWSSDK.Kinesis 4.x) · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Split a real-time banking flow into a synchronous decision path with a hard latency budget and an asynchronous stream path that follows it.
- Build a latency budget table, allocate it across components and enforce it in code with deadline propagation.
- Reserve funds with a conditional DynamoDB update that is safe under concurrency and under retries.
- Keep per-account ordering with Kinesis partition keys, and detect duplicates and gaps with a sequence guard.
- Explain back-pressure and apply admission control, bounded queues, load shedding by priority and iterator-age alarms.
- Design graceful degradation: what the system decides when a dependency is too slow.
- Write the real-time architecture using the 15-section template.
Prerequisites
- Payment processing for idempotency and the ledger
- Delivery guarantees and ordering and retry strategies
- Streaming ingestion with Kinesis for producer and consumer mechanics
- Auto scaling everywhere
- PCI DSS on AWS for what card data you must keep out of your workload
- Tools: .NET 10 SDK or Docker, AWS CLI v2 (optional)
Why this matters in banking
When a customer taps a card at a till, the authorisation decision has to come back in a fraction of a second. Card networks and issuers set timeouts, and a response that arrives late is treated as a failure, even if your system eventually approves it. A late approval can then become a reversal, a duplicate attempt and an annoyed customer at a checkout queue.
Real-time instant payments and wallet top-ups share the pattern. Four things go wrong in practice:
- A slow dependency eats the whole budget. The risk scorer takes 400 ms once, and every authorisation times out.
- Concurrency double-spends. Two simultaneous authorisations each see GBP 100 available and both approve GBP 80.
- Order is lost. A reversal is processed before the original authorisation, so the balance goes wrong.
- A traffic spike turns into an outage. Queues grow without limit, latency climbs, retries pile on, and the system collapses under its own recovery traffic.
Educational content, not scheme or legal advice
Card scheme timeouts, stand-in processing rules, message formats (card networks commonly use ISO 8583 ⚠ VERIFY) and PCI DSS obligations are set by the networks and your acquirer or issuer processor. The numbers in this lesson are fictional teaching values. Keep primary account numbers out of general workloads: use tokens, and put any workload that touches card data under your PCI scope decisions (PCI DSS on AWS).
The idea in plain language
Split the work in two:
- Decision path (synchronous): "May this transaction go ahead?" It uses only data that is small and hot: the account's available balance, its status, a few rules and, if time allows, a risk score. It must answer within the budget. It writes one thing: a reservation of funds.
- Stream path (asynchronous): everything that can wait: posting to the ledger, notifications, fraud analytics, statements, reporting. It reads an ordered stream of events and may take seconds.
The decision path is deliberately narrow. Every extra dependency on it multiplies the chances of missing the budget.
Analogy: a supermarket checkout. The cashier (decision path) only checks that the card works and the amount is allowed, then says "approved". Restocking shelves, updating sales reports and emailing the receipt (stream path) happen later, in the order the sales happened, without making the queue wait.
Four ideas to carry through the design
| Idea | One-line version |
|---|---|
| Latency budget | Total time allowed, divided among components; the sum of worst realistic cases must fit |
| Deadline propagation | Each call is given only the time remaining, not a fresh full timeout |
| Back-pressure | When downstream is full, push the signal upstream (slow down, reject, shed) instead of buffering forever |
| Ordering by key | Order matters per account, not globally; route each account's events through one ordered lane |
What problem the AWS services solve
| Service | Role | Limits and sharp edges |
|---|---|---|
| Amazon API Gateway, or an NLB/ALB to ECS | Entry for the decision path | API Gateway adds latency and has an integration timeout; for very tight budgets a load balancer in front of containers is often used ⚠ VERIFY latency expectations by measurement |
| Amazon ECS on AWS Fargate | Runs the authorisation service with warm capacity | Scale-out takes tens of seconds; keep headroom for spikes |
| Amazon DynamoDB | Single-digit-millisecond reads and writes for balances, reservations, idempotency | Each partition has throughput limits ⚠ VERIFY current figures; a hot account key can throttle |
| DynamoDB Accelerator (DAX) | In-memory cache for eventually consistent reads | Not suitable for balance decisions that need strong consistency |
| Amazon ElastiCache (Valkey) | Sub-millisecond lookups for rules, velocity counters | A cache can be cold or lost; never the only copy of money |
| Amazon Kinesis Data Streams | Ordered, durable stream per partition key; multiple consumers | Per-shard write about 1 MB/s or 1,000 records/s, read about 2 MB/s shared ⚠ VERIFY; a failing record blocks its shard until retries or age limits end it |
| AWS Lambda (Kinesis event source mapping) | Stream consumers with batching, retries, partial failure reporting, parallelisation | Concurrency per shard is bounded by the parallelisation factor; poison records must be handled deliberately |
| Amazon SQS FIFO | Alternative when ordering per group and dedup are needed with simpler consumers | Throughput limits per queue or group ⚠ VERIFY; no replay as with a stream |
| Amazon CloudWatch | Latency percentiles, iterator age, throttles, alarms | Percentile metrics need to be published deliberately; alarm on p99, not averages |
| AWS Auto Scaling / Application Auto Scaling | Scale tasks on CPU, request count or custom metrics | Reactive, so it cannot absorb a sudden vertical spike by itself |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Event Hubs with partition keys | Kinesis Data Streams with partition keys | Both preserve order per partition. Kinesis offers on-demand mode that resizes shards for you |
| Event Hubs consumer groups | Kinesis consumers (shared throughput) or enhanced fan-out (dedicated throughput per consumer) | Enhanced fan-out has its own cost ⚠ VERIFY |
| Azure Functions Event Hubs trigger | Lambda Kinesis event source mapping | Retry, bisect, max age and partial batch response are mapping settings |
| Cosmos DB with strong consistency and RU limits | DynamoDB strongly consistent reads and conditional writes | Capacity modes differ: on-demand versus provisioned with auto scaling |
| Polly timeout and bulkhead policies | Same in .NET on AWS, plus CancellationToken deadlines | Nothing AWS-specific: the budget code in this lesson uses plain .NET |
| Azure Front Door load shedding rules | API Gateway throttling, WAF rate rules, application-level shedding | Shedding by business priority needs application logic |
Architecture
Two paths
The latency budget
Teaching example: total budget 150 ms as seen at the load balancer, with a 30 ms reserve.
| Step | Budget (ms) | Realistic p99 (ms) | Notes |
|---|---|---|---|
| Network and TLS from terminal to load balancer | outside | outside | Out of your control; keep your own part small |
| Load balancer and service routing | 5 | 3 | Connection reuse; avoid cold TLS handshakes |
| Parse and validate message | 5 | 2 | Pre-allocated parsers; no reflection-heavy serialisation |
| Account status and rule lookup (cache) | 10 | 4 | Rules are cached in memory and refreshed in the background |
| Balance reservation (DynamoDB conditional update) | 25 | 12 | Strongly consistent by nature; one round trip |
| Risk score (optional) | 40 | 25 | Skipped when less than 40 ms remain |
| Publish event to Kinesis | 15 | 10 | Asynchronous: do not block the response on it if you have an outbox (see below) |
| Build and send response | 5 | 2 | |
| Total committed | 105 | 58 | |
| Reserve | 45 | For GC pauses, retries, noisy neighbours |
The rule: the sum of budgets must leave a reserve; the sum of p99s is only a sanity check, because p99s do not add up linearly. Measure end to end with load tests at peak and p99.9 as well.
Security-boundary view
| Zone | Control |
|---|---|
| Network edge (acquirer or switch) | Private connectivity (VPN/Direct Connect/PrivateLink), mTLS, allow-listed addresses |
| Authorisation workload | Private subnets, task role for exactly the tables and stream used, KMS-encrypted data stores |
| Card data | Tokens only inside the AWS workload; PAN handling inside a separate PCI-scoped component if required ⚠ VERIFY scope with your QSA |
| Stream | Encrypted with KMS; consumer roles can read only; producer role can only put |
| Logs | No PAN, no full request bodies; correlation id and decision only |
Step-by-step implementation
Set the requirements (sections 1 to 3)
| Quality | Target (fictional) |
|---|---|
| Decision latency | p99 under 150 ms measured at the load balancer; p99.9 under 300 ms |
| Availability | 99.99% monthly for the decision path |
| Correctness | No approved amount may exceed available balance; no duplicate reservation for one authorisation id |
| Ordering | Events for one account are processed in the order they were decided |
| Peak | 2,000 decisions per second, with a 3 times surge for 5 minutes |
| Degraded mode | If risk scoring is unavailable, apply rule-based limits and approve only below a stand-in limit ⚠ VERIFY policy |
Reserve funds with one conditional update (section 9)
The balance item holds available and version. A reservation is a conditional UpdateItem: subtract the amount only if available >= :amount. This is atomic: two concurrent requests cannot both succeed beyond the balance. To make retries safe, include the authorisation id in a transaction with an idempotency item (attribute_not_exists), as in DynamoDB for banking correctness.
Enforce the budget in code (section 12)
Create a Deadline when the request arrives. Give each downstream call a slice of the remaining time. If the remaining time is too short for an optional step (risk scoring), skip it and note degraded = true. The "Complete code" section has a full implementation.
Publish decisions to the stream (section 10)
Write each decision as an event with the account id as the partition key, a monotonically increasing per-account sequence (from the balance item's version) and the authorisation id. If publishing to Kinesis fails, the decision still stands (money is reserved); write the event to an outbox and retry, or use DynamoDB Streams as the source of events so the database commit is the single source of truth.
Consume with ordering, idempotency and bounded retries (section 10 and 11)
A Lambda consumer processes records for one shard in order. Apply the sequence guard: the next expected sequence applies; an old one is a duplicate and is skipped; a gap is a signal to wait or to alert. Configure the event source mapping with bounded retries and a maximum record age so one poison record cannot stall the shard indefinitely, and send failed records to a dead-letter destination.
Add back-pressure (section 11 and 12)
Three layers: (1) admission control at the service: a concurrency limit and a short bounded queue; beyond it return 429 or a decline with a retry hint; (2) load shedding by priority: drop low-value work first (for example balance-enquiry refreshes, marketing events) to protect authorisations; (3) stream-side limits: alarm on IteratorAge and scale consumers (parallelisation factor, shards) before the backlog exceeds your freshness target.
Rehearse failure (section 11)
Use chaos experiments in a non-production environment to add latency to the risk scorer and to DynamoDB, and prove the degraded-mode decision and the alarms.
AWS Console instructions
- Console → Kinesis → Data streams → Create data stream. Name
hl-m14-authorisations, capacity mode On-demand. - Stream → Configuration → Encryption: turn on server-side encryption with the AWS managed key or your own key; add tags
Project=cea-lab,Lab=m14-l6,Environment=learn,Owner=<you>. - Stream → Monitoring: look at
WriteProvisionedThroughputExceeded,GetRecords.IteratorAgeMilliseconds,IncomingRecords. - Console → Lambda → Add trigger → Kinesis: set batch size 100, Report batch item failures on, Retry attempts 3, Maximum age of record 1 hour, Split batch on error on, plus an on-failure destination SQS queue.
AWS CLI
# Create an on-demand stream (billed while it exists: delete it within the hour).
aws kinesis create-stream --stream-name hl-m14-authorisations \
--stream-mode-details StreamMode=ON_DEMAND --profile learn-sandbox --region us-east-1
# Tag it with the course tags.
aws kinesis add-tags-to-stream --stream-name hl-m14-authorisations \
--tags Project=cea-lab,Lab=m14-l6,Environment=learn,Owner=learner \
--profile learn-sandbox --region us-east-1
# Put a fictional decision event. The partition key is the account id, so one account maps to one ordered lane.
aws kinesis put-record --stream-name hl-m14-authorisations \
--partition-key ACC-0000000001 \
--data "$(echo -n '{"authId":"AUTH-0001","accountId":"ACC-0000000001","sequence":1,"amount":"12.50","currency":"GBP","decision":"approved"}' | base64)" \
--profile learn-sandbox --region us-east-1
# Check how far behind the slowest consumer is (iterator age is in milliseconds) after you attach one.
aws cloudwatch get-metric-statistics --namespace AWS/Kinesis --metric-name GetRecords.IteratorAgeMilliseconds \
--dimensions Name=StreamName,Value=hl-m14-authorisations --statistics Maximum --period 60 \
--start-time "$(date -u -d '-15 minutes' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
--profile learn-sandbox --region us-east-1date -u -d is GNU date (Git Bash on Windows includes it). The --data value is base64 because the CLI expects it so for blobs under v2 ⚠ VERIFY: v2 CLI treats --data as base64 by default.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.Kinesis;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.Lambda.EventSources;
using Amazon.CDK.AWS.SQS;
using Constructs;
public sealed class AuthorisationStreamStack : Stack
{
public AuthorisationStreamStack(Construct scope, string id, IStackProps props, IFunction ledgerConsumer) : base(scope, id, props)
{
var stream = new Amazon.CDK.AWS.Kinesis.Stream(this, "Authorisations", new StreamProps
{
StreamName = "hl-m14-authorisations",
StreamMode = StreamMode.ON_DEMAND,
RetentionPeriod = Duration.Hours(24),
Encryption = StreamEncryption.MANAGED,
});
var deadLetters = new Queue(this, "ConsumerDlq", new QueueProps
{
QueueName = "hl-m14-authorisations-dlq",
RetentionPeriod = Duration.Days(14),
Encryption = QueueEncryption.SQS_MANAGED,
});
ledgerConsumer.AddEventSource(new KinesisEventSource(stream, new KinesisEventSourceProps
{
StartingPosition = StartingPosition.TRIM_HORIZON,
BatchSize = 100,
ParallelizationFactor = 2, // up to 2 concurrent batches per shard, still ordered per partition key
BisectBatchOnError = true,
RetryAttempts = 3,
MaxRecordAge = Duration.Hours(1),
ReportBatchItemFailures = true,
OnFailure = new SqsDlq(deadLetters),
}));
}
}Complete code
Three small, dependency-free classes carry the ideas: a Deadline, a balance Reserver over DynamoDB, and a SequenceGuard.
namespace Harbourline.Authorisations;
public sealed class BudgetExceededException(string step)
: Exception($"Latency budget exhausted before step '{step}'.");
/// <summary>A request's total time budget. Create one per request and slice it for each downstream call.</summary>
public sealed class Deadline(TimeProvider clock, TimeSpan total)
{
private readonly long _start = clock.GetTimestamp();
public TimeSpan Remaining => total - clock.GetElapsedTime(_start);
public bool HasAtLeast(TimeSpan needed) => Remaining >= needed;
/// <summary>
/// A cancellation source that fires after the smaller of the step's own cap and the time still left.
/// Dispose it when the call finishes.
/// </summary>
public CancellationTokenSource Slice(string step, TimeSpan cap, CancellationToken outer = default)
{
var remaining = Remaining;
if (remaining <= TimeSpan.Zero)
{
throw new BudgetExceededException(step);
}
var cts = CancellationTokenSource.CreateLinkedTokenSource(outer);
cts.CancelAfter(remaining < cap ? remaining : cap);
return cts;
}
}using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
namespace Harbourline.Authorisations;
public enum ReservationResult { Reserved, InsufficientFunds, AlreadyReserved }
public sealed class BalanceReserver(IAmazonDynamoDB dynamo, string table)
{
/// <summary>
/// Atomically reserves funds and records the authorisation id so a retry cannot reserve twice.
/// Both writes succeed or neither does.
/// </summary>
public async Task<ReservationResult> ReserveAsync(
string accountId, string authId, decimal amount, CancellationToken ct)
{
var amountText = amount.ToString("F2", System.Globalization.CultureInfo.InvariantCulture);
try
{
await dynamo.TransactWriteItemsAsync(new TransactWriteItemsRequest
{
TransactItems =
[
new TransactWriteItem
{
// One item per authorisation: its existence IS the idempotency check.
Put = new Put
{
TableName = table,
Item = new Dictionary<string, AttributeValue>
{
["pk"] = new() { S = $"AUTH#{authId}" },
["sk"] = new() { S = "RESERVATION" },
["accountId"] = new() { S = accountId },
["amount"] = new() { N = amountText },
},
ConditionExpression = "attribute_not_exists(pk)",
},
},
new TransactWriteItem
{
Update = new Update
{
TableName = table,
Key = new Dictionary<string, AttributeValue>
{
["pk"] = new() { S = $"ACCOUNT#{accountId}" },
["sk"] = new() { S = "BALANCE" },
},
UpdateExpression = "SET available = available - :amt, version = version + :one",
ConditionExpression = "available >= :amt",
ExpressionAttributeValues = new Dictionary<string, AttributeValue>
{
[":amt"] = new() { N = amountText },
[":one"] = new() { N = "1" },
},
},
},
],
}, ct);
return ReservationResult.Reserved;
}
catch (TransactionCanceledException ex)
{
// CancellationReasons are in the same order as TransactItems.
var reasons = ex.CancellationReasons ?? [];
if (reasons.Count > 0 && reasons[0].Code == "ConditionalCheckFailed")
{
return ReservationResult.AlreadyReserved;
}
if (reasons.Count > 1 && reasons[1].Code == "ConditionalCheckFailed")
{
return ReservationResult.InsufficientFunds;
}
throw;
}
}
}namespace Harbourline.Authorisations;
public enum SequenceVerdict { Apply, Duplicate, Gap }
public static class SequenceGuard
{
/// <summary>
/// Events for one account carry sequence numbers 1, 2, 3... A consumer remembers the last one it applied.
/// </summary>
public static SequenceVerdict Check(long lastApplied, long incoming) =>
incoming == lastApplied + 1 ? SequenceVerdict.Apply
: incoming <= lastApplied ? SequenceVerdict.Duplicate
: SequenceVerdict.Gap;
}Code explained line by line
Deadlinecaptures a start timestamp from the injectedTimeProvider.Remainingis the total minus the elapsed time, so every step sees the truth about what is left, not a fresh timeout.Slicecreates a linkedCancellationTokenSourcethat cancels after the smaller of the step's own cap and the remaining time. If nothing remains it throwsBudgetExceededException, which the caller turns into a degraded or declined response.HasAtLeastlets the caller skip an optional step. For example:if (deadline.HasAtLeast(TimeSpan.FromMilliseconds(40))) score = await risk.ScoreAsync(...).BalanceReserverbuilds one DynamoDB transaction with two actions. ThePutofAUTH#<id>fails if the item exists, which makes the reservation idempotent. TheUpdatesubtracts the amount only ifavailable >= :amt.- Both actions succeed or neither does, so a crash cannot leave "reserved but not recorded".
TransactionCanceledException.CancellationReasonsis a list in the same order as the actions. Reason 0 failing means the authorisation was already seen; reason 1 failing means the balance was too low. Anything else is rethrown (throttling, validation).- Amounts are formatted with two decimals in the invariant culture and sent as DynamoDB numbers. Never use
doublein the money path. SequenceGuard.CheckreturnsApplyonly for the next number. A smaller or equal number is a duplicate (safe to skip), and a larger number means something was missed and needs attention. TheGaphandling is a design choice: wait briefly, then alert and reconcile.
Security considerations
⚠ Learning shortcut — not for production
The reservation code trusts that available and version exist on the balance item and does not separate the balance table by account type, set per-item size limits, or log the decision to an immutable audit stream. In production: validate the request before any write, keep an audit event for every decision including declines, use a customer managed KMS key, restrict the task role to dynamodb:TransactWriteItems, UpdateItem and PutItem on the specific table with condition keys, and never log PANs or full messages.
- Declines must be as well-logged as approvals. Fraud analysis depends on them.
- Replay and duplicate protection: every authorisation message has a unique id; the reservation item makes it idempotent.
- Time-based attacks: do not take the decision time from the message alone; use server time and compare.
- Least privilege per consumer: the notifications Lambda can read the stream and write to SNS, but cannot touch the ledger; the ledger Lambda cannot publish notifications.
- Poison data: validate stream records strictly; a malformed record must not crash a consumer loop.
- Stand-in rules are policy: the limits you approve when risk scoring is down are a business and compliance decision ⚠ VERIFY. Record them in an ADR and in configuration the risk team can see.
Banking use case
Harbourline's card-not-present payments (fictional numbers): 600 authorisations per second on a normal evening, 1,800 during a flash sale.
- The decision path answers in about 45 ms p50 and 90 ms p99. Risk scoring is skipped when less than 40 ms remain; those decisions are flagged
degradedand their share is on a dashboard. - A customer's two quick purchases hit the same DynamoDB item; the conditional update serialises them. The second sees the reduced balance and is declined if funds are short.
- Each decision becomes a stream event keyed by account id. The ledger consumer posts in order; a reversal that arrives after its authorisation applies correctly because both share the partition key.
- During the flash sale the stream's iterator age rises to 20 seconds on the notification consumer. The team's freshness target is 30 seconds, so the alarm warns at 20 and they raise the parallelisation factor. Authorisations were never slowed, because the decision path does not wait on consumers.
Common mistakes
- Giving every call a fresh timeout → the sum exceeds the budget → propagate one deadline and slice it.
- Putting everything on the synchronous path → one slow dependency fails every decision → keep it narrow; defer the rest to the stream.
- Reading the balance, then writing → double-spend under concurrency → single conditional update.
- A global partition key → one hot shard, no ordering benefit → partition by account.
- Unbounded retries on a stream consumer → a poison record blocks the shard for 24 hours → bounded attempts, maximum age, dead-letter.
- Buffering without limit during a spike → memory grows, latency explodes → bounded queue plus shedding.
- Measuring averages → hides the tail where timeouts happen → alarm on p99 and p99.9.
- Assuming exactly-once → duplicate or reordered events surface later → idempotent consumers with sequence checks.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
ProvisionedThroughputExceededException on one account | Hot key: one account receives too many writes | Spread by design where possible (not for one balance); add a retry with jitter and a short queue; investigate abuse |
TransactionCanceledException with TransactionConflict | Concurrent transactions touch the same item | Retry with jitter within the deadline; keep transactions short |
Stream IteratorAge climbing steadily | Consumer slower than producers, or a poison record | Raise parallelisation or shards; check function errors; set maximum age |
WriteProvisionedThroughputExceeded on Kinesis | Hot partition key or insufficient shards | Use on-demand mode; fix key distribution; batch with PutRecords and retry failed entries |
| Duplicate notifications | At-least-once delivery | Deduplicate on authorisation id before sending |
| Decisions slow only after deployments | Cold starts or empty caches | Pre-warm caches, use rolling deployments with health checks, keep spare capacity |
| p99 spikes every few minutes | GC pauses or connection churn | Use Server GC settings suited to the container, reuse HttpClient and SDK clients, check allocations |
BudgetExceededException frequent | A dependency regressed | Check per-step timings; the exception names the step that ran out of time |
Cost and cleanup
Label: 🟡 credits. Kinesis bills per hour while the stream exists (on-demand has a per-stream hourly charge plus data charges ⚠ VERIFY); at a few records it costs cents per hour. DynamoDB on-demand is pay-per-request. Run the stream for under an hour or simulate the stream locally by feeding events into the consumer class in a unit test (free alternative). Real production cost is dominated by Fargate capacity kept warm for the decision path, Kinesis data volume and DynamoDB write units (a transaction costs twice a normal write ⚠ VERIFY).
# Delete the stream (and any registered consumers).
aws kinesis delete-stream --stream-name hl-m14-authorisations --enforce-consumer-deletion \
--profile learn-sandbox --region us-east-1
# Delete lab queues and tables you created.
aws sqs delete-queue --queue-url <dlq-url> --profile learn-sandbox --region us-east-1
# Confirm nothing tagged for the course remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1Exercises
- 🟢 Guided: write unit tests for
SequenceGuardand forDeadlineusingFakeTimeProvider(Microsoft.Extensions.TimeProvider.Testing): remaining time shrinks as the fake clock advances,Slicethrows when none remains, and the slice is capped by the smaller value. - 🟡 Challenge: extend the decision path so that when risk scoring is skipped, only amounts below GBP 100 are approved. Write the decision function with a
degradedflag and test the boundary values (99.99, 100.00). - 🔴 Live problem: after a marketing campaign the notification consumer's iterator age reaches 40 minutes and customers receive "payment received" messages after they have left the shop. Find the cause in the metrics you would check, apply an immediate fix, apply a lasting fix and say how you would prove that authorisations were never affected.
Assignment
Write the 15-section real-time authorisation architecture in architecture/real-time-authorisation/: the two-path diagram, the latency budget table with measurements you plan to take, the reservation design, the ordering and sequence design, the back-pressure rules, the degraded-mode policy as an ADR, a failure table (dependency slowdown, hot account, poison record, shard hot spot, deployment cold start, Availability Zone loss) and the compliance touchpoints (PCI scope, scheme rules, operational resilience, data protection).
Solutions
using Harbourline.Authorisations;
using Microsoft.Extensions.Time.Testing;
using Xunit;
public class DeadlineTests
{
[Fact]
public void Remaining_shrinks_as_time_passes()
{
var clock = new FakeTimeProvider();
var deadline = new Deadline(clock, TimeSpan.FromMilliseconds(150));
clock.Advance(TimeSpan.FromMilliseconds(60));
Assert.Equal(TimeSpan.FromMilliseconds(90), deadline.Remaining);
}
[Fact]
public void Slice_throws_when_budget_is_gone()
{
var clock = new FakeTimeProvider();
var deadline = new Deadline(clock, TimeSpan.FromMilliseconds(50));
clock.Advance(TimeSpan.FromMilliseconds(60));
Assert.Throws<BudgetExceededException>(() => deadline.Slice("risk", TimeSpan.FromMilliseconds(40)));
}
[Theory]
[InlineData(0, 1, SequenceVerdict.Apply)]
[InlineData(5, 6, SequenceVerdict.Apply)]
[InlineData(5, 5, SequenceVerdict.Duplicate)]
[InlineData(5, 3, SequenceVerdict.Duplicate)]
[InlineData(5, 8, SequenceVerdict.Gap)]
public void Sequence_guard_classifies_events(long last, long incoming, SequenceVerdict expected) =>
Assert.Equal(expected, SequenceGuard.Check(last, incoming));
}Why: a fake clock removes real waiting from the test, and the theory pins the three outcomes. The package Microsoft.Extensions.TimeProvider.Testing provides FakeTimeProvider.
public enum Decision { Approve, Decline }
public static Decision Decide(decimal amount, bool fundsReserved, decimal? riskScore, bool degraded)
{
if (!fundsReserved) return Decision.Decline;
if (degraded) return amount < 100.00m ? Decision.Approve : Decision.Decline; // stand-in limit
return riskScore is < 0.80m ? Decision.Approve : Decision.Decline;
}Boundary tests: 99.99 approves and 100.00 declines in degraded mode. Why: the stand-in limit is a conservative rule that protects the bank when intelligence is missing. The limit value is a business and risk decision, so keep it in reviewed configuration rather than hard-coded ⚠ VERIFY with risk.
Check: IteratorAge per consumer, Lambda duration and error count, throttles, the batch size, and whether a failing record is retried (error rate steady while iterator age grows suggests a poison record). Immediate: raise the parallelisation factor, bisect on error, skip the poison record to the dead-letter queue, and temporarily send low-value notifications to a lower-priority path. Lasting: bounded retries and maximum record age, a freshness alarm at half the target, load testing of the consumer at campaign volume, and a rule that notifications older than N minutes are dropped or reworded. Proof that authorisations were unaffected: decision-path p99 and error rate stayed flat during the incident, because they do not depend on consumers.
Knowledge check
Answer: Each dependency adds latency and a way to fail. The decision needs only hot, small data. Everything else (ledger, notifications, analytics) follows through an ordered stream, so slow consumers cannot slow decisions.
Answer: Passing the remaining time to each downstream call instead of a new full timeout. It prevents a chain of calls each taking its own maximum and exceeding the total budget.
Answer: A conditional update subtracts the amount only if available >= amount. DynamoDB applies it atomically to the item, so the second request sees the reduced balance and fails the condition.
Answer: Records with the same key go to the same shard and are read in order, so one account's events keep their sequence while different accounts spread across shards. A constant key would serialise everything on one shard.
Answer: The mechanism by which an overloaded component signals upstream to slow down (reject, queue with a limit, shed load) instead of buffering without limit. Without it, queues grow, latency rises and retries amplify the overload.
Interview questions
- How do you design for a 150 ms authorisation budget? Allocate the budget per step with a reserve, keep the path to hot data, propagate one deadline, skip optional steps when time is short, and measure p99 and p99.9 under load.
- How do you prevent double spending in a distributed system? A single atomic conditional write on the balance plus an idempotency record for the request, in one transaction.
- Kinesis or SQS for transaction events? Kinesis when you need ordered replayable streams with several independent consumers; SQS FIFO for simpler per-group ordering and queue semantics. Say what you give up in each.
- What do you do when a stream consumer gets a poison record? Bounded retries, bisect to isolate, maximum record age, partial batch responses and a dead-letter destination, with an alarm and a replay procedure.
- What is graceful degradation in payments? Predefined, risk-approved behaviour when a dependency is slow or down, such as stand-in limits, with the decision flagged and monitored.
Certification practice
A banking application must process transaction events in order per account and allow several independent consumers to read the same events. Which service and design fit best?
A. Amazon SNS with a standard topic B. Amazon Kinesis Data Streams using the account id as the partition key C. Amazon S3 with event notifications D. Amazon SQS standard queue
Answer: B. Kinesis preserves order per partition key and supports multiple consumers. A and D do not guarantee order. C is for object events, not ordered streams.
A Lambda function consuming a Kinesis stream repeatedly fails on one malformed record, and the stream's iterator age keeps rising. Which settings help the function make progress without losing the other records?
A. Increase the function memory only B. Configure bisect-on-error, a maximum retry count, a maximum record age and an on-failure destination, and return partial batch failures C. Switch to a larger shard count only D. Disable the event source mapping
Answer: B. These settings isolate and divert the bad record while the rest continue. A and C do not remove the poison record, and D stops processing altogether.
A payments platform's authorisation latency spikes whenever an optional risk-scoring service is slow, causing timeouts. What design change best protects the authorisation SLA?
A. Increase the timeout for the risk service B. Propagate a total deadline, skip the optional risk call when little time remains, and fall back to rule-based stand-in limits C. Make risk scoring mandatory and synchronous for every request D. Remove all fraud checks
Answer: B. Deadline propagation and graceful degradation keep the decision within budget with a controlled risk posture. A makes timeouts more likely, C increases dependency risk, and D removes a control.
Completion checklist
- I can explain the two-path design and why the decision path is narrow
- I produced a latency budget with a reserve
- I can explain a conditional reservation and an idempotency item in one transaction
- I can say how ordering is preserved per account and how duplicates are detected
- I described back-pressure and degraded mode in the architecture
- I deleted any stream I created and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- Decide on a small hot data set within a budget; stream everything else after.
- Propagate one deadline, skip optional work when time is short, and degrade by pre-approved policy.
- Reserve funds with a single conditional write plus an idempotency record.
- Ordering is per key; consumers must still handle duplicates and gaps.
- Bounded retries, bounded queues and shedding stop a spike turning into an outage.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Lab: payment saga failure scenarios (design studies 14.3 and 14.4)
Run and extend a unit-tested payment saga in C# that survives a sanctions provider outage, duplicate client retries, a scheme timeout and a partial settlement, then optionally deploy a minimal Step Functions version and destroy it the same day.
Fraud-detection integration: rules, ML scoring and human review
Integrate fraud signals into banking flows without relying on services closed to new customers: layered rules on Kinesis and Lambda, optional ML scoring through SageMaker AI, Bedrock for analyst summaries (never for the decision), a human review queue built from SQS and DynamoDB, and a feedback loop.