Cloud architecture fundamentals
Design for scale and failure from day one: scalability, elasticity, availability, loose coupling, stateless services, retries with backoff and jitter, idempotency and managed services, with a runnable C# example.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.S3 4.1.1) · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-17.
Learning objectives
By the end of this lesson you can:
- Explain the difference between scalability and elasticity, and between vertical and horizontal scaling.
- Distinguish high availability, fault tolerance and disaster recovery, and calculate the downtime budget of an availability target.
- Explain loose coupling, stateless services and "cattle, not pets" with a banking example.
- Configure retries on an AWS SDK for .NET v4 client and write your own retry loop with exponential backoff and jitter.
- Make a payment endpoint idempotent with an
Idempotency-Key, so a retry never moves money twice. - Argue for managed services and automation over self-managed servers, and name their limits.
Prerequisites
- What cloud computing really is and AWS global infrastructure (Regions and Availability Zones)
- The Shared Responsibility Model
- Tools: .NET 10 SDK. AWS access is optional: the code runs offline.
Why this matters in banking
On payday morning, Harbourline Bank's mobile app sees five times its normal traffic between 06:00 and 08:00. In the on-premises world the bank bought servers for that peak, and they sat mostly idle for the rest of the month. One year, a single overloaded SQL Server node slowed the whole payments flow; the mobile app timed out, customers pressed Pay again, and the payments team spent two days reversing duplicate transfers.
Every problem in that story is an architecture problem: capacity planned for a guess, a single point of failure, tight coupling between the app and the database, and a non-idempotent payment API. The cloud does not fix these automatically. It gives you building blocks (Availability Zones, queues, auto scaling, managed databases) that make the right design cheap. This lesson gives you the vocabulary and patterns every later module assumes.
The idea in plain language
A few ideas carry most of cloud architecture:
- Scalability: the system can handle more load by adding resources. Vertical scaling means a bigger machine (more CPU and RAM). Horizontal scaling means more machines sharing the work.
- Elasticity: the system automatically adds resources when load rises and removes them when it falls, so you pay only for what you use.
- High availability (HA): the service stays usable despite the failure of a component, usually by running copies in more than one Availability Zone. Short blips may occur during failover.
- Fault tolerance: the service keeps working with no visible interruption when a component fails, usually through full redundancy. More expensive than HA.
- Disaster recovery (DR): the plan to restore the service after a large event (a Region outage, ransomware, a bad deployment that corrupts data). Measured by RTO (how long to recover) and RPO (how much data you can lose). Module 12 covers DR in depth.
- Loose coupling: components talk through well-defined interfaces, often queues or events, so one can fail, slow down or change without breaking the others.
- Stateless services: a server keeps no customer session in memory or on local disk. Any server can handle any request, so you can add, remove or replace servers freely.
- Design for failure: assume everything fails eventually. Use timeouts, retries with backoff, idempotency, health checks and redundancy.
Analogy: A bank branch on payday. Vertical scaling is hiring one super-fast cashier; when they are ill, the branch closes. Horizontal scaling is opening more tills. Elasticity is the branch manager opening tills when the queue grows and closing them when it shrinks. Loose coupling is the ticket machine: customers take a number and wait, so a slow till never blocks the door. Statelessness means any cashier can serve any ticket, because the customer's details live in the core system, not in one cashier's head. Idempotency is the cashier checking the payment slip's reference number, so a customer who hands over the same slip twice is not charged twice.
What problem cloud architecture principles solve
On-premises architecture is constrained by capacity you bought months ago. Cloud architecture is constrained by design: capacity is an API call away, but only if your application can use it.
| Principle | Problem it solves | Limitation or sharp edge |
|---|---|---|
| Horizontal scaling | One big server is a single point of failure and has a ceiling | Needs stateless app tiers; databases are harder to scale out |
| Elasticity (auto scaling) | Paying for peak capacity all month | Scaling takes time (seconds for Lambda, minutes for EC2); quotas still apply |
| Multi-AZ deployment | Data-centre-level failures | Cross-AZ data transfer costs money; synchronous replication adds latency |
| Loose coupling (queues, events) | Slow or failed dependencies cascading to users | Eventual consistency; you must handle duplicates and ordering |
| Stateless services | Lost sessions when a server dies; sticky sessions limit scaling | Session state moves to a cache or database you must run well |
| Retries with backoff and jitter | Transient network errors and throttling | Unsafe without idempotency; can amplify an outage if unbounded |
| Idempotency | Duplicate side effects (double payments) after retries | Needs a durable store and a key-retention policy |
| Managed services | Patching, replication, backups done by hand | Less control, service quotas, some lock-in, per-request pricing |
| Infrastructure as code | Snowflake servers nobody can rebuild | Learning curve; state files and drift need discipline |
| Caching | Repeated expensive reads | Stale data; cache invalidation; a cache outage can overload the database |
The cost of a nine
Availability is usually expressed as a percentage of time. Each extra "nine" cuts the allowed downtime by ten, and usually costs a lot more:
| Availability | Downtime per year | Downtime per 30-day month | Typical design at Harbourline |
|---|---|---|---|
| 99% | about 3 days 15.6 hours | about 7.2 hours | Single instance, manual recovery: internal reporting tools |
| 99.9% | about 8 hours 46 minutes | about 43 minutes | Two AZs, automated failover: staff-facing apps |
| 99.95% | about 4 hours 23 minutes | about 22 minutes | Multi-AZ with health checks and auto scaling: customer web banking |
| 99.99% | about 53 minutes | about 4.3 minutes | Multi-AZ, no single points of failure, automated deployment with fast rollback: card authorisation, faster payments |
| 99.999% | about 5.3 minutes | about 26 seconds | Multi-Region active-active, extensive testing: rarely justified outside payment-scheme infrastructure |
Two pieces of arithmetic matter:
- Components in series multiply. An API at 99.95% that depends on a database at 99.95% has at most about 99.90% (0.9995 × 0.9995). Every hard dependency lowers your ceiling.
- Redundant components in parallel add nines. Two independent copies at 99% each, where either can serve, give about 99.99% (1 − 0.01 × 0.01), if failures really are independent and failover works. Separate Availability Zones exist to make failures independent.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Scale up an App Service plan / bigger VM size | Change EC2 instance type, RDS instance class | Usually needs a restart; vertical scaling has a ceiling |
| Scale out App Service / VM Scale Sets | EC2 Auto Scaling groups, ECS service auto scaling, Lambda concurrency | Lambda scales per request with no servers; you control it with concurrency limits |
| Availability Zones and zone-redundant services | Availability Zones; Multi-AZ RDS; ALB across AZs | Many AWS services are Regional and span AZs for you (S3, DynamoDB, SQS, Lambda) |
| Azure Service Bus queues | Amazon SQS (queues), Amazon SNS (pub/sub), Amazon EventBridge (event bus) | SQS has no sessions by default; FIFO queues give ordering and deduplication |
Polly retry policies / Microsoft.Extensions.Http.Resilience | AWS SDK built-in retries (RetryMode, MaxErrorRetry) plus your own policies for non-AWS calls | The SDK retries AWS calls for you; don't wrap SDK calls in a second retry layer without thought |
| ASP.NET session state in SQL Server / Redis | Amazon ElastiCache, Amazon DynamoDB | Keep web tiers stateless; put session state in a managed store |
| ARM / Bicep templates | AWS CloudFormation, AWS CDK, Terraform | CDK lets you write infrastructure in C# |
| Azure Cache for Redis | Amazon ElastiCache (Valkey, Redis OSS, Memcached), DynamoDB Accelerator, CloudFront | CloudFront caches at the edge, close to users |
Architecture
Harbourline's payment initiation, designed with the principles above. Each numbered idea maps to a principle.
What each part demonstrates:
- Stateless API in two or more AZs: if an AZ fails, the load balancer or API Gateway routes to the other copies. Auto scaling adds copies on payday.
- Idempotency table: the API records each
Idempotency-Keywith a conditional write before doing any work. A retry with the same key returns the original result. - SQS queue: the API answers "accepted" quickly; the slow work happens in workers. A slow core banking system makes the queue longer, not the app slower.
- Retries with backoff and jitter: workers retry transient failures from the sanctions API without overwhelming it.
- Dead-letter queue (DLQ): messages that keep failing are set aside for investigation instead of blocking the queue forever.
Failure-domain view: a Region contains several AZs, each with independent power and networking. Putting copies in different AZs makes their failures largely independent. Putting two copies in the same AZ protects you from a server failure but not an AZ failure.
Step-by-step implementation
You will build the pieces that live in your code, because those are the ones AWS cannot do for you: SDK retry settings, a retry loop for a non-AWS dependency, and an idempotent handler.
Create the project
mkdir Harbourline.Resilience.Demo; cd Harbourline.Resilience.Demo
dotnet new console --framework net10.0
dotnet add package AWSSDK.S3 --version 4.1.1
dotnet add package AWSSDK.SSO --version 4.1.0.2
dotnet add package AWSSDK.SSOOIDC --version 4.1.0.2The SSO packages are only needed if you run the optional --list-buckets call with your learn-sandbox profile.
Configure SDK retries explicitly
Replace Program.cs with the code in Complete code. The first block creates an AmazonS3Config with RetryMode.Standard, three retries and a 10-second request timeout. The SDK retries throttling and transient errors itself, using backoff with jitter.
Add your own retry loop for non-AWS calls
The Backoff.RetryAsync helper retries a fictional FX-rate service that times out twice. The SDK's retries only cover AWS calls; anything else (a sanctions-screening API, an internal service) needs its own policy.
Make the payment handler idempotent
PaymentHandler.Submit stores the result of the first request per Idempotency-Key. The second call with the same key returns the same PAY- id without executing again. A reused key with a different body is rejected.
Run it
dotnet run
# optional, makes one real S3 call using your SSO session:
aws sso login --profile learn-sandbox
$env:AWS_PROFILE = "learn-sandbox"
dotnet run -- --list-bucketsAWS Console instructions
This lesson's code runs locally. To see these principles in AWS services:
- Availability Zones: Console → EC2 → Dashboard, in the service health / zones panel (⚠ VERIFY the exact panel name), lists the AZs of the Region selected in the top-right Region selector. Note that AZ names (such as
us-east-1a) map to different physical zones in different accounts; the AZ ID (such asuse1-az1) is the same everywhere. - Managed scaling: Console → Lambda → Functions → (a function) → Configuration → Concurrency shows how Lambda scales by concurrency instead of servers.
- Dead-letter queues: Console → Simple Queue Service → Create queue → Dead-letter queue section shows the "maximum receives" setting before a message moves to the DLQ. You do not need to create one now.
AWS CLI
# Which AZs exist in your Region, with their account-independent AZ IDs?
aws ec2 describe-availability-zones --region us-east-1 --profile learn-sandbox \
--query "AvailabilityZones[].[ZoneName, ZoneId, State]" --output table
# The SDK and CLI read the same retry settings from the environment or the config file.
# Show what the CLI will use for a profile (empty means the default: standard mode).
aws configure get retry_mode --profile learn-sandbox
aws configure get max_attempts --profile learn-sandbox
# Set them for one profile in %USERPROFILE%\.aws\config (applies to CLI and SDKs that read the file).
aws configure set retry_mode standard --profile learn-sandbox
aws configure set max_attempts 4 --profile learn-sandboxdescribe-availability-zonesis free and read-only.ZoneIdlets two teams in different accounts agree which physical AZ they mean.retry_modeandmax_attemptsare shared config-file settings understood by the AWS CLI and the SDKs, including .NET (asAWS_RETRY_MODE/AWS_MAX_ATTEMPTSenvironment variables or file settings). A value set in code on the client config takes precedence.- Careful with names: the config file's
max_attemptscounts all attempts including the first; the .NET propertyMaxErrorRetrycounts retries after the first attempt.
Infrastructure as Code
Loose coupling in code: an encrypted SQS queue with a dead-letter queue. You can read this now and deploy it in Module 8. If you do deploy it, SQS's always-free allowance (1 million requests per month ⚠ VERIFY on the SQS pricing page) makes an idle queue cost $0.
using Amazon.CDK;
using Amazon.CDK.AWS.SQS;
using Constructs;
namespace Harbourline.Infrastructure;
public sealed class PaymentQueueStack : Stack
{
public PaymentQueueStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
// Messages that fail repeatedly land here for investigation; keep them 14 days.
var dlq = new Queue(this, "PaymentsDlq", new QueueProps
{
QueueName = "hl-m01-payments-dlq",
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
RetentionPeriod = Duration.Days(14),
});
// Main queue: buffers payday bursts between the API and the workers.
_ = new Queue(this, "Payments", new QueueProps
{
QueueName = "hl-m01-payments",
Encryption = QueueEncryption.SQS_MANAGED,
EnforceSSL = true,
VisibilityTimeout = Duration.Seconds(60), // longer than a worker needs to process one payment
DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 },
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m01-architecture");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "alex");
}
}MaxReceiveCount = 5 means a message that fails five times moves to the DLQ, so one "poison" payment cannot block the others. In production you would add an alarm on the DLQ's message count.
Complete code
The project file:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<RootNamespace>Harbourline.Resilience.Demo</RootNamespace>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AWSSDK.S3" Version="4.1.1" />
<PackageReference Include="AWSSDK.SSO" Version="4.1.0.2" />
<PackageReference Include="AWSSDK.SSOOIDC" Version="4.1.0.2" />
</ItemGroup>
</Project>// Harbourline.Resilience.Demo: design-for-failure basics without extra libraries.
// 1. AWS SDK retries: configure RetryMode, MaxErrorRetry and a timeout on a client config.
// 2. Your own retries: exponential backoff with "full jitter" for a flaky downstream call.
// 3. Idempotency: a duplicate payment request with the same Idempotency-Key is not paid twice.
// Runs fully offline. Pass --list-buckets to make one real S3 call with the configured client.
using System.Collections.Concurrent;
using System.Security.Cryptography;
using System.Text;
using Amazon;
using Amazon.Runtime;
using Amazon.S3;
// ---- 1. SDK retry configuration ----------------------------------------------------------
var s3Config = new AmazonS3Config
{
RegionEndpoint = RegionEndpoint.USEast1,
RetryMode = RequestRetryMode.Standard, // backoff with jitter, consistent across AWS SDKs
MaxErrorRetry = 3, // retries AFTER the first attempt (4 attempts in total)
Timeout = TimeSpan.FromSeconds(10), // per HTTP request; keep it below your caller's timeout
};
Console.WriteLine($"S3 client: RetryMode={s3Config.RetryMode}, MaxErrorRetry={s3Config.MaxErrorRetry}, Timeout={s3Config.Timeout}");
if (args.Contains("--list-buckets"))
{
using var s3 = new AmazonS3Client(s3Config); // credentials from the default chain (AWS_PROFILE=learn-sandbox)
var buckets = await s3.ListBucketsAsync();
Console.WriteLine($"Buckets visible: {buckets.Buckets?.Count ?? 0}");
}
// ---- 2. Your own retry loop with exponential backoff and full jitter ----------------------
var flaky = new FlakyFxRateService(failuresBeforeSuccess: 2);
var rate = await Backoff.RetryAsync(
operation: _ => flaky.GetGbpEurRateAsync(),
isTransient: ex => ex is TimeoutException,
maxAttempts: 4,
baseDelay: TimeSpan.FromMilliseconds(100),
maxDelay: TimeSpan.FromSeconds(2));
Console.WriteLine($"GBP->EUR rate after retries: {rate} (calls made: {flaky.Calls})");
// ---- 3. Idempotent payment handler -------------------------------------------------------
var handler = new PaymentHandler(new InMemoryIdempotencyStore());
var request = new PaymentRequest("ACC-0000012345", "Northwind Lettings Ltd", "GB33BUKB20201555555555", 1250.00m, "GBP");
var key = "7f8a1c2e-4b6d-4e0f-9a1b-2c3d4e5f6a7b"; // client-generated Idempotency-Key
var first = handler.Submit(key, request);
var retry = handler.Submit(key, request); // network timeout on the client side -> it sends the same request again
Console.WriteLine($"First: {first.Outcome} {first.PaymentId}");
Console.WriteLine($"Retry: {retry.Outcome} {retry.PaymentId} (same id, money moved once: {handler.PaymentsExecuted == 1})");
var tampered = handler.Submit(key, request with { Amount = 9999.00m });
Console.WriteLine($"Reused key, different body: {tampered.Outcome}");
// ===========================================================================================
/// <summary>Exponential backoff with full jitter: sleep a random time between 0 and min(maxDelay, base * 2^attempt).</summary>
public static class Backoff
{
public static async Task<T> RetryAsync<T>(
Func<int, Task<T>> operation,
Func<Exception, bool> isTransient,
int maxAttempts,
TimeSpan baseDelay,
TimeSpan maxDelay,
CancellationToken ct = default)
{
for (var attempt = 1; ; attempt++)
{
try
{
return await operation(attempt);
}
catch (Exception ex) when (attempt < maxAttempts && isTransient(ex))
{
var ceiling = Math.Min(maxDelay.TotalMilliseconds, baseDelay.TotalMilliseconds * Math.Pow(2, attempt - 1));
var delay = TimeSpan.FromMilliseconds(Random.Shared.NextDouble() * ceiling);
Console.WriteLine($" attempt {attempt} failed ({ex.GetType().Name}); waiting {delay.TotalMilliseconds:F0} ms");
await Task.Delay(delay, ct);
}
}
}
}
/// <summary>Simulates a downstream FX-rate API that times out a few times before answering.</summary>
public sealed class FlakyFxRateService(int failuresBeforeSuccess)
{
public int Calls { get; private set; }
public Task<decimal> GetGbpEurRateAsync()
{
Calls++;
if (Calls <= failuresBeforeSuccess) throw new TimeoutException("FX service did not answer in time");
return Task.FromResult(1.168200m); // fictional rate, decimal(18,6)
}
}
public sealed record PaymentRequest(string DebtorAccountId, string CreditorName, string CreditorIban, decimal Amount, string Currency);
public enum SubmitOutcome { Created, Replayed, KeyReusedWithDifferentRequest }
public sealed record SubmitResult(SubmitOutcome Outcome, string? PaymentId);
public sealed record IdempotencyRecord(string RequestHash, string PaymentId);
public interface IIdempotencyStore
{
/// <summary>Atomically stores the record if the key is new; otherwise returns the existing record.</summary>
IdempotencyRecord GetOrAdd(string key, Func<IdempotencyRecord> create);
}
/// <summary>Learning version. Production: a DynamoDB table with a conditional write (attribute_not_exists) and a TTL.</summary>
public sealed class InMemoryIdempotencyStore : IIdempotencyStore
{
private readonly ConcurrentDictionary<string, Lazy<IdempotencyRecord>> _records = new();
public IdempotencyRecord GetOrAdd(string key, Func<IdempotencyRecord> create) =>
_records.GetOrAdd(key, _ => new Lazy<IdempotencyRecord>(create)).Value;
}
public sealed class PaymentHandler(IIdempotencyStore store)
{
private int _executed;
public int PaymentsExecuted => _executed;
public SubmitResult Submit(string idempotencyKey, PaymentRequest request)
{
var hash = Hash(request);
var created = false;
var record = store.GetOrAdd(idempotencyKey, () =>
{
created = true;
Interlocked.Increment(ref _executed); // the side effect (moving money) happens only here
return new IdempotencyRecord(hash, $"PAY-{Guid.CreateVersion7()}");
});
if (record.RequestHash != hash) return new SubmitResult(SubmitOutcome.KeyReusedWithDifferentRequest, null);
return new SubmitResult(created ? SubmitOutcome.Created : SubmitOutcome.Replayed, record.PaymentId);
}
private static string Hash(PaymentRequest r) =>
Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(
$"{r.DebtorAccountId}|{r.CreditorName}|{r.CreditorIban}|{r.Amount:F2}|{r.Currency}")));
}Output from a real run on 2026-11-17 (delays and the id vary each run):
S3 client: RetryMode=Standard, MaxErrorRetry=3, Timeout=00:00:10
attempt 1 failed (TimeoutException); waiting 40 ms
attempt 2 failed (TimeoutException); waiting 193 ms
GBP->EUR rate after retries: 1.168200 (calls made: 3)
First: Created PAY-01a1eb46-8b48-705d-8942-53a23873fb4b
Retry: Replayed PAY-01a1eb46-8b48-705d-8942-53a23873fb4b (same id, money moved once: True)
Reused key, different body: KeyReusedWithDifferentRequestAll data is fictional. The IBAN is the SWIFT registry's published UK example.
Code explained line by line
SDK retry configuration
new AmazonS3Config { ... }— every service client has a config class that inherits fromAmazon.Runtime.ClientConfig. Retry and timeout settings live there and are fixed once the client is created.RetryMode = RequestRetryMode.Standard— in SDK v4 the enum has two values,StandardandAdaptive. Standard uses exponential backoff with jitter and a retry quota that stops retrying when many calls are failing. Adaptive adds client-side rate limiting and is meant for special cases. Checked locally on 2026-11-17: a newAmazonS3Configin AWSSDK.S3 4.1.1 defaults toStandardwithMaxErrorRetry = 2.MaxErrorRetry = 3— the number of retries after the first attempt. Raise it for background jobs; keep it low for user-facing calls where waiting is worse than failing fast.Timeout = TimeSpan.FromSeconds(10)— the per-request HTTP timeout. The default printed by a fresh config is extremely large, so set your own in services that have a response-time budget.new AmazonS3Client(s3Config)— credentials still come from the default chain; the config only controls behaviour.
Your own retry loop
Backoff.RetryAsynctakes the operation, a predicate saying which exceptions are transient, a maximum attempt count and delay limits.catch (Exception ex) when (attempt < maxAttempts && isTransient(ex))— the exception filter retries only transient errors and only while attempts remain. A validation error or the last failure propagates immediately.baseDelay * 2^(attempt-1)— exponential growth: 100 ms, 200 ms, 400 ms, capped bymaxDelay.Random.Shared.NextDouble() * ceiling— full jitter: a random wait between zero and the ceiling. Without jitter, thousands of clients that failed at the same moment retry at the same moment and knock the service over again (the "thundering herd").FlakyFxRateServicesimulates a dependency that fails twice, so you can watch the loop work without a network.
Idempotent payment handler
PaymentRequestis a record withdecimal Amountand an ISO currency code: neverdoublefor money.IIdempotencyStore.GetOrAddis the key contract: atomically store a record for a new key or return the existing one. The interface lets you swap the in-memory store for DynamoDB without changing the handler.ConcurrentDictionary<string, Lazy<IdempotencyRecord>>—GetOrAddon its own can call the factory twice under a race; wrapping the value inLazyguarantees the factory (and so the money movement) runs once per key.Hash(request)— a SHA-256 of the business fields. It detects a client that reuses a key for a different payment, which is a bug or an attack and must be rejected, not replayed.- Inside the factory,
Interlocked.Increment(ref _executed)stands in for the real side effect: posting the payment. It runs only for the first request with that key. Guid.CreateVersion7()creates a time-ordered UUID v7, matching the course'sPAY-+ UUID v7 convention.- The return logic distinguishes
Created(first time),Replayed(same key, same body: return the stored result) andKeyReusedWithDifferentRequest(reject, typically HTTP 422 with ProblemDetails in an API).
Security considerations
- Retries can become an attack amplifier. Cap attempts and total time. Never retry authentication or authorisation failures (
AccessDenied): they will not succeed and they generate noise in CloudTrail. - Idempotency keys are not secrets but they are identifiers. Scope them per customer (store
customerId + key) so one customer cannot replay or probe another customer's payment result. - Do not log full request bodies for payments when recording idempotency data. Store a hash, the result id and minimal metadata. Personal data in logs extends your UK GDPR footprint.
- Security at every layer: loose coupling and managed services do not remove the need for least-privilege IAM per component (the API may only
sqs:SendMessage, the worker onlysqs:ReceiveMessageandsqs:DeleteMessage), encryption on queues and tables, and private networking. - Caches hold data too. A cache of account balances is a copy of confidential data: encrypt it, restrict access and set short expiry times.
⚠ Learning shortcut — not for production
The idempotency store here is an in-memory dictionary. It loses everything on restart and is not shared between API copies, so it only works for one process. In production Harbourline uses a DynamoDB table: a conditional PutItem with attribute_not_exists(pk) claims the key atomically across all API instances, a TTL attribute expires keys after an agreed window (for example 24 hours), and the stored response is returned on replay. Powertools for AWS Lambda (.NET) offers an idempotency utility that implements this pattern; Module 3 and Module 8 use it.
⚠ Learning shortcut — not for production
The retry helper prints to the console and has no overall time budget or circuit breaker. A production client logs structured events with a correlation ID, enforces a total deadline (via CancellationToken), and stops calling a dependency that is clearly down (circuit breaker) so queues can absorb the backlog instead.
Banking use case
Harbourline's Payments Service (fictional) applies every principle in this lesson:
- Elasticity for payday: the stateless Payments API runs as an ECS service on Fargate in three AZs, scaling on request count. Workers scale on the SQS queue's backlog, so a payday burst becomes a longer queue rather than timeouts.
- Idempotency end to end: the mobile app generates an
Idempotency-Keyper payment attempt and reuses it on retry. The API's DynamoDB idempotency table guarantees one payment per key. Downstream, the Faster Payments submission uses the ISO 20022EndToEndIdso the scheme itself can spot duplicates. - Design for failure: the sanctions-screening provider is a third party with its own outages. Workers retry with backoff and jitter for two minutes, then park the payment in a
screeningstate and raise an operational alert, instead of failing the customer's payment outright. - Managed over self-managed: the old design ran RabbitMQ on two VMs that the team patched monthly. SQS removed that work. The trade-off accepted by the architecture board: no message priorities, and at-least-once delivery, which the idempotent workers handle.
- Availability target as a business decision: the business asked for "100%". The team showed the cost-of-a-nine table and agreed 99.95% for payment initiation, with the service designed so that a queued payment is never lost even when processing is delayed.
- Automation: every component is defined in CDK (C#) and deployed through a pipeline. A failed AZ is not "repaired"; replacement tasks start automatically elsewhere ("cattle, not pets").
Whether this design meets a specific regulatory resilience requirement (for example impact tolerances under the UK operational-resilience rules) is assessed by the bank's resilience and risk teams; architecture provides the capability, not the sign-off.
Common mistakes
- Scaling vertically until it hurts. → Bigger instances feel simpler → you hit the largest size and still have one point of failure. Design stateless tiers from the start.
- Retrying everything. → "Retry fixes flakiness" → retrying
400 Bad RequestorAccessDeniedwastes time and money. Retry only transient errors (timeouts, throttling, 5xx). - Retrying without jitter. → Fixed delays are easier to reason about → synchronised retries cause retry storms. Use full jitter.
- Double retry layers. → Wrapping SDK calls in Polly on top of the SDK's own retries → 3 × 4 = 12 attempts and long hangs. Configure the SDK's retries, and add your own only for non-AWS calls.
- Non-idempotent writes behind retries. → The client times out after the server already processed the payment → the retry pays twice. Require an
Idempotency-Keyon every POST that moves money. - Calling two copies in the same AZ "highly available". → It survived a server reboot in testing → an AZ failure takes both. Spread across AZs.
- Session state in memory. → It worked on one IIS server → with several copies users get logged out randomly. Store session state in ElastiCache or DynamoDB, or use stateless tokens.
- Caching without expiry. → Cache hit rates look great → customers see yesterday's balance. Set TTLs that match the data's tolerance for staleness.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Calls to AWS take 20+ seconds before failing | High MaxErrorRetry combined with long timeouts | Lower retries for user-facing paths; set Timeout; pass a CancellationToken with a deadline |
ThrottlingException / SlowDown errors despite retries | Sustained request rate above the service quota | Reduce concurrency, batch requests, request a quota increase, or consider RetryMode.Adaptive for a single-tenant batch job |
| Duplicate payments after a network blip | Endpoint not idempotent, or idempotency store not shared between instances | Use a durable, shared store with a conditional write; scope keys per customer |
KeyReusedWithDifferentRequest for a genuine retry | Client changed a field (for example a timestamp) inside the hashed body | Hash only business fields, and make the client resend the identical body |
| Messages stuck reappearing in SQS | Visibility timeout shorter than processing time, so a second worker picks the message up | Set visibility timeout above worst-case processing time; make workers idempotent |
| Users randomly logged out after scaling out | Session state stored in one instance's memory | Externalise session state or use stateless tokens |
| Retries continue after the app shut down | No cancellation passed to Task.Delay | Pass the host's CancellationToken through to the retry loop |
Cost and cleanup
Cost: $0. The code runs locally. The optional --list-buckets call is one free-tier-sized S3 request. The IaC examples are for reading; if you deploy the SQS queues, an idle queue costs nothing and the first million requests per month are in the always-free tier ⚠ VERIFY.
Cleanup if you deployed the queue stack:
# CDK
cdk destroy
# or Terraform
terraform destroy
# Confirm nothing tagged for this lesson remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m01-architecture --profile learn-sandbox --query "ResourceTagMappingList[].ResourceARN"Delete the local project folder when you are done, or keep it in your learning-journal repository.
Exercises
- 🟢 Guided: Run the demo. Change
failuresBeforeSuccessto 5 and run again. What happens, and why? Then changemaxAttemptsto 6 and confirm it succeeds. - 🟡 Challenge: Add a total time budget to
Backoff.RetryAsync: if the next delay would exceed adeadlineparameter, stop retrying and rethrow. Write two xUnit v3 tests: one where the call succeeds on the third attempt, one where the deadline stops retries. - 🔴 Live problem: The operations channel reports: "Since the sanctions provider had a 10-minute outage this morning, our payment workers keep failing even though the provider is back. CPU is at 100% on the provider side according to their status page." Explain what likely happened, what in your client design made it worse, and what you would change.
Assignment
Write a one-page design note, docs/payments-resilience.md, for a "submit payment" endpoint at Harbourline: availability target with its downtime budget, which components sit in series (and the resulting ceiling), where the idempotency key is stored and for how long, which calls are retried and with what limits, and what happens when the sanctions provider is down for an hour.
Solutions
With failuresBeforeSuccess: 5 and maxAttempts: 4, the fourth attempt fails, the exception filter's attempt < maxAttempts check is false, and the TimeoutException propagates and crashes the demo. Raising maxAttempts to 6 allows the sixth call to succeed. Why: bounded retries are deliberate. An unbounded loop would hide a real outage and keep load on a struggling service.
Add a TimeSpan deadline parameter, start a Stopwatch before the loop, and in the filter add && stopwatch.Elapsed + delay < deadline (calculate the delay before the filter, or move the check inside the catch block and throw; when exceeded). Tests: use a fake operation that throws TimeoutException twice then returns a value and assert the result and call count; for the deadline test, set baseDelay high (for example 500 ms) and deadline to 300 ms and assert TimeoutException is thrown after one call. Why: retries must fit inside the caller's timeout; a retry that completes after the user has given up only adds load.
Likely a retry storm: during the outage, every worker kept retrying; when the provider recovered, all the queued retries (plus new work) arrived at once, overloading it again. If retries had no jitter, they also arrived in synchronised waves. Changes: full jitter and capped exponential backoff; a total time budget per message; a circuit breaker that stops calling the provider for a period after repeated failures; limit worker concurrency against the provider (it is a shared third-party dependency); leave unprocessed work in SQS (it is a buffer, not a failure); and agree a rate limit with the provider. Why: backoff protects the dependency, jitter spreads load, and queues let you recover at a controlled pace.
Knowledge check
Answer: Scalability is the ability to handle more load by adding resources. Elasticity is doing that automatically in both directions, adding resources as load rises and removing them as it falls, so cost follows demand.
Answer: About 53 minutes per year (about 4.3 minutes per 30-day month).
Answer: About 99.8% (0.999 × 0.999 = 0.998). Hard dependencies in series multiply, so each one lowers the ceiling.
Answer: To spread retries out in time. Without jitter, clients that failed together retry together, creating waves of load that can keep a recovering service down.
Answer: The client sends a unique Idempotency-Key; the server atomically records the key with the result of the first execution and returns that stored result for any repeat. It matters because retries are only safe if repeating a request cannot repeat its side effect, such as moving money.
Answer: Standard and Adaptive. Standard is the default, with MaxErrorRetry defaulting to 2 for most services (checked with AWSSDK.S3 4.1.1).
Interview questions
- "How would you design a service for 99.99% availability on AWS?" — Remove single points of failure: stateless compute across at least two (preferably three) AZs behind a load balancer or API Gateway, Multi-AZ managed data stores, health checks and automated replacement, automated deployments with fast rollback, timeouts and retries with backoff, and load testing. Then check the series dependencies: every hard dependency must also meet the target, or be made asynchronous.
- "What's the difference between high availability, fault tolerance and disaster recovery?" — HA keeps the service usable through component failures, possibly with brief interruptions; fault tolerance means no visible interruption at all, usually through full redundancy; DR is restoring service after a major event, measured by RTO and RPO.
- "How do you prevent duplicate payments when clients retry?" — Client-generated idempotency keys stored atomically in a shared store (for example DynamoDB conditional writes) with a TTL, hash comparison to reject reused keys with different bodies, idempotent consumers on queues, and scheme-level identifiers such as the ISO 20022 EndToEndId.
- "When would you not use a managed service?" — When a hard requirement cannot be met (a specific engine version or extension, data-location or key-control constraints, latency), or when per-request pricing at very high steady volume costs significantly more than well-run self-managed capacity. Even then, compare the operational and staffing cost honestly.
Certification practice
A bank's online statement portal is busy for three days at the start of each month and quiet otherwise. Which cloud benefit lets the bank pay mainly for the busy days?
A. Global reach B. Elasticity C. High availability D. Economies of scale
Answer: B. Elasticity adds capacity when demand rises and removes it when it falls, so cost follows usage. A concerns serving users in many geographies. C concerns surviving failures. D explains why AWS's prices can be low, not why the bill follows demand.
A payments API calls a core banking system that slows down unpredictably during end-of-day processing, causing API timeouts. Payments may be processed a few minutes later, but none may be lost. Which design is most appropriate?
A. Increase the API's instance size B. Place an Amazon SQS queue between the API and workers that call the core system, with a dead-letter queue C. Add a second API instance in the same Availability Zone D. Cache core banking responses in Amazon ElastiCache
Answer: B. The queue decouples the API from the slow dependency, durably buffers payments and lets workers process at the core system's pace; the DLQ isolates messages that keep failing. A does not address the slow dependency. C adds capacity but not decoupling, and stays in one AZ. D does not help write operations such as payments.
After adding client retries, a team sees occasional duplicate fund transfers when the network is slow. What should they implement?
A. Increase the client timeout so retries happen less often B. Remove retries entirely C. Require an idempotency key on transfer requests and store processed keys in a shared table using conditional writes D. Switch the SDK to adaptive retry mode
Answer: C. Idempotency makes repeats harmless. A reduces but does not eliminate duplicates. B sacrifices resilience to transient errors. D adds client-side rate limiting but does nothing to prevent a second execution.
An ASP.NET Core application keeps user sessions in server memory. When the Auto Scaling group adds instances, users are randomly signed out. What change best supports horizontal scaling?
A. Enable sticky sessions permanently and disable scale-in B. Store session state in Amazon ElastiCache or Amazon DynamoDB C. Move to a larger instance type and stop scaling out D. Store sessions on each instance's EBS volume
Answer: B. Externalising state makes every instance interchangeable. A reduces symptoms but breaks when instances are replaced and blocks elasticity. C gives up horizontal scaling and creates a single point of failure. D still ties state to one instance.
Completion checklist
- I can explain scalability, elasticity, HA, fault tolerance and DR in one sentence each
- I can work out the downtime budget for 99.9%, 99.95% and 99.99%
- I ran the demo and saw retries with jitter and an idempotent replay
- I know the default
RetryModeandMaxErrorRetryin SDK for .NET v4 - I can name three managed services that replace self-managed components in a .NET estate
- I destroyed anything I deployed and checked the tagging API
Key takeaways
- Design stateless, horizontally scalable tiers across Availability Zones; elasticity then follows demand and cost.
- Every hard dependency in series lowers your availability ceiling; queues turn hard dependencies into soft ones.
- Retries need limits, backoff and jitter, and they are only safe on idempotent operations.
- An
Idempotency-Keywith an atomic, shared store is the core defence against double payments. - Prefer managed services and infrastructure as code: replace, don't repair.
Further reading
- AWS Well-Architected Framework — next lesson: Well-Architected introduction
- Reliability pillar and its availability section
- Retries and timeouts (AWS SDK for .NET v4) and Retry behavior (AWS SDKs and Tools reference)
- Timeouts, retries, and backoff with jitter (Amazon Builders' Library)
- Making retries safe with idempotent APIs (Amazon Builders' Library)
- Availability and Beyond: Understanding and improving the resilience of distributed systems on AWS (whitepaper)
- Disaster recovery of workloads on AWS (whitepaper)
- Deeper: Module 8 — Messaging and microservices · Module 12 — High availability and DR
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
The Shared Responsibility Model
Draw the line between what AWS secures and what Harbourline Bank still owns, for EC2, RDS, Fargate, Lambda and S3, and collect evidence for your side of the line.
Well-Architected Framework introduction
Learn the six pillars and their design principles, use lenses including the Financial Services Industry Lens, and run a free review of your module-exercise workload in the AWS Well-Architected Tool.