Skip to Content
Cloud Engineering Academy
1. Cloud Computing & AWS Fundamentals

Cloud architecture fundamentals

Design for scale and failure from day one: scalability, elasticity, availability, loose coupling, stateless services, retries with backoff and jitter, idempotency and managed services, with a runnable C# example.

beginner⏱ 1 h 30 min🟢 FreeCLF-C02SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.S3 4.1.1) · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-17.

Learning objectives

By the end of this lesson you can:

  • Explain the difference between scalability and elasticity, and between vertical and horizontal scaling.
  • Distinguish high availability, fault tolerance and disaster recovery, and calculate the downtime budget of an availability target.
  • Explain loose coupling, stateless services and "cattle, not pets" with a banking example.
  • Configure retries on an AWS SDK for .NET v4 client and write your own retry loop with exponential backoff and jitter.
  • Make a payment endpoint idempotent with an Idempotency-Key, so a retry never moves money twice.
  • Argue for managed services and automation over self-managed servers, and name their limits.

Prerequisites

Why this matters in banking

On payday morning, Harbourline Bank's mobile app sees five times its normal traffic between 06:00 and 08:00. In the on-premises world the bank bought servers for that peak, and they sat mostly idle for the rest of the month. One year, a single overloaded SQL Server node slowed the whole payments flow; the mobile app timed out, customers pressed Pay again, and the payments team spent two days reversing duplicate transfers.

Every problem in that story is an architecture problem: capacity planned for a guess, a single point of failure, tight coupling between the app and the database, and a non-idempotent payment API. The cloud does not fix these automatically. It gives you building blocks (Availability Zones, queues, auto scaling, managed databases) that make the right design cheap. This lesson gives you the vocabulary and patterns every later module assumes.

The idea in plain language

A few ideas carry most of cloud architecture:

  • Scalability: the system can handle more load by adding resources. Vertical scaling means a bigger machine (more CPU and RAM). Horizontal scaling means more machines sharing the work.
  • Elasticity: the system automatically adds resources when load rises and removes them when it falls, so you pay only for what you use.
  • High availability (HA): the service stays usable despite the failure of a component, usually by running copies in more than one Availability Zone. Short blips may occur during failover.
  • Fault tolerance: the service keeps working with no visible interruption when a component fails, usually through full redundancy. More expensive than HA.
  • Disaster recovery (DR): the plan to restore the service after a large event (a Region outage, ransomware, a bad deployment that corrupts data). Measured by RTO (how long to recover) and RPO (how much data you can lose). Module 12 covers DR in depth.
  • Loose coupling: components talk through well-defined interfaces, often queues or events, so one can fail, slow down or change without breaking the others.
  • Stateless services: a server keeps no customer session in memory or on local disk. Any server can handle any request, so you can add, remove or replace servers freely.
  • Design for failure: assume everything fails eventually. Use timeouts, retries with backoff, idempotency, health checks and redundancy.

Analogy: A bank branch on payday. Vertical scaling is hiring one super-fast cashier; when they are ill, the branch closes. Horizontal scaling is opening more tills. Elasticity is the branch manager opening tills when the queue grows and closing them when it shrinks. Loose coupling is the ticket machine: customers take a number and wait, so a slow till never blocks the door. Statelessness means any cashier can serve any ticket, because the customer's details live in the core system, not in one cashier's head. Idempotency is the cashier checking the payment slip's reference number, so a customer who hands over the same slip twice is not charged twice.

What problem cloud architecture principles solve

On-premises architecture is constrained by capacity you bought months ago. Cloud architecture is constrained by design: capacity is an API call away, but only if your application can use it.

PrincipleProblem it solvesLimitation or sharp edge
Horizontal scalingOne big server is a single point of failure and has a ceilingNeeds stateless app tiers; databases are harder to scale out
Elasticity (auto scaling)Paying for peak capacity all monthScaling takes time (seconds for Lambda, minutes for EC2); quotas still apply
Multi-AZ deploymentData-centre-level failuresCross-AZ data transfer costs money; synchronous replication adds latency
Loose coupling (queues, events)Slow or failed dependencies cascading to usersEventual consistency; you must handle duplicates and ordering
Stateless servicesLost sessions when a server dies; sticky sessions limit scalingSession state moves to a cache or database you must run well
Retries with backoff and jitterTransient network errors and throttlingUnsafe without idempotency; can amplify an outage if unbounded
IdempotencyDuplicate side effects (double payments) after retriesNeeds a durable store and a key-retention policy
Managed servicesPatching, replication, backups done by handLess control, service quotas, some lock-in, per-request pricing
Infrastructure as codeSnowflake servers nobody can rebuildLearning curve; state files and drift need discipline
CachingRepeated expensive readsStale data; cache invalidation; a cache outage can overload the database

The cost of a nine

Availability is usually expressed as a percentage of time. Each extra "nine" cuts the allowed downtime by ten, and usually costs a lot more:

AvailabilityDowntime per yearDowntime per 30-day monthTypical design at Harbourline
99%about 3 days 15.6 hoursabout 7.2 hoursSingle instance, manual recovery: internal reporting tools
99.9%about 8 hours 46 minutesabout 43 minutesTwo AZs, automated failover: staff-facing apps
99.95%about 4 hours 23 minutesabout 22 minutesMulti-AZ with health checks and auto scaling: customer web banking
99.99%about 53 minutesabout 4.3 minutesMulti-AZ, no single points of failure, automated deployment with fast rollback: card authorisation, faster payments
99.999%about 5.3 minutesabout 26 secondsMulti-Region active-active, extensive testing: rarely justified outside payment-scheme infrastructure

Two pieces of arithmetic matter:

  • Components in series multiply. An API at 99.95% that depends on a database at 99.95% has at most about 99.90% (0.9995 × 0.9995). Every hard dependency lowers your ceiling.
  • Redundant components in parallel add nines. Two independent copies at 99% each, where either can serve, give about 99.99% (1 − 0.01 × 0.01), if failures really are independent and failover works. Separate Availability Zones exist to make failures independent.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Scale up an App Service plan / bigger VM sizeChange EC2 instance type, RDS instance classUsually needs a restart; vertical scaling has a ceiling
Scale out App Service / VM Scale SetsEC2 Auto Scaling groups, ECS service auto scaling, Lambda concurrencyLambda scales per request with no servers; you control it with concurrency limits
Availability Zones and zone-redundant servicesAvailability Zones; Multi-AZ RDS; ALB across AZsMany AWS services are Regional and span AZs for you (S3, DynamoDB, SQS, Lambda)
Azure Service Bus queuesAmazon SQS (queues), Amazon SNS (pub/sub), Amazon EventBridge (event bus)SQS has no sessions by default; FIFO queues give ordering and deduplication
Polly retry policies / Microsoft.Extensions.Http.ResilienceAWS SDK built-in retries (RetryMode, MaxErrorRetry) plus your own policies for non-AWS callsThe SDK retries AWS calls for you; don't wrap SDK calls in a second retry layer without thought
ASP.NET session state in SQL Server / RedisAmazon ElastiCache, Amazon DynamoDBKeep web tiers stateless; put session state in a managed store
ARM / Bicep templatesAWS CloudFormation, AWS CDK, TerraformCDK lets you write infrastructure in C#
Azure Cache for RedisAmazon ElastiCache (Valkey, Redis OSS, Memcached), DynamoDB Accelerator, CloudFrontCloudFront caches at the edge, close to users

Architecture

Harbourline's payment initiation, designed with the principles above. Each numbered idea maps to a principle.

What each part demonstrates:

  • Stateless API in two or more AZs: if an AZ fails, the load balancer or API Gateway routes to the other copies. Auto scaling adds copies on payday.
  • Idempotency table: the API records each Idempotency-Key with a conditional write before doing any work. A retry with the same key returns the original result.
  • SQS queue: the API answers "accepted" quickly; the slow work happens in workers. A slow core banking system makes the queue longer, not the app slower.
  • Retries with backoff and jitter: workers retry transient failures from the sanctions API without overwhelming it.
  • Dead-letter queue (DLQ): messages that keep failing are set aside for investigation instead of blocking the queue forever.

Failure-domain view: a Region contains several AZs, each with independent power and networking. Putting copies in different AZs makes their failures largely independent. Putting two copies in the same AZ protects you from a server failure but not an AZ failure.

Step-by-step implementation

You will build the pieces that live in your code, because those are the ones AWS cannot do for you: SDK retry settings, a retry loop for a non-AWS dependency, and an idempotent handler.

Create the project

mkdir Harbourline.Resilience.Demo; cd Harbourline.Resilience.Demo
dotnet new console --framework net10.0
dotnet add package AWSSDK.S3 --version 4.1.1
dotnet add package AWSSDK.SSO --version 4.1.0.2
dotnet add package AWSSDK.SSOOIDC --version 4.1.0.2

The SSO packages are only needed if you run the optional --list-buckets call with your learn-sandbox profile.

Configure SDK retries explicitly

Replace Program.cs with the code in Complete code. The first block creates an AmazonS3Config with RetryMode.Standard, three retries and a 10-second request timeout. The SDK retries throttling and transient errors itself, using backoff with jitter.

Add your own retry loop for non-AWS calls

The Backoff.RetryAsync helper retries a fictional FX-rate service that times out twice. The SDK's retries only cover AWS calls; anything else (a sanctions-screening API, an internal service) needs its own policy.

Make the payment handler idempotent

PaymentHandler.Submit stores the result of the first request per Idempotency-Key. The second call with the same key returns the same PAY- id without executing again. A reused key with a different body is rejected.

Run it

dotnet run
# optional, makes one real S3 call using your SSO session:
aws sso login --profile learn-sandbox
$env:AWS_PROFILE = "learn-sandbox"
dotnet run -- --list-buckets

AWS Console instructions

This lesson's code runs locally. To see these principles in AWS services:

  • Availability Zones: Console → EC2 → Dashboard, in the service health / zones panel (⚠ VERIFY the exact panel name), lists the AZs of the Region selected in the top-right Region selector. Note that AZ names (such as us-east-1a) map to different physical zones in different accounts; the AZ ID (such as use1-az1) is the same everywhere.
  • Managed scaling: Console → Lambda → Functions → (a function) → Configuration → Concurrency shows how Lambda scales by concurrency instead of servers.
  • Dead-letter queues: Console → Simple Queue Service → Create queue → Dead-letter queue section shows the "maximum receives" setting before a message moves to the DLQ. You do not need to create one now.

AWS CLI

# Which AZs exist in your Region, with their account-independent AZ IDs?
aws ec2 describe-availability-zones --region us-east-1 --profile learn-sandbox \
  --query "AvailabilityZones[].[ZoneName, ZoneId, State]" --output table

# The SDK and CLI read the same retry settings from the environment or the config file.
# Show what the CLI will use for a profile (empty means the default: standard mode).
aws configure get retry_mode --profile learn-sandbox
aws configure get max_attempts --profile learn-sandbox

# Set them for one profile in %USERPROFILE%\.aws\config (applies to CLI and SDKs that read the file).
aws configure set retry_mode standard --profile learn-sandbox
aws configure set max_attempts 4 --profile learn-sandbox
  • describe-availability-zones is free and read-only. ZoneId lets two teams in different accounts agree which physical AZ they mean.
  • retry_mode and max_attempts are shared config-file settings understood by the AWS CLI and the SDKs, including .NET (as AWS_RETRY_MODE / AWS_MAX_ATTEMPTS environment variables or file settings). A value set in code on the client config takes precedence.
  • Careful with names: the config file's max_attempts counts all attempts including the first; the .NET property MaxErrorRetry counts retries after the first attempt.

Infrastructure as Code

Loose coupling in code: an encrypted SQS queue with a dead-letter queue. You can read this now and deploy it in Module 8. If you do deploy it, SQS's always-free allowance (1 million requests per month ⚠ VERIFY on the SQS pricing page) makes an idle queue cost $0.

PaymentQueueStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.SQS;
using Constructs;

namespace Harbourline.Infrastructure;

public sealed class PaymentQueueStack : Stack
{
    public PaymentQueueStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        // Messages that fail repeatedly land here for investigation; keep them 14 days.
        var dlq = new Queue(this, "PaymentsDlq", new QueueProps
        {
            QueueName = "hl-m01-payments-dlq",
            Encryption = QueueEncryption.SQS_MANAGED,
            EnforceSSL = true,
            RetentionPeriod = Duration.Days(14),
        });

        // Main queue: buffers payday bursts between the API and the workers.
        _ = new Queue(this, "Payments", new QueueProps
        {
            QueueName = "hl-m01-payments",
            Encryption = QueueEncryption.SQS_MANAGED,
            EnforceSSL = true,
            VisibilityTimeout = Duration.Seconds(60), // longer than a worker needs to process one payment
            DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 },
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m01-architecture");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        Amazon.CDK.Tags.Of(this).Add("Owner", "alex");
    }
}

MaxReceiveCount = 5 means a message that fails five times moves to the DLQ, so one "poison" payment cannot block the others. In production you would add an alarm on the DLQ's message count.

Complete code

The project file:

Harbourline.Resilience.Demo.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <RootNamespace>Harbourline.Resilience.Demo</RootNamespace>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="AWSSDK.S3" Version="4.1.1" />
    <PackageReference Include="AWSSDK.SSO" Version="4.1.0.2" />
    <PackageReference Include="AWSSDK.SSOOIDC" Version="4.1.0.2" />
  </ItemGroup>
</Project>
Program.cs
// Harbourline.Resilience.Demo: design-for-failure basics without extra libraries.
//  1. AWS SDK retries: configure RetryMode, MaxErrorRetry and a timeout on a client config.
//  2. Your own retries: exponential backoff with "full jitter" for a flaky downstream call.
//  3. Idempotency: a duplicate payment request with the same Idempotency-Key is not paid twice.
// Runs fully offline. Pass --list-buckets to make one real S3 call with the configured client.
using System.Collections.Concurrent;
using System.Security.Cryptography;
using System.Text;
using Amazon;
using Amazon.Runtime;
using Amazon.S3;

// ---- 1. SDK retry configuration ----------------------------------------------------------
var s3Config = new AmazonS3Config
{
    RegionEndpoint = RegionEndpoint.USEast1,
    RetryMode = RequestRetryMode.Standard, // backoff with jitter, consistent across AWS SDKs
    MaxErrorRetry = 3,                     // retries AFTER the first attempt (4 attempts in total)
    Timeout = TimeSpan.FromSeconds(10),    // per HTTP request; keep it below your caller's timeout
};
Console.WriteLine($"S3 client: RetryMode={s3Config.RetryMode}, MaxErrorRetry={s3Config.MaxErrorRetry}, Timeout={s3Config.Timeout}");

if (args.Contains("--list-buckets"))
{
    using var s3 = new AmazonS3Client(s3Config); // credentials from the default chain (AWS_PROFILE=learn-sandbox)
    var buckets = await s3.ListBucketsAsync();
    Console.WriteLine($"Buckets visible: {buckets.Buckets?.Count ?? 0}");
}

// ---- 2. Your own retry loop with exponential backoff and full jitter ----------------------
var flaky = new FlakyFxRateService(failuresBeforeSuccess: 2);
var rate = await Backoff.RetryAsync(
    operation: _ => flaky.GetGbpEurRateAsync(),
    isTransient: ex => ex is TimeoutException,
    maxAttempts: 4,
    baseDelay: TimeSpan.FromMilliseconds(100),
    maxDelay: TimeSpan.FromSeconds(2));
Console.WriteLine($"GBP->EUR rate after retries: {rate} (calls made: {flaky.Calls})");

// ---- 3. Idempotent payment handler -------------------------------------------------------
var handler = new PaymentHandler(new InMemoryIdempotencyStore());
var request = new PaymentRequest("ACC-0000012345", "Northwind Lettings Ltd", "GB33BUKB20201555555555", 1250.00m, "GBP");
var key = "7f8a1c2e-4b6d-4e0f-9a1b-2c3d4e5f6a7b"; // client-generated Idempotency-Key

var first = handler.Submit(key, request);
var retry = handler.Submit(key, request); // network timeout on the client side -> it sends the same request again
Console.WriteLine($"First:  {first.Outcome} {first.PaymentId}");
Console.WriteLine($"Retry:  {retry.Outcome} {retry.PaymentId}  (same id, money moved once: {handler.PaymentsExecuted == 1})");

var tampered = handler.Submit(key, request with { Amount = 9999.00m });
Console.WriteLine($"Reused key, different body: {tampered.Outcome}");

// ===========================================================================================

/// <summary>Exponential backoff with full jitter: sleep a random time between 0 and min(maxDelay, base * 2^attempt).</summary>
public static class Backoff
{
    public static async Task<T> RetryAsync<T>(
        Func<int, Task<T>> operation,
        Func<Exception, bool> isTransient,
        int maxAttempts,
        TimeSpan baseDelay,
        TimeSpan maxDelay,
        CancellationToken ct = default)
    {
        for (var attempt = 1; ; attempt++)
        {
            try
            {
                return await operation(attempt);
            }
            catch (Exception ex) when (attempt < maxAttempts && isTransient(ex))
            {
                var ceiling = Math.Min(maxDelay.TotalMilliseconds, baseDelay.TotalMilliseconds * Math.Pow(2, attempt - 1));
                var delay = TimeSpan.FromMilliseconds(Random.Shared.NextDouble() * ceiling);
                Console.WriteLine($"  attempt {attempt} failed ({ex.GetType().Name}); waiting {delay.TotalMilliseconds:F0} ms");
                await Task.Delay(delay, ct);
            }
        }
    }
}

/// <summary>Simulates a downstream FX-rate API that times out a few times before answering.</summary>
public sealed class FlakyFxRateService(int failuresBeforeSuccess)
{
    public int Calls { get; private set; }

    public Task<decimal> GetGbpEurRateAsync()
    {
        Calls++;
        if (Calls <= failuresBeforeSuccess) throw new TimeoutException("FX service did not answer in time");
        return Task.FromResult(1.168200m); // fictional rate, decimal(18,6)
    }
}

public sealed record PaymentRequest(string DebtorAccountId, string CreditorName, string CreditorIban, decimal Amount, string Currency);

public enum SubmitOutcome { Created, Replayed, KeyReusedWithDifferentRequest }

public sealed record SubmitResult(SubmitOutcome Outcome, string? PaymentId);

public sealed record IdempotencyRecord(string RequestHash, string PaymentId);

public interface IIdempotencyStore
{
    /// <summary>Atomically stores the record if the key is new; otherwise returns the existing record.</summary>
    IdempotencyRecord GetOrAdd(string key, Func<IdempotencyRecord> create);
}

/// <summary>Learning version. Production: a DynamoDB table with a conditional write (attribute_not_exists) and a TTL.</summary>
public sealed class InMemoryIdempotencyStore : IIdempotencyStore
{
    private readonly ConcurrentDictionary<string, Lazy<IdempotencyRecord>> _records = new();

    public IdempotencyRecord GetOrAdd(string key, Func<IdempotencyRecord> create) =>
        _records.GetOrAdd(key, _ => new Lazy<IdempotencyRecord>(create)).Value;
}

public sealed class PaymentHandler(IIdempotencyStore store)
{
    private int _executed;

    public int PaymentsExecuted => _executed;

    public SubmitResult Submit(string idempotencyKey, PaymentRequest request)
    {
        var hash = Hash(request);
        var created = false;

        var record = store.GetOrAdd(idempotencyKey, () =>
        {
            created = true;
            Interlocked.Increment(ref _executed); // the side effect (moving money) happens only here
            return new IdempotencyRecord(hash, $"PAY-{Guid.CreateVersion7()}");
        });

        if (record.RequestHash != hash) return new SubmitResult(SubmitOutcome.KeyReusedWithDifferentRequest, null);
        return new SubmitResult(created ? SubmitOutcome.Created : SubmitOutcome.Replayed, record.PaymentId);
    }

    private static string Hash(PaymentRequest r) =>
        Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(
            $"{r.DebtorAccountId}|{r.CreditorName}|{r.CreditorIban}|{r.Amount:F2}|{r.Currency}")));
}

Output from a real run on 2026-11-17 (delays and the id vary each run):

S3 client: RetryMode=Standard, MaxErrorRetry=3, Timeout=00:00:10
  attempt 1 failed (TimeoutException); waiting 40 ms
  attempt 2 failed (TimeoutException); waiting 193 ms
GBP->EUR rate after retries: 1.168200 (calls made: 3)
First:  Created PAY-01a1eb46-8b48-705d-8942-53a23873fb4b
Retry:  Replayed PAY-01a1eb46-8b48-705d-8942-53a23873fb4b  (same id, money moved once: True)
Reused key, different body: KeyReusedWithDifferentRequest

All data is fictional. The IBAN is the SWIFT registry's published UK example.

Code explained line by line

SDK retry configuration

  1. new AmazonS3Config { ... } — every service client has a config class that inherits from Amazon.Runtime.ClientConfig. Retry and timeout settings live there and are fixed once the client is created.
  2. RetryMode = RequestRetryMode.Standard — in SDK v4 the enum has two values, Standard and Adaptive. Standard uses exponential backoff with jitter and a retry quota that stops retrying when many calls are failing. Adaptive adds client-side rate limiting and is meant for special cases. Checked locally on 2026-11-17: a new AmazonS3Config in AWSSDK.S3 4.1.1 defaults to Standard with MaxErrorRetry = 2.
  3. MaxErrorRetry = 3 — the number of retries after the first attempt. Raise it for background jobs; keep it low for user-facing calls where waiting is worse than failing fast.
  4. Timeout = TimeSpan.FromSeconds(10) — the per-request HTTP timeout. The default printed by a fresh config is extremely large, so set your own in services that have a response-time budget.
  5. new AmazonS3Client(s3Config) — credentials still come from the default chain; the config only controls behaviour.

Your own retry loop

  1. Backoff.RetryAsync takes the operation, a predicate saying which exceptions are transient, a maximum attempt count and delay limits.
  2. catch (Exception ex) when (attempt < maxAttempts && isTransient(ex)) — the exception filter retries only transient errors and only while attempts remain. A validation error or the last failure propagates immediately.
  3. baseDelay * 2^(attempt-1) — exponential growth: 100 ms, 200 ms, 400 ms, capped by maxDelay.
  4. Random.Shared.NextDouble() * ceiling — full jitter: a random wait between zero and the ceiling. Without jitter, thousands of clients that failed at the same moment retry at the same moment and knock the service over again (the "thundering herd").
  5. FlakyFxRateService simulates a dependency that fails twice, so you can watch the loop work without a network.

Idempotent payment handler

  1. PaymentRequest is a record with decimal Amount and an ISO currency code: never double for money.
  2. IIdempotencyStore.GetOrAdd is the key contract: atomically store a record for a new key or return the existing one. The interface lets you swap the in-memory store for DynamoDB without changing the handler.
  3. ConcurrentDictionary<string, Lazy<IdempotencyRecord>> — GetOrAdd on its own can call the factory twice under a race; wrapping the value in Lazy guarantees the factory (and so the money movement) runs once per key.
  4. Hash(request) — a SHA-256 of the business fields. It detects a client that reuses a key for a different payment, which is a bug or an attack and must be rejected, not replayed.
  5. Inside the factory, Interlocked.Increment(ref _executed) stands in for the real side effect: posting the payment. It runs only for the first request with that key.
  6. Guid.CreateVersion7() creates a time-ordered UUID v7, matching the course's PAY- + UUID v7 convention.
  7. The return logic distinguishes Created (first time), Replayed (same key, same body: return the stored result) and KeyReusedWithDifferentRequest (reject, typically HTTP 422 with ProblemDetails in an API).

Security considerations

  • Retries can become an attack amplifier. Cap attempts and total time. Never retry authentication or authorisation failures (AccessDenied): they will not succeed and they generate noise in CloudTrail.
  • Idempotency keys are not secrets but they are identifiers. Scope them per customer (store customerId + key) so one customer cannot replay or probe another customer's payment result.
  • Do not log full request bodies for payments when recording idempotency data. Store a hash, the result id and minimal metadata. Personal data in logs extends your UK GDPR footprint.
  • Security at every layer: loose coupling and managed services do not remove the need for least-privilege IAM per component (the API may only sqs:SendMessage, the worker only sqs:ReceiveMessage and sqs:DeleteMessage), encryption on queues and tables, and private networking.
  • Caches hold data too. A cache of account balances is a copy of confidential data: encrypt it, restrict access and set short expiry times.

⚠ Learning shortcut — not for production

The idempotency store here is an in-memory dictionary. It loses everything on restart and is not shared between API copies, so it only works for one process. In production Harbourline uses a DynamoDB table: a conditional PutItem with attribute_not_exists(pk) claims the key atomically across all API instances, a TTL attribute expires keys after an agreed window (for example 24 hours), and the stored response is returned on replay. Powertools for AWS Lambda (.NET) offers an idempotency utility that implements this pattern; Module 3 and Module 8 use it.

⚠ Learning shortcut — not for production

The retry helper prints to the console and has no overall time budget or circuit breaker. A production client logs structured events with a correlation ID, enforces a total deadline (via CancellationToken), and stops calling a dependency that is clearly down (circuit breaker) so queues can absorb the backlog instead.

Banking use case

Harbourline's Payments Service (fictional) applies every principle in this lesson:

  • Elasticity for payday: the stateless Payments API runs as an ECS service on Fargate in three AZs, scaling on request count. Workers scale on the SQS queue's backlog, so a payday burst becomes a longer queue rather than timeouts.
  • Idempotency end to end: the mobile app generates an Idempotency-Key per payment attempt and reuses it on retry. The API's DynamoDB idempotency table guarantees one payment per key. Downstream, the Faster Payments submission uses the ISO 20022 EndToEndId so the scheme itself can spot duplicates.
  • Design for failure: the sanctions-screening provider is a third party with its own outages. Workers retry with backoff and jitter for two minutes, then park the payment in a screening state and raise an operational alert, instead of failing the customer's payment outright.
  • Managed over self-managed: the old design ran RabbitMQ on two VMs that the team patched monthly. SQS removed that work. The trade-off accepted by the architecture board: no message priorities, and at-least-once delivery, which the idempotent workers handle.
  • Availability target as a business decision: the business asked for "100%". The team showed the cost-of-a-nine table and agreed 99.95% for payment initiation, with the service designed so that a queued payment is never lost even when processing is delayed.
  • Automation: every component is defined in CDK (C#) and deployed through a pipeline. A failed AZ is not "repaired"; replacement tasks start automatically elsewhere ("cattle, not pets").

Whether this design meets a specific regulatory resilience requirement (for example impact tolerances under the UK operational-resilience rules) is assessed by the bank's resilience and risk teams; architecture provides the capability, not the sign-off.

Common mistakes

  • Scaling vertically until it hurts. → Bigger instances feel simpler → you hit the largest size and still have one point of failure. Design stateless tiers from the start.
  • Retrying everything. → "Retry fixes flakiness" → retrying 400 Bad Request or AccessDenied wastes time and money. Retry only transient errors (timeouts, throttling, 5xx).
  • Retrying without jitter. → Fixed delays are easier to reason about → synchronised retries cause retry storms. Use full jitter.
  • Double retry layers. → Wrapping SDK calls in Polly on top of the SDK's own retries → 3 × 4 = 12 attempts and long hangs. Configure the SDK's retries, and add your own only for non-AWS calls.
  • Non-idempotent writes behind retries. → The client times out after the server already processed the payment → the retry pays twice. Require an Idempotency-Key on every POST that moves money.
  • Calling two copies in the same AZ "highly available". → It survived a server reboot in testing → an AZ failure takes both. Spread across AZs.
  • Session state in memory. → It worked on one IIS server → with several copies users get logged out randomly. Store session state in ElastiCache or DynamoDB, or use stateless tokens.
  • Caching without expiry. → Cache hit rates look great → customers see yesterday's balance. Set TTLs that match the data's tolerance for staleness.

Troubleshooting

SymptomLikely causeFix
Calls to AWS take 20+ seconds before failingHigh MaxErrorRetry combined with long timeoutsLower retries for user-facing paths; set Timeout; pass a CancellationToken with a deadline
ThrottlingException / SlowDown errors despite retriesSustained request rate above the service quotaReduce concurrency, batch requests, request a quota increase, or consider RetryMode.Adaptive for a single-tenant batch job
Duplicate payments after a network blipEndpoint not idempotent, or idempotency store not shared between instancesUse a durable, shared store with a conditional write; scope keys per customer
KeyReusedWithDifferentRequest for a genuine retryClient changed a field (for example a timestamp) inside the hashed bodyHash only business fields, and make the client resend the identical body
Messages stuck reappearing in SQSVisibility timeout shorter than processing time, so a second worker picks the message upSet visibility timeout above worst-case processing time; make workers idempotent
Users randomly logged out after scaling outSession state stored in one instance's memoryExternalise session state or use stateless tokens
Retries continue after the app shut downNo cancellation passed to Task.DelayPass the host's CancellationToken through to the retry loop

Cost and cleanup

Cost: $0. The code runs locally. The optional --list-buckets call is one free-tier-sized S3 request. The IaC examples are for reading; if you deploy the SQS queues, an idle queue costs nothing and the first million requests per month are in the always-free tier ⚠ VERIFY.

Cleanup if you deployed the queue stack:

# CDK
cdk destroy

# or Terraform
terraform destroy

# Confirm nothing tagged for this lesson remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m01-architecture --profile learn-sandbox --query "ResourceTagMappingList[].ResourceARN"

Delete the local project folder when you are done, or keep it in your learning-journal repository.

Exercises

  • 🟢 Guided: Run the demo. Change failuresBeforeSuccess to 5 and run again. What happens, and why? Then change maxAttempts to 6 and confirm it succeeds.
  • 🟡 Challenge: Add a total time budget to Backoff.RetryAsync: if the next delay would exceed a deadline parameter, stop retrying and rethrow. Write two xUnit v3 tests: one where the call succeeds on the third attempt, one where the deadline stops retries.
  • 🔴 Live problem: The operations channel reports: "Since the sanctions provider had a 10-minute outage this morning, our payment workers keep failing even though the provider is back. CPU is at 100% on the provider side according to their status page." Explain what likely happened, what in your client design made it worse, and what you would change.

Assignment

Write a one-page design note, docs/payments-resilience.md, for a "submit payment" endpoint at Harbourline: availability target with its downtime budget, which components sit in series (and the resulting ceiling), where the idempotency key is stored and for how long, which calls are retried and with what limits, and what happens when the sanctions provider is down for an hour.

Solutions

Knowledge check

Interview questions

  • "How would you design a service for 99.99% availability on AWS?" — Remove single points of failure: stateless compute across at least two (preferably three) AZs behind a load balancer or API Gateway, Multi-AZ managed data stores, health checks and automated replacement, automated deployments with fast rollback, timeouts and retries with backoff, and load testing. Then check the series dependencies: every hard dependency must also meet the target, or be made asynchronous.
  • "What's the difference between high availability, fault tolerance and disaster recovery?" — HA keeps the service usable through component failures, possibly with brief interruptions; fault tolerance means no visible interruption at all, usually through full redundancy; DR is restoring service after a major event, measured by RTO and RPO.
  • "How do you prevent duplicate payments when clients retry?" — Client-generated idempotency keys stored atomically in a shared store (for example DynamoDB conditional writes) with a TTL, hash comparison to reject reused keys with different bodies, idempotent consumers on queues, and scheme-level identifiers such as the ISO 20022 EndToEndId.
  • "When would you not use a managed service?" — When a hard requirement cannot be met (a specific engine version or extension, data-location or key-control constraints, latency), or when per-request pricing at very high steady volume costs significantly more than well-run self-managed capacity. Even then, compare the operational and staffing cost honestly.

Certification practice

Completion checklist

  • I can explain scalability, elasticity, HA, fault tolerance and DR in one sentence each
  • I can work out the downtime budget for 99.9%, 99.95% and 99.99%
  • I ran the demo and saw retries with jitter and an idempotent replay
  • I know the default RetryMode and MaxErrorRetry in SDK for .NET v4
  • I can name three managed services that replace self-managed components in a .NET estate
  • I destroyed anything I deployed and checked the tagging API

Key takeaways

  • Design stateless, horizontally scalable tiers across Availability Zones; elasticity then follows demand and cost.
  • Every hard dependency in series lowers your availability ceiling; queues turn hard dependencies into soft ones.
  • Retries need limits, backoff and jitter, and they are only safe on idempotent operations.
  • An Idempotency-Key with an atomic, shared store is the core defence against double payments.
  • Prefer managed services and infrastructure as code: replace, don't repair.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page