Skip to Content
Cloud Engineering Academy
12. High Availability & Disaster Recovery

Caching for performance and protection

Use CloudFront and ElastiCache to speed up reads and shield back ends from load spikes and partial outages, implement cache-aside with HybridCache in .NET, and apply a data-classification rule for what must never be cached.

intermediate⏱ 2 h 🟢 FreeSAA-C03SOA-C03SAP-C03

Assumes: .NET 10 LTS · Microsoft.Extensions.Caching.Hybrid · AWS CDK v2 (C#) · Terraform AWS provider 6.x · Valkey in Docker (local) · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain the two jobs of a cache: performance (lower latency) and protection (keeping load off a back end that is slow or failing).
  • Place caches at the right layer: browser, CloudFront edge, application memory, ElastiCache, and the database's own buffers.
  • Write cache-aside code with HybridCache that avoids stampedes, and add a fail-open guard so a cache outage does not become a database outage.
  • Set Cache-Control headers from a data-classification rule and configure CloudFront cache policies that never cache personal or sensitive banking data.
  • Use CloudFront origin groups to serve static content when a primary origin fails.

Prerequisites

Why this matters in banking

At 08:55 on payday, a million customers open the app to see their balance. The Accounts API calls the core banking system for each request, which handles 2,000 requests a second. Without a cache the core system collapses and every other channel (branch, call centre, payments) goes down with it. With the right caches, the product list, interest rates and branch finder are served from the edge, the exchange-rate widget from memory, and only the truly personal data reaches the back end, behind a limiter.

Caching is also where banks get hurt in the other direction. A shared cache that stores one customer's account summary under a key that does not include the customer returns it to another customer: a data breach built from a performance optimisation. This lesson teaches both halves: use caches to protect availability, and never let them break confidentiality.

The idea in plain language

A cache keeps a copy of an answer close to the asker so the next identical question is quick and does not bother the original source. Every cache trades freshness for speed and protection: the answer may be a little old.

Caches stack like the layers of a bank's information desk:

  1. The customer remembers yesterday's opening hours (browser cache).
  2. The branch has a printed leaflet (CloudFront edge cache, close to the user).
  3. The clerk remembers the common answers (in-process memory cache).
  4. A shared notice board in the back office (ElastiCache, shared by every clerk).
  5. Only then does anyone walk to the records room (the database or core system).

Analogy: The records room is the core banking system. Everyone would prefer not to queue there. But some records are confidential: a leaflet at the front desk may list savings rates, never someone's balance.

What problem each layer solves

LayerWhat it cachesSpeeds upProtectsSharp edges
Browser (Cache-Control)Static assets, public JSONPage loadEverything behind itCannot be invalidated remotely; keep TTLs short for data
CloudFrontStatic files, public API responses, error pagesLatency worldwideOrigin load, origin outages (serve stale), DDoS surfaceCache keys, invalidation cost and delay, never for personal data
In-process memory (IMemoryCache, HybridCache L1)Reference data, parsed configMicrosecondsDistributed cache and DBEach instance has its own copy; stale between instances
ElastiCache (Valkey)Session data, computed results, rate countersMillisecondsDatabaseCost per node, network hop, needs TLS, auth and a fallback plan
Database buffers / read replicasHot pages, read trafficQuery speedPrimary writerReplica lag; see Multi-AZ and replicas

The protection angle: caches and failure

A cache can hide a back-end problem for its TTL. CloudFront can even return a stale copy when the origin errors (the stale-if-error directive, and origin groups for failover). This turns a total outage into a degraded one for read-only content. But there is a catch: if your system quietly depends on the cache to survive normal load, then losing the cache (a node failure, a flush, a deployment) sends the entire load to the database at once. That is a cold-cache stampede, and it is a classic cause of cascading outages. The defences:

  • Request coalescing: only one caller recomputes a missing value while others wait (HybridCache does this per key).
  • Jittered TTLs: so a million keys do not all expire in the same second.
  • Bulkhead around the origin: a concurrency limit protecting the database when the cache is cold or down.
  • Warm-up and gradual rollout for planned flushes.
  • Capacity planning: know your database load at 0% cache hits; if it cannot survive it, say so in the risk register.

What must never be cached in a shared cache

Use a data classification and make caching depend on it:

ClassExamplesEdge (CloudFront)Shared cache (ElastiCache)Browser
PublicInterest rates, branch locator, product descriptionsYes, minutes to hoursYesYes, short
Internal referenceCurrency list, error catalogues, feature flagsNo (not public) or shortYesShort or no
PersonalAccount summary, statement list, profileNoOnly per-customer keys, short TTL, encrypted, after legal reviewprivate, short or no-store
Sensitive / secretBalances in flight, card tokens, one-time codes, auth tokens, PAN-adjacent dataNeverNever (unless the design explicitly allows it, for example server-side session with strict controls)no-store

Never cache responses that depend on the Authorization header or a cookie at a shared edge unless the cache key includes that identity, and even then prefer not to: the failure mode is a cross-customer leak. For PCI DSS scope, cache and logs that touch card data come into scope (PCI DSS on AWS); the simplest control is that card data is never cached.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Front Door / Azure CDNAmazon CloudFrontCache policy, origin request policy and response headers policy are separate, reusable objects
Azure Cache for RedisElastiCache for Valkey or Redis OSS (or MemoryDB for durable data)Run inside your VPC only; no public endpoint; TLS and auth are your job
IMemoryCache and IDistributedCacheSame interfaces with AddStackExchangeRedisCache pointed at ElastiCacheUnchanged in .NET
HybridCache in .NET 9 and laterSame, with ElastiCache as the L2Gives stampede protection and tag invalidation across layers
ASP.NET Core ResponseCache / output cachingCache-Control headers plus CloudFront policiesThe edge obeys the headers your app sends, subject to the policy
Front Door origin groups and health probesCloudFront origin groupsFailover triggers on selected HTTP status codes for safe methods

Architecture

Security-boundary view: CloudFront and the ALB are the internet-facing edge (add AWS WAF there, see WAF and Shield). The S3 origin allows only CloudFront via origin access control. ElastiCache sits in private subnets, reachable only from the API's security group, with encryption in transit and at rest and authentication enabled.

Step-by-step implementation

Classify the endpoints

List each API route and static path with its class from the table above. Mark each public, internal, personal or sensitive. This list drives every other step.

Set the HTTP caching headers in the API

Add a small policy so each endpoint declares its own Cache-Control. Personal and sensitive responses get private, no-store; public reference data gets public with s-maxage for the edge and a stale-if-error allowance.

Configure CloudFront

Static site and public reference data: cache policy with query strings in the key only where needed. API paths: the managed CachingDisabled policy and an origin request policy that forwards the headers the API needs. Add an origin group for static content failover.

Add application caching with HybridCache

Cache reference data (exchange rates, product catalogue) with a short TTL and tags. Cache personal data only if you have a design decision to do so, and then per customer key with a very short TTL.

Protect the origin

Wrap the cache-miss path in a concurrency limit (a bulkhead) and a timeout, so a cold or failed cache does not melt the database.

Test the failure of the cache

Stop the local Valkey container while load runs and confirm the API continues to answer, slower, with the bulkhead shedding excess load with 503 and Retry-After.

AWS Console instructions

  • CloudFront distribution: Console → CloudFront → Distributions → Create distribution. Choose the origin (S3 bucket with Origin access control (OAC) or the ALB DNS name), pick a cache policy (managed CachingOptimized for static assets, CachingDisabled for API paths), add behaviors for path patterns such as /api/* and /assets/*, and attach an AWS WAF web ACL.
  • Origin group: Distribution → Origins tab → Create origin group → choose primary and secondary origins and the status codes that trigger failover; then point a behavior at the origin group.
  • Invalidation: Distribution → Invalidations → Create invalidation → path such as /assets/app.*.js or /rates.json. Prefer versioned file names so you rarely need this.
  • ElastiCache: Console → ElastiCache → Valkey caches (or Redis OSS caches) → Create. Choose serverless or node-based, private subnet group, encryption in transit and at rest, and authentication. See the ElastiCache lesson for the full walkthrough.

AWS CLI

# List the managed cache policies, to find CachingOptimized and CachingDisabled IDs.
aws cloudfront list-cache-policies --type managed \
  --query "CachePolicyList.Items[].CachePolicy.{Id:Id,Name:CachePolicyConfig.Name}" \
  --profile learn-sandbox

# See distributions and their status.
aws cloudfront list-distributions \
  --query "DistributionList.Items[].{Id:Id,Domain:DomainName,Status:Status}" --profile learn-sandbox

# Invalidate a small number of paths (the first 1,000 paths a month per account are free ⚠ VERIFY).
aws cloudfront create-invalidation --distribution-id EDISTID123 \
  --paths "/rates.json" "/products/*" --profile learn-sandbox

# Check a response header from the edge: look for X-Cache and Age.
curl -sI https://d111111abcdef8.cloudfront.net/rates.json | grep -i -E "x-cache|age|cache-control"

# ElastiCache: list replication groups and check encryption settings.
aws elasticache describe-replication-groups \
  --query "ReplicationGroups[].{Id:ReplicationGroupId,TLS:TransitEncryptionEnabled,AtRest:AtRestEncryptionEnabled}" \
  --region us-east-1 --profile learn-sandbox

X-Cache: Hit from cloudfront and a rising Age header confirm an edge hit; Miss from cloudfront is a request that went to the origin; Error from cloudfront is an origin or edge error. The distribution ID and domain above are placeholders.

Infrastructure as Code

infra/cdk/EdgeStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.CloudFront;
using Amazon.CDK.AWS.CloudFront.Origins;
using Amazon.CDK.AWS.S3;
using Constructs;

namespace Harbourline.Ha.Infra;

public sealed class EdgeStack : Stack
{
    public EdgeStack(Construct scope, string id, string albDnsName, IStackProps props)
        : base(scope, id, props)
    {
        var primary = new Bucket(this, "StaticPrimary", new BucketProps
        {
            BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
            Encryption = BucketEncryption.S3_MANAGED,
            EnforceSSL = true,
            RemovalPolicy = RemovalPolicy.DESTROY,
            AutoDeleteObjects = true
        });
        // In a real design the fallback bucket lives in another Region, replicated by S3 CRR.
        var fallback = new Bucket(this, "StaticFallback", new BucketProps
        {
            BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
            Encryption = BucketEncryption.S3_MANAGED,
            EnforceSSL = true,
            RemovalPolicy = RemovalPolicy.DESTROY,
            AutoDeleteObjects = true
        });

        // S3 origins with origin access control (⚠ VERIFY the helper name in your CDK version).
        var staticGroup = new OriginGroup(new OriginGroupProps
        {
            PrimaryOrigin = S3BucketOrigin.WithOriginAccessControl(primary),
            FallbackOrigin = S3BucketOrigin.WithOriginAccessControl(fallback),
            FallbackStatusCodes = new double[] { 500, 502, 503, 504 }
        });

        var apiOrigin = new HttpOrigin(albDnsName, new HttpOriginProps
        {
            ProtocolPolicy = OriginProtocolPolicy.HTTPS_ONLY
        });

        var distribution = new Distribution(this, "Edge", new DistributionProps
        {
            DefaultBehavior = new BehaviorOptions
            {
                Origin = staticGroup,
                ViewerProtocolPolicy = ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
                CachePolicy = CachePolicy.CACHING_OPTIMIZED,
                Compress = true
            },
            AdditionalBehaviors = new Dictionary<string, IBehaviorOptions>
            {
                // Public reference data: cached. Listed first: the specific pattern must precede /api/*.
                ["/api/public/*"] = new BehaviorOptions
                {
                    Origin = apiOrigin,
                    ViewerProtocolPolicy = ViewerProtocolPolicy.HTTPS_ONLY,
                    CachePolicy = new CachePolicy(this, "PublicRefData", new CachePolicyProps
                    {
                        CachePolicyName = "hl-m12-public-ref",
                        MinTtl = Duration.Seconds(0),
                        DefaultTtl = Duration.Minutes(5),
                        MaxTtl = Duration.Hours(1),
                        // Only these query strings are part of the key; no cookies, no Authorization.
                        QueryStringBehavior = CacheQueryStringBehavior.AllowList("currency"),
                        CookieBehavior = CacheCookieBehavior.None(),
                        HeaderBehavior = CacheHeaderBehavior.None(),
                        EnableAcceptEncodingGzip = true,
                        EnableAcceptEncodingBrotli = true
                    })
                },
                // Personal and sensitive data: never cached at the edge.
                ["/api/*"] = new BehaviorOptions
                {
                    Origin = apiOrigin,
                    ViewerProtocolPolicy = ViewerProtocolPolicy.HTTPS_ONLY,
                    AllowedMethods = AllowedMethods.ALLOW_ALL,
                    CachePolicy = CachePolicy.CACHING_DISABLED,
                    OriginRequestPolicy = OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER
                }
            },
            MinimumProtocolVersion = SecurityPolicyProtocol.TLS_V1_2_2021,
            HttpVersion = HttpVersion.HTTP2_AND_3
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m12-caching");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        _ = new CfnOutput(this, "DistributionDomain", new CfnOutputProps { Value = distribution.DomainName });
    }
}

A distribution with an HTTP origin also needs WAF, logging and a certificate for your own domain; those are omitted here for focus.

Complete code

First the classification rule, as a pure function that a unit test can check exhaustively:

src/Harbourline.Ha.Caching/Cacheability.cs
namespace Harbourline.Ha.Caching;

public enum DataClass { Public, InternalReference, Personal, Sensitive }

public static class Cacheability
{
    /// <summary>The Cache-Control header value an endpoint of this class must send.</summary>
    public static string CacheControl(DataClass dataClass) => dataClass switch
    {
        // Edge may keep it 5 minutes (s-maxage), browsers 1 minute, stale allowed on errors.
        DataClass.Public => "public, max-age=60, s-maxage=300, stale-while-revalidate=30, stale-if-error=600",
        // Not for shared caches; the browser may keep it briefly.
        DataClass.InternalReference => "private, max-age=60",
        // Shared caches must not store it; browser should revalidate.
        DataClass.Personal => "private, no-cache",
        // Never store anywhere.
        DataClass.Sensitive => "no-store",
        _ => "no-store"
    };

    /// <summary>May a shared cache (CloudFront, ElastiCache) ever hold it?</summary>
    public static bool SharedCacheAllowed(DataClass dataClass) =>
        dataClass is DataClass.Public or DataClass.InternalReference;
}

Then an endpoint filter that applies it, and a cache-aside service with a bulkhead:

src/Harbourline.Ha.Caching/CachingExtensions.cs
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Http;
using Microsoft.Net.Http.Headers;

namespace Harbourline.Ha.Caching;

public static class CachingExtensions
{
    /// <summary>Sets Cache-Control (and Vary) on the response according to the data class.</summary>
    public static RouteHandlerBuilder WithDataClass(this RouteHandlerBuilder builder, DataClass dataClass) =>
        builder.AddEndpointFilter(async (ctx, next) =>
        {
            var result = await next(ctx);
            var headers = ctx.HttpContext.Response.Headers;
            headers[HeaderNames.CacheControl] = Cacheability.CacheControl(dataClass);
            if (dataClass is not DataClass.Public)
                headers[HeaderNames.Vary] = "Authorization, Cookie"; // belt and braces for shared caches
            return result;
        });
}
src/Harbourline.Ha.Caching/RateCache.cs
using Microsoft.Extensions.Caching.Hybrid;
using Microsoft.Extensions.Logging;

namespace Harbourline.Ha.Caching;

public interface IRateSource
{
    Task<decimal> GetRateAsync(string baseCurrency, string quoteCurrency, CancellationToken ct);
}

/// <summary>Cache-aside for exchange rates with stampede protection and an origin bulkhead.</summary>
public sealed class RateCache(HybridCache cache, IRateSource source, ILogger<RateCache> log)
{
    // At most 20 concurrent origin calls: protects the rate source when the cache is cold or down.
    private static readonly SemaphoreSlim OriginBulkhead = new(20, 20);

    public async Task<decimal> GetAsync(string baseCcy, string quoteCcy, CancellationToken ct)
    {
        var key = $"rate:{baseCcy}:{quoteCcy}";
        // Jitter the expiry by +/-10% so keys written together do not expire together.
        var seconds = 60 + Random.Shared.Next(-6, 7);
        var options = new HybridCacheEntryOptions
        {
            Expiration = TimeSpan.FromSeconds(seconds),         // L2 (ElastiCache)
            LocalCacheExpiration = TimeSpan.FromSeconds(15)     // L1 (in-process)
        };

        try
        {
            return await cache.GetOrCreateAsync(
                key,
                async token => await LoadFromOriginAsync(baseCcy, quoteCcy, token),
                options,
                tags: ["rates"],
                cancellationToken: ct);
        }
        catch (Exception ex) when (ex is not OperationCanceledException and not BulkheadFullException)
        {
            // Fail open: the cache layer is unhealthy, so go straight to the origin, still bulkheaded.
            log.LogWarning(ex, "Cache unavailable for {Key}; using origin", key);
            return await LoadFromOriginAsync(baseCcy, quoteCcy, ct);
        }
    }

    private async ValueTask<decimal> LoadFromOriginAsync(string b, string q, CancellationToken ct)
    {
        if (!await OriginBulkhead.WaitAsync(TimeSpan.FromMilliseconds(200), ct))
            throw new BulkheadFullException();   // shed load: the endpoint maps this to 503 + Retry-After
        try { return await source.GetRateAsync(b, q, ct); }
        finally { OriginBulkhead.Release(); }
    }

    /// <summary>Call after a rates refresh job: removes every cached rate, in all instances.</summary>
    public ValueTask InvalidateAllAsync(CancellationToken ct) => cache.RemoveByTagAsync("rates", ct);
}

public sealed class BulkheadFullException : Exception;

And the wiring:

src/Harbourline.Ha.Caching/Program.cs
using Harbourline.Ha.Caching;

var builder = WebApplication.CreateBuilder(args);

// L2: Valkey/Redis. Locally: docker run -p 6379:6379 valkey/valkey:8
// On AWS: the ElastiCache endpoint, with TLS and auth taken from configuration/Secrets Manager.
builder.Services.AddStackExchangeRedisCache(o =>
    o.Configuration = builder.Configuration.GetConnectionString("Cache") ?? "localhost:6379");
builder.Services.AddHybridCache();
builder.Services.AddSingleton<IRateSource, FakeRateSource>();
builder.Services.AddSingleton<RateCache>();

var app = builder.Build();

// Public reference data: cacheable at the edge.
app.MapGet("/api/public/rates/{baseCcy}/{quoteCcy}",
        async (string baseCcy, string quoteCcy, RateCache rates, CancellationToken ct) =>
        {
            try { return Results.Ok(new { baseCcy, quoteCcy, rate = await rates.GetAsync(baseCcy, quoteCcy, ct) }); }
            catch (BulkheadFullException)
            {
                return Results.Problem(statusCode: 503, title: "Busy, retry shortly",
                    extensions: new Dictionary<string, object?> { ["retryAfterSeconds"] = 2 });
            }
        })
    .WithDataClass(DataClass.Public);

// Personal data: never stored by shared caches.
app.MapGet("/api/accounts/{id}/summary",
        (string id) => Results.Ok(new { id, balance = 1250.40m, currency = "GBP" }))
    .WithDataClass(DataClass.Personal);

app.Run();

sealed class FakeRateSource : IRateSource
{
    public async Task<decimal> GetRateAsync(string b, string q, CancellationToken ct)
    {
        await Task.Delay(100, ct);          // pretend the origin is slow
        return b == q ? 1m : 1.17m;         // fictional rate
    }
}

Run it with docker run --rm -p 6379:6379 valkey/valkey:8 in one terminal and dotnet run in another. Package references: Microsoft.Extensions.Caching.Hybrid and Microsoft.Extensions.Caching.StackExchangeRedis (look up current versions on NuGet; the ElastiCache lesson pins tested versions).

Code explained line by line

  1. DataClass and Cacheability encode the table from earlier as code. Changing a policy becomes a one-line diff with a test, not a hunt through controllers.
  2. Public gets s-maxage=300 so the edge keeps it for 5 minutes, max-age=60 for browsers, and stale-if-error=600 so the edge can serve a stale copy for up to 10 minutes if the origin fails.
  3. Personal is private, no-cache: a shared cache must not store it, and a browser must revalidate. Sensitive is no-store: nobody keeps it.
  4. WithDataClass is an endpoint filter that runs after the handler and sets the headers. The Vary header is an extra safeguard for any intermediary that might otherwise cache personal responses.
  5. RateCache takes HybridCache through a primary constructor. GetOrCreateAsync returns the cached value or runs the factory once per key at a time (stampede protection across callers in the same process, and with L2 shared between instances).
  6. The expiry jitter spreads expirations. LocalCacheExpiration is shorter than the distributed Expiration so each instance revalidates from L2 often.
  7. The catch is the fail-open guard: when the cache layer throws (Valkey down), the code still serves from the origin, still limited by the bulkhead. It deliberately excludes cancellation and bulkhead exceptions so those are not retried.
  8. OriginBulkhead allows 20 concurrent origin calls and waits at most 200 ms for a slot. Beyond that it sheds load by throwing BulkheadFullException, which the endpoint turns into HTTP 503.
  9. RemoveByTagAsync("rates") removes all entries tagged rates so a refresh job can invalidate everything at once.
  10. /api/public/... and /api/accounts/... illustrate the two classes. The CloudFront behaviours earlier match these paths.

Security considerations

⚠ Learning shortcut — not for production

The local Valkey container runs with no password and no TLS, and the CloudFront example uses the default certificate. In production ElastiCache needs TLS, RBAC or IAM authentication, private subnets and a security group that allows only the API; CloudFront needs your own certificate, WAF, logging and a response headers policy.

  • Cross-customer leakage is the main risk. Keys for personal data must include the customer identity, and shared edge caching of authenticated responses must be off. Write a test that fails if a Personal or Sensitive endpoint ever sends public or s-maxage.
  • Cache poisoning. Keep the cache key small and controlled: only forward the query strings and headers you need, and never use the raw Host header or user-supplied headers in the key.
  • Stale data after a security change. If a customer's account is frozen, a cached Personal response could keep showing it as open. This is another reason personal data is not cached, or only for seconds.
  • Origin access. Use origin access control so S3 content is reachable only through CloudFront. Keep ALB security groups restricted to CloudFront's managed prefix list ⚠ VERIFY.
  • Encryption and logging. ElastiCache with encryption at rest and in transit; CloudFront standard logs or real-time logs to a protected bucket, with retention set.
  • Data protection. Cached personal data is still personal data under UK GDPR: it needs a retention period and must be covered by erasure and access processes. Ask your data-protection officer before caching it at all.

Banking use case

Harbourline's rule of thumb after a review by security and operations:

ContentClassWhere cached
Marketing site, app static files (versioned names)PublicCloudFront, 1 year, immutable
Savings rates, exchange rates, branch finderPublicCloudFront 5 min, HybridCache 60 s
Feature flags, bank holiday calendarInternalHybridCache only
Account summary, transaction listPersonalNot cached at the edge; origin only, with a database read replica
Card tokens, one-time codes, balances in a payment flowSensitiveNever cached
Rate limit countersOperationalElastiCache (atomic counters, see Module 6)

During an incident drill the team stops the cache and confirms the API still answers within its SLO with a smaller concurrency limit, shedding load with a clear 503 and Retry-After. That is evidence for the operational-resilience file that the cache is an optimisation and not a single point of failure.

Common mistakes

  • Caching by URL only for authenticated endpoints. Two customers calling /api/accounts/me/summary share one URL, so a shared cache gives the first customer's data to the second. Do not cache these.
  • Wrong behaviour order. CloudFront picks the first matching path pattern in order, so /api/* before /api/public/* silently disables caching for public data.
  • Invalidating on every deploy. Invalidations are slow and, beyond the free allowance, cost money. Use versioned filenames (app.4f3a.js) and long TTLs.
  • A cache with no fallback. If the application throws when ElastiCache is down, you converted a performance layer into a dependency. Fail open with a bulkhead.
  • Identical TTLs for everything. Synchronised expiry causes thundering herds; add jitter.
  • Caching errors. A cached 500 serves a failure for the whole TTL. CloudFront has a separate error caching setting; keep error TTLs short ⚠ VERIFY defaults.
  • Treating the cache as a database. ElastiCache can lose data on failover or flush. Keep the source of truth elsewhere (MemoryDB is the durable option ⚠ VERIFY suitability).

Troubleshooting

SymptomLikely causeFix
Always Miss from cloudfrontOrigin sends no-store, private or cookies in the cache keySend public, s-maxage=...; remove cookies and headers from the cache key
Users see old content after a releaseLong TTL on non-versioned file namesVersion file names, or invalidate specific paths
Different users see each other's dataShared cache stored an authenticated responseRemove caching of the path, add private, no-store, and review cache keys; treat as a security incident
403 from CloudFront to S3Bucket policy missing OAC permissionAllow the distribution through the bucket policy for the OAC
Origin group never fails overFailover status code not listed, or method not GET/HEAD/OPTIONSAdd the status codes; use origin groups only for safe methods
API slow after a cache restartCold-cache stampede on the databaseBulkhead, jitter, warm-up, scale the database temporarily
RedisConnectionException in logsWrong endpoint, TLS mismatch or security groupCheck the endpoint, ssl=true, and that the API's group is allowed
HybridCache serves stale local valuesL1 LocalCacheExpiration too longShorten it or use tag invalidation

Cost and cleanup

Cost: free for everything you run locally (Docker Valkey, .NET). The CloudFront and ElastiCache IaC are optional and not required for the exercises. If you deploy them: CloudFront bills for data transfer and requests with an always-free monthly allowance ⚠ VERIFY current numbers; ElastiCache nodes bill hourly whether used or not and are the part to delete promptly. A serverless cache has a minimum charge as well ⚠ VERIFY.

Teardown:

# Local: stop and remove the Valkey container.
docker rm -f $(docker ps -q --filter ancestor=valkey/valkey:8)

# CDK or Terraform deployment: destroy it.
cdk destroy --all --profile learn-sandbox
# or: terraform destroy

# Verify nothing tagged Project=cea-lab is left. CloudFront is a global service, so use us-east-1.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --region us-east-1 --profile learn-sandbox --query "ResourceTagMappingList[].ResourceARN"

CloudFront distributions are disabled first, then deleted, which can take 15 minutes or more.

Exercises

  • 🟢 Guided: Run the sample API with a local Valkey container. Call the rates endpoint twice and compare timings; then check the response headers with curl -i. Confirm the account summary endpoint returns private, no-cache.
  • 🟡 Challenge: Write xUnit v3 tests for Cacheability (every class), and a test with a fake IRateSource that blocks, proving the bulkhead throws BulkheadFullException when more than 20 callers are waiting. Stop the Valkey container during a load test and capture the behaviour.
  • 🔴 Live problem: After a "performance improvement" ticket, a tester reports seeing another customer's first name on the home screen, intermittently. CloudFront was set to cache /api/* for 60 seconds with the Authorization header excluded from the key. Triage: stop the leak, scope the exposure, fix the root cause, and write the incident note for the data-protection officer.

Assignment

Produce a caching standard for one Harbourline service: a table of endpoints with data class, edge policy, application cache policy and TTL; the CloudFront configuration; the fail-open behaviour; and a test plan for cache-down and cache-cold scenarios. Include the unit tests for the classification rule.

Solutions

Knowledge check

Interview questions

  • How would you decide what to cache in a banking API? Classify data first; only public and reference data go to shared caches. Personal data stays at the origin, or is cached per customer for seconds after legal sign-off. Always include a plan for cache loss.
  • What is a cache stampede and how do you prevent it? Many requests recompute the same missing value simultaneously. Use request coalescing (HybridCache), jittered expiry, a bulkhead and stale-while-revalidate.
  • CloudFront versus ElastiCache? CloudFront caches HTTP responses at the edge near users; ElastiCache caches application data inside your VPC for your own services.
  • How do you invalidate a cache across many instances? Tag-based invalidation with HybridCache and a distributed L2, short L1 TTLs, and versioned keys.

Certification practice

Completion checklist

  • I can explain the two jobs of a cache and the four data classes
  • I know which classes may be cached by CloudFront and which may never be
  • I completed the guided exercise and the cache-down test
  • I ran the cleanup steps and confirmed nothing is still billing

Key takeaways

  • A cache is a trade of freshness for speed and for protection of the back end.
  • Classify data first. Personal and sensitive data never go to a shared edge cache.
  • Design for cache loss: coalesce requests, jitter TTLs, bulkhead the origin, fail open.
  • CloudFront behaviour order matters; specific patterns go first.
  • Version file names; avoid mass invalidations.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page