Caching for performance and protection
Use CloudFront and ElastiCache to speed up reads and shield back ends from load spikes and partial outages, implement cache-aside with HybridCache in .NET, and apply a data-classification rule for what must never be cached.
Assumes: .NET 10 LTS · Microsoft.Extensions.Caching.Hybrid · AWS CDK v2 (C#) · Terraform AWS provider 6.x · Valkey in Docker (local) · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain the two jobs of a cache: performance (lower latency) and protection (keeping load off a back end that is slow or failing).
- Place caches at the right layer: browser, CloudFront edge, application memory, ElastiCache, and the database's own buffers.
- Write cache-aside code with
HybridCachethat avoids stampedes, and add a fail-open guard so a cache outage does not become a database outage. - Set
Cache-Controlheaders from a data-classification rule and configure CloudFront cache policies that never cache personal or sensitive banking data. - Use CloudFront origin groups to serve static content when a primary origin fails.
Prerequisites
- Amazon ElastiCache (Valkey/Redis OSS), where cache-aside, Lua rate limits, TLS and RBAC are covered in depth. This lesson uses it but does not repeat it.
- Performance optimisation
- Elastic Load Balancing
- Tools: .NET 10 SDK and Docker (for a local Valkey). No AWS account is needed for the code; the IaC is for reading or an optional deploy.
Why this matters in banking
At 08:55 on payday, a million customers open the app to see their balance. The Accounts API calls the core banking system for each request, which handles 2,000 requests a second. Without a cache the core system collapses and every other channel (branch, call centre, payments) goes down with it. With the right caches, the product list, interest rates and branch finder are served from the edge, the exchange-rate widget from memory, and only the truly personal data reaches the back end, behind a limiter.
Caching is also where banks get hurt in the other direction. A shared cache that stores one customer's account summary under a key that does not include the customer returns it to another customer: a data breach built from a performance optimisation. This lesson teaches both halves: use caches to protect availability, and never let them break confidentiality.
The idea in plain language
A cache keeps a copy of an answer close to the asker so the next identical question is quick and does not bother the original source. Every cache trades freshness for speed and protection: the answer may be a little old.
Caches stack like the layers of a bank's information desk:
- The customer remembers yesterday's opening hours (browser cache).
- The branch has a printed leaflet (CloudFront edge cache, close to the user).
- The clerk remembers the common answers (in-process memory cache).
- A shared notice board in the back office (ElastiCache, shared by every clerk).
- Only then does anyone walk to the records room (the database or core system).
Analogy: The records room is the core banking system. Everyone would prefer not to queue there. But some records are confidential: a leaflet at the front desk may list savings rates, never someone's balance.
What problem each layer solves
| Layer | What it caches | Speeds up | Protects | Sharp edges |
|---|---|---|---|---|
Browser (Cache-Control) | Static assets, public JSON | Page load | Everything behind it | Cannot be invalidated remotely; keep TTLs short for data |
| CloudFront | Static files, public API responses, error pages | Latency worldwide | Origin load, origin outages (serve stale), DDoS surface | Cache keys, invalidation cost and delay, never for personal data |
In-process memory (IMemoryCache, HybridCache L1) | Reference data, parsed config | Microseconds | Distributed cache and DB | Each instance has its own copy; stale between instances |
| ElastiCache (Valkey) | Session data, computed results, rate counters | Milliseconds | Database | Cost per node, network hop, needs TLS, auth and a fallback plan |
| Database buffers / read replicas | Hot pages, read traffic | Query speed | Primary writer | Replica lag; see Multi-AZ and replicas |
The protection angle: caches and failure
A cache can hide a back-end problem for its TTL. CloudFront can even return a stale copy when the origin errors (the stale-if-error directive, and origin groups for failover). This turns a total outage into a degraded one for read-only content. But there is a catch: if your system quietly depends on the cache to survive normal load, then losing the cache (a node failure, a flush, a deployment) sends the entire load to the database at once. That is a cold-cache stampede, and it is a classic cause of cascading outages. The defences:
- Request coalescing: only one caller recomputes a missing value while others wait (
HybridCachedoes this per key). - Jittered TTLs: so a million keys do not all expire in the same second.
- Bulkhead around the origin: a concurrency limit protecting the database when the cache is cold or down.
- Warm-up and gradual rollout for planned flushes.
- Capacity planning: know your database load at 0% cache hits; if it cannot survive it, say so in the risk register.
What must never be cached in a shared cache
Use a data classification and make caching depend on it:
| Class | Examples | Edge (CloudFront) | Shared cache (ElastiCache) | Browser |
|---|---|---|---|---|
| Public | Interest rates, branch locator, product descriptions | Yes, minutes to hours | Yes | Yes, short |
| Internal reference | Currency list, error catalogues, feature flags | No (not public) or short | Yes | Short or no |
| Personal | Account summary, statement list, profile | No | Only per-customer keys, short TTL, encrypted, after legal review | private, short or no-store |
| Sensitive / secret | Balances in flight, card tokens, one-time codes, auth tokens, PAN-adjacent data | Never | Never (unless the design explicitly allows it, for example server-side session with strict controls) | no-store |
Never cache responses that depend on the Authorization header or a cookie at a shared edge unless the cache key includes that identity, and even then prefer not to: the failure mode is a cross-customer leak. For PCI DSS scope, cache and logs that touch card data come into scope (PCI DSS on AWS); the simplest control is that card data is never cached.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Front Door / Azure CDN | Amazon CloudFront | Cache policy, origin request policy and response headers policy are separate, reusable objects |
| Azure Cache for Redis | ElastiCache for Valkey or Redis OSS (or MemoryDB for durable data) | Run inside your VPC only; no public endpoint; TLS and auth are your job |
IMemoryCache and IDistributedCache | Same interfaces with AddStackExchangeRedisCache pointed at ElastiCache | Unchanged in .NET |
HybridCache in .NET 9 and later | Same, with ElastiCache as the L2 | Gives stampede protection and tag invalidation across layers |
ASP.NET Core ResponseCache / output caching | Cache-Control headers plus CloudFront policies | The edge obeys the headers your app sends, subject to the policy |
| Front Door origin groups and health probes | CloudFront origin groups | Failover triggers on selected HTTP status codes for safe methods |
Architecture
Security-boundary view: CloudFront and the ALB are the internet-facing edge (add AWS WAF there, see WAF and Shield). The S3 origin allows only CloudFront via origin access control. ElastiCache sits in private subnets, reachable only from the API's security group, with encryption in transit and at rest and authentication enabled.
Step-by-step implementation
Classify the endpoints
List each API route and static path with its class from the table above. Mark each public, internal, personal or sensitive. This list drives every other step.
Set the HTTP caching headers in the API
Add a small policy so each endpoint declares its own Cache-Control. Personal and sensitive responses get private, no-store; public reference data gets public with s-maxage for the edge and a stale-if-error allowance.
Configure CloudFront
Static site and public reference data: cache policy with query strings in the key only where needed. API paths: the managed CachingDisabled policy and an origin request policy that forwards the headers the API needs. Add an origin group for static content failover.
Add application caching with HybridCache
Cache reference data (exchange rates, product catalogue) with a short TTL and tags. Cache personal data only if you have a design decision to do so, and then per customer key with a very short TTL.
Protect the origin
Wrap the cache-miss path in a concurrency limit (a bulkhead) and a timeout, so a cold or failed cache does not melt the database.
Test the failure of the cache
Stop the local Valkey container while load runs and confirm the API continues to answer, slower, with the bulkhead shedding excess load with 503 and Retry-After.
AWS Console instructions
- CloudFront distribution: Console → CloudFront → Distributions → Create distribution. Choose the origin (S3 bucket with Origin access control (OAC) or the ALB DNS name), pick a cache policy (managed
CachingOptimizedfor static assets,CachingDisabledfor API paths), add behaviors for path patterns such as/api/*and/assets/*, and attach an AWS WAF web ACL. - Origin group: Distribution → Origins tab → Create origin group → choose primary and secondary origins and the status codes that trigger failover; then point a behavior at the origin group.
- Invalidation: Distribution → Invalidations → Create invalidation → path such as
/assets/app.*.jsor/rates.json. Prefer versioned file names so you rarely need this. - ElastiCache: Console → ElastiCache → Valkey caches (or Redis OSS caches) → Create. Choose serverless or node-based, private subnet group, encryption in transit and at rest, and authentication. See the ElastiCache lesson for the full walkthrough.
AWS CLI
# List the managed cache policies, to find CachingOptimized and CachingDisabled IDs.
aws cloudfront list-cache-policies --type managed \
--query "CachePolicyList.Items[].CachePolicy.{Id:Id,Name:CachePolicyConfig.Name}" \
--profile learn-sandbox
# See distributions and their status.
aws cloudfront list-distributions \
--query "DistributionList.Items[].{Id:Id,Domain:DomainName,Status:Status}" --profile learn-sandbox
# Invalidate a small number of paths (the first 1,000 paths a month per account are free ⚠ VERIFY).
aws cloudfront create-invalidation --distribution-id EDISTID123 \
--paths "/rates.json" "/products/*" --profile learn-sandbox
# Check a response header from the edge: look for X-Cache and Age.
curl -sI https://d111111abcdef8.cloudfront.net/rates.json | grep -i -E "x-cache|age|cache-control"
# ElastiCache: list replication groups and check encryption settings.
aws elasticache describe-replication-groups \
--query "ReplicationGroups[].{Id:ReplicationGroupId,TLS:TransitEncryptionEnabled,AtRest:AtRestEncryptionEnabled}" \
--region us-east-1 --profile learn-sandboxX-Cache: Hit from cloudfront and a rising Age header confirm an edge hit; Miss from cloudfront is a request that went to the origin; Error from cloudfront is an origin or edge error. The distribution ID and domain above are placeholders.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.CloudFront;
using Amazon.CDK.AWS.CloudFront.Origins;
using Amazon.CDK.AWS.S3;
using Constructs;
namespace Harbourline.Ha.Infra;
public sealed class EdgeStack : Stack
{
public EdgeStack(Construct scope, string id, string albDnsName, IStackProps props)
: base(scope, id, props)
{
var primary = new Bucket(this, "StaticPrimary", new BucketProps
{
BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
Encryption = BucketEncryption.S3_MANAGED,
EnforceSSL = true,
RemovalPolicy = RemovalPolicy.DESTROY,
AutoDeleteObjects = true
});
// In a real design the fallback bucket lives in another Region, replicated by S3 CRR.
var fallback = new Bucket(this, "StaticFallback", new BucketProps
{
BlockPublicAccess = BlockPublicAccess.BLOCK_ALL,
Encryption = BucketEncryption.S3_MANAGED,
EnforceSSL = true,
RemovalPolicy = RemovalPolicy.DESTROY,
AutoDeleteObjects = true
});
// S3 origins with origin access control (⚠ VERIFY the helper name in your CDK version).
var staticGroup = new OriginGroup(new OriginGroupProps
{
PrimaryOrigin = S3BucketOrigin.WithOriginAccessControl(primary),
FallbackOrigin = S3BucketOrigin.WithOriginAccessControl(fallback),
FallbackStatusCodes = new double[] { 500, 502, 503, 504 }
});
var apiOrigin = new HttpOrigin(albDnsName, new HttpOriginProps
{
ProtocolPolicy = OriginProtocolPolicy.HTTPS_ONLY
});
var distribution = new Distribution(this, "Edge", new DistributionProps
{
DefaultBehavior = new BehaviorOptions
{
Origin = staticGroup,
ViewerProtocolPolicy = ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
CachePolicy = CachePolicy.CACHING_OPTIMIZED,
Compress = true
},
AdditionalBehaviors = new Dictionary<string, IBehaviorOptions>
{
// Public reference data: cached. Listed first: the specific pattern must precede /api/*.
["/api/public/*"] = new BehaviorOptions
{
Origin = apiOrigin,
ViewerProtocolPolicy = ViewerProtocolPolicy.HTTPS_ONLY,
CachePolicy = new CachePolicy(this, "PublicRefData", new CachePolicyProps
{
CachePolicyName = "hl-m12-public-ref",
MinTtl = Duration.Seconds(0),
DefaultTtl = Duration.Minutes(5),
MaxTtl = Duration.Hours(1),
// Only these query strings are part of the key; no cookies, no Authorization.
QueryStringBehavior = CacheQueryStringBehavior.AllowList("currency"),
CookieBehavior = CacheCookieBehavior.None(),
HeaderBehavior = CacheHeaderBehavior.None(),
EnableAcceptEncodingGzip = true,
EnableAcceptEncodingBrotli = true
})
},
// Personal and sensitive data: never cached at the edge.
["/api/*"] = new BehaviorOptions
{
Origin = apiOrigin,
ViewerProtocolPolicy = ViewerProtocolPolicy.HTTPS_ONLY,
AllowedMethods = AllowedMethods.ALLOW_ALL,
CachePolicy = CachePolicy.CACHING_DISABLED,
OriginRequestPolicy = OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER
}
},
MinimumProtocolVersion = SecurityPolicyProtocol.TLS_V1_2_2021,
HttpVersion = HttpVersion.HTTP2_AND_3
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m12-caching");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
_ = new CfnOutput(this, "DistributionDomain", new CfnOutputProps { Value = distribution.DomainName });
}
}A distribution with an HTTP origin also needs WAF, logging and a certificate for your own domain; those are omitted here for focus.
Complete code
First the classification rule, as a pure function that a unit test can check exhaustively:
namespace Harbourline.Ha.Caching;
public enum DataClass { Public, InternalReference, Personal, Sensitive }
public static class Cacheability
{
/// <summary>The Cache-Control header value an endpoint of this class must send.</summary>
public static string CacheControl(DataClass dataClass) => dataClass switch
{
// Edge may keep it 5 minutes (s-maxage), browsers 1 minute, stale allowed on errors.
DataClass.Public => "public, max-age=60, s-maxage=300, stale-while-revalidate=30, stale-if-error=600",
// Not for shared caches; the browser may keep it briefly.
DataClass.InternalReference => "private, max-age=60",
// Shared caches must not store it; browser should revalidate.
DataClass.Personal => "private, no-cache",
// Never store anywhere.
DataClass.Sensitive => "no-store",
_ => "no-store"
};
/// <summary>May a shared cache (CloudFront, ElastiCache) ever hold it?</summary>
public static bool SharedCacheAllowed(DataClass dataClass) =>
dataClass is DataClass.Public or DataClass.InternalReference;
}Then an endpoint filter that applies it, and a cache-aside service with a bulkhead:
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Http;
using Microsoft.Net.Http.Headers;
namespace Harbourline.Ha.Caching;
public static class CachingExtensions
{
/// <summary>Sets Cache-Control (and Vary) on the response according to the data class.</summary>
public static RouteHandlerBuilder WithDataClass(this RouteHandlerBuilder builder, DataClass dataClass) =>
builder.AddEndpointFilter(async (ctx, next) =>
{
var result = await next(ctx);
var headers = ctx.HttpContext.Response.Headers;
headers[HeaderNames.CacheControl] = Cacheability.CacheControl(dataClass);
if (dataClass is not DataClass.Public)
headers[HeaderNames.Vary] = "Authorization, Cookie"; // belt and braces for shared caches
return result;
});
}using Microsoft.Extensions.Caching.Hybrid;
using Microsoft.Extensions.Logging;
namespace Harbourline.Ha.Caching;
public interface IRateSource
{
Task<decimal> GetRateAsync(string baseCurrency, string quoteCurrency, CancellationToken ct);
}
/// <summary>Cache-aside for exchange rates with stampede protection and an origin bulkhead.</summary>
public sealed class RateCache(HybridCache cache, IRateSource source, ILogger<RateCache> log)
{
// At most 20 concurrent origin calls: protects the rate source when the cache is cold or down.
private static readonly SemaphoreSlim OriginBulkhead = new(20, 20);
public async Task<decimal> GetAsync(string baseCcy, string quoteCcy, CancellationToken ct)
{
var key = $"rate:{baseCcy}:{quoteCcy}";
// Jitter the expiry by +/-10% so keys written together do not expire together.
var seconds = 60 + Random.Shared.Next(-6, 7);
var options = new HybridCacheEntryOptions
{
Expiration = TimeSpan.FromSeconds(seconds), // L2 (ElastiCache)
LocalCacheExpiration = TimeSpan.FromSeconds(15) // L1 (in-process)
};
try
{
return await cache.GetOrCreateAsync(
key,
async token => await LoadFromOriginAsync(baseCcy, quoteCcy, token),
options,
tags: ["rates"],
cancellationToken: ct);
}
catch (Exception ex) when (ex is not OperationCanceledException and not BulkheadFullException)
{
// Fail open: the cache layer is unhealthy, so go straight to the origin, still bulkheaded.
log.LogWarning(ex, "Cache unavailable for {Key}; using origin", key);
return await LoadFromOriginAsync(baseCcy, quoteCcy, ct);
}
}
private async ValueTask<decimal> LoadFromOriginAsync(string b, string q, CancellationToken ct)
{
if (!await OriginBulkhead.WaitAsync(TimeSpan.FromMilliseconds(200), ct))
throw new BulkheadFullException(); // shed load: the endpoint maps this to 503 + Retry-After
try { return await source.GetRateAsync(b, q, ct); }
finally { OriginBulkhead.Release(); }
}
/// <summary>Call after a rates refresh job: removes every cached rate, in all instances.</summary>
public ValueTask InvalidateAllAsync(CancellationToken ct) => cache.RemoveByTagAsync("rates", ct);
}
public sealed class BulkheadFullException : Exception;And the wiring:
using Harbourline.Ha.Caching;
var builder = WebApplication.CreateBuilder(args);
// L2: Valkey/Redis. Locally: docker run -p 6379:6379 valkey/valkey:8
// On AWS: the ElastiCache endpoint, with TLS and auth taken from configuration/Secrets Manager.
builder.Services.AddStackExchangeRedisCache(o =>
o.Configuration = builder.Configuration.GetConnectionString("Cache") ?? "localhost:6379");
builder.Services.AddHybridCache();
builder.Services.AddSingleton<IRateSource, FakeRateSource>();
builder.Services.AddSingleton<RateCache>();
var app = builder.Build();
// Public reference data: cacheable at the edge.
app.MapGet("/api/public/rates/{baseCcy}/{quoteCcy}",
async (string baseCcy, string quoteCcy, RateCache rates, CancellationToken ct) =>
{
try { return Results.Ok(new { baseCcy, quoteCcy, rate = await rates.GetAsync(baseCcy, quoteCcy, ct) }); }
catch (BulkheadFullException)
{
return Results.Problem(statusCode: 503, title: "Busy, retry shortly",
extensions: new Dictionary<string, object?> { ["retryAfterSeconds"] = 2 });
}
})
.WithDataClass(DataClass.Public);
// Personal data: never stored by shared caches.
app.MapGet("/api/accounts/{id}/summary",
(string id) => Results.Ok(new { id, balance = 1250.40m, currency = "GBP" }))
.WithDataClass(DataClass.Personal);
app.Run();
sealed class FakeRateSource : IRateSource
{
public async Task<decimal> GetRateAsync(string b, string q, CancellationToken ct)
{
await Task.Delay(100, ct); // pretend the origin is slow
return b == q ? 1m : 1.17m; // fictional rate
}
}Run it with docker run --rm -p 6379:6379 valkey/valkey:8 in one terminal and dotnet run in another. Package references: Microsoft.Extensions.Caching.Hybrid and Microsoft.Extensions.Caching.StackExchangeRedis (look up current versions on NuGet; the ElastiCache lesson pins tested versions).
Code explained line by line
DataClassandCacheabilityencode the table from earlier as code. Changing a policy becomes a one-line diff with a test, not a hunt through controllers.Publicgetss-maxage=300so the edge keeps it for 5 minutes,max-age=60for browsers, andstale-if-error=600so the edge can serve a stale copy for up to 10 minutes if the origin fails.Personalisprivate, no-cache: a shared cache must not store it, and a browser must revalidate.Sensitiveisno-store: nobody keeps it.WithDataClassis an endpoint filter that runs after the handler and sets the headers. TheVaryheader is an extra safeguard for any intermediary that might otherwise cache personal responses.RateCachetakesHybridCachethrough a primary constructor.GetOrCreateAsyncreturns the cached value or runs the factory once per key at a time (stampede protection across callers in the same process, and with L2 shared between instances).- The expiry jitter spreads expirations.
LocalCacheExpirationis shorter than the distributedExpirationso each instance revalidates from L2 often. - The
catchis the fail-open guard: when the cache layer throws (Valkey down), the code still serves from the origin, still limited by the bulkhead. It deliberately excludes cancellation and bulkhead exceptions so those are not retried. OriginBulkheadallows 20 concurrent origin calls and waits at most 200 ms for a slot. Beyond that it sheds load by throwingBulkheadFullException, which the endpoint turns into HTTP 503.RemoveByTagAsync("rates")removes all entries taggedratesso a refresh job can invalidate everything at once./api/public/...and/api/accounts/...illustrate the two classes. The CloudFront behaviours earlier match these paths.
Security considerations
⚠ Learning shortcut — not for production
The local Valkey container runs with no password and no TLS, and the CloudFront example uses the default certificate. In production ElastiCache needs TLS, RBAC or IAM authentication, private subnets and a security group that allows only the API; CloudFront needs your own certificate, WAF, logging and a response headers policy.
- Cross-customer leakage is the main risk. Keys for personal data must include the customer identity, and shared edge caching of authenticated responses must be off. Write a test that fails if a
PersonalorSensitiveendpoint ever sendspublicors-maxage. - Cache poisoning. Keep the cache key small and controlled: only forward the query strings and headers you need, and never use the raw
Hostheader or user-supplied headers in the key. - Stale data after a security change. If a customer's account is frozen, a cached
Personalresponse could keep showing it as open. This is another reason personal data is not cached, or only for seconds. - Origin access. Use origin access control so S3 content is reachable only through CloudFront. Keep ALB security groups restricted to CloudFront's managed prefix list ⚠ VERIFY.
- Encryption and logging. ElastiCache with encryption at rest and in transit; CloudFront standard logs or real-time logs to a protected bucket, with retention set.
- Data protection. Cached personal data is still personal data under UK GDPR: it needs a retention period and must be covered by erasure and access processes. Ask your data-protection officer before caching it at all.
Banking use case
Harbourline's rule of thumb after a review by security and operations:
| Content | Class | Where cached |
|---|---|---|
| Marketing site, app static files (versioned names) | Public | CloudFront, 1 year, immutable |
| Savings rates, exchange rates, branch finder | Public | CloudFront 5 min, HybridCache 60 s |
| Feature flags, bank holiday calendar | Internal | HybridCache only |
| Account summary, transaction list | Personal | Not cached at the edge; origin only, with a database read replica |
| Card tokens, one-time codes, balances in a payment flow | Sensitive | Never cached |
| Rate limit counters | Operational | ElastiCache (atomic counters, see Module 6) |
During an incident drill the team stops the cache and confirms the API still answers within its SLO with a smaller concurrency limit, shedding load with a clear 503 and Retry-After. That is evidence for the operational-resilience file that the cache is an optimisation and not a single point of failure.
Common mistakes
- Caching by URL only for authenticated endpoints. Two customers calling
/api/accounts/me/summaryshare one URL, so a shared cache gives the first customer's data to the second. Do not cache these. - Wrong behaviour order. CloudFront picks the first matching path pattern in order, so
/api/*before/api/public/*silently disables caching for public data. - Invalidating on every deploy. Invalidations are slow and, beyond the free allowance, cost money. Use versioned filenames (
app.4f3a.js) and long TTLs. - A cache with no fallback. If the application throws when ElastiCache is down, you converted a performance layer into a dependency. Fail open with a bulkhead.
- Identical TTLs for everything. Synchronised expiry causes thundering herds; add jitter.
- Caching errors. A cached 500 serves a failure for the whole TTL. CloudFront has a separate error caching setting; keep error TTLs short ⚠ VERIFY defaults.
- Treating the cache as a database. ElastiCache can lose data on failover or flush. Keep the source of truth elsewhere (MemoryDB is the durable option ⚠ VERIFY suitability).
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Always Miss from cloudfront | Origin sends no-store, private or cookies in the cache key | Send public, s-maxage=...; remove cookies and headers from the cache key |
| Users see old content after a release | Long TTL on non-versioned file names | Version file names, or invalidate specific paths |
| Different users see each other's data | Shared cache stored an authenticated response | Remove caching of the path, add private, no-store, and review cache keys; treat as a security incident |
403 from CloudFront to S3 | Bucket policy missing OAC permission | Allow the distribution through the bucket policy for the OAC |
| Origin group never fails over | Failover status code not listed, or method not GET/HEAD/OPTIONS | Add the status codes; use origin groups only for safe methods |
| API slow after a cache restart | Cold-cache stampede on the database | Bulkhead, jitter, warm-up, scale the database temporarily |
RedisConnectionException in logs | Wrong endpoint, TLS mismatch or security group | Check the endpoint, ssl=true, and that the API's group is allowed |
| HybridCache serves stale local values | L1 LocalCacheExpiration too long | Shorten it or use tag invalidation |
Cost and cleanup
Cost: free for everything you run locally (Docker Valkey, .NET). The CloudFront and ElastiCache IaC are optional and not required for the exercises. If you deploy them: CloudFront bills for data transfer and requests with an always-free monthly allowance ⚠ VERIFY current numbers; ElastiCache nodes bill hourly whether used or not and are the part to delete promptly. A serverless cache has a minimum charge as well ⚠ VERIFY.
Teardown:
# Local: stop and remove the Valkey container.
docker rm -f $(docker ps -q --filter ancestor=valkey/valkey:8)
# CDK or Terraform deployment: destroy it.
cdk destroy --all --profile learn-sandbox
# or: terraform destroy
# Verify nothing tagged Project=cea-lab is left. CloudFront is a global service, so use us-east-1.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--region us-east-1 --profile learn-sandbox --query "ResourceTagMappingList[].ResourceARN"CloudFront distributions are disabled first, then deleted, which can take 15 minutes or more.
Exercises
- 🟢 Guided: Run the sample API with a local Valkey container. Call the rates endpoint twice and compare timings; then check the response headers with
curl -i. Confirm the account summary endpoint returnsprivate, no-cache. - 🟡 Challenge: Write xUnit v3 tests for
Cacheability(every class), and a test with a fakeIRateSourcethat blocks, proving the bulkhead throwsBulkheadFullExceptionwhen more than 20 callers are waiting. Stop the Valkey container during a load test and capture the behaviour. - 🔴 Live problem: After a "performance improvement" ticket, a tester reports seeing another customer's first name on the home screen, intermittently. CloudFront was set to cache
/api/*for 60 seconds with theAuthorizationheader excluded from the key. Triage: stop the leak, scope the exposure, fix the root cause, and write the incident note for the data-protection officer.
Assignment
Produce a caching standard for one Harbourline service: a table of endpoints with data class, edge policy, application cache policy and TTL; the CloudFront configuration; the fail-open behaviour; and a test plan for cache-down and cache-cold scenarios. Include the unit tests for the classification rule.
Solutions
The first call takes about 100 ms (origin delay); the second returns in a few ms from L1 or L2. curl -i http://localhost:5000/api/public/rates/GBP/EUR shows Cache-Control: public, max-age=60, s-maxage=300, .... The account summary shows Cache-Control: private, no-cache and a Vary header. The reason: the rate is public data, safe at the edge, the summary is personal, so the app says "do not store this in a shared cache".
Tests: CacheControl(DataClass.Sensitive) == "no-store"; SharedCacheAllowed(DataClass.Personal) is false; CacheControl(DataClass.Public) contains s-maxage and stale-if-error; an exhaustive loop over Enum.GetValues<DataClass>() asserts that any class for which SharedCacheAllowed is false does not contain public or s-maxage. For the bulkhead, expose the limit through the constructor, call the origin 25 times with a source that awaits a TaskCompletionSource that never completes, and assert at least 5 calls throw BulkheadFullException within about 300 ms. During the Valkey outage you should see warnings logged and slower answers, but no 5xx apart from deliberate 503s.
Knowledge check
Answer: If the system depends on the cache to survive normal load, losing the cache (failure, flush, restart) sends all traffic to the database at once, causing a stampede and a cascading failure. Mitigate with request coalescing, jittered TTLs, a bulkhead and warm-up.
Answer: Only public data, such as rates, static assets and product information. Personal and sensitive responses must not be cached at a shared edge, and the API should send private, no-cache or no-store.
Answer: It lets CloudFront try a secondary origin when the primary returns selected error status codes (or is unreachable), for GET, HEAD and OPTIONS requests. It is a simple failover for read-only content such as static sites.
Answer: s-maxage sets a lifetime for shared caches like CloudFront separately from max-age for browsers. stale-if-error lets a cache serve an expired copy for a limited time when the origin fails, which keeps public content visible during an outage.
Answer: A new name is a new cache key, so there is nothing to invalidate; old files age out. Invalidations take time and are billed beyond a free allowance.
Interview questions
- How would you decide what to cache in a banking API? Classify data first; only public and reference data go to shared caches. Personal data stays at the origin, or is cached per customer for seconds after legal sign-off. Always include a plan for cache loss.
- What is a cache stampede and how do you prevent it? Many requests recompute the same missing value simultaneously. Use request coalescing (
HybridCache), jittered expiry, a bulkhead and stale-while-revalidate. - CloudFront versus ElastiCache? CloudFront caches HTTP responses at the edge near users; ElastiCache caches application data inside your VPC for your own services.
- How do you invalidate a cache across many instances? Tag-based invalidation with
HybridCacheand a distributed L2, short L1 TTLs, and versioned keys.
Certification practice
A global bank serves a static web app and public exchange-rate JSON from S3 and an ALB. Pages load slowly for users in Asia and the ALB is overloaded at peak. Which change gives the largest improvement with the least change to the application?
A. Add more ALB instances in Asia. B. Put CloudFront in front, cache static files and the public JSON, and send API traffic that is personal to the ALB uncached. C. Move the application to a larger EC2 instance type. D. Enable S3 Transfer Acceleration for the static bucket.
Answer: B. CloudFront serves cacheable content from edge locations, reducing latency and origin load. A and C do not reduce global latency. D accelerates uploads to S3, not downloads through a CDN.
A site must continue to show its static pages when the primary S3 bucket's Region has a problem. Which CloudFront feature supports this?
A. Field-level encryption. B. An origin group with a replicated fallback bucket. C. Lambda@Edge viewer request function that retries. D. Geo restriction.
Answer: B. Origin groups fail over to a secondary origin on configured status codes, and a replicated bucket holds the content. A protects data fields. C is more complex and is not designed for this. D controls where users can access from.
After an ElastiCache node replacement the database CPU hits 100% for 10 minutes. What is the most appropriate long-term fix?
A. Disable caching. B. Add request coalescing, jittered TTLs and a concurrency limit on database calls, and consider warm-up. C. Increase TTLs to 24 hours. D. Move the cache to the same Availability Zone as the database.
Answer: B. The incident is a cold-cache stampede; limiting concurrent origin calls and spreading expirations prevents it. A removes the benefit. C increases staleness without fixing cold starts. D does not address the cause and reduces resilience.
Completion checklist
- I can explain the two jobs of a cache and the four data classes
- I know which classes may be cached by CloudFront and which may never be
- I completed the guided exercise and the cache-down test
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- A cache is a trade of freshness for speed and for protection of the back end.
- Classify data first. Personal and sensitive data never go to a shared edge cache.
- Design for cache loss: coalesce requests, jitter TTLs, bulkhead the origin, fail open.
- CloudFront behaviour order matters; specific patterns go first.
- Version file names; avoid mass invalidations.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
AWS Backup in depth
Build a backup plan with vaults, Vault Lock, cross-Region and cross-account copies, and automated restore testing with validation, then produce restore evidence a bank can show an auditor.
Performance and cost optimisation for resilient systems
Right-size auto-scaled fleets, move .NET services to Graviton, use Spot and Savings Plans safely, and put a price on idle disaster-recovery capacity with a small C# cost model so resilience spending is a decision, not a surprise.