Service-to-service communication: IAM auth, private APIs and service discovery
Connect Harbourline's internal services securely on AWS: choose between private API Gateway, VPC Lattice, ECS Service Connect and events, sign calls with IAM from C#, discover services with Cloud Map, and propagate customer context without trusting it blindly.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · Microsoft.Extensions.Http.Resilience 10.11.0 · Amazon.CDK.Lib 2.277.0 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Choose a communication style per interaction — synchronous HTTP, asynchronous events or messages — and justify it for banking workloads.
- Compare private REST APIs, Amazon VPC Lattice, ECS Service Connect, internal load balancers and AWS Cloud Map for service-to-service traffic and discovery.
- Sign service-to-service HTTP calls with IAM (SigV4) from C# using a reusable
DelegatingHandler, for bothexecute-apiandvpc-lattice-svcs. - Write VPC Lattice auth policies and API Gateway resource policies that allow one service's role to call one route of another.
- Propagate customer context and correlation IDs between services without letting a compromised service impersonate any customer.
Prerequisites
- Calling AWS APIs from C#: the SDK and SigV4 — the signing algorithm
- IAM/SigV4 authorisation and mTLS —
execute-api:Invoke, resource policies, private APIs - Calling partner APIs safely — timeouts, retries, circuit breakers
- Module 2: Networking — VPCs, endpoints, load balancers
- Lab: SigV4 signer walkthrough, code in
labs/m04-sigv4-signer/
Why this matters in banking
Harbourline's payment journey crosses at least five internal services: the Accounts API checks the debtor account, the Payments Service validates the instruction, the Sanctions Service screens the creditor, the Limits Service checks daily limits, and the Notifications Service tells the customer. In a typical on-premises bank these talk over a flat internal network where "inside the firewall" means "trusted". That assumption is how one compromised service becomes a breach of everything: if the Notifications Service can call POST /payments on the Payments Service, so can an attacker who gets code execution in the Notifications Service.
Modern regulators and security frameworks expect zero-trust internal traffic: every call authenticated with a workload identity, authorised for that specific operation, encrypted, logged and attributable. On AWS the natural workload identity is the IAM role each service already has. This lesson shows how to use it for HTTP between services, and when not to use HTTP at all.
The idea in plain language
Services need to answer three questions when they talk:
- Where is the other service? (discovery) — a stable name that resolves to healthy instances, even as tasks come and go.
- Who is calling me, and may they do this? (authentication and authorisation) — the caller's role, checked against a policy for this route.
- Do I need the answer now? (coupling) — if not, send an event or a message and let the other service process it in its own time.
AWS has several building blocks for these:
- Private REST API (API Gateway) — an API with no public endpoint, reachable through an interface VPC endpoint. IAM auth, resource policies, throttling, request validation and usage plans. Best when the internal API is a product with many consumers.
- Amazon VPC Lattice — an application networking service. You register services (ECS, EKS, Lambda, EC2, IP targets) in a service network, associate VPCs (even across accounts), and every service gets a DNS name. Lattice handles routing, health checks and IAM auth policies for each request. No sidecars, no VPC peering.
- ECS Service Connect — service-to-service networking for ECS services in the same namespace, with a managed Envoy proxy that adds discovery, retries, outlier detection and per-service metrics. Authentication is not IAM per request; it can add TLS between services.
- AWS Cloud Map — a service registry. Services register instances; clients discover them by DNS or the
DiscoverInstancesAPI. ECS integrates with it, and Service Connect uses Cloud Map namespaces. - Events and messages — Amazon EventBridge, SNS and SQS (covered in depth in later modules). No discovery problem, no synchronous dependency, IAM on the queue or bus.
Analogy: a bank's internal post room versus phone calls. A phone call (synchronous HTTP) needs the other person's extension (discovery), they must pick up now (availability), and they should check who's calling before reading out a balance (authentication). An internal memo (event) only needs the right pigeonhole, and the recipient deals with it when they can. VPC Lattice is the switchboard that knows every extension and checks the caller's badge before connecting. A private API is a dedicated, staffed help desk with its own rules and queue limits.
What problem the AWS service solves
| Option | Discovery | Per-request IAM auth | Cross-VPC / cross-account | Extra cost | Watch out |
|---|---|---|---|---|---|
| Private REST API | DNS via the execute-api endpoint | Yes (AWS_IAM + resource policy) | Yes, via endpoints in each VPC | Interface endpoint hours + API requests | 29 s default integration timeout; REST features only |
| VPC Lattice | Built-in DNS per service | Yes (auth policies, SigV4 with vpc-lattice-svcs) | Yes, service network associations; RAM sharing across accounts | Per service-hour + per GB + per request above a free hourly allowance | Payload is not signed: send x-amz-content-sha256: UNSIGNED-PAYLOAD |
| ECS Service Connect | Cloud Map namespace, short names (http://limits:8080) | No (network-level; optional TLS) | Same namespace; not a cross-account mesh | Proxy sidecar resources in each task | ECS only |
| Internal ALB | DNS of the load balancer | No (add your own: JWT, mTLS on ALB, or SigV4 checked in code) | Via peering / Transit Gateway | Hourly + LCU | Another hop to operate and pay for |
| Cloud Map (alone) | DNS or API discovery, health checks | No | Namespace per VPC (private DNS) | Per registered instance and per lookup ⚠ VERIFY | Discovery only — you still need auth |
| EventBridge / SNS / SQS | Bus or queue name | IAM on the resource | Yes, with resource policies | Per event/request | Eventual consistency; design for duplicates |
Verified facts worth remembering:
- VPC Lattice auth policies are IAM policy documents attached to a service network and/or a service. With auth type
AWS_IAM, requests are denied unless both the identity-based policy (if signed) and the auth policies allow them; explicit deny wins. Condition keys includevpc-lattice-svcs:SourceVpc,vpc-lattice-svcs:RequestMethod,vpc-lattice-svcs:RequestPathandvpc-lattice-svcs:ServiceNetworkArn. - VPC Lattice accepts SigV4 or SigV4A and doesn't support payload signing: the request must carry
x-amz-content-sha256: UNSIGNED-PAYLOAD. - VPC Lattice pricing in us-east-1: USD 0.025 per service per hour, USD 0.025 per GB processed, the first 300,000 requests per hour free, then USD 0.10 per million requests (VPC Lattice pricing page, checked 2026-11-18).
- AWS App Mesh reached end of support on 30 September 2026. AWS's migration guidance for ECS points to ECS Service Connect; VPC Lattice covers the cross-VPC, cross-account and IAM-auth cases. Don't start new designs on App Mesh.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Managed identity calling an internal API with an Entra token | IAM role + SigV4-signed request to a private API or Lattice service | No token endpoint, no audience to configure; the signature carries the identity |
| APIM in internal mode | Private REST API | Interface endpoint per VPC instead of VNet injection |
| Azure Container Apps internal ingress + Dapr service invocation | ECS Service Connect | Envoy is managed for you; no app-level SDK |
| Azure Service Fabric naming / Consul | AWS Cloud Map | DNS and API discovery with health checks |
| Private Link Service between VNets | VPC Lattice service network, or PrivateLink endpoint services | Lattice adds per-request auth and HTTP routing |
| Azure Service Bus topics | SNS + SQS, EventBridge | Separate services for fan-out, queueing and routing |
| Open Service Mesh / Istio on AKS | ECS Service Connect, VPC Lattice (App Mesh is discontinued) | No sidecar to run yourself with Lattice |
Architecture
Security-boundary view. Each arrow is authorised by IAM: the Lattice auth policy allows only hl-payments-task to POST /v1/screenings; the Accounts API resource policy allows only that role on GET /internal/accounts/* through the VPC endpoint; the event bus policy allows the Payments Service to put PaymentSettled events and the Notifications Service's rule to receive them. The Notifications Service has no path to call Payments at all — it only consumes events.
Step-by-step implementation
Decide synchronous or asynchronous per interaction
| Interaction | Style | Why |
|---|---|---|
| Payments → Sanctions: "may I send this payment?" | Synchronous (with a strict timeout and a fallback to manual review) | The answer gates the next step |
| Payments → Accounts: "is the debtor account open and does it have funds?" | Synchronous | Needed now, read-only |
| Payments → Notifications: "payment settled" | Event | The payment doesn't wait for an SMS; many consumers may care |
| Payments → Ledger: "post these entries" | Message on a queue (SQS FIFO) | Must happen exactly once per payment, in order, even if the ledger is down |
Rule of thumb: synchronous calls multiply availability (99.9% × 99.9% × 99.9% ≈ 99.7%) and latency; events trade immediacy for resilience. Keep synchronous chains short — ideally one hop.
Pick the transport for synchronous calls
- Same ECS cluster family, same team, no per-request auth requirement: ECS Service Connect.
- Across VPCs or accounts, mixed compute (ECS, Lambda, EKS), IAM per request: VPC Lattice.
- An internal API product with many consumers, throttling and usage plans, request validation: private REST API.
- Existing ALB-based estate: internal ALB, with JWT or mTLS checked by ALB or in code.
Give every service its own role and sign every call
Each ECS task definition gets a dedicated task role (hl-payments-task, hl-sanctions-task), never shared. The SDK finds those credentials automatically. For AWS services, the SDK signs. For your own HTTP services behind API Gateway or Lattice, plug a SigV4 DelegatingHandler into HttpClientFactory (complete code below).
Authorise at the destination
- Private REST API: method auth
AWS_IAM; resource policy allows the caller's role ARN andaws:SourceVpce; the caller's identity policy allowsexecute-api:Invokeon the exact method ARN. - VPC Lattice: auth type
AWS_IAMon the service network (coarse: "only principals in our organisation") and on the service (fine: "onlyhl-payments-taskmayPOST /v1/screenings"). The caller's identity policy allowsvpc-lattice-svcs:Invokeon the service ARN.
Propagate context deliberately
Pass a correlation ID on every hop (X-Correlation-ID, plus W3C traceparent from OpenTelemetry). Pass customer context only when the downstream service needs it, and make the downstream service decide whether to trust it:
- Option A — forward the customer's access token. Simple, but the token was issued for the public API's scopes and client, so the downstream service must accept a token not meant for it, and the token's lifetime can expire mid-flow.
- Option B — token exchange (RFC 8693): swap the customer token for a downstream-specific token. Strong, but needs an authorization server that supports it (Cognito doesn't natively ⚠ VERIFY).
- Option C — trusted header from an authenticated caller: the caller sends
X-Harbourline-Customer-Id; the downstream service honours it only when the IAM-authenticated caller is on an allow-list for that operation, and logs both identities. Harbourline uses C internally and A or B at the edge.
Make the call resilient and observable
Timeouts shorter than the caller's own deadline, retries only for idempotent operations, a circuit breaker per dependency, and metrics per destination. Service Connect and Lattice publish per-service metrics to CloudWatch; with private APIs, use API Gateway access logs and metrics.
AWS Console instructions
- VPC Lattice: Console → VPC → PrivateLink and Lattice → Service networks → Create service network → name
hl-internal, Auth type: AWS IAM → associate VPCs (Payments VPC, Sanctions VPC) with security groups → associate services. Then Services → Create servicehl-sanctions→ Auth type AWS IAM → listener HTTPS 443 → default action forward to a target group (ECS service, IP, Lambda or ALB) → Auth policy tab → paste the policy. - ECS Service Connect: Console → ECS → Clusters → (cluster) → Services → (service) → Update → Service Connect → turn on → namespace (Cloud Map) → client and server mode → port mapping name and discovery name (for example
limits). - Cloud Map: Console → Cloud Map → Namespaces → Create namespace (
harbourline.internal, private DNS in your VPC) → Create service → DNS records A/SRV, health check. - Private API: see the IAM/SigV4 and mTLS lesson.
AWS CLI
# --- VPC Lattice: service network with IAM auth, a service, and an auth policy ---
SN_ID=$(aws vpc-lattice create-service-network --name hl-internal --auth-type AWS_IAM \
--tags Project=cea-lab,Lab=m04-l16,Environment=learn,Owner=learner \
--query id --output text --profile learn-sandbox --region us-east-1)
SVC_ID=$(aws vpc-lattice create-service --name hl-sanctions --auth-type AWS_IAM \
--tags Project=cea-lab,Lab=m04-l16,Environment=learn,Owner=learner \
--query id --output text --profile learn-sandbox --region us-east-1)
aws vpc-lattice create-service-network-service-association \
--service-network-identifier $SN_ID --service-identifier $SVC_ID --profile learn-sandbox --region us-east-1
aws vpc-lattice create-service-network-vpc-association \
--service-network-identifier $SN_ID --vpc-identifier vpc-0payments0000000 \
--security-group-ids sg-0payments000000 --profile learn-sandbox --region us-east-1
# Only the Payments task role may POST /v1/screenings. The policy JSON must be on one line.
aws vpc-lattice put-auth-policy --resource-identifier $SVC_ID --policy file://sanctions-auth-policy.json \
--profile learn-sandbox --region us-east-1
# --- Cloud Map: discover healthy instances of a service (read-only) ---
aws servicediscovery discover-instances --namespace-name harbourline.internal --service-name limits \
--health-status HEALTHY --profile learn-sandbox --region us-east-1sanctions-auth-policy.json (one line in the real file; formatted here):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "AWS": "arn:aws:iam::111122223333:role/hl-payments-task" },
"Action": "vpc-lattice-svcs:Invoke",
"Resource": "arn:aws:vpc-lattice:us-east-1:444455556666:service/svc-0123456789abcdef0/v1/screenings",
"Condition": {
"StringEquals": { "vpc-lattice-svcs:RequestMethod": "POST" }
}
}
]
}And on the network: allow only authenticated principals from the organisation:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "vpc-lattice-svcs:Invoke",
"Resource": "*",
"Condition": {
"StringEquals": { "aws:PrincipalOrgID": "o-exampleorgid" },
"StringNotEquals": { "aws:PrincipalType": "Anonymous" }
}
}
]
}Infrastructure as Code
The stable CDK library exposes VPC Lattice through L1 constructs (CfnServiceNetwork, CfnService, CfnAuthPolicy and friends).
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.VpcLattice;
using Constructs;
namespace Harbourline.Internal.Infra;
public sealed class LatticeStack : Stack
{
public LatticeStack(Construct scope, string id, IStackProps props, string paymentsVpcId, string paymentsSgId,
IRole paymentsTaskRole, string sanctionsTargetGroupArn) : base(scope, id, props)
{
var network = new CfnServiceNetwork(this, "Network", new CfnServiceNetworkProps
{
Name = "hl-internal",
AuthType = "AWS_IAM",
});
var sanctions = new CfnService(this, "Sanctions", new CfnServiceProps
{
Name = "hl-sanctions",
AuthType = "AWS_IAM",
});
_ = new CfnListener(this, "Https", new CfnListenerProps
{
ServiceIdentifier = sanctions.AttrId,
Protocol = "HTTPS",
Port = 443,
DefaultAction = new CfnListener.DefaultActionProperty
{
Forward = new CfnListener.ForwardProperty
{
TargetGroups = new object[] { new CfnListener.WeightedTargetGroupProperty { TargetGroupIdentifier = sanctionsTargetGroupArn, Weight = 100 } },
},
},
});
_ = new CfnServiceNetworkServiceAssociation(this, "SanctionsInNetwork", new CfnServiceNetworkServiceAssociationProps
{
ServiceNetworkIdentifier = network.AttrId,
ServiceIdentifier = sanctions.AttrId,
});
_ = new CfnServiceNetworkVpcAssociation(this, "PaymentsVpc", new CfnServiceNetworkVpcAssociationProps
{
ServiceNetworkIdentifier = network.AttrId,
VpcIdentifier = paymentsVpcId,
SecurityGroupIds = [paymentsSgId],
});
// Fine-grained: only the Payments task role, only POST /v1/screenings.
_ = new CfnAuthPolicy(this, "SanctionsPolicy", new CfnAuthPolicyProps
{
ResourceIdentifier = sanctions.AttrId,
Policy = new PolicyDocument(new PolicyDocumentProps
{
Statements =
[
new PolicyStatement(new PolicyStatementProps
{
Principals = [new ArnPrincipal(paymentsTaskRole.RoleArn)],
Actions = ["vpc-lattice-svcs:Invoke"],
Resources = [$"{sanctions.AttrArn}/v1/screenings"],
Conditions = new Dictionary<string, object>
{
["StringEquals"] = new Dictionary<string, object> { ["vpc-lattice-svcs:RequestMethod"] = "POST" },
},
}),
],
}).ToJSON(),
});
// The caller needs an identity-based allow as well.
paymentsTaskRole.AddToPrincipalPolicy(new PolicyStatement(new PolicyStatementProps
{
Actions = ["vpc-lattice-svcs:Invoke"],
Resources = [$"{sanctions.AttrArn}/*"],
}));
_ = new CfnOutput(this, "SanctionsDns", new CfnOutputProps { Value = sanctions.AttrDnsEntryDomainName });
}
}Complete code
The signing handler lives in the lab project labs/m04-sigv4-signer/src/Harbourline.SigV4/ (SigV4Signer, SigV4Options, SigV4DelegatingHandler, AwsSdkCredentialSource), where it is tested against AWS's published SigV4 vectors. Here is how a service wires it into HttpClientFactory, together with resilience, correlation propagation and a trusted customer header.
using Amazon.Runtime.Credentials;
using Harbourline.SigV4;
namespace Harbourline.Payments;
public static class InternalClients
{
public static IServiceCollection AddInternalClients(this IServiceCollection services, IConfiguration config)
{
services.AddSingleton<IAwsCredentialSource>(_ =>
new AwsSdkCredentialSource(DefaultAWSCredentialsIdentityResolver.GetCredentials()));
services.AddSingleton(TimeProvider.System);
services.AddSingleton<SigV4Signer>();
services.AddHttpContextAccessor();
services.AddTransient<ContextPropagationHandler>();
// Handlers run outermost → innermost in registration order.
services.AddHttpClient<SanctionsClient>(c => c.BaseAddress = new Uri(config["Services:Sanctions"]!))
.AddHttpMessageHandler<ContextPropagationHandler>()
.AddResilienceHandler("sanctions", p =>
{
p.AddTimeout(TimeSpan.FromSeconds(3));
p.AddCircuitBreaker(new Microsoft.Extensions.Http.Resilience.HttpCircuitBreakerStrategyOptions
{
BreakDuration = TimeSpan.FromSeconds(15),
});
})
.AddHttpMessageHandler(sp => Signing(sp, "vpc-lattice-svcs", SigV4Options.VpcLattice));
// AddStandardResilienceHandler returns a pipeline builder, so the signing handler is added
// with a second AddHttpClient call for the same typed client: it lands innermost.
services.AddHttpClient<AccountsClient>(c => c.BaseAddress = new Uri(config["Services:Accounts"]!))
.AddHttpMessageHandler<ContextPropagationHandler>()
.AddStandardResilienceHandler().Configure(o => o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2));
services.AddHttpClient<AccountsClient>()
.AddHttpMessageHandler(sp => Signing(sp, "execute-api", SigV4Options.Default));
return services;
}
// Typed clients: thin wrappers you write per downstream API (methods omitted here).
public sealed class SanctionsClient(HttpClient http) { public HttpClient Http => http; }
public sealed class AccountsClient(HttpClient http) { public HttpClient Http => http; }
private static SigV4DelegatingHandler Signing(IServiceProvider sp, string service, SigV4Options options) =>
new(sp.GetRequiredService<SigV4Signer>(), sp.GetRequiredService<IAwsCredentialSource>(), "us-east-1", service, options);
}using System.Security.Claims;
namespace Harbourline.Payments;
/// <summary>
/// Copies the correlation id and the server-resolved customer id from the current request to the
/// outgoing call. The customer id comes from our own validated token processing, never from the
/// incoming request's headers.
/// </summary>
public sealed class ContextPropagationHandler(IHttpContextAccessor accessor) : DelegatingHandler
{
public const string CorrelationHeader = "X-Correlation-ID";
public const string CustomerHeader = "X-Harbourline-Customer-Id";
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct)
{
var http = accessor.HttpContext;
if (http is not null)
{
if (http.Items["CorrelationId"] is string correlationId)
request.Headers.TryAddWithoutValidation(CorrelationHeader, correlationId);
if (http.User.FindFirstValue("customer_id") is { } customerId)
request.Headers.TryAddWithoutValidation(CustomerHeader, customerId);
}
return base.SendAsync(request, ct);
}
}On the receiving side, the Accounts API trusts X-Harbourline-Customer-Id only from callers it has authenticated with IAM and allow-listed. Behind an AWS_IAM method with a Lambda proxy integration, API Gateway puts the caller's ARN in the request context:
using Amazon.Lambda.APIGatewayEvents;
namespace Harbourline.Accounts.Internal;
public static class TrustedCaller
{
// Roles allowed to act on behalf of a customer. Everything else may only read its own data.
private static readonly string[] OnBehalfOfAllowList =
[
"arn:aws:sts::111122223333:assumed-role/hl-payments-task/",
];
/// <summary>Returns the customer id to act for, or null if the caller may not assert one.</summary>
public static string? CustomerFor(HttpContext http)
{
if (http.Items["LambdaRequestObject"] is not APIGatewayProxyRequest request) return null;
// Set by API Gateway after SigV4 verification: the caller cannot forge it.
var callerArn = request.RequestContext?.Identity?.UserArn;
if (callerArn is null || !OnBehalfOfAllowList.Any(p => callerArn.StartsWith(p, StringComparison.Ordinal)))
return null;
var customerId = http.Request.Headers["X-Harbourline-Customer-Id"].ToString();
return System.Text.RegularExpressions.Regex.IsMatch(customerId, "^CUS-[0-9]{8}$") ? customerId : null;
}
}Code explained line by line
DefaultAWSCredentialsIdentityResolver.GetCredentials()— the SDK v4 default credential chain: environment, SSO profile, ECS container credentials, EC2 instance profile ⚠ VERIFY the exact type name in your SDK version (v3 called itFallbackCredentialsFactory).AwsSdkCredentialSourceturns SDK credentials into the signer'sAwsCredentialson every call, so rotated task-role credentials are picked up automatically.- Handler order in
HttpClientFactoryis outermost first. Context propagation → resilience → signing means each retry passes through the signing handler again and gets a freshX-Amz-Dateand signature. Signing outside the retry would replay a signature that may be minutes old. SigV4Options.VpcLattice—UNSIGNED-PAYLOAD, thex-amz-content-sha256header, and single-encoded paths, as VPC Lattice requires.- The Sanctions client has no retry:
POST /v1/screeningsisn't safe to repeat unless the Sanctions Service de-duplicates on a key. The timeout and circuit breaker still protect the caller. AddStandardResilienceHandler()gives the Accounts client a sensible default pipeline (rate limiter, total timeout, retry, circuit breaker, attempt timeout) for GETs.AddHttpMessageHandler(sp => ...)with a factory builds a new signing handler per handler chain; don't try to wrap handlers inside each other by hand.ContextPropagationHandlerreadscustomer_idfrom the current request's validated principal (resolved server-side, as in the module project), never by copying an incoming header — otherwise a caller could inject any customer ID.TrustedCaller.CustomerForreadsrequestContext.identity.userArn, which API Gateway fills in after verifying the SigV4 signature. Only allow-listed roles may assert a customer; everyone else getsnulland must be treated as "service acting for itself".- The regex check on
CUS-+ 8 digits keeps junk out of queries and logs.
Security considerations
⚠ Learning shortcut — not for production
The allow-list of roles that may act on behalf of customers is hard-coded in TrustedCaller, and the Lattice policies use one account ID in examples. In production, keep the allow-list in configuration managed through code review, use aws:PrincipalOrgID and SCPs as guard rails, and review the trust relationships in your threat model. For the highest-risk flows (moving money), prefer a downstream-specific token from token exchange, so the downstream service sees proof of the customer's consent rather than another service's assertion.
- One role per service, never shared. Shared roles destroy attribution and least privilege.
- Authorise at the destination, not just the network. Security groups and private subnets are necessary but not sufficient; Lattice auth policies and API resource policies decide per request.
- Encrypt in transit everywhere. Lattice HTTPS listeners, TLS on Service Connect, HTTPS on private APIs. Internal doesn't mean plaintext.
- Events need authorisation too. Event bus and queue policies should allow specific producer roles and specific consumer rules. Consumers must validate event schemas and de-duplicate.
- Log both identities on every on-behalf-of call: the service principal (ARN) and the customer ID, plus the correlation ID. This is what auditors and incident responders need.
- Timeouts are a security control as well as a resilience one: unbounded calls let one slow dependency exhaust threads across the estate.
Banking use case
Harbourline moves its payment orchestration to Fargate. The Payments Service calls the Sanctions Service (owned by Financial Crime, in a separate account) through a VPC Lattice service network shared with AWS RAM; the auth policy allows only the Payments task role to POST /v1/screenings. It reads account status from the Accounts team's private REST API with AWS_IAM, passing the customer ID in a trusted header that the Accounts API honours only from hl-payments-task. Settlement publishes PaymentSettled to EventBridge; Notifications and the Data Platform subscribe. When Financial Crime rotates its Fargate tasks, nothing changes for Payments: the Lattice DNS name stays the same. The previous App Mesh setup is retired, as App Mesh reached end of support on 30 September 2026.
Common mistakes
- Signing outside the retry handler → retries fail with
Signature expiredor replay old signatures → put signing innermost. - Signing the payload for VPC Lattice → requests rejected →
UNSIGNED-PAYLOADand thex-amz-content-sha256header. - Trusting
X-Customer-Idfrom anyone → any internal service can impersonate any customer → trust only allow-listed IAM callers. - Chaining five synchronous calls → latency and availability multiply → use events for anything that doesn't gate the response.
- Using Cloud Map alone as "security" → discovery isn't authorisation → add IAM auth (Lattice, private API) or mTLS.
- Starting new work on App Mesh → discontinued → Service Connect or Lattice.
- Forgetting the caller's identity policy for Lattice or execute-api → 403 even though the destination policy allows it → both sides need an allow.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Lattice returns 403 AccessDeniedException | Auth policy doesn't allow the caller's role, path or method; or caller lacks vpc-lattice-svcs:Invoke | Compare the role ARN, RequestPath and RequestMethod with both policies |
Lattice 403 for every request after enabling AWS_IAM | No auth policy attached yet (default deny) | Attach a policy to the network and service |
Signature mismatch only for paths with spaces or % | Double-encoding the path for Lattice | Use SigV4Options.VpcLattice (single-encoded path) |
| Name resolves outside the VPC but times out | VPC not associated with the service network, or security group blocks it | Associate the VPC; allow outbound 443 and the Lattice prefix list ⚠ VERIFY |
| Service Connect short name doesn't resolve | Client service not in the same namespace or not in client mode | Enable Service Connect on the client service |
| Private API call from Lambda times out | Lambda not attached to the VPC with the endpoint | Configure VPC access and endpoint security group |
Cost and cleanup
This lesson is labelled free because everything in the complete code runs locally with the lab's tests (labs/m04-sigv4-signer), and the console/CLI steps are optional. If you do create Lattice resources, a service costs about USD 0.025 per hour (USD 0.60 per day) plus negligible data and request charges at lab volumes — delete it the same day. Private API endpoints cost about USD 0.01 per AZ-hour ⚠ VERIFY.
Teardown (in this order: associations before the network):
aws vpc-lattice delete-auth-policy --resource-identifier $SVC_ID --profile learn-sandbox --region us-east-1
for a in $(aws vpc-lattice list-service-network-service-associations --service-network-identifier $SN_ID \
--query "items[].id" --output text --profile learn-sandbox --region us-east-1); do
aws vpc-lattice delete-service-network-service-association --service-network-service-association-identifier $a --profile learn-sandbox --region us-east-1
done
for a in $(aws vpc-lattice list-service-network-vpc-associations --service-network-identifier $SN_ID \
--query "items[].id" --output text --profile learn-sandbox --region us-east-1); do
aws vpc-lattice delete-service-network-vpc-association --service-network-vpc-association-identifier $a --profile learn-sandbox --region us-east-1
done
aws vpc-lattice delete-service --service-identifier $SVC_ID --profile learn-sandbox --region us-east-1
aws vpc-lattice delete-service-network --service-network-identifier $SN_ID --profile learn-sandbox --region us-east-1
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m04-l16 \
--query "ResourceTagMappingList[].ResourceARN" --profile learn-sandbox --region us-east-1Associations take a short time to delete; if a delete fails with a conflict, wait and retry.
Exercises
- 🟢 Guided: Run the SigV4 lab tests. Then, in
SigV4DelegatingHandlerTests, add a test proving that a request with a path containing a space is canonicalised differently forexecute-apiand for Lattice. - 🟡 Challenge: Draw Harbourline's payment flow (Accounts, Payments, Sanctions, Limits, Notifications, Ledger) and label each arrow with style (sync/event/queue), transport (Lattice, private API, Service Connect, EventBridge, SQS) and the IAM policy that authorises it.
- 🔴 Live problem: After a deployment, the Payments Service gets intermittent 403s from the Sanctions Service through Lattice, about 1 in 20 requests, only under load. Nothing changed in the policies. Investigate and fix.
Assignment
Write an internal communication standard for Harbourline (one or two pages): when to use events vs synchronous calls, approved transports, identity and authorisation rules (one role per service, auth at the destination, on-behalf-of rules), required headers (correlation, trace), timeouts and retry rules, and a migration note for any App Mesh usage.
Solutions
Build a request to https://x.execute-api.us-east-1.amazonaws.com/learn/files/q3 report (the URI escapes the space to %20). With SigV4Options.Default the canonical path contains q3%2520report (each segment encoded again); with SigV4Options.VpcLattice it contains q3%20report. Assert on SigningResult.CanonicalRequest. Why: most AWS services expect the path double-encoded in the canonical request; S3 and VPC Lattice don't — a classic source of "works until a customer uses a space" bugs.
Typical result: Payments → Accounts (sync, private API, execute-api:Invoke on GET /internal/accounts/*), Payments → Sanctions (sync, Lattice, auth policy allowing POST /v1/screenings), Payments → Limits (sync, Service Connect within the payments cluster, no per-request IAM, TLS on), Payments → Ledger (SQS FIFO, queue policy allowing sqs:SendMessage from the Payments role), Payments → Notifications and Data Platform (EventBridge, bus policy for events:PutEvents, rules per consumer). The key insight is that only the steps that gate the customer's response are synchronous.
Intermittent, load-related 403s with unchanged policies point to signing, not authorisation. Likely causes: (1) signing happens outside the retry handler, so retries reuse an old signature — move signing innermost; (2) credentials expire mid-flight because they were cached at startup instead of refreshed — use the SDK credential object on every call; (3) clock skew on some tasks. Check Lattice access logs for the failure reason, compare X-Amz-Date against the receive time, and confirm handler order. Add a test that runs a retry and asserts a new X-Amz-Date.
Knowledge check
Answer: The caller's identity-based IAM policy (allow vpc-lattice-svcs:Invoke) and the Lattice auth policies on the service network and the service (where configured). An explicit deny anywhere wins.
Answer: So every retry attempt is signed fresh with a new timestamp. A signature generated once outside the retry loop can expire or be rejected as a replay, and changing a header after signing breaks it.
Answer: For ECS services in the same namespace that need simple discovery, retries and metrics without per-request IAM authorisation or cross-account reach. Choose Lattice when you need IAM auth policies per request, cross-VPC/cross-account connectivity or mixed compute types.
Answer: App Mesh reached end of support on 30 September 2026. For ECS, AWS points to ECS Service Connect; VPC Lattice is the option for IAM-authorised, cross-VPC and cross-account service networking.
Interview questions
- "How do microservices authenticate to each other on AWS?" Each service has its own IAM role; calls are SigV4-signed; the destination (private API Gateway, VPC Lattice) authorises with resource/auth policies plus the caller's identity policy. No shared secrets, full attribution.
- "How do you pass the end user's identity to a downstream service?" Either forward or exchange the user's token so the downstream service can verify the user's consent, or pass a customer ID in a header that the downstream service accepts only from authenticated, allow-listed service identities, logging both. Never trust an unauthenticated header.
- "Sync or async?" Sync only when the caller needs the answer to respond; otherwise events or queues, with idempotent consumers. Keep sync chains short and protected by timeouts and circuit breakers.
- "What did you use instead of a service mesh?" Service Connect for intra-cluster ECS traffic, Lattice for cross-VPC/account with IAM auth, plus OpenTelemetry for tracing.
Certification practice
Services in three AWS accounts and several VPCs (ECS, Lambda and EKS) must call each other over HTTPS. Each call must be authorised per caller role and per path, without managing VPC peering or a self-hosted proxy. What should the architect use?
A. VPC peering with security groups referencing each other. B. Amazon VPC Lattice with a shared service network, auth type AWS_IAM and auth policies per service. C. AWS App Mesh with Envoy sidecars. D. Public ALBs with AWS WAF IP allow-lists.
Answer: B. Lattice connects VPCs and accounts without peering and authorises each request with IAM. A gives connectivity but no per-request auth. C is discontinued. D exposes services publicly.
A .NET service signs requests to a VPC Lattice service with SigV4 and the service name vpc-lattice-svcs. GET requests succeed, but POST requests with a JSON body are rejected. What should the developer change?
A. Use the service name execute-api.
B. Send x-amz-content-sha256: UNSIGNED-PAYLOAD and don't sign the payload.
C. Add an API key header.
D. Switch to HTTP instead of HTTPS.
Answer: B. Lattice doesn't support payload signing. A is the wrong service. C is unrelated. D weakens security and doesn't fix signing.
An internal Accounts API accepts a X-Customer-Id header to decide which customer's data to return. Several internal services can reach it. What is the most secure design?
A. Accept the header from any caller inside the VPC. B. Require IAM authorisation on the API and accept the header only from an allow-list of caller role ARNs taken from the verified request context, logging both identities. C. Encrypt the header with a shared key. D. Rate-limit the header per IP.
Answer: B. The caller's identity must be verified (IAM) and authorised to assert customer context. A trusts the network. C adds a shared secret every service would hold. D doesn't address impersonation.
Completion checklist
- I can choose between events, Service Connect, VPC Lattice and private APIs for a given interaction
- I ran the SigV4 lab tests, including the
execute-apiand Lattice handler tests - I can write a Lattice auth policy for one role, one path and one method
- I can explain the trusted-header pattern and its risks
- I deleted any Lattice or endpoint resources I created
Key takeaways
- Decide sync vs async first; keep synchronous chains short.
- On AWS the workload identity is the IAM role; sign service calls with SigV4 and authorise at the destination with resource/auth policies plus identity policies.
- VPC Lattice gives discovery, routing and per-request IAM auth across VPCs and accounts; Service Connect gives simple ECS discovery and resilience; private APIs suit internal API products. App Mesh is discontinued.
- Propagate correlation IDs always; propagate customer context only through an authenticated, allow-listed path.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Calling AWS APIs from C#: the SDK and a hand-written SigV4 signer
Configure AWS SDK for .NET v4 clients properly (credentials, Region, retries, timeouts, paginators, DI), then sign an AWS request by hand with Signature Version 4 so you understand exactly what the SDK does and can debug SignatureDoesNotMatch errors.
Lab: Hand-sign an AWS request with SigV4 (Lab 4.5)
Implement AWS Signature Version 4 in plain C#, prove it against AWS's published test vectors, sign a real STS GetCallerIdentity call and compare it with the SDK, then reuse the signer as a DelegatingHandler for IAM-protected APIs.