Skip to Content
Cloud Engineering Academy
4. API Development & Integration

Service-to-service communication: IAM auth, private APIs and service discovery

Connect Harbourline's internal services securely on AWS: choose between private API Gateway, VPC Lattice, ECS Service Connect and events, sign calls with IAM from C#, discover services with Cloud Map, and propagate customer context without trusting it blindly.

advanced⏱ 1 h 30 min🟢 FreeSAA-C03DVA-C03SCS-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 · Microsoft.Extensions.Http.Resilience 10.11.0 · Amazon.CDK.Lib 2.277.0 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Choose a communication style per interaction — synchronous HTTP, asynchronous events or messages — and justify it for banking workloads.
  • Compare private REST APIs, Amazon VPC Lattice, ECS Service Connect, internal load balancers and AWS Cloud Map for service-to-service traffic and discovery.
  • Sign service-to-service HTTP calls with IAM (SigV4) from C# using a reusable DelegatingHandler, for both execute-api and vpc-lattice-svcs.
  • Write VPC Lattice auth policies and API Gateway resource policies that allow one service's role to call one route of another.
  • Propagate customer context and correlation IDs between services without letting a compromised service impersonate any customer.

Prerequisites

Why this matters in banking

Harbourline's payment journey crosses at least five internal services: the Accounts API checks the debtor account, the Payments Service validates the instruction, the Sanctions Service screens the creditor, the Limits Service checks daily limits, and the Notifications Service tells the customer. In a typical on-premises bank these talk over a flat internal network where "inside the firewall" means "trusted". That assumption is how one compromised service becomes a breach of everything: if the Notifications Service can call POST /payments on the Payments Service, so can an attacker who gets code execution in the Notifications Service.

Modern regulators and security frameworks expect zero-trust internal traffic: every call authenticated with a workload identity, authorised for that specific operation, encrypted, logged and attributable. On AWS the natural workload identity is the IAM role each service already has. This lesson shows how to use it for HTTP between services, and when not to use HTTP at all.

The idea in plain language

Services need to answer three questions when they talk:

  1. Where is the other service? (discovery) — a stable name that resolves to healthy instances, even as tasks come and go.
  2. Who is calling me, and may they do this? (authentication and authorisation) — the caller's role, checked against a policy for this route.
  3. Do I need the answer now? (coupling) — if not, send an event or a message and let the other service process it in its own time.

AWS has several building blocks for these:

  • Private REST API (API Gateway) — an API with no public endpoint, reachable through an interface VPC endpoint. IAM auth, resource policies, throttling, request validation and usage plans. Best when the internal API is a product with many consumers.
  • Amazon VPC Lattice — an application networking service. You register services (ECS, EKS, Lambda, EC2, IP targets) in a service network, associate VPCs (even across accounts), and every service gets a DNS name. Lattice handles routing, health checks and IAM auth policies for each request. No sidecars, no VPC peering.
  • ECS Service Connect — service-to-service networking for ECS services in the same namespace, with a managed Envoy proxy that adds discovery, retries, outlier detection and per-service metrics. Authentication is not IAM per request; it can add TLS between services.
  • AWS Cloud Map — a service registry. Services register instances; clients discover them by DNS or the DiscoverInstances API. ECS integrates with it, and Service Connect uses Cloud Map namespaces.
  • Events and messages — Amazon EventBridge, SNS and SQS (covered in depth in later modules). No discovery problem, no synchronous dependency, IAM on the queue or bus.

Analogy: a bank's internal post room versus phone calls. A phone call (synchronous HTTP) needs the other person's extension (discovery), they must pick up now (availability), and they should check who's calling before reading out a balance (authentication). An internal memo (event) only needs the right pigeonhole, and the recipient deals with it when they can. VPC Lattice is the switchboard that knows every extension and checks the caller's badge before connecting. A private API is a dedicated, staffed help desk with its own rules and queue limits.

What problem the AWS service solves

OptionDiscoveryPer-request IAM authCross-VPC / cross-accountExtra costWatch out
Private REST APIDNS via the execute-api endpointYes (AWS_IAM + resource policy)Yes, via endpoints in each VPCInterface endpoint hours + API requests29 s default integration timeout; REST features only
VPC LatticeBuilt-in DNS per serviceYes (auth policies, SigV4 with vpc-lattice-svcs)Yes, service network associations; RAM sharing across accountsPer service-hour + per GB + per request above a free hourly allowancePayload is not signed: send x-amz-content-sha256: UNSIGNED-PAYLOAD
ECS Service ConnectCloud Map namespace, short names (http://limits:8080)No (network-level; optional TLS)Same namespace; not a cross-account meshProxy sidecar resources in each taskECS only
Internal ALBDNS of the load balancerNo (add your own: JWT, mTLS on ALB, or SigV4 checked in code)Via peering / Transit GatewayHourly + LCUAnother hop to operate and pay for
Cloud Map (alone)DNS or API discovery, health checksNoNamespace per VPC (private DNS)Per registered instance and per lookup ⚠ VERIFYDiscovery only — you still need auth
EventBridge / SNS / SQSBus or queue nameIAM on the resourceYes, with resource policiesPer event/requestEventual consistency; design for duplicates

Verified facts worth remembering:

  • VPC Lattice auth policies are IAM policy documents attached to a service network and/or a service. With auth type AWS_IAM, requests are denied unless both the identity-based policy (if signed) and the auth policies allow them; explicit deny wins. Condition keys include vpc-lattice-svcs:SourceVpc, vpc-lattice-svcs:RequestMethod, vpc-lattice-svcs:RequestPath and vpc-lattice-svcs:ServiceNetworkArn.
  • VPC Lattice accepts SigV4 or SigV4A and doesn't support payload signing: the request must carry x-amz-content-sha256: UNSIGNED-PAYLOAD.
  • VPC Lattice pricing in us-east-1: USD 0.025 per service per hour, USD 0.025 per GB processed, the first 300,000 requests per hour free, then USD 0.10 per million requests (VPC Lattice pricing page, checked 2026-11-18).
  • AWS App Mesh reached end of support on 30 September 2026. AWS's migration guidance for ECS points to ECS Service Connect; VPC Lattice covers the cross-VPC, cross-account and IAM-auth cases. Don't start new designs on App Mesh.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Managed identity calling an internal API with an Entra tokenIAM role + SigV4-signed request to a private API or Lattice serviceNo token endpoint, no audience to configure; the signature carries the identity
APIM in internal modePrivate REST APIInterface endpoint per VPC instead of VNet injection
Azure Container Apps internal ingress + Dapr service invocationECS Service ConnectEnvoy is managed for you; no app-level SDK
Azure Service Fabric naming / ConsulAWS Cloud MapDNS and API discovery with health checks
Private Link Service between VNetsVPC Lattice service network, or PrivateLink endpoint servicesLattice adds per-request auth and HTTP routing
Azure Service Bus topicsSNS + SQS, EventBridgeSeparate services for fan-out, queueing and routing
Open Service Mesh / Istio on AKSECS Service Connect, VPC Lattice (App Mesh is discontinued)No sidecar to run yourself with Lattice

Architecture

Security-boundary view. Each arrow is authorised by IAM: the Lattice auth policy allows only hl-payments-task to POST /v1/screenings; the Accounts API resource policy allows only that role on GET /internal/accounts/* through the VPC endpoint; the event bus policy allows the Payments Service to put PaymentSettled events and the Notifications Service's rule to receive them. The Notifications Service has no path to call Payments at all — it only consumes events.

Step-by-step implementation

Decide synchronous or asynchronous per interaction

InteractionStyleWhy
Payments → Sanctions: "may I send this payment?"Synchronous (with a strict timeout and a fallback to manual review)The answer gates the next step
Payments → Accounts: "is the debtor account open and does it have funds?"SynchronousNeeded now, read-only
Payments → Notifications: "payment settled"EventThe payment doesn't wait for an SMS; many consumers may care
Payments → Ledger: "post these entries"Message on a queue (SQS FIFO)Must happen exactly once per payment, in order, even if the ledger is down

Rule of thumb: synchronous calls multiply availability (99.9% × 99.9% × 99.9% ≈ 99.7%) and latency; events trade immediacy for resilience. Keep synchronous chains short — ideally one hop.

Pick the transport for synchronous calls

  • Same ECS cluster family, same team, no per-request auth requirement: ECS Service Connect.
  • Across VPCs or accounts, mixed compute (ECS, Lambda, EKS), IAM per request: VPC Lattice.
  • An internal API product with many consumers, throttling and usage plans, request validation: private REST API.
  • Existing ALB-based estate: internal ALB, with JWT or mTLS checked by ALB or in code.

Give every service its own role and sign every call

Each ECS task definition gets a dedicated task role (hl-payments-task, hl-sanctions-task), never shared. The SDK finds those credentials automatically. For AWS services, the SDK signs. For your own HTTP services behind API Gateway or Lattice, plug a SigV4 DelegatingHandler into HttpClientFactory (complete code below).

Authorise at the destination

  • Private REST API: method auth AWS_IAM; resource policy allows the caller's role ARN and aws:SourceVpce; the caller's identity policy allows execute-api:Invoke on the exact method ARN.
  • VPC Lattice: auth type AWS_IAM on the service network (coarse: "only principals in our organisation") and on the service (fine: "only hl-payments-task may POST /v1/screenings"). The caller's identity policy allows vpc-lattice-svcs:Invoke on the service ARN.

Propagate context deliberately

Pass a correlation ID on every hop (X-Correlation-ID, plus W3C traceparent from OpenTelemetry). Pass customer context only when the downstream service needs it, and make the downstream service decide whether to trust it:

  • Option A — forward the customer's access token. Simple, but the token was issued for the public API's scopes and client, so the downstream service must accept a token not meant for it, and the token's lifetime can expire mid-flow.
  • Option B — token exchange (RFC 8693): swap the customer token for a downstream-specific token. Strong, but needs an authorization server that supports it (Cognito doesn't natively ⚠ VERIFY).
  • Option C — trusted header from an authenticated caller: the caller sends X-Harbourline-Customer-Id; the downstream service honours it only when the IAM-authenticated caller is on an allow-list for that operation, and logs both identities. Harbourline uses C internally and A or B at the edge.

Make the call resilient and observable

Timeouts shorter than the caller's own deadline, retries only for idempotent operations, a circuit breaker per dependency, and metrics per destination. Service Connect and Lattice publish per-service metrics to CloudWatch; with private APIs, use API Gateway access logs and metrics.

AWS Console instructions

  • VPC Lattice: Console → VPC → PrivateLink and Lattice → Service networks → Create service network → name hl-internal, Auth type: AWS IAM → associate VPCs (Payments VPC, Sanctions VPC) with security groups → associate services. Then Services → Create service hl-sanctions → Auth type AWS IAM → listener HTTPS 443 → default action forward to a target group (ECS service, IP, Lambda or ALB) → Auth policy tab → paste the policy.
  • ECS Service Connect: Console → ECS → Clusters → (cluster) → Services → (service) → Update → Service Connect → turn on → namespace (Cloud Map) → client and server mode → port mapping name and discovery name (for example limits).
  • Cloud Map: Console → Cloud Map → Namespaces → Create namespace (harbourline.internal, private DNS in your VPC) → Create service → DNS records A/SRV, health check.
  • Private API: see the IAM/SigV4 and mTLS lesson.

AWS CLI

# --- VPC Lattice: service network with IAM auth, a service, and an auth policy ---
SN_ID=$(aws vpc-lattice create-service-network --name hl-internal --auth-type AWS_IAM \
  --tags Project=cea-lab,Lab=m04-l16,Environment=learn,Owner=learner \
  --query id --output text --profile learn-sandbox --region us-east-1)

SVC_ID=$(aws vpc-lattice create-service --name hl-sanctions --auth-type AWS_IAM \
  --tags Project=cea-lab,Lab=m04-l16,Environment=learn,Owner=learner \
  --query id --output text --profile learn-sandbox --region us-east-1)

aws vpc-lattice create-service-network-service-association \
  --service-network-identifier $SN_ID --service-identifier $SVC_ID --profile learn-sandbox --region us-east-1

aws vpc-lattice create-service-network-vpc-association \
  --service-network-identifier $SN_ID --vpc-identifier vpc-0payments0000000 \
  --security-group-ids sg-0payments000000 --profile learn-sandbox --region us-east-1

# Only the Payments task role may POST /v1/screenings. The policy JSON must be on one line.
aws vpc-lattice put-auth-policy --resource-identifier $SVC_ID --policy file://sanctions-auth-policy.json \
  --profile learn-sandbox --region us-east-1

# --- Cloud Map: discover healthy instances of a service (read-only) ---
aws servicediscovery discover-instances --namespace-name harbourline.internal --service-name limits \
  --health-status HEALTHY --profile learn-sandbox --region us-east-1

sanctions-auth-policy.json (one line in the real file; formatted here):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:role/hl-payments-task" },
      "Action": "vpc-lattice-svcs:Invoke",
      "Resource": "arn:aws:vpc-lattice:us-east-1:444455556666:service/svc-0123456789abcdef0/v1/screenings",
      "Condition": {
        "StringEquals": { "vpc-lattice-svcs:RequestMethod": "POST" }
      }
    }
  ]
}

And on the network: allow only authenticated principals from the organisation:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "vpc-lattice-svcs:Invoke",
      "Resource": "*",
      "Condition": {
        "StringEquals": { "aws:PrincipalOrgID": "o-exampleorgid" },
        "StringNotEquals": { "aws:PrincipalType": "Anonymous" }
      }
    }
  ]
}

Infrastructure as Code

The stable CDK library exposes VPC Lattice through L1 constructs (CfnServiceNetwork, CfnService, CfnAuthPolicy and friends).

LatticeStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.VpcLattice;
using Constructs;

namespace Harbourline.Internal.Infra;

public sealed class LatticeStack : Stack
{
    public LatticeStack(Construct scope, string id, IStackProps props, string paymentsVpcId, string paymentsSgId,
        IRole paymentsTaskRole, string sanctionsTargetGroupArn) : base(scope, id, props)
    {
        var network = new CfnServiceNetwork(this, "Network", new CfnServiceNetworkProps
        {
            Name = "hl-internal",
            AuthType = "AWS_IAM",
        });

        var sanctions = new CfnService(this, "Sanctions", new CfnServiceProps
        {
            Name = "hl-sanctions",
            AuthType = "AWS_IAM",
        });

        _ = new CfnListener(this, "Https", new CfnListenerProps
        {
            ServiceIdentifier = sanctions.AttrId,
            Protocol = "HTTPS",
            Port = 443,
            DefaultAction = new CfnListener.DefaultActionProperty
            {
                Forward = new CfnListener.ForwardProperty
                {
                    TargetGroups = new object[] { new CfnListener.WeightedTargetGroupProperty { TargetGroupIdentifier = sanctionsTargetGroupArn, Weight = 100 } },
                },
            },
        });

        _ = new CfnServiceNetworkServiceAssociation(this, "SanctionsInNetwork", new CfnServiceNetworkServiceAssociationProps
        {
            ServiceNetworkIdentifier = network.AttrId,
            ServiceIdentifier = sanctions.AttrId,
        });

        _ = new CfnServiceNetworkVpcAssociation(this, "PaymentsVpc", new CfnServiceNetworkVpcAssociationProps
        {
            ServiceNetworkIdentifier = network.AttrId,
            VpcIdentifier = paymentsVpcId,
            SecurityGroupIds = [paymentsSgId],
        });

        // Fine-grained: only the Payments task role, only POST /v1/screenings.
        _ = new CfnAuthPolicy(this, "SanctionsPolicy", new CfnAuthPolicyProps
        {
            ResourceIdentifier = sanctions.AttrId,
            Policy = new PolicyDocument(new PolicyDocumentProps
            {
                Statements =
                [
                    new PolicyStatement(new PolicyStatementProps
                    {
                        Principals = [new ArnPrincipal(paymentsTaskRole.RoleArn)],
                        Actions = ["vpc-lattice-svcs:Invoke"],
                        Resources = [$"{sanctions.AttrArn}/v1/screenings"],
                        Conditions = new Dictionary<string, object>
                        {
                            ["StringEquals"] = new Dictionary<string, object> { ["vpc-lattice-svcs:RequestMethod"] = "POST" },
                        },
                    }),
                ],
            }).ToJSON(),
        });

        // The caller needs an identity-based allow as well.
        paymentsTaskRole.AddToPrincipalPolicy(new PolicyStatement(new PolicyStatementProps
        {
            Actions = ["vpc-lattice-svcs:Invoke"],
            Resources = [$"{sanctions.AttrArn}/*"],
        }));

        _ = new CfnOutput(this, "SanctionsDns", new CfnOutputProps { Value = sanctions.AttrDnsEntryDomainName });
    }
}

Complete code

The signing handler lives in the lab project labs/m04-sigv4-signer/src/Harbourline.SigV4/ (SigV4Signer, SigV4Options, SigV4DelegatingHandler, AwsSdkCredentialSource), where it is tested against AWS's published SigV4 vectors. Here is how a service wires it into HttpClientFactory, together with resilience, correlation propagation and a trusted customer header.

src/Harbourline.Payments/InternalClients.cs
using Amazon.Runtime.Credentials;
using Harbourline.SigV4;

namespace Harbourline.Payments;

public static class InternalClients
{
    public static IServiceCollection AddInternalClients(this IServiceCollection services, IConfiguration config)
    {
        services.AddSingleton<IAwsCredentialSource>(_ =>
            new AwsSdkCredentialSource(DefaultAWSCredentialsIdentityResolver.GetCredentials()));
        services.AddSingleton(TimeProvider.System);
        services.AddSingleton<SigV4Signer>();
        services.AddHttpContextAccessor();
        services.AddTransient<ContextPropagationHandler>();

        // Handlers run outermost → innermost in registration order.
        services.AddHttpClient<SanctionsClient>(c => c.BaseAddress = new Uri(config["Services:Sanctions"]!))
            .AddHttpMessageHandler<ContextPropagationHandler>()
            .AddResilienceHandler("sanctions", p =>
            {
                p.AddTimeout(TimeSpan.FromSeconds(3));
                p.AddCircuitBreaker(new Microsoft.Extensions.Http.Resilience.HttpCircuitBreakerStrategyOptions
                {
                    BreakDuration = TimeSpan.FromSeconds(15),
                });
            })
            .AddHttpMessageHandler(sp => Signing(sp, "vpc-lattice-svcs", SigV4Options.VpcLattice));

        // AddStandardResilienceHandler returns a pipeline builder, so the signing handler is added
        // with a second AddHttpClient call for the same typed client: it lands innermost.
        services.AddHttpClient<AccountsClient>(c => c.BaseAddress = new Uri(config["Services:Accounts"]!))
            .AddHttpMessageHandler<ContextPropagationHandler>()
            .AddStandardResilienceHandler().Configure(o => o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2));
        services.AddHttpClient<AccountsClient>()
            .AddHttpMessageHandler(sp => Signing(sp, "execute-api", SigV4Options.Default));

        return services;
    }

    // Typed clients: thin wrappers you write per downstream API (methods omitted here).
    public sealed class SanctionsClient(HttpClient http) { public HttpClient Http => http; }
    public sealed class AccountsClient(HttpClient http) { public HttpClient Http => http; }

    private static SigV4DelegatingHandler Signing(IServiceProvider sp, string service, SigV4Options options) =>
        new(sp.GetRequiredService<SigV4Signer>(), sp.GetRequiredService<IAwsCredentialSource>(), "us-east-1", service, options);
}
src/Harbourline.Payments/ContextPropagationHandler.cs
using System.Security.Claims;

namespace Harbourline.Payments;

/// <summary>
/// Copies the correlation id and the server-resolved customer id from the current request to the
/// outgoing call. The customer id comes from our own validated token processing, never from the
/// incoming request's headers.
/// </summary>
public sealed class ContextPropagationHandler(IHttpContextAccessor accessor) : DelegatingHandler
{
    public const string CorrelationHeader = "X-Correlation-ID";
    public const string CustomerHeader = "X-Harbourline-Customer-Id";

    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct)
    {
        var http = accessor.HttpContext;
        if (http is not null)
        {
            if (http.Items["CorrelationId"] is string correlationId)
                request.Headers.TryAddWithoutValidation(CorrelationHeader, correlationId);

            if (http.User.FindFirstValue("customer_id") is { } customerId)
                request.Headers.TryAddWithoutValidation(CustomerHeader, customerId);
        }
        return base.SendAsync(request, ct);
    }
}

On the receiving side, the Accounts API trusts X-Harbourline-Customer-Id only from callers it has authenticated with IAM and allow-listed. Behind an AWS_IAM method with a Lambda proxy integration, API Gateway puts the caller's ARN in the request context:

src/Harbourline.Accounts.Internal/TrustedCaller.cs
using Amazon.Lambda.APIGatewayEvents;

namespace Harbourline.Accounts.Internal;

public static class TrustedCaller
{
    // Roles allowed to act on behalf of a customer. Everything else may only read its own data.
    private static readonly string[] OnBehalfOfAllowList =
    [
        "arn:aws:sts::111122223333:assumed-role/hl-payments-task/",
    ];

    /// <summary>Returns the customer id to act for, or null if the caller may not assert one.</summary>
    public static string? CustomerFor(HttpContext http)
    {
        if (http.Items["LambdaRequestObject"] is not APIGatewayProxyRequest request) return null;

        // Set by API Gateway after SigV4 verification: the caller cannot forge it.
        var callerArn = request.RequestContext?.Identity?.UserArn;
        if (callerArn is null || !OnBehalfOfAllowList.Any(p => callerArn.StartsWith(p, StringComparison.Ordinal)))
            return null;

        var customerId = http.Request.Headers["X-Harbourline-Customer-Id"].ToString();
        return System.Text.RegularExpressions.Regex.IsMatch(customerId, "^CUS-[0-9]{8}$") ? customerId : null;
    }
}

Code explained line by line

  1. DefaultAWSCredentialsIdentityResolver.GetCredentials() — the SDK v4 default credential chain: environment, SSO profile, ECS container credentials, EC2 instance profile ⚠ VERIFY the exact type name in your SDK version (v3 called it FallbackCredentialsFactory).
  2. AwsSdkCredentialSource turns SDK credentials into the signer's AwsCredentials on every call, so rotated task-role credentials are picked up automatically.
  3. Handler order in HttpClientFactory is outermost first. Context propagation → resilience → signing means each retry passes through the signing handler again and gets a fresh X-Amz-Date and signature. Signing outside the retry would replay a signature that may be minutes old.
  4. SigV4Options.VpcLattice — UNSIGNED-PAYLOAD, the x-amz-content-sha256 header, and single-encoded paths, as VPC Lattice requires.
  5. The Sanctions client has no retry: POST /v1/screenings isn't safe to repeat unless the Sanctions Service de-duplicates on a key. The timeout and circuit breaker still protect the caller.
  6. AddStandardResilienceHandler() gives the Accounts client a sensible default pipeline (rate limiter, total timeout, retry, circuit breaker, attempt timeout) for GETs.
  7. AddHttpMessageHandler(sp => ...) with a factory builds a new signing handler per handler chain; don't try to wrap handlers inside each other by hand.
  8. ContextPropagationHandler reads customer_id from the current request's validated principal (resolved server-side, as in the module project), never by copying an incoming header — otherwise a caller could inject any customer ID.
  9. TrustedCaller.CustomerFor reads requestContext.identity.userArn, which API Gateway fills in after verifying the SigV4 signature. Only allow-listed roles may assert a customer; everyone else gets null and must be treated as "service acting for itself".
  10. The regex check on CUS- + 8 digits keeps junk out of queries and logs.

Security considerations

⚠ Learning shortcut — not for production

The allow-list of roles that may act on behalf of customers is hard-coded in TrustedCaller, and the Lattice policies use one account ID in examples. In production, keep the allow-list in configuration managed through code review, use aws:PrincipalOrgID and SCPs as guard rails, and review the trust relationships in your threat model. For the highest-risk flows (moving money), prefer a downstream-specific token from token exchange, so the downstream service sees proof of the customer's consent rather than another service's assertion.

  • One role per service, never shared. Shared roles destroy attribution and least privilege.
  • Authorise at the destination, not just the network. Security groups and private subnets are necessary but not sufficient; Lattice auth policies and API resource policies decide per request.
  • Encrypt in transit everywhere. Lattice HTTPS listeners, TLS on Service Connect, HTTPS on private APIs. Internal doesn't mean plaintext.
  • Events need authorisation too. Event bus and queue policies should allow specific producer roles and specific consumer rules. Consumers must validate event schemas and de-duplicate.
  • Log both identities on every on-behalf-of call: the service principal (ARN) and the customer ID, plus the correlation ID. This is what auditors and incident responders need.
  • Timeouts are a security control as well as a resilience one: unbounded calls let one slow dependency exhaust threads across the estate.

Banking use case

Harbourline moves its payment orchestration to Fargate. The Payments Service calls the Sanctions Service (owned by Financial Crime, in a separate account) through a VPC Lattice service network shared with AWS RAM; the auth policy allows only the Payments task role to POST /v1/screenings. It reads account status from the Accounts team's private REST API with AWS_IAM, passing the customer ID in a trusted header that the Accounts API honours only from hl-payments-task. Settlement publishes PaymentSettled to EventBridge; Notifications and the Data Platform subscribe. When Financial Crime rotates its Fargate tasks, nothing changes for Payments: the Lattice DNS name stays the same. The previous App Mesh setup is retired, as App Mesh reached end of support on 30 September 2026.

Common mistakes

  • Signing outside the retry handler → retries fail with Signature expired or replay old signatures → put signing innermost.
  • Signing the payload for VPC Lattice → requests rejected → UNSIGNED-PAYLOAD and the x-amz-content-sha256 header.
  • Trusting X-Customer-Id from anyone → any internal service can impersonate any customer → trust only allow-listed IAM callers.
  • Chaining five synchronous calls → latency and availability multiply → use events for anything that doesn't gate the response.
  • Using Cloud Map alone as "security" → discovery isn't authorisation → add IAM auth (Lattice, private API) or mTLS.
  • Starting new work on App Mesh → discontinued → Service Connect or Lattice.
  • Forgetting the caller's identity policy for Lattice or execute-api → 403 even though the destination policy allows it → both sides need an allow.

Troubleshooting

SymptomLikely causeFix
Lattice returns 403 AccessDeniedExceptionAuth policy doesn't allow the caller's role, path or method; or caller lacks vpc-lattice-svcs:InvokeCompare the role ARN, RequestPath and RequestMethod with both policies
Lattice 403 for every request after enabling AWS_IAMNo auth policy attached yet (default deny)Attach a policy to the network and service
Signature mismatch only for paths with spaces or %Double-encoding the path for LatticeUse SigV4Options.VpcLattice (single-encoded path)
Name resolves outside the VPC but times outVPC not associated with the service network, or security group blocks itAssociate the VPC; allow outbound 443 and the Lattice prefix list ⚠ VERIFY
Service Connect short name doesn't resolveClient service not in the same namespace or not in client modeEnable Service Connect on the client service
Private API call from Lambda times outLambda not attached to the VPC with the endpointConfigure VPC access and endpoint security group

Cost and cleanup

This lesson is labelled free because everything in the complete code runs locally with the lab's tests (labs/m04-sigv4-signer), and the console/CLI steps are optional. If you do create Lattice resources, a service costs about USD 0.025 per hour (USD 0.60 per day) plus negligible data and request charges at lab volumes — delete it the same day. Private API endpoints cost about USD 0.01 per AZ-hour ⚠ VERIFY.

Teardown (in this order: associations before the network):

aws vpc-lattice delete-auth-policy --resource-identifier $SVC_ID --profile learn-sandbox --region us-east-1
for a in $(aws vpc-lattice list-service-network-service-associations --service-network-identifier $SN_ID \
  --query "items[].id" --output text --profile learn-sandbox --region us-east-1); do
  aws vpc-lattice delete-service-network-service-association --service-network-service-association-identifier $a --profile learn-sandbox --region us-east-1
done
for a in $(aws vpc-lattice list-service-network-vpc-associations --service-network-identifier $SN_ID \
  --query "items[].id" --output text --profile learn-sandbox --region us-east-1); do
  aws vpc-lattice delete-service-network-vpc-association --service-network-vpc-association-identifier $a --profile learn-sandbox --region us-east-1
done
aws vpc-lattice delete-service --service-identifier $SVC_ID --profile learn-sandbox --region us-east-1
aws vpc-lattice delete-service-network --service-network-identifier $SN_ID --profile learn-sandbox --region us-east-1
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m04-l16 \
  --query "ResourceTagMappingList[].ResourceARN" --profile learn-sandbox --region us-east-1

Associations take a short time to delete; if a delete fails with a conflict, wait and retry.

Exercises

  • 🟢 Guided: Run the SigV4 lab tests. Then, in SigV4DelegatingHandlerTests, add a test proving that a request with a path containing a space is canonicalised differently for execute-api and for Lattice.
  • 🟡 Challenge: Draw Harbourline's payment flow (Accounts, Payments, Sanctions, Limits, Notifications, Ledger) and label each arrow with style (sync/event/queue), transport (Lattice, private API, Service Connect, EventBridge, SQS) and the IAM policy that authorises it.
  • 🔴 Live problem: After a deployment, the Payments Service gets intermittent 403s from the Sanctions Service through Lattice, about 1 in 20 requests, only under load. Nothing changed in the policies. Investigate and fix.

Assignment

Write an internal communication standard for Harbourline (one or two pages): when to use events vs synchronous calls, approved transports, identity and authorisation rules (one role per service, auth at the destination, on-behalf-of rules), required headers (correlation, trace), timeouts and retry rules, and a migration note for any App Mesh usage.

Solutions

Knowledge check

Interview questions

  • "How do microservices authenticate to each other on AWS?" Each service has its own IAM role; calls are SigV4-signed; the destination (private API Gateway, VPC Lattice) authorises with resource/auth policies plus the caller's identity policy. No shared secrets, full attribution.
  • "How do you pass the end user's identity to a downstream service?" Either forward or exchange the user's token so the downstream service can verify the user's consent, or pass a customer ID in a header that the downstream service accepts only from authenticated, allow-listed service identities, logging both. Never trust an unauthenticated header.
  • "Sync or async?" Sync only when the caller needs the answer to respond; otherwise events or queues, with idempotent consumers. Keep sync chains short and protected by timeouts and circuit breakers.
  • "What did you use instead of a service mesh?" Service Connect for intra-cluster ECS traffic, Lattice for cross-VPC/account with IAM auth, plus OpenTelemetry for tracing.

Certification practice

Completion checklist

  • I can choose between events, Service Connect, VPC Lattice and private APIs for a given interaction
  • I ran the SigV4 lab tests, including the execute-api and Lattice handler tests
  • I can write a Lattice auth policy for one role, one path and one method
  • I can explain the trusted-header pattern and its risks
  • I deleted any Lattice or endpoint resources I created

Key takeaways

  • Decide sync vs async first; keep synchronous chains short.
  • On AWS the workload identity is the IAM role; sign service calls with SigV4 and authorise at the destination with resource/auth policies plus identity policies.
  • VPC Lattice gives discovery, routing and per-request IAM auth across VPCs and accounts; Service Connect gives simple ECS discovery and resilience; private APIs suit internal API products. App Mesh is discontinued.
  • Propagate correlation IDs always; propagate customer context only through an authenticated, allow-listed path.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page