Private networking patterns: keeping traffic off the internet end to end
Design workloads whose traffic never touches the public internet — NAT-free subnets, gateway and interface endpoints, private DNS, private API Gateway APIs, PrivateLink services, VPC Lattice — and enforce it with endpoint and resource policies.
Assumes: .NET 10 LTS (file-based app) · AWS SDK for .NET v4 (AWSSDK.EC2 4.1.4) · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain the difference between "in a private subnet" and "never on the internet", and trace every hop of a request to see which one you have.
- Choose the right private path for each kind of traffic: gateway endpoint, interface endpoint, PrivateLink endpoint service, private API Gateway API, VPC Lattice, Transit Gateway or Direct Connect.
- Make private DNS work so that applications use normal service names (
secretsmanager.us-east-1.amazonaws.com) and still stay private. - Enforce "private only" with endpoint policies, resource policies (
aws:SourceVpce,aws:SourceVpc) and service control policies — so a mistake can't quietly route around you. - Audit a VPC with a small C# tool that lists every route that leaves the VPC and every endpoint that keeps traffic in.
Prerequisites
- Amazon VPC fundamentals and CIDR planning.
- The lessons on subnets, route tables, NAT gateways and VPC endpoints listed in the module overview.
- Tools: AWS CLI v2 with the
learn-sandboxprofile, .NET 10 SDK.
Why this matters in banking
A bank's security architecture usually contains a sentence like "Customer data in transit between internal components must not traverse the public internet." It sounds simple. In practice, a .NET service in a "private" subnet that calls Secrets Manager, S3, SQS and a partner API through a NAT gateway is sending all of that traffic to public AWS endpoints over public IP addresses. It is encrypted with TLS and it stays on the AWS backbone in most cases, but from the architecture review's point of view it is internet-routed: it leaves through an internet gateway, it needs a public IP, and it can reach anything on the internet, including an attacker's server.
There are also practical reasons:
- Data exfiltration. If an attacker gets code execution inside a workload, a NAT gateway is an open door. A NAT-free design with endpoints and policies limits what stolen credentials can reach.
- Cost. NAT gateways charge per GB processed. A data pipeline pulling terabytes from S3 through NAT pays twice for nothing; a gateway endpoint is free.
- Regulators and auditors ask for network diagrams that show every path out. "Private by default" is far easier to evidence than "public but locked down".
The idea in plain language
"Private" is a property of the whole path, not of the subnet. A request is private end to end only if:
- the client has no public IP and no route to an internet gateway for that traffic,
- the destination is reached by a private address (inside your VPC, a peered network, or an endpoint ENI), and
- the destination refuses requests that arrive any other way.
AWS gives you several building blocks for step 2, depending on what you are calling:
| You are calling… | Private path |
|---|---|
| S3 or DynamoDB in the same Region | Gateway endpoint — a route table entry, free |
| Most other AWS service APIs (Secrets Manager, SQS, KMS, ECR, CloudWatch Logs, STS, …) | Interface endpoint — ENIs with private IPs in your subnets (AWS PrivateLink), hourly + per GB |
| Your own service in another VPC or account | PrivateLink endpoint service (behind an NLB), or VPC Lattice |
| Your own REST API on API Gateway | Private API reached through the execute-api interface endpoint |
| Another VPC's whole network | VPC peering or Transit Gateway (next lesson) |
| On-premises | Site-to-Site VPN or Direct Connect (hybrid connectivity) |
| A partner on the internet | Not private by nature. Use egress filtering (AWS Network Firewall) or ask the partner for PrivateLink |
Step 3 is where policies come in: an S3 bucket policy can deny any request that did not come through your endpoint, so even a valid credential used from a laptop is refused.
Analogy: a bank's internal post. A letter between two branches can go by public mail (encrypted envelope, but it travels through the public postal system and any postbox can send one) or by the bank's own courier van (private route, only staff can put letters in, and the receiving branch refuses anything not delivered by the van). Endpoints are the van; policies are the receiving branch's rule.
What problem the AWS services solve
| Building block | What it is | Price (us-east-1, checked 2026-11-18) | Sharp edges |
|---|---|---|---|
| Gateway endpoint (S3, DynamoDB) | A prefix-list route in chosen route tables | Free | Same Region only; not reachable from on-premises or peered VPCs |
| Interface endpoint | One ENI per chosen AZ, private DNS optional | About USD 0.01 per endpoint-AZ-hour + USD 0.01 per GB ⚠ VERIFY on the PrivateLink pricing page | Cost adds up: 10 services × 3 AZs ≈ USD 219/month per VPC |
| PrivateLink endpoint service | Expose your NLB-fronted service to consumers' VPCs | Consumers pay the endpoint price | One-directional (consumer → provider); NLB required (or GWLB for appliances) |
| Private REST API (API Gateway) | API reachable only via execute-api interface endpoints | API Gateway request pricing + endpoint | Needs a resource policy; private DNS on the endpoint affects all API Gateway calls from the VPC |
| VPC Lattice | Application-layer service network across VPCs/accounts with IAM auth | Per service-hour, per GB and per request ⚠ VERIFY | Newer; not every protocol; overlaps with PrivateLink and service meshes |
Policies (aws:SourceVpce, aws:SourceVpc) | Conditions in resource, endpoint and SCP policies | Free | Easy to lock yourself out of the console; test in a sandbox |
A useful rule of thumb from the cost column: gateway endpoints for S3/DynamoDB always; interface endpoints for the handful of services each workload really uses; centralise interface endpoints in a hub VPC when you have many VPCs (see multi-account networking).
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Service endpoints | Gateway endpoints (S3, DynamoDB only) | AWS gateway endpoints are route-table entries; Azure service endpoints are subnet settings |
Private Endpoint + Private DNS zone privatelink.* | Interface endpoint with private DNS enabled | AWS manages the private hosted zone for you when private DNS is on |
| Private Link Service (behind Standard LB) | PrivateLink endpoint service (behind NLB) | Similar model; AWS uses "acceptance" and allowed principals |
| API Management internal mode / private endpoint | API Gateway private REST API | Resource policy decides which VPC endpoints may call it |
| Azure Policy "deny public network access" | SCPs + resource policies with aws:SourceVpce / aws:SourceVpc | AWS enforces at request time by condition keys, not at resource creation |
| NAT gateway | NAT gateway | Same role; AWS charges per GB processed as well as hourly |
Architecture
Anti-pattern: "private subnet, public path"
Target pattern: NAT-free and policy-enforced
Private DNS: how normal names stay private
The application code does not change. The AWS SDK for .NET calls the normal Regional endpoint name; the VPC resolver answers with the endpoint ENIs' private IPs because private DNS is enabled on the interface endpoint and the VPC has enableDnsSupport and enableDnsHostnames turned on. TLS still validates, because the certificate is for the public service name.
Step-by-step implementation
1. Inventory what each workload calls
List every outbound dependency of the service: AWS services, internal services, partners, package registries, telemetry. For a typical Harbourline ASP.NET Core API on ECS Fargate: ECR (ecr.api, ecr.dkr, plus S3 for image layers), CloudWatch Logs, Secrets Manager, STS (for IAM roles), SQS, KMS, S3 and DynamoDB.
2. Remove the default route from the app tier
Take 0.0.0.0/0 → NAT out of the app route tables (or never add it). From now on, anything the app needs must have an explicit private path. Build failures and timeouts will tell you what you missed — faster in dev than in an audit.
3. Add gateway endpoints for S3 and DynamoDB
Associate them with the app route tables. They are free and remove the most expensive NAT traffic.
4. Add interface endpoints for the services you listed
Put them in a dedicated endpoint subnet per AZ (the "spare" tier block in the IP plan is ideal), with a security group that allows HTTPS 443 only from the app security group. Enable private DNS.
5. Write endpoint policies
Every endpoint can carry a policy (gateway endpoints and most interface endpoints). Restrict them to your own resources, for example aws:ResourceAccount equals your account, or aws:ResourceOrgID equals your Organization. This stops a compromised workload from copying data to an attacker's bucket through your endpoint.
6. Lock the destinations with resource policies
Add Deny statements to bucket policies, secret policies, KMS key policies or API resource policies so requests must come through your endpoint (aws:SourceVpce) or VPC (aws:SourceVpc). Include an exception for break-glass and deployment roles, or you will lock out your pipeline.
7. Prove it
Run the audit tool below, then test from an instance: nslookup secretsmanager.us-east-1.amazonaws.com must return private IPs; a call to an S3 bucket in another account must be denied; curl https://example.com must time out.
AWS Console instructions
- Interface endpoint: Console → VPC → Endpoints → Create endpoint → AWS services → search
secretsmanager→ choosecom.amazonaws.us-east-1.secretsmanager→ VPC → Additional settings: Enable DNS name ticked → subnets (one per AZ, endpoint tier) → security grouphl-endpoints-sg→ policy Custom → tags → Create endpoint. - Gateway endpoint: same page, choose
com.amazonaws.us-east-1.s3with type Gateway, tick the app route tables. - Private REST API: Console → API Gateway → Create API → REST API → Build → Endpoint type Private, then Resource policy (left menu) to allow only your
execute-apiendpoint, then Deploy API. - Endpoint service: Console → VPC → Endpoint services → Create endpoint service → choose your NLB → Require acceptance on → add allowed principals (account or role ARNs).
AWS CLI
$p = '--profile', 'learn-sandbox', '--region', 'us-east-1'
$vpc = 'vpc-0123456789abcdef0'
# Gateway endpoint for S3 on the app route tables (free)
aws ec2 create-vpc-endpoint @p --vpc-id $vpc --vpc-endpoint-type Gateway `
--service-name com.amazonaws.us-east-1.s3 --route-table-ids rtb-0app0000000000a rtb-0app0000000000b `
--policy-document file://s3-endpoint-policy.json `
--tag-specifications "ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=hl-s3-gw},{Key=Project,Value=cea-lab},{Key=Lab,Value=m02-l11}]"
# Interface endpoint for Secrets Manager with private DNS, in the endpoint subnets
aws ec2 create-vpc-endpoint @p --vpc-id $vpc --vpc-endpoint-type Interface `
--service-name com.amazonaws.us-east-1.secretsmanager `
--subnet-ids subnet-0ep000000000000a subnet-0ep000000000000b `
--security-group-ids sg-0endpoints000000 --private-dns-enabled `
--tag-specifications "ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=hl-secretsmanager},{Key=Project,Value=cea-lab},{Key=Lab,Value=m02-l11}]"
# Which interface endpoints exist, and is private DNS on?
aws ec2 describe-vpc-endpoints @p --filters "Name=vpc-id,Values=$vpc" `
--query "VpcEndpoints[].{Service:ServiceName,Type:VpcEndpointType,PrivateDns:PrivateDnsEnabled,State:State}" --output table
# Which services offer interface endpoints in this Region?
aws ec2 describe-vpc-endpoint-services @p --query "ServiceNames[?contains(@, 'secretsmanager') || contains(@, 'execute-api')]"An endpoint policy that only allows access to buckets owned by your own account:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "OnlyOurAccountsBuckets",
"Effect": "Allow",
"Principal": "*",
"Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
"Resource": "*",
"Condition": { "StringEquals": { "aws:ResourceAccount": "111122223333" } }
}
]
}A bucket policy that refuses anything not arriving through the endpoint (with a break-glass exception):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DenyUnlessThroughOurEndpoint",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::hl-statements-111122223333",
"arn:aws:s3:::hl-statements-111122223333/*"
],
"Condition": {
"StringNotEquals": { "aws:SourceVpce": "vpce-0s3gateway0000000" },
"ArnNotLike": { "aws:PrincipalArn": "arn:aws:iam::111122223333:role/hl-breakglass" }
}
}
]
}Infrastructure as Code
using Amazon.CDK.AWS.EC2;
using Amazon.CDK.AWS.IAM;
using Constructs;
namespace Harbourline.Networking.Private;
/// <summary>Gateway endpoints for S3/DynamoDB plus interface endpoints for the services a .NET app on Fargate needs.</summary>
public sealed class PrivateEndpoints : Construct
{
public PrivateEndpoints(Construct scope, string id, IVpc vpc, ISecurityGroup appSecurityGroup)
: base(scope, id)
{
var appSubnets = new SubnetSelection { SubnetGroupName = "app" };
var endpointSubnets = new SubnetSelection { SubnetGroupName = "endpoints" };
vpc.AddGatewayEndpoint("S3", new GatewayVpcEndpointOptions { Service = GatewayVpcEndpointAwsService.S3, Subnets = [appSubnets] });
vpc.AddGatewayEndpoint("DynamoDb", new GatewayVpcEndpointOptions { Service = GatewayVpcEndpointAwsService.DYNAMODB, Subnets = [appSubnets] });
var endpointSg = new SecurityGroup(this, "EndpointSg", new SecurityGroupProps
{
Vpc = vpc,
Description = "Interface endpoints: HTTPS from the app tier only",
AllowAllOutbound = false,
});
endpointSg.AddIngressRule(appSecurityGroup, Port.Tcp(443), "HTTPS from app tier");
InterfaceVpcEndpointAwsService[] services =
[
InterfaceVpcEndpointAwsService.SECRETS_MANAGER,
InterfaceVpcEndpointAwsService.SQS,
InterfaceVpcEndpointAwsService.CLOUDWATCH_LOGS,
InterfaceVpcEndpointAwsService.STS,
InterfaceVpcEndpointAwsService.ECR,
InterfaceVpcEndpointAwsService.ECR_DOCKER,
InterfaceVpcEndpointAwsService.KMS,
];
foreach (var service in services)
{
var endpoint = vpc.AddInterfaceEndpoint(service.ShortName, new InterfaceVpcEndpointOptions
{
Service = service,
Subnets = endpointSubnets,
SecurityGroups = [endpointSg],
PrivateDnsEnabled = true,
Open = false, // we wrote the one ingress rule we want
});
// Endpoint policy: only principals and resources in our Organization.
endpoint.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
Effect = Effect.ALLOW,
Principals = [new AnyPrincipal()],
Actions = ["*"],
Resources = ["*"],
Conditions = new Dictionary<string, object>
{
["StringEquals"] = new Dictionary<string, object>
{
["aws:PrincipalOrgID"] = "o-exampleorgid",
["aws:ResourceOrgID"] = "o-exampleorgid",
},
},
}));
}
}
}Not every service supports endpoint policies, and some (for example ECR image layer pulls) also need the S3 gateway endpoint because layers are stored in S3 ⚠ VERIFY the per-service list before relying on a policy.
Private DNS on the execute-api endpoint affects every API Gateway call from the VPC
With private DNS enabled on an execute-api interface endpoint, all *.execute-api.us-east-1.amazonaws.com names resolve to that endpoint from inside the VPC — including public APIs you might call (a partner's API Gateway API, for example). Those calls will now fail, because public APIs don't accept traffic through your endpoint. Either disable private DNS and call the private API by its endpoint-specific DNS name (or a custom domain), or keep public API calls out of that VPC.
Complete code
A .NET 10 file-based app that audits a VPC for privacy: for each subnet, it reports whether the subnet has a route to the internet (IGW, NAT, egress-only IGW), which gateway endpoints its route table uses, and which interface endpoints exist in the VPC with private DNS. Use it before a design review, or in a pipeline after deployment.
#:package AWSSDK.EC2@4.1.4
#:package AWSSDK.SSO@4.1.0.2
#:package AWSSDK.SSOOIDC@4.1.0.2
using Amazon;
using Amazon.EC2;
using Amazon.EC2.Model;
using Amazon.Runtime.CredentialManagement;
// Usage: dotnet run privacyaudit.cs -- vpc-0123456789abcdef0 [--strict app,data]
// --strict lists Tier tag values that must have NO internet route; the exit code is 3 if any do.
if (args.Length < 1)
{
Console.Error.WriteLine("Usage: dotnet run privacyaudit.cs -- <vpc-id> [--strict tier1,tier2]");
return 1;
}
var vpcId = args[0];
var strictTiers = args.Length >= 3 && args[1] == "--strict"
? args[2].Split(',', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries).ToHashSet(StringComparer.OrdinalIgnoreCase)
: new HashSet<string>(StringComparer.OrdinalIgnoreCase);
if (!new CredentialProfileStoreChain().TryGetAWSCredentials("learn-sandbox", out var credentials))
{
Console.Error.WriteLine("Profile learn-sandbox not found. Run: aws sso login --profile learn-sandbox");
return 1;
}
using var ec2 = new AmazonEC2Client(credentials, RegionEndpoint.USEast1);
var vpcFilter = new List<Filter> { new() { Name = "vpc-id", Values = [vpcId] } };
var subnets = await ec2.DescribeSubnetsAsync(new DescribeSubnetsRequest { Filters = vpcFilter });
var tables = await ec2.DescribeRouteTablesAsync(new DescribeRouteTablesRequest { Filters = vpcFilter });
var endpoints = await ec2.DescribeVpcEndpointsAsync(new DescribeVpcEndpointsRequest { Filters = vpcFilter });
// A subnet without an explicit association uses the VPC's main route table.
var mainTable = tables.RouteTables?.FirstOrDefault(t => t.Associations?.Any(a => a.Main == true) == true);
RouteTable TableFor(string subnetId) =>
tables.RouteTables?.FirstOrDefault(t => t.Associations?.Any(a => a.SubnetId == subnetId) == true)
?? mainTable
?? throw new InvalidOperationException($"No route table for {subnetId}");
// Map gateway endpoint prefix lists (pl-...) to service names for readable output.
var prefixToService = (endpoints.VpcEndpoints ?? [])
.Where(e => e.VpcEndpointType == VpcEndpointType.Gateway)
.SelectMany(e => (e.RouteTableIds ?? []).Select(rt => (rt, e.ServiceName)))
.ToLookup(x => x.rt, x => x.ServiceName);
var violations = 0;
Console.WriteLine($"{"Subnet",-26} {"Tier",-8} {"AZ",-11} {"Internet route",-26} Gateway endpoints");
foreach (var subnet in (subnets.Subnets ?? []).OrderBy(s => s.CidrBlock))
{
var tier = subnet.Tags?.FirstOrDefault(t => t.Key == "Tier")?.Value ?? "-";
var table = TableFor(subnet.SubnetId);
var internetRoutes = (table.Routes ?? [])
.Where(r => r.State == RouteState.Active)
.Where(r => r.DestinationCidrBlock == "0.0.0.0/0" || r.DestinationIpv6CidrBlock == "::/0")
.Select(r => r.NatGatewayId ?? r.EgressOnlyInternetGatewayId ?? r.GatewayId ?? r.TransitGatewayId ?? r.NetworkInterfaceId ?? r.VpcPeeringConnectionId ?? "?")
.Where(target => target.StartsWith("igw-", StringComparison.Ordinal)
|| target.StartsWith("nat-", StringComparison.Ordinal)
|| target.StartsWith("eigw-", StringComparison.Ordinal))
.ToList();
var gatewayServices = string.Join(", ", prefixToService[table.RouteTableId].Select(s => s.Split('.').Last()));
var internet = internetRoutes.Count == 0 ? "none" : string.Join(",", internetRoutes);
var flag = internetRoutes.Count > 0 && strictTiers.Contains(tier) ? " <-- VIOLATION" : string.Empty;
if (flag.Length > 0)
{
violations++;
}
Console.WriteLine($"{subnet.SubnetId,-26} {tier,-8} {subnet.AvailabilityZone,-11} {internet,-26} {gatewayServices}{flag}");
}
Console.WriteLine();
Console.WriteLine("Interface endpoints:");
foreach (var e in (endpoints.VpcEndpoints ?? []).Where(e => e.VpcEndpointType == VpcEndpointType.Interface).OrderBy(e => e.ServiceName))
{
// Crude but useful: the default endpoint policy has no Condition block and allows everything.
var policyNote = e.PolicyDocument?.Contains("Condition", StringComparison.Ordinal) == true
? "conditional policy"
: "no conditions (likely full access)";
Console.WriteLine($" {e.ServiceName,-48} privateDns={e.PrivateDnsEnabled,-5} subnets={e.SubnetIds?.Count ?? 0} {policyNote}");
}
Console.WriteLine();
Console.WriteLine(violations == 0
? "RESULT: no strict tier has an internet route."
: $"RESULT: {violations} subnet(s) in strict tiers have an internet route.");
return violations == 0 ? 0 : 3;Run it against the secure VPC lab VPC (the Terraform version tags subnets with Tier):
dotnet run privacyaudit.cs -- vpc-0123456789abcdef0 --strict app,dataExpected (IDs differ):
Subnet Tier AZ Internet route Gateway endpoints
subnet-0aaa... public us-east-1a igw-0abc...
subnet-0bbb... public us-east-1b igw-0abc...
subnet-0ccc... app us-east-1a none s3, dynamodb
subnet-0ddd... app us-east-1b none s3, dynamodb
subnet-0eee... data us-east-1a none
subnet-0fff... data us-east-1b none
Interface endpoints:
RESULT: no strict tier has an internet route.Code explained line by line
#:package AWSSDK.EC2@4.1.4— the EC2 client from SDK v4; the SSO packages let the SDK use your Identity Center profile at runtime.strictTiers— the tiers you claim are private. The tool is a test of a claim, not just a report.CredentialProfileStoreChain().TryGetAWSCredentials("learn-sandbox", …)— loads the SSO profile; no keys in code.- Three
Describe…calls filtered byvpc-idfetch subnets, route tables and endpoints. In SDK v4, collection properties can benullwhen empty, hence?? []everywhere. mainTableandTableFor— a subnet with no explicit association uses the main route table. Forgetting this is a classic audit bug: the main table may have a NAT route nobody looks at.prefixToService— gateway endpoints appear in route tables as prefix-list routes (pl-…); the lookup maps each route table to the services it reaches privately.- The
internetRoutesquery keeps only active default routes (0.0.0.0/0or::/0) whose target is an internet gateway (igw-), NAT gateway (nat-) or egress-only internet gateway (eigw-). A default route to a Transit Gateway is not counted — in a centralised design that is the private path to an inspection VPC. flagmarks a violation only when a strict tier has an internet route; the exit code becomes 3, so a pipeline step can fail the deployment.- The interface-endpoint loop prints private DNS status and a crude policy hint: the default endpoint policy has no
Conditionand allows everything, which is worth flagging in a review.
Security considerations
⚠ Learning shortcut — not for production
The examples put interface endpoints in every workload VPC. At bank scale (hundreds of VPCs × a dozen services × three AZs) that is expensive and hard to govern. Production usually centralises interface endpoints in a shared-services VPC and shares private DNS through Route 53 Resolver rules or private hosted zones associated across accounts — see multi-account network architecture.
⚠ Learning shortcut — not for production
The bucket policy example has one break-glass exception. A real data perimeter also covers the deployment pipeline's role, AWS services acting on your behalf (aws:PrincipalIsAWSService, aws:ViaAWSService), and replication — and it is rolled out in audit mode first (using IAM Access Analyzer and CloudTrail) so you don't break production.
- Endpoint security groups should admit 443 only from the tiers that need the service, not from the whole VPC CIDR.
- Interface endpoints are not a firewall for the internet. They only cover AWS services (and PrivateLink services). If any tier still has a NAT route, data can leave by that path; remove it or filter it with Network Firewall.
- Policies are the enforcement; routes are the convenience. Without
aws:SourceVpceconditions, a stolen credential works perfectly well from outside your network. - DNS is part of the attack surface. Use Route 53 Resolver DNS Firewall to block lookups of unknown domains, and resolver query logging for investigations.
- Encryption still matters. Private paths do not replace TLS. Keep TLS on for every call; private does not mean trusted.
Banking use case
Harbourline's Accounts API (Module 4) runs on ECS Fargate in the Accounts VPC (10.18.0.0/16). The target design:
- The app tier has no NAT route. It reaches ECR, CloudWatch Logs, Secrets Manager, STS and KMS through interface endpoints in a shared-services VPC, and S3/DynamoDB through local gateway endpoints.
- The internal Payments API in the Payments account is exposed to the Accounts VPC as a PrivateLink endpoint service behind an NLB. Only the Accounts account's principals are allowed, and acceptance is manual so the Payments team approves each new consumer.
- Branch staff tools call an internal private REST API on API Gateway through the
execute-apiendpoint; the resource policy allows only that endpoint. - The statements bucket denies any request not arriving through the Accounts VPC's S3 gateway endpoint, except a break-glass role whose use pages the security team.
- The design review evidence is the audit tool's output (
RESULT: no strict tier has an internet route) plus a Network Access Analyzer scope showing no path from any internet gateway to the app or data tiers.
Common mistakes
- Calling a subnet "private" because it has no public IPs, while its route table sends
0.0.0.0/0to a NAT gateway. - Forgetting the main route table, which subnets use when they have no explicit association.
- Interface endpoint without private DNS (or VPC DNS attributes off) → the SDK still resolves public IPs, and with no NAT route the call times out.
- ECR pulls failing because the
ecr.apiandecr.dkrendpoints exist but the S3 gateway endpoint for image layers does not. - Endpoint security group allows the whole VPC — fine for a lab, too broad for a bank.
- Private DNS on
execute-apibreaks public API calls from the same VPC (see callout). - Locking a bucket to a VPC endpoint without exceptions, then discovering the deployment pipeline and the console can no longer manage it.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| SDK call times out after removing NAT | No endpoint for that service, or private DNS off | Create the interface endpoint with private DNS; check nslookup returns private IPs |
nslookup still returns public IPs | enableDnsSupport/enableDnsHostnames off, or custom DNS server not forwarding to the VPC resolver | Turn both VPC attributes on; forward to the VPC+2 resolver |
AccessDenied through the endpoint | Endpoint policy too narrow | Add the action/resource; check aws:ResourceAccount value |
AccessDenied from the console on a bucket | aws:SourceVpce deny without an exception | Use the break-glass role or add a principal exception |
| ECS task cannot pull image | Missing ecr.dkr, ecr.api or S3 gateway endpoint | Add all three; check the endpoint SG allows 443 from tasks |
| Partner's public API Gateway API fails from VPC | execute-api endpoint with private DNS | Disable private DNS on that endpoint, or call partner from a different VPC |
Endpoint service connection stuck in pendingAcceptance | Provider has acceptance required | Provider accepts in Endpoint services → Endpoint connections |
Cost and cleanup
Cost (us-east-1, checked 2026-11-18 unless marked):
- Gateway endpoints: free.
- Interface endpoints: about USD 0.01 per endpoint per AZ per hour plus USD 0.01 per GB processed ⚠ VERIFY on the PrivateLink pricing page. Seven services × two AZs × 730 hours ≈ USD 102/month per VPC.
- NAT gateway (for comparison): USD 0.045 per hour plus USD 0.045 per GB. Moving 2 TB/month of S3 traffic from NAT to a gateway endpoint saves about USD 90/month in processing alone.
- API Gateway private REST API: normal REST API request pricing; no extra charge for "private" beyond the endpoint.
Free path for this lesson: add only the S3 and DynamoDB gateway endpoints (free) to the lab VPC, run privacyaudit.cs, and design the interface-endpoint layout on paper. If you create interface endpoints, delete them in the same session (about USD 0.02 per endpoint per hour for two AZs).
Cleanup:
$p = '--profile', 'learn-sandbox', '--region', 'us-east-1'
aws ec2 describe-vpc-endpoints @p --filters Name=tag:Lab,Values=m02-l11 --query "VpcEndpoints[].VpcEndpointId" --output text
aws ec2 delete-vpc-endpoints @p --vpc-endpoint-ids vpce-0aaa0000000000001 vpce-0bbb0000000000002
aws resourcegroupstaggingapi get-resources @p --tag-filters Key=Project,Values=cea-lab --query "ResourceTagMappingList[].ResourceARN"Exercises
- 🟢 Guided: In the lab VPC, run
privacyaudit.cswith--strict app,data. Then temporarily add a0.0.0.0/0route from the data route table to the internet gateway, re-run, see the violation and exit code 3, and remove the route. - 🟡 Challenge: Design the endpoint list, subnets, security groups and endpoint policies for an ECS Fargate service that pulls images from ECR, writes logs to CloudWatch, reads secrets, sends to SQS and reads S3 — with no NAT. Estimate the monthly endpoint cost for two AZs.
- 🔴 Live problem: After the network team removed the NAT gateway from the Accounts VPC, the Accounts API works, but the nightly statement job (a .NET worker) fails with
Unable to get IAM security credentials from EC2 Instance Metadata Serviceon ECS and, after that is fixed, with timeouts callingsts.amazonaws.com. Explain both failures and the fix.
Assignment
Write a "private by default" design note (one page) for a Harbourline service of your choice: dependency inventory, private path per dependency, endpoint policies, resource policies, the audit command and its expected output, and the monthly cost. Add privacyaudit.cs to your portfolio repository with a README.
Solutions
aws ec2 create-route --profile learn-sandbox --route-table-id <data-rt> --destination-cidr-block 0.0.0.0/0 --gateway-id <igw>
dotnet run privacyaudit.cs -- <vpc-id> --strict app,data # data subnets show igw-... and VIOLATION, exit code 3
aws ec2 delete-route --profile learn-sandbox --route-table-id <data-rt> --destination-cidr-block 0.0.0.0/0Why it matters: the tool turns "the data tier is private" into a check that fails loudly. Note that the CDK test Data_tier_route_tables_have_no_default_route in the lab catches the same mistake before deployment; the audit catches out-of-band changes after.
| Dependency | Path | Notes |
|---|---|---|
| ECR API | Interface ecr.api | Auth and manifests |
| ECR registry | Interface ecr.dkr | Docker registry API |
| Image layers | S3 gateway | Layers are served from S3 |
| CloudWatch Logs | Interface logs | awslogs driver |
| Secrets Manager | Interface secretsmanager | Task secrets |
| SQS | Interface sqs | |
| S3 data | S3 gateway (same endpoint) | Policy: our account's buckets only |
| STS | Interface sts | If the app assumes roles explicitly |
Six interface endpoints × 2 AZs × 730 h × USD 0.01 ≈ USD 88/month plus per-GB charges. Endpoint SG: 443 from the task SG only. Endpoint policies: aws:PrincipalOrgID and aws:ResourceOrgID equal to your Organization, or aws:ResourceAccount for single-account designs.
- Credentials error: the worker was using the EC2 instance metadata credential provider instead of the ECS task role, because an old configuration forced
AWS_EC2_METADATA_*settings or the task had no task role. On Fargate the SDK should use the container credentials endpoint (AWS_CONTAINER_CREDENTIALS_RELATIVE_URI), which is local and needs no NAT. Fix: assign a task role and remove the override. - STS timeouts: the worker calls the global STS endpoint
sts.amazonaws.com, which resolves to a public IP (the interface endpoint coverssts.us-east-1.amazonaws.com). Without NAT the call times out. Fix: use the Regional STS endpoint (SDK v4 uses Regional STS by default; check for an explicit global configuration such asAWS_STS_REGIONAL_ENDPOINTS=legacy⚠ VERIFY), and add thestsinterface endpoint.
Lesson: removing NAT is a great way to discover hidden internet dependencies — do it in dev first.
Knowledge check
Answer: Amazon S3 and Amazon DynamoDB. Gateway endpoints are free.
Answer: The VPC resolver answers the normal service name with the endpoint ENIs' private IPs, so the SDK connects privately to the same name and TLS still validates.
Answer: A bucket policy that denies requests not arriving through your endpoint (aws:SourceVpce) or VPC (aws:SourceVpc). Endpoints create a path; policies close the others.
Answer: An endpoint policy limits what can be done through that endpoint (any principal, any resource). A resource policy limits who can access that resource and from where. Use both for a data perimeter.
Answer: When consumers should reach one service, not a whole network; when CIDRs overlap; or when the provider wants to approve each consumer. Peering connects whole VPCs in both directions.
Interview questions
- "What does 'private' mean for a workload on AWS?" — No internet route for the traffic, private destinations (endpoints, PrivateLink, peering, TGW, DX), and resource policies that refuse anything else. A private subnet with NAT is only half of that.
- "How would you stop a compromised container exfiltrating data to the attacker's S3 bucket?" — No NAT, an S3 gateway endpoint whose policy only allows our Organization's buckets (
aws:ResourceOrgID), and a VPC with no other route out. Add Network Firewall if some internet egress is required. - "Gateway or interface endpoint for S3?" — Gateway for in-VPC workloads (free). Interface endpoints for S3 when on-premises or peered networks must reach S3 privately, since gateway endpoints aren't reachable from outside the VPC.
- "How do you expose an internal API to another account without peering?" — A PrivateLink endpoint service behind an NLB with allowed principals and acceptance, or a private API Gateway API with a resource policy naming the consumer's endpoint, or VPC Lattice with IAM auth.
Certification practice
An application runs on EC2 instances in private subnets with no NAT gateway. It must read objects from Amazon S3 and secrets from AWS Secrets Manager in the same Region, without traffic traversing the internet, at the lowest cost. What should a solutions architect do?
A. Add a NAT gateway and restrict its security group to S3 and Secrets Manager B. Create an S3 gateway endpoint and a Secrets Manager interface endpoint with private DNS enabled C. Create interface endpoints for both S3 and Secrets Manager without private DNS D. Peer the VPC with a VPC that has a NAT gateway
Answer: B. The S3 gateway endpoint is free; Secrets Manager needs an interface endpoint, and private DNS lets the SDK use the normal name. A uses the internet path (and NAT gateways have no security groups). C works for S3 but costs more, and without private DNS the SDK won't use the Secrets Manager endpoint unless reconfigured. D: peering does not provide transitive NAT access.
A security engineer must ensure that objects in a sensitive S3 bucket can be accessed only from the company's production VPC, even by principals with valid credentials. Which control achieves this?
A. An S3 gateway endpoint policy that allows only the sensitive bucket
B. A bucket policy that denies all S3 actions when aws:SourceVpce is not the production VPC's S3 gateway endpoint, with an exception for a break-glass role
C. Enable S3 Block Public Access on the bucket
D. Enable default encryption with an AWS KMS key
Answer: B. The resource policy is evaluated for every request, whichever network it comes from. A only restricts what passes through that endpoint; requests from outside still work. C blocks public grants, not authenticated access from elsewhere. D protects data at rest, not the access path.
Developers created an API Gateway private REST API and an execute-api interface endpoint. Calls from the VPC return HTTP 403 Forbidden. DNS resolves to the endpoint's private IPs. What is the most likely fix?
A. Enable a NAT gateway
B. Attach a resource policy to the API that allows execute-api:Invoke from the VPC endpoint, then redeploy the API
C. Change the endpoint type to Regional
D. Add an inbound rule for port 80 to the endpoint security group
Answer: B. Private APIs require a resource policy allowing the endpoint or VPC, and the API must be redeployed for the policy to take effect. A is unrelated. C makes the API public. D: the API uses HTTPS 443; DNS and connectivity already work, so the 403 is authorisation.
Completion checklist
- I can explain why "private subnet + NAT" is not "private end to end"
- I can pick the right private path for an AWS service, an internal service, an API and on-premises
- I know what private DNS does and when it causes surprises
- I wrote an endpoint policy and a resource policy with
aws:SourceVpce - I ran
privacyaudit.csand deleted any interface endpoints I created
Key takeaways
- Privacy is a property of the whole path: no internet route, private destination, and a destination that refuses other paths.
- Gateway endpoints (S3, DynamoDB) are free — always add them. Interface endpoints cost money — add what you need, centralise at scale.
- Private DNS keeps code unchanged; check its side effects (especially
execute-api). - Endpoint policies limit what flows through; resource policies limit where requests can come from. Use both.
- Make the claim testable: an audit with an exit code beats a diagram.
Further reading
- AWS PrivateLink concepts
- Gateway endpoints
- Private REST APIs in API Gateway
- AWS PrivateLink pricing
- Amazon VPC Lattice
- Segmentation, private endpoints and egress filtering (Module 5)
Next steps: connect whole networks with VPC peering and Transit Gateway.
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Elastic Load Balancing: ALB, NLB and Gateway Load Balancer
Choose between Application, Network and Gateway Load Balancers, configure listeners, rules, target groups, health checks and TLS termination, make ASP.NET Core behave correctly behind a load balancer, and audit load balancers with C#.
VPC peering and Transit Gateway: connecting many VPCs
Connect VPCs with peering and with AWS Transit Gateway, understand non-transitive routing, design segmented Transit Gateway route tables for dev, test and prod, and see why peering stops scaling in a bank.