Hybrid connectivity: Site-to-Site VPN, Direct Connect and hybrid DNS
Connect AWS to on-premises data centres with Site-to-Site VPN and AWS Direct Connect, choose a resilient connection model for a bank, and make names resolve both ways with Route 53 Resolver endpoints and forwarding rules.
Assumes: .NET 10 LTS (file-based app) ยท AWS SDK for .NET v4 (AWSSDK.EC2 4.1.4) ยท AWS CLI v2 ยท CDK v2 (Amazon.CDK.Lib 2.277.0) ยท Terraform AWS provider v6 ยท Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain how a Site-to-Site VPN works on AWS โ customer gateway, two tunnels, virtual private gateway or Transit Gateway โ and choose static or dynamic (BGP) routing.
- Explain AWS Direct Connect: dedicated versus hosted connections, private/transit/public virtual interfaces, Direct Connect gateways, LAGs and MACsec.
- Pick a resiliency model (development and test, high, maximum) and design VPN as a backup for Direct Connect.
- Make DNS work in both directions with Route 53 Resolver inbound and outbound endpoints and forwarding rules shared across accounts.
- Monitor tunnel health from C# and estimate the monthly cost of a bank's hybrid network.
Prerequisites
- CIDR planning โ the on-premises ranges
10.0.0.0/14(LDN-DC1) and10.4.0.0/14(MAN-DC2). - VPC peering and Transit Gateway โ VPN and Direct Connect usually terminate on a Transit Gateway.
- The DNS and Route 53 lesson listed in the module overview.
- Tools: AWS CLI v2 with the
learn-sandboxprofile, .NET 10 SDK.
Why this matters in banking
Harbourline's core banking platform runs on-premises in London (LDN-DC1) and Manchester (MAN-DC2), and it will for years. The Accounts API on AWS must read balances from it; the payments hub on-premises must call fraud scoring on AWS; nightly batch files flow both ways; staff in branches use internal web apps hosted on AWS. All of this traffic carries customer data, and much of it is time-critical (a payment that misses a scheme cut-off is a customer complaint and sometimes a regulatory report).
So the bank needs connections to AWS that are private (not across the open internet unprotected), predictable (latency and bandwidth that don't depend on internet weather), resilient (no single cable, device or building whose failure stops payments) and observable (alarms before customers notice). And it needs DNS to work across the boundary: AWS workloads must resolve corebanking.ldn.harbourline.internal, and on-premises systems must resolve the private names of AWS services.
The idea in plain language
There are two ways to connect a data centre to AWS:
- Site-to-Site VPN builds encrypted IPsec tunnels over the internet between your router and AWS. Quick to set up (hours), cheap, encrypted by design โ but performance depends on the internet path and each tunnel is limited (about 1.25 Gbps for a standard tunnel).
- AWS Direct Connect is a physical, dedicated connection from your network to AWS at a Direct Connect location (a colocation data centre), often via a telecoms partner. Predictable latency and high bandwidth (1 to 400 Gbps), lower data-out prices โ but it takes weeks to provision, and it is not encrypted by default.
Banks typically use both: Direct Connect as the primary path, VPN as an encrypted backup (or IPsec running over Direct Connect when encryption is required on the link).
Routing between your network and AWS is usually exchanged with BGP (Border Gateway Protocol) โ the protocol routers use to tell each other which address ranges they can reach. For this course you only need the idea: each side advertises its prefixes, and when a link fails, its routes are withdrawn and traffic moves to the remaining links automatically.
Analogy: VPN is an armoured van on public roads โ secure and available today, but stuck in traffic sometimes. Direct Connect is the bank's own private rail line to the cash centre โ fast and predictable, but it takes months to lay, and you still want guards on board (encryption) if the cargo is sensitive. A sensible bank has the rail line and a van on standby.
What problem the AWS services solve
| Site-to-Site VPN | AWS Direct Connect | |
|---|---|---|
| Transport | IPsec over the internet (or over DX) | Private fibre to a DX location |
| Time to provision | Minutes to hours | Weeks (partner, cross-connect, LOA-CFA) |
| Bandwidth | Up to 1.25 Gbps per standard tunnel; up to 5 Gbps per large-bandwidth tunnel; ECMP across tunnels on Transit Gateway | 1, 10, 100, 400 Gbps dedicated; 50 Mbpsโ25 Gbps hosted (partner) |
| Encryption | Always (IPsec) | Not by default โ add MACsec (layer 2, on supported dedicated ports) or IPsec over DX |
| Terminates on | Virtual private gateway (one VPC) or Transit Gateway / Cloud WAN | Private VIF โ VGW or DX gateway; transit VIF โ DX gateway โ Transit Gateway; public VIF โ AWS public endpoints |
| Routing | Static or BGP | BGP only |
| MTU | 1446 bytes (MSS 1406), no Path MTU Discovery | Up to 9001 (private VIF) / 8500 (transit VIF) โ VERIFY |
| Price (checked 2026-11-18) | USD 0.05 per connection-hour (two tunnels included); large-bandwidth 5 Gbps USD 0.60/hour; data out at internet rates | Port-hours: 1G USD 0.30/h, 10G USD 2.25/h, 100G USD 22.50/h, 400G USD 85/h (dedicated); data out from US to a US location USD 0.02/GB; data in free |
Sharp edges:
- A VPN connection to a virtual private gateway reaches one VPC. For many VPCs, terminate on a Transit Gateway.
- VPN to a VGW is limited to 100 routes advertised from your side; a VPN on a Transit Gateway accepts up to 1,000 dynamic routes. Summarise your on-premises ranges.
- ECMP (spreading traffic across several tunnels) works only on Transit Gateway with dynamic (BGP) routing.
- Direct Connect alone is a single point of failure unless you buy redundancy (see resiliency models).
- Direct Connect is a layer-2/3 circuit, not a security control: you still need encryption decisions, firewalls and monitoring.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| VPN Gateway (S2S) | Site-to-Site VPN on a VGW or Transit Gateway | AWS gives two tunnels per connection, each in a different AZ endpoint |
| Local network gateway | Customer gateway (CGW) resource | Describes your router's public IP and BGP ASN |
| ExpressRoute circuit | Direct Connect connection (dedicated or hosted) | DX ports are per location; you order cross-connects |
| ExpressRoute private peering | Private VIF / transit VIF | Transit VIF connects to Transit Gateway via a DX gateway |
| ExpressRoute Microsoft peering | Public VIF | Reaches AWS public endpoints (e.g. S3) over DX |
| ExpressRoute Global Reach | DX SiteLink | Site-to-site over the AWS backbone โ VERIFY pricing |
| ExpressRoute Gateway | Direct Connect gateway | Global object; associates with VGWs or Transit Gateways in many Regions |
| ExpressRoute MACsec / IPsec over ER | MACsec on DX dedicated ports / VPN over DX | Same choices, similar trade-offs |
| Azure DNS Private Resolver (inbound/outbound endpoints, forwarding rulesets) | Route 53 Resolver inbound/outbound endpoints, forwarding rules | Rules shared across accounts with AWS RAM |
Architecture
Harbourline's target hybrid design
What makes this "maximum resiliency": two connections at each of two separate DX locations, terminating on different AWS devices, each data centre feeding a different location, plus VPN backup over the internet. Any single router, cable, DX device or whole DX location can fail without losing connectivity.
Resiliency models
| Model | Shape | Protects against | AWS SLA (per the DX SLA, if all its conditions are met) |
|---|---|---|---|
| Development and test | Two connections, separate devices, one location | Device failure | โ (not for critical workloads) |
| High resiliency | One connection at each of two locations | Device, fibre cut, one location failure | 99.9% |
| Maximum resiliency | Two connections at each of two locations (four total) | Device and location failure at the same time | 99.99% |
| VPN only | Two tunnels, one connection (ideally two CGWs) | Tunnel endpoint failure | Separate VPN SLA โ VERIFY |
The Direct Connect Resiliency Toolkit in the console orders connections to match a model, prevents two connections landing on the same AWS device, and offers a failover test that brings down a BGP session so you can prove traffic moves.
Routing preference: DX first, VPN second
When the same prefix is learned over DX and VPN, AWS prefers Direct Connect over VPN for traffic to on-premises. On your side, you influence how AWS sends traffic with BGP attributes (for example, AS-path prepending to make a path less attractive) and how your network sends traffic to AWS with local preference. Keep the prefixes identical on both paths (or the VPN prefixes less specific): a more specific route over VPN would win over DX because longest-prefix match beats everything โ VERIFY the full route-priority order in the Transit Gateway and DX documentation.
Hybrid DNS with Route 53 Resolver
- Inbound endpoint: IP addresses in your VPC that on-premises DNS servers forward to. They answer from Route 53 private hosted zones and the VPC's view of AWS names (for example, private-DNS names of interface endpoints in the hub).
- Outbound endpoint + forwarding rules: the VPC resolver forwards queries for chosen domains (for example
ldn.harbourline.internal) to your on-premises DNS servers. - Share rules with AWS RAM so every workload account's VPCs use the same forwarding rules, while the endpoints live once in the hub account.
- Each endpoint has at least two IP addresses (ENIs) in different AZs.
Step-by-step implementation
1. Agree the addressing and BGP plan
Prefixes AWS will learn: 10.0.0.0/13 (both data centres summarised) or the two /14s. Prefixes on-premises will learn: 10.16.0.0/12 (all of AWS). ASNs: Harbourline on-premises 65010/65020; Transit Gateway 64600. Write it down โ mismatched prefixes and ASNs cause most turn-up failures.
2. Create the customer gateway
One customer gateway per on-premises router: its public IP and BGP ASN (or a certificate for devices with dynamic IPs).
3. Create the VPN connection on the Transit Gateway
Choose dynamic routing, let AWS generate pre-shared keys (store them in Secrets Manager), and pick modern tunnel options (IKEv2, AES-256-GCM, SHA-384, DH groups 19/20 โ VERIFY current recommended set). Download the device configuration for your router vendor.
4. Bring up both tunnels and BGP
Configure the router from the downloaded file. Both tunnels should show UP with BGP routes received. Configure the router to use both tunnels (ECMP on TGW requires dynamic routing on both sides).
5. Order Direct Connect
Use the Resiliency Toolkit, choose the model, locations and speed. You receive a Letter of Authorization and Connecting Facility Assignment (LOA-CFA); your partner or colocation provider uses it to order the cross-connect. Then create a Direct Connect gateway, transit VIFs on each connection, and associate the DX gateway with the Transit Gateway with allowed prefixes (what AWS advertises to you).
6. Route in the Transit Gateway
Associate the DX gateway and VPN attachments with rt-onprem and propagate them into the route tables that should reach on-premises (prod and shared โ see VPC peering and Transit Gateway).
7. Hybrid DNS
Create inbound and outbound Resolver endpoints in the hub VPC, forwarding rules for on-premises domains, share the rules with RAM, associate them with every spoke VPC, and configure conditional forwarders on-premises to the inbound endpoint IPs.
8. Test failover and monitor
Run the DX failover test, shut a VPN tunnel, and watch traffic move. Alarm on TunnelState, DX ConnectionState and BGP status.
AWS Console instructions
- Customer gateway: Console โ VPC โ Virtual private network โ Customer gateways โ Create โ BGP ASN, IP address, name
hl-cgw-ldn-a, tags. - VPN connection: Site-to-Site VPN connections โ Create โ Target gateway type Transit gateway โ choose the TGW and CGW โ Routing Dynamic (requires BGP) โ tunnel options โ Create. Then Download configuration for your device.
- Direct Connect: Console โ Direct Connect โ Create a connection โ Connection wizard (Resiliency Toolkit) โ choose model, speed, locations. Direct Connect gateways โ Create. Virtual interfaces โ Create โ Transit.
- Resolver endpoints: Console โ Route 53 โ Resolver โ Inbound endpoints / Outbound endpoints โ Create โ VPC, security group (TCP and UDP 53), two subnets in different AZs. Rules โ Create rule โ Forward, domain, outbound endpoint, target IPs. Console โ RAM โ Create resource share to share the rule.
AWS CLI
$p = '--profile', 'learn-sandbox', '--region', 'us-east-1'
# Customer gateway for the London edge router (documentation IP range)
$cgw = aws ec2 create-customer-gateway @p --type ipsec.1 --bgp-asn 65010 --ip-address 198.51.100.10 `
--tag-specifications "ResourceType=customer-gateway,Tags=[{Key=Name,Value=hl-cgw-ldn-a},{Key=Project,Value=cea-lab},{Key=Lab,Value=m02-l13}]" `
--query CustomerGateway.CustomerGatewayId --output text
# VPN on the Transit Gateway with dynamic routing
$vpn = aws ec2 create-vpn-connection @p --type ipsec.1 --customer-gateway-id $cgw --transit-gateway-id tgw-0hub00000000000 `
--options "StaticRoutesOnly=false,TunnelOptions=[{IKEVersions=[{Value=ikev2}]},{IKEVersions=[{Value=ikev2}]}]" `
--tag-specifications "ResourceType=vpn-connection,Tags=[{Key=Name,Value=hl-vpn-ldn-a},{Key=Project,Value=cea-lab},{Key=Lab,Value=m02-l13}]" `
--query VpnConnection.VpnConnectionId --output text
# Tunnel health (UP/DOWN) and routes learned
aws ec2 describe-vpn-connections @p --vpn-connection-ids $vpn `
--query "VpnConnections[0].VgwTelemetry[].{Outside:OutsideIpAddress,Status:Status,Routes:AcceptedRouteCount,Since:LastStatusChange}" --output table
# Device configuration for a vendor (list types, then download)
aws ec2 get-vpn-connection-device-types @p --query "VpnConnectionDeviceTypes[?Vendor=='Generic']"
# Direct Connect: list connections and virtual interfaces (read-only)
aws directconnect describe-connections @p --query "connections[].{Id:connectionId,Name:connectionName,State:connectionState,Bandwidth:bandwidth,Location:location}"
aws directconnect describe-virtual-interfaces @p --query "virtualInterfaces[].{Id:virtualInterfaceId,Type:virtualInterfaceType,State:virtualInterfaceState,Bgp:bgpPeers[0].bgpStatus}"
# Route 53 Resolver: outbound endpoint and a forwarding rule for the London data centre zone
$out = aws route53resolver create-resolver-endpoint @p --creator-request-id hl-out-1 --name hl-outbound `
--direction OUTBOUND --security-group-ids sg-0resolver0000000 `
--ip-addresses SubnetId=subnet-0hubres00000000a SubnetId=subnet-0hubres00000000b `
--query ResolverEndpoint.Id --output text
aws route53resolver create-resolver-rule @p --creator-request-id hl-rule-ldn --name hl-ldn-internal `
--rule-type FORWARD --domain-name ldn.harbourline.internal --resolver-endpoint-id $out `
--target-ips Ip=10.0.10.53,Port=53 Ip=10.0.11.53,Port=53Infrastructure as Code
using Amazon.CDK.AWS.EC2;
using Amazon.CDK.AWS.Route53Resolver;
using Constructs;
namespace Harbourline.Networking.Hybrid;
/// <summary>Inbound and outbound Resolver endpoints in the hub VPC plus one forwarding rule per on-premises zone.</summary>
public sealed class HybridDns : Construct
{
public HybridDns(Construct scope, string id, IVpc hubVpc, string[] onPremDnsServers, string[] forwardedDomains)
: base(scope, id)
{
var sg = new SecurityGroup(this, "ResolverSg", new SecurityGroupProps
{
Vpc = hubVpc,
Description = "Route 53 Resolver endpoints: DNS to and from Harbourline networks only",
AllowAllOutbound = false,
});
var harbourline = Peer.Ipv4("10.0.0.0/8");
sg.AddIngressRule(harbourline, Port.Tcp(53), "DNS/TCP from Harbourline");
sg.AddIngressRule(harbourline, Port.Udp(53), "DNS/UDP from Harbourline");
foreach (var server in onPremDnsServers)
{
sg.AddEgressRule(Peer.Ipv4($"{server}/32"), Port.Tcp(53), "DNS/TCP to on-prem resolver");
sg.AddEgressRule(Peer.Ipv4($"{server}/32"), Port.Udp(53), "DNS/UDP to on-prem resolver");
}
var subnets = hubVpc.SelectSubnets(new SubnetSelection { SubnetGroupName = "resolver" }).SubnetIds;
var ipAddresses = subnets.Select(s => (object)new CfnResolverEndpoint.IpAddressRequestProperty { SubnetId = s }).ToArray();
Inbound = new CfnResolverEndpoint(this, "Inbound", new CfnResolverEndpointProps
{
Name = "hl-inbound",
Direction = "INBOUND",
SecurityGroupIds = [sg.SecurityGroupId],
IpAddresses = ipAddresses,
});
var outbound = new CfnResolverEndpoint(this, "Outbound", new CfnResolverEndpointProps
{
Name = "hl-outbound",
Direction = "OUTBOUND",
SecurityGroupIds = [sg.SecurityGroupId],
IpAddresses = ipAddresses,
});
foreach (var domain in forwardedDomains)
{
var rule = new CfnResolverRule(this, $"Rule-{domain}", new CfnResolverRuleProps
{
Name = "fwd-" + domain.Replace('.', '-'),
RuleType = "FORWARD",
DomainName = domain,
ResolverEndpointId = outbound.AttrResolverEndpointId,
TargetIps = onPremDnsServers.Select(ip => (object)new CfnResolverRule.TargetAddressProperty { Ip = ip, Port = "53" }).ToArray(),
});
_ = new CfnResolverRuleAssociation(this, $"Assoc-{domain}", new CfnResolverRuleAssociationProps
{
ResolverRuleId = rule.AttrResolverRuleId,
VpcId = hubVpc.VpcId,
});
}
}
public CfnResolverEndpoint Inbound { get; }
}Share the rules to workload accounts with Amazon.CDK.AWS.RAM.CfnResourceShare and associate them in each spoke VPC (in that account's stack).
Complete code
The SOC needs to know a tunnel is down before payments slow down. This .NET 10 file-based app checks every Site-to-Site VPN connection in the account, prints each tunnel's state and accepted routes, and exits with 3 if any connection has fewer than two tunnels up โ suitable for a scheduled pipeline job or a container on a timer. (In production you would also alarm on the CloudWatch TunnelState metric.)
#:package AWSSDK.EC2@4.1.4
#:package AWSSDK.SSO@4.1.0.2
#:package AWSSDK.SSOOIDC@4.1.0.2
using Amazon;
using Amazon.EC2;
using Amazon.EC2.Model;
using Amazon.Runtime.CredentialManagement;
// Usage: dotnet run vpnhealth.cs [--min-up 2] [--tag-key Name --tag-prefix hl-vpn-]
var minUp = 2;
string? tagPrefix = null;
for (var i = 0; i + 1 < args.Length; i += 2)
{
switch (args[i])
{
case "--min-up": minUp = int.Parse(args[i + 1]); break;
case "--tag-prefix": tagPrefix = args[i + 1]; break;
}
}
if (!new CredentialProfileStoreChain().TryGetAWSCredentials("learn-sandbox", out var credentials))
{
Console.Error.WriteLine("Profile learn-sandbox not found. Run: aws sso login --profile learn-sandbox");
return 1;
}
using var ec2 = new AmazonEC2Client(credentials, RegionEndpoint.USEast1);
var response = await ec2.DescribeVpnConnectionsAsync(new DescribeVpnConnectionsRequest
{
Filters = [new Filter { Name = "state", Values = ["available"] }],
});
var degraded = 0;
foreach (var vpn in response.VpnConnections ?? [])
{
var name = vpn.Tags?.FirstOrDefault(t => t.Key == "Name")?.Value ?? vpn.VpnConnectionId;
if (tagPrefix is not null && !name.StartsWith(tagPrefix, StringComparison.Ordinal))
{
continue;
}
var tunnels = vpn.VgwTelemetry ?? [];
var up = tunnels.Count(t => t.Status == TelemetryStatus.UP);
var target = vpn.TransitGatewayId ?? vpn.VpnGatewayId ?? "unattached";
var routing = vpn.Options?.StaticRoutesOnly == true ? "static" : "BGP";
var healthy = up >= minUp;
if (!healthy)
{
degraded++;
}
Console.WriteLine($"{(healthy ? "OK " : "WARN")} {name,-22} {vpn.VpnConnectionId} -> {target} ({routing}) tunnels up {up}/{tunnels.Count}");
foreach (var t in tunnels.OrderBy(t => t.OutsideIpAddress))
{
var since = t.LastStatusChange is { } changed ? $"{DateTime.UtcNow - changed.ToUniversalTime():d\\.hh\\:mm} ago" : "unknown";
Console.WriteLine($" {t.OutsideIpAddress,-16} {t.Status,-5} routes={t.AcceptedRouteCount,-4} changed {since} {t.StatusMessage}");
}
}
if (response.VpnConnections is null or { Count: 0 })
{
Console.WriteLine("No available VPN connections in us-east-1.");
}
Console.WriteLine(degraded == 0 ? "RESULT: all VPN connections have redundant tunnels up." : $"RESULT: {degraded} connection(s) degraded.");
return degraded == 0 ? 0 : 3;Example output against a lab VPN whose customer gateway does not exist (both tunnels down, as expected):
WARN hl-vpn-ldn-a vpn-0abc12345def67890 -> tgw-0hub00000000000 (BGP) tunnels up 0/2
3.210.x.x DOWN routes=0 changed 0.00:12 ago
52.4.x.x DOWN routes=0 changed 0.00:12 ago
RESULT: 1 connection(s) degraded.Code explained line by line
#:package AWSSDK.EC2@4.1.4โ VPN connections are part of the EC2 API in the SDK.- The
forloop reads two optional flags:--min-up(how many tunnels must be up; two is the redundant baseline) and--tag-prefix(to check only Harbourline's connections). - Credentials come from the SSO profile, never from code.
DescribeVpnConnectionsAsyncwith astate = availablefilter skips connections being deleted.VgwTelemetryholds one entry per tunnel โ despite the name, it is filled for VPNs on a Transit Gateway too. In SDK v4 it may benull, hence?? [].TelemetryStatus.UPis the SDK's constant class for the tunnel state.AcceptedRouteCountshows how many BGP routes AWS learned over that tunnel โ a tunnel that is UP with zero routes usually means BGP is not established or the router filters advertisements.LastStatusChangetells you whether a tunnel is flapping (changed minutes ago) or stable.- Exit code 3 when any connection has fewer than
minUptunnels up, so a scheduler can page someone.
Security considerations
โ Learning shortcut โ not for production
The optional lab VPN uses AWS-generated pre-shared keys that you can see in the downloaded configuration file. In production, store and rotate them in Secrets Manager (or use certificate-based authentication with AWS Private CA), never email the configuration file, and restrict ec2:GetVpnConnectionDeviceSampleConfiguration and ec2:DescribeVpnConnections (which can expose keys) to the network team.
- Encrypt Direct Connect traffic if policy requires encryption in transit on all external links. Options: MACsec on supported dedicated ports (line-rate, layer 2, between your device and the AWS device), or IPsec VPN over a transit/private VIF (limited by VPN tunnel throughput). Many banks also rely on TLS end to end; agree the policy with security.
- Filter routes. Advertise only the summary prefixes you intend; on the DX gateway association use allowed prefixes; on your routers filter what you accept. A leaked default route can send AWS traffic through your data centre, or vice versa.
- Segment at the Transit Gateway. The on-premises attachments sit in their own route table and are propagated only into segments that should reach the data centres (see the previous lesson).
- Inspect. Route on-premises traffic to sensitive VPCs through an inspection VPC (AWS Network Firewall).
- DNS is a trust boundary. Restrict Resolver endpoint security groups to known DNS servers; log queries with Resolver query logging; consider DNS Firewall on outbound queries.
- Monitor. Alarms on
TunnelState, DXConnectionState,VirtualInterfaceBpsEgress, BGP state changes, and CloudTrail events for VPN/DX/route changes.
Banking use case
Harbourline's network hub account owns the Transit Gateway, the DX gateway and the Resolver endpoints:
- Primary path: maximum resiliency DX โ two 10G connections at each of two DX locations, in LAGs, with MACsec enabled; transit VIFs to the DX gateway; DX gateway associated with the TGW advertising
10.16.0.0/12. - Backup path: two Site-to-Site VPN connections (London and Manchester edge routers) to the TGW, BGP with ECMP, advertising the same summaries. Quarterly failover tests use the DX failover test and a planned VPN tunnel shutdown; results go to the operational-resilience team as evidence for the bank's important business services mapping.
- DNS: on-premises DNS forwards
aws.harbourline.internaland AWS service names to the inbound endpoint; forwarding rules forldn.harbourline.internalandman.harbourline.internalare shared to all accounts with RAM. - Change control: BGP and prefix changes are pull requests against the network repo; the TGW simulator from the previous lesson includes the
dcattachment so segmentation is tested before each change.
Whether this design satisfies the bank's operational-resilience or outsourcing obligations is for Harbourline's risk and compliance functions to decide; the architecture provides the redundancy and evidence they need.
Common mistakes
- One DX connection and calling it resilient. A single port, cross-connect or location is a single point of failure.
- VPN on a VGW for many VPCs. A VGW serves one VPC; use a Transit Gateway.
- More specific prefixes on the VPN than on DX, so traffic unexpectedly prefers the VPN.
- Static routing with two tunnels and assuming automatic failover/ECMP. Use BGP.
- Forgetting MTU/MSS on VPN (1446/1406) โ large packets fail mysteriously.
- Treating DX as encrypted. It isn't by default.
- Resolver forwarding loops: on-premises forwards a zone to the inbound endpoint, and a rule in AWS forwards the same zone back on-premises.
- Resolver endpoint security group that blocks UDP 53 or TCP 53 (large responses and zone transfers use TCP).
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Tunnel DOWN | IKE mismatch, wrong PSK, firewall blocking UDP 500/4500 or ESP | Compare tunnel options with the device config; check StatusMessage; enable VPN tunnel logs to CloudWatch Logs |
| Tunnel UP, 0 accepted routes | BGP not established or routes filtered | Check BGP neighbour/ASN; advertise prefixes; check route filters |
| Traffic only uses one tunnel | Static routing, or ECMP not configured | Dynamic routing on TGW, ECMP on the router |
| Large file transfers fail over VPN | MTU/MSS | Set MSS clamp 1406 or lower per algorithm guidance |
DX VIF down | BGP not up, VLAN or IP mismatch, LOA not completed | Check VIF BGP peer config and cross-connect status |
| On-prem can't resolve AWS names | No conditional forwarder to inbound endpoint, or SG blocks 53 | Configure forwarder; allow TCP/UDP 53 from on-prem DNS |
| AWS can't resolve on-prem names | Rule not associated with the VPC, or outbound SG blocks egress | Associate the shared rule; allow egress 53 to target IPs |
| Traffic prefers VPN over DX | More specific prefixes over VPN | Advertise identical prefixes on both |
Cost and cleanup
Prices (checked 2026-11-18):
- Site-to-Site VPN: USD 0.05 per connection-hour (about USD 36/month); large-bandwidth (5 Gbps) USD 0.60/hour; accelerated VPN adds two Global Accelerators (USD 0.05/hour total) plus data-transfer premium; each tunnel's public IPv4 address bills at standard rates. Data out at internet rates (first 100 GB/month free, then from USD 0.09/GB).
- Transit Gateway VPN attachment: USD 0.05 per hour plus USD 0.02 per GB processed (see previous lesson).
- Direct Connect: dedicated port-hours 1G USD 0.30, 10G USD 2.25, 100G USD 22.50, 400G USD 85; hosted from USD 0.03/h (50 Mbps); data out from US Regions to US locations USD 0.02/GB; data in free. Partner and cross-connect fees are extra and outside AWS pricing.
- Route 53 Resolver endpoints: USD 0.125 per ENI-hour (an endpoint needs at least two: USD 182.50/month per endpoint) plus USD 0.40 per million queries through endpoints.
Harbourline estimate: 4 ร 10G DX ports = 4 ร 2.25 ร 730 = USD 6,570/month; 2 VPN connections + 2 TGW VPN attachments โ USD 146; 1 DX gateway attachment USD 36.50; inbound + outbound Resolver endpoints (2 ENIs each) = USD 365; data out 30 TB via DX at USD 0.02 = USD 600. Roughly USD 7,700/month before partner and colocation charges.
Free alternative: design-only โ the BGP/addressing plan, the resiliency diagram, the Resolver rule design and the cost estimate above. All exam-relevant knowledge is in the design.
Optional deploy-and-destroy (about USD 0.06 for an hour): create a customer gateway with a documentation IP and a VPN connection to a virtual private gateway on a lab VPC, download the configuration, run vpnhealth.cs (tunnels DOWN is expected), then delete. No Transit Gateway needed.
$p = '--profile', 'learn-sandbox', '--region', 'us-east-1'
aws ec2 delete-vpn-connection @p --vpn-connection-id vpn-0abc12345def67890
# wait until the VPN connection is 'deleted', then:
aws ec2 delete-customer-gateway @p --customer-gateway-id cgw-0abc12345def67890
aws ec2 detach-vpn-gateway @p --vpn-gateway-id vgw-0abc12345def67890 --vpc-id vpc-0123456789abcdef0
aws ec2 delete-vpn-gateway @p --vpn-gateway-id vgw-0abc12345def67890
aws resourcegroupstaggingapi get-resources @p --tag-filters Key=Project,Values=cea-lab --query "ResourceTagMappingList[].ResourceARN"Resolver endpoints bill per ENI-hour from creation: if you try them, delete them in the same session (aws route53resolver delete-resolver-endpoint --resolver-endpoint-id โฆ) after removing rule associations.
Exercises
- ๐ข Guided: Draw Harbourline's hybrid design for the high resiliency model (one connection at each of two locations) with VPN backup, including prefixes advertised in each direction and the TGW route table each attachment is associated with.
- ๐ก Challenge: Design hybrid DNS for three zones:
aws.harbourline.internal(Route 53 private hosted zone in the hub),ldn.harbourline.internalandman.harbourline.internal(on-premises). List the endpoints, rules, RAM shares, VPC associations and on-premises conditional forwarders, and show how you avoid a forwarding loop. - ๐ด Live problem: On Monday morning branch staff report that the internal statements portal (hosted on AWS) is slow and sometimes times out when downloading PDFs larger than about 2 MB; small pages are fine. The network team says "DX had planned maintenance at location 1 overnight". Diagnose.
Assignment
Write a two-page hybrid connectivity design for Harbourline: addressing and BGP plan, resiliency model and justification, encryption decision, DNS design, monitoring and alarms, failover test plan, and monthly cost. Add the vpnhealth.cs tool with a README to your portfolio repository.
Solutions
- Connections: LDN-DC1 โ DX location 1 (one 10G); MAN-DC2 โ DX location 2 (one 10G). Transit VIF on each to the DX gateway; DX gateway associated with the TGW, allowed prefix
10.16.0.0/12. - VPN backup: one VPN connection from each data centre to the TGW, BGP, both tunnels up.
- Advertised to AWS:
10.0.0.0/14from LDN,10.4.0.0/14from MAN, over both DX and VPN (identical prefixes, so DX is preferred). - Advertised to on-premises:
10.16.0.0/12. - TGW: DX gateway and VPN attachments associated with
rt-onprem; propagated intort-prodandrt-sharedonly.
Why: high resiliency survives a device, fibre or location failure for DX; VPN covers simultaneous DX failures at a lower bandwidth.
| Item | Where | Detail |
|---|---|---|
| Inbound endpoint | Hub VPC, 2 AZs | On-prem forwards aws.harbourline.internal and amazonaws.com endpoint names here |
| Outbound endpoint | Hub VPC, 2 AZs | Used by forwarding rules |
Rule ldn.harbourline.internal | Hub account | FORWARD โ LDN DNS servers |
Rule man.harbourline.internal | Hub account | FORWARD โ MAN DNS servers |
| RAM share | Hub account โ Organization | Both rules |
| Rule associations | Every spoke VPC | Done in each account's baseline stack |
Private hosted zone aws.harbourline.internal | Hub account | Associated with every spoke VPC (cross-account association) |
| On-prem conditional forwarders | LDN and MAN DNS | aws.harbourline.internal โ inbound endpoint IPs |
Loop avoidance: never create a forwarding rule in AWS for aws.harbourline.internal, and never configure on-premises to forward ldn/man zones to AWS. Each zone has exactly one authoritative side.
"Small responses fine, large downloads fail, after DX maintenance" points to MTU. During the maintenance window traffic failed over to the VPN backup, and it may not have failed back (for example, BGP on location 1 is still down, or the VPN now has more specific prefixes). Over VPN the MTU is 1446 bytes and there is no Path MTU Discovery; if the portal's servers or the on-premises routers don't clamp MSS, full-size packets are dropped. Small pages fit in one packet; large PDFs don't.
Check: vpnhealth.cs (tunnels UP with accepted routes?), DX VIF BGP status, TGW route table rt-onprem (are routes via the DX gateway or the VPN attachment?), and VPC Flow Logs showing retransmissions. Fix immediately: MSS clamp (1406 or lower) on the VPN path; then restore the DX BGP session so traffic fails back. Afterwards: add an alarm on DX BGP state and on "traffic over VPN" so the next failover is noticed.
Knowledge check
Answer: Two, terminating on different AWS endpoints, so one can be maintained or fail without losing the connection.
Answer: A transit VIF, connected to a Direct Connect gateway that is associated with the Transit Gateway.
Answer: Not by default. Use MACsec on supported dedicated connections or run IPsec VPN over DX; and keep TLS end to end.
Answer: Inbound lets on-premises DNS query AWS (private hosted zones, endpoint names). Outbound lets the VPC resolver forward queries for chosen domains to on-premises DNS, using forwarding rules.
Answer: Maximum resiliency: separate connections on separate devices in at least two locations (two per location), meeting the conditions in the DX SLA.
Interview questions
- "VPN or Direct Connect for a bank?" โ Both: DX for predictable bandwidth and latency, VPN as an encrypted backup and for quick starts. Encryption on DX via MACsec or IPsec depending on policy.
- "How do you make Direct Connect highly available?" โ Two DX locations, separate devices, ideally two connections per location (maximum resiliency), BGP failover, VPN backup, and regular failover tests with evidence.
- "How does an app on AWS resolve an on-premises hostname?" โ Route 53 Resolver outbound endpoint with a forwarding rule for the on-prem domain, shared to accounts via RAM and associated with the VPC.
- "Why might traffic prefer the VPN over Direct Connect?" โ More specific prefixes advertised over the VPN; longest-prefix match wins before path preference.
Certification practice
A bank has one 10 Gbps AWS Direct Connect connection from its data centre to a transit gateway. A business-continuity review requires that connectivity survive the failure of the Direct Connect location, while keeping costs moderate. Encryption over the internet is acceptable during failures. What should a solutions architect recommend?
A. Add a second virtual interface on the same Direct Connect connection B. Create a Site-to-Site VPN connection to the transit gateway with BGP as a backup path C. Order a second connection at the same Direct Connect location D. Replace Direct Connect with VPC peering
Answer: B. A VPN to the TGW with BGP survives the loss of the DX location at low cost and encrypts traffic. A and C share the same location (and A the same port). D is not a hybrid technology.
EC2 instances in several VPCs across multiple accounts must resolve names in the on-premises domain corp.example.internal. The on-premises DNS servers are reachable over Direct Connect. What should the CloudOps engineer configure with the LEAST operational effort?
A. A Route 53 public hosted zone for corp.example.internal
B. A Route 53 Resolver outbound endpoint in a shared-services VPC, a forwarding rule for corp.example.internal, shared with AWS RAM and associated with each VPC
C. A custom DHCP options set in each VPC pointing to the on-premises DNS servers
D. An inbound Resolver endpoint in each VPC
Answer: B. One outbound endpoint and one shared rule serve all VPCs. A would publish internal names publicly. C sends all DNS to on-premises, breaking AWS private DNS features and adding dependency. D handles the opposite direction.
A financial services company requires that all traffic on links leaving its premises be encrypted, including its 100 Gbps dedicated Direct Connect connections, without reducing throughput significantly. Which solution meets the requirement?
A. Enable MACsec on the dedicated Direct Connect connections with a compatible on-premises device B. Run a Site-to-Site VPN over a public VIF C. Rely on TLS between applications only D. Use AWS PrivateLink over Direct Connect
Answer: A. MACsec provides layer-2 encryption at line rate on supported dedicated ports. B encrypts, but VPN tunnels are limited to 1.25โ5 Gbps each, far below 100 Gbps. C does not cover all traffic on the link. D is a service-exposure mechanism, not link encryption.
Completion checklist
- I can explain VPN tunnels, customer gateways and VGW versus TGW termination
- I can describe dedicated/hosted DX, the three VIF types and the DX gateway
- I can draw the high and maximum resiliency models with VPN backup
- I can design Resolver inbound/outbound endpoints and shared forwarding rules
- If I created a lab VPN or Resolver endpoint, I deleted it and the tag check is clean
Key takeaways
- VPN: encrypted, fast to set up, bandwidth-limited per tunnel; DX: private, predictable, high bandwidth, not encrypted by default.
- Terminate on a Transit Gateway for many VPCs; use BGP for failover and ECMP.
- Resilience comes from separate devices and separate locations, plus a tested backup path.
- Hybrid DNS: inbound for on-prem โ AWS, outbound + rules (shared via RAM) for AWS โ on-prem.
- Monitor tunnels, BGP and DX state, and test failover with evidence.
Further reading
- AWS Site-to-Site VPN quotas
- AWS Site-to-Site VPN pricing
- AWS Direct Connect Resiliency Toolkit
- AWS Direct Connect pricing
- Amazon Route 53 pricing (Resolver endpoints)
- AWS Transit Gateway quotas
Next steps: record and investigate what crosses these links with VPC Flow Logs, then put it all together in multi-account network architecture.
Independent learning resource โ not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
VPC peering and Transit Gateway: connecting many VPCs
Connect VPCs with peering and with AWS Transit Gateway, understand non-transitive routing, design segmented Transit Gateway route tables for dev, test and prod, and see why peering stops scaling in a bank.
VPC Flow Logs and Reachability Analyzer: evidence of what can talk to what
Enable, read and query VPC Flow Logs for investigations, choose the right destination and format, and use Reachability Analyzer and Network Access Analyzer to prove which paths exist without sending a packet.