VPC Flow Logs and Reachability Analyzer: evidence of what can talk to what
Enable, read and query VPC Flow Logs for investigations, choose the right destination and format, and use Reachability Analyzer and Network Access Analyzer to prove which paths exist without sending a packet.
Assumes: .NET 10 LTS (file-based apps) ยท AWS SDK for .NET v4 (AWSSDK.CloudWatchLogs 4.1.0.2) ยท AWS CLI v2 ยท CDK v2 (Amazon.CDK.Lib 2.277.0) ยท Terraform AWS provider v6 ยท Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Enable VPC Flow Logs on a VPC, subnet or network interface, and choose between CloudWatch Logs, Amazon S3 and Amazon Data Firehose as the destination.
- Read a default flow-log record field by field, and recognise the signatures of a security-group block, a network-ACL block and a NACL that forgot return traffic.
- Add custom-format fields (
vpc-id,subnet-id,tcp-flags,pkt-srcaddr,flow-direction,traffic-path) when the default record cannot answer your question. - Query flow logs with CloudWatch Logs Insights and with Athena over S3, and from C# with the AWS SDK for .NET v4.
- Use Reachability Analyzer to prove a single path exists or does not, and Network Access Analyzer to prove a rule holds across a whole account.
- Estimate and control the cost of flow logging at bank scale.
Prerequisites
- Networking refresher โ five-tuples, ports, TCP handshake.
- Amazon VPC fundamentals and the security group / network ACL lesson (module overview).
- Tools: AWS CLI v2 with the
learn-sandboxSSO profile, .NET 10 SDK.
Why this matters in banking
At 02:10 on a Saturday, Harbourline's security operations centre gets an alert: an EC2 instance in the Payments test VPC has been talking to an IP address that a threat-intelligence feed lists as a command-and-control server. The questions arrive immediately:
- When did it start, and how much data left?
- Did that instance talk to anything else inside the bank โ the card tokenisation service, the core banking VPN?
- Could anything else in the estate reach that address?
Application logs cannot answer these questions; the attacker is not going to log their own activity. VPC Flow Logs can answer the first two, because they record every IP flow on every network interface whether or not the application cooperates. Reachability Analyzer and Network Access Analyzer answer the third, from configuration alone.
There is also a regulatory angle. PCI DSS expects logging of access to cardholder-data networks, and financial regulators expect a bank to reconstruct incidents. Flow logs are a cheap, always-on network record. Whether a given logging setup meets a specific requirement is a decision for your compliance team and assessor; this lesson shows you how to produce the data.
The idea in plain language
A flow log is a summary, per network interface, of who talked to whom. It is not a recording of the conversation. Every aggregation interval (one minute or ten), AWS writes one line per flow โ a unique combination of source IP, destination IP, source port, destination port and protocol (the "5-tuple") โ with packet and byte counts and one word saying whether the traffic was ACCEPTed or REJECTed by security groups and network ACLs.
Reachability Analyzer works the other way round: you name a source and a destination, and it reads every route table, security group, NACL, gateway and load balancer on the way to work out whether a packet could get through โ without sending one.
Analogy: flow logs are a bank's door-access log: badge number, door, time, granted or denied. They don't tell you what was said in the meeting room. Reachability Analyzer is the building plan review: given the locks and corridors, could badge X ever open door Y?
What problem the AWS services solve
| Tool | Question it answers | Sends traffic? | Price (us-east-1, checked 2026-11-18) |
|---|---|---|---|
| VPC Flow Logs | What flows actually happened, and were they accepted? | No (passive) | Vended-log ingestion, e.g. USD 0.50 per GB to CloudWatch Logs for the first 10 TB; S3 delivery is cheaper โ VERIFY S3 rate |
| Reachability Analyzer | Can source A reach destination B on this protocol/port? Which component blocks it? | No | USD 0.10 per analysis |
| Network Access Analyzer | Does any path violate my rule, e.g. "no internet gateway can reach a database subnet"? | No | USD 0.002 per network interface analysed |
| Traffic Mirroring | What exactly was in the packets? | Copies real packets | USD 0.015 per session-hour (plus the appliance) |
What flow logs do not capture
Honest limits matter in an investigation. Flow logs do not record:
- Traffic to and from the Amazon DNS resolver (the VPC+2 address). If you run your own DNS servers, that traffic is logged. Use Route 53 Resolver query logging for DNS questions.
- Instance metadata (
169.254.169.254), Amazon Time Sync (169.254.169.123), DHCP traffic and traffic to the reserved VPC router address. - Windows licence activation traffic for AWS-provided Windows AMIs.
- Mirrored traffic and some traffic between endpoint network interfaces and Network Load Balancer interfaces โ VERIFY the current full list on the flow-logs limitations page.
They are also not real-time: records arrive after the aggregation interval plus a delivery delay of several minutes. And a flow record cannot tell you which control rejected a packet โ only that something did.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| NSG flow logs (being retired) / VNet flow logs | VPC Flow Logs (VPC, subnet, ENI, also Transit Gateway) | AWS records per ENI; destination is CloudWatch Logs, S3 or Firehose rather than a storage account |
| Traffic Analytics (Log Analytics workspace) | CloudWatch Logs Insights, Athena over S3, or a SIEM | No built-in analytics dashboard; you write queries (or use Amazon Security Lake / OpenSearch) |
| KQL | Logs Insights query language, or SQL in Athena | Similar pipe-style syntax in Logs Insights |
| Network Watcher IP flow verify | Reachability Analyzer (single path) | Analyzer covers the whole path (routes, gateways, LBs), not just one NSG |
| Network Watcher Connection troubleshoot | Reachability Analyzer | Analyzer never sends probes |
| Azure Policy / Network Manager security admin rules (for audit) | Network Access Analyzer scopes | Scopes describe forbidden paths; findings list every match |
| Virtual network TAP | Traffic Mirroring | Both copy packets to an appliance |
Architecture
How to choose:
| Destination | Best for | Watch out for |
|---|---|---|
| CloudWatch Logs | Quick investigations, Logs Insights, metric filters and alarms on REJECT spikes, small estates | Highest ingestion price; set retention or you keep (and pay for) everything forever |
| S3 | Bank-scale retention, central log-archive account, Athena, Security Lake | Query latency higher; set lifecycle rules; bucket policy must allow the log delivery service |
| Firehose | Streaming into a SIEM | Firehose charges on top; buffering adds delay |
A common bank pattern is both: S3 (Parquet) in the log-archive account for long retention and forensics, and CloudWatch Logs with short retention on the few VPCs where the SOC wants live alarms.
Reading a flow-log record
The default format (version 2) has 14 space-separated fields:
version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-statusA record from the secure VPC lab:
2 111122223333 eni-0db000000000000a1 10.20.64.25 10.20.128.40 41822 5432 6 6 360 1763452800 1763452860 ACCEPT OK| Field | Value | Meaning |
|---|---|---|
| version | 2 | Default format |
| account-id | 111122223333 | Account that owns the ENI (fictional) |
| interface-id | eni-0dbโฆa1 | The database's network interface |
| srcaddr / dstaddr | 10.20.64.25 โ 10.20.128.40 | App instance โ database |
| srcport / dstport | 41822 โ 5432 | Ephemeral client port โ PostgreSQL |
| protocol | 6 | IANA protocol number: 6 = TCP, 17 = UDP, 1 = ICMP |
| packets / bytes | 6 / 360 | In this interval, this direction |
| start / end | Unix seconds | The aggregation window |
| action | ACCEPT | SG and NACL both allowed it |
| log-status | OK | Data was recorded (NODATA = no traffic, SKIPDATA = records dropped by capacity limits) |
Recognising who blocked what
Security groups are stateful; network ACLs are stateless. That difference leaves fingerprints:
| Pattern on the destination ENI | Most likely cause |
|---|---|
| One REJECT for the inbound request, no reply record | The SG or the inbound NACL rule dropped the request |
| ACCEPT inbound request, then REJECT for the reply (source port = service port) | The outbound NACL has no rule for the ephemeral return ports. The SG would have let the reply out automatically |
| ACCEPT inbound and ACCEPT reply, but the client still times out | Problem is elsewhere: the client's own NACL inbound, a route, or the application |
| REJECTs from many sources to many ports | Someone scanning; check whether the ENI is reachable from where they are |
To tell an SG reject from a NACL reject, ask Reachability Analyzer (below) โ its explanation codes name the component.
Custom formats: when the default is not enough
You can choose any field set in any order when you create the flow log (you cannot change it later โ create a new flow log). Useful additions:
| Field | Version | Why you'd add it |
|---|---|---|
vpc-id, subnet-id, instance-id | 3 | Correlate without a lookup table; essential in a central log archive |
tcp-flags | 3 | See who opened the connection: 2 = SYN, 18 = SYN-ACK, 1 = FIN, 4 = RST |
type | 3 | IPv4 / IPv6 / EFA |
pkt-srcaddr, pkt-dstaddr | 3 | The original addresses when traffic passes through a NAT gateway, Transit Gateway or other intermediate ENI |
region, az-id | 4 | Cross-AZ traffic analysis (it costs money) |
pkt-src-aws-service, pkt-dst-aws-service | 5 | Names the AWS service behind a public IP, e.g. S3 |
flow-direction | 5 | ingress / egress relative to the ENI |
traffic-path | 5 | For egress: how it left, e.g. 8 = through an internet gateway, 1 = within the VPC |
AWS has added further fields since version 5 (for example ECS task metadata) โ VERIFY the latest field list before designing a format. A bank-friendly custom format:
${version} ${account-id} ${vpc-id} ${subnet-id} ${instance-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${tcp-flags} ${pkt-srcaddr} ${pkt-dstaddr} ${flow-direction} ${traffic-path} ${pkt-dst-aws-service}Reachability Analyzer and Network Access Analyzer
Reachability Analyzer takes a path โ source, destination, protocol and optional ports/IPs โ and an analysis of that path at a point in time. Sources and destinations can be instances, ENIs, internet gateways, VPC endpoints, peering connections, Transit Gateways, VPN gateways and more. The result is NetworkPathFound: true with the hop-by-hop path, or false with explanation codes such as:
| Code | Plain meaning |
|---|---|
ENI_SG_RULES_MISMATCH | A security group on this ENI has no rule that matches |
SUBNET_ACL_RESTRICTION | The subnet's network ACL denies it |
NO_ROUTE_TO_DESTINATION | A route table has no applicable route |
IGW_PRIVATE_IP_ASSOCIATION_FOR_INGRESS | The internet gateway only delivers to public IPs mapped to an ENI |
VPC_BLOCK_PUBLIC_ACCESS_ENABLED | VPC Block Public Access stopped it |
TGW_RTB_NO_ROUTE_TO_TGW_ATTACHMENT | A Transit Gateway route table has no route to the attachment |
Analyses can span accounts in an AWS Organization once you enable trusted access and register a delegated administrator.
Network Access Analyzer inverts the question. You define a Network Access Scope โ "match any path from an internet gateway to a network interface tagged Tier=data" โ and the analyzer lists every path in the account that matches. An empty finding list is evidence that the rule holds everywhere, not just for the one path you thought of. AWS ships a few starter scopes in the console.
| Use Reachability Analyzer whenโฆ | Use Network Access Analyzer whenโฆ |
|---|---|
| Debugging one connection ("why can't app reach DB?") | Proving a policy ("no data subnet is internet-reachable") |
| You need a positive and a negative control in a test | You want continuous assurance across many VPCs |
| Charged per analysis | Charged per ENI analysed |
Step-by-step implementation
1. Decide scope, traffic type and interval
- Scope: create flow logs at the VPC level so new subnets and ENIs are covered automatically. Add ENI-level logs only for special cases.
- Traffic type:
ALLfor security work.REJECTonly is cheaper but hides successful exfiltration. - Interval: 60 seconds for investigations and labs; 600 seconds (the default) for low-risk VPCs at scale.
2. Create the destination and permissions
For CloudWatch Logs, create a log group with retention and an IAM role that the vpc-flow-logs.amazonaws.com service can assume. Add an aws:SourceAccount condition to the trust policy so another account's flow log cannot use your role (the "confused deputy" problem).
For S3, no role is needed; the bucket policy allows the log delivery service principal. AWS adds the required statements automatically if you have permission to change the bucket policy.
3. Create the flow log
Use the CLI or IaC below. Confirm with describe-flow-logs that FlowLogStatus is ACTIVE and DeliverLogsStatus is SUCCESS.
4. Generate and find traffic
Wait at least one interval plus a few minutes. In CloudWatch Logs, each ENI gets its own log stream named after the ENI ID.
5. Query, then prove
Run Logs Insights queries for the investigation, then use Reachability Analyzer to confirm why rejected flows were rejected.
AWS Console instructions
Create a flow log: Console โ VPC โ Your VPCs โ select the VPC โ Flow logs tab โ Create flow log. Name hl-m02-vpc-flow-log; Filter All; Maximum aggregation interval 1 minute; Destination Send to CloudWatch Logs; log group /hl/m02/flow-logs; IAM role (create one or pick an existing one); Log record format AWS default or Custom format; add the lab tags. Create flow log.
Query: Console โ CloudWatch โ Logs โ Logs Insights, choose the log group. Logs Insights recognises VPC flow-log fields automatically (srcAddr, dstAddr, srcPort, dstPort, protocol, action, bytes, packets, interfaceId, logStatus).
Reachability Analyzer: Console โ VPC โ Reachability Analyzer โ Create and analyze path. Network Access Analyzer: Console โ VPC โ Network Access Analyzer โ Network Access Scopes.
AWS CLI
$p = '--profile', 'learn-sandbox', '--region', 'us-east-1'
# 1. Log group with a 7-day retention (never leave retention at "never expire")
aws logs create-log-group @p --log-group-name /hl/m02/flow-logs --tags Project=cea-lab,Lab=m02-l14,Environment=learn
aws logs put-retention-policy @p --log-group-name /hl/m02/flow-logs --retention-in-days 7
# 2. Flow log on the whole VPC, all traffic, 1-minute aggregation, to CloudWatch Logs
aws ec2 create-flow-logs @p --resource-type VPC --resource-ids vpc-0123456789abcdef0 `
--traffic-type ALL --max-aggregation-interval 60 `
--log-destination-type cloud-watch-logs --log-group-name /hl/m02/flow-logs `
--deliver-logs-permission-arn arn:aws:iam::111122223333:role/hl-m02-flow-logs `
--tag-specifications "ResourceType=vpc-flow-log,Tags=[{Key=Project,Value=cea-lab},{Key=Lab,Value=m02-l14}]"
# 3. Alternative: S3 in Parquet with hourly Hive-style partitions (cheaper at scale, Athena-friendly)
aws ec2 create-flow-logs @p --resource-type VPC --resource-ids vpc-0123456789abcdef0 `
--traffic-type ALL --max-aggregation-interval 600 `
--log-destination-type s3 --log-destination arn:aws:s3:::hl-log-archive-111122223333/vpc-flow-logs/ `
--destination-options FileFormat=parquet,HiveCompatiblePartitions=true,PerHourPartition=true
# 4. Is it working?
aws ec2 describe-flow-logs @p --filter Name=resource-id,Values=vpc-0123456789abcdef0 `
--query "FlowLogs[].{Id:FlowLogId,Status:FlowLogStatus,Delivery:DeliverLogsStatus,Dest:LogDestinationType}"
# 5. A Logs Insights query from the CLI: top rejected sources in the last hour
$end = [DateTimeOffset]::UtcNow.ToUnixTimeSeconds(); $start = $end - 3600
$q = aws logs start-query @p --log-group-name /hl/m02/flow-logs --start-time $start --end-time $end `
--query-string "filter action = 'REJECT' | stats count(*) as attempts by srcAddr, dstPort | sort attempts desc | limit 20" `
--query queryId --output text
aws logs get-query-results @p --query-id $qUseful Logs Insights queries:
# Who talked to the database port, and was it allowed?
fields @timestamp, srcAddr, dstAddr, dstPort, action
| filter dstPort = 5432
| stats count(*) as flows, sum(bytes) as bytes by srcAddr, action
| sort flows desc
# Top talkers leaving the VPC (possible exfiltration) - destination not in 10.0.0.0/8
filter action = 'ACCEPT' and not isIpv4InSubnet(dstAddr, '10.0.0.0/8')
| stats sum(bytes) as bytesOut by srcAddr, dstAddr
| sort bytesOut desc
| limit 20
# NACL return-path problem: replies rejected (source port is a well-known service port)
filter action = 'REJECT' and srcPort in [443, 5432, 1433]
| stats count(*) by interfaceId, srcAddr, dstAddr, srcPortisIpv4InSubnet is a Logs Insights IP function โ VERIFY availability in your Region's query language version.
Athena over the Parquet output (create the table from the console's Generate Athena integration option on the flow log, or define it yourself):
-- Bytes sent from the Payments test VPC to a suspicious IP, per hour, last two days
SELECT date_trunc('hour', from_unixtime("start")) AS hour,
srcaddr, dstaddr, SUM(bytes) AS bytes_out
FROM vpc_flow_logs
WHERE dstaddr = '203.0.113.66'
AND action = 'ACCEPT'
AND "date" >= date_format(current_date - interval '2' day, '%Y/%m/%d')
GROUP BY 1, 2, 3
ORDER BY 1;Partition column names depend on how the table was created โ VERIFY against your generated DDL. Partition filters are what keep Athena cheap: it charges per TB scanned.
Reachability Analyzer from the CLI:
$path = aws ec2 create-network-insights-path @p --source i-0aaaabbbbccccdddd0 --destination eni-0db000000000000a1 `
--protocol tcp --destination-port 5432 --query NetworkInsightsPath.NetworkInsightsPathId --output text
$an = aws ec2 start-network-insights-analysis @p --network-insights-path-id $path `
--query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text
aws ec2 describe-network-insights-analyses @p --network-insights-analysis-ids $an `
--query "NetworkInsightsAnalyses[0].{Status:Status,Found:NetworkPathFound,Why:Explanations[].ExplanationCode}"
aws ec2 delete-network-insights-analysis @p --network-insights-analysis-id $an
aws ec2 delete-network-insights-path @p --network-insights-path-id $pathInfrastructure as Code
var flowLogGroup = new LogGroup(this, "FlowLogGroup", new LogGroupProps
{
LogGroupName = "/hl/m02/flow-logs",
Retention = RetentionDays.ONE_WEEK, // bounded retention
RemovalPolicy = RemovalPolicy.DESTROY, // LEARNING SHORTCUT: production keeps evidence
});
vpc.AddFlowLog("FlowLog", new FlowLogOptions
{
Destination = FlowLogDestination.ToCloudWatchLogs(flowLogGroup), // CDK creates the IAM role
TrafficType = FlowLogTrafficType.ALL,
MaxAggregationInterval = FlowLogMaxAggregationInterval.ONE_MINUTE,
});CDK generates the delivery role for you. If your security team requires the aws:SourceAccount condition on every service role, pass your own role with FlowLogDestination.ToCloudWatchLogs(logGroup, role).
Complete code
Two runnable pieces. First, a .NET 10 file-based app that runs a Logs Insights query with the SDK and prints the result โ the same thing the SOC would automate:
#:package AWSSDK.CloudWatchLogs@4.1.0.2
#:package AWSSDK.SSO@4.1.0.2
#:package AWSSDK.SSOOIDC@4.1.0.2
using Amazon;
using Amazon.CloudWatchLogs;
using Amazon.CloudWatchLogs.Model;
using Amazon.Runtime.CredentialManagement;
// Usage: dotnet run flowquery.cs -- /hl/m02-secure-vpc/flow-logs 60
var logGroup = args.Length > 0 ? args[0] : "/hl/m02-secure-vpc/flow-logs";
var minutes = args.Length > 1 ? int.Parse(args[1]) : 60;
if (!new CredentialProfileStoreChain().TryGetAWSCredentials("learn-sandbox", out var credentials))
{
Console.Error.WriteLine("Profile learn-sandbox not found. Run: aws sso login --profile learn-sandbox");
return 1;
}
using var logs = new AmazonCloudWatchLogsClient(credentials, RegionEndpoint.USEast1);
var end = DateTimeOffset.UtcNow;
var start = await logs.StartQueryAsync(new StartQueryRequest
{
LogGroupName = logGroup,
StartTime = end.AddMinutes(-minutes).ToUnixTimeSeconds(),
EndTime = end.ToUnixTimeSeconds(),
QueryString = """
filter dstPort = 5432
| stats count(*) as flows by srcAddr, action
| sort flows desc
""",
});
GetQueryResultsResponse result;
do
{
await Task.Delay(TimeSpan.FromSeconds(2));
result = await logs.GetQueryResultsAsync(new GetQueryResultsRequest { QueryId = start.QueryId });
}
while (result.Status == QueryStatus.Scheduled || result.Status == QueryStatus.Running);
if (result.Status != QueryStatus.Complete)
{
Console.Error.WriteLine($"Query ended with status {result.Status}");
return 2;
}
Console.WriteLine($"{"srcAddr",-16} {"action",-7} {"flows",6}");
foreach (var row in result.Results ?? [])
{
var cells = row.ToDictionary(f => f.Field, f => f.Value);
Console.WriteLine($"{cells.GetValueOrDefault("srcAddr"),-16} {cells.GetValueOrDefault("action"),-7} {cells.GetValueOrDefault("flows"),6}");
}
Console.WriteLine($"Scanned {result.Statistics?.BytesScanned:N0} bytes.");
return 0;Second, the offline evidence checker from the lab, labs/m02-secure-vpc/src/Harbourline.Networking.FlowLogs. It parses default-format records and decides whether anything other than the allowed sources was accepted into a protected range. It is covered by unit tests with fixture files, so it runs in CI without AWS:
public static bool TryParse(string? line, out FlowLogRecord? record)
{
record = null;
if (string.IsNullOrWhiteSpace(line))
{
return false;
}
var f = line.Split(' ', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries);
if (f.Length != DefaultFieldCount
|| !int.TryParse(f[0], NumberStyles.None, CultureInfo.InvariantCulture, out var version)
|| !TryUnixSeconds(f[10], out var start)
|| !TryUnixSeconds(f[11], out var end)
|| !TryOptionalInt(f[5], out var srcPort)
|| !TryOptionalInt(f[6], out var dstPort)
|| !TryOptionalInt(f[7], out var protocol)
|| !TryOptionalLong(f[8], out var packets)
|| !TryOptionalLong(f[9], out var bytes)
|| !TryAction(f[12], out var action))
{
return false;
}
record = new FlowLogRecord(version, f[1], f[2], OrNull(f[3]), OrNull(f[4]),
srcPort, dstPort, protocol, packets, bytes, start, end, action, f[13]);
return true;
}cd labs/m02-secure-vpc
dotnet run --project src/Harbourline.Networking.FlowLogCheck -- `
--file tests/Harbourline.Networking.FlowLogs.Tests/fixtures/isolated.txt `
--protected 10.20.128.0/18 --allowed 10.20.64.0/18 --port 5432Expected:
Records parsed: 9 (skipped 0 lines that are not default-format records)
Protected range 10.20.128.0/18 port 5432; allowed sources: 10.20.64.0/18
ACCEPT from allowed sources : 2
REJECT (blocked attempts) : 4
rejected from 10.20.0.17 x3
rejected from 10.20.16.9 x1
VERDICT: ISOLATED - no accepted connection from outside the allowed sources.Code explained line by line
flowquery.cs:
#:package โฆโ .NET 10 file-based apps declare NuGet packages inline; no.csprojis needed.AWSSDK.SSOandAWSSDK.SSOOIDCare needed at runtime so the SDK can turn an IAM Identity Center profile into credentials.CredentialProfileStoreChain().TryGetAWSCredentials("learn-sandbox", โฆ)โ loads the SSO profile from~/.aws/config. Never hard-code keys.new AmazonCloudWatchLogsClient(credentials, RegionEndpoint.USEast1)โ an SDK v4 client;using vardisposes it.StartQueryAsyncโ Logs Insights is asynchronous: you start a query and get aQueryId. Times are Unix seconds.- The raw string literal (
"""โฆ""") holds the query; flow-log fields are discovered automatically. - The
do โฆ whileloop pollsGetQueryResultsAsyncevery two seconds until the status leavesScheduled/Running. In production, add a timeout and cancellation token. - Each result row is a list of
ResultField(Field,Value) pairs;ToDictionarymakes them addressable by name. Statistics.BytesScannedโ what you are billed for. Print it; it teaches cost awareness.
FlowLogRecord.TryParse:
- Returns
falseinstead of throwing โ log files contain headers, blank lines and custom formats, and an investigation tool must not stop at line 3. Split(' ', RemoveEmptyEntries | TrimEntries)tolerates double spaces and trailing whitespace.f.Length != 14rejects anything that is not the default format, so you never misread a custom record's fields.TryOptionalInt/Longtreat-as "no value", which is howNODATAandSKIPDATArecords look.TryUnixSecondsturnsstart/endintoDateTimeOffsetโ always UTC.TryActionmapsACCEPT/REJECTto an enum; anything else (except-) is invalid.
The IsolationChecker that consumes these records is explained line by line in the secure VPC lab.
Security considerations
โ Learning shortcut โ not for production
The lab keeps flow logs in the same account with 7-day retention and RemovalPolicy.DESTROY, so teardown is clean. A bank writes flow logs to a central log-archive account (S3 with Object Lock, bucket policy denying deletes, KMS encryption), with retention set by its records policy, so an attacker with admin rights in a workload account cannot erase the evidence.
- Protect the delivery role. Use the
aws:SourceAccount(and optionallyaws:SourceArn) condition in the trust policy; scope permissions to the one log group. - Encrypt. Use a KMS key on the log group or bucket; restrict
kms:Decryptto the SOC role. - Flow logs contain personal data. IP addresses can be personal data under GDPR / UK GDPR (for example, customer IPs hitting an internet-facing load balancer). Agree retention and access with your data-protection officer; do not copy flow logs into ad-hoc spreadsheets.
- Detect "logging turned off". Alert on CloudTrail
DeleteFlowLogsevents, and use an AWS Config rule (vpc-flow-logs-enabled) or Security Hub CSPM control to flag VPCs without flow logs. - Reachability Analyzer permissions let someone map your network. Grant them to network and security roles, not to every developer.
Banking use case
Harbourline's SOC runs three standing jobs on flow logs from every production VPC, stored as Parquet in the log-archive account:
- Egress anomaly: an Athena query each hour totals bytes from each workload subnet to destinations outside the bank's ranges and outside AWS service prefixes; more than three standard deviations above the 30-day baseline opens an incident.
- Data-tier guard: the
FlowLogChecklogic runs nightly over the data subnets of the Payments and Accounts VPCs. Any accepted flow into10.17.128.0/18on 5432 from outside the app tier is a P1 incident. - Scanning detection: a CloudWatch metric filter on REJECT records feeds an alarm when one source is rejected on more than 50 distinct ports within five minutes (Amazon GuardDuty also analyses flow logs independently for this kind of behaviour).
During the 02:10 incident above, the analyst used Athena to bound the exfiltration window (first ACCEPT to the bad IP at 01:52, 38 MB out), Logs Insights to see that the instance had also attempted, and been REJECTed by, the tokenisation service, and Reachability Analyzer to confirm no path existed from that subnet to the core banking VPN attachment. The incident report cited all three โ dated, reproducible evidence.
Common mistakes
- No retention on the log group. CloudWatch Logs keeps data forever by default โ the bill grows every month. Always set retention (or lifecycle rules on S3).
- REJECT-only logging for security. Most real breaches are accepted traffic. Capture
ALL. - Expecting DNS lookups in flow logs. Queries to the Amazon resolver are not logged; use Resolver query logging.
- Reading
srcaddrbehind a NAT gateway or Transit Gateway as the true source. On intermediate ENIs,srcaddris the gateway's own address; addpkt-srcaddr. - Trying to edit a flow log's format. You can't; create a new one and delete the old.
- Calling a single "not reachable" analysis proof. Always pair it with a positive control (see the lab).
- Forgetting Reachability Analyzer paths and analyses. They are cheap but clutter; tag and delete them.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
DeliverLogsStatus: FAILED | Role trust or permissions wrong; KMS key policy missing the service | Check the role trusts vpc-flow-logs.amazonaws.com and can PutLogEvents; add the logs service to the key policy |
| No log streams after 15 minutes | No traffic on any ENI, or flow log on the wrong VPC | Generate traffic; describe-flow-logs and check ResourceId |
S3 delivery Access Denied | Bucket policy does not allow delivery.logs.amazonaws.com | Add the documented bucket-policy statements (with aws:SourceAccount) |
| Logs Insights fields are empty | Custom format in CloudWatch Logs | Auto-discovery covers the default format; use parse for custom formats โ VERIFY |
Many SKIPDATA records | Internal capacity limits during the interval | Expected occasionally; if frequent, raise a support case โ and note the gap in your evidence |
Reachability analysis failed | Unsupported component or permissions | Read StatusMessage; check ec2:Describe* and tiros:* permissions โ VERIFY action prefix |
| Path found but app still fails | Analyzer is layer 3/4 only | Check TLS, credentials, DNS, the application |
| Athena query scans terabytes | No partition filter | Filter on partition columns (date/hour) |
Cost and cleanup
Cost (us-east-1, checked 2026-11-18):
- Flow logs to CloudWatch Logs: vended-log ingestion USD 0.50 per GB for the first 10 TB per month, then tiered down (0.25, 0.10, 0.05). Storage after ingestion is about USD 0.03 per GB-month โ VERIFY. A lab VPC for an hour produces kilobytes โ effectively free.
- Flow logs to S3: a lower vended-log rate (about USD 0.25 per GB for the first 10 TB โ VERIFY) plus optional Parquet conversion per GB โ VERIFY, plus S3 storage.
- Logs Insights: USD 0.005 per GB scanned โ VERIFY.
- Reachability Analyzer: USD 0.10 per analysis. Network Access Analyzer: USD 0.002 per ENI analysed.
Bank-scale example: 200 VPCs producing 5 TB of flow records a month. To CloudWatch Logs: 5,000 GB ร USD 0.50 = USD 2,500/month before storage. To S3: roughly half that. This is why banks send the bulk to S3 and only selected VPCs to CloudWatch Logs.
Free alternative: use the C# checker with the fixture files, and read the flow log record examples in the AWS docs โ no deployment needed.
Cleanup:
$p = '--profile', 'learn-sandbox', '--region', 'us-east-1'
aws ec2 describe-flow-logs @p --filter Name=tag:Lab,Values=m02-l14 --query "FlowLogs[].FlowLogId"
aws ec2 delete-flow-logs @p --flow-log-ids fl-0123456789abcdef0
aws logs delete-log-group @p --log-group-name /hl/m02/flow-logs
aws iam delete-role-policy --profile learn-sandbox --role-name hl-m02-flow-logs --policy-name write-flow-logs
aws iam delete-role --profile learn-sandbox --role-name hl-m02-flow-logs
aws ec2 describe-network-insights-paths @p --query "NetworkInsightsPaths[].NetworkInsightsPathId"
aws resourcegroupstaggingapi get-resources @p --tag-filters Key=Project,Values=cea-lab --query "ResourceTagMappingList[].ResourceARN"Exercises
- ๐ข Guided: In the secure VPC lab, run the three Logs Insights queries above. For each, write one sentence on what it shows.
- ๐ก Challenge: Create a second flow log on the lab VPC with the custom format above, sent to S3 in Parquet with hourly partitions. After 20 minutes, generate the Athena table and write a query that shows, per hour, accepted versus rejected flows into the data tier.
- ๐ด Live problem: The Accounts API team says: "Since yesterday's NACL change, about 2% of calls to the database time out, but the security group looks fine." Flow logs show
ACCEPTfor10.18.70.x โ 10.18.130.12:5432and someREJECTrecords for10.18.130.12:5432 โ 10.18.70.xwith destination ports between 61000 and 65535. Diagnose and propose the fix.
Assignment
Write a one-page flow-log standard for Harbourline: scope, traffic type, interval, format (with the field list), destination(s), retention, encryption, who can read the logs, and the three detections you would build first. Add the cost estimate for 50 VPCs at 20 GB/month each. Commit it to your portfolio repository.
Solutions
- Port 5432 by source and action: two sources โ the app instance with
ACCEPTand the probe withREJECT. This is the lab's core evidence. - Top talkers leaving the VPC: in a NAT-free lab, probably nothing beyond AWS service prefixes reached through gateway endpoints โ confirming the app tier has no internet egress.
- Rejected replies on service ports: should be empty. If not, a NACL is missing an ephemeral-port rule.
aws ec2 create-flow-logs --profile learn-sandbox --resource-type VPC --resource-ids <VpcId> `
--traffic-type ALL --max-aggregation-interval 60 --log-destination-type s3 `
--log-destination arn:aws:s3:::<your-bucket>/flow/ `
--log-format '${version} ${account-id} ${vpc-id} ${subnet-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${tcp-flags} ${flow-direction}' `
--destination-options FileFormat=parquet,HiveCompatiblePartitions=true,PerHourPartition=trueSELECT date_trunc('hour', from_unixtime("start")) AS hour, action, COUNT(*) AS flows
FROM vpc_flow_logs
WHERE dstport = 5432 AND dstaddr LIKE '10.20.1%' -- 10.20.128-191 starts with 10.20.1
GROUP BY 1, 2
ORDER BY 1, 2;Why: the custom format adds subnet-id and flow-direction, so you can filter by subnet instead of a fragile LIKE; try rewriting it with subnet_id IN (...). Single quotes around the format stop PowerShell expanding ${...}.
The ACCEPT records show requests reaching the database. The REJECT records are the database's replies going back to client ports 61000โ65535. Yesterday's change narrowed the data-tier NACL outbound rule to the Linux default ephemeral range (32768โ60999). Most app hosts are Linux, but a few Windows-based batch hosts use 49152โ65535, and some of their connections land above 60999. Because NACLs are stateless, those replies are dropped โ about 2% of connections.
Fix: widen the outbound NACL rule to 1024โ65535 towards the app subnets (the range that covers every common client OS and NAT gateways), keeping the destination restricted to the app tier CIDRs. Confirm with Logs Insights that the reply REJECTs stop. A security group would never have had this problem, which is why the SG is the primary control and the NACL is kept deliberately simple.
Knowledge check
Answer: The network ACL (outbound rule on the server's subnet, or inbound rule on the client's). Security groups are stateful and would have allowed the reply automatically.
Answer: Traffic to the Amazon-provided DNS server is excluded from flow logs by design. Use Route 53 Resolver query logging.
Answer: When traffic passes through an intermediate ENI (NAT gateway, Transit Gateway attachment, load balancer) and you need the original source rather than the gateway's address.
Answer: Reachability Analyzer checks one sourceโdestination path. Network Access Analyzer checks a rule (a scope) against every path in the account and lists violations.
Answer: Send the bulk to S3 (cheaper vended rate, Parquet, lifecycle to colder storage); use a 10-minute interval for low-risk VPCs; reduce scope to the VPCs that need live alarms. Avoid REJECT-only for security VPCs.
Interview questions
- "How do you tell from flow logs whether a security group or a NACL blocked traffic?" โ Look for the stateless signature: accepted request plus rejected reply means NACL. A single reject could be either; confirm with Reachability Analyzer's explanation codes.
- "Where would you store flow logs in a multi-account bank, and why?" โ S3 in a central log-archive account, Parquet with hourly partitions, Object Lock, KMS, lifecycle rules; workload accounts cannot delete evidence. CloudWatch Logs only where live alarms justify the cost.
- "What don't flow logs show?" โ Payload, DNS queries to the Amazon resolver, metadata and Time Sync traffic, and which control rejected a flow. They're also delayed by minutes.
- "How would you continuously prove no database is internet-reachable?" โ A Network Access Analyzer scope (internet gateway โ ENIs tagged as data tier) run on a schedule, plus a Config rule for public RDS, plus flow-log detections.
Certification practice
An application in subnet A cannot connect to a database in subnet B on TCP 1433. VPC Flow Logs on the database ENI show ACCEPT records for traffic from the application to port 1433 and REJECT records for traffic from port 1433 back to the application. What should the CloudOps engineer change?
A. Add an outbound rule to the database security group allowing all traffic to subnet A
B. Add an outbound rule to subnet B's network ACL allowing TCP ephemeral ports to subnet A
C. Add an inbound rule to the application security group allowing 1433 from subnet B
D. Enable tcp-flags in the flow log format
Answer: B. The reply is rejected, which is the stateless NACL signature. A and C change stateful security groups, which already permit replies. D adds diagnostics, not a fix.
A bank must keep VPC flow logs for all workload accounts for seven years and ensure that administrators of workload accounts cannot delete them. Which solution meets the requirement with the LEAST operational overhead?
A. Enable flow logs to CloudWatch Logs in each account with retention set to 10 years B. Enable flow logs in each account to an S3 bucket in a central log-archive account with S3 Object Lock in compliance mode and a lifecycle rule to an archive storage class C. Use Traffic Mirroring to copy packets to an appliance in a security account D. Export flow logs weekly from each account with a Lambda function to an on-premises SIEM
Answer: B. Cross-account S3 delivery puts the evidence outside the workload admins' reach, Object Lock prevents deletion, lifecycle reduces cost. A keeps logs where local admins can delete them. C is expensive and captures payloads. D is operationally heavy and leaves a window for tampering.
A security team must verify, across all VPCs in an account, that no network interface in subnets tagged Tier=data can be reached from any internet gateway. They want a repeatable check that lists any violating paths. Which AWS feature fits best?
A. VPC Reachability Analyzer, with one path per database ENI B. Network Access Analyzer with a scope matching paths from internet gateways to the tagged resources C. VPC Flow Logs filtered for REJECT D. AWS Firewall Manager security group policies
Answer: B. Network Access Analyzer evaluates a rule against all paths and reports matches. A works but needs one path per ENI and misses new ones. C shows only traffic that happened. D manages security groups; it does not analyse paths.
Completion checklist
- I can read every field of a default flow-log record
- I can spot the NACL "accept request, reject reply" signature
- I ran at least two Logs Insights queries on the lab VPC
- I ran a Reachability Analyzer analysis and read its explanation codes
- I deleted my extra flow logs, log groups, roles and analyzer paths
Key takeaways
- Flow logs are a door-access log for every ENI: who, where, when, how much, allowed or not โ but not what was said.
- Capture
ALL, log at VPC scope, and send the bulk to S3 in a central log-archive account; set retention everywhere. - Stateless NACLs leave a recognisable "accept request, reject reply" pattern.
- Reachability Analyzer proves one path from configuration; Network Access Analyzer proves a rule across the account.
- Evidence beats opinion: pair configuration analysis with recorded traffic and a positive control.
Further reading
- Logging IP traffic using VPC Flow Logs
- Flow log record examples
- What is Reachability Analyzer?
- Reachability Analyzer explanation codes
- Amazon VPC pricing
- Amazon CloudWatch pricing (vended logs)
Next steps: filter what is allowed to leave the VPC in the first place with AWS Network Firewall, or put the evidence tools to work in the secure VPC lab.
Independent learning resource โ not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Hybrid connectivity: Site-to-Site VPN, Direct Connect and hybrid DNS
Connect AWS to on-premises data centres with Site-to-Site VPN and AWS Direct Connect, choose a resilient connection model for a bank, and make names resolve both ways with Route 53 Resolver endpoints and forwarding rules.
AWS Network Firewall: centralised inspection and domain-based egress filtering
Understand how AWS Network Firewall inspects traffic, write stateless and Suricata-compatible stateful rules, route traffic through firewall endpoints symmetrically, and choose between distributed and centralised inspection designs.