Skip to Content
Cloud Engineering Academy
2. AWS Networking & Infrastructure

VPC Flow Logs and Reachability Analyzer: evidence of what can talk to what

Enable, read and query VPC Flow Logs for investigations, choose the right destination and format, and use Reachability Analyzer and Network Access Analyzer to prove which paths exist without sending a packet.

intermediateโฑ 1 h 40 min๐ŸŸก Low costSAA-C03SOA-C03SCS-C03

Assumes: .NET 10 LTS (file-based apps) ยท AWS SDK for .NET v4 (AWSSDK.CloudWatchLogs 4.1.0.2) ยท AWS CLI v2 ยท CDK v2 (Amazon.CDK.Lib 2.277.0) ยท Terraform AWS provider v6 ยท Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Enable VPC Flow Logs on a VPC, subnet or network interface, and choose between CloudWatch Logs, Amazon S3 and Amazon Data Firehose as the destination.
  • Read a default flow-log record field by field, and recognise the signatures of a security-group block, a network-ACL block and a NACL that forgot return traffic.
  • Add custom-format fields (vpc-id, subnet-id, tcp-flags, pkt-srcaddr, flow-direction, traffic-path) when the default record cannot answer your question.
  • Query flow logs with CloudWatch Logs Insights and with Athena over S3, and from C# with the AWS SDK for .NET v4.
  • Use Reachability Analyzer to prove a single path exists or does not, and Network Access Analyzer to prove a rule holds across a whole account.
  • Estimate and control the cost of flow logging at bank scale.

Prerequisites

Why this matters in banking

At 02:10 on a Saturday, Harbourline's security operations centre gets an alert: an EC2 instance in the Payments test VPC has been talking to an IP address that a threat-intelligence feed lists as a command-and-control server. The questions arrive immediately:

  • When did it start, and how much data left?
  • Did that instance talk to anything else inside the bank โ€” the card tokenisation service, the core banking VPN?
  • Could anything else in the estate reach that address?

Application logs cannot answer these questions; the attacker is not going to log their own activity. VPC Flow Logs can answer the first two, because they record every IP flow on every network interface whether or not the application cooperates. Reachability Analyzer and Network Access Analyzer answer the third, from configuration alone.

There is also a regulatory angle. PCI DSS expects logging of access to cardholder-data networks, and financial regulators expect a bank to reconstruct incidents. Flow logs are a cheap, always-on network record. Whether a given logging setup meets a specific requirement is a decision for your compliance team and assessor; this lesson shows you how to produce the data.

The idea in plain language

A flow log is a summary, per network interface, of who talked to whom. It is not a recording of the conversation. Every aggregation interval (one minute or ten), AWS writes one line per flow โ€” a unique combination of source IP, destination IP, source port, destination port and protocol (the "5-tuple") โ€” with packet and byte counts and one word saying whether the traffic was ACCEPTed or REJECTed by security groups and network ACLs.

Reachability Analyzer works the other way round: you name a source and a destination, and it reads every route table, security group, NACL, gateway and load balancer on the way to work out whether a packet could get through โ€” without sending one.

Analogy: flow logs are a bank's door-access log: badge number, door, time, granted or denied. They don't tell you what was said in the meeting room. Reachability Analyzer is the building plan review: given the locks and corridors, could badge X ever open door Y?

What problem the AWS services solve

ToolQuestion it answersSends traffic?Price (us-east-1, checked 2026-11-18)
VPC Flow LogsWhat flows actually happened, and were they accepted?No (passive)Vended-log ingestion, e.g. USD 0.50 per GB to CloudWatch Logs for the first 10 TB; S3 delivery is cheaper โš  VERIFY S3 rate
Reachability AnalyzerCan source A reach destination B on this protocol/port? Which component blocks it?NoUSD 0.10 per analysis
Network Access AnalyzerDoes any path violate my rule, e.g. "no internet gateway can reach a database subnet"?NoUSD 0.002 per network interface analysed
Traffic MirroringWhat exactly was in the packets?Copies real packetsUSD 0.015 per session-hour (plus the appliance)

What flow logs do not capture

Honest limits matter in an investigation. Flow logs do not record:

  • Traffic to and from the Amazon DNS resolver (the VPC+2 address). If you run your own DNS servers, that traffic is logged. Use Route 53 Resolver query logging for DNS questions.
  • Instance metadata (169.254.169.254), Amazon Time Sync (169.254.169.123), DHCP traffic and traffic to the reserved VPC router address.
  • Windows licence activation traffic for AWS-provided Windows AMIs.
  • Mirrored traffic and some traffic between endpoint network interfaces and Network Load Balancer interfaces โš  VERIFY the current full list on the flow-logs limitations page.

They are also not real-time: records arrive after the aggregation interval plus a delivery delay of several minutes. And a flow record cannot tell you which control rejected a packet โ€” only that something did.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
NSG flow logs (being retired) / VNet flow logsVPC Flow Logs (VPC, subnet, ENI, also Transit Gateway)AWS records per ENI; destination is CloudWatch Logs, S3 or Firehose rather than a storage account
Traffic Analytics (Log Analytics workspace)CloudWatch Logs Insights, Athena over S3, or a SIEMNo built-in analytics dashboard; you write queries (or use Amazon Security Lake / OpenSearch)
KQLLogs Insights query language, or SQL in AthenaSimilar pipe-style syntax in Logs Insights
Network Watcher IP flow verifyReachability Analyzer (single path)Analyzer covers the whole path (routes, gateways, LBs), not just one NSG
Network Watcher Connection troubleshootReachability AnalyzerAnalyzer never sends probes
Azure Policy / Network Manager security admin rules (for audit)Network Access Analyzer scopesScopes describe forbidden paths; findings list every match
Virtual network TAPTraffic MirroringBoth copy packets to an appliance

Architecture

How to choose:

DestinationBest forWatch out for
CloudWatch LogsQuick investigations, Logs Insights, metric filters and alarms on REJECT spikes, small estatesHighest ingestion price; set retention or you keep (and pay for) everything forever
S3Bank-scale retention, central log-archive account, Athena, Security LakeQuery latency higher; set lifecycle rules; bucket policy must allow the log delivery service
FirehoseStreaming into a SIEMFirehose charges on top; buffering adds delay

A common bank pattern is both: S3 (Parquet) in the log-archive account for long retention and forensics, and CloudWatch Logs with short retention on the few VPCs where the SOC wants live alarms.

Reading a flow-log record

The default format (version 2) has 14 space-separated fields:

version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status

A record from the secure VPC lab:

2 111122223333 eni-0db000000000000a1 10.20.64.25 10.20.128.40 41822 5432 6 6 360 1763452800 1763452860 ACCEPT OK
FieldValueMeaning
version2Default format
account-id111122223333Account that owns the ENI (fictional)
interface-ideni-0dbโ€ฆa1The database's network interface
srcaddr / dstaddr10.20.64.25 โ†’ 10.20.128.40App instance โ†’ database
srcport / dstport41822 โ†’ 5432Ephemeral client port โ†’ PostgreSQL
protocol6IANA protocol number: 6 = TCP, 17 = UDP, 1 = ICMP
packets / bytes6 / 360In this interval, this direction
start / endUnix secondsThe aggregation window
actionACCEPTSG and NACL both allowed it
log-statusOKData was recorded (NODATA = no traffic, SKIPDATA = records dropped by capacity limits)

Recognising who blocked what

Security groups are stateful; network ACLs are stateless. That difference leaves fingerprints:

Pattern on the destination ENIMost likely cause
One REJECT for the inbound request, no reply recordThe SG or the inbound NACL rule dropped the request
ACCEPT inbound request, then REJECT for the reply (source port = service port)The outbound NACL has no rule for the ephemeral return ports. The SG would have let the reply out automatically
ACCEPT inbound and ACCEPT reply, but the client still times outProblem is elsewhere: the client's own NACL inbound, a route, or the application
REJECTs from many sources to many portsSomeone scanning; check whether the ENI is reachable from where they are

To tell an SG reject from a NACL reject, ask Reachability Analyzer (below) โ€” its explanation codes name the component.

Custom formats: when the default is not enough

You can choose any field set in any order when you create the flow log (you cannot change it later โ€” create a new flow log). Useful additions:

FieldVersionWhy you'd add it
vpc-id, subnet-id, instance-id3Correlate without a lookup table; essential in a central log archive
tcp-flags3See who opened the connection: 2 = SYN, 18 = SYN-ACK, 1 = FIN, 4 = RST
type3IPv4 / IPv6 / EFA
pkt-srcaddr, pkt-dstaddr3The original addresses when traffic passes through a NAT gateway, Transit Gateway or other intermediate ENI
region, az-id4Cross-AZ traffic analysis (it costs money)
pkt-src-aws-service, pkt-dst-aws-service5Names the AWS service behind a public IP, e.g. S3
flow-direction5ingress / egress relative to the ENI
traffic-path5For egress: how it left, e.g. 8 = through an internet gateway, 1 = within the VPC

AWS has added further fields since version 5 (for example ECS task metadata) โš  VERIFY the latest field list before designing a format. A bank-friendly custom format:

${version} ${account-id} ${vpc-id} ${subnet-id} ${instance-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${tcp-flags} ${pkt-srcaddr} ${pkt-dstaddr} ${flow-direction} ${traffic-path} ${pkt-dst-aws-service}

Reachability Analyzer and Network Access Analyzer

Reachability Analyzer takes a path โ€” source, destination, protocol and optional ports/IPs โ€” and an analysis of that path at a point in time. Sources and destinations can be instances, ENIs, internet gateways, VPC endpoints, peering connections, Transit Gateways, VPN gateways and more. The result is NetworkPathFound: true with the hop-by-hop path, or false with explanation codes such as:

CodePlain meaning
ENI_SG_RULES_MISMATCHA security group on this ENI has no rule that matches
SUBNET_ACL_RESTRICTIONThe subnet's network ACL denies it
NO_ROUTE_TO_DESTINATIONA route table has no applicable route
IGW_PRIVATE_IP_ASSOCIATION_FOR_INGRESSThe internet gateway only delivers to public IPs mapped to an ENI
VPC_BLOCK_PUBLIC_ACCESS_ENABLEDVPC Block Public Access stopped it
TGW_RTB_NO_ROUTE_TO_TGW_ATTACHMENTA Transit Gateway route table has no route to the attachment

Analyses can span accounts in an AWS Organization once you enable trusted access and register a delegated administrator.

Network Access Analyzer inverts the question. You define a Network Access Scope โ€” "match any path from an internet gateway to a network interface tagged Tier=data" โ€” and the analyzer lists every path in the account that matches. An empty finding list is evidence that the rule holds everywhere, not just for the one path you thought of. AWS ships a few starter scopes in the console.

Use Reachability Analyzer whenโ€ฆUse Network Access Analyzer whenโ€ฆ
Debugging one connection ("why can't app reach DB?")Proving a policy ("no data subnet is internet-reachable")
You need a positive and a negative control in a testYou want continuous assurance across many VPCs
Charged per analysisCharged per ENI analysed

Step-by-step implementation

1. Decide scope, traffic type and interval

  • Scope: create flow logs at the VPC level so new subnets and ENIs are covered automatically. Add ENI-level logs only for special cases.
  • Traffic type: ALL for security work. REJECT only is cheaper but hides successful exfiltration.
  • Interval: 60 seconds for investigations and labs; 600 seconds (the default) for low-risk VPCs at scale.

2. Create the destination and permissions

For CloudWatch Logs, create a log group with retention and an IAM role that the vpc-flow-logs.amazonaws.com service can assume. Add an aws:SourceAccount condition to the trust policy so another account's flow log cannot use your role (the "confused deputy" problem).

For S3, no role is needed; the bucket policy allows the log delivery service principal. AWS adds the required statements automatically if you have permission to change the bucket policy.

3. Create the flow log

Use the CLI or IaC below. Confirm with describe-flow-logs that FlowLogStatus is ACTIVE and DeliverLogsStatus is SUCCESS.

4. Generate and find traffic

Wait at least one interval plus a few minutes. In CloudWatch Logs, each ENI gets its own log stream named after the ENI ID.

5. Query, then prove

Run Logs Insights queries for the investigation, then use Reachability Analyzer to confirm why rejected flows were rejected.

AWS Console instructions

Create a flow log: Console โ†’ VPC โ†’ Your VPCs โ†’ select the VPC โ†’ Flow logs tab โ†’ Create flow log. Name hl-m02-vpc-flow-log; Filter All; Maximum aggregation interval 1 minute; Destination Send to CloudWatch Logs; log group /hl/m02/flow-logs; IAM role (create one or pick an existing one); Log record format AWS default or Custom format; add the lab tags. Create flow log.

Query: Console โ†’ CloudWatch โ†’ Logs โ†’ Logs Insights, choose the log group. Logs Insights recognises VPC flow-log fields automatically (srcAddr, dstAddr, srcPort, dstPort, protocol, action, bytes, packets, interfaceId, logStatus).

Reachability Analyzer: Console โ†’ VPC โ†’ Reachability Analyzer โ†’ Create and analyze path. Network Access Analyzer: Console โ†’ VPC โ†’ Network Access Analyzer โ†’ Network Access Scopes.

AWS CLI

$p = '--profile', 'learn-sandbox', '--region', 'us-east-1'

# 1. Log group with a 7-day retention (never leave retention at "never expire")
aws logs create-log-group @p --log-group-name /hl/m02/flow-logs --tags Project=cea-lab,Lab=m02-l14,Environment=learn
aws logs put-retention-policy @p --log-group-name /hl/m02/flow-logs --retention-in-days 7

# 2. Flow log on the whole VPC, all traffic, 1-minute aggregation, to CloudWatch Logs
aws ec2 create-flow-logs @p --resource-type VPC --resource-ids vpc-0123456789abcdef0 `
  --traffic-type ALL --max-aggregation-interval 60 `
  --log-destination-type cloud-watch-logs --log-group-name /hl/m02/flow-logs `
  --deliver-logs-permission-arn arn:aws:iam::111122223333:role/hl-m02-flow-logs `
  --tag-specifications "ResourceType=vpc-flow-log,Tags=[{Key=Project,Value=cea-lab},{Key=Lab,Value=m02-l14}]"

# 3. Alternative: S3 in Parquet with hourly Hive-style partitions (cheaper at scale, Athena-friendly)
aws ec2 create-flow-logs @p --resource-type VPC --resource-ids vpc-0123456789abcdef0 `
  --traffic-type ALL --max-aggregation-interval 600 `
  --log-destination-type s3 --log-destination arn:aws:s3:::hl-log-archive-111122223333/vpc-flow-logs/ `
  --destination-options FileFormat=parquet,HiveCompatiblePartitions=true,PerHourPartition=true

# 4. Is it working?
aws ec2 describe-flow-logs @p --filter Name=resource-id,Values=vpc-0123456789abcdef0 `
  --query "FlowLogs[].{Id:FlowLogId,Status:FlowLogStatus,Delivery:DeliverLogsStatus,Dest:LogDestinationType}"

# 5. A Logs Insights query from the CLI: top rejected sources in the last hour
$end = [DateTimeOffset]::UtcNow.ToUnixTimeSeconds(); $start = $end - 3600
$q = aws logs start-query @p --log-group-name /hl/m02/flow-logs --start-time $start --end-time $end `
  --query-string "filter action = 'REJECT' | stats count(*) as attempts by srcAddr, dstPort | sort attempts desc | limit 20" `
  --query queryId --output text
aws logs get-query-results @p --query-id $q

Useful Logs Insights queries:

# Who talked to the database port, and was it allowed?
fields @timestamp, srcAddr, dstAddr, dstPort, action
| filter dstPort = 5432
| stats count(*) as flows, sum(bytes) as bytes by srcAddr, action
| sort flows desc

# Top talkers leaving the VPC (possible exfiltration) - destination not in 10.0.0.0/8
filter action = 'ACCEPT' and not isIpv4InSubnet(dstAddr, '10.0.0.0/8')
| stats sum(bytes) as bytesOut by srcAddr, dstAddr
| sort bytesOut desc
| limit 20

# NACL return-path problem: replies rejected (source port is a well-known service port)
filter action = 'REJECT' and srcPort in [443, 5432, 1433]
| stats count(*) by interfaceId, srcAddr, dstAddr, srcPort

isIpv4InSubnet is a Logs Insights IP function โš  VERIFY availability in your Region's query language version.

Athena over the Parquet output (create the table from the console's Generate Athena integration option on the flow log, or define it yourself):

-- Bytes sent from the Payments test VPC to a suspicious IP, per hour, last two days
SELECT date_trunc('hour', from_unixtime("start")) AS hour,
       srcaddr, dstaddr, SUM(bytes) AS bytes_out
FROM   vpc_flow_logs
WHERE  dstaddr = '203.0.113.66'
  AND  action = 'ACCEPT'
  AND  "date" >= date_format(current_date - interval '2' day, '%Y/%m/%d')
GROUP  BY 1, 2, 3
ORDER  BY 1;

Partition column names depend on how the table was created โš  VERIFY against your generated DDL. Partition filters are what keep Athena cheap: it charges per TB scanned.

Reachability Analyzer from the CLI:

$path = aws ec2 create-network-insights-path @p --source i-0aaaabbbbccccdddd0 --destination eni-0db000000000000a1 `
  --protocol tcp --destination-port 5432 --query NetworkInsightsPath.NetworkInsightsPathId --output text
$an = aws ec2 start-network-insights-analysis @p --network-insights-path-id $path `
  --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text
aws ec2 describe-network-insights-analyses @p --network-insights-analysis-ids $an `
  --query "NetworkInsightsAnalyses[0].{Status:Status,Found:NetworkPathFound,Why:Explanations[].ExplanationCode}"
aws ec2 delete-network-insights-analysis @p --network-insights-analysis-id $an
aws ec2 delete-network-insights-path @p --network-insights-path-id $path

Infrastructure as Code

FlowLogs in a CDK stack
var flowLogGroup = new LogGroup(this, "FlowLogGroup", new LogGroupProps
{
    LogGroupName = "/hl/m02/flow-logs",
    Retention = RetentionDays.ONE_WEEK,      // bounded retention
    RemovalPolicy = RemovalPolicy.DESTROY,   // LEARNING SHORTCUT: production keeps evidence
});

vpc.AddFlowLog("FlowLog", new FlowLogOptions
{
    Destination = FlowLogDestination.ToCloudWatchLogs(flowLogGroup), // CDK creates the IAM role
    TrafficType = FlowLogTrafficType.ALL,
    MaxAggregationInterval = FlowLogMaxAggregationInterval.ONE_MINUTE,
});

CDK generates the delivery role for you. If your security team requires the aws:SourceAccount condition on every service role, pass your own role with FlowLogDestination.ToCloudWatchLogs(logGroup, role).

Complete code

Two runnable pieces. First, a .NET 10 file-based app that runs a Logs Insights query with the SDK and prints the result โ€” the same thing the SOC would automate:

flowquery.cs
#:package AWSSDK.CloudWatchLogs@4.1.0.2
#:package AWSSDK.SSO@4.1.0.2
#:package AWSSDK.SSOOIDC@4.1.0.2

using Amazon;
using Amazon.CloudWatchLogs;
using Amazon.CloudWatchLogs.Model;
using Amazon.Runtime.CredentialManagement;

// Usage: dotnet run flowquery.cs -- /hl/m02-secure-vpc/flow-logs 60
var logGroup = args.Length > 0 ? args[0] : "/hl/m02-secure-vpc/flow-logs";
var minutes = args.Length > 1 ? int.Parse(args[1]) : 60;

if (!new CredentialProfileStoreChain().TryGetAWSCredentials("learn-sandbox", out var credentials))
{
    Console.Error.WriteLine("Profile learn-sandbox not found. Run: aws sso login --profile learn-sandbox");
    return 1;
}

using var logs = new AmazonCloudWatchLogsClient(credentials, RegionEndpoint.USEast1);

var end = DateTimeOffset.UtcNow;
var start = await logs.StartQueryAsync(new StartQueryRequest
{
    LogGroupName = logGroup,
    StartTime = end.AddMinutes(-minutes).ToUnixTimeSeconds(),
    EndTime = end.ToUnixTimeSeconds(),
    QueryString = """
        filter dstPort = 5432
        | stats count(*) as flows by srcAddr, action
        | sort flows desc
        """,
});

GetQueryResultsResponse result;
do
{
    await Task.Delay(TimeSpan.FromSeconds(2));
    result = await logs.GetQueryResultsAsync(new GetQueryResultsRequest { QueryId = start.QueryId });
}
while (result.Status == QueryStatus.Scheduled || result.Status == QueryStatus.Running);

if (result.Status != QueryStatus.Complete)
{
    Console.Error.WriteLine($"Query ended with status {result.Status}");
    return 2;
}

Console.WriteLine($"{"srcAddr",-16} {"action",-7} {"flows",6}");
foreach (var row in result.Results ?? [])
{
    var cells = row.ToDictionary(f => f.Field, f => f.Value);
    Console.WriteLine($"{cells.GetValueOrDefault("srcAddr"),-16} {cells.GetValueOrDefault("action"),-7} {cells.GetValueOrDefault("flows"),6}");
}

Console.WriteLine($"Scanned {result.Statistics?.BytesScanned:N0} bytes.");
return 0;

Second, the offline evidence checker from the lab, labs/m02-secure-vpc/src/Harbourline.Networking.FlowLogs. It parses default-format records and decides whether anything other than the allowed sources was accepted into a protected range. It is covered by unit tests with fixture files, so it runs in CI without AWS:

FlowLogRecord.cs (parsing excerpt)
public static bool TryParse(string? line, out FlowLogRecord? record)
{
    record = null;
    if (string.IsNullOrWhiteSpace(line))
    {
        return false;
    }

    var f = line.Split(' ', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries);
    if (f.Length != DefaultFieldCount
        || !int.TryParse(f[0], NumberStyles.None, CultureInfo.InvariantCulture, out var version)
        || !TryUnixSeconds(f[10], out var start)
        || !TryUnixSeconds(f[11], out var end)
        || !TryOptionalInt(f[5], out var srcPort)
        || !TryOptionalInt(f[6], out var dstPort)
        || !TryOptionalInt(f[7], out var protocol)
        || !TryOptionalLong(f[8], out var packets)
        || !TryOptionalLong(f[9], out var bytes)
        || !TryAction(f[12], out var action))
    {
        return false;
    }

    record = new FlowLogRecord(version, f[1], f[2], OrNull(f[3]), OrNull(f[4]),
        srcPort, dstPort, protocol, packets, bytes, start, end, action, f[13]);
    return true;
}
cd labs/m02-secure-vpc
dotnet run --project src/Harbourline.Networking.FlowLogCheck -- `
  --file tests/Harbourline.Networking.FlowLogs.Tests/fixtures/isolated.txt `
  --protected 10.20.128.0/18 --allowed 10.20.64.0/18 --port 5432

Expected:

Records parsed: 9 (skipped 0 lines that are not default-format records)
Protected range 10.20.128.0/18 port 5432; allowed sources: 10.20.64.0/18
ACCEPT from allowed sources : 2
REJECT (blocked attempts)   : 4
  rejected from 10.20.0.17      x3
  rejected from 10.20.16.9      x1
VERDICT: ISOLATED - no accepted connection from outside the allowed sources.

Code explained line by line

flowquery.cs:

  1. #:package โ€ฆ โ€” .NET 10 file-based apps declare NuGet packages inline; no .csproj is needed. AWSSDK.SSO and AWSSDK.SSOOIDC are needed at runtime so the SDK can turn an IAM Identity Center profile into credentials.
  2. CredentialProfileStoreChain().TryGetAWSCredentials("learn-sandbox", โ€ฆ) โ€” loads the SSO profile from ~/.aws/config. Never hard-code keys.
  3. new AmazonCloudWatchLogsClient(credentials, RegionEndpoint.USEast1) โ€” an SDK v4 client; using var disposes it.
  4. StartQueryAsync โ€” Logs Insights is asynchronous: you start a query and get a QueryId. Times are Unix seconds.
  5. The raw string literal ("""โ€ฆ""") holds the query; flow-log fields are discovered automatically.
  6. The do โ€ฆ while loop polls GetQueryResultsAsync every two seconds until the status leaves Scheduled/Running. In production, add a timeout and cancellation token.
  7. Each result row is a list of ResultField (Field, Value) pairs; ToDictionary makes them addressable by name.
  8. Statistics.BytesScanned โ€” what you are billed for. Print it; it teaches cost awareness.

FlowLogRecord.TryParse:

  1. Returns false instead of throwing โ€” log files contain headers, blank lines and custom formats, and an investigation tool must not stop at line 3.
  2. Split(' ', RemoveEmptyEntries | TrimEntries) tolerates double spaces and trailing whitespace.
  3. f.Length != 14 rejects anything that is not the default format, so you never misread a custom record's fields.
  4. TryOptionalInt/Long treat - as "no value", which is how NODATA and SKIPDATA records look.
  5. TryUnixSeconds turns start/end into DateTimeOffset โ€” always UTC.
  6. TryAction maps ACCEPT/REJECT to an enum; anything else (except -) is invalid.

The IsolationChecker that consumes these records is explained line by line in the secure VPC lab.

Security considerations

โš  Learning shortcut โ€” not for production

The lab keeps flow logs in the same account with 7-day retention and RemovalPolicy.DESTROY, so teardown is clean. A bank writes flow logs to a central log-archive account (S3 with Object Lock, bucket policy denying deletes, KMS encryption), with retention set by its records policy, so an attacker with admin rights in a workload account cannot erase the evidence.

  • Protect the delivery role. Use the aws:SourceAccount (and optionally aws:SourceArn) condition in the trust policy; scope permissions to the one log group.
  • Encrypt. Use a KMS key on the log group or bucket; restrict kms:Decrypt to the SOC role.
  • Flow logs contain personal data. IP addresses can be personal data under GDPR / UK GDPR (for example, customer IPs hitting an internet-facing load balancer). Agree retention and access with your data-protection officer; do not copy flow logs into ad-hoc spreadsheets.
  • Detect "logging turned off". Alert on CloudTrail DeleteFlowLogs events, and use an AWS Config rule (vpc-flow-logs-enabled) or Security Hub CSPM control to flag VPCs without flow logs.
  • Reachability Analyzer permissions let someone map your network. Grant them to network and security roles, not to every developer.

Banking use case

Harbourline's SOC runs three standing jobs on flow logs from every production VPC, stored as Parquet in the log-archive account:

  1. Egress anomaly: an Athena query each hour totals bytes from each workload subnet to destinations outside the bank's ranges and outside AWS service prefixes; more than three standard deviations above the 30-day baseline opens an incident.
  2. Data-tier guard: the FlowLogCheck logic runs nightly over the data subnets of the Payments and Accounts VPCs. Any accepted flow into 10.17.128.0/18 on 5432 from outside the app tier is a P1 incident.
  3. Scanning detection: a CloudWatch metric filter on REJECT records feeds an alarm when one source is rejected on more than 50 distinct ports within five minutes (Amazon GuardDuty also analyses flow logs independently for this kind of behaviour).

During the 02:10 incident above, the analyst used Athena to bound the exfiltration window (first ACCEPT to the bad IP at 01:52, 38 MB out), Logs Insights to see that the instance had also attempted, and been REJECTed by, the tokenisation service, and Reachability Analyzer to confirm no path existed from that subnet to the core banking VPN attachment. The incident report cited all three โ€” dated, reproducible evidence.

Common mistakes

  • No retention on the log group. CloudWatch Logs keeps data forever by default โ€” the bill grows every month. Always set retention (or lifecycle rules on S3).
  • REJECT-only logging for security. Most real breaches are accepted traffic. Capture ALL.
  • Expecting DNS lookups in flow logs. Queries to the Amazon resolver are not logged; use Resolver query logging.
  • Reading srcaddr behind a NAT gateway or Transit Gateway as the true source. On intermediate ENIs, srcaddr is the gateway's own address; add pkt-srcaddr.
  • Trying to edit a flow log's format. You can't; create a new one and delete the old.
  • Calling a single "not reachable" analysis proof. Always pair it with a positive control (see the lab).
  • Forgetting Reachability Analyzer paths and analyses. They are cheap but clutter; tag and delete them.

Troubleshooting

SymptomLikely causeFix
DeliverLogsStatus: FAILEDRole trust or permissions wrong; KMS key policy missing the serviceCheck the role trusts vpc-flow-logs.amazonaws.com and can PutLogEvents; add the logs service to the key policy
No log streams after 15 minutesNo traffic on any ENI, or flow log on the wrong VPCGenerate traffic; describe-flow-logs and check ResourceId
S3 delivery Access DeniedBucket policy does not allow delivery.logs.amazonaws.comAdd the documented bucket-policy statements (with aws:SourceAccount)
Logs Insights fields are emptyCustom format in CloudWatch LogsAuto-discovery covers the default format; use parse for custom formats โš  VERIFY
Many SKIPDATA recordsInternal capacity limits during the intervalExpected occasionally; if frequent, raise a support case โ€” and note the gap in your evidence
Reachability analysis failedUnsupported component or permissionsRead StatusMessage; check ec2:Describe* and tiros:* permissions โš  VERIFY action prefix
Path found but app still failsAnalyzer is layer 3/4 onlyCheck TLS, credentials, DNS, the application
Athena query scans terabytesNo partition filterFilter on partition columns (date/hour)

Cost and cleanup

Cost (us-east-1, checked 2026-11-18):

  • Flow logs to CloudWatch Logs: vended-log ingestion USD 0.50 per GB for the first 10 TB per month, then tiered down (0.25, 0.10, 0.05). Storage after ingestion is about USD 0.03 per GB-month โš  VERIFY. A lab VPC for an hour produces kilobytes โ€” effectively free.
  • Flow logs to S3: a lower vended-log rate (about USD 0.25 per GB for the first 10 TB โš  VERIFY) plus optional Parquet conversion per GB โš  VERIFY, plus S3 storage.
  • Logs Insights: USD 0.005 per GB scanned โš  VERIFY.
  • Reachability Analyzer: USD 0.10 per analysis. Network Access Analyzer: USD 0.002 per ENI analysed.

Bank-scale example: 200 VPCs producing 5 TB of flow records a month. To CloudWatch Logs: 5,000 GB ร— USD 0.50 = USD 2,500/month before storage. To S3: roughly half that. This is why banks send the bulk to S3 and only selected VPCs to CloudWatch Logs.

Free alternative: use the C# checker with the fixture files, and read the flow log record examples in the AWS docs โ€” no deployment needed.

Cleanup:

$p = '--profile', 'learn-sandbox', '--region', 'us-east-1'
aws ec2 describe-flow-logs @p --filter Name=tag:Lab,Values=m02-l14 --query "FlowLogs[].FlowLogId"
aws ec2 delete-flow-logs @p --flow-log-ids fl-0123456789abcdef0
aws logs delete-log-group @p --log-group-name /hl/m02/flow-logs
aws iam delete-role-policy --profile learn-sandbox --role-name hl-m02-flow-logs --policy-name write-flow-logs
aws iam delete-role --profile learn-sandbox --role-name hl-m02-flow-logs
aws ec2 describe-network-insights-paths @p --query "NetworkInsightsPaths[].NetworkInsightsPathId"
aws resourcegroupstaggingapi get-resources @p --tag-filters Key=Project,Values=cea-lab --query "ResourceTagMappingList[].ResourceARN"

Exercises

  • ๐ŸŸข Guided: In the secure VPC lab, run the three Logs Insights queries above. For each, write one sentence on what it shows.
  • ๐ŸŸก Challenge: Create a second flow log on the lab VPC with the custom format above, sent to S3 in Parquet with hourly partitions. After 20 minutes, generate the Athena table and write a query that shows, per hour, accepted versus rejected flows into the data tier.
  • ๐Ÿ”ด Live problem: The Accounts API team says: "Since yesterday's NACL change, about 2% of calls to the database time out, but the security group looks fine." Flow logs show ACCEPT for 10.18.70.x โ†’ 10.18.130.12:5432 and some REJECT records for 10.18.130.12:5432 โ†’ 10.18.70.x with destination ports between 61000 and 65535. Diagnose and propose the fix.

Assignment

Write a one-page flow-log standard for Harbourline: scope, traffic type, interval, format (with the field list), destination(s), retention, encryption, who can read the logs, and the three detections you would build first. Add the cost estimate for 50 VPCs at 20 GB/month each. Commit it to your portfolio repository.

Solutions

Knowledge check

Interview questions

  • "How do you tell from flow logs whether a security group or a NACL blocked traffic?" โ€” Look for the stateless signature: accepted request plus rejected reply means NACL. A single reject could be either; confirm with Reachability Analyzer's explanation codes.
  • "Where would you store flow logs in a multi-account bank, and why?" โ€” S3 in a central log-archive account, Parquet with hourly partitions, Object Lock, KMS, lifecycle rules; workload accounts cannot delete evidence. CloudWatch Logs only where live alarms justify the cost.
  • "What don't flow logs show?" โ€” Payload, DNS queries to the Amazon resolver, metadata and Time Sync traffic, and which control rejected a flow. They're also delayed by minutes.
  • "How would you continuously prove no database is internet-reachable?" โ€” A Network Access Analyzer scope (internet gateway โ†’ ENIs tagged as data tier) run on a schedule, plus a Config rule for public RDS, plus flow-log detections.

Certification practice

Completion checklist

  • I can read every field of a default flow-log record
  • I can spot the NACL "accept request, reject reply" signature
  • I ran at least two Logs Insights queries on the lab VPC
  • I ran a Reachability Analyzer analysis and read its explanation codes
  • I deleted my extra flow logs, log groups, roles and analyzer paths

Key takeaways

  • Flow logs are a door-access log for every ENI: who, where, when, how much, allowed or not โ€” but not what was said.
  • Capture ALL, log at VPC scope, and send the bulk to S3 in a central log-archive account; set retention everywhere.
  • Stateless NACLs leave a recognisable "accept request, reject reply" pattern.
  • Reachability Analyzer proves one path from configuration; Network Access Analyzer proves a rule across the account.
  • Evidence beats opinion: pair configuration analysis with recorded traffic and a positive control.

Further reading

Next steps: filter what is allowed to leave the VPC in the first place with AWS Network Firewall, or put the evidence tools to work in the secure VPC lab.

Independent learning resource โ€” not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page