Marko Cupa??(marko.cu...@mimar.rs) on 2014.05.31 13:03:18 +0200:
> Hi,
>
> I'm trying to understand and measure traffic on relatively large and
> complicated pf firewall, and for this purpose I am exporting netflow
> data with pflow to nfsen/nfdump.
>
> For the time being, I have set pflow on external interface in outbound
> direction:
>
> pass out on $if_ext inet all keep state (pflow)
>
> On collector (nfsen), I want to see interface numbers so i can create
> interface filter:
>
> % nfdump -R 2014 -s if/bytes
> Top 10 In/Out If ordered by bytes:
> If Flows(%) Packets(%) Bytes(%) pps bps bpp
> 5 19396(100.0) 300683(100.0) 186.7 M(100.0) 3 16984 620
> 7 19109(98.5) 299769(99.7) 186.6 M(100.0) 3 16976 622
> 0 287( 1.5) 914( 0.3) 83170( 0.0) 0 330 90
>
> Another mailing list member told me I can find about interface numbers
> with snmpwalk:
>
> % snmpwalk -v2c -c community -On IP.ADD.RE.SS
> .1.3.6.1.2.1.2.2.1.2.5 = STRING: bnx1
> .1.3.6.1.2.1.2.2.1.2.7 = STRING: carp2
>
> Ok, now I know interface 5 is bnx1 ($if_ext), and I want to know what
> comes in:
>
> % nfdump -R 2014 -s dstip/bytes 'in if 5'
> Top 10 Dst IP Addr ordered by bytes:
> Dst IP Addr Flows(%) Packets(%) Bytes(%)
> 10.20.0.15 10754(62.9) 323834(52.9) 324.9 M(63.7)
> 10.20.4.99 462( 2.7) 10496( 1.7) 9.4 M( 1.8)
> 178.148.77.73 4( 0.0) 6681( 1.1) 7.7 M( 1.5)
>
> First two addresses really are on my internal network, and I know first
> one is return web traffic to my proxy, and the second one return web
> traffic to another internal host.
>
> But the last address is not on my network. Let's see records for this
> address:
>
> nfdump -R 2014 -n 100000 -s record/bytes 'in if 5' | grep 178.148.77.73
> TCP 193.53.106.35:443 -> 178.148.77.73:49193 5606 7.6 M
> TCP 193.53.106.35:443 -> 178.148.77.73:49191 313 95342
> TCP 193.53.106.35:443 -> 178.148.77.73:49192 404 18674
> TCP 193.53.106.35:443 -> 178.148.77.73:49190 358 16798
>
> Ok, these are redirected incoming requests to HTTPS server on my
> internal network:
>
> pass in on $if_ext inet proto tcp from any to $pub_web port { 80 443 } \
> rdr-to $priv_web keep state
>
> But source and destination IP adresses are reversed!
>
> Here's what pf's state table shows:
> $ sudo pfctl -ss | grep 178.148.77.73
> all tcp 10.20.0.36:443 (193.53.106.35:443) <- 178.148.77.73:49377
> all tcp 178.148.77.73:49377 -> 10.20.0.36:443
> all tcp 10.20.0.36:443 (193.53.106.35:443) <- 178.148.77.73:49378
> all tcp 178.148.77.73:49378 -> 10.20.0.36:443
> all tcp 10.20.0.36:443 (193.53.106.35:443) <- 178.148.77.73:49379
> all tcp 178.148.77.73:49379 -> 10.20.0.36:443
> all tcp 10.20.0.36:443 (193.53.106.35:443) <- 178.148.77.73:49380
> all tcp 178.148.77.73:49380 -> 10.20.0.36:443
>
> How could this be corrected? Am I configuring pf incorrectly? Or is
> there a problem with how pflow exports data? Or is pfdump parsing the
> data incorrectly?
The simple answer: It's complicated.
The complicated answer: the pf state is used to keep track of both
directions of the traffic flow. When the state times out, _two_ flows are
created, one for each direction of traffic, you can see this in
copy_flow_ipfix_4_data() in /usr/src/sys/net/if_pflow.c.
For NAT/RDR its a bit more complicated, so what you are seeing might be
'normal' or a problem.
nfdump should be able to show you both directions of this traffic. Please
check what in and out interface is recorded for each flow, ie grep for
178.148.77.73 but dont restrict on the interface.
Also, please show a dmesg - we need to know what version you are running.
/Benno
