On Sat, 31 May 2014 20:01:25 +0200
Sebastian Benoit <benoit-li...@fb12.de> wrote:
> The simple answer: It's complicated.
>
> The complicated answer: the pf state is used to keep track of both
> directions of the traffic flow. When the state times out, _two_ flows
> are created, one for each direction of traffic, you can see this in
> copy_flow_ipfix_4_data() in /usr/src/sys/net/if_pflow.c.
>
> For NAT/RDR its a bit more complicated, so what you are seeing might
> be 'normal' or a problem.
>
> nfdump should be able to show you both directions of this traffic.
> Please check what in and out interface is recorded for each flow, ie
> grep for 178.148.77.73 but dont restrict on the interface.
>
> Also, please show a dmesg - we need to know what version you are
> running.
>
> /Benno
>
I have enabled pflow for outbound traffic on $int_if and $ext_if first,
and it appears that in this setup no redirected traffic is recorded by
nfdump, either entering $ext_if and leaving $int_if on arrival, or
entering $int_if and leaving $ext_if on return. Other kinds of traffic
appear to be recorded correctly by pflow, including NAT traffic.
Next, I enabled pflow for one additional inbound redirected rule:
pass in on $if_ext inet proto tcp from any to $pub_srv port 1002 \
rdr-to $priv_srv keep state (pflow)
In this setup flows appear to be recorded by nfdump fine on $int_if,
both leaving it on arrival and entering it on return. Direction is
correct.
% nfdump -R 2014 -s srcip/bytes 'out if 5 and port 1002'
Src IP Addr Flows(%) Packets(%) Bytes(%)
212.200.65.243 3678(34.9) 24554(36.0) 2.1 M(35.2)
212.200.65.244 2393(22.7) 15331(22.5) 1.4 M(23.3)
212.200.65.241 2457(23.3) 15488(22.7) 1.3 M(22.5)
212.200.65.242 2025(19.2) 12765(18.7) 1.1 M(19.0)
% nfdump -R 2014 -s dstip/bytes 'in if 5 and port 1002'
Dst IP Addr Flows(%) Packets(%) Bytes(%)
212.200.65.243 3678(34.9) 20699(34.9) 1.0 M(36.3)
212.200.65.241 2457(23.3) 13572(22.9) 638520(22.5)
212.200.65.244 2393(22.7) 13590(22.9) 619420(21.9)
212.200.65.242 2025(19.2) 11496(19.4) 547616(19.3)
However, on external interface the direction appears to be reversed
(notice I need to request '$ext_if outbound srcip' in order to get
'$ext_if outbound dstip':
% nfdump -R 2014 -s srcip/bytes 'out if 4 and port 1002'
Src IP Addr Flows(%) Packets(%) Bytes(%)
212.200.65.243 4051(35.0) 26862(36.4) 2.3 M(35.7)
212.200.65.244 2654(23.0) 16771(22.7) 1.5 M(23.4)
212.200.65.241 2683(23.2) 16731(22.7) 1.4 M(22.4)
212.200.65.242 2175(18.8) 13475(18.2) 1.2 M(18.5)
Also I need to request '$ext_if inbound dstip' in order to get '$ext_if
inbound srcip':
% nfdump -R 2014 -s dstip/bytes 'in if 4 and port 1002'
Dst IP Addr Flows(%) Packets(%) Bytes(%)
212.200.65.243 4051(35.0) 22767(35.0) 1.1 M(36.5)
212.200.65.241 2683(23.2) 14756(22.7) 692652(22.4)
212.200.65.244 2654(23.0) 15024(23.1) 683824(22.1)
212.200.65.242 2175(18.8) 12409(19.1) 586820(19.0)
I am using quite recent snapshot:
OpenBSD 5.5-current (GENERIC.MP) #150: Mon May 26 11:50:31 MDT 2014
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 2128887808 (2030MB)
avail mem = 2063499264 (1967MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xee000 (69 entries)
bios0: vendor HP version "P58" date 05/02/2011
bios0: HP ProLiant DL360 G5
acpi0 at bios0: rev 2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP SPCR MCFG HPET SPMI ERST APIC FFFF BERT HEST
SSDT acpi0: wakeup devices PCI0(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2500.38 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,LONG,LAHF,PERF
cpu0: 6MB 64b/line 16-way L2 cache cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 333MHz
cpu0: mwait min=64, max=64, C-substates=0.2.2.2.0, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2000.08 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,LONG,LAHF,PERF
cpu1: 6MB 64b/line 16-way L2 cache cpu1: smt 0, core 2, package 0
cpu2 at mainbus0: apid 1 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2500.09 MHz
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,LONG,LAHF,PERF
cpu2: 6MB 64b/line 16-way L2 cache cpu2: smt 0, core 1, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2500.09 MHz
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,LONG,LAHF,PERF
cpu3: 6MB 64b/line 16-way L2 cache cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0: apid 9 pa 0xfec80000, version 20, 24 pins
acpiprt0 at acpi0: bus 1 (IP2P)
acpiprt1 at acpi0: bus 11 (IPE1)
acpiprt2 at acpi0: bus 10 (IPE4)
acpiprt3 at acpi0: bus 16 (P2P2)
acpiprt4 at acpi0: bus 9 (PT02)
acpiprt5 at acpi0: bus 6 (PT03)
acpiprt6 at acpi0: bus 19 (PT04)
acpiprt7 at acpi0: bus 3 (NB01)
acpiprt8 at acpi0: bus 5 (NB02)
acpiprt9 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C3, C1
acpicpu1 at acpi0: C3, C1
acpicpu2 at acpi0: C3, C1
acpicpu3 at acpi0: C3, C1
acpitz0 at acpi0: critical temperature is 31 degC
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 5000P Host" rev 0xb1
ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE" rev 0xb1
pci1 at ppb0 bus 9
ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci2 at ppb1 bus 10
ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci3 at ppb2 bus 11
ppb3 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01
pci4 at ppb3 bus 14
ppb4 at pci2 dev 2 function 0 "Intel 6321ESB PCIE" rev 0x01
pci5 at ppb4 bus 15
ppb5 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01
pci6 at ppb5 bus 16
ppb6 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0xb1
pci7 at ppb6 bus 6
ciss0 at pci7 dev 0 function 0 "Hewlett-Packard Smart Array" rev 0x04:
apic 8 int 16 ciss0: 1 LD, HW rev 4, FW 7.24/7.24, 64bit fifo
scsibus1 at ciss0: 1 targets
sd0 at scsibus1 targ 0 lun 0: <HP, LOGICAL VOLUME, 7.24> SCSI3 0/direct
fixed sd0: 139979MB, 512 bytes/sector, 286677120 sectors
ppb7 at pci0 dev 4 function 0 "Intel 5000 PCIE x8" rev 0xb1
pci8 at ppb7 bus 19
em0 at pci8 dev 0 function 0 "Intel 82574L" rev 0x00: msi, address
2c:27:d7:15:21:95 ppb8 at pci0 dev 5 function 0 "Intel 5000 PCIE" rev
0xb1 pci9 at ppb8 bus 22
ppb9 at pci0 dev 6 function 0 "Intel 5000 PCIE" rev 0xb1
pci10 at ppb9 bus 2
ppb10 at pci10 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
pci11 at ppb10 bus 3
bnx0 at pci11 dev 0 function 0 "Broadcom BCM5708" rev 0x12: apic 8 int
18 ppb11 at pci0 dev 7 function 0 "Intel 5000 PCIE" rev 0xb1
pci12 at ppb11 bus 4
ppb12 at pci12 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
pci13 at ppb12 bus 5
bnx1 at pci13 dev 0 function 0 "Broadcom BCM5708" rev 0x12: apic 8 int
19 pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0xb1
pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0xb1
pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0xb1
pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0xb1
pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0xb1
pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0xb1
pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0xb1
uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: apic 8
int 16 uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09:
apic 8 int 17 uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev
0x09: apic 8 int 18 uhci3 at pci0 dev 29 function 3 "Intel 6321ESB USB"
rev 0x09: apic 8 int 19 ehci0 at pci0 dev 29 function 7 "Intel 6321ESB
USB" rev 0x09: apic 8 int 16 usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb13 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xd9
pci14 at ppb13 bus 1
radeondrm0 at pci14 dev 3 function 0 "ATI ES1000" rev 0x02
drm0 at radeondrm0
radeondrm0: apic 8 int 23
"Compaq iLO" rev 0x03 at pci14 dev 4 function 0 not configured
"Compaq iLO" rev 0x03 at pci14 dev 4 function 2 not configured
uhci4 at pci14 dev 4 function 4 "Hewlett-Packard USB" rev 0x00: apic 8
int 22 "Hewlett-Packard IPMI" rev 0x00 at pci14 dev 4 function 6 not
configured usb1 at uhci4: USB revision 1.0
uhub1 at usb1 "Hewlett-Packard UHCI root hub" rev 1.00/1.00 addr 1
pcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09
pciide0 at pci0 dev 31 function 1 "Intel 6321ESB IDE" rev 0x09: DMA,
channel 0 configured to compatibility, channel 1 configured to
compatibility atapiscsi0 at pciide0 channel 0 drive 0 scsibus2 at
atapiscsi0: 2 targets cd0 at scsibus2 targ 0 lun 0: <HL-DT-ST, DVDRAM
GSA-T40L, KS03> ATAPI 5/cdrom removable cd0(pciide0:0:0): using PIO
mode 4, Ultra-DMA mode 2 pciide0: channel 1 disabled (no drives)
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
com1: probed fifo depth: 0 bytes
pckbc0 at isa0 port 0x60/5
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
uhidev0 at uhub1 port 1 configuration 1 interface 0 "HP Virtual
Keyboard" rev 1.10/0.02 addr 2 uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes, country code 33
wskbd1 at ukbd0 mux 1
uhidev1 at uhub1 port 1 configuration 1 interface 1 "HP Virtual
Keyboard" rev 1.10/0.02 addr 2 uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons
wsmouse1 at ums0 mux 0
uhidev2 at uhub4 port 1 configuration 1 interface 0 "NOVATEK USB
Keyboard" rev 1.10/1.04 addr 2 uhidev2: iclass 3/1
ukbd1 at uhidev2: 8 variable keys, 6 key codes
wskbd2 at ukbd1 mux 1
uhidev3 at uhub4 port 1 configuration 1 interface 1 "NOVATEK USB
Keyboard" rev 1.10/1.04 addr 2 uhidev3: iclass 3/0, 2 report ids
uhid0 at uhidev3 reportid 1: input=1, output=0, feature=0
uhid1 at uhidev3 reportid 2: input=3, output=0, feature=0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (ac6a2b6d6cc53aac.a) swap on sd0b dump on sd0b
bnx0: address 00:22:64:a1:dd:e8
brgphy0 at bnx0 phy 1: BCM5708C 10/100/1000baseT PHY, rev. 6
bnx1: address 00:22:64:a1:dd:e6
brgphy1 at bnx1 phy 1: BCM5708C 10/100/1000baseT PHY, rev. 6
drm: initializing kernel modesetting (RV100 0x1002:0x515E
0x103C:0x31FB). radeondrm0: VRAM: 128M 0x00000000D8000000 -
0x00000000DFFFFFFF (32M used) radeondrm0: GTT: 512M 0x00000000B8000000
- 0x00000000D7FFFFFF drm: PCI GART of 512M enabled (table at
0x00000000056A1000). drm: No TV DAC info found in BIOS
radeondrm0: 1024x768
wsdisplay0 at radeondrm0 mux 1: console (std, vt100 emulation), using
wskbd0 wskbd1: connecting to wsdisplay0
wskbd2: connecting to wsdisplay0
wsdisplay0: screen 1-5 added (std, vt100 emulation)
carp1: state transition: BACKUP -> MASTER
carp2: state transition: BACKUP -> MASTER
Regards,
--
Marko Cupać
