i have a site-to-site vpn setup across a 40Mbps wan link (average ~30ms
latency). one of its uses is for san replication, but of course management
traffic (ssh sessions, etc.) have to cross the link as well. without using
queues, at times the replication traffic is such that management traffic
suffers to the point of being unusable.
so i setup queues, which "fixed" the management problem. but despite the
management bandwidth requirements being minimal, the san replication
traffic was then seen to plateau well below where i believe it should have
been.
one specific thing i'm seeing with this particular configuration is that
san replication traffic tops out at 24Mbps, as seen on the wan circuit
itself (outside of openbsd). removing the queues results in 100% wan
utilization, even up to 100Mbps when the circuit is temporarily
reconfigured to allow it.
i have to assume that i've misunderstood the documentation and am looking
for some help. i'll paste the pf.conf below, followed by dmesg. we have a
fairly complex network on each end of the vpn, and we do in fact need the
nat that you will see though not for reasons related to the san replication
traffic. i have no doubt that i've done things incorrectly even in areas
seemingly unrelated to san replication, so feel free to fire away ...
pf.conf:
=======
##########################
# macros
LANIF = em0
WANIF = em1
PFSYNC = em2
INETIF = vlan2
TWP2PIF = vlan3
table <vpnendpoints> persist { $PUBLIC1 $PUBLIC2 $REMOTEPUB3 $REMOTEPUB4
172.30.255.240/28 }
table <carpmembers> persist { $PUBLIC1 $PUBLIC2 172.28.0.251
172.28.0.252 }
table <trustednets> persist { 10.200.0.0/16 192.168.0.0/16 172.16.0.0/12
}
table <recoverpoint> persist { 10.200.80.0/24 172.28.0.247 172.28.0.248
10.200.72.0/24 172.28.0.10 172.28.0.11 172.28.0.12 172.28.2.0/24 }
##########################
# queues
altq on $INETIF cbq bandwidth 20Mb queue { ssh, sansync, std }
altq on $TWP2PIF cbq bandwidth 35Mb queue { ssh, sansync, std }
queue sansync bandwidth 35% priority 1 cbq(borrow ecn)
queue std bandwidth 50% priority 2 cbq(default borrow)
queue ssh bandwidth 15% priority 3 cbq(borrow ecn)
##########################
# options
set skip on lo
set skip on enc0
set skip on gif
set skip on $PFSYNC
set block-policy return
set loginterface $WANIF
##########################
# ftp proxy
anchor "ftp-proxy/*"
pass in quick on $LANIF inet proto tcp to any port ftp \
divert-to 127.0.0.1 port 8021
##########################
# match rules
match in from <trustednets> scrub (no-df random-id max-mss 1200)
##########################
# filter rules
block in log
pass out
pass out proto tcp all modulate state
# site-to-site vpn
pass in quick log proto esp from <vpnendpoints>
pass in quick log proto udp from <vpnendpoints> port isakmp
antispoof quick for { lo $LANIF }
pass in quick proto carp from any to any
pass in quick inet proto icmp from any to any icmp-type { echoreq echorep
timex unreach }
pass in quick on $LANIF to <recoverpoint> queue sansync label sansync
pass in quick on $LANIF proto tcp to port { ssh } queue ssh label ssh
pass in quick on $LANIF proto tcp to port { 3389 } queue ssh label rdp
pass in log on $LANIF queue std label std
dmesg:
=====
OpenBSD 5.4 (GENERIC.MP) #41: Tue Jul 30 15:30:02 MDT 2013
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 8403169280 (8013MB)
avail mem = 8171749376 (7793MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xb7fcb000 (82 entries)
bios0: vendor HP version "P80" date 11/08/2013
bios0: HP ProLiant DL320e Gen8 v2
acpi0 at bios0: rev 2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP SPCR MCFG HPET FFFF SPMI ERST APIC FFFF BERT HEST
DMAR FFFF SSDT SSDT SSDT SSDT SSDT
acpi0: wakeup devices PCI0(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimcfg0 at acpi0 addr 0xb8000000, bus 0-63
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz, 3492.44 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX
,SMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT
,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1
,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
cpu0: apic clock running at 99MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz, 3491.92 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX
,SMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT
,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1
,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz, 3491.92 MHz
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX
,SMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT
,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1
,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz, 3491.92 MHz
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX
,SMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT
,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1
,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 1 (application processor)
cpu4: Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz, 3491.92 MHz
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX
,SMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT
,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1
,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 1, core 0, package 0
cpu5 at mainbus0: apid 3 (application processor)
cpu5: Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz, 3491.92 MHz
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX
,SMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT
,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1
,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: smt 1, core 1, package 0
cpu6 at mainbus0: apid 5 (application processor)
cpu6: Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz, 3491.92 MHz
cpu6:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX
,SMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT
,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1
,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM
cpu6: 256KB 64b/line 8-way L2 cache
cpu6: smt 1, core 2, package 0
cpu7 at mainbus0: apid 7 (application processor)
cpu7: Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz, 3491.92 MHz
cpu7:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX
,SMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT
,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1
,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM
cpu7: 256KB 64b/line 8-way L2 cache
cpu7: smt 1, core 3, package 0
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 8
acpiprt0 at acpi0: bus 10 (IPT1)
acpiprt1 at acpi0: bus -1 (IPT2)
acpiprt2 at acpi0: bus -1 (IPT3)
acpiprt3 at acpi0: bus -1 (IPT4)
acpiprt4 at acpi0: bus -1 (IPT5)
acpiprt5 at acpi0: bus 3 (IPT6)
acpiprt6 at acpi0: bus -1 (IPT7)
acpiprt7 at acpi0: bus 1 (IPT8)
acpiprt8 at acpi0: bus 4 (PT02)
acpiprt9 at acpi0: bus 7 (PT03)
acpiprt10 at acpi0: bus 2 (PT04)
acpiprt11 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C2, C1
acpicpu1 at acpi0: C2, C1
acpicpu2 at acpi0: C2, C1
acpicpu3 at acpi0: C2, C1
acpicpu4 at acpi0: C2, C1
acpicpu5 at acpi0: C2, C1
acpicpu6 at acpi0: C2, C1
acpitz0 at acpi0: critical temperature is 31 degC
ipmi at mainbus0 not configured
cpu0: unknown i686 model 0x3c, can't get bus clock
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 vendor "Intel", unknown product 0x0c08 rev
0x06
ppb0 at pci0 dev 1 function 0 vendor "Intel", unknown product 0x0c01 rev
0x06: msi
pci1 at ppb0 bus 4
ciss0 at pci1 dev 0 function 0 "Hewlett-Packard Smart Array" rev 0x01: apic
8 int 16
ciss0: 1 LD, HW rev 1, FW 5.22/5.22, 64bit fifo rro
scsibus0 at ciss0: 1 targets
sd0 at scsibus0 targ 0 lun 0: <HP, LOGICAL VOLUME, 5.22> SCSI3 0/direct
fixed
sd0: 286070MB, 512 bytes/sector, 585871964 sectors
ppb1 at pci0 dev 1 function 1 vendor "Intel", unknown product 0x0c05 rev
0x06: msi
pci2 at ppb1 bus 7
em0 at pci2 dev 0 function 0 "Intel 82580" rev 0x01: msi, address
ac:16:2d:9d:40:48
em1 at pci2 dev 0 function 1 "Intel 82580" rev 0x01: msi, address
ac:16:2d:9d:40:49
em2 at pci2 dev 0 function 2 "Intel 82580" rev 0x01: msi, address
ac:16:2d:9d:40:4a
em3 at pci2 dev 0 function 3 "Intel 82580" rev 0x01: msi, address
ac:16:2d:9d:40:4b
ppb2 at pci0 dev 1 function 2 vendor "Intel", unknown product 0x0c09 rev
0x06: msi
pci3 at ppb2 bus 2
unknown vendor 0x1590 product 0x005f (class memory subclass miscellaneous,
rev 0x00) at pci3 dev 0 function 0 not configured
vendor "Intel", unknown product 0x8c31 (class serial bus subclass USB, rev
0x04) at pci0 dev 20 function 0 not configured
ehci0 at pci0 dev 26 function 0 vendor "Intel", unknown product 0x8c2d rev
0x04: apic 8 int 21
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb3 at pci0 dev 28 function 0 vendor "Intel", unknown product 0x8c10 rev
0xd4
pci4 at ppb3 bus 10
ppb4 at pci0 dev 28 function 5 vendor "Intel", unknown product 0x8c1a rev
0xd4
pci5 at ppb4 bus 3
bge0 at pci5 dev 0 function 0 "Broadcom BCM5720" rev 0x00, BCM5720 A0
(0x5720000), APE firmware NCSI 1.1.15.0: msi, address 9c:b6:54:bb:91:8c
brgphy0 at bge0 phy 1: BCM5720C 10/100/1000baseT PHY, rev. 0
bge1 at pci5 dev 0 function 1 "Broadcom BCM5720" rev 0x00, BCM5720 A0
(0x5720000), APE firmware NCSI 1.1.15.0: msi, address 9c:b6:54:bb:91:8d
brgphy1 at bge1 phy 2: BCM5720C 10/100/1000baseT PHY, rev. 0
ppb5 at pci0 dev 28 function 7 vendor "Intel", unknown product 0x8c1e rev
0xd4
pci6 at ppb5 bus 1
"Hewlett-Packard iLO3 Slave" rev 0x05 at pci6 dev 0 function 0 not
configured
vga1 at pci6 dev 0 function 1 vendor "Matrox", unknown product 0x0533 rev
0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
"Hewlett-Packard iLO3 Management" rev 0x05 at pci6 dev 0 function 2 not
configured
uhci0 at pci6 dev 0 function 4 "Hewlett-Packard USB" rev 0x02: apic 8 int 16
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Hewlett-Packard UHCI root hub" rev 1.00/1.00 addr 1
ehci1 at pci0 dev 29 function 0 vendor "Intel", unknown product 0x8c26 rev
0x04: apic 8 int 20
usb2 at ehci1: USB revision 2.0
uhub2 at usb2 "Intel EHCI root hub" rev 2.00/1.00 addr 1
pcib0 at pci0 dev 31 function 0 vendor "Intel", unknown product 0x8c52 rev
0x04
vendor "Intel", unknown product 0x8c04 (class mass storage subclass RAID,
rev 0x04) at pci0 dev 31 function 2 not configured
isa0 at pcib0
isadma0 at isa0
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
com1: probed fifo depth: 0 bytes
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
mtrr: Pentium Pro MTRR support
uhub3 at uhub0 port 1 "Intel product 0x8008" rev 2.00/0.04 addr 2
umass0 at uhub3 port 3 configuration 1 interface 0 "HP 8GB_EM_USB_Drive"
rev 2.00/1.00 addr 3
umass0: using SCSI over Bulk-Only
scsibus1 at umass0: 2 targets, initiator 0
sd1 at scsibus1 targ 1 lun 0: <HP, 8GB_EM_USB_Drive, PMAP> SCSI2 0/direct
removable serial.13fe41003C16C4DBC903
sd1: 7644MB, 512 bytes/sector, 15654912 sectors
uhub4 at uhub2 port 1 "Intel product 0x8000" rev 2.00/0.04 addr 2
uhub5 at uhub4 port 3 "Standard Microsystems product 0x2660" rev 2.00/8.01
addr 3
uhidev0 at uhub1 port 1 configuration 1 interface 0 "HP Virtual Keyboard"
rev 1.10/0.02 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes, country code 33
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub1 port 1 configuration 1 interface 1 "HP Virtual Keyboard"
rev 1.10/0.02 addr 2
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (1ae5d0692c17e9ae.a) swap on sd0b dump on sd0b
