hello,
Last week, I upgraded a couple of firewalls using carp/pfsync and sasyncd
from 6.0 to 6.7 (yes, big jump !).
I also applied all the 6.7 published patches.
When some heavy traffic takes one of the IPSec tunnel, I noticed that :
- all network connections are slowed down
- unused network bandwidth increase instead of decrease
- idle CPU move towards 0, and spinning increase to take about 50% of the
CPU
When I stop the IPSec traffic :
- network connections increase immediatly
- unused network bandwidth cecreases immediately
- spinning CPU is low.
Yes I know, my hardware is a bit old. I understand that CPU raises due to
IPSec crypto, but I do not understand why network performance decrease.
1) Situation before doing anything:
# pktstat -ntT -m 100000000 -i em1
interface: em1 total: 122.6Mb (7m18s)
cur: 260.1k (0%) min: 0.0 max: 100.0M avg: 279.3k bps
bps % b desc
69.6k 0% 348.6k tcp 109.7.96.229:54880 <-> 52.113.194.132:443
60.0k 0% 36.1M ip proto 50 109.7.96.226 <-> 92.174.146.73
36.5k 0% 182.8k tcp 109.7.96.229:59950 <-> 52.113.194.132:443
12.3k 0% 61.5k tcp 109.7.96.229:51009 <-> 216.58.214.78:443
11.8k 0% 58.9k tcp 109.7.96.229:61287 <-> 216.58.206.229:443
# top
load averages: 0.14, 0.12, 0.14 xxxx.xxxx.fr
20:00:05
81 processes: 2 running, 77 idle, 2 on processor up
10:53
CPU0: 31.9% user, 0.0% nice, 21.4% sys, 5.8% spin, 0.4% intr, 40.5% idle
CPU1: 30.9% user, 0.0% nice, 17.2% sys, 5.2% spin, 0.0% intr, 46.7% idle
Memory: Real: 166M/403M act/tot Free: 561M Cache: 128M Swap: 0K/0K
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
35828 osadmin 52 0 1676K 3504K run/0 - 0:03 8.35% sshd
68723 _openvpn 2 0 4016K 6404K sleep/1 poll 11:41 1.12% openvpn
16143 root 2 0 1372K 4056K sleep/0 poll 0:00 0.49% sshd
95804 root 28 0 5440K 6892K run/0 - 0:05 0.34% pktstat
2) Making heavy traffic NOT using IPSec :
Notice bandwidth usage.
heavy traffic NOT using the IPSec tunnel
# ssh ardee dd if=/dev/urandom bs=1M | dd of=/dev/null bs=1M
0+12031 records in
0+12031 records out
198180864 bytes (198 MB, 189 MiB) copied, 23.3799 s, 8.5 MB/s
0+19257 records in
0+19257 records out
316571648 bytes (317 MB, 302 MiB) copied, 37.167 s, 8.5 MB/s
# pktstat -ntT -m 100000000 -i em1
interface: em1 total: 8.2Gb (11m49s)
cur: 72.6M (72%) min: 0.0 max: 100.0M avg: 11.5M bps
bps % b desc
72.4M 72% 8.0G tcp 109.7.96.226:63663 <-> 212.83.131.76:22222
66.4k 0% 60.2M ip proto 50 109.7.96.226 <-> 92.174.146.73
33.5k 0% 167.7k tcp 109.7.96.229:52670 <-> 52.97.168.210:443
10.3k 0% 7.5M ip proto 112 109.7.96.227 <-> 224.0.0.18
9.2k 0% 46.3k tcp 109.7.96.229:56973 <-> 40.101.92.178:443
# top
load averages: 1.11, 0.61, 0.34 billy.basystemes.fr
20:04:41
76 processes: 75 idle, 1 on processor up
10:58
CPU0: 13.8% user, 0.0% nice, 18.6% sys, 1.2% spin, 11.2% intr, 55.3% idle
CPU1: 10.2% user, 0.0% nice, 29.3% sys, 0.6% spin, 0.0% intr, 59.9% idle
Memory: Real: 166M/390M act/tot Free: 574M Cache: 115M Swap: 0K/0K
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
95804 root 2 0 9760K 8696K sleep/1 poll 0:36 15.77% pktstat
68723 _openvpn 2 0 4012K 6332K sleep/1 poll 11:46 1.17% openvpn
33560 _isakmpd 2 0 11M 15M sleep/0 select 7:28 0.59% isakmpd
83650 _openvpn 2 0 3928K 6388K sleep/0 poll 20:10 0.00% openvpn
3) Making heavy traffic using the IPSec tunnel in addition to the previous
heavy traffic :
Notice bandwidth usage, which has decreased, and spinning value in top.
Also notice the weak rate tranfer in the IPSec tunnel.
heavy traffic NOT using the IPSec tunnel
# ssh ardee dd if=/dev/urandom bs=1M | dd of=/dev/null bs=1M
0+11902 records in
0+11902 records out
231751680 bytes (232 MB, 221 MiB) copied, 109.809 s, 2.1 MB/s
0+12372 records in
0+12372 records out
247152640 bytes (247 MB, 236 MiB) copied, 131.151 s, 1.9 MB/s
heavy traffic using the IPSec tunnel
# ssh doon dd if=/dev/urandom bs=1M | dd of=/tmp/null bs=1M
0+2496 records in
0+2496 records out
81723392 bytes (82 MB, 78 MiB) copied, 91.6991 s, 891 kB/s
0+3078 records in
0+3078 records out
100794368 bytes (101 MB, 96 MiB) copied, 113.042 s, 892 kB/s
# pktstat -ntT -m 100000000 -i em1
interface: em1 total: 15.3Gb (13m44s)
cur: 11.1M (11%) min: 0.0 max: 100.0M avg: 18.5M bps
bps % b desc
6.2M 6% 163.3M ip proto 50 109.7.96.226 <-> 92.174.146.73
4.7M 4% 1.2G tcp 109.7.96.226:52734 <-> 212.83.131.76:22222
33.7k 0% 474.5k ip fragments
25.8k 0% 2.5M udp 109.7.96.228:1195 <-> 92.135.30.8:52978
18.2k 0% 9.8M udp 109.7.96.228:1195 <-> 91.166.166.68:17587
17.6k 0% 88.3k tcp 109.7.96.229:443 <-> 213.32.72.115:47700
# top
load averages: 2.59, 1.39, 0.70 billy.basystemes.fr
20:08:22
79 processes: 78 idle, 1 on processor up
11:01
CPU0: 7.2% user, 0.0% nice, 50.6% sys, 21.1% spin, 2.4% intr, 18.7% idle
CPU1: 8.2% user, 0.0% nice, 55.5% sys, 18.4% spin, 0.0% intr, 18.0% idle
Memory: Real: 173M/402M act/tot Free: 563M Cache: 115M Swap: 0K/0K
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
95804 root 2 0 14M 17M sleep/1 poll 1:22 21.34% pktstat
68723 _openvpn 2 0 4000K 6364K sleep/1 poll 11:52 2.98% openvpn
83650 _openvpn 2 0 3928K 6388K sleep/1 poll 20:17 2.83% openvpn
33560 _isakmpd 2 0 11M 15M sleep/1 select 7:32 0.88% isakmpd
4) After stopping heavy traffic using the IPSec tunnel :
Notice that bandwidth usage raises.
heavy traffic not using the IPSec tunnel
# ssh ardee dd if=/dev/urandom bs=1M | dd of=/dev/null bs=1M
0+66256 records in
0+66256 records out
1086308352 bytes (1.1 GB, 1.0 GiB) copied, 127.389 s, 8.5 MB/s
0+66705 records in
0+66705 records out
1093664768 bytes (1.1 GB, 1.0 GiB) copied, 128.265 s, 8.5 MB/s
# pktstat -ntT -m 100000000 -i em1
interface: em1 total: 28.6Gb (20m10s)
cur: 70.3M (70%) min: 0.0 max: 100.0M avg: 23.6M bps
bps % b desc
70.2M 70% 10.8G tcp 109.7.96.226:63823 <-> 212.83.131.76:22222
46.4k 0% 1.5G ip proto 50 109.7.96.226 <-> 92.174.146.73
9.6k 0% 48.0k tcp 109.7.96.229:55137 <-> 216.58.215.42:443
9.2k 0% 45.9k tcp 109.7.96.229:65011 <-> 52.97.173.2:443
9.1k 0% 151.2k tcp 109.7.96.229:59164 <-> 40.101.93.226:443
# top
load averages: 1.28, 1.45, 0.94 billy.basystemes.fr
20:12:51
77 processes: 2 running, 74 idle, 1 on processor up
11:06
CPU0: 8.8% user, 0.0% nice, 18.4% sys, 1.2% spin, 10.4% intr, 61.3% idle
CPU1: 9.4% user, 0.0% nice, 29.7% sys, 0.2% spin, 0.0% intr, 60.7% idle
Memory: Real: 173M/403M act/tot Free: 562M Cache: 117M Swap: 0K/0K
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
95804 root 2 0 14M 18M sleep/0 poll 2:23 19.48% pktstat
68723 _openvpn 2 0 4000K 6364K sleep/0 poll 11:59 1.46% openvpn
52284 root 2 0 1336K 4004K sleep/1 poll 0:00 0.39% sshd
33560 _isakmpd 10 0 11M 15M run/0 - 7:36 0.34% isakmpd
80804 sshd 2 0 1304K 2948K sleep/1 select 0:00 0.24% sshd
How could I correct this situation ?
Thank you for your help.
OpenBSD 6.7 (GENERIC.MP) #4: Wed Jul 15 11:16:20 MDT 2020
r...@syspatch-67-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/
GENERIC.MP
real mem = 1047134208 (998MB)
avail mem = 1002844160 (956MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xfc690 (23 entries)
bios0: vendor American Megatrends Inc. version "080015" date 09/15/2010
bios0: AXIOMTEK NA-320
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC MCFG OEMB GSCI SSDT
acpi0: wakeup devices P0P1(S4) USB0(S4) USB1(S4) USB2(S4) USB3(S4) EUSB(S4)
P0P4(S4) P0P5(S4) P0P6(S4) P0P7(S4) P0P8(S4) P0P9(S4) USB4(S4) USB5(S4)
USBE(S4) GBEC(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Atom(TM) CPU D510 @ 1.66GHz, 1662.70 MHz, 06-1c-0a
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN
cpu0: 512KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 166MHz
cpu0: mwait min=64, max=64, C-substates=0.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Atom(TM) CPU D510 @ 1.66GHz, 1662.51 MHz, 06-1c-0a
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN
cpu1: 512KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 1 (application processor)
cpu2: Intel(R) Atom(TM) CPU D510 @ 1.66GHz, 1662.50 MHz, 06-1c-0a
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN
cpu2: 512KB 64b/line 8-way L2 cache
cpu2: smt 1, core 0, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Atom(TM) CPU D510 @ 1.66GHz, 1662.51 MHz, 06-1c-0a
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN
cpu3: 512KB 64b/line 8-way L2 cache
cpu3: smt 1, core 1, package 0
ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 20, 24 pins, remapped
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 2 (P0P4)
acpiprt2 at acpi0: bus 3 (P0P5)
acpiprt3 at acpi0: bus 4 (P0P6)
acpiprt4 at acpi0: bus 5 (P0P7)
acpiprt5 at acpi0: bus 6 (P0P8)
acpiprt6 at acpi0: bus 7 (P0P9)
acpicpu0 at acpi0: C1(1000@1 mwait.1)
acpicpu1 at acpi0: C1(1000@1 mwait.1)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
acpipci0 at acpi0 PCI0: _OSC failed
acpicmos0 at acpi0
acpibtn0 at acpi0: PWRB
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Pineview DMI" rev 0x02
inteldrm0 at pci0 dev 2 function 0 "Intel Pineview Video" rev 0x02
drm0 at inteldrm0
intagp0 at inteldrm0
agp0 at intagp0: aperture at 0xd0000000, size 0x10000000
inteldrm0: apic 4 int 16, PINEVIEW, gen 3
ppb0 at pci0 dev 28 function 0 "Intel 82801H PCIE" rev 0x04: msi
pci1 at ppb0 bus 2
em0 at pci1 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
00:60:e0:56:24:5d
ppb1 at pci0 dev 28 function 1 "Intel 82801H PCIE" rev 0x04: msi
pci2 at ppb1 bus 3
em1 at pci2 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
00:60:e0:56:24:5e
ppb2 at pci0 dev 28 function 2 "Intel 82801H PCIE" rev 0x04: msi
pci3 at ppb2 bus 4
em2 at pci3 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
00:60:e0:56:24:5f
ppb3 at pci0 dev 28 function 3 "Intel 82801H PCIE" rev 0x04: msi
pci4 at ppb3 bus 5
em3 at pci4 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
00:60:e0:56:24:60
ppb4 at pci0 dev 28 function 4 "Intel 82801H PCIE" rev 0x04: msi
pci5 at ppb4 bus 6
em4 at pci5 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
00:60:e0:56:24:61
ppb5 at pci0 dev 28 function 5 "Intel 82801H PCIE" rev 0x04: msi
pci6 at ppb5 bus 7
em5 at pci6 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
00:60:e0:56:24:62
uhci0 at pci0 dev 29 function 0 "Intel 82801H USB" rev 0x04: apic 4 int 23
uhci1 at pci0 dev 29 function 1 "Intel 82801H USB" rev 0x04: apic 4 int 19
uhci2 at pci0 dev 29 function 2 "Intel 82801H USB" rev 0x04: apic 4 int 18
ehci0 at pci0 dev 29 function 7 "Intel 82801H USB" rev 0x04: apic 4 int 23
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
ppb6 at pci0 dev 30 function 0 "Intel 82801BAM Hub-to-PCI" rev 0xf4
pci7 at ppb6 bus 1
pcib0 at pci0 dev 31 function 0 "Intel 82801HBM LPC" rev 0x04
pciide0 at pci0 dev 31 function 1 "Intel 82801HBM IDE" rev 0x04: DMA,
channel 0 configured to compatibility, channel 1 configured to compatibility
wd0 at pciide0 channel 0 drive 0: <TS4GCF133>
wd0: 1-sector PIO, LBA, 3823MB, 7831152 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 4
pciide0: channel 1 ignored (disabled)
pciide1 at pci0 dev 31 function 2 "Intel 82801HBM SATA" rev 0x04: DMA,
channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide1: using apic 4 int 18 for native-PCI interrupt
ichiic0 at pci0 dev 31 function 3 "Intel 82801H SMBus" rev 0x04: apic 4 int
17
iic0 at ichiic0
spdmem0 at iic0 addr 0x50: 1GB DDR2 SDRAM non-parity PC2-5300CL5 SO-DIMM
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
wbsio0 at isa0 port 0x2e/2: W83627DHG-P rev 0x73
lm1 at wbsio0 port 0xa00/8: W83627DHG
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on wd0a (a3db199668db825a.a) swap on wd0b dump on wd0b
drm:pid0:connector_bad_edid *WARNING* VGA-1: EDID is invalid:
[00] BAD f0 f0 f0 f0 f0 f0 f0 f0 f0 f0 f0 f0 e1 e1 e1 e1
[00] BAD c3 c3 c3 c3 87 87 87 87 0f 0f 0f 0f 1f 1f 1f 1f
[00] BAD 3f 3f 3f 3f 7f 7f 7f 7f ff ff ff ff ff ff ff ff
[00] BAD ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
[00] BAD ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
[00] BAD ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
[00] BAD ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
[00] BAD ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
inteldrm0: 1024x768, 32bpp
wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation), using wskbd0
wsdisplay0: screen 1-5 added (std, vt100 emulation)
--
Jean-Yves Boisiaud
