Hello everyone,
We've a few OpenBSD running -release at work and acting as IKE "road
warrior" VPN servers, serving traffic to Android (strongswan), Linux
(also strongswan) and Windows 10/11 ("default" Windows WAN Miniport
IKEv2 driver) clients.
Up until 7.5 everything was running smoothly. Last week we upgraded two
of our machines to 7.6 (nothing more than sysupgrade > pkg_add -u,
since sysmerge returned nothing and the configuration changes weren't
relevant for these servers).
Connections engaged with Linux or Android clients didn't show any
noticeable difference, while Windows users started to complain that
every download (whether it be on shared folders over SMB or straight
from an host on the broader Internet) comes almost to an halt.
Upon further investigation on multiple Windows clients (each from
different location and connected to our VPN endpoint through varying
media/ISP), we confirmed that download rates went down as low as 10kb/s
(from 10Mb/s on average on 7.5) as soon as SMB or any "barely big" http
download was involved.
Interestingly, "light" Web traffic wasn't affected, and Windows clients
were still able to get consistently north of 50Mb/s each when surfing
(on youtube for instance).
Both servers were affected the same, albeit being on slightly different
locations in our network.
Therefore, MTU and TCP MSS were the first thing we checked: MTU was
still set at its default value (on both servers), but MSS (as well as
randomizing TCP id and unsetting the DF bit) was already taken care of
in pf.conf, since well before the upgrade (and left unchanged after,
here again on both servers).
We lowered the MTU and MSS (to 1400 / 1350), and even dropped the
"scrub" line in pf.conf, to see if it has an impact at all, but to no
avail.
Everything else being equal (no Windows update on clients, no
meaningful network maintenance, ...) except for the 7.5>7.6 upgrade, we
decided to restore one server to its 7.5 version. Performance on
SMB/HTTP download got back to "normal" immediately.
As seen in dmesg (see below), both servers are guest machines, and as
such use vio(4) (more precisely a vlan(4) on top of vio(4)).
We did notice in the 7.6 changelog that some changes have been made to
vio(4), but are unsure if they come at play here.
There is also a "Fixed IPsec in use with IP forwarding 2 logic" in
changelog, but since we had this sysctl set to 2 in 7.5 not sure here also if
that's relevant.
Some configuration details (same on both servers):
iked.conf:
ikev2 "ike_user" passive esp \
from any to dynamic \
config address 10.0.24.1/23 \
config netmask 255.255.254.0 \
config dhcp-server 172.31.12.101 \
config name-server 172.31.12.101 \
config netbios-server 172.31.12.101 \
tag "IKExx"
pf.conf:
[...]
pass in log on $ext_if proto udp from any to self port {isakmp, ipsec-nat-t}
match on any scrub (random-id no-df max-mss 1394)
pass log on enc0 tagged IKExx
match out log on $ext_if inet tagged IKExx
[...]
sysctl.conf
net.inet.ip.forwarding=2
dmesg (7.5):
OpenBSD 7.5 (GENERIC.MP) #2: Mon Sep 16 15:56:43 CEST 2024
r...@syspatch-75-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 2130563072 (2031MB)
avail mem = 2045104128 (1950MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5220 (10 entries)
bios0: vendor SeaBIOS version "1.16.3-debian-1.16.3-2~bpo12+1" date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S5
acpi0: tables DSDT FACP APIC HPET WAET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD EPYC-Milan Processor, 1996.56 MHz, 19-01-01
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LA
HF,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,TOPEXT,CPCTR,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,PKU,IBRS,IBPB,SSBD,IBPB,IBRS,STIBP,SSBD,VIRTSSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB
64b/line 8-way L2 cache, 32MB 64b/line 16-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD EPYC-Milan Processor, 1996.62 MHz, 19-01-01
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LA
HF,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,TOPEXT,CPCTR,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,PKU,IBRS,IBPB,SSBD,IBPB,IBRS,STIBP,SSBD,VIRTSSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB
64b/line 8-way L2 cache, 32MB 64b/line 16-way L3 cache
cpu1: smt 0, core 0, package 1
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
acpicmos0 at acpi0
"ACPI0010" at acpi0 not configured
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA,
channel 0 wired to compatibility, channel 1 wired to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
cd0(pciide0:0:0): using PIO mode 4, DMA mode 2
pciide0: channel 1 disabled (no drives)
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Bochs VGA" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address 52:54:00:c9:03:28
virtio0: msix per-VQ
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio1
scsibus2 at vioblk0: 1 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 15360MB, 512 bytes/sector, 31457280 sectors
virtio1: msix per-VQ
virtio2 at pci0 dev 5 function 0 "Qumranet Virtio Memory Balloon" rev 0x00
viomb0 at virtio2
virtio2: apic 0 int 10
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (571c338be1b81e5d.a) swap on sd0b dump on sd0b
dmesg (7.6):
OpenBSD 7.6 (GENERIC.MP) #338: Mon Sep 30 08:55:35 MDT 2024
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 2130563072 (2031MB)
avail mem = 2042884096 (1948MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5220 (10 entries)
bios0: vendor SeaBIOS version "1.16.3-debian-1.16.3-2~bpo12+1" date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S5
acpi0: tables DSDT FACP APIC HPET WAET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD EPYC-Milan Processor, 1996.54 MHz, 19-01-01
cpu0: cpuid 1
edx=78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2>
ecx=f6fa3203<SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV>
cpu0: cpuid 6 eax=4<ARAT>
cpu0: cpuid 7.0
ebx=219c07a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA>
ecx=40000c<UMIP,PKU> edx=84000010<IBRS,IBPB,SSBD>
cpu0: cpuid d.1 eax=f<XSAVEOPT,XSAVEC,XGETBV1,XSAVES>
cpu0: cpuid 80000001 edx=2fd3fbff<NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG>
ecx=c003f1<LAHF,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,TOPEXT,CPCTR>
cpu0: cpuid 80000008 ebx=300d205<IBPB,IBRS,STIBP,SSBD,VIRTSSBD>
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB
64b/line 8-way L2 cache, 32MB 64b/line 16-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD EPYC-Milan Processor, 1996.66 MHz, 19-01-01
cpu1: smt 0, core 0, package 1
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
acpicmos0 at acpi0
"ACPI0010" at acpi0 not configured
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA,
channel 0 wired to compatibility, channel 1 wired to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
cd0(pciide0:0:0): using PIO mode 4, DMA mode 2
pciide0: channel 1 disabled (no drives)
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Bochs VGA" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0, address 52:54:00:0f:37:07
virtio0: msix per-VQ
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio1
scsibus2 at vioblk0: 1 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 15360MB, 512 bytes/sector, 31457280 sectors
virtio1: msix per-VQ
virtio2 at pci0 dev 5 function 0 "Qumranet Virtio Memory Balloon" rev 0x00
viomb0 at virtio2
virtio2: apic 0 int 10
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (1f2435ee5375e4cb.a) swap on sd0b dump on sd0b
fd0 at fdc0 drive 1: density unknown
We've kept one IKE server upgraded to 7.6 on purpose, to help with
troubleshooting.
Loïc REVEST
