Synopsis: Experiencing frequent but intermittent network delays for
host bound and transit traffic on my OpenBSD router/firewall after
upgrading from 7.2 to 7.3
Category: system kernel amd64 Environment:
System : OpenBSD 7.3
Details : OpenBSD 7.3 (GENERIC.MP) #1125: Sat Mar 25 10:36:29 MDT
2023
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
Architecture: OpenBSD.amd64
Machine : amd64
Description:
I noticed that after upgrading from 7.2 to 7.3, frequent but
intermittent network delays. These delays are very brief but when they
happen they typically last from 500 - 900 ms. If I ping my gateway, I
generally get sub millisecond pings (0.400 - 0.700 ms) but
intermittently I'll get an echo replies (typically 1 echo reply) that
can range from 200 - 900ms. Then it returns to the sub millisecond echo
responses for a period of time (no pattern). Because these "delays" are
noticable for both host bound traffic (eg. SSH to the OpenBSD router)
and transit traffic, it affects persistent connections destined to the
router or passing through it.
It's noticible for things like SSH sessions to the OpenBSD router,
video and voice calls, or any other persistent connection. I've looked
at the system processes to see if there are any noticable spikes in CPU
usage, but I'm not seeing anything that stands out.
I'm not noticing these same symptoms for other layer2 traffic. I can
ping other hosts on my LAN and my switch and do not notice this
behaviour, so it's only for connections to or transiting through the
router.
Here is a sample output from pinging my gateway:
64 bytes from 10.77.1.1: icmp_seq=629 ttl=255 time=687.979 ms
64 bytes from 10.77.1.1: icmp_seq=630 ttl=255 time=0.420 ms
64 bytes from 10.77.1.1: icmp_seq=631 ttl=255 time=0.754 ms
64 bytes from 10.77.1.1: icmp_seq=632 ttl=255 time=0.418 ms
64 bytes from 10.77.1.1: icmp_seq=633 ttl=255 time=0.682 ms
64 bytes from 10.77.1.1: icmp_seq=634 ttl=255 time=0.435 ms
64 bytes from 10.77.1.1: icmp_seq=635 ttl=255 time=0.490 ms
64 bytes from 10.77.1.1: icmp_seq=636 ttl=255 time=0.780 ms
64 bytes from 10.77.1.1: icmp_seq=637 ttl=255 time=0.546 ms
64 bytes from 10.77.1.1: icmp_seq=638 ttl=255 time=0.697 ms
64 bytes from 10.77.1.1: icmp_seq=639 ttl=255 time=0.690 ms
64 bytes from 10.77.1.1: icmp_seq=640 ttl=255 time=658.827 ms
64 bytes from 10.77.1.1: icmp_seq=641 ttl=255 time=0.571 ms
64 bytes from 10.77.1.1: icmp_seq=642 ttl=255 time=0.383 ms
64 bytes from 10.77.1.1: icmp_seq=643 ttl=255 time=0.591 ms
64 bytes from 10.77.1.1: icmp_seq=644 ttl=255 time=0.555 ms
64 bytes from 10.77.1.1: icmp_seq=645 ttl=255 time=0.634 ms
64 bytes from 10.77.1.1: icmp_seq=646 ttl=255 time=0.541 ms
64 bytes from 10.77.1.1: icmp_seq=647 ttl=255 time=0.638 ms
64 bytes from 10.77.1.1: icmp_seq=648 ttl=255 time=0.689 ms
64 bytes from 10.77.1.1: icmp_seq=649 ttl=255 time=0.740 ms
64 bytes from 10.77.1.1: icmp_seq=650 ttl=255 time=0.667 ms
64 bytes from 10.77.1.1: icmp_seq=651 ttl=255 time=0.615 ms
64 bytes from 10.77.1.1: icmp_seq=652 ttl=255 time=0.737 ms
64 bytes from 10.77.1.1: icmp_seq=653 ttl=255 time=0.412 ms
64 bytes from 10.77.1.1: icmp_seq=654 ttl=255 time=0.625 ms
64 bytes from 10.77.1.1: icmp_seq=655 ttl=255 time=0.334 ms
64 bytes from 10.77.1.1: icmp_seq=656 ttl=255 time=0.446 ms
64 bytes from 10.77.1.1: icmp_seq=657 ttl=255 time=0.719 ms
64 bytes from 10.77.1.1: icmp_seq=658 ttl=255 time=0.760 ms
64 bytes from 10.77.1.1: icmp_seq=659 ttl=255 time=0.745 ms
64 bytes from 10.77.1.1: icmp_seq=660 ttl=255 time=0.749 ms
64 bytes from 10.77.1.1: icmp_seq=661 ttl=255 time=0.765 ms
64 bytes from 10.77.1.1: icmp_seq=662 ttl=255 time=589.051 ms
64 bytes from 10.77.1.1: icmp_seq=663 ttl=255 time=0.359 ms
64 bytes from 10.77.1.1: icmp_seq=664 ttl=255 time=0.657 ms
64 bytes from 10.77.1.1: icmp_seq=665 ttl=255 time=0.492 ms
64 bytes from 10.77.1.1: icmp_seq=666 ttl=255 time=0.711 ms
64 bytes from 10.77.1.1: icmp_seq=667 ttl=255 time=0.850 ms
64 bytes from 10.77.1.1: icmp_seq=668 ttl=255 time=0.664 ms
64 bytes from 10.77.1.1: icmp_seq=669 ttl=255 time=0.464 ms
64 bytes from 10.77.1.1: icmp_seq=670 ttl=255 time=0.620 ms
64 bytes from 10.77.1.1: icmp_seq=671 ttl=255 time=0.333 ms
64 bytes from 10.77.1.1: icmp_seq=672 ttl=255 time=0.353 ms
My network configuration is as follows:
ls -las hostname.*
4 -rw-r----- 1 root wheel 31 May 15 2021 hostname.aggr0
4 -rw-r----- 1 root wheel 3 May 15 2021 hostname.em0
4 -rw-r----- 1 root wheel 3 May 18 2020 hostname.em1
4 -rw-r----- 1 root wheel 32 May 18 2020 hostname.lo1
4 -rw-r----- 1 root wheel 69 Apr 12 06:26 hostname.vlan100
4 -rw-r----- 1 root wheel 65 May 15 2021 hostname.vlan105
4 -rw-r----- 1 root wheel 125 May 16 2021 hostname.vlan2000
4 -rw-r----- 1 root wheel 80 Apr 12 06:23 hostname.vlan99
4 -rw-r----- 1 root wheel 91 Apr 12 06:27 hostname.vlan999
4 -rw-r----- 1 root wheel 247 Mar 4 11:45 hostname.wg0
/etc/hostname.aggr0
trunkport em0
trunkport em1
up
/etc/hostname.em0
up
/etc/hostname.em1
up
/etc/hostname.lo1
inet 10.199.0.7 255.255.255.255
/etc/hostname.vlan100
parent aggr0
vnetid 100
inet 10.77.100.1/24
up
/etc/hostname.vlan105
parent aggr0
vnetid 105
inet 10.77.105.1/24
up
/etc/hostname.vlan2000
parent aggr0
vnetid 2000
inet 10.254.0.1/30
up
!route add -net 10.76.8/24 10.254.0.2
/etc/hostname.vlan99 (Home)
parent aggr0
vnetid 99
inet 10.77.1.1/24
inet6 autoconf
up
/etc/hostname.vlan999 (Internet circuit)
parent aggr0
vnetid 999
inet autoconf
inet6 autoconf
up
How-To-Repeat:
Happens consistently and even after reboots
Fix:
No fix that I've been able to figure out
dmesg:
OpenBSD 7.3 (GENERIC.MP) #1125: Sat Mar 25 10:36:29 MDT 2023
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 8519643136 (8124MB)
avail mem = 8242040832 (7860MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0x7cd0b020 (14 entries)
bios0: vendor coreboot version "v4.12.0.3" date 10/28/2020
bios0: Protectli FW4B
acpi0 at bios0: ACPI 6.0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP SSDT MCFG APIC
acpi0: wakeup devices XHCI(S3)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Celeron(R) CPU J3160 @ 1.60GHz, 2240.46 MHz, 06-4c-04
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,AES,RDRAND,NXE,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,TSC_ADJUST,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
cpu0: 24KB 64b/line 6-way D-cache, 32KB 64b/line 8-way I-cache, 1MB
64b/line 16-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 79MHz
cpu0: mwait min=64, max=64, C-substates=0.2, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Celeron(R) CPU J3160 @ 1.60GHz, 2240.95 MHz, 06-4c-04
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,AES,RDRAND,NXE,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,TSC_ADJUST,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
cpu1: 24KB 64b/line 6-way D-cache, 32KB 64b/line 8-way I-cache, 1MB
64b/line 16-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Celeron(R) CPU J3160 @ 1.60GHz, 2240.68 MHz, 06-4c-04
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,AES,RDRAND,NXE,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,TSC_ADJUST,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
cpu2: 24KB 64b/line 6-way D-cache, 32KB 64b/line 8-way I-cache, 1MB
64b/line 16-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Celeron(R) CPU J3160 @ 1.60GHz, 2241.25 MHz, 06-4c-04
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,AES,RDRAND,NXE,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,TSC_ADJUST,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
cpu3: 24KB 64b/line 6-way D-cache, 32KB 64b/line 8-way I-cache, 1MB
64b/line 16-way L2 cache
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 115 pins,
remapped
acpiprt0 at acpi0: bus 0 (PCI0)
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
"INT33BD" at acpi0 not configured
acpicmos0 at acpi0
com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
com0: console
chvgpio0 at acpi0 GPSW uid 1 addr 0xfed80000/0x8000 irq 49, 56 pins
chvgpio1 at acpi0 GPNC uid 2 addr 0xfed88000/0x8000 irq 48, 59 pins
chvgpio2 at acpi0 GPEC uid 3 addr 0xfed90000/0x8000 irq 50, 24 pins
chvgpio3 at acpi0 GPSE uid 4 addr 0xfed98000/0x8000 irq 91, 55 pins
"BOOT0000" at acpi0 not configured
acpicpu0 at acpi0: C2 bad (state 6 has no substates): C3 bad (state 6
has no substates): C1(1000@1 mwait.1), PSS
acpicpu1 at acpi0: C2 bad (state 6 has no substates): C3 bad (state 6
has no substates): C1(1000@1 mwait.1), PSS
acpicpu2 at acpi0: C2 bad (state 6 has no substates): C3 bad (state 6
has no substates): C1(1000@1 mwait.1), PSS
acpicpu3 at acpi0: C2 bad (state 6 has no substates): C3 bad (state 6
has no substates): C1(1000@1 mwait.1), PSS
cpu0: using VERW MDS workaround
cpu0: Enhanced SpeedStep 2240 MHz: speeds: 1601, 1600, 1520, 1440, 1360,
1280, 1200, 1120, 1040, 960, 880, 800, 720, 640, 560, 480 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Braswell Host" rev 0x35
inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics" rev 0x35
drm0 at inteldrm0
inteldrm0: msi, CHERRYVIEW, gen 8
ahci0 at pci0 dev 19 function 0 "Intel Braswell AHCI" rev 0x35: msi,
AHCI 1.3.1
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, KINGSTON SUV500M, 0030>
naa.50026b77836de374
sd0: 228936MB, 512 bytes/sector, 468862128 sectors, thin
xhci0 at pci0 dev 20 function 0 "Intel Braswell xHCI" rev 0x35: msi,
xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev
3.00/1.00 addr 1
azalia0 at pci0 dev 27 function 0 "Intel Braswell HD Audio" rev 0x35:
msi
azalia0: codecs: Realtek ALC662, Intel/0x2883, using Realtek ALC662
audio0 at azalia0
ppb0 at pci0 dev 28 function 0 "Intel Braswell PCIE" rev 0x35: msi
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel I210" rev 0x03: msi, address
00:e0:67:22:36:a4
ppb1 at pci0 dev 28 function 1 "Intel Braswell PCIE" rev 0x35: msi
pci2 at ppb1 bus 2
em1 at pci2 dev 0 function 0 "Intel I210" rev 0x03: msi, address
00:e0:67:22:36:a5
ppb2 at pci0 dev 28 function 2 "Intel Braswell PCIE" rev 0x35: msi
pci3 at ppb2 bus 3
em2 at pci3 dev 0 function 0 "Intel I210" rev 0x03: msi, address
00:e0:67:22:36:a6
ppb3 at pci0 dev 28 function 3 "Intel Braswell PCIE" rev 0x35: msi
pci4 at ppb3 bus 4
em3 at pci4 dev 0 function 0 "Intel I210" rev 0x03: msi, address
00:e0:67:22:36:a7
pcib0 at pci0 dev 31 function 0 "Intel Braswell PCU LPC" rev 0x35
ichiic0 at pci0 dev 31 function 3 "Intel Braswell SMBus" rev 0x35: apic
2 int 18
iic0 at ichiic0
spdmem0 at iic0 addr 0x50: 8GB DDR3 SDRAM PC3-12800 SO-DIMM
isa0 at pcib0
isadma0 at isa0
vga0 at isa0 port 0x3b0/48 iomem 0xa0000/131072
wsdisplay at vga0 not configured
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT (using slow L1TF mitigation)
umass0 at uhub0 port 1 configuration 1 interface 0 "SanDisk Cruzer Fit"
rev 2.00/1.27 addr 2
umass0: using SCSI over Bulk-Only
scsibus2 at umass0: 2 targets, initiator 0
sd1 at scsibus2 targ 1 lun 0: <SanDisk, Cruzer Fit, 1.27> removable
serial.07815571870804115450
sd1: 7633MB, 512 bytes/sector, 15633408 sectors
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (993900eb51631b95.a) swap on sd0b dump on sd0b
inteldrm0: 1024x768, 32bpp
wsdisplay0 at inteldrm0 mux 1
wsdisplay0: screen 0-5 added (std, vt100 emulation)
usbdevs:
Controller /dev/usb0:
addr 01: 8086:0000 Intel, xHCI root hub
super speed, self powered, config 1, rev 1.00
driver: uhub0
addr 02: 0781:5571 SanDisk, Cruzer Fit
high speed, power 200 mA, config 1, rev 1.27, iSerial
4C530102870804115450
driver: umass0
