Hi all,
I'm testing forwarding over em with plain with snapshot
em0 at pci7 dev 0 function 0 "Intel 82576" rev 0x01: msi,
em1 at pci7 dev 0 function 1 "Intel 82576" rev 0x01: msi,
em2 at pci8 dev 0 function 0 "Intel I210" rev 0x03: msi,
em3 at pci9 dev 0 function 0 "Intel I210" rev 0x03: msi,
em4 at pci12 dev 0 function 0 "Intel I350" rev 0x01: msi,
em5 at pci12 dev 0 function 1 "Intel I350" rev 0x01: msi,
em6 at pci12 dev 0 function 2 "Intel I350" rev 0x01: msi,
em7 at pci12 dev 0 function 3 "Intel I350" rev 0x01: msi,
ix0 at pci5 dev 0 function 0 "Intel X552 SFP+" rev 0x00, msix, 4 queues,
ix1 at pci5 dev 0 function 1 "Intel X552 SFP+" rev 0x00, msix, 4 queues,
em0 to em5 is for testing, em6 is for ssh and other interfaces are in shut.
em0 and em1 interfaces are pci card and em2-em5 are onboard.
I'm sending traffic on all 6 em interfaces at the same time and when
doing that I'm seeing 4 active softnet task
sleep/2 bored 1:12 44.73% softnet
sleep/1 bored 2:55 35.99% softnet
sleep/1 bored 0:18 31.20% softnet
sleep/0 bored 0:05 5.86% softnet
is this ok ? I thought that parallel forwarding is active only over
multiqueue interfaces.
Other funny thing is that if I leave traffic for some time I can get
softnet tasks over 100%.
13281 469454 64 0 0K 860K onproc/3 - 554:59 5071.39%
softnet
53429 575885 64 0 0K 860K onproc/0 - 88:27 4955.62%
softnet
57655 540258 10 0 0K 860K sleep/2 bored 129:23 4683.54%
softnet
61778 462853 64 0 0K 860K onproc/1 - 503:16 2583.94%
softnet
70216 420909 105 20 0K 860K sleep/2 pgzero 6:48 489.60%
zerothread
22474 596339 -18 0 0K 860K sleep/2 reaper 6:47 6.15% reaper
56297 350215 28 0 0K 860K run/0 - 0:12 1.76%
softclock
73318 605906 10 0 0K 860K sleep/1 bored 0:02 1.46% systq
I am aware that this is extreme but I've never seen something like that,
so I'm posting it here if someone could give me rough explanation what
I'm seeing here ..
Thank you
OpenBSD 7.2-beta (GENERIC.MP) #692: Mon Aug 15 11:36:43 MDT 2022
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 17052663808 (16262MB)
avail mem = 16518471680 (15753MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xed9b0 (48 entries)
bios0: vendor American Megatrends Inc. version "2.3" date 05/07/2021
bios0: Supermicro Super Server
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG UEFI DBG2 HPET WDDT
SSDT SSDT SSDT PRAD DMAR HEST BERT ERST EINJ
acpi0: wakeup devices IP2P(S4) EHC1(S4) EHC2(S4) RP07(S4) RP08(S4)
BR1A(S4) BR1B(S4) BR2A(S4) BR2B(S4) BR2C(S4) BR2D(S4) BR3A(S4) BR3B(S4)
BR3C(S4) BR3D(S4) RP01(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz, 2200.31 MHz, 06-56-03
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
64b/line 8-way L2 cache, 6MB 64b/line 12-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz, 2200.01 MHz, 06-56-03
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
64b/line 8-way L2 cache, 6MB 64b/line 12-way L3 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz, 2200.01 MHz, 06-56-03
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
64b/line 8-way L2 cache, 6MB 64b/line 12-way L3 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz, 2200.01 MHz, 06-56-03
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
64b/line 8-way L2 cache, 6MB 64b/line 12-way L3 cache
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0x80000000, bus 0-255
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 255 (UNC0)
acpiprt1 at acpi0: bus 0 (PCI0)
acpiprt2 at acpi0: bus 1 (BR1A)
acpiprt3 at acpi0: bus 2 (BR1B)
acpiprt4 at acpi0: bus 4 (BR2C)
acpiprt5 at acpi0: bus 6 (BR3A)
acpiprt6 at acpi0: bus 9 (RP01)
acpiprt7 at acpi0: bus 10 (RP02)
acpiprt8 at acpi0: bus 11 (RP04)
acpiprt9 at acpi0: bus 12 (BR3A)
acpiprt10 at acpi0: bus 13 (RP05)
acpipci0 at acpi0 UNC0: 0x00000010 0x00000011 0x00000000
"ACPI0004" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"PNP0C33" at acpi0 not configured
"PNP0C14" at acpi0 not configured
acpipci1 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
"PNP0003" at acpi0 not configured
"PNP0C14" at acpi0 not configured
acpicmos0 at acpi0
com0 at acpi0 UAR1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
com1 at acpi0 UAR2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo
com1: console
"IPI0001" at acpi0 not configured
acpibtn0 at acpi0: PWRB
acpicpu0 at acpi0: C1(1000@1 halt), PSS
acpicpu1 at acpi0: C1(1000@1 halt), PSS
acpicpu2 at acpi0: C1(1000@1 halt), PSS
acpicpu3 at acpi0: C1(1000@1 halt), PSS
ipmi at mainbus0 not configured
cpu0: using VERW MDS workaround (except on vmm entry)
cpu0: Enhanced SpeedStep 2200 MHz: speeds: 2200, 2100, 2000, 1900, 1800,
1700, 1600, 1500, 1400, 1300, 1200, 1100, 1000, 900, 800 MHz
pci0 at mainbus0 bus 255
"Intel Xeon-D QPI Link" rev 0x03 at pci0 dev 11 function 0 not configured
"Intel Xeon-D QPI Link" rev 0x03 at pci0 dev 11 function 1 not configured
"Intel Xeon-D QPI Link" rev 0x03 at pci0 dev 11 function 2 not configured
"Intel Xeon-D QPI Debug" rev 0x03 at pci0 dev 11 function 3 not configured
"Intel Xeon-D Cache" rev 0x03 at pci0 dev 12 function 0 not configured
"Intel Xeon-D Cache" rev 0x03 at pci0 dev 12 function 1 not configured
"Intel Xeon-D Cache" rev 0x03 at pci0 dev 12 function 2 not configured
"Intel Xeon-D Cache" rev 0x03 at pci0 dev 12 function 3 not configured
"Intel Xeon-D Cache" rev 0x03 at pci0 dev 15 function 0 not configured
"Intel Xeon-D Cache" rev 0x03 at pci0 dev 15 function 4 not configured
"Intel Xeon-D Cache" rev 0x03 at pci0 dev 15 function 5 not configured
"Intel Xeon-D Cache" rev 0x03 at pci0 dev 15 function 6 not configured
"Intel Xeon-D PCIE" rev 0x03 at pci0 dev 16 function 0 not configured
"Intel E5 v4 R2PCIe Agent" rev 0x03 at pci0 dev 16 function 1 not configured
"Intel Xeon-D Ubox" rev 0x03 at pci0 dev 16 function 5 not configured
"Intel Xeon-D Ubox" rev 0x03 at pci0 dev 16 function 6 not configured
"Intel Xeon-D Ubox" rev 0x03 at pci0 dev 16 function 7 not configured
"Intel Xeon-D Home Agent" rev 0x03 at pci0 dev 18 function 0 not configured
"Intel Xeon-D Home Agent" rev 0x03 at pci0 dev 18 function 1 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 0 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 1 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 2 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 3 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 4 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 5 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 6 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 7 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 0 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 1 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 2 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 3 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 4 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 5 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 6 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 7 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 21 function 0 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 21 function 1 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 21 function 2 not configured
"Intel Xeon-D Memory" rev 0x03 at pci0 dev 21 function 3 not configured
"Intel Xeon-D PCU" rev 0x03 at pci0 dev 30 function 0 not configured
"Intel Xeon-D PCU" rev 0x03 at pci0 dev 30 function 1 not configured
"Intel Xeon-D PCU" rev 0x03 at pci0 dev 30 function 2 not configured
"Intel Xeon-D PCU" rev 0x03 at pci0 dev 30 function 3 not configured
"Intel Xeon-D PCU" rev 0x03 at pci0 dev 30 function 4 not configured
"Intel Xeon-D PCU" rev 0x03 at pci0 dev 31 function 0 not configured
"Intel Xeon-D PCU" rev 0x03 at pci0 dev 31 function 2 not configured
pci1 at mainbus0 bus 0
pchb0 at pci1 dev 0 function 0 "Intel Xeon-D Host" rev 0x03
ppb0 at pci1 dev 1 function 0 "Intel Xeon-D PCIE" rev 0x03
pci2 at ppb0 bus 1
ppb1 at pci1 dev 1 function 1 "Intel Xeon-D PCIE" rev 0x03
pci3 at ppb1 bus 2
ppb2 at pci1 dev 2 function 0 "Intel Xeon-D PCIE" rev 0x03
pci4 at ppb2 bus 3
"Intel Xeon-D QuickData" rev 0x00 at pci4 dev 0 function 0 not configured
"Intel Xeon-D QuickData" rev 0x00 at pci4 dev 0 function 1 not configured
"Intel Xeon-D QuickData" rev 0x00 at pci4 dev 0 function 2 not configured
"Intel Xeon-D QuickData" rev 0x00 at pci4 dev 0 function 3 not configured
ppb3 at pci1 dev 2 function 2 "Intel Xeon-D PCIE" rev 0x03
pci5 at ppb3 bus 4
ix0 at pci5 dev 0 function 0 "Intel X552 SFP+" rev 0x00, msix, 4 queues,
address 00:25:90:5d:ca:38
ix1 at pci5 dev 0 function 1 "Intel X552 SFP+" rev 0x00, msix, 4 queues,
address 00:25:90:5d:ca:39
ppb4 at pci1 dev 3 function 0 "Intel Xeon-D PCIE" rev 0x03: msi
pci6 at ppb4 bus 6
ppb5 at pci1 dev 3 function 2 "Intel Xeon-D PCIE" rev 0x03
pci7 at ppb5 bus 7
em0 at pci7 dev 0 function 0 "Intel 82576" rev 0x01: msi, address
00:1b:21:61:8a:94
em1 at pci7 dev 0 function 1 "Intel 82576" rev 0x01: msi, address
00:1b:21:61:8a:95
"Intel Xeon-D Address Map" rev 0x03 at pci1 dev 5 function 0 not configured
"Intel Xeon-D Hot Plug" rev 0x03 at pci1 dev 5 function 1 not configured
"Intel Xeon-D RAS" rev 0x03 at pci1 dev 5 function 2 not configured
xhci0 at pci1 dev 20 function 0 "Intel 8 Series xHCI" rev 0x05: msi,
xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev
3.00/1.00 addr 1
"Intel 8 Series MEI" rev 0x04 at pci1 dev 22 function 0 not configured
"Intel 8 Series MEI" rev 0x04 at pci1 dev 22 function 1 not configured
ehci0 at pci1 dev 26 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 18
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
ppb6 at pci1 dev 28 function 0 "Intel 8 Series PCIE" rev 0xd5: msi
pci8 at ppb6 bus 9
em2 at pci8 dev 0 function 0 "Intel I210" rev 0x03: msi, address
00:25:90:5d:c9:98
ppb7 at pci1 dev 28 function 1 "Intel 8 Series PCIE" rev 0xd5: msi
pci9 at ppb7 bus 10
em3 at pci9 dev 0 function 0 "Intel I210" rev 0x03: msi, address
00:25:90:5d:c9:99
ppb8 at pci1 dev 28 function 3 "Intel 8 Series PCIE" rev 0xd5: msi
pci10 at ppb8 bus 11
ppb9 at pci10 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x03
pci11 at ppb9 bus 12
vga1 at pci11 dev 0 function 0 "ASPEED Technology AST2000" rev 0x30
wsdisplay at vga1 not configured
ppb10 at pci1 dev 28 function 4 "Intel 8 Series PCIE" rev 0xd5: msi
pci12 at ppb10 bus 13
em4 at pci12 dev 0 function 0 "Intel I350" rev 0x01: msi, address
00:25:90:5d:c9:9a
em5 at pci12 dev 0 function 1 "Intel I350" rev 0x01: msi, address
00:25:90:5d:c9:9b
em6 at pci12 dev 0 function 2 "Intel I350" rev 0x01: msi, address
00:25:90:5d:c9:9c
em7 at pci12 dev 0 function 3 "Intel I350" rev 0x01: msi, address
00:25:90:5d:c9:9d
ehci1 at pci1 dev 29 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 18
usb2 at ehci1: USB revision 2.0
uhub2 at usb2 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
pcib0 at pci1 dev 31 function 0 "Intel C224 LPC" rev 0x05
ahci0 at pci1 dev 31 function 2 "Intel 8 Series AHCI" rev 0x05: msi,
AHCI 1.3
ahci0: port 4: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 4 lun 0: <ATA, INTEL SSDSCKJF24, LT2i>
naa.55cd2e414c94a064
sd0: 228936MB, 512 bytes/sector, 468862128 sectors, thin
ichiic0 at pci1 dev 31 function 3 "Intel 8 Series SMBus" rev 0x05: apic
8 int 18
iic0 at ichiic0
"Intel 8 Series Thermal" rev 0x05 at pci1 dev 31 function 6 not configured
isa0 at pcib0
isadma0 at isa0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT
uhub3 at uhub0 port 3 configuration 1 interface 0 "Genesys Logic USB2.0
Hub" rev 2.00/32.98 addr 2
uhub4 at uhub3 port 2 configuration 1 interface 0 "Genesys Logic USB2.0
Hub" rev 2.00/32.98 addr 3
uhub5 at uhub0 port 4 configuration 1 interface 0 "ATEN International
product 0x7000" rev 2.00/0.00 addr 4
uhidev0 at uhub5 port 1 configuration 1 interface 0 "ATEN International
product 0x2419" rev 1.10/1.00 addr 5
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0 mux 1
uhidev1 at uhub5 port 1 configuration 1 interface 1 "ATEN International
product 0x2419" rev 1.10/1.00 addr 5
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
uhub6 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching
Hub" rev 2.00/0.05 addr 2
uhub7 at uhub2 port 1 configuration 1 interface 0 "Intel Rate Matching
Hub" rev 2.00/0.05 addr 2
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (a4e2e39914e3ace8.a) swap on sd0b dump on sd0b
