Greetings:
I am troubleshooting an issue: users complaining about network performance. The
firewall
is an OpenBSD 6.7 system with patches applied. I've traced the issue and I'm
seeing the
congestion counter incrementing on system. The problems that we're seeing fit
with what
I have been able to find about congestion - when the firewall is congested it
continues
passing packets that match existing state entries but it will not create any
new state
entries until the congestion clears.
I'm having trouble troubleshooting it beyond that point because I have not been
able to
find any additional information about what the congestion counter is counting.
There is
the information in the pfctl man page: "congestion: network interface queue
congested",
but beyond that I can't really find any information about exactly what network
interface
queue is congested.
I'm not seeing packets being dropped, either on the switch side or firewall
side that
correspond with the congestion counter going up. The average on the congestion
counter
stays around 10/s, but what it's really doing is going up by 100-300/s for
short periods
and then not moving for longer periods.
If anyone could spare a couple of sentences or a share a link to a page
detailing what
state causes the system to consider itself contested, I would appreciate it.
Thanks for your time.
-Scott
System dmesg:
OpenBSD 6.7 (GENERIC.MP) #6: Thu Sep 3 14:08:18 MDT 2020
r...@syspatch-67-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 8386699264 (7998MB)
avail mem = 8119902208 (7743MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7fb76000 (62 entries)
bios0: vendor American Megatrends Inc. version "2.2" date 05/23/2018
bios0: Supermicro X11SSL-F
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG HPET LPIT SSDT SSDT SSDT DBGP
DBG2 SSDT SSDT UEFI SSDT DMAR EINJ ERST BERT HEST
acpi0: wakeup devices PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) PEGP(S4)
RP09(S4) PXSX(S4) RP10(S4) PXSX(S4) RP11(S4) PXSX(S4) RP12(S4) PXSX(S4)
RP13(S4) PXSX(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3901.62 MHz, 06-9e-09
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 24MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 1 (application processor)
cpu4: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 1, core 0, package 0
cpu5 at mainbus0: apid 3 (application processor)
cpu5: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.00 MHz, 06-9e-09
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: smt 1, core 1, package 0
cpu6 at mainbus0: apid 5 (application processor)
cpu6: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
cpu6:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu6: 256KB 64b/line 8-way L2 cache
cpu6: smt 1, core 2, package 0
cpu7 at mainbus0: apid 7 (application processor)
cpu7: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
cpu7:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu7: 256KB 64b/line 8-way L2 cache
cpu7: smt 1, core 3, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 23999999 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEG0)
acpiprt2 at acpi0: bus -1 (PEG1)
acpiprt3 at acpi0: bus -1 (PEG2)
acpiprt4 at acpi0: bus 2 (RP09)
acpiprt5 at acpi0: bus 3 (RP10)
acpiprt6 at acpi0: bus 4 (RP11)
acpiprt7 at acpi0: bus 5 (BR51)
acpiprt8 at acpi0: bus -1 (RP12)
acpiprt9 at acpi0: bus -1 (RP13)
acpiprt10 at acpi0: bus -1 (RP01)
acpiprt11 at acpi0: bus -1 (RP05)
acpiprt12 at acpi0: bus -1 (RP17)
acpiprt13 at acpi0: bus -1 (RP18)
acpiprt14 at acpi0: bus -1 (RP19)
acpiprt15 at acpi0: bus -1 (RP20)
acpiprt16 at acpi0: bus -1 (RP14)
acpiprt17 at acpi0: bus -1 (RP15)
acpiprt18 at acpi0: bus -1 (RP16)
acpiec0 at acpi0: not present
acpicpu0 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpicpu1 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpicpu2 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpicpu3 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpicpu4 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpicpu5 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpicpu6 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpicpu7 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpipwrres0 at acpi0: PG00, resource for PEG0
acpipwrres1 at acpi0: PG01, resource for PEG1
acpipwrres2 at acpi0: PG02, resource for PEG2
acpipwrres3 at acpi0: WRST
acpipwrres4 at acpi0: WRST
acpipwrres5 at acpi0: WRST
acpipwrres6 at acpi0: WRST
acpipwrres7 at acpi0: WRST
acpipwrres8 at acpi0: WRST
acpipwrres9 at acpi0: WRST
acpipwrres10 at acpi0: WRST
acpipwrres11 at acpi0: WRST
acpipwrres12 at acpi0: WRST
acpipwrres13 at acpi0: WRST
acpipwrres14 at acpi0: WRST
acpipwrres15 at acpi0: WRST
acpipwrres16 at acpi0: WRST
acpipwrres17 at acpi0: WRST
acpipwrres18 at acpi0: WRST
acpipwrres19 at acpi0: WRST
acpipwrres20 at acpi0: WRST
acpipwrres21 at acpi0: WRST
acpipwrres22 at acpi0: WRST
acpipwrres23 at acpi0: FN00, resource for FAN0
acpipwrres24 at acpi0: FN01, resource for FAN1
acpipwrres25 at acpi0: FN02, resource for FAN2
acpipwrres26 at acpi0: FN03, resource for FAN3
acpipwrres27 at acpi0: FN04, resource for FAN4
acpitz0 at acpi0: critical temperature is 119 degC
acpitz1 at acpi0: critical temperature is 119 degC
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
"IPI0001" at acpi0 not configured
"INT0E0C" at acpi0 not configured
acpibtn0 at acpi0: SLPB
"INT33A1" at acpi0 not configured
acpibtn1 at acpi0: PWRB
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
acpivideo0 at acpi0: GFX0
acpivout0 at acpivideo0: DD1F
ipmi at mainbus0 not configured
cpu0: using Skylake AVX MDS workaround
cpu0: Enhanced SpeedStep 3901 MHz: speeds: 3901, 3900, 3700, 3500, 3200, 3000,
2800, 2600, 2300, 2100, 1900, 1700, 1500, 1200, 1000, 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 v6/7 Host" rev 0x05
ppb0 at pci0 dev 1 function 0 "Intel Core 6G PCIE" rev 0x05: msi
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel I350" rev 0x01: msi, address
ac:1f:6b:96:bd:4c
em1 at pci1 dev 0 function 1 "Intel I350" rev 0x01: msi, address
ac:1f:6b:96:bd:4d
em2 at pci1 dev 0 function 2 "Intel I350" rev 0x01: msi, address
ac:1f:6b:96:bd:4e
em3 at pci1 dev 0 function 3 "Intel I350" rev 0x01: msi, address
ac:1f:6b:96:bd:4f
"Intel 100 Series ISH" rev 0x31 at pci0 dev 19 function 0 not configured
xhci0 at pci0 dev 20 function 0 "Intel 100 Series xHCI" rev 0x31: msi, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00
addr 1
pchtemp0 at pci0 dev 20 function 2 "Intel 100 Series Thermal" rev 0x31
"Intel 100 Series MEI" rev 0x31 at pci0 dev 22 function 0 not configured
ahci0 at pci0 dev 23 function 0 "Intel 100 Series AHCI" rev 0x31: msi, AHCI
1.3.1
ahci0: port 2: 6.0Gb/s
ahci0: port 3: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 2 lun 0: <ATA, WDC WD5000LPLX-0, 01.0> naa.50014ee6b30cf538
sd0: 476940MB, 512 bytes/sector, 976773168 sectors
sd1 at scsibus1 targ 3 lun 0: <ATA, WDC WD5000LPLX-0, 01.0> naa.50014ee6b30182ec
sd1: 476940MB, 512 bytes/sector, 976773168 sectors
ppb1 at pci0 dev 29 function 0 "Intel 100 Series PCIE" rev 0xf1: msi
pci2 at ppb1 bus 2
em4 at pci2 dev 0 function 0 "Intel I210" rev 0x03: msi, address
ac:1f:6b:97:4a:bc
ppb2 at pci0 dev 29 function 1 "Intel 100 Series PCIE" rev 0xf1: msi
pci3 at ppb2 bus 3
em5 at pci3 dev 0 function 0 "Intel I210" rev 0x03: msi, address
ac:1f:6b:97:4a:bd
ppb3 at pci0 dev 29 function 2 "Intel 100 Series PCIE" rev 0xf1: msi
pci4 at ppb3 bus 4
ppb4 at pci4 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x03
pci5 at ppb4 bus 5
vga1 at pci5 dev 0 function 0 "ASPEED Technology AST2000" rev 0x30
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel C232 LPC" rev 0x31
"Intel 100 Series PMC" rev 0x31 at pci0 dev 31 function 2 not configured
ichiic0 at pci0 dev 31 function 4 "Intel 100 Series SMBus" rev 0x31: apic 2 int
16
iic0 at ichiic0
iic0: addr 0x19 00=00 01=00 02=04 03=00 04=06 05=01 06=1c 07=22 08=00 words
00=00ef 01=000d 02=04a0 03=0000 04=0600 05=0150 06=1c85 07=2221
iic0: addr 0x1b 00=00 01=00 02=04 03=00 04=06 05=01 06=1c 07=22 08=00 words
00=00ef 01=000d 02=04a0 03=0000 04=0600 05=0154 06=1c85 07=2221
iic0: addr 0x4c 04=01 44=01 84=01 c0=03 c1=01 c4=01 c6=01 ca=01 cb=01 cc=01
cd=02 cf=01 d1=02 d4=07 da=0f db=02 dd=01 de=03 words 00=ffff 01=ffff 02=ffff
03=ffff 04=01ff 05=ffff 06=ffff 07=ffff
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT
uhub1 at uhub0 port 9 configuration 1 interface 0 "ATEN International product
0x7000" rev 2.00/0.00 addr 2
uhidev0 at uhub1 port 1 configuration 1 interface 0 "ATEN International product
0x2419" rev 1.10/1.00 addr 3
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub1 port 1 configuration 1 interface 1 "ATEN International product
0x2419" rev 1.10/1.00 addr 3
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
uhidev2 at uhub0 port 11 configuration 1 interface 0 "Dell Dell USB Keyboard"
rev 1.10/3.52 addr 4
uhidev2: iclass 3/1
ukbd1 at uhidev2: 8 variable keys, 6 key codes
wskbd1 at ukbd1 mux 1
wskbd1: connecting to wsdisplay0
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006>
sd2: 476937MB, 512 bytes/sector, 976767473 sectors
root on sd2a (9e550c811878345c.a) swap on sd2b dump on sd2b
carp52: state transition: BACKUP -> MASTER
carp302: state transition: BACKUP -> MASTER
carp99: state transition: BACKUP -> MASTER
carp56: state transition: BACKUP -> MASTER
carp48: state transition: BACKUP -> MASTER
carp303: state transition: BACKUP -> MASTER
carp22: state transition: BACKUP -> MASTER
carp20: state transition: BACKUP -> MASTER
carp17: state transition: BACKUP -> MASTER
carp17: state transition: MASTER -> BACKUP
carp20: state transition: MASTER -> BACKUP
carp22: state transition: MASTER -> BACKUP
carp302: state transition: MASTER -> BACKUP
carp303: state transition: MASTER -> BACKUP
carp48: state transition: MASTER -> BACKUP
carp120: state transition: BACKUP -> MASTER
carp52: state transition: MASTER -> BACKUP
carp99: state transition: MASTER -> BACKUP
carp56: state transition: MASTER -> BACKUP
