We're current running a 1U platform (dmesg below) that currentl serves
as our network gateway to a 3Gbps fiber based circuit. After operating
for a number of hours the ix0 device becomes inaccessible and when a
ping/traceroute/... command is attempted from the device through the
external network interface facing the Internet ie ix0 the following
error occurs
# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendmsg: No buffer space available
ping: wrote 8.8.8.8 64 chars, ret=-1
ping: sendmsg: No buffer space available
ping: wrote 8.8.8.8 64 chars, ret=-1
a netstat -nI for the both ix0(external) and em0(internal) interface
produces the following
Name Mtu Network Address Ipkts Ifail Opkts
Ofail Colls
ix0 1500 198.64.189. 198.64.189.200 24396906 0 13619148
906752 0
em0 1500 10.10.210/24 10.10.210.50 15138143 0 25142717
285044 0
Aslo while the servcies are inoperable nestat -m produces the following
$ netstat -m
2544 mbufs in use:
2461 mbufs allocated to data
1 mbuf allocated to packet headers
82 mbufs allocated to socket names and addresses
16/88 mbuf 2048 byte clusters in use (current/peak)
2315/2415 mbuf 2112 byte clusters in use (current/peak)
0/56 mbuf 4096 byte clusters in use (current/peak)
0/32 mbuf 8192 byte clusters in use (current/peak)
0/0 mbuf 9216 byte clusters in use (current/peak)
0/0 mbuf 12288 byte clusters in use (current/peak)
0/0 mbuf 16384 byte clusters in use (current/peak)
0/0 mbuf 65536 byte clusters in use (current/peak)
6476/6476/524288 Kbytes allocated to network (current/peak/max)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
A reboot clears things up and the services will then work for a period
of time. Although a pfctl -F all( or some derivative of the modifier)
allows the device to once again pass traffic all of its network
functionality does not return in that certain network requests such as
pkg_add are not successfully processed.
While snagging traffic on the ix0 interface while the device in
malfunctioning noticed a response from the one-hop away/default gateway
for the services making an arp request, namely
tcpdump: listening on ix0, link-type EN10MB
20:28:13.811695 arp who-has 198.64.189.200 tell 198.64.189.199
20:28:14.611323 arp who-has 198.64.189.200 tell 198.64.189.199
the following results are produced from
# sysctl | grep buf
kern.msgbufsize=131032
kern.malloc.kmemnames=free,,devbuf,,pcb,rtable,,,,ifaddr,,sysctl,counters,,ioctlops,,,,,iov,mount,,NFS_req,NFS_mount,log,vnodes,,UFS_quota,UFS_mount,shm,VM_map,sem,dirhash,ACPI,VM_pmap,,,,,file_desc,sigio,proc,subproc,,,,MFS_node,,,Export_Host,NFS_srvsock,,NFS_daemon,ip_moptions,in_multi,ether_multi,mrt,ISOFS_mount,ISOFS_node,MSDOSFS_mount,MSDOSFS_fat,MSDOSFS_node,ttys,exec,miscfs_mount,fusefs_mount,,,,,,,,,pfkey_data,tdb,xform_data,,pagedep,inodedep,newblk,,,indirdep,,,,,,,,,VM_swap,,,,,,UVM_amap,UVM_aobj,,USB,USB_device,USB_HC,witness,memdesc,,,crypto_data,,IPsec_creds,,,,,,,,,,,,,ip6_options,NDP,,,temp,NTFS_mount,NTFS_node,NTFS_fnode,NTFS_dir,NTFS_hash,NTFS_attr,NTFS_data,NTFS_decomp,NTFS_vrun,kqueue,,SYN_cache,UDF_mount,UDF_file_entry,UDF_file_id,,AGP_Memory,DRM
kern.malloc.kmemstat.devbuf=(inuse = 6179, calls = 7341, memuse = 8537K,
limblocks = 0, maxused = 8602K, limit = 78644K, spare = 0, sizes =
(16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072,262144))
kern.bufcachepercent=20
kern.consbufsize=16344
net.bpf.bufsize=32768
net.bpf.maxbufsize=2097152
vfs.fuse.fusefs_fbufs_in=0
vfs.fuse.fusefs_fbufs_wait=0
We utilize similar platform with similar deployments (1Gbps Internet
circuits) and have not seen this issue previously.
sAny thoughts on the culprit of the buffer exhaustion?
---
Boyd Stephens
dmesg:
OpenBSD 7.1 (GENERIC.MP) #3: Sun May 15 10:27:01 MDT 2022
r...@syspatch-71-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 17058750464 (16268MB)
avail mem = 16524451840 (15758MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.2 @ 0x8ec16000 (47 entries)
bios0: vendor American Megatrends Inc. version "1.5" date 10/05/2020
bios0: Supermicro Super Server
acpi0 at bios0: ACPI 6.1
acpi0: sleep states S0 S5
acpi0: tables DSDT FACP APIC FPDT FIDT MCFG SPMI SSDT SSDT SSDT HPET
SSDT SSDT NHLT UEFI LPIT SSDT SSDT DBGP DBG2 SSDT DMAR SSDT WSMT EINJ
ERST BERT HEST SSDT
acpi0: wakeup devices PEG0(S0) PEGP(S0) PEG1(S0) PEGP(S0) PEG2(S0)
PEGP(S0) RP01(S0) PXSX(S0) RP02(S0) PXSX(S0) RP03(S0) PXSX(S0) RP04(S0)
PXSX(S0) RP05(S0) PXSX(S0) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) E-2126G CPU @ 3.30GHz, 3292.33 MHz, 06-9e-0a
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x
2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA
,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 24MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) E-2126G CPU @ 3.30GHz, 3292.33 MHz, 06-9e-0a
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x
2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA
,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) E-2126G CPU @ 3.30GHz, 3292.34 MHz, 06-9e-0a
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x
2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA
,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) E-2126G CPU @ 3.30GHz, 3292.34 MHz, 06-9e-0a
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x
2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA
,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 8 (application processor)
cpu4: Intel(R) Xeon(R) E-2126G CPU @ 3.30GHz, 3292.34 MHz, 06-9e-0a
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x
2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA
,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 0, core 4, package 0
cpu5 at mainbus0: apid 10 (application processor)
cpu5: Intel(R) Xeon(R) E-2126G CPU @ 3.30GHz, 3292.34 MHz, 06-9e-0a
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x
2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA
,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: smt 0, core 5, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 120 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 23999999 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEG0)
acpiprt2 at acpi0: bus -1 (PEG1)
acpiprt3 at acpi0: bus -1 (PEG2)
acpiprt4 at acpi0: bus 4 (RP01)
acpiprt5 at acpi0: bus 5 (RP02)
acpiprt6 at acpi0: bus -1 (RP03)
acpiprt7 at acpi0: bus -1 (RP04)
acpiprt8 at acpi0: bus -1 (RP05)
acpiprt9 at acpi0: bus -1 (RP06)
acpiprt10 at acpi0: bus -1 (RP07)
acpiprt11 at acpi0: bus -1 (RP08)
acpiprt12 at acpi0: bus -1 (RP09)
acpiprt13 at acpi0: bus -1 (RP10)
acpiprt14 at acpi0: bus -1 (RP11)
acpiprt15 at acpi0: bus -1 (RP12)
acpiprt16 at acpi0: bus -1 (RP13)
acpiprt17 at acpi0: bus -1 (RP14)
acpiprt18 at acpi0: bus -1 (RP15)
acpiprt19 at acpi0: bus -1 (RP16)
acpiprt20 at acpi0: bus -1 (RP18)
acpiprt21 at acpi0: bus -1 (RP19)
acpiprt22 at acpi0: bus -1 (RP20)
acpiprt23 at acpi0: bus -1 (RP21)
acpiprt24 at acpi0: bus 3 (RP23)
acpiprt25 at acpi0: bus -1 (RP24)
acpiprt26 at acpi0: bus -1 (RP17)
acpiec0 at acpi0: not present
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
com0 at acpi0 UAR1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
com0: console
"IPI0001" at acpi0 not configured
pchgpio0 at acpi0 GPI0 addr 0xfd6e0000/0x10000 0xfd6d0000/0x10000
0xfd6b0000/0x10000 0xfd6a0000/0x10000 irq 14, 384 pins
"ACPI000E" at acpi0 not configured
acpibtn0 at acpi0: SLPB
"PNP0C14" at acpi0 not configured
"PNP0C14" at acpi0 not configured
"INT33A1" at acpi0 not configured
"PNP0C33" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
acpipwrres0 at acpi0: USBC, resource for XDCI
acpipwrres1 at acpi0: V0PR
acpipwrres2 at acpi0: V1PR
acpipwrres3 at acpi0: V2PR
acpipwrres4 at acpi0: WRST
acpicpu0 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpicpu1 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpicpu2 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpicpu3 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpicpu4 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpicpu5 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpipwrres5 at acpi0: FN00, resource for FAN0
acpipwrres6 at acpi0: FN01, resource for FAN1
acpipwrres7 at acpi0: FN02, resource for FAN2
acpipwrres8 at acpi0: FN03, resource for FAN3
acpipwrres9 at acpi0: FN04, resource for FAN4
acpitz0 at acpi0: critical temperature is 119 degC
acpipwrres10 at acpi0: PIN_
acpivideo0 at acpi0: GFX0
acpivout0 at acpivideo0: DD1F
ipmi at mainbus0 not configured
cpu0: using VERW MDS workaround (except on vmm entry)
cpu0: Enhanced SpeedStep 3292 MHz: speeds: 3301, 3300, 3100, 2900, 2800,
2600, 2400, 2200, 2000, 1900, 1700, 1500, 1300, 1200, 1000, 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Core 8G Host" rev 0x07
ppb0 at pci0 dev 1 function 0 "Intel Core 6G PCIE" rev 0x07: msi
pci1 at ppb0 bus 1
ix0 at pci1 dev 0 function 0 "Intel 82599" rev 0x01, msix, 6 queues,
address 0c:c4:7a:87:21:c4
ix1 at pci1 dev 0 function 1 "Intel 82599" rev 0x01, msix, 6 queues,
address 0c:c4:7a:87:21:c5
"Intel Core GMM" rev 0x00 at pci0 dev 8 function 0 not configured
pchtemp0 at pci0 dev 18 function 0 "Intel 300 Series Thermal" rev 0x10
xhci0 at pci0 dev 20 function 0 "Intel 300 Series xHCI" rev 0x10: msi,
xHCI 1.10
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev
3.00/1.00 addr 1
"Intel 300 Series Shared SRAM" rev 0x10 at pci0 dev 20 function 2 not
configured
dwiic0 at pci0 dev 21 function 0 "Intel 300 Series I2C" rev 0x10: apic 2
int 16
iic0 at dwiic0
dwiic1 at pci0 dev 21 function 1 "Intel 300 Series I2C" rev 0x10: apic 2
int 17
iic1 at dwiic1
"Intel 300 Series HECI" rev 0x10 at pci0 dev 22 function 0 not configured
"Intel 300 Series HECI" rev 0x10 at pci0 dev 22 function 4 not configured
ahci0 at pci0 dev 23 function 0 "Intel 300 Series AHCI" rev 0x10: msi,
AHCI 1.3.1
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, PNY CS900 250GB, CS90>
naa.5f8db4c2133011dd
sd0: 238475MB, 512 bytes/sector, 488397168 sectors, thin
ppb1 at pci0 dev 27 function 0 "Intel 300 Series PCIE" rev 0xf0: msi
pci2 at ppb1 bus 2
em0 at pci2 dev 0 function 0 "Intel I210" rev 0x03: msi, address
3c:ec:ef:f0:f6:ee
ppb2 at pci0 dev 27 function 6 "Intel 300 Series PCIE" rev 0xf0: msi
pci3 at ppb2 bus 3
em1 at pci3 dev 0 function 0 "Intel I210" rev 0x03: msi, address
3c:ec:ef:f0:f6:ef
ppb3 at pci0 dev 28 function 0 "Intel 300 Series PCIE" rev 0xf0: msi
pci4 at ppb3 bus 4
ppb4 at pci0 dev 28 function 1 "Intel 300 Series PCIE" rev 0xf0: msi
pci5 at ppb4 bus 5
ppb5 at pci5 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x04
pci6 at ppb5 bus 6
vga1 at pci6 dev 0 function 0 "ASPEED Technology AST2000" rev 0x41
wsdisplay at vga1 not configured
"Intel 300 Series UART" rev 0x10 at pci0 dev 30 function 0 not configured
pcib0 at pci0 dev 31 function 0 "Intel C242 LPC" rev 0x10
ichiic0 at pci0 dev 31 function 4 "Intel 300 Series SMBus" rev 0x10:
apic 2 int 16
iic2 at ichiic0
"Intel 300 Series SPI" rev 0x10 at pci0 dev 31 function 5 not configured
isa0 at pcib0
isadma0 at isa0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT
uhidev0 at uhub0 port 1 configuration 1 interface 0 "American Power
Conversion Smart-UPS 1500 FW:UPS 09.3 / ID=18" rev 2.00/1.06 addr 2
uhidev0: iclass 3/0, 146 report ids
upd0 at uhidev0
uhid0 at uhidev0 reportid 1: input=0, output=0, feature=1
uhid1 at uhidev0 reportid 2: input=0, output=0, feature=1
uhid2 at uhidev0 reportid 3: input=0, output=0, feature=1
uhid3 at uhidev0 reportid 4: input=0, output=0, feature=1
uhid4 at uhidev0 reportid 5: input=0, output=0, feature=1
uhid5 at uhidev0 reportid 6: input=0, output=0, feature=2
uhid6 at uhidev0 reportid 8: input=2, output=0, feature=2
uhid7 at uhidev0 reportid 9: input=0, output=0, feature=2
uhid8 at uhidev0 reportid 10: input=0, output=0, feature=2
uhid9 at uhidev0 reportid 11: input=0, output=0, feature=2
uhid10 at uhidev0 reportid 12: input=1, output=0, feature=1
uhid11 at uhidev0 reportid 13: input=2, output=0, feature=2
uhid12 at uhidev0 reportid 14: input=0, output=0, feature=2
uhid13 at uhidev0 reportid 15: input=0, output=0, feature=1
uhid14 at uhidev0 reportid 16: input=0, output=0, feature=2
uhid15 at uhidev0 reportid 17: input=0, output=0, feature=1
uhid16 at uhidev0 reportid 18: input=0, output=0, feature=2
uhid17 at uhidev0 reportid 19: input=0, output=0, feature=3
uhid18 at uhidev0 reportid 20: input=1, output=0, feature=1
uhid19 at uhidev0 reportid 137: input=63, output=0, feature=0
uhid20 at uhidev0 reportid 141: input=0, output=0, feature=2
uhid21 at uhidev0 reportid 142: input=0, output=0, feature=2
uhid22 at uhidev0 reportid 144: input=0, output=63, feature=0
uhid23 at uhidev0 reportid 146: input=0, output=0, feature=2
uhub1 at uhub0 port 14 configuration 1 interface 0 "ATEN International
product 0x7000" rev 2.00/0.00 addr 3
uhidev1 at uhub1 port 1 configuration 1 interface 0 "ATEN International
product 0x2419" rev 1.10/1.00 addr 4
uhidev1: iclass 3/1
ukbd0 at uhidev1: 8 variable keys, 6 key codes
wskbd0 at ukbd0 mux 1
uhidev2 at uhub1 port 1 configuration 1 interface 1 "ATEN International
product 0x2419" rev 1.10/1.00 addr 4
uhidev2: iclass 3/1
ums0 at uhidev2: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (97229a9c59cddc5c.a) swap on sd0b dump on sd0b
