Hi!
I've recently up'd my processor count on our poudriere box and have
started noticing the error
"witness_lock_list_get: witness exhausted" on the console. The kernel
*DOES NOT* crash but I
thought the report may be useful to someone.
$ uname -a
FreeBSD poudriere 12.0-CURRENT FreeBSD 12.0-CURRENT #1 r325999: Sun Nov
19 18:41:20 EST 2017
mikej@poudriere:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
The machine is pretty busy running four poudriere build instances.
last pid: 76584; load averages: 115.07, 115.96, 98.30
up 6+07:32:59 14:44:03
763 processes: 117 running, 581 sleeping, 2 zombie, 63 lock
CPU: 59.0% user, 0.0% nice, 40.7% system, 0.1% interrupt, 0.1% idle
Mem: 12G Active, 2003M Inact, 44G Wired, 29G Free
ARC: 28G Total, 11G MFU, 16G MRU, 122M Anon, 359M Header, 1184M Other
25G Compressed, 32G Uncompressed, 1.24:1 Ratio
Let me know what additional information I might supply.
Thanks!
--mikej
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #1 r325999: Sun Nov 19 18:41:20 EST 2017
mikej@poudriere:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 5.0.0 (tags/RELEASE_500/final 312559) (based on LLVM
5.0.0svn)
WARNING: WITNESS option enabled, expect reduced performance.
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU E7- 4850 @ 2.00GHz (1995.00-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x206f2 Family=0x6 Model=0x2f Stepping=2
Features=0x1fa3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,HTT>
Features2=0x83b82203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,HV>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
Structured Extended Features=0x2<TSCADJ>
TSC: P-state invariant
Hypervisor: Origin = "VMwareVMware"
real memory = 96636764160 (92160 MB)
avail memory = 93873786880 (89525 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <PTLTD APIC >
FreeBSD/SMP: Multiprocessor System Detected: 60 CPUs
FreeBSD/SMP: 6 package(s) x 10 core(s)
random: unblocking device.
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
SMP: AP CPU #46 Launched!
SMP: AP CPU #16 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #51 Launched!
SMP: AP CPU #20 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #48 Launched!
SMP: AP CPU #23 Launched!
SMP: AP CPU #27 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #55 Launched!
SMP: AP CPU #17 Launched!
SMP: AP CPU #52 Launched!
SMP: AP CPU #18 Launched!
SMP: AP CPU #39 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #41 Launched!
SMP: AP CPU #58 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #26 Launched!
SMP: AP CPU #37 Launched!
SMP: AP CPU #45 Launched!
SMP: AP CPU #40 Launched!
SMP: AP CPU #35 Launched!
SMP: AP CPU #53 Launched!
SMP: AP CPU #50 Launched!
SMP: AP CPU #44 Launched!
SMP: AP CPU #34 Launched!
SMP: AP CPU #30 Launched!
SMP: AP CPU #31 Launched!
SMP: AP CPU #47 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #28 Launched!
SMP: AP CPU #49 Launched!
SMP: AP CPU #54 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #57 Launched!
SMP: AP CPU #29 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #24 Launched!
SMP: AP CPU #33 Launched!
SMP: AP CPU #38 Launched!
SMP: AP CPU #59 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #42 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #43 Launched!
SMP: AP CPU #14 Launched!
SMP: AP CPU #19 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #25 Launched!
SMP: AP CPU #56 Launched!
SMP: AP CPU #36 Launched!
SMP: AP CPU #22 Launched!
SMP: AP CPU #32 Launched!
SMP: AP CPU #21 Launched!
random: entropy device external interface
netmap: loaded module
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0xffffffff80f95c20, 0) error 19
kbd1 at kbdmux0
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <INTEL 440BX> on motherboard
acpi0: Power Button (fixed)
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
cpu0: <ACPI CPU> numa-domain 0 on acpi0
cpu1: <ACPI CPU> numa-domain 5 on acpi0
cpu2: <ACPI CPU> numa-domain 5 on acpi0
cpu3: <ACPI CPU> numa-domain 5 on acpi0
cpu4: <ACPI CPU> numa-domain 5 on acpi0
cpu5: <ACPI CPU> numa-domain 5 on acpi0
cpu6: <ACPI CPU> numa-domain 5 on acpi0
cpu7: <ACPI CPU> numa-domain 5 on acpi0
cpu8: <ACPI CPU> numa-domain 5 on acpi0
cpu9: <ACPI CPU> numa-domain 5 on acpi0
cpu10: <ACPI CPU> numa-domain 5 on acpi0
cpu11: <ACPI CPU> numa-domain 4 on acpi0
cpu12: <ACPI CPU> numa-domain 4 on acpi0
cpu13: <ACPI CPU> numa-domain 4 on acpi0
cpu14: <ACPI CPU> numa-domain 4 on acpi0
cpu15: <ACPI CPU> numa-domain 4 on acpi0
cpu16: <ACPI CPU> numa-domain 4 on acpi0
cpu17: <ACPI CPU> numa-domain 4 on acpi0
cpu18: <ACPI CPU> numa-domain 4 on acpi0
cpu19: <ACPI CPU> numa-domain 4 on acpi0
cpu20: <ACPI CPU> numa-domain 4 on acpi0
cpu21: <ACPI CPU> numa-domain 3 on acpi0
cpu22: <ACPI CPU> numa-domain 3 on acpi0
cpu23: <ACPI CPU> numa-domain 3 on acpi0
cpu24: <ACPI CPU> numa-domain 3 on acpi0
cpu25: <ACPI CPU> numa-domain 3 on acpi0
cpu26: <ACPI CPU> numa-domain 3 on acpi0
cpu27: <ACPI CPU> numa-domain 3 on acpi0
cpu28: <ACPI CPU> numa-domain 3 on acpi0
cpu29: <ACPI CPU> numa-domain 3 on acpi0
cpu30: <ACPI CPU> numa-domain 3 on acpi0
cpu31: <ACPI CPU> numa-domain 2 on acpi0
cpu32: <ACPI CPU> numa-domain 2 on acpi0
cpu33: <ACPI CPU> numa-domain 2 on acpi0
cpu34: <ACPI CPU> numa-domain 2 on acpi0
cpu35: <ACPI CPU> numa-domain 2 on acpi0
cpu36: <ACPI CPU> numa-domain 2 on acpi0
cpu37: <ACPI CPU> numa-domain 2 on acpi0
cpu38: <ACPI CPU> numa-domain 2 on acpi0
cpu39: <ACPI CPU> numa-domain 2 on acpi0
cpu40: <ACPI CPU> numa-domain 2 on acpi0
cpu41: <ACPI CPU> numa-domain 1 on acpi0
cpu42: <ACPI CPU> numa-domain 1 on acpi0
cpu43: <ACPI CPU> numa-domain 1 on acpi0
cpu44: <ACPI CPU> numa-domain 1 on acpi0
cpu45: <ACPI CPU> numa-domain 1 on acpi0
cpu46: <ACPI CPU> numa-domain 1 on acpi0
cpu47: <ACPI CPU> numa-domain 1 on acpi0
cpu48: <ACPI CPU> numa-domain 1 on acpi0
cpu49: <ACPI CPU> numa-domain 1 on acpi0
cpu50: <ACPI CPU> numa-domain 1 on acpi0
cpu51: <ACPI CPU> numa-domain 0 on acpi0
cpu52: <ACPI CPU> numa-domain 0 on acpi0
cpu53: <ACPI CPU> numa-domain 0 on acpi0
cpu54: <ACPI CPU> numa-domain 0 on acpi0
cpu55: <ACPI CPU> numa-domain 0 on acpi0
cpu56: <ACPI CPU> numa-domain 0 on acpi0
cpu57: <ACPI CPU> numa-domain 0 on acpi0
cpu58: <ACPI CPU> numa-domain 0 on acpi0
cpu59: <ACPI CPU> numa-domain 0 on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 7.3 (no driver attached)
vgapci0: <VGA-compatible display> port 0x1070-0x107f mem
0xe8000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0
vgapci0: Boot video device
pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
pcib3: [GIANT-LOCKED]
pci3: <ACPI PCI bus> on pcib3
mpt0: <LSILogic SAS/SATA Adapter> port 0x4000-0x40ff mem
0xfd5ec000-0xfd5effff,0xfd5f0000-0xfd5fffff irq 18 at device 0.0 on pci3
mpt0: MPI Version=1.5.0.0
pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
pcib4: [GIANT-LOCKED]
pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0
pcib5: [GIANT-LOCKED]
pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0
pcib6: [GIANT-LOCKED]
pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0
pcib7: [GIANT-LOCKED]
pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0
pcib8: [GIANT-LOCKED]
pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0
pcib9: [GIANT-LOCKED]
pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0
pcib10: [GIANT-LOCKED]
pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0
pcib11: [GIANT-LOCKED]
pci4: <ACPI PCI bus> on pcib11
vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x5000-0x500f mem
0xfd4fc000-0xfd4fcfff,0xfd4fd000-0xfd4fdfff,0xfd4fe000-0xfd4fffff irq 19 at
device 0.0 on pci4
vmx0: Ethernet address: 00:50:56:90:0e:51
pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0
pcib12: [GIANT-LOCKED]
pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0
pcib13: [GIANT-LOCKED]
pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0
pcib14: [GIANT-LOCKED]
pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0
pcib15: [GIANT-LOCKED]
pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0
pcib16: [GIANT-LOCKED]
pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0
pcib17: [GIANT-LOCKED]
pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0
pcib18: [GIANT-LOCKED]
pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0
pcib19: [GIANT-LOCKED]
pci5: <ACPI PCI bus> on pcib19
vmx1: <VMware VMXNET3 Ethernet Adapter> port 0x6000-0x600f mem
0xfd3fc000-0xfd3fcfff,0xfd3fd000-0xfd3fdfff,0xfd3fe000-0xfd3fffff irq 16 at
device 0.0 on pci5
vmx1: Ethernet address: 00:50:56:90:cb:41
pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0
pcib20: [GIANT-LOCKED]
pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0
pcib21: [GIANT-LOCKED]
pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0
pcib22: [GIANT-LOCKED]
pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0
pcib23: [GIANT-LOCKED]
pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0
pcib24: [GIANT-LOCKED]
pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0
pcib25: [GIANT-LOCKED]
pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0
pcib26: [GIANT-LOCKED]
pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0
pcib27: [GIANT-LOCKED]
pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0
pcib28: [GIANT-LOCKED]
pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0
pcib29: [GIANT-LOCKED]
pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0
pcib30: [GIANT-LOCKED]
pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0
pcib31: [GIANT-LOCKED]
pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0
pcib32: [GIANT-LOCKED]
pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0
pcib33: [GIANT-LOCKED]
pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0
pcib34: [GIANT-LOCKED]
acpi_acad0: <AC Adapter> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse, device ID 3
acpi_syscontainer0: <System Container> on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
orm0: <ISA Option ROMs> at iomem
0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcafff,0xcb000-0xcbfff,0xdc000-0xdffff,0xe0000-0xe7fff
on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 10.000 msec
usb_needs_explore_all: no devclass
da0 at mpt0 bus 0 scbus2 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 409600MB (838860800 512 byte sectors)
da0: quirks=0x140<RETRY_BUSY,STRICT_UNMAP>
da1 at mpt0 bus 0 scbus2 target 1 lun 0
da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 131072MB (268435456 512 byte sectors)
da1: quirks=0x140<RETRY_BUSY,STRICT_UNMAP>
cd0 at ata1 bus 0 scbus1 target 0 lun 0
cd0: <NECVMWar VMware IDE CDR10 1.00> Removable CD-ROM SCSI device
cd0: Serial Number 10000000000000000001
cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt
cd0: quirks=0x40<RETRY_BUSY>
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from zfs:zroot/ROOT/default []...
VMware memory control driver initialized
vmx0: link state changed to UP
vmx1: link state changed to UP
lock order reversal:
1st 0xfffff801eb8ec7c8 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:852
2nd 0xfffff801c49699a0 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2606
stack backtrace:
#0 0xffffffff80acf933 at witness_debugger+0x73
#1 0xffffffff80acf7b2 at witness_checkorder+0xe02
#2 0xffffffff80a41abe at lockmgr_lock_fast_path+0x1ae
#3 0xffffffff81094309 at VOP_LOCK1_APV+0xd9
#4 0xffffffff80b4ab66 at _vn_lock+0x66
#5 0xffffffff80b395d2 at vget+0x82
#6 0xffffffff80936fdd at devfs_allocv+0xcd
#7 0xffffffff80936ae3 at devfs_root+0x43
#8 0xffffffff80b30144 at vfs_donmount+0x11f4
#9 0xffffffff80b2ef22 at sys_nmount+0x72
#10 0xffffffff80f16d2b at amd64_syscall+0x79b
#11 0xffffffff80ef5aab at Xfast_syscall+0xfb
lock order reversal:
1st 0xfffff803b7d235f0 syncer (syncer) @ /usr/src/sys/kern/vfs_subr.c:2156
2nd 0xfffff802dbf88240 zfs (zfs) @ /usr/src/sys/kern/vfs_subr.c:2606
stack backtrace:
#0 0xffffffff80acf933 at witness_debugger+0x73
#1 0xffffffff80acf7b2 at witness_checkorder+0xe02
#2 0xffffffff80a41abe at lockmgr_lock_fast_path+0x1ae
#3 0xffffffff81094309 at VOP_LOCK1_APV+0xd9
#4 0xffffffff80b4ab66 at _vn_lock+0x66
#5 0xffffffff80b395d2 at vget+0x82
#6 0xffffffff80b3b656 at vfs_msync+0xa6
#7 0xffffffff80b40366 at sync_fsync+0xc6
#8 0xffffffff81093279 at VOP_FSYNC_APV+0xd9
#9 0xffffffff80b3e314 at sched_sync+0x3c4
#10 0xffffffff80a2d584 at fork_exit+0x84
#11 0xffffffff80ef5d3e at fork_trampoline+0xe
lock order reversal:
1st 0xfffff8154354b8e0 filedesc structure (filedesc structure) @
/usr/src/sys/kern/sys_generic.c:1565
2nd 0xfffff8037c406418 devfs (devfs) @ /usr/src/sys/kern/vfs_vnops.c:1524
stack backtrace:
#0 0xffffffff80acf933 at witness_debugger+0x73
#1 0xffffffff80acf7b2 at witness_checkorder+0xe02
#2 0xffffffff80a41abe at lockmgr_lock_fast_path+0x1ae
#3 0xffffffff81094309 at VOP_LOCK1_APV+0xd9
#4 0xffffffff80b4ab66 at _vn_lock+0x66
#5 0xffffffff80b4996b at vn_poll+0x3b
#6 0xffffffff8093a27d at devfs_poll_f+0xcd
#7 0xffffffff80ad68d5 at kern_poll+0x385
#8 0xffffffff80ad6540 at sys_poll+0x50
#9 0xffffffff80f16d2b at amd64_syscall+0x79b
#10 0xffffffff80ef5aab at Xfast_syscall+0xfb
witness_lock_list_get: witness exhausted
_______________________________________________
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
