>Synopsis: panic on vm hosted with VMM - missing dquot
>Category: unknown
>Environment:
System : OpenBSD 6.6
Details : OpenBSD 6.6-current (GENERIC) #85: Sun Mar 29
10:50:36 MDT 2020
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
Architecture: OpenBSD.amd64
Machine : amd64
>Description: I noticed my VM was unresponsive, and upon accessing via
>console, received the ddb prompt.
>How-To-Repeat: unknown
>Fix: unknown
server9$ vmctl console 16
Connected to /dev/ttypg (speed 115200)
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb> trace
db_enter() at db_enter+0x10
panic(ffffffff81c52ce1) at panic+0x128
chkdquot(fffffd80363fa968) at chkdquot+0xac
ufs_quota_alloc_blocks2(fffffd80363fa968,20,fffffd801b174918,0) at ufs_quota_al
loc_blocks2+0x2f
ffs_alloc(fffffd80363fa968,23,230648,4000,fffffd801b174918,ffff8000149cd458) at
ffs_alloc+0x11e
ffs1_balloc(fffffd80363fa968,8c000,50,fffffd801b174918,1,ffff8000149cd550) at f
fs1_balloc+0xc46
ffs_write(ffff8000149cd5c0) at ffs_write+0x262
VOP_WRITE(fffffd803aff95c0,ffff8000149cd650,3,fffffd801b174918) at VOP_WRITE+0x
4f
ktrwriteraw(ffff8000149ce4f0,fffffd803aff95c0,fffffd801b174918,ffff8000149cd710
,ffff8000149cd6f0) at ktrwriteraw+0xf3
ktrsyscall(ffff8000149ce4f0,3,18,ffff8000149cd7d0) at ktrsyscall+0x20c
syscall(ffff8000149cd8a0) at syscall+0x1e8
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7ffffd6300, count: -12
ddb>
ddb> ps
PID TID PPID UID S FLAGS WAIT COMMAND
81456 209606 92191 83 3 0x100092 poll ntpd
92191 250231 77184 83 3 0x100092 poll ntpd
77184 191748 1 0 3 0x100080 poll ntpd
23959 413905 25382 1000 3 0x100083 ttyin ksh
25382 436054 23023 1000 3 0x90 select sshd
23023 62155 12842 0 3 0x92 poll sshd
6671 55545 1 0 3 0x80 kqread relayd
62186 10628 1 89 3 0x100092 kqread relayd
1993 7611 1 89 3 0x100092 kqread relayd
60621 523468 1 89 3 0x100092 kqread relayd
89874 514298 1 89 3 0x100092 kqread relayd
24726 348586 1 89 3 0x100092 kqread relayd
91422 445911 1 89 3 0x100092 kqread relayd
92658 252289 1 89 3 0x100092 kqread relayd
88052 189012 1 89 3 0x100092 kqread relayd
40313 102543 53424 1001 3 0x83 thrsleep pic-uploader
40313 224510 53424 1001 3 0x4000083 thrsleep pic-uploader
40313 329647 53424 1001 3 0x4000083 kqread pic-uploader
40313 155810 53424 1001 3 0x4000083 thrsleep pic-uploader
40313 39956 53424 1001 3 0x4000083 thrsleep pic-uploader
53424 221662 1 1000 3 0x80 select screen
37797 358849 0 0 3 0x14200 bored sosplice
39784 174006 57747 1000 3 0x100083 ttyin ksh
57747 80274 96976 1000 3 0x90 select sshd
96976 413266 12842 0 3 0x92 poll sshd
58019 465341 1 0 3 0x100083 ttyin getty
62694 507888 1 0 3 0x100098 poll cron
72159 298694 1 0 3 0x100080 kqread httpd
79097 358617 1 67 3 0x100092 kqread httpd
40001 382867 1 67 3 0x100092 kqread httpd
70306 368045 1 67 3 0x100092 kqread httpd
34723 64169 1 67 3 0x100092 kqread httpd
49595 481674 27251 95 3 0x100092 kqread smtpd
62245 261196 27251 103 3 0x100092 kqread smtpd
78278 138538 27251 95 3 0x100092 kqread smtpd
92404 102960 27251 95 3 0x100092 kqread smtpd
90460 303274 27251 95 3 0x100092 kqread smtpd
87323 91411 27251 95 3 0x100092 kqread smtpd
27251 77183 1 0 3 0x100080 kqread smtpd
12842 81499 1 0 3 0x80 select sshd
*96640 485954 51034 74 7 0x100013 pflogd
51034 235286 1 0 3 0x80 netio pflogd
60945 148844 2475 73 3 0x100090 kqread syslogd
2475 233565 1 0 3 0x100082 netio syslogd
13924 80282 1 77 3 0x100090 poll dhclient
93521 254355 1 0 3 0x80 poll dhclient
83747 389586 76706 115 3 0x100092 kqread slaacd
33094 326559 76706 115 3 0x100092 kqread slaacd
76706 480047 1 0 3 0x100080 kqread slaacd
58959 410884 0 0 3 0x14200 bored smr
78739 126850 0 0 3 0x14200 pgzero zerothread
30038 111777 0 0 3 0x14200 aiodoned aiodoned
17416 51642 0 0 3 0x14200 syncer update
7166 490681 0 0 3 0x14200 cleaner cleaner
4779 258210 0 0 3 0x14200 reaper reaper
29760 392847 0 0 3 0x14200 pgdaemon pagedaemon
57537 337303 0 0 3 0x14200 bored crynlk
76922 414222 0 0 3 0x14200 bored crypto
82654 92795 0 0 2 0x14200 softnet
15571 52197 0 0 3 0x14200 bored systqmp
13208 471867 0 0 3 0x14200 bored systq
22489 256853 0 0 3 0x40014200 bored softclock
21807 350209 0 0 3 0x40014200 idle0
1 476317 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb>
ddb> show uvm
Current UVM status:
pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
249258 VM pages: 12673 active, 16970 inactive, 0 wired, 159229 free (19918 ze
ro)
min 10% (25) anon, 10% (25) vnode, 5% (12) vtext
freemin=8308, free-target=11077, inactive-target=0, wired-max=83086
faults=3063811, traps=3125241, intrs=10024337, ctxswitch=2548842 fpuswitch=0
softint=4716945, syscalls=30420084, kmapent=12
fault counts:
noram=0, noanon=0, noamap=0, pgwait=0, pgrele=0
ok relocks(total)=116604(116604), anget(retries)=1516737(0), amapcopy=12358
13
neighbor anon/obj pg=213748/2421323, gets(lock/unlock)=799993/116604
cases: anon=1185884, anoncow=330853, obj=700369, prcopy=99624, przero=74708
0
daemon and swap counts:
woke=0, revs=0, scans=0, obscans=0, anscans=0
busy=0, freed=0, reactivate=0, deactivate=0
pageouts=0, pending=0, nswget=0
nswapdev=1
swpages=323583, swpginuse=0, swpgonly=0 paging=0
kernel pointers:
objs(kern)=0xffffffff81f37dd8
ddb>
ddb> show bcstats
Current Buffer Cache status:
numbufs 12071 busymapped 2, delwri 8
kvaslots 3115 avail kva slots 3113
bufpages 48248, dmapages 48248, dirtypages 32
pendingreads 0, pendingwrites 0
highflips 0, highflops 0, dmaflips 0
ddb>
ddb> show panic
missing dquot
ddb>
dmesg:
OpenBSD 6.6-current (GENERIC) #85: Sun Mar 29 10:50:36 MDT 2020
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 1056956416 (1007MB)
avail mem = 1012465664 (965MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf3f40 (10 entries)
bios0: vendor SeaBIOS version "1.11.0p2-OpenBSD-vmm" date 01/01/2011
bios0: OpenBSD VMM
acpi at bios0 not configured
cpu0 at mainbus0: (uniprocessor)
cpu0: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz, 3110.77 MHz, 06-2c-02
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,HV,NXE,PAGE1GB,LONG,LAHF,ITSC,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
cpu0: using IvyBridge MDS workaround
pvbus0 at mainbus0: OpenBSD
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
viornd0 at virtio0
virtio0: irq 3
virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio1: address fe:e1:bb:d4:c9:16
virtio1: irq 5
virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio2
scsibus1 at vioblk0: 2 targets
sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 51200MB, 512 bytes/sector, 104857600 sectors
virtio2: irq 6
virtio3 at pci0 dev 4 function 0 "OpenBSD VMM Control" rev 0x00
vmmci0 at virtio3
virtio3: irq 7
isa0 at mainbus0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
com0: console
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (f10723e9888c49b1.a) swap on sd0b dump on sd0b
usbdevs:
usbdevs: no USB controllers found
pcidump:
Domain /dev/pci0:
0:0:0: OpenBSD VMM Host
0x0000: Vendor ID: 0b5d, Product ID: 0666
0x0004: Command: 0103, Status: 0000
0x0008: Class: 06 Bridge, Subclass: 00 Host,
Interface: 00, Revision: 00
0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
Cache Line Size: 00
0x0010: BAR empty (00000000)
0x0014: BAR empty (00000000)
0x0018: BAR empty (00000000)
0x001c: BAR empty (00000000)
0x0020: BAR empty (00000000)
0x0024: BAR empty (00000000)
0x0028: Cardbus CIS: 00000000
0x002c: Subsystem Vendor ID: 0b5d Product ID: 0000
0x0030: Expansion ROM Base Address: 00000000
0x0038: 00000000
0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
0x0000: 06660b5d 00000103 06000000 00000000
0x0010: 00000000 00000000 00000000 00000000
0x0020: 00000000 00000000 00000000 00000b5d
0x0030: 00000000 00000000 00000000 00000000
0x0040: 00000000 00000000 00000000 00000000
0x0050: 00000000 00000000 00000000 00000000
0x0060: 00000000 00000000 00000000 00000000
0x0070: 00000000 00000000 00000000 00000000
0x0080: 00000000 00000000 00000000 00000000
0x0090: 00000000 00000000 00000000 00000000
0x00a0: 00000000 00000000 00000000 00000000
0x00b0: 00000000 00000000 00000000 00000000
0x00c0: 00000000 00000000 00000000 00000000
0x00d0: 00000000 00000000 00000000 00000000
0x00e0: 00000000 00000000 00000000 00000000
0x00f0: 00000000 00000000 00000000 00000000
0:1:0: Qumranet Virtio RNG
0x0000: Vendor ID: 1af4, Product ID: 1005
0x0004: Command: 0107, Status: 0000
0x0008: Class: 08 System, Subclass: 80 Miscellaneous,
Interface: 00, Revision: 00
0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
Cache Line Size: 00
0x0010: BAR io addr: 0x00001000/0x1000
0x0014: BAR empty (00000000)
0x0018: BAR empty (00000000)
0x001c: BAR empty (00000000)
0x0020: BAR empty (00000000)
0x0024: BAR empty (00000000)
0x0028: Cardbus CIS: 00000000
0x002c: Subsystem Vendor ID: 0b5d Product ID: 0004
0x0030: Expansion ROM Base Address: 00000000
0x0038: 00000000
0x003c: Interrupt Pin: 01 Line: 03 Min Gnt: 00 Max Lat: 00
0x0000: 10051af4 00000107 08800000 00000000
0x0010: 00001001 00000000 00000000 00000000
0x0020: 00000000 00000000 00000000 00040b5d
0x0030: 00000000 00000000 00000000 00000103
0x0040: 00000001 00000000 00000000 00000000
0x0050: 00000010 00000000 00000000 00000000
0x0060: 00000000 b0000000 5e77e20a 00000002
0x0070: 00000000 00000000 00000000 00000000
0x0080: 00000000 00000000 00000000 00000000
0x0090: 00000000 00000000 00000000 00000000
0x00a0: 00000000 00000000 00000000 00000000
0x00b0: 00000000 00000000 00000000 00000000
0x00c0: 00000000 00000000 00000000 00000000
0x00d0: 00000000 00000000 00000000 00000000
0x00e0: 00000000 00000000 00000000 00000000
0x00f0: 00000000 00000000 00000000 00000000
0:2:0: Qumranet Virtio Network
0x0000: Vendor ID: 1af4, Product ID: 1000
0x0004: Command: 0107, Status: 0000
0x0008: Class: 08 System, Subclass: 80 Miscellaneous,
Interface: 00, Revision: 00
0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
Cache Line Size: 00
0x0010: BAR io addr: 0x00002000/0x1000
0x0014: BAR empty (00000000)
0x0018: BAR empty (00000000)
0x001c: BAR empty (00000000)
0x0020: BAR empty (00000000)
0x0024: BAR empty (00000000)
0x0028: Cardbus CIS: 00000000
0x002c: Subsystem Vendor ID: 0b5d Product ID: 0001
0x0030: Expansion ROM Base Address: 00000000
0x0038: 00000000
0x003c: Interrupt Pin: 01 Line: 05 Min Gnt: 00 Max Lat: 00
0x0000: 10001af4 00000107 08800000 00000000
0x0010: 00002001 00000000 00000000 00000000
0x0020: 00000000 00000000 00000000 00010b5d
0x0030: 00000000 00000000 00000000 00000105
0x0040: 00000001 00000000 00000000 00000000
0x0050: 00000010 00000000 00000000 00000000
0x0060: 00000000 a0000000 5e77e21e 00000002
0x0070: 00000000 00000000 00000000 00000000
0x0080: 00000000 00000000 00000000 00000000
0x0090: 00000000 00000000 614d850e 00000002
0x00a0: 00000000 00000000 00000000 00000000
0x00b0: 00000000 00000000 00000000 00000000
0x00c0: 00000000 00000000 00000000 00000000
0x00d0: 00000000 00000000 00000000 00000000
0x00e0: 00000000 00000000 00000000 00000000
0x00f0: 00000000 00000000 00000000 00000000
0:3:0: Qumranet Virtio Storage
0x0000: Vendor ID: 1af4, Product ID: 1001
0x0004: Command: 0107, Status: 0000
0x0008: Class: 01 Mass Storage, Subclass: 00 SCSI,
Interface: 00, Revision: 00
0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
Cache Line Size: 00
0x0010: BAR io addr: 0x00003000/0x1000
0x0014: BAR empty (00000000)
0x0018: BAR empty (00000000)
0x001c: BAR empty (00000000)
0x0020: BAR empty (00000000)
0x0024: BAR empty (00000000)
0x0028: Cardbus CIS: 00000000
0x002c: Subsystem Vendor ID: 0b5d Product ID: 0002
0x0030: Expansion ROM Base Address: 00000000
0x0038: 00000000
0x003c: Interrupt Pin: 01 Line: 06 Min Gnt: 00 Max Lat: 00
0x0000: 10011af4 00000107 01000000 00000000
0x0010: 00003001 00000000 00000000 00000000
0x0020: 00000000 00000000 00000000 00020b5d
0x0030: 00000000 00000000 00000000 00000106
0x0040: 00000001 00000000 00000000 00000000
0x0050: 00000010 00000000 00000000 00000000
0x0060: 00000000 60000000 5e77e217 00000002
0x0070: 00000000 00000000 00000000 00000000
0x0080: 00000000 00000000 00000000 00000000
0x0090: 00000000 00000000 60b12df1 00000002
0x00a0: 00000000 00000000 00000000 00000000
0x00b0: 00000000 00000000 00000000 00000000
0x00c0: 00000000 00000000 00000000 00000000
0x00d0: 00000000 00000000 00000000 00000000
0x00e0: 00000000 00000000 00000000 00000000
0x00f0: 00000000 00000000 00000000 00000000
0:4:0: OpenBSD VMM Control
0x0000: Vendor ID: 0b5d, Product ID: 0777
0x0004: Command: 0107, Status: 0000
0x0008: Class: 07 Communications, Subclass: 80 Miscellaneous,
Interface: 00, Revision: 00
0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
Cache Line Size: 00
0x0010: BAR io addr: 0x00004000/0x1000
0x0014: BAR empty (00000000)
0x0018: BAR empty (00000000)
0x001c: BAR empty (00000000)
0x0020: BAR empty (00000000)
0x0024: BAR empty (00000000)
0x0028: Cardbus CIS: 00000000
0x002c: Subsystem Vendor ID: 0b5d Product ID: ffff
0x0030: Expansion ROM Base Address: 00000000
0x0038: 00000000
0x003c: Interrupt Pin: 01 Line: 07 Min Gnt: 00 Max Lat: 00
0x0000: 07770b5d 00000107 07800000 00000000
0x0010: 00004001 00000000 00000000 00000000
0x0020: 00000000 00000000 00000000 ffff0b5d
0x0030: 00000000 00000000 00000000 00000107
0x0040: 00000001 00000000 00000000 00000000
0x0050: 00000010 00000000 00000000 00000000
0x0060: 00000000 90000000 5e77e231 00000002
0x0070: 00000000 00000000 00000000 00000000
0x0080: 00000000 00000000 00000000 00000000
0x0090: 00000000 00000000 00000000 00000000
0x00a0: 00000000 00000000 00000000 00000000
0x00b0: 00000000 00000000 00000000 00000000
0x00c0: 00000000 00000000 00000000 00000000
0x00d0: 00000000 00000000 00000000 00000000
0x00e0: 00000000 00000000 00000000 00000000
0x00f0: 00000000 00000000 00000000 00000000
acpidump:
