On 2017/05/04 00:31, Mark Kettenis wrote:
> > Date: Wed, 3 May 2017 21:05:24 +0100
> > From: Stuart Henderson <s...@spacehopper.org>
> >
> > On 2017/05/03 15:12, Mark Kettenis wrote:
> > > > Date: Wed, 3 May 2017 13:51:22 +0100
> > > > From: Stuart Henderson <s...@spacehopper.org>
> > > >
> > > > On 2017/05/01 22:18, Mark Kettenis wrote:
> > > > > > Date: Mon, 1 May 2017 20:58:29 +0100
> > > > > > From: Stuart Henderson <s...@spacehopper.org>
> > > > > >
> > > > > > Userland is non-responsive, machine is pingable, tcp connections
> > > > > > open
> > > > > > but no banner from ssh. No failed pool requests. This kernel is from
> > > > > > today's snapshot but I saw the same with one from a couple of days
> > > > > > ago. Is there anything else I can get that might be useful?
> > > > > >
> > > > ..
> > > > > > 71034 186155 65198 0 3 0x11 vp perl
> > > > ..
> > > > >
> > > > > The diff below might fix thise. Or it might actually turn this into a
> > > > > hard hang...
> > > > >
> > > > > Nevertheless, could you try running with it?
> > > >
> > > > I haven't seen this happen again with your diff, and haven't seen any
> > > > hangs. Probably still too early to say for sure that it fixes things,
> > > > but it looks promising so far.
> > >
> > > Thanks. Since Dale ok'ed it and I had been running with it for a
> > > while already, I committed it last night.
> > >
> >
> > Ha. As is traditional, not long after sending that message I've hit
> > a hard lock - no DDB.
>
> I'm sure it wouldn't have happened if I hand't committed it ;).
>
> Could you change the PR_NOWAIT back into P_WAITOK and see if you can
> reproduce the hang and break into ddb? Meanwhile I'll think about
> what information to print once you've hit it ;).
>
I've hit it:
[halt sent]
Stopped at pluart_intr+0x164:
ddb> ps
PID TID PPID UID S FLAGS WAIT COMMAND
625 437040 78755 55 3 0x2 vp cc
78755 385259 7516 55 3 0x82 wait cc
7516 136325 5903 55 3 0x82 wait bash
19959 322579 19904 0 3 0x11 vp perl
5903 139810 38260 55 3 0x82 wait gmake
38260 218618 2854 55 3 0x82 wait bash
2854 90787 9324 55 3 0x82 wait gmake
9324 170231 48868 55 3 0x82 wait gmake
48868 117527 14396 55 3 0x80 wait bash
14396 515415 1643 55 3 0x82 wait bash
1643 416463 68987 55 3 0x82 wait gmake
68987 461754 3515 55 3 0x82 wait bash
3515 418174 60280 55 3 0x82 wait gmake
60280 191595 96060 55 3 0x10008a pause sh
96060 102351 39653 55 3 0x10008a pause sh
39653 519956 69874 55 3 0x10008a pause make
69874 105490 67943 55 3 0x10008a pause make
67943 8933 3490 55 3 0x10008a pause sh
3490 32658 91100 55 3 0x10008a pause make
91100 232620 7725 55 3 0x10008a pause sh
7725 345849 19904 55 3 0x10008a pause make
74896 137698 47027 1000 3 0x100083 kqread tmux
47027 296497 97822 1000 3 0x10008b pause ksh
97822 245510 69858 1000 3 0x10 vp sshd
69858 304169 7139 0 3 0x92 poll sshd
52214 79445 1 0 3 0x100003 vp getty
65809 151855 1 77 3 0x100090 poll dhclient
69118 177062 1 0 3 0x80 poll dhclient
74448 202290 83042 1000 3 0x100083 ttyin ksh
19904 489750 99750 0 3 0x93 wait perl
99750 481138 83042 1000 3 0x10008b pause ksh
83042 493811 1 1000 3 0x100080 kqread tmux
60522 271756 1 0 3 0x100010 vp cron
49702 312331 1 110 3 0x100090 poll sndiod
81577 333393 1 99 3 0x100090 poll sndiod
3727 393780 19238 95 3 0x100092 kqread smtpd
88472 480120 19238 103 3 0x100092 kqread smtpd
87719 273623 19238 95 3 0x100092 kqread smtpd
8252 227745 19238 95 3 0x100092 kqread smtpd
30445 236541 19238 95 3 0x100092 kqread smtpd
23622 99340 19238 95 3 0x100092 kqread smtpd
19238 382063 1 0 3 0x100080 kqread smtpd
7139 54914 1 0 3 0x80 select sshd
63805 211190 0 0 3 0x14280 nfsidl nfsio
56273 341955 0 0 3 0x14280 nfsidl nfsio
13129 36316 0 0 3 0x14280 nfsidl nfsio
43056 41917 0 0 3 0x14280 nfsidl nfsio
65234 94439 1 0 3 0x80 poll rpc.statd
36708 372206 1 28 3 0x100090 poll portmap
67184 8464 1 0 3 0x100000 vp ntpd
63530 142156 9255 83 3 0x100092 poll ntpd
9255 419504 1 83 3 0x100092 poll ntpd
21344 103894 37793 74 3 0x100090 bpf pflogd
37793 380774 1 0 3 0x80 netio pflogd
88367 407407 15455 73 3 0x100090 kqread syslogd
15455 126289 1 0 3 0x100082 netio syslogd
72434 358805 1 0 3 0x80 mfsidl mount_mfs
50290 48406 0 0 3 0x14200 pgzero zerothread
59015 520358 0 0 3 0x14200 aiodoned aiodoned
17801 463867 0 0 3 0x14200 syncer update
44767 310853 0 0 3 0x14200 cleaner cleaner
54975 292383 0 0 3 0x14200 reaper reaper
91922 407515 0 0 3 0x14200 pgdaemon pagedaemon
94512 251496 0 0 3 0x14200 bored crynlk
98657 227030 0 0 3 0x14200 bored crypto
39112 254079 0 0 3 0x14200 pftm pfpurge
2397 79268 0 0 3 0x14200 usbtsk usbtask
19773 474134 0 0 3 0x14200 usbatsk usbatsk
35790 24431 0 0 3 0x14200 bored softnet
90639 498542 0 0 3 0x14200 bored systqmp
82735 387084 0 0 3 0x14200 bored systq
61640 110443 0 0 3 0x40014200 bored softclock
*85150 137825 0 0 7 0x40014200 idle0
16105 122058 0 0 3 0x14200 kmalloc kmthread
1 52157 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb> sh uvmexp
Current UVM status:
pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
2026672 VM pages: 102147 active, 13022 inactive, 0 wired, 169818 free (21243 z
ero)
min 10% (25) anon, 10% (25) vnode, 5% (12) vtext
pages 0 anon, 0 vnode, 0 vtext
freemin=67555, free-target=90073, inactive-target=100540, wired-max=675557
faults=-81014170, traps=0, intrs=0, ctxswitch=50458791 fpuswitch=0
softint=38669549, syscalls=-1850493479, kmapent=16
fault counts:
noram=0, noanon=0, noamap=0, pgwait=0, pgrele=0
ok relocks(total)=3698813(3698816), anget(retries)=1732428043(0), amapcopy=
1087830774
neighbor anon/obj pg=193841849/2039583775, gets(lock/unlock)=1651547778/369
8816
cases: anon=1308260659, anoncow=424167384, obj=1504268658, prcopy=147279117
, przero=829977286
daemon and swap counts:
woke=13, revs=6, scans=256981, obscans=256981, anscans=0
busy=0, freed=256981, reactivate=0, deactivate=0
pageouts=0, pending=0, nswget=0
nswapdev=1, nanon=0, nanonneeded=0 nfreeanon=0
swpages=2162687, swpginuse=0, swpgonly=0 paging=0
kernel pointers:
objs(kern)=0xffffff80008a53d0
ddb> sh bcst
Current Buffer Cache status:
numbufs 319739 busymapped 1, delwri 0
kvaslots 2047 avail kva slots 2046
bufpages 1621259, dmapages 1621259, dirtypages 0
pendingreads 51, pendingwrites 56
highflips 0, highflops 0, dmaflips 0
ddb> sh all pools
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
arp 56 118 0 115 1 0 1 1 0 8 0
inpcbpl 288 1242 0 1221 2 0 2 2 0 8 0
plimitpl 152 379997 0 379965 277 275 2 2 0 8 0
rtentry 112 153 0 125 1 0 1 1 0 8 0
syncache 264 3 0 3 3 3 0 1 0 8 0
sackhl 24 241 0 241 122 122 0 1 0 8 0
tcpqe 32 83 0 83 3 3 0 1 0 8 0
tcpcb 560 442 0 433 1 0 1 1 0 8 0
rttmr 72 7 0 7 2 2 0 1 0 8 0
pffrag 112 706515 0 706515 1071 1071 0 6 0 4 0
pffrent 40 4222832 0 4222832 1075 1075 0 9 0 8 0
pfosfp 40 846 0 423 5 0 5 5 0 8 0
pfosfpen 112 1428 0 714 21 0 21 21 0 8 0
pfstitem 24 45558 0 45547 1 0 1 1 0 8 0
pfstkey 112 45558 0 45547 1 0 1 1 0 8 0
pfstate 312 45558 0 45547 546 544 2 3 0 8 0
pfrule 1360 19 0 16 2 1 1 2 0 8 0
art_heap8 4096 1 0 0 1 0 1 1 0 8 0
art_heap4 256 145 0 46 7 0 7 7 0 8 0
art_table 32 146 0 46 1 0 1 1 0 8 0
art_node 16 153 0 126 1 0 1 1 0 8 0
shmpl 112 3 0 3 3 3 0 1 0 8 0
nfsnodepl 472 25397 0 24696 3083 2966 117 3023 0 8 0
dirhash 1024 32403 0 30688 779 558 221 234 0 8 0
newdirblk 32 12720 0 12720 659 659 0 2 0 8 0
dirrem 64 3543218 0 3543218 30042 30042 0 159 0 8 0
mkdir 56 691184 0 691184 5258 5258 0 158 0 8 0
diradd 56 4164681 0 4164681 26975 26975 0 705 0 8 0
freefile 48 2651321 0 2651321 17326 17326 0 126 0 8 0
freeblks 192 2517659 0 2517659 70389 70389 0 523 0 8 0
freefrag 64 348387 0 348387 3755 3755 0 13 0 8 0
allocindir 104 1942505 0 1942505 35043 35043 0 1950 0 8 0
indirdep 56 91210 0 91210 809 809 0 9 0 8 0
allocdir 128 5732228 0 5732228 88640 88640 0 1376 0 8 0
bmsafemap 64 88912 0 88912 468 468 0 12 0 8 0
newblk 64 7674733 0 7674733 1379 1379 0 1 0 8 0
inodedep 152 5069285 0 5069285 112909 112909 0 1924 0 8 0
pagedep 128 506342 0 506342 7213 7213 0 196 0 8 0
dino1pl 128 4205590 0 3809924 28453 15689 12764 12782 0 8 0
ffsino 240 4205590 0 3809924 59665 36390 23275 23309 0 8 0
nchpl 144 23568508 0 23565846 464 365 99 99 0 8 0
uvmvnodes 72 396384 0 0 7207 0 7207 7207 0 8 0
vnodes 200 396384 0 0 20863 0 20863 20863 0 8 0
namei 1024 217778646 0 217778646 1285 1285 0 1 0 8 0
nfsreqpl 96 861495 0 861495 1891 1891 0 1 0 8 0
xhcixfer 240 21 0 20 1 0 1 1 0 8 0
extentpl 40 21 0 8 1 0 1 1 0 8 0
scxspl 192 7276210 0 7276210 5491 5491 0 2 0 8 0
sigapl 432 4298647 0 4298591 2533 2526 7 10 0 8 0
knotepl 112 84714 0 84652 2 0 2 2 0 8 0
kqueuepl 320 20 0 10 1 0 1 1 0 8 0
pipepl 120 3013088 0 3013086 12 11 1 1 0 8 0
fdescpl 440 4298648 0 4298591 2482 2475 7 10 0 8 0
filepl 112 62418208 0 62418050 131 126 5 7 0 8 0
lockfpl 88 15305 0 15304 2 1 1 1 0 8 0
sessionpl 112 7571 0 7548 4 3 1 3 0 8 0
pgrppl 40 16196 0 16171 1 0 1 1 0 8 0
ucredpl 96 9711138 0 9710438 25 4 21 23 0 8 0
zombiepl 144 4298591 0 4298591 1704 1704 0 3 0 8 0
processpl 744 4298668 0 4298591 1833 1826 7 12 0 8 0
procpl 560 4301777 0 4301700 1097 1091 6 10 0 8 0
sockpl 392 2122 0 2042 216 207 9 10 0 8 0
mcl64k 65536 17971 0 17935 8 2 6 6 0 8 0
mcl16k 16384 445 0 445 95 95 0 1 0 8 0
mcl12k 12288 539 0 538 70 69 1 1 0 8 0
mcl9k 9216 5813596 2 5813575 1258 1256 2 61 0 8 0
mcl8k 8192 1117813 0 1117724 1121 1104 17 19 0 8 0
mcl4k 4096 777974 0 777953 402 395 7 11 0 8 0
mcl2k2 2112 2082 0 2081 13 12 1 1 0 8 0
mcl2k 2048 762612 0 762575 831 821 10 13 0 8 0
mtagpl 80 4 0 4 3 3 0 1 0 8 0
mbufpl 256 22374084 0 22373071 519 446 73 132 0 8 0
bufpl 256 8440406 0 8120667 62093 41539 20554 24775 0 8 0
anonpl 16 1401423787 0 1401328213 36475 36006 469 1314 0 509 0
amapchunkpl 152 52380888 0 52379947 21868 21831 37 328 0 158 0
amappl16 192 18130777 0 18127729 10621 10468 153 485 0 8 0
amappl15 184 1099073 0 1099061 160 159 1 5 0 8 0
amappl14 176 1330113 0 1330091 162 160 2 9 0 8 0
amappl13 168 679381 0 679372 243 242 1 5 0 8 0
amappl12 160 1416544 0 1416530 539 538 1 8 0 8 0
amappl11 152 1590057 0 1589986 735 731 4 19 0 8 0
amappl10 144 3054686 0 3054662 458 456 2 13 0 8 0
amappl9 136 2344945 0 2344908 1215 1212 3 54 0 8 0
amappl8 128 5182108 0 5182050 4656 4651 5 50 0 8 0
amappl7 120 1293031 0 1293005 1186 1185 1 9 0 8 0
amappl6 112 1959850 0 1959810 1598 1595 3 11 0 8 0
amappl5 104 2321931 0 2321857 1215 1211 4 75 0 8 0
amappl4 96 22182740 0 22182352 4569 4557 12 38 0 8 0
amappl3 88 11579040 0 11578606 4197 4183 14 158 0 8 0
amappl2 80 134445905 0 134436218 66781 66583 198 407 0 8 0
amappl1 72 690128483 0 690104527 386848 386412 436 2186 0 8 0
amappl 72 20671350 0 20670910 1665 1653 12 51 0 75 0
dma4096 4096 1 0 1 1 1 0 1 0 8 0
dma512 512 3 0 2 1 0 1 1 0 8 0
dma256 256 7 0 7 1 1 0 1 0 8 0
dma64 64 2 0 2 1 1 0 1 0 8 0
dma32 32 6 0 6 1 1 0 1 0 8 0
dma16 16 2 0 2 1 1 0 1 0 8 0
vp 8192 629404920 0 629392032 3763956 3762345 1611 1863 3 8 0
pted 40 6442073370 0 6441973556 1665838 1664829 1009 3372 1 8 0
pmap 48 4298647 0 4298591 2 0 2 2 1 8 1
aobjpl 64 4 0 3 1 0 1 1 0 8 0
uaddrrnd 24 4298646 0 4298591 1 0 1 1 0 8 0
uaddrbest 32 2 0 0 1 0 1 1 0 8 0
uaddr 24 4298646 0 4298591 1 0 1 1 0 8 0
vmmpekpl 168 20317967 0 20315946 1134 1044 90 183 0 8 0
vmmpepl 168 1742275877 0 1742232060 2006359 2004453 1906 6364 0 357 0
vmsppl 256 4298647 0 4298591 1037 1033 4 6 0 8 0
phpool 112 3894186 0 3848386 1599 261 1338 1347 0 8 0
ddb> sh malloc
Type InUse MemUse HighUse Limit Requests Type Lim Kern Lim
devbuf 2578 1399K 1399K 78643K 4589 0 0
pcb 72 15K 16K 78643K 896 0 0
rtable 74 2K 2K 78643K 604 0 0
ifaddr 32 8K 8K 78643K 43 0 0
counters 14 15K 15K 78643K 14 0 0
ioctlops 0 0K 4K 78643K 1737 0 0
iov 0 0K 16K 78643K 39 0 0
mount 11 11K 11K 78643K 11 0 0
NFS mount 1 0K 0K 78643K 1 0 0
vnodes 36 2K 65K 78643K 3052 0 0
UFS quota 1 16K 16K 78643K 1 0 0
UFS mount 41 82K 82K 78643K 41 0 0
shm 2 1K 5K 78643K 3 0 0
VM map 2 0K 0K 78643K 2 0 0
sem 2 0K 0K 78643K 2 0 0
dirhash 1834 452K 493K 78643K 30525 0 0
file desc 3 8K 16K 78643K 8369 0 0
proc 21 7K 7K 78643K 21 0 0
MFS node 1 0K 0K 78643K 1 0 0
NFS srvsock 1 0K 0K 78643K 1 0 0
NFS daemon 1 16K 16K 78643K 1 0 0
in_multi 14 0K 0K 78643K 16 0 0
ether_multi 1 0K 0K 78643K 3 0 0
ISOFS mount 1 16K 16K 78643K 1 0 0
MSDOSFS mount 1 8K 8K 78643K 1 0 0
ttys 378 1656K 1656K 78643K 378 0 0
exec 0 0K 1K 78643K 4302477 0 0
pfkey data 1 0K 0K 78643K 2 0 0
pagedep 1 4K 4K 78643K 1 0 0
inodedep 1 16K 1862K 78643K 1586598 0 0
newblk 1 0K 0K 78643K 1 0 0
indirdep 0 0K 208K 78643K 733 0 0
VM swap 7 598K 598K 78643K 7 0 0
UVM amap 851 47K 360K 78643K 44517235 0 0
UVM aobj 2 2K 2K 78643K 5 0 0
USB 11 10K 10K 78643K 13 0 0
USB device 4 0K 0K 78643K 4 0 0
crypto data 1 1K 1K 78643K 1 0 0
ip6_options 0 0K 0K 78643K 1 0 0
NDP 4 0K 0K 78643K 4 0 0
temp 70 41K 5718K 78643K 32558468 0 0
kqueue 0 0K 0K 78643K 1 0 0
SYN cache 2 16K 16K 78643K 2 0 0
ddb> sh reg
x0 0x500
x1 0xffffff8022a8a000
x2 0
x3 0xffffffff
x4 0
x5 0xffffff8032b09000
x6 0
x7 0
x8 0x1
x9 0xffffff800058309c pluart_intr
x10 0xb
x11 0x60000005
x12 0
x13 0
x14 0x1
x15 0xe3cd8949c
x16 0x9b82c0100
x17 0xe33f4eacc
x18 0xffffff8032b0dc40
x19 0xffffff8010374c00 _end+0xfa55d00
x20 0xffffff8010358470 _end+0xfa39570
x21 0xffffff8022a8a000
x22 0xffffff8010374c70 _end+0xfa55d70
x23 0xffffff8010374d08 _end+0xfa55e08
x24 0xffffff800080b004 hz
x25 0x3c
x26 0xffffff8000884b94 db_console
x27 0x3
x28 0xffffffffffffffff
x29 0xffffff8032b0dc80
x30 0xffffff8010370400 _end+0xfa51500
sp 0xffffff8032b0dc40
spsr 0xa00003c5
elr 0xffffff800046817c db_enter
lr 0xffffff8000583200 pluart_intr+0x164
pluart_intr+0x164:
