Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-18 Thread Henri Hennebert
On 11/18/2016 13:30, Andriy Gapon wrote: On 14/11/2016 14:00, Henri Hennebert wrote: On 11/14/2016 12:45, Andriy Gapon wrote: Okay. Luckily for us, it seems that 'm' is available in frame 5. It also happens to be the first field of 'struct faultstate'. So, could you please go to frame and

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-18 Thread Andriy Gapon
On 14/11/2016 14:00, Henri Hennebert wrote: > On 11/14/2016 12:45, Andriy Gapon wrote: >> Okay. Luckily for us, it seems that 'm' is available in frame 5. It also >> happens to be the first field of 'struct faultstate'. So, could you please >> go >> to frame and print '*m' and '*(struct faultst

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-14 Thread Henri Hennebert
On 11/14/2016 12:45, Andriy Gapon wrote: On 14/11/2016 11:35, Henri Hennebert wrote: On 11/14/2016 10:07, Andriy Gapon wrote: Hmm, I've just noticed another interesting thread: Thread 668 (Thread 101245): #0 sched_switch (td=0xf800b642aa00, newtd=0xf8000285f000, flags=) at /usr/src

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-14 Thread Andriy Gapon
On 14/11/2016 11:35, Henri Hennebert wrote: > > > On 11/14/2016 10:07, Andriy Gapon wrote: >> Hmm, I've just noticed another interesting thread: >> Thread 668 (Thread 101245): >> #0 sched_switch (td=0xf800b642aa00, newtd=0xf8000285f000, >> flags=> optimized out>) at /usr/src/sys/kern/sc

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-14 Thread Henri Hennebert
On 11/14/2016 10:07, Andriy Gapon wrote: On 13/11/2016 15:28, Henri Hennebert wrote: On 11/13/2016 11:06, Andriy Gapon wrote: On 12/11/2016 14:40, Henri Hennebert wrote: [snip] Could you please show 'info local' in frame 14? I expected that 'nd' variable would be defined there and it may

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-14 Thread Andriy Gapon
On 13/11/2016 15:28, Henri Hennebert wrote: > On 11/13/2016 11:06, Andriy Gapon wrote: >> On 12/11/2016 14:40, Henri Hennebert wrote: >>> I attatch it >> >> Thank you! >> So, these two threads are trying to get the lock in the exclusive mode: >> Thread 687 (Thread 101243): >> #0 sched_switch (td=0

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-13 Thread Henri Hennebert
On 11/13/2016 14:28, Henri Hennebert wrote: This 2 threads are innd processes. In core.txt.4: 8 14789 29165 0 24 4 40040 6612 zfs DN- 0:00.00 [innd] 8 29165 1 0 20 0 42496 6888 select Ds- 0:01.33 [innd] 8 49778 29165 0 24 4 40040 6900 zfs

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-13 Thread Henri Hennebert
On 11/13/2016 11:06, Andriy Gapon wrote: On 12/11/2016 14:40, Henri Hennebert wrote: I attatch it Thank you! So, these two threads are trying to get the lock in the exclusive mode: Thread 687 (Thread 101243): #0 sched_switch (td=0xf800b642b500, newtd=0xf8000285ea00, flags=) at /usr/sr

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-13 Thread Andriy Gapon
On 12/11/2016 14:40, Henri Hennebert wrote: > I attatch it Thank you! So, these two threads are trying to get the lock in the exclusive mode: Thread 687 (Thread 101243): #0 sched_switch (td=0xf800b642b500, newtd=0xf8000285ea00, flags=) at /usr/src/sys/kern/sched_ule.c:1973 #1 0x8

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-12 Thread Andriy Gapon
On 11/11/2016 16:50, Henri Hennebert wrote: > > > On 11/11/2016 12:24, Andriy Gapon wrote: >> >> At this stage I would try to get a system crash dump for post-mortem >> analysis. >> There are a few way to do that. You can enter ddb and then run 'dump' and >> 'reset' commands. Or you can just d

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-11 Thread Henri Hennebert
On 11/11/2016 12:24, Andriy Gapon wrote: At this stage I would try to get a system crash dump for post-mortem analysis. There are a few way to do that. You can enter ddb and then run 'dump' and 'reset' commands. Or you can just do `sysctl debug.kdb.panic=1`. In either case, please double-che

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-11 Thread Andriy Gapon
On 10/11/2016 21:41, Henri Hennebert wrote: > On 11/10/2016 19:40, Andriy Gapon wrote: >> On 10/11/2016 19:55, Henri Hennebert wrote: >>> >>> >>> On 11/10/2016 18:33, Andriy Gapon wrote: On 10/11/2016 18:12, Henri Hennebert wrote: > On 11/10/2016 16:54, Andriy Gapon wrote: >> On 10/11/

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Henri Hennebert
On 11/10/2016 19:40, Andriy Gapon wrote: On 10/11/2016 19:55, Henri Hennebert wrote: On 11/10/2016 18:33, Andriy Gapon wrote: On 10/11/2016 18:12, Henri Hennebert wrote: On 11/10/2016 16:54, Andriy Gapon wrote: On 10/11/2016 17:20, Henri Hennebert wrote: On 11/10/2016 15:00, Andriy Gapon w

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Andriy Gapon
On 10/11/2016 19:55, Henri Hennebert wrote: > > > On 11/10/2016 18:33, Andriy Gapon wrote: >> On 10/11/2016 18:12, Henri Hennebert wrote: >>> On 11/10/2016 16:54, Andriy Gapon wrote: On 10/11/2016 17:20, Henri Hennebert wrote: > On 11/10/2016 15:00, Andriy Gapon wrote: >> Interesting

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Henri Hennebert
On 11/10/2016 18:33, Andriy Gapon wrote: On 10/11/2016 18:12, Henri Hennebert wrote: On 11/10/2016 16:54, Andriy Gapon wrote: On 10/11/2016 17:20, Henri Hennebert wrote: On 11/10/2016 15:00, Andriy Gapon wrote: Interesting. I can not spot any suspicious thread that would hold the vnode loc

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Andriy Gapon
On 10/11/2016 18:12, Henri Hennebert wrote: > On 11/10/2016 16:54, Andriy Gapon wrote: >> On 10/11/2016 17:20, Henri Hennebert wrote: >>> On 11/10/2016 15:00, Andriy Gapon wrote: Interesting. I can not spot any suspicious thread that would hold the vnode lock. Could you please run

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Henri Hennebert
On 11/10/2016 16:54, Andriy Gapon wrote: On 10/11/2016 17:20, Henri Hennebert wrote: On 11/10/2016 15:00, Andriy Gapon wrote: Interesting. I can not spot any suspicious thread that would hold the vnode lock. Could you please run kgdb (just like that, no arguments), then execute 'bt' command a

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Andriy Gapon
On 10/11/2016 17:20, Henri Hennebert wrote: > On 11/10/2016 15:00, Andriy Gapon wrote: >> Interesting. I can not spot any suspicious thread that would hold the vnode >> lock. Could you please run kgdb (just like that, no arguments), then execute >> 'bt' command and then select a frame when _vn_lo

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Henri Hennebert
On 11/10/2016 15:00, Andriy Gapon wrote: On 10/11/2016 12:30, Henri Hennebert wrote: On 11/10/2016 11:21, Andriy Gapon wrote: On 09/11/2016 15:58, Eric van Gyzen wrote: On 11/09/2016 07:48, Henri Hennebert wrote: I encounter a strange deadlock on FreeBSD avoriaz.restart.bel 11.0-RELEASE-p3 F

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Andriy Gapon
On 10/11/2016 12:30, Henri Hennebert wrote: > On 11/10/2016 11:21, Andriy Gapon wrote: >> On 09/11/2016 15:58, Eric van Gyzen wrote: >>> On 11/09/2016 07:48, Henri Hennebert wrote: I encounter a strange deadlock on FreeBSD avoriaz.restart.bel 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Henri Hennebert
On 11/10/2016 11:21, Andriy Gapon wrote: On 09/11/2016 15:58, Eric van Gyzen wrote: On 11/09/2016 07:48, Henri Hennebert wrote: I encounter a strange deadlock on FreeBSD avoriaz.restart.bel 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #0 r308260: Fri Nov 4 02:51:33 CET 2016 r...@avoriaz.restart.be

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Andriy Gapon
On 09/11/2016 15:58, Eric van Gyzen wrote: > On 11/09/2016 07:48, Henri Hennebert wrote: >> I encounter a strange deadlock on >> >> FreeBSD avoriaz.restart.bel 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #0 >> r308260: >> Fri Nov 4 02:51:33 CET 2016 >> r...@avoriaz.restart.bel:/usr/obj/usr/src/sys/AV

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-10 Thread Henri Hennebert
On 11/09/2016 19:23, Thierry Thomas wrote: Le mer. 9 nov. 16 à 15:03:49 +0100, Henri Hennebert écrivait : [root@avoriaz ~]# procstat -kk 85656 PIDTID COMM TDNAME KSTACK 85656 101112 find -mi_switch+0xd2 sleepq_wait+0x3a sleeplk+0x1b4 __lockmgr_a

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-09 Thread Thierry Thomas
Le mer. 9 nov. 16 à 15:03:49 +0100, Henri Hennebert écrivait : > [root@avoriaz ~]# procstat -kk 85656 >PIDTID COMM TDNAME KSTACK > 85656 101112 find -mi_switch+0xd2 > sleepq_wait+0x3a sleeplk+0x1b4 __lockmgr_args+0x356 vop_stdlock+0x3c > VOP_LOC

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-09 Thread Henri Hennebert
On 11/09/2016 14:58, Eric van Gyzen wrote: On 11/09/2016 07:48, Henri Hennebert wrote: I encounter a strange deadlock on FreeBSD avoriaz.restart.bel 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #0 r308260: Fri Nov 4 02:51:33 CET 2016 r...@avoriaz.restart.bel:/usr/obj/usr/src/sys/AVORIAZ amd64 Thi

Re: Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-09 Thread Eric van Gyzen
On 11/09/2016 07:48, Henri Hennebert wrote: > I encounter a strange deadlock on > > FreeBSD avoriaz.restart.bel 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #0 > r308260: > Fri Nov 4 02:51:33 CET 2016 > r...@avoriaz.restart.bel:/usr/obj/usr/src/sys/AVORIAZ amd64 > > This system is exclusively runni

Freebsd 11.0 RELEASE - ZFS deadlock

2016-11-09 Thread Henri Hennebert
I encounter a strange deadlock on FreeBSD avoriaz.restart.bel 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #0 r308260: Fri Nov 4 02:51:33 CET 2016 r...@avoriaz.restart.bel:/usr/obj/usr/src/sys/AVORIAZ amd64 This system is exclusively running on zfs. After 3 or 4 days, `periodic daily` is locked