Re: Process stuck in "vnread"

2016-03-29 Thread Maxim Sobolev
Hey Adrian, unfortunately, this is kinda our production build/test box, it's not particularly beefy and we also run some heavily contended apps on it too just to see how it performs on this size of machine, so we would not really want to put any heavy debug into kernel unless it's some issue that c

Re: Process stuck in "vnread"

2016-03-28 Thread Maxim Sobolev
Andriy, this is file that gets copies from md(4)-baked UFS to file that is located on ZFS zraid volume that is mounted via NULLFS. The file that backs up md(4) is located on ZFS, so in a sense we have full cycle with the backing block starting on ZFS and ending up on ZFS too. cp(1) calls write(), w

Re: Process stuck in "vnread"

2016-03-28 Thread Andriy Gapon
On 28/03/2016 19:23, Konstantin Belousov wrote: > On Mon, Mar 28, 2016 at 08:52:03AM -0700, Maxim Sobolev wrote: >> Done some head scratching, it looks like it's got page fault in the >> copyin() (cp(1) AFAIK mmaps source file). There might be some interlock >> issue between competing write to the

Re: Process stuck in "vnread"

2016-03-28 Thread Konstantin Belousov
On Mon, Mar 28, 2016 at 08:52:03AM -0700, Maxim Sobolev wrote: > Done some head scratching, it looks like it's got page fault in the > copyin() (cp(1) AFAIK mmaps source file). There might be some interlock > issue between competing write to the same ZFS, the md0 device is locked > forever waiting

Re: Process stuck in "vnread"

2016-03-28 Thread Adrian Chadd
Hi, I think a bunch of the lock order checks with witness get disabled for ZFS ? Maybe compile up a witness kernel so you can get 'show alllocks' in ddb. That should help narrow down exactly what's going on. Thanks! -a ___ freebsd-stable@freebsd.org

Re: Process stuck in "vnread"

2016-03-28 Thread Maxim Sobolev
P.S. That being said, I am not sure if that write operation on md(4) happen before or after the offending lockup in the zfs_freebsd_write(), so it might be just as well be the result of that, not cause. On Mon, Mar 28, 2016 at 8:52 AM, Maxim Sobolev wrote: > Done some head scratching, it looks l

Re: Process stuck in "vnread"

2016-03-28 Thread Maxim Sobolev
Done some head scratching, it looks like it's got page fault in the copyin() (cp(1) AFAIK mmaps source file). There might be some interlock issue between competing write to the same ZFS, the md0 device is locked forever waiting for the write operation to complete at the very same time. I am curious

Re: Process stuck in "vnread"

2016-03-28 Thread Maxim Sobolev
OK, that happened again. Now it's 10.3-RC3, funny enough, it's the same "cp" command. Any ideas about what can be wrong here? The box is still up, so if you need me to do something specific inside kgdb please let me know. Thanks! $ uname -a FreeBSD abc01.sippysoft.com 10.3-RC3 FreeBSD 10.3-RC3 #0

Re: Process stuck in "vnread"

2016-03-02 Thread Maxim Sobolev
Thanks Ronald, I think this is at least a possibility that it's related, I've bumped myself up to the latest FreeBSD 10.3-BETA3 (svn revision 296326) and compiled nullfs statically, so I'll run those builds and see how it goes. -Max On Wed, Mar 2, 2016 at 1:32 AM, Ronald Klop wrote: > Hello, >

Re: Process stuck in "vnread"

2016-03-02 Thread Maxim Sobolev
Sorry gmail hit set too early. Backtrace from the md worker: [Switching to thread 357 (Thread 101131)]#0 0x8095244e in sched_switch () (kgdb) bt #0 0x8095244e in sched_switch () #1 0x809313b1 in mi_switch () #2 0x8097089a in sleepq_wait () #3 0x808d344d

Re: Process stuck in "vnread"

2016-03-02 Thread Konstantin Belousov
On Wed, Mar 02, 2016 at 03:02:02AM -0800, Maxim Sobolev wrote: > Thanks, Konstantin. > > Re: md(4) state: > >0 88688 0 0 -8 0 0 16 tx->tx_s DL- 0:45.43 > [md0] > > Its backtrace: So md is stuck in ZFS. ___ freebsd-stabl

Re: Process stuck in "vnread"

2016-03-02 Thread Maxim Sobolev
Thanks, Konstantin. Re: md(4) state: 0 88688 0 0 -8 0 0 16 tx->tx_s DL- 0:45.43 [md0] Its backtrace: About the backtrace, indeed, looks like you are right and some portion of it is not decoded properly, as it's loaded as a kernel module. The setup is somewhat ev

Re: Process stuck in "vnread"

2016-03-02 Thread Konstantin Belousov
On Wed, Mar 02, 2016 at 01:12:31AM -0800, Maxim Sobolev wrote: > Hi, I've encountered cp(1) process stuck in the vnread state on one of my > build machines that got recently upgraded to 10.3. > >0 79596 1 0 20 0 170921396 wait I 1 0:00.00 > /bin/sh /usr/local/bin/au

Re: Process stuck in "vnread"

2016-03-02 Thread Ronald Klop
Hello, Would it be possible this has to do with the resolved 'system hangs when using ZFS caused by VFS' in 10.3-BETA3? https://lists.freebsd.org/pipermail/freebsd-stable/2016-February/084238.html Regards, Ronald. On Wed, 02 Mar 2016 10:12:31 +0100, Maxim Sobolev wrote: Hi, I've encoun

Process stuck in "vnread"

2016-03-02 Thread Maxim Sobolev
Hi, I've encountered cp(1) process stuck in the vnread state on one of my build machines that got recently upgraded to 10.3. 0 79596 1 0 20 0 170921396 wait I 1 0:00.00 /bin/sh /usr/local/bin/autoreconf -f -i 0 79602 79596 0 52 0 414889036 wait I