Re: [ceph-users] ls/file access hangs on a single ceph directory

2013-10-24 Thread Michael
On 24/10/2013 14:55, Yan, Zheng wrote: On Thu, Oct 24, 2013 at 9:13 PM, Michael wrote: On 24/10/2013 13:53, Yan, Zheng wrote: On Thu, Oct 24, 2013 at 5:43 PM, Michael wrote: On 24/10/2013 03:09, Yan, Zheng wrote: On Thu, Oct 24, 2013 at 6:44 AM, Michael wrote: Tying to gather some more in

Re: [ceph-users] ls/file access hangs on a single ceph directory

2013-10-24 Thread Yan, Zheng
On Thu, Oct 24, 2013 at 9:13 PM, Michael wrote: > On 24/10/2013 13:53, Yan, Zheng wrote: >> >> On Thu, Oct 24, 2013 at 5:43 PM, Michael >> wrote: >>> >>> On 24/10/2013 03:09, Yan, Zheng wrote: On Thu, Oct 24, 2013 at 6:44 AM, Michael wrote: > > Tying to gather some more in

[ceph-users] ls/file access hangs on a single ceph directory

2013-10-24 Thread Michael
I have a filesystem shared by several systems mounted on 2 ceph nodes with a 3rd as a reference monitor. It's been used for a couple of months now but suddenly the root directory for the mount has become inaccessible and requests to files in it just hang, there's no ceph errors reported before/a

Re: [ceph-users] ls/file access hangs on a single ceph directory

2013-10-24 Thread Michael
On 24/10/2013 13:53, Yan, Zheng wrote: On Thu, Oct 24, 2013 at 5:43 PM, Michael wrote: On 24/10/2013 03:09, Yan, Zheng wrote: On Thu, Oct 24, 2013 at 6:44 AM, Michael wrote: Tying to gather some more info. CentOS - hanging ls [root@srv ~]# cat /proc/14614/stack [] wait_answer_interruptible+

Re: [ceph-users] ls/file access hangs on a single ceph directory

2013-10-24 Thread Yan, Zheng
On Thu, Oct 24, 2013 at 5:43 PM, Michael wrote: > On 24/10/2013 03:09, Yan, Zheng wrote: >> >> On Thu, Oct 24, 2013 at 6:44 AM, Michael >> wrote: >>> >>> Tying to gather some more info. >>> >>> CentOS - hanging ls >>> [root@srv ~]# cat /proc/14614/stack >>> [] wait_answer_interruptible+0x81/0xc0

Re: [ceph-users] ls/file access hangs on a single ceph directory

2013-10-24 Thread Michael
On 24/10/2013 03:09, Yan, Zheng wrote: On Thu, Oct 24, 2013 at 6:44 AM, Michael wrote: Tying to gather some more info. CentOS - hanging ls [root@srv ~]# cat /proc/14614/stack [] wait_answer_interruptible+0x81/0xc0 [fuse] [] fuse_request_send+0x1cb/0x290 [fuse] [] fuse_do_getattr+0x10c/0x2c0 [f

Re: [ceph-users] ls/file access hangs on a single ceph directory

2013-10-23 Thread Yan, Zheng
On Thu, Oct 24, 2013 at 6:44 AM, Michael wrote: > Tying to gather some more info. > > CentOS - hanging ls > [root@srv ~]# cat /proc/14614/stack > [] wait_answer_interruptible+0x81/0xc0 [fuse] > [] fuse_request_send+0x1cb/0x290 [fuse] > [] fuse_do_getattr+0x10c/0x2c0 [fuse] > [] fuse_update_attribu

Re: [ceph-users] ls/file access hangs on a single ceph directory

2013-10-23 Thread Sage Weil
If you do ceph mds tell 0 dumpcache /tmp/foo it will dump the dms cache, and ceph-post-file /tmp/foo will send the file to ceph.com so we can get some clue what happened. I suspect that restarting the ceph-mds process will resolve the hang. Thanks! sage On Wed, 23 Oct 2013, Michael wrot

Re: [ceph-users] ls/file access hangs on a single ceph directory

2013-10-23 Thread Michael
Tying to gather some more info. CentOS - hanging ls [root@srv ~]# cat /proc/14614/stack [] wait_answer_interruptible+0x81/0xc0 [fuse] [] fuse_request_send+0x1cb/0x290 [fuse] [] fuse_do_getattr+0x10c/0x2c0 [fuse] [] fuse_update_attributes+0x75/0x80 [fuse] [] fuse_getattr+0x53/0x60 [fuse] [] vfs_ge