On Mon, Dec 1, 2014 at 1:39 PM, Andrei Mikhailovsky wrote:
> Ilya,
>
> I will try doing that once again tonight as this is a production cluster and
> when dds trigger that dmesg error the cluster's io becomes very bad and I
> have to reboot the server to get things on track. Most of my vms start
>
On Sun, Nov 30, 2014 at 1:15 PM, Andrei Mikhailovsky wrote:
> Greg, thanks for your comment. Could you please share what OS, kernel and
> any nfs/cephfs settings you've used to achieve the pretty well stability?
> Also, what kind of tests have you ran to check that?
We're just doing it on our te
December, 2014 11:06:37 AM
> Subject: Re: [ceph-users] Giant + nfs over cephfs hang tasks
> On Mon, Dec 1, 2014 at 1:39 PM, Andrei Mikhailovsky
> wrote:
> > Ilya,
> >
> > I will try doing that once again tonight as this is a production
> > cluster and
> > w
On Mon, Dec 1, 2014 at 1:39 PM, Andrei Mikhailovsky wrote:
> Ilya,
>
> I will try doing that once again tonight as this is a production cluster and
> when dds trigger that dmesg error the cluster's io becomes very bad and I
> have to reboot the server to get things on track. Most of my vms start
>
nal Message -
> From: "Ilya Dryomov"
> To: "Andrei Mikhailovsky"
> Cc: "ceph-users" , "Gregory Farnum"
>
> Sent: Monday, 1 December, 2014 8:22:08 AM
> Subject: Re: [ceph-users] Giant + nfs over cephfs hang tasks
> On Mon, Dec 1
On Mon, Dec 1, 2014 at 12:30 AM, Andrei Mikhailovsky wrote:
>
> Ilya, further to your email I have switched back to the 3.18 kernel that
> you've sent and I got similar looking dmesg output as I had on the 3.17
> kernel. Please find it attached for your reference. As before, this is the
> command
gt; To: "Ilya Dryomov" , "Andrei Mikhailovsky"
>
> Cc: "ceph-users"
> Sent: Saturday, 29 November, 2014 10:19:32 PM
> Subject: Re: [ceph-users] Giant + nfs over cephfs hang tasks
> Ilya, do you have a ticket reference for the bug?
> Andrei, we run NFS
On Sun, Nov 30, 2014 at 1:19 AM, Gregory Farnum wrote:
> Ilya, do you have a ticket reference for the bug?
Opened a ticket, assigned to myself.
http://tracker.ceph.com/issues/10208
> Andrei, we run NFS tests on CephFS in our nightlies and it does pretty well
> so in the general case we expect i
Ilya, do you have a ticket reference for the bug?
Andrei, we run NFS tests on CephFS in our nightlies and it does pretty well
so in the general case we expect it to work. Obviously not at the moment
with whatever bug Ilya is looking at, though. ;)
-Greg
On Sat, Nov 29, 2014 at 4:51 AM Ilya Dryomov
On Sat, Nov 29, 2014 at 3:49 PM, Ilya Dryomov wrote:
> On Sat, Nov 29, 2014 at 3:22 PM, Andrei Mikhailovsky
> wrote:
>> Ilya,
>>
>> I think i spoke too soon in my last message. I've not given it more load
>> (running 8 concurrent dds with bs=4M) and about a minute or so after
>> starting i've se
On Sat, Nov 29, 2014 at 3:22 PM, Andrei Mikhailovsky wrote:
> Ilya,
>
> I think i spoke too soon in my last message. I've not given it more load
> (running 8 concurrent dds with bs=4M) and about a minute or so after
> starting i've seen problems in dmesg output. I am attaching kern.log file
> for
On Sat, Nov 29, 2014 at 3:10 PM, Andrei Mikhailovsky wrote:
> Ilya,
>
> The 3.17.4 kernel that you've given is also good so far. No hang tasks as
> seen before. However, I do have the same message in dmesg as with the 3.18
> kernel that you've sent. This message I've not seen in the past while usi
> From: "Ilya Dryomov"
> To: "Andrei Mikhailovsky"
> Cc: "ceph-users"
> Sent: Saturday, 29 November, 2014 10:40:48 AM
> Subject: Re: [ceph-users] Giant + nfs over cephfs hang tasks
> On Sat, Nov 29, 2014 at 2:33 AM, Andrei Mikhailovsky
> wrote:
if this message should be treated as alarming.
Andrei
- Original Message -
> From: "Ilya Dryomov"
> To: "Andrei Mikhailovsky"
> Cc: "ceph-users"
> Sent: Saturday, 29 November, 2014 10:40:48 AM
> Subject: Re: [ceph-users] Giant + nfs over
Ilya, I will give it a try and get back to you shortly,
Andrei
- Original Message -
> From: "Ilya Dryomov"
> To: "Andrei Mikhailovsky"
> Cc: "ceph-users"
> Sent: Saturday, 29 November, 2014 10:40:48 AM
> Subject: Re: [ceph-users] Giant +
On Sat, Nov 29, 2014 at 2:33 AM, Andrei Mikhailovsky wrote:
> Ilya,
>
> not sure if dmesg output in the previous is related to the cephfs, but from
> what I can see it looks good with your kernel. I would have seen hang tasks
> by now, but not anymore. I've ran a bunch of concurrent dd tests and a
odule?
Thanks
- Original Message -
> From: "Ilya Dryomov"
> To: "Andrei Mikhailovsky"
> Cc: "ceph-users"
> Sent: Saturday, 29 November, 2014 8:45:54 AM
> Subject: Re: [ceph-users] Giant + nfs over cephfs hang tasks
> On Sat, Nov 29, 2014 at
On Sat, Nov 29, 2014 at 2:13 AM, Andrei Mikhailovsky wrote:
> Ilya, here is what I got shortly after starting the dd test:
>
>
>
> [ 288.307993]
> [ 288.308004] =
> [ 288.308008] [ INFO: possible irq lock inversion dependency detected ]
>
14 5:28:28 PM
> Subject: Re: [ceph-users] Giant + nfs over cephfs hang tasks
> On Fri, Nov 28, 2014 at 8:20 PM, Ilya Dryomov
> wrote:
> > On Fri, Nov 28, 2014 at 8:19 PM, Ilya Dryomov
> > wrote:
> >> On Fri, Nov 28, 2014 at 8:13 PM, Andrei M
and get back to you with more info.
Andrei
- Original Message -
> From: "Ilya Dryomov"
> To: "Andrei Mikhailovsky"
> Cc: "ceph-users"
> Sent: Friday, 28 November, 2014 5:28:28 PM
> Subject: Re: [ceph-users] Giant + nfs over cephfs hang
I will give it a go and let you know.
Cheers
- Original Message -
> From: "Ilya Dryomov"
> To: "Andrei Mikhailovsky"
> Cc: "ceph-users"
> Sent: Friday, 28 November, 2014 5:28:28 PM
> Subject: Re: [ceph-users] Giant + nfs over cephfs
On Fri, Nov 28, 2014 at 8:20 PM, Ilya Dryomov wrote:
> On Fri, Nov 28, 2014 at 8:19 PM, Ilya Dryomov
> wrote:
>> On Fri, Nov 28, 2014 at 8:13 PM, Andrei Mikhailovsky
>> wrote:
>>> Ilya, yes I do! LIke these from different osds:
>>>
>>> [ 4422.212204] libceph: osd13 192.168.168.201:6819 socket
On Fri, Nov 28, 2014 at 8:19 PM, Ilya Dryomov wrote:
> On Fri, Nov 28, 2014 at 8:13 PM, Andrei Mikhailovsky
> wrote:
>> Ilya, yes I do! LIke these from different osds:
>>
>> [ 4422.212204] libceph: osd13 192.168.168.201:6819 socket closed (con state
>> OPEN)
>
> Can you by any chance try a kerne
On Fri, Nov 28, 2014 at 8:13 PM, Andrei Mikhailovsky wrote:
> Ilya, yes I do! LIke these from different osds:
>
> [ 4422.212204] libceph: osd13 192.168.168.201:6819 socket closed (con state
> OPEN)
Can you by any chance try a kernel from [1] ? It's based on Ubuntu
config and unless you are doing
Sent: Friday, 28 November, 2014 4:58:41 PM
> Subject: Re: [ceph-users] Giant + nfs over cephfs hang tasks
> On Fri, Nov 28, 2014 at 3:02 PM, Andrei Mikhailovsky
> wrote:
> > I've just tried the latest ubuntu-vivid kernel and also seeing hang
> > tasks
> > with dd tests:
f=/dev/zero of=/tmp/cephfs/4G bs=1M count=4K oflag=direct & dd
> if=/dev/zero of=/tmp/cephfs/4G1 bs=1M count=4K oflag=direct & dd
> if=/dev/zero of=/tmp/cephfs/4G2 bs=1M count=4K oflag=direct & dd
> if=/dev/zero of=/tmp/cephfs/4G3 bs=1M count=4K oflag=direct &
>
> Cheers
>
> __
?
Thanks
- Original Message -
> From: "Andrei Mikhailovsky"
> To: "ceph-users"
> Sent: Friday, 28 November, 2014 12:02:57 PM
> Subject: Re: [ceph-users] Giant + nfs over cephfs hang tasks
> I've just tried the latest ubuntu-vivid kernel and also seeing hang
&
; dd if=/dev/zero
of=/tmp/cephfs/4G3 bs=1M count=4K oflag=direct &
Cheers
- Original Message -
> From: "Andrei Mikhailovsky"
> To: "ceph-users"
> Sent: Friday, 28 November, 2014 11:22:07 AM
> Subject: Re: [ceph-users] Giant + nfs over cephfs hang tas
back again.
Has anyone experienced similar issues?
Andrei
- Original Message -
> From: "Andrei Mikhailovsky"
> To: "ceph-users"
> Sent: Friday, 28 November, 2014 9:08:17 AM
> Subject: [ceph-users] Giant + nfs over cephfs hang tasks
> Hello guys,
Hello guys,
I've got a bunch of hang tasks of the nfsd service running over the cephfs
(kernel) mounted file system. Here is an example of one of them.
[433079.991218] INFO: task nfsd:32625 blocked for more than 120 seconds.
[433080.029685] Not tainted 3.15.10-031510-generic #201408132333
[
30 matches
Mail list logo
