[ceph-users] Re: mds crash loop

2019-12-05 Thread Karsten Nielsen
:Re: [ceph-users] Re: mds crash loop To: Karsten Nielsen ; CC: ceph-users@ceph.io; > On Tue, Nov 12, 2019 at 6:18 PM Karsten Nielsen wrote: > > > > -Original message- > > From: Karsten Nielsen > > Sent: Tue 12-11-2019 10:30 > > Subject:

[ceph-users] Re: mds crash loop

2019-11-12 Thread Yan, Zheng
On Tue, Nov 12, 2019 at 6:18 PM Karsten Nielsen wrote: > > -Original message- > From: Karsten Nielsen > Sent: Tue 12-11-2019 10:30 > Subject: [ceph-users] Re: mds crash loop > To: Yan, Zheng ; > CC: ceph-users@ceph.io; > > -Original

[ceph-users] Re: mds crash loop

2019-11-12 Thread Karsten Nielsen
-Original message- From: Karsten Nielsen Sent: Tue 12-11-2019 10:30 Subject:[ceph-users] Re: mds crash loop To: Yan, Zheng ; CC: ceph-users@ceph.io; > -Original message- > From: Yan, Zheng > Sent: Mon 11-11-2019 15:09 > Subject: Re: [ceph-us

[ceph-users] Re: mds crash loop

2019-11-12 Thread Karsten Nielsen
-Original message- From: Yan, Zheng Sent: Mon 11-11-2019 15:09 Subject:Re: [ceph-users] Re: mds crash loop To: Karsten Nielsen ; CC: ceph-users@ceph.io; > On Mon, Nov 11, 2019 at 5:09 PM Karsten Nielsen wrote: > > > > I started a job that moved some

[ceph-users] Re: mds crash loop

2019-11-11 Thread Yan, Zheng
arsten > > -Original message- > From: Yan, Zheng > Sent: Thu 07-11-2019 14:20 > Subject:Re: [ceph-users] Re: mds crash loop > To: Karsten Nielsen ; > CC: ceph-users@ceph.io; > > On Thu, Nov 7, 2019 at 6:40 PM Karsten Nielsen wrote: > > > &g

[ceph-users] Re: mds crash loop

2019-11-11 Thread Karsten Nielsen
-2019 14:20 Subject:Re: [ceph-users] Re: mds crash loop To: Karsten Nielsen ; CC: ceph-users@ceph.io; > On Thu, Nov 7, 2019 at 6:40 PM Karsten Nielsen wrote: > > > > That is awesome. > > > > Now I just need to figure out where the lost+found files needs t

[ceph-users] Re: mds crash loop

2019-11-08 Thread Karsten Nielsen
-Original message- From: Yan, Zheng Sent: Thu 07-11-2019 14:20 Subject:Re: [ceph-users] Re: mds crash loop To: Karsten Nielsen ; CC: ceph-users@ceph.io; > On Thu, Nov 7, 2019 at 6:40 PM Karsten Nielsen wrote: > > > > That is awesome. > > &

[ceph-users] Re: mds crash loop

2019-11-07 Thread Yan, Zheng
minute, the rm -rf /mnt/cephfs/missing_obj_dirs > Any tool that is able to do that ? > > Thanks > - Karsten > > -Original message- > From: Yan, Zheng > Sent: Thu 07-11-2019 09:22 > Subject:Re: [ceph-users] Re: mds crash loop > To: Karsten Niels

[ceph-users] Re: mds crash loop

2019-11-07 Thread Karsten Nielsen
: [ceph-users] Re: mds crash loop To: Karsten Nielsen ; CC: ceph-users@ceph.io; > I have tracked down the root cause. See https://tracker.ceph.com/issues/42675 > > Regards > Yan, Zheng > > On Thu, Nov 7, 2019 at 4:01 PM Karsten Nielsen wrote: > > > > -Origina

[ceph-users] Re: mds crash loop

2019-11-07 Thread Yan, Zheng
I have tracked down the root cause. See https://tracker.ceph.com/issues/42675 Regards Yan, Zheng On Thu, Nov 7, 2019 at 4:01 PM Karsten Nielsen wrote: > > -Original message- > From: Yan, Zheng > Sent: Thu 07-11-2019 07:21 > Subject: Re: [ceph-users] Re: mds

[ceph-users] Re: mds crash loop

2019-11-07 Thread Karsten Nielsen
-Original message- From: Yan, Zheng Sent: Thu 07-11-2019 07:21 Subject:Re: [ceph-users] Re: mds crash loop To: Karsten Nielsen ; CC: ceph-users@ceph.io; > On Thu, Nov 7, 2019 at 5:50 AM Karsten Nielsen wrote: > > > > -Original message- > &g

[ceph-users] Re: mds crash loop

2019-11-06 Thread Yan, Zheng
On Thu, Nov 7, 2019 at 5:50 AM Karsten Nielsen wrote: > > -Original message- > From: Yan, Zheng > Sent: Wed 06-11-2019 14:16 > Subject:Re: [ceph-users] mds crash loop > To: Karsten Nielsen ; > CC: ceph-users@ceph.io; > > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen

[ceph-users] Re: mds crash loop

2019-11-06 Thread Karsten Nielsen
-Original message- From: Yan, Zheng Sent: Wed 06-11-2019 14:16 Subject:Re: [ceph-users] mds crash loop To: Karsten Nielsen ; CC: ceph-users@ceph.io; > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen wrote: > > > > -Original message- > > From: Yan, Zheng > >

[ceph-users] Re: mds crash loop

2019-11-06 Thread Karsten Nielsen
-Original message- From: Yan, Zheng Sent: Wed 06-11-2019 14:16 Subject:Re: [ceph-users] mds crash loop To: Karsten Nielsen ; CC: ceph-users@ceph.io; > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen wrote: > > > > -Original message- > > From: Yan, Zheng > >

[ceph-users] Re: mds crash loop

2019-11-06 Thread Yan, Zheng
On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen wrote: > > -Original message- > From: Yan, Zheng > Sent: Wed 06-11-2019 08:15 > Subject:Re: [ceph-users] mds crash loop > To: Karsten Nielsen ; > CC: ceph-users@ceph.io; > > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen

[ceph-users] Re: mds crash loop

2019-11-06 Thread Karsten Nielsen
-Original message- From: Yan, Zheng Sent: Wed 06-11-2019 08:15 Subject:Re: [ceph-users] mds crash loop To: Karsten Nielsen ; CC: ceph-users@ceph.io; > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen wrote: > > > > Hi, > > > > Last week I upgraded my ceph cluster from

[ceph-users] Re: mds crash loop

2019-11-05 Thread Yan, Zheng
On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen wrote: > > Hi, > > Last week I upgraded my ceph cluster from luminus to mimic 13.2.6 > It was running fine for a while but yesterday my mds went into a crash loop. > > I have 1 active and 1 standby mds for my cephfs both of which is running the > sam

[ceph-users] Re: mds crash loop

2019-11-05 Thread Karsten Nielsen
from ceph -w [root@k8s-node-01 /]# ceph -w cluster: id: 571d4bfe-2c5d-45ca-8da1-91dcaf69942c health: HEALTH_WARN 1 filesystem is degraded services: mon: 3 daemons, quorum k8s-node-00,k8s-node-01,k8s-node-02 mgr: k8s-node-01(active) mds: cephfs-1/1/1 up rec