Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

you can run 13.2.1 mds on another machine. kill all client sessions and wait until purge queue is empty. then it's safe to run 13.2.2 mds. run command "cephfs-journal-tool --rank=cephfs_name:rank --journal=purge_queue header get" purge queue is empty when write_pos == expire_pos On Wed, Nov 21,

[ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

I am also having this problem. Zheng (or anyone else), any idea how to perform this downgrade on a node that is also a monitor and an OSD node? dpkg complains of a dependency conflict when I try to install ceph-mds_13.2.1-1xenial_amd64.deb: ``` dpkg: dependency problems prevent configuration of c

Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

On 08/10/18 11:47, Yan, Zheng wrote: On Mon, Oct 8, 2018 at 9:46 PM Alfredo Daniel Rezinovsky wrote: On 08/10/18 10:20, Yan, Zheng wrote: On Mon, Oct 8, 2018 at 9:07 PM Alfredo Daniel Rezinovsky wrote: On 08/10/18 09:45, Yan, Zheng wrote: On Mon, Oct 8, 2018 at 6:40 PM Alfredo Daniel R

Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

On Mon, Oct 8, 2018 at 9:46 PM Alfredo Daniel Rezinovsky wrote: > > > > On 08/10/18 10:20, Yan, Zheng wrote: > > On Mon, Oct 8, 2018 at 9:07 PM Alfredo Daniel Rezinovsky > > wrote: > >> > >> > >> On 08/10/18 09:45, Yan, Zheng wrote: > >>> On Mon, Oct 8, 2018 at 6:40 PM Alfredo Daniel Rezinovsky >

Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

On 08/10/18 10:20, Yan, Zheng wrote: On Mon, Oct 8, 2018 at 9:07 PM Alfredo Daniel Rezinovsky wrote: On 08/10/18 09:45, Yan, Zheng wrote: On Mon, Oct 8, 2018 at 6:40 PM Alfredo Daniel Rezinovsky wrote: On 08/10/18 07:06, Yan, Zheng wrote: On Mon, Oct 8, 2018 at 5:43 PM Sergey Malinin

Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

On 08/10/18 10:32, Sergey Malinin wrote: On 8.10.2018, at 16:07, Alfredo Daniel Rezinovsky mailto:alfrenov...@gmail.com>> wrote: So I can stopt  cephfs-data-scan, run the import, downgrade, and then reset the purge queue? I suggest that you backup metadata pool so that in case of failure

Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

> On 8.10.2018, at 16:07, Alfredo Daniel Rezinovsky > wrote: > > So I can stopt cephfs-data-scan, run the import, downgrade, and then reset > the purge queue? I suggest that you backup metadata pool so that in case of failure you can continue with data scan from where you stopped. I've read

Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

On Mon, Oct 8, 2018 at 9:07 PM Alfredo Daniel Rezinovsky wrote: > > > > On 08/10/18 09:45, Yan, Zheng wrote: > > On Mon, Oct 8, 2018 at 6:40 PM Alfredo Daniel Rezinovsky > > wrote: > >> On 08/10/18 07:06, Yan, Zheng wrote: > >>> On Mon, Oct 8, 2018 at 5:43 PM Sergey Malinin wrote: > > >

Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

On 08/10/18 09:45, Yan, Zheng wrote: On Mon, Oct 8, 2018 at 6:40 PM Alfredo Daniel Rezinovsky wrote: On 08/10/18 07:06, Yan, Zheng wrote: On Mon, Oct 8, 2018 at 5:43 PM Sergey Malinin wrote: On 8.10.2018, at 12:37, Yan, Zheng wrote: On Mon, Oct 8, 2018 at 4:37 PM Sergey Malinin wrote:

Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

On Mon, Oct 8, 2018 at 5:43 PM Sergey Malinin wrote: > > > > > On 8.10.2018, at 12:37, Yan, Zheng wrote: > > > > On Mon, Oct 8, 2018 at 4:37 PM Sergey Malinin wrote: > >> > >> What additional steps need to be taken in order to (try to) regain access > >> to the fs providing that I backed up met

Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

> On 8.10.2018, at 12:37, Yan, Zheng wrote: > > On Mon, Oct 8, 2018 at 4:37 PM Sergey Malinin wrote: >> >> What additional steps need to be taken in order to (try to) regain access to >> the fs providing that I backed up metadata pool, created alternate metadata >> pool and ran scan_extent

Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

On Mon, Oct 8, 2018 at 4:37 PM Sergey Malinin wrote: > > What additional steps need to be taken in order to (try to) regain access to > the fs providing that I backed up metadata pool, created alternate metadata > pool and ran scan_extents, scan_links, scan_inodes, and somewhat recursive > scru

Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

What additional steps need to be taken in order to (try to) regain access to the fs providing that I backed up metadata pool, created alternate metadata pool and ran scan_extents, scan_links, scan_inodes, and somewhat recursive scrub. After that I only mounted the fs read-only to backup the data

Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

Sorry. this is caused wrong backport. downgrading mds to 13.2.1 and marking mds repaird can resolve this. Yan, Zheng On Sat, Oct 6, 2018 at 8:26 AM Sergey Malinin wrote: > > Update: > I discovered http://tracker.ceph.com/issues/24236 and > https://github.com/ceph/ceph/pull/22146 > Make sure that

Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

I was able to start MDS and mount the fs with broken ownership/permissions and 8k out of millions files in lost+found. > On 7.10.2018, at 02:04, Sergey Malinin wrote: > > I'm at scan_links now, will post an update once it has finished. > Have you reset the journal after fs recovery as suggeste

Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

I'm at scan_links now, will post an update once it has finished. Have you reset the journal after fs recovery as suggested in the doc? quote: If the damaged filesystem contains dirty journal data, it may be recovered next with: cephfs-journal-tool --rank=:0 event recover_dentries list --altern

Re: [ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

Update: I discovered http://tracker.ceph.com/issues/24236 and https://github.com/ceph/ceph/pull/22146 Make sure that it is not relevant in your case before proceeding to operations that modify on-disk data. > On

[ceph-users] MDS damaged after mimic 13.2.1 to 13.2.2 upgrade

Hello, Followed standard upgrade procedure to upgrade from 13.2.1 to 13.2.2. After upgrade MDS cluster is down, mds rank 0 and purge_queue journal are damaged. Resetting purge_queue does not seem to work well as journal still appears to be damaged. Can anybody help? mds log: -789> 2018-09-26