[ceph-users] Re: [Ceph-announce] Re: v16.2.6 Pacific released

Hi Fyodor, > As I understand, command > > ceph orch upgrade start --ceph-version 16.2.6 > > is broken and will not be able to update the ceph? You should be able to use ceph orch upgrade start --image quay.io/ceph/ceph:v16.2.6 Greetings, Tom On 9/17/21 8:40 AM, Fyodor Ustinov wrote: Hi! Co

[ceph-users] Re: Is it normal Ceph reports "Degraded data redundancy" in normal use?

On 16.09.2021 15:51, Josh Baergen wrote: I assume it's the balancer module. If you write lots of data quickly into the cluster the distribution can vary and the balancer will try to even out the placement. The balancer won't cause degradation, only misplaced objects. Since I'm trying to test

[ceph-users] Re: CephFS optimizated for machine learning workload

On Fri, Sep 17, 2021 at 12:14 AM Mark Nelson wrote: > > > > On 9/15/21 11:05 PM, Yan, Zheng wrote: > > On Wed, Sep 15, 2021 at 8:36 PM Mark Nelson wrote: > >> > >> Hi Zheng, > >> > >> > >> This looks great! Have you noticed any slow performance during > >> directory splitting? One of the things

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

Hi, On 20.08.21 23:58, Patrick Donnelly wrote: Your MDSMap compat is probably what's preventing promotion of standbys. That's a new change in master (which is also being backported to Pacific). Did you downgrade back to Pacific? Try: for i in $(seq 1 10); do ceph fs compat add_incompat $i; d

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

Hi, I had to run ceph fs set cephfs max_mds 1 ceph fs set cephfs allow_standby_replay false and stop all MDS and NFS containers and start one after the other again to clear this issue. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-s

[ceph-users] HEALTH_WARN: failed to probe daemons or devices after upgrade to 16.2.6

Hi! After upgrading to version 16.2.6, my cluster is in this state: root@s-26-9-19-mon-m1:~# ceph -s cluster: id: 1ef45b26-dbac-11eb-a357-616c355f48cb health: HEALTH_WARN failed to probe daemons or devices In logs: 9/17/21 1:30:40 PM[ERR]cephadm exited with an error co

[ceph-users] Re: v16.2.6 Pacific released

Hi, For some reason backport bot is't created backport issue for this, then ticket just closed without pacific backport k > On 17 Sep 2021, at 13:34, Adrian Nicolae wrote: > > Hi, > > Does the 16.2.6 version fixed the following bug : > > https://github.com/ceph/ceph/pull/42690 >

[ceph-users] Re: v16.2.6 Pacific released

Hi, Have you released a procedure to safely upgrade the cluster? Or just launching "apt upgrade" randomly we do the job? Thanks. F. Il 16.09.21 21:48, David Galloway ha scritto: We're happy to announce the 6th backport release in the Pacific series. We recommend users to update to this rele

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

Thanks Patrick, Similar to Robert, when trying that, I simply receive "Error EINVAL: adding a feature requires a feature string" 10x times. I attempted to downgrade, but wasn't able to successfully get my mons to come back up, as they had quincy specific "mon data structure changes" or something

[ceph-users] Re: HEALTH_WARN: failed to probe daemons or devices after upgrade to 16.2.6

Was there a MON running previously on that host? Do you see the daemon when running 'cephadm ls'? If so, remove it with 'cephadm rm-daemon --name mon.s-26-9-17' Zitat von Fyodor Ustinov : Hi! After upgrading to version 16.2.6, my cluster is in this state: root@s-26-9-19-mon-m1:~# ceph -s

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

Hi, I get the same after upgrading to 16.2.6. All mds daemons are standby. After setting ceph fs set cephfs max_mds 1 ceph fs set cephfs allow_standby_replay false the mds still wants to be standby. 2021-09-17T14:40:59.371+0200 7f810a58f600 0 ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c1449

[ceph-users] Re: v16.2.6 Pacific released

Hi Konstantin, Orchestrator issues don't get their own backport trackers because the team lead handles these backports and does them in batches. This patch did make it into the 16.2.6 release via this batch backport PR: https://github.com/ceph/ceph/pull/43029 -Cory On Fri, Sep 17, 2021 at 6:

[ceph-users] Re: v16.2.6 Pacific released

Thanks Cory, Adrian, FYI k > On 17 Sep 2021, at 16:15, Cory Snyder wrote: > > Orchestrator issues don't get their own backport trackers because the team > lead handles these backports and does them in batches. This patch did make it > into the 16.2.6 release via this batch backport PR: > >

[ceph-users] September Ceph Science Virtual User Group Meeting

Hey all, We will be having a Ceph science/research/big cluster call on Wednesday September 22nd. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mos

[ceph-users] Re: CentOS Linux 8 EOL

Currently, we on CentOS8 Stream use usual Ceph repo: [root@k8s-prod-worker0 /]# dnf info ceph-osd Last metadata expiration check: 0:00:06 ago on Fri 17 Sep 2021 08:44:30 PM +07. Available Packages Name : ceph-osd Epoch: 2 Version : 16.2.5 Release : 0.el8 Architecture :

[ceph-users] Re: Is it normal Ceph reports "Degraded data redundancy" in normal use?

Since I'm trying to test different erasure encoding plugin and technique I don't want the balancer active. So I tried setting it to none as Eguene suggested, and to my surprise I did not get any degraded messages at all, and the cluster was in HEALTH_OK the whole time. Interesting, maybe th

[ceph-users] debugging radosgw sync errors

Hello again, as my tests with some fresh clusters answerd most of my config questions, I now wanted to start with our production cluster and the basic setup looks good, but the sync does not work: [root@3cecef5afb05 ~]# radosgw-admin sync status realm 5d6f2ea4-b84a-459b-bce2-bccac338b3e

[ceph-users] Re: v16.2.6 Pacific released

Hi, Does the 16.2.6 version fixed the following bug : https://github.com/ceph/ceph/pull/42690 ? It's not listed in the changelog. Message: 3 Date: Thu, 16 Sep 2021 15:48:42 -0400 From: David Galloway Subject: [ceph-users] v16.2.6 Pacific released To: ceph-annou...@ceph.io, ceph-users@ceph

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

On Fri, Sep 17, 2021 at 5:54 AM Robert Sander wrote: > > Hi, > > I had to run > > ceph fs set cephfs max_mds 1 > ceph fs set cephfs allow_standby_replay false > > and stop all MDS and NFS containers and start one after the other again > to clear this issue. Stopping NFS should not have been neces

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

On Fri, Sep 17, 2021 at 8:54 AM Eric Dold wrote: > > Hi, > > I get the same after upgrading to 16.2.6. All mds daemons are standby. > > After setting > ceph fs set cephfs max_mds 1 > ceph fs set cephfs allow_standby_replay false > the mds still wants to be standby. > > 2021-09-17T14:40:59.371+0200

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

On Fri, Sep 17, 2021 at 8:19 AM Joshua West wrote: > > Thanks Patrick, > > Similar to Robert, when trying that, I simply receive "Error EINVAL: > adding a feature requires a feature string" 10x times. > > I attempted to downgrade, but wasn't able to successfully get my mons > to come back up, as t

[ceph-users] 回复: Re: Cephfs - MDS all up:standby, not becoming up:active

We are experiencing the same when upgrading to 16.2.6 with cephadm. I tried ceph fs set cephfs max_mds 1 ceph fs set cephfs allow_standby_replay false , but still all MDS goes to standby. It seems all ranks are marked failed. Do we have a way to clear this flag? Please help. Our cluster is dow

[ceph-users] Re: radosgw find buckets which use the s3website feature

Found it: for bucket in `radosgw-admin metadata list bucket.instance | jq .[] | cut -f2 -d\"`; do if radosgw-admin metadata get --metadata-key=bucket.instance:$bucket | grep --silent website_conf; then echo $bucket fi done Am Do., 16. Sept. 2021 um 09:49 Uhr schrieb Boris Behrens : > Hi

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

On 17.09.21 16:40, Patrick Donnelly wrote: Stopping NFS should not have been necessary. But, yes, reducing max_mds to 1 and disabling allow_standby_replay is required. See: https://docs.ceph.com/en/pacific/cephfs/upgrading/#upgrading-the-mds-cluster I do no read upgrade notes any more because

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

> > Stopping NFS should not have been necessary. But, yes, reducing > > max_mds to 1 and disabling allow_standby_replay is required. See: > > https://docs.ceph.com/en/pacific/cephfs/upgrading/#upgrading-the-mds- > cluster > > I do no read upgrade notes any more because I just run > > ceph orch up

[ceph-users] Re: debugging radosgw sync errors

While searching for other things I came across this: [root ~]# radosgw-admin metadata list bucket | grep www1 "www1", [root ~]# radosgw-admin metadata list bucket.instance | grep www1 "www1:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.81095307.31103", "www1.company.dev", [root ~]# radosgw-admin

[ceph-users] aws-sdk-cpp-s3 alternative for ceph

I was wondering if there some patched aws-sdk that allows it to be used with ceph rgw. For instance removes such things: :EC2MetadataClient: Can not retrieve resource from http://169.254.169.254/latest/meta-data/placement/availability-zone ___ ce

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

On Fri, Sep 17, 2021 at 11:11 AM 胡 玮文 wrote: > > We are experiencing the same when upgrading to 16.2.6 with cephadm. > > > > I tried > > > > ceph fs set cephfs max_mds 1 > > ceph fs set cephfs allow_standby_replay false > > > > , but still all MDS goes to standby. It seems all ranks are marked fai

[ceph-users] Ceph Community Ambassador Sync

Hi everyone, We first introduced the Ceph Community Ambassador program in Ceph Month back in June. The group is planning to meet for the first time on September 23rd at 6:00 UTC to sync on ideas and what's going on in their particular region. This is an open event on the Ceph community calendar:

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

Thank you very much. But the mds still don’t go active. While trying to resolve this, I run: ceph mds rmfailed 0 --yes-i-really-mean-it ceph mds rmfailed 1 --yes-i-really-mean-it Then 3 out of 5 MONs crashed. I was able to keep MON up by making MDSMonitor::maybe_resize_cluster return false direc

[ceph-users] Re: Ceph Community Ambassador Sync

Hello Mike Where can we find a list of ambassadors and their perspective region? I ask that to know if our region has someone who present us. Thank you On Fri, 17 Sep 2021, 19:25 Mike Perez, wrote: > Hi everyone, > > We first introduced the Ceph Community Ambassador program in Ceph > Month

[ceph-users] Re: Replacing swift with RGW

Hello Eugen Thank you very much for your guidance and support , now everything is working fine , RGW has been replaced swift as I wanted. Michel On Thu, 9 Sep 2021, 13:59 Michel Niyoyita, wrote: > Hello Eugen, > > Are there other config done on the OpenStack side except creating > endpoints?I

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

On Fri, Sep 17, 2021 at 2:32 PM 胡 玮文 wrote: > > Thank you very much. But the mds still don’t go active. Did you run the command I suggested before or after you executed `rmfailed` below? > While trying to resolve this, I run: > > ceph mds rmfailed 0 --yes-i-really-mean-it > > ceph mds rmfailed 1

[ceph-users] 回复: Cephfs - MDS all up:standby, not becoming up:active

> Did you run the command I suggested before or after you executed `rmfailed` > below? I run “rmfailed” before reading your mail. Then I got MON crashed. I fixed the crash by setting max_mds=2. Then I tried the command you suggested. By reading the code[1], I think I really need to undo the “rm

[ceph-users] Re: HEALTH_WARN: failed to probe daemons or devices after upgrade to 16.2.6

Hi! > Was there a MON running previously on that host? Do you see the daemon > when running 'cephadm ls'? If so, remove it with 'cephadm rm-daemon > --name mon.s-26-9-17' Hmm. 'cephadm ls' running directly on the node does show that there is mon. I don't quite understand where it came from and I

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

On Fri, Sep 17, 2021 at 3:17 PM 胡 玮文 wrote: > > > Did you run the command I suggested before or after you executed `rmfailed` > > below? > > > > I run “rmfailed” before reading your mail. Then I got MON crashed. I fixed > the crash by setting max_mds=2. Then I tried the command you suggested. >

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

Thanks again. Now my CephFS is back online! I ended up build ceph-mon from source myself, with the following patch applied. and only replacing the mon leader seems sufficient. Now I’m interested in why such a routine automated minor version upgrade could get the cluster into such a state in the

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

Hi Patrick Here's the output of ceph fs dump: e226256 enable_multiple, ever_enabled_multiple: 0,1 default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

On Fri, Sep 17, 2021 at 6:57 PM Eric Dold wrote: > > Hi Patrick > > Here's the output of ceph fs dump: > > e226256 > enable_multiple, ever_enabled_multiple: 0,1 > default compat: compat={},rocompat={},incompat={1=base v0.20,2=client > writeable ranges,3=default file layouts on dirs,4=dir inode in

[ceph-users] Re: Cephfs - MDS all up:standby, not becoming up:active

On Fri, Sep 17, 2021 at 11:30 AM Robert Sander wrote: > > On 17.09.21 16:40, Patrick Donnelly wrote: > > > Stopping NFS should not have been necessary. But, yes, reducing > > max_mds to 1 and disabling allow_standby_replay is required. See: > > https://docs.ceph.com/en/pacific/cephfs/upgrading/#up