[ceph-users] Re: "ceph orch restart mgr" creates manager daemon restart loop
Hi Roman, what ceph version are you on? Also, when you ran the restart command originally, did you get a message about scheduling the restarts or no output? On Tue, Nov 23, 2021 at 6:04 AM Roman Steinhart wrote: > Hi all, > > while digging down another issue I had with the managers I restarted them > using "ceph orch restart mgr". > After running that command the main manager is now in a restart loop. > The only way for me to stop this is running "ceph orch pause", as soon as I > do "ceph orch resume" the loop starts again. > > Has anyone a suggestion on how I can remove that stuck "restart" job? > "ceph orch cancel" does not work, it returns "This Orchestrator does not > support `orch cancel`" > > While googling for this issue it seems I'm not the first one having that > issue: > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/IRC4ZPSSZYELWPU5D2FHKWJ2VU7IP3JG/ > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/WZUWZGVYPFII5HTUTW6OAQXIZ6VNT2E2/ > > See the logs below: > user@ceph1:~# journalctl -fu > ceph-6d588189-f434-4cb1-8c60-6e48cbf43a2a@mgr.ceph1.service --since "1 day > ago" -g "Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a" > Nov 22 15:31:38 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:31:51 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:31:51 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:34:49 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:35:00 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:35:00 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:38:39 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:38:50 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:38:50 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:43:35 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:43:46 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:43:46 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:50:19 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:50:30 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:50:30 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:58:26 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:58:37 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:58:37 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:10:16 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:10:28 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:10:28 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:17:17 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:17:29 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:17:29 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:40:52 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:41:07 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:41:07 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:45:34 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:45:47 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:45:47 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:49:34 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:49:39 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:49:39 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:51:47 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:51:51 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:51:51 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:54:02 ceph1 systemd[1]: Stopping Ceph mgr.c
[ceph-users] Re: have buckets with low number of shards
Hi Josh Thanks for your response. Do you have any advice how to reshard these big buckets so it doesn't cause any down time in our cluster? Resharding these buckets makes a lots of slow ops in deleting old shard phase and the cluster can't responde to any requests till resharding is completely done. Regards, Mahnoosh On Tue, Nov 23, 2021, 5:28 PM Josh Baergen wrote: > Hey Mahnoosh, > > > Running cluster in octopus 15.2.12 . We have a big bucket with about 800M > > objects and resharding this bucket makes many slow ops in our bucket > index > > osds. I wanna know what happens if I don't reshard this bucket any more? > > How does it affect the performance? The performance problem would be only > > for that bucket or it affects the entire bucket index pool? > > Unfortunately, if you don't reshard the bucket, it's likely that > you'll see widespread index pool performance and stability issues, > generally manifesting as one or more OSDs becoming very busy to the > point of holding up traffic for multiple buckets or even flapping (the > OSD briefly gets marked down), leading to recovery. Recovering large > index shards can itself cause issues like this to occur. Although the > official recommendation, IIRC, is 100K objects per index shard, the > exact objects per shard count at which one starts to experience these > sorts of issues highly depends on the hardware involved and user > workload. > > Josh > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: have buckets with low number of shards
Manoosh; You can't reshard a bucket without downtime. During a reshard RGW creates new RADOS objects to match the new shard number. Then all the RGW objects are moved from the old RADOS objects to the new RADOS objects, and the original RADOS objects are destroyed. The reshard locks the bucket for the duration. Thank you, Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. dhils...@performair.com www.PerformAir.com -Original Message- From: mahnoosh shahidi [mailto:mahnooosh@gmail.com] Sent: Tuesday, November 23, 2021 8:20 AM To: Josh Baergen Cc: Ceph Users Subject: [ceph-users] Re: have buckets with low number of shards Hi Josh Thanks for your response. Do you have any advice how to reshard these big buckets so it doesn't cause any down time in our cluster? Resharding these buckets makes a lots of slow ops in deleting old shard phase and the cluster can't responde to any requests till resharding is completely done. Regards, Mahnoosh On Tue, Nov 23, 2021, 5:28 PM Josh Baergen wrote: > Hey Mahnoosh, > > > Running cluster in octopus 15.2.12 . We have a big bucket with about 800M > > objects and resharding this bucket makes many slow ops in our bucket > index > > osds. I wanna know what happens if I don't reshard this bucket any more? > > How does it affect the performance? The performance problem would be only > > for that bucket or it affects the entire bucket index pool? > > Unfortunately, if you don't reshard the bucket, it's likely that > you'll see widespread index pool performance and stability issues, > generally manifesting as one or more OSDs becoming very busy to the > point of holding up traffic for multiple buckets or even flapping (the > OSD briefly gets marked down), leading to recovery. Recovering large > index shards can itself cause issues like this to occur. Although the > official recommendation, IIRC, is 100K objects per index shard, the > exact objects per shard count at which one starts to experience these > sorts of issues highly depends on the hardware involved and user > workload. > > Josh > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW support IAM user authentication
Hi Pritha - or anyone who knows,
I too have problems with IAM, in particular with AssumeRoleWithWebIdentity.
I am running the master branch version of Ceph because it looks like it
includes code related to the functionality described at
https://docs.ceph.com/en/latest/radosgw/STS/ - code which is not in any
released version, even 17.0.
Looking at the code on that page, there appear to be at least two errors:
(1) an instance of "client" should be "sts_client" (or vice versa)
(2) an access key and secret key are specified when creating sts_client,
which is unnecessary and therefore confusing: only the access token is used
or should be required for assume_role_with_web_identity
But I still cannot get the AssumeRoleWithWebIdentity code example to work.
The RGW debug logs show
debug 2021-11-23T15:51:22.247+ 7fad6e351700 0 evaluating policy for
op: 93 returned deny/pass
In my case, the policy_document and role_policy are
policy_document =
'''{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Federated":["arn:aws:iam:::oidc-provider/proteus.ves.corp/auth/realms/cno"]},"Action":["sts:AssumeRoleWithWebIdentity"],"Condition":{"StringEquals":{"proteus.ves.corp/auth/realms/cno:app_id":"ceph_rgw"}}}]}'''
role_policy =
'''{"Version":"2012-10-17","Statement":{"Effect":"Allow","Action":"s3:*","Resource":"arn:aws:s3:::*"}}'''
(I assume it is only the former that may be relevant here, but maybe I'm
wrong.)
In /etc/ceph/ceph.conf I have
[client.radosgw.gateway]
rgw sts key = abcdefghijklmnop
rgw s3 auth use sts = true
In the debug I can see the token from Keycloak looks like (after formatting
it)
{
"exp": 1637677729,
"iat": 1637677429,
"jti": "06e5422e-8395-4727-9366-a851c3f5930f",
"iss": "https://proteus.ves.corp/auth/realms/cno";,
"aud": "account",
"sub": "f45bae70-1517-48f6-9d75-af7f421f4a0c",
"typ": "Bearer",
"azp": "ceph_rgw",
"session_state": "1413beec-9785-4e63-947f-72eb26da9daf",
"acr": "1",
"allowed-origins": [
"*"
],
"realm_access": {
"roles": [
"offline_access",
"uma_authorization"
]
},
"resource_access": {
"ceph_rgw": {
"roles": [
"arn:aws:iam:::role/S3Access",
"S3Access"
]
},
"account": {
"roles": [
"manage-account",
"manage-account-links",
"view-profile"
]
}
},
"scope": "openid profile email",
"email_verified": true,
"name": "testuser",
"preferred_username": "testuser",
"given_name": "testuser",
"email": "test-u...@help-me-please.com"
}
Please, if you are familiar with this, can you tell me what step is
missing? There is no description on that page of what should be done at
Keycloak, so I'm guessing the problem may be there. (Keycloak screens are
shown elsewhere, but for a different example.) I have spent a good deal of
time trying to understand this, so if you could help I would greatly
appreciate it.
Kind regards,
Michael
On Tue, 23 Nov 2021 at 06:22, Pritha Srivastava wrote:
> Hi Nio,
>
> Can you provide more details around what you are trying to do?
>
> RGW supports attaching IAM policies to users that aid in managing their
> permissions.
>
> Thanks,
> Pritha
>
> On Tue, Nov 23, 2021 at 11:43 AM nio wrote:
>
> > hi，all:
> > In the process of using RGW, I still cannot authenticate users
> through
> > IAM. In the near future, will RGW support IAM to manage user permissions
> > and authentication functions?
> >
> >
> > Looking forward to your reply 😁
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
--
CONFIDENTIALITY
This e-mail message and any attachments thereto, is
intended only for use by the addressee(s) named herein and may contain
legally privileged and/or confidential information. If you are not the
intended recipient of this e-mail message, you are hereby notified that any
dissemination, distribution or copying of this e-mail message, and any
attachments thereto, is strictly prohibited. If you have received this
e-mail message in error, please immediately notify the sender and
permanently delete the original and any copies of this email and any prints
thereof.
ABSENT AN EXPRESS STATEMENT TO THE CONTRARY HEREINABOVE, THIS
E-MAIL IS NOT INTENDED AS A SUBSTITUTE FOR A WRITING. Notwithstanding the
Uniform Electronic Transactions Act or the applicability of any other law
of similar substance and effect, absent an express statement to the
contrary hereinabove, this e-mail message its contents, and any attachments
hereto are not intended to represent an offer or acceptance to enter into a
contract and are not otherwise intended to bind the sender, Sanmina
Corporation (or any of its subsidiaries), or any other person or entity.
___
[ceph-users] Re: have buckets with low number of shards
Hi Dominic, Thanks for explanation but I didn't mean the bucket lock which happens during the reshard. My problem is when number of objects in a bucket is about 500M and more than that, deleting those old RADOS objects in the reshard process, causes slow ops which results in osd failures so we experience down time in the whole cluster not only in the resharded bucket. Thanks On Tue, Nov 23, 2021, 7:43 PM wrote: > Manoosh; > > You can't reshard a bucket without downtime. During a reshard RGW creates > new RADOS objects to match the new shard number. Then all the RGW objects > are moved from the old RADOS objects to the new RADOS objects, and the > original RADOS objects are destroyed. The reshard locks the bucket for the > duration. > > Thank you, > > Dominic L. Hilsbos, MBA > Vice President - Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > > > -Original Message- > From: mahnoosh shahidi [mailto:mahnooosh@gmail.com] > Sent: Tuesday, November 23, 2021 8:20 AM > To: Josh Baergen > Cc: Ceph Users > Subject: [ceph-users] Re: have buckets with low number of shards > > Hi Josh > > Thanks for your response. Do you have any advice how to reshard these big > buckets so it doesn't cause any down time in our cluster? Resharding these > buckets makes a lots of slow ops in deleting old shard phase and the > cluster can't responde to any requests till resharding is completely done. > > Regards, > Mahnoosh > > On Tue, Nov 23, 2021, 5:28 PM Josh Baergen > wrote: > > > Hey Mahnoosh, > > > > > Running cluster in octopus 15.2.12 . We have a big bucket with about > 800M > > > objects and resharding this bucket makes many slow ops in our bucket > > index > > > osds. I wanna know what happens if I don't reshard this bucket any > more? > > > How does it affect the performance? The performance problem would be > only > > > for that bucket or it affects the entire bucket index pool? > > > > Unfortunately, if you don't reshard the bucket, it's likely that > > you'll see widespread index pool performance and stability issues, > > generally manifesting as one or more OSDs becoming very busy to the > > point of holding up traffic for multiple buckets or even flapping (the > > OSD briefly gets marked down), leading to recovery. Recovering large > > index shards can itself cause issues like this to occur. Although the > > official recommendation, IIRC, is 100K objects per index shard, the > > exact objects per shard count at which one starts to experience these > > sorts of issues highly depends on the hardware involved and user > > workload. > > > > Josh > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] DACH Ceph Meetup
Hi everyone, There will be a virtual Ceph Meetup taking place on November 30th at 16:00 UTC. Take a look at the excellent lineup of speakers we have and register. https://ceph.io/en/community/events/2021/meetup-dach-2021-11-30/ P.S. This is an opportunity to claim a free Ceph Pacific release shirt. More details to follow during the event. -- Mike Perez ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW support IAM user authentication
Hi Michael,
My responses are inline:
On Tue, Nov 23, 2021 at 10:07 PM Michael Breen <
michael.br...@vikingenterprise.com> wrote:
> Hi Pritha - or anyone who knows,
>
> I too have problems with IAM, in particular with AssumeRoleWithWebIdentity.
>
> I am running the master branch version of Ceph because it looks like it
> includes code related to the functionality described at
> https://docs.ceph.com/en/latest/radosgw/STS/ - code which is not in any
> released version, even 17.0.
>
> Looking at the code on that page, there appear to be at least two errors:
> (1) an instance of "client" should be "sts_client" (or vice versa)
> (2) an access key and secret key are specified when creating sts_client,
> which is unnecessary and therefore confusing: only the access token is used
> or should be required for assume_role_with_web_identity
>
I agree, these can be corrected. However I don't understand what you mean
by saying that this - "code which is not in any released version, even
17.0."
>
> But I still cannot get the AssumeRoleWithWebIdentity code example to work.
> The RGW debug logs show
>
> debug 2021-11-23T15:51:22.247+ 7fad6e351700 0 evaluating policy for
> op: 93 returned deny/pass
>
> In my case, the policy_document and role_policy are
>
> policy_document =
> '''{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Federated":["arn:aws:iam:::oidc-provider/proteus.ves.corp/auth/realms/cno"]},"Action":["sts:AssumeRoleWithWebIdentity"],"Condition":{"StringEquals":{"proteus.ves.corp/auth/realms/cno:app_id":"ceph_rgw"}}}]}'''
> role_policy =
> '''{"Version":"2012-10-17","Statement":{"Effect":"Allow","Action":"s3:*","Resource":"arn:aws:s3:::*"}}'''
>
>
> (I assume it is only the former that may be relevant here, but maybe I'm
> wrong.)
>
The former is only relevant here. And have you created an openid connect
provider in RGW? ( I am assuming you must have, since it is there in the
examples given). You have set the condition as app_id: ceph_rgw. Whereas
the documentation says that - "The app_id in the condition above must match
the ‘aud’ claim of the incoming token." in the example which uses "app_id"
as a condition element, and the value of "aud" is "account" in the web
token. So please modify the condition accordingly. Also note that other
claims can also now be used in the condition element of the trust policy.
>
> In /etc/ceph/ceph.conf I have
> [client.radosgw.gateway]
> rgw sts key = abcdefghijklmnop
> rgw s3 auth use sts = true
>
> In the debug I can see the token from Keycloak looks like (after
> formatting it)
>
> {
> "exp": 1637677729,
> "iat": 1637677429,
> "jti": "06e5422e-8395-4727-9366-a851c3f5930f",
> "iss": "https://proteus.ves.corp/auth/realms/cno";,
> "aud": "account",
> "sub": "f45bae70-1517-48f6-9d75-af7f421f4a0c",
> "typ": "Bearer",
> "azp": "ceph_rgw",
> "session_state": "1413beec-9785-4e63-947f-72eb26da9daf",
> "acr": "1",
> "allowed-origins": [
> "*"
> ],
> "realm_access": {
> "roles": [
> "offline_access",
> "uma_authorization"
> ]
> },
> "resource_access": {
> "ceph_rgw": {
> "roles": [
> "arn:aws:iam:::role/S3Access",
> "S3Access"
> ]
> },
> "account": {
> "roles": [
> "manage-account",
> "manage-account-links",
> "view-profile"
> ]
> }
> },
> "scope": "openid profile email",
> "email_verified": true,
> "name": "testuser",
> "preferred_username": "testuser",
> "given_name": "testuser",
> "email": "test-u...@help-me-please.com"
> }
>
> Please, if you are familiar with this, can you tell me what step is
> missing? There is no description on that page of what should be done at
> Keycloak, so I'm guessing the problem may be there. (Keycloak screens are
> shown elsewhere, but for a different example.) I have spent a good deal of
> time trying to understand this, so if you could help I would greatly
> appreciate it.
> Kind regards,
> Michael
>
If you can paste further logs here then I will be able to help you. Also
Ceph documentation explains how to get a token from Keycloak and proceed
with that. For any other Keycloak configurations, please refer to Keycloak
documentation.
Thanks,
Pritha
>
> On Tue, 23 Nov 2021 at 06:22, Pritha Srivastava
> wrote:
>
>> Hi Nio,
>>
>> Can you provide more details around what you are trying to do?
>>
>> RGW supports attaching IAM policies to users that aid in managing their
>> permissions.
>>
>> Thanks,
>> Pritha
>>
>> On Tue, Nov 23, 2021 at 11:43 AM nio wrote:
>>
>> > hi，all:
>> > In the process of using RGW, I still cannot authenticate users
>> through
>> > IAM. In the near future, will RGW support IAM to manage user permissions
>> > and authentication functions?
>> >
>> >
>> > Looking forward to your reply 😁
>> > ___
>> > ceph-users mailing list -- ceph-users@ceph.io
>> > To unsubscribe send an email to ceph-users-le...@ceph.io
>>
[ceph-users] Re: One pg stuck in active+undersized+degraded after OSD down
Fiddling with the crush weights sorted this out and I was able to remove the OSD from the cluster. I set all the big weights down to 1 ceph osd crush reweight osd.7 1.0 etc. Tx for all the help On Tue, Nov 23, 2021 at 9:35 AM Stefan Kooman wrote: > On 11/23/21 08:21, David Tinker wrote: > > Yes it recovered when I put the OSD back in. The issue is that it fails > > to sort itself out when I remove that OSD even though I have loads of > > space and 8 other OSDs in 4 different zones to choose from. The weights > > are very different (some 3.2 others 0.36) and that post I found > > suggested that this might cause trouble for the crush algorithm. So I > > was thinking about making them more even before removing the OSD. > > Sure, that sounds like a good plan. If the balancer is not able to > optimize at this point, the improved balancer bij Jonas Jelten might do > the trick here [1], also see ML thread [2]. > > Do you have many different pools? > > What Weiwen Hu said in reply to this threads sounds very plausible. You > might dump your crush map, remove OSD.7, and check if the mappings can > still be made. > > Gr. Stefan > > [1]: https://github.com/TheJJ/ceph-balancer > [2]: > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/QXTFTYRD43XB3JRDSTMH657ISSMZ6QTU/ > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
