[ceph-users] Re: Handling out-of-balance OSD?

ceph pg ls-by-osd k Sent from my iPhone > On 28 Jul 2021, at 12:46, Manuel Holtgrewe wrote: > > How can I find out which pgs are actually on osd.0? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ce

[ceph-users] Re: inbalancing data distribution for osds with custom device class

could anyone has similar experience? BRS | | renjianxinlover | | renjianxinlo...@163.com | 签名由网易邮箱大师定制 On 7/21/2021 11:28，renjianxinlover wrote： Ceph: ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) OS: Linux *** 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3 (2019-09-0

[ceph-users] iSCSI HA (ALUA): Single disk image shared by multiple iSCSI gateways

Hi, I would like to have a Ceph rbd single disk image shared by multiple iSCSI gateways, providing a redundant access mode (active/passive,  ALUA) to a VMware cluster, like an old iSCSI storage array would perform. I've been searching, but is has been hard to find related documentation or any simi

[ceph-users] Orchestrator terminating mgr services

I have two managers deployed, active-stdby, on a couple of the monitors. The managers were stopped with the orchestrator, and as expected, cannot be started without a manager running. I started a manager instance manually using ‘systemctl start ceph-f...@mgr.host.xxx.service

[ceph-users] Re: pool removed_snaps

Hi, IIRC there was a discussion related in the past (in my jewel/luminous days) and the outcome something like this is neccessary for ceph and have to stay. Hth a bit Mehmet Am 14. Juli 2021 11:15:39 MESZ schrieb Seena Fallah : >Hi, > >In ceph osd dump I see many removed_snaps in order of 500k.

[ceph-users] Cephadm and multipath.

I have a number of disk trays, with 25 ssd's in them, these are attached to my servers via a pair of sas cables, so that multipath is used to join the together again and maximize speed etc. Using cephadm how can I create the osd's? It looks like it should be possible to use ceph-volume but I've n

[ceph-users] Re: OSD failed to load OSD map for epoch

OK, thanks! This is the same package as in the Octopus images, so I would expect Pacific to fail just as spectacularly. What's the best way to have this fixed? New issue on the Ceph tracker? I understand the Ceph images use CentOS packages, so should they be poked as well? // Best wishes;

[ceph-users] Re: large directory /var/lib/ceph/$FSID/removed/

Alright, thanks for the confirmation. Zitat von E Taka <0eta...@gmail.com>: Thanks, Eugen! You are right, these directories are the left-overs from playing around with "ceph orch apply …", trying to get rid of the wrong warning message CEPHADM_STRAY_DAEMON. I hadn't noticed that before, that

[ceph-users] Re: Handling out-of-balance OSD?

Ah, the `debug_osd=10` was the missing piece of information for me. It looks like repeering actually triggered the necessary (chaotic ;-)) deletion. With increased log level I'm now seeing the following in the logs: 2021-07-28 12:07:56.661 7f688f301700 10 osd.0 pg_epoch: 284426 pg[3.114s1( v 2798

[ceph-users] Re: Handling out-of-balance OSD?

Yes I expect `ceph pg repeer` would work. Instead of doing all PGs at once, which is sort of chaotic, just pick one PG which is on osd.0 but shouldn't be there. To find that, you need to restart the osd.0 with debug_osd=10 and look for lines like: 10 osd.74 47719 load_pgs loaded pg[2.6d( The

[ceph-users] Re: Handling out-of-balance OSD?

Hi, would it not be simpler to find the "bad" pgs and call "ceph pg repeer" on them to force them to peer? Or is this a different kind of peering than the one you are describing? My approach would be to get a list of ALL pgs and then call "ceph pg repeer" on them. The first command line call gets

[ceph-users] Re: Can single Ceph cluster run on various OS families

icy chan (icy.kf.chan) writes: > > Are there any practices for OS upgrade/migration that can be found from the > official site? > > My drafted plan is: > 1. [CentOS 7] Adopt the cluster by cephadm via upgrading it from Nautilus > to Octopus. > 2. Reinstall the nodes one by one (with new OS)

[ceph-users] Re: Handling out-of-balance OSD?

Wait, after re-reading my own ticket I realized you can more easily remove the leftover PGs by re-peering the *other* osds. "I found a way to remove those leftover PGs (without using ceph-objectstore-tool): If the PG re-peers, then osd.74 notices he's not in the up/acting set then starts deleting

[ceph-users] Re: Handling out-of-balance OSD?

Cool, looks like the second problem is the real issue here :) IIRC, you can remove the leftover PGs with ceph-objectstore-tool. I don't recall the exact syntax, but you'd need to find out which PGs are not mapped there by the current crush rule and remove the others. Or, you can zap and re-create

[ceph-users] Re: Handling out-of-balance OSD?

How "wide" is "wide". I have 4 nodes and 140 HDD OSDs. Here is the info as from the Ceph system: # ceph osd erasure-code-profile get hdd_ec crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=1 plugin=jerasure technique=reed_sol_van w=8 Her

[ceph-users] Re: Handling out-of-balance OSD?

How wide is hdd_ec? With a wide EC rule and relatively few OSDs and relatively few PGs per OSD for the pool, it can be impossible for the balancer to make things perfect. It would help to look at the PG distribution for only the hdd_ec pool -- this script can help https://github.com/cernceph/ceph-s

[ceph-users] Re: Handling out-of-balance OSD?

Hi, thanks for your quick response. I already did this earlier this week: # ceph config dump | grep upmap_max_deviation mgr advanced mgr/balancer/upmap_max_deviation1 Cheers, Manuel On Wed, Jul 28, 2021 at 9:15 AM Dan van der Ster wrote: > Hi, > > Start by setting

[ceph-users] Re: Handling out-of-balance OSD?

Hi, Start by setting: ceph config set mgr mgr/balancer/upmap_max_deviation 1 This configures the balancer to squeeze the OSDs to within 1 PG of eachother. I'm starting to think this should be the default. Cheers, dan On Wed, Jul 28, 2021 at 9:08 AM Manuel Holtgrewe wrote: > > Dear all,

[ceph-users] Re: large directory /var/lib/ceph/$FSID/removed/

Hi, the docs [1] only state: /var/lib/ceph//removed contains old daemon data directories for stateful daemons (e.g., monitor, prometheus) that have been removed by cephadm. So that directory should not grow, I'm not sure if does in your case because you write "now 12 GB". Are you playin

[ceph-users] Handling out-of-balance OSD?

Dear all, I'm running Ceph 14.2.11. I have 140 HDDs in my cluster of 4 nodes, 35 HDDs per node. I am observing fill ratios of 66% to 70% of OSDs and then one with 82% (see attached ceph-osd-df.txt for output of "ceph osd df"). Previously, I had problems with single OSDs filling up to 85% and then