[ceph-users] How to reset compat weight-set changes caused by PG balancer module?

Apparently the PG balancer crush-compat mode adds some crush bucket weights. Those cause major havoc in our cluster, our PG distribution is all over the place. Seeing things like this:... 97 hdd 9.09470 1.0 9.1 TiB 6.3 TiB 6.3 TiB 32 KiB 17 GiB 2.8 TiB 69.03 1.08 28 up 98 hdd

[ceph-users] TCMU Runner: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown

Hi, i'm running a ceph cluster with 4x ISCSI exporter nodes and oVirt on the client side. In the tcmu-runner logs i the the following happening every few seconds: ### 2019-10-22 10:11:11.231 1710 [WARN] tcmu_rbd_lock:762 rbd/image.lun0: Acquired exclusive lock. 2019-10-22 10:11:11.395 1710

[ceph-users] Updating crush location on all nodes of a cluster

Hello everyone, We have a Ceph cluster (running 14.2.2) which has already dozens of TB of data and... we did not set the location of the OSD hosts. The hosts are located in 2 datacenters. We would like to update the locations of all the hosts so not all replicas end up in a single DC. We are wond

Re: [ceph-users] Decreasing the impact of reweighting osds

getting rid of filestore solves most latency spike issues during recovery because they are often caused by random XFS hangs (splitting dirs or just xfs having a bad day) Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h

Re: [ceph-users] Updating crush location on all nodes of a cluster

Hello Alexandre, maybe you take a look into https://www.youtube.com/watch?v=V33f7ipw9d4 where you can see how easy Ceph CRUSH can be managed. 1. Changing the locations of all hosts at once > We are worried that this will generate too much IO and network activity > (and there is no way to pause /

Re: [ceph-users] Updating crush location on all nodes of a cluster

Hey Martin, Alright then, we'll just go with the update of every osd's location at once then; just wanted to be sure this was not a problem. :) On Tue, Oct 22, 2019 at 1:21 PM Martin Verges wrote: > Hello Alexandre, > > maybe you take a look into https://www.youtube.com/watch?v=V33f7ipw9d4 wher

Re: [ceph-users] TCMU Runner: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown

On 10/22/2019 03:20 AM, Kilian Ries wrote: > Hi, > > > i'm running a ceph cluster with 4x ISCSI exporter nodes and oVirt on the > client side. In the tcmu-runner logs i the the following happening every > few seconds: > > Are you exporting a LUN to one client or multiple clients at the same ti

Re: [ceph-users] clust recovery stuck

hi, On 2019-10-22 08:05, Eugen Block wrote: > Hi, > > can you share `ceph osd tree`? What crush rules are in use in your > cluster? I assume that the two failed OSDs prevent the remapping because > the rules can't be applied. > ceph osd tree gives: ID WEIGHT TYPE NAMEUP/DOWN REWE

Re: [ceph-users] clust recovery stuck

Hi Philipp, Given 256 PG's triple replicated onto 4 OSD's you might be encountering the "PG overdose protection" of OSDs.  Take a look at 'ceph osd df' and see the number of PG's that are mapped to each OSD (last column or near the last).  The default limit is 200, so if any OSD exceeds that,

Re: [ceph-users] TCMU Runner: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown

- Each LUN is exported to multiple clients (at the same time) - yes, IO is done to the LUNs (read and write); (oVirt runs VMs on each of the LUNs) Ok, i'll update this tomorrow with the logs you asked for ... Von: Mike Christie Gesendet: Dienstag, 22. Oktober

Re: [ceph-users] TCMU Runner: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown

Ignore my log request. I think I know what is going on. Let me do some testing here and I will make a test rpm for you. On 10/22/2019 04:38 PM, Kilian Ries wrote: > - Each LUN is exported to multiple clients (at the same time) > > - yes, IO is done to the LUNs (read and write); (oVirt runs VMs

Re: [ceph-users] Decreasing the impact of reweighting osds

Thanks - that's a good suggestion! However I'd still like to know the answers to my 2 questions. regards Mark On 22/10/19 11:22 pm, Paul Emmerich wrote: getting rid of filestore solves most latency spike issues during recovery because they are often caused by random XFS hangs (splitting dirs

Re: [ceph-users] How to reset compat weight-set changes caused by PG balancer module?

Apparently the PG balancer crush-compat mode adds some crush bucket weights. Those cause major havoc in our cluster, our PG distribution is all over the place. Seeing things like this:... 97 hdd 9.09470 1.0 9.1 TiB 6.3 TiB 6.3 TiB 32 KiB 17 GiB 2.8 TiB 69.03 1.08 28 up 98 hd

Re: [ceph-users] Decreasing the impact of reweighting osds

Most times you are better served with simpler settings like osd_recovery_sleep, which has 3 variants if you have multiple types of OSDs in your cluster (osd_recovery_sleep_hdd, osd_recovery_sleep_sdd, osd_recovery_sleep_hybrid). Using those you can tweak a specific type of OSD that might be having

Re: [ceph-users] ceph balancer do not start

Of the top of my head, if say your cluster might have the wrong tunables for crush-compat. I know I ran into that when I first set up the balancer and nothing obviously said that was the problem. Only researching find it for me. My real question, though, is why aren't you using upmap? It is signif

[ceph-users] How does IOPS/latency scale for additional OSDs? (Intel S3610 SATA SSD, for block storage use case)

Hi, I'm running a 3-node Ceph cluster for VM block storage (Proxmox/KVM). Replication is set to 3. Previously, we were running 1 x Intel Optane 905P 960B disk p

Re: [ceph-users] Crashed MDS (segfault)

On Tue, Oct 22, 2019 at 1:49 AM Gustavo Tonini wrote: > > Is there a possibility to lose data if I use "cephfs-data-scan init > --force-init"? > It only causes incorrect stat on root inode, can't cause data lose. running 'ceph daemon mds.a scrub_path / force repair' after mds restart can fix t

Re: [ceph-users] ceph balancer do not start

 Hi David, thank you. Unfortunately i have the problem that i can't use the upmap mode at the moment.   root@ceph-mgr:~# ceph balancer mode upmap root@ceph-mgr:~# ceph balancer optimize myplan root@ceph-mgr:~# ceph balancer show myplan # starting osdmap epoch 409753 # starting crush version 84