[ceph-users] Re: MDS stuck ops

Hi Venky and Patrick, thanks Venky for your explanation. Now I understand, hopefully. The difference is that: - octopus: every immediate child of /home gets an individual directory fragment that is pinned - pacific: every immediate (not recursive) directory fragment of /home gets an individual

[ceph-users] Re: PGs stuck down

Hi, while I basically agree with Frank's response (e. g. min_size = 2) I disagree that it won't work without the stretch mode. We have a customer with a similar setup, two datacenters and a third mon in a different location. And this setup has proven multiple times the resiliency of ceph.

[ceph-users] Re: PGs stuck down

Hi Eugen, power outage is one thing, a cable cut is another. With power outages you will have OSDs down and only one sub-cluster up at a time. OSD's will peer locally on a single DC and stuff moves on. With a cable cut you have a split brain. Have you actually tested your setup with everything

[ceph-users] Re: PGs stuck down

Hi all, It's difficult to say exactly what happened here without cluster logs. Dale, would you be able to share the ceph.log showing the start of the incident? Cheers, dan On Wed, Nov 30, 2022 at 10:30 AM Frank Schilder wrote: > > Hi Eugen, > > power outage is one thing, a cable cut is another.

[ceph-users] Re: PGs stuck down

Hi, thanks for the clarification, I missed the cable cut part, I should read more carefully before responding. ;-) I don't really know which tests were performed because we joined that project at a later phase, but it makes sense with the two subclusters. Thanks, Eugen Zitat von Frank Sc

[ceph-users] osd set-require-min-compat-client

Hey guys, our ceph cluster is on pacific, but started on jewel years ago. While i was going through the logs of the mrg daemon i stumbled about the following entry: = [balancer ERROR root] execute error: r = -1, detail = min_compat_client jewel < luminous, which is required for pg-upmap

[ceph-users] Re: osd set-require-min-compat-client

Hi Felix, This change won't trigger any rebalancing. It will prevent older clients from connecting, but since this isn't a crush tunable it won't directly affect data placement. Best, Dan On Wed, Nov 30, 2022, 12:33 Stolte, Felix wrote: > Hey guys, > > our ceph cluster is on pacific, but sta

[ceph-users] Cannot create snapshots if RBD image is mapped with -oexclusive

Hello, creating snapshots of RBD images that are mapped with -oexclusive seems not to be possible: # rbd map -oexclusive rbd.blu1/andreasspielt-share11 /dev/rbd7 # rbd snap create rbd.blu1/andreasspielt-share11@ateuchert_test01 Creating snap: 0% complete...failed. rbd: failed to create snapsho

[ceph-users] Re: osd set-require-min-compat-client

Hi Dan, thanks for your reply. I wasn’t worried about the setting itself, but about the balancer starting to use the pg-upmap feature (which currently fails, because of the jewel setting). I would assume though, that the balancer is using pg-upmap in a throttled way to avoid performance issues.

[ceph-users] Re: osd set-require-min-compat-client

Hi Felix, With `ceph balancer off` the upmap balancer will not move any PGs around. https://docs.ceph.com/en/latest/rados/operations/balancer/ Cheers, Dan On Wed, Nov 30, 2022 at 1:20 PM Stolte, Felix wrote: > > Hi Dan, > > thanks for your reply. I wasn’t worried about the setting itself, but

[ceph-users] Cache modes libvirt

Hi, I was wondering... In Ceph/Libvirt docs only cachmodes writetrough and writeback are discussed. My clients's disks are all set to writeback in the libvirt client xml-definition. For a backup operation, I notice a severe lag on one of my VM's. Such a backup operation that takes 1 to 2 hours

[ceph-users] Quincy 17.2.5: proper way to replace OSD (HDD with Wal/DB on SSD)

Ubuntu 20.04, Ceph 17.2.5, dockerized Hello all, this is frequently asked, but the answers I found are either old or do not cover an extra WAL/DB device. Given an OSD that is located on a HDD, where WAL/DB is located on a SSD, which is used by all OSDs of the host. The OSD is in, up and running.

[ceph-users] Re: osd removal leaves 'stray daemon'

I typically don't see this when I do OSD replacement. If you do a mgr failover ("ceph mgr fail") and wait a few minutes does this still show up? The stray daemon/host warning is roughly equivalent to comparing the daemons in `ceph node ls` and `ceph orch ps` and seeing if there's anything in the fo

[ceph-users] Re: Cache modes libvirt

Some information is missing to give a helpful answer. How do you backup? (Files? RBD via Ceph? Block Device with qemu-img? Which Device driver do you use (Virtio? SATA?). In our production we use Virtio RBD and the Hypervisor standard cache mode. The Disks are snapshoted before the backup with 'q

[ceph-users] Re: osd set-require-min-compat-client

When we switched (Was using the compat balancer previously), I: 1. turned off the balancer 2. forced the client minimum (new centos7 clients are ok being forced to luminious even though they report as jewel. There's an email thread elsewhere describing it) 3. slowly reweighted the crush compat w

[ceph-users] MDS crashes to damaged metadata

Hey guys, our mds daemons are crashing constantly when someone is trying to delete a file: -26> 2022-11-29T12:32:58.807+0100 7f081b458700 -1 /build/ceph-16.2.10/src/mds/Server.cc: In function 'void Server::_unlink_local(MDRequestRef&, CDentry*, CDentry*)' thread 7f081b458700

[ceph-users] opnesuse rpm repos

Hello, I ran the following on OpenSuse cephadm add-repo --release quincy The command landed /etc/zypp/repos.d/ceph.repo and contents [Ceph] name=Ceph $basearch baseurl=https://download.ceph.com/rpm-quincy/opensuse/$basearch enabled=1 gpgcheck=1 gpgkey=https://download.ceph.com/keys/release.gpg

[ceph-users] Re: MDS crashes to damaged metadata

On Wed, Nov 30, 2022 at 3:10 PM Stolte, Felix wrote: > > Hey guys, > > our mds daemons are crashing constantly when someone is trying to delete a > file: > > -26> 2022-11-29T12:32:58.807+0100 7f081b458700 -1 > /build/ceph-16.2.10/src/mds/Server.cc: In function 'void > Server:

[ceph-users] Re: MDS crashes to damaged metadata

Hi Patrick, it does seem like it. We are not using postgres on cephfs as far as i know. We narrowed it down to three damaged inodes, but files in question had been xlsx, pdf or pst. Do you have any suggestion how to fix this? Is there a way to scan the cephfs for damaged inodes?

[ceph-users] Re: MDS crashes to damaged metadata

You can run this tool. Be sure to read the comments. https://github.com/ceph/ceph/blob/main/src/tools/cephfs/first-damage.py As of now what causes the damage is not yet known but we are trying to reproduce it. If your workload reliably produces the damage, a debug_mds=20 MDS log would be extremel

[ceph-users] OSD booting gets stuck after log_to_monitors step

Dear experts, Recently, we suffered network problem due to switch H/W failure and caused massive OSD offline. However, after the recovery of network, several of OSD are unable to join back, resulting some down or unknown PGs. Restarting OSD doesn't help. It will end up with sticking at booting

[ceph-users] Re: OSD booting gets stuck after log_to_monitors step

Dear experts, Sorry, I missed to mention that the initial symptom is that those OSDs will suffer: "wait_auth_rotating timed out" and "unable to obtain rotating service keys; retrying" I then increased rotating_keys_bootstrap_timeout, but it doesn't really help. Best regards, Felix Lee ~ On

[ceph-users] Tuning CephFS on NVME for HPC / IO500

Dear all, I am currently creating a CephFS setup for a HPC setting. I have a Ceph v17.2.5 Cluster on Rocky Linux 8.7 (Kernel 4.18.0-425.3.1.el8.x86_64) deployed with cephadm. I have 10 Ceph nodes with 2x100GbE LAG interconnect and 36 client nodes with 2x25GbE LAG interconnect. We have Dell NOS10 s

[ceph-users] Re: Tuning CephFS on NVME for HPC / IO500

Hi, On 2022-12-01 8:26, Manuel Holtgrewe wrote: The Ceph cluster nodes have 10x enterprise NVMEs each (all branded as "Dell enterprise disks"), 8 older nodes (last year) have "Dell Ent NVMe v2 AGN RI U.2 15.36TB" which are Samsung disks, 2 newer nodes (just delivered) have "Dell Ent NVMe CM6