[ceph-users] Re: Cephadm - Adding host to migrated cluster

I didn’t have a chance to run this while things were down. I went ahead and purged the OSDs again, then went to the OSD server and deleted all the osd instance folders in /var/lib/ceph. Waited for the cluster to clear out the daemons and then zapped the drives. The one thing I did differently

[ceph-users] Re: How to remove remaining bucket index shard objects

Thank you for your reply. > the object need only for OMAP data, not for actual data. I believe so. However, OMAP is set for an object, so I think that at least one object exists in PG. Below it appears that the OMAP exists even though the object does not exist. I feel it is strange. Is this co

[ceph-users] cephadm error: add-repo does not have a release file

I followed the instruction in ceph doc But, to summarize, this is why I did. Please note that this error occurs with the previous release, such as Pacific and Nautilus. curl --silent --remote-n

[ceph-users] Understanding rbd objects, with snapshots

Hi, Is there anywhere that describes exactly how rbd data (including snapshots) are stored within a pool? I can see how a rbd broadly stores its data in rados objects in the pool, although the object map is opaque. But once an rbd snap is created and new data written to the rbd, where is the

[ceph-users] Re: Updating Git Submodules -- a documentation question

I think if you are changing commits and/or branches a lot the submodules can end up dirty. An alternative approach is to descend into the submodule root directory and use git commands to work out why it's dirty and fix it but I've always found that more trouble than it's worth. YMMV. On Mon, Oct 1

[ceph-users] kafka notifications

ributes.entry.1.key': 'push-endpoint', 'Attributes.entry.1.value': 'kafka://my-user:@172.24.5.223:9094', 'Attributes.entry.2.key': 'use-ssl', 'Attributes.entry.2.value': 'true', 'Attributes.entry.3.key': '

[ceph-users] Getting started with cephfs-top, how to install

I'd like to see what CephFS clients are doing the most IO. According to this page: https://docs.ceph.com/en/quincy/cephfs/cephfs-top/ - cephfs-top is the simplest way to do this? I enabled 'ceph mgr module enable stats' today, but I'm a bit confused about what the be

[ceph-users] Re: Cephadm - Adding host to migrated cluster

Do the journal logs for the OSDs say anything about why they couldn't start up? ("cephadm ls --no-detail" run on the host will give the systemd units for each daemon on the host so you can get them easier). On Mon, Oct 17, 2022 at 1:37 PM Brent Kennedy wrote: > Below is what the ceph mgr log is

[ceph-users] Re: Cephadm - Adding host to migrated cluster

Sorry, didn't include the cephadm log on the osd node: 2022-10-17 03:38:45,571 7f5e65b33b80 DEBUG /usr/bin/podman: ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable) 2022-10-17 03:38:45,673 7f5e65b33b80 DEBUG systemctl: enabled 2022-10-17 03:38:45,691 7f5e65b33b80 DEBUG

[ceph-users] Re: Cephadm - Adding host to migrated cluster

Below is what the ceph mgr log is saying as soon as I zap the disks and it tries to add them. Note, the crash and node exporter containers were started from the cluster when the node was added to the cluster( no issues or manual involvement ). 0376a72700 0 log_channel(cephadm) log [INF] : Dete

[ceph-users] Re: Cephadm - Adding host to migrated cluster

Does the cephadm.log on that node reveal anything useful? What about the (active) mgr log? Zitat von Brent Kennedy : Greetings everyone, We recently moved a ceph-ansible cluster running pacific on centos 8 to centos 8 stream and then upgraded to quincy using cephadm after converting to cep

[ceph-users] Cephadm - Adding host to migrated cluster

Greetings everyone, We recently moved a ceph-ansible cluster running pacific on centos 8 to centos 8 stream and then upgraded to quincy using cephadm after converting to cephadm. Everything with the transition worked but recently we decided to add another node to the cluster with 10 more drive

[ceph-users] Re: rgw with unix socket

Hi Rok, It was I think planned years ago to drop the fcgi front-end. I'm unsure if Pacific does have it. Matt On Mon, Oct 17, 2022 at 11:31 AM Rok Jaklič wrote: > Hi, > > I try to configure ceph with rgw and unix socket (based on > https://docs.ceph.com/en/pacific/man/8/radosgw/?highlight=ra

[ceph-users] rgw with unix socket

Hi, I try to configure ceph with rgw and unix socket (based on https://docs.ceph.com/en/pacific/man/8/radosgw/?highlight=radosgw). I have in ceph.conf something like this: [client.radosgw.ctplmon3] host = ctplmon3 rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock log file = /var/lo

[ceph-users] Re: disable stretch_mode possible?

On Mon, Oct 17, 2022 at 4:40 AM Enrico Bocchi wrote: > > Hi, > > I have played with stretch clusters a bit but never managed to > un-stretch them fully. > > From my experience (using Pacific 16.2.9), once the stretch mode is > enabled, the replicated pools switch to the stretch_rule with size 4,

[ceph-users] Re: Rgw compression any experience?

On Mon, Oct 17, 2022 at 6:12 AM Szabo, Istvan (Agoda) wrote: > > Hi, > > I’m looking in ceph octopus in my existing cluster to have object compression. > Any feedback/experience appreciated. > Also I’m curious is it possible to set after cluster setup or need to setup > at the beginning? it's fi

[ceph-users] Re: Updating Git Submodules -- a documentation question

Here is an example of dealing with untracked files, which Brad discusses at the end of the most recent email in this thread: [zdover@fedora src]$ git status > On branch main > Your branch is up to date with 'origin/main'. > > Untracked files: > (use "git add ..." to include in what will be commi

[ceph-users] Re: Cluster crashing when stopping some host

If this is reproducable I'd recommend to create a tracker issue: https://tracker.ceph.com/ Zitat von Murilo Morais : Eugen, it worked and it didn't. I had to bootstrap in v17.2.3, using v17.2.4 this behavior is occurring. I did numerous tests with 3 VMs, two with disks and another only for M

[ceph-users] Re: disable stretch_mode possible?

Hi, I have played with stretch clusters a bit but never managed to un-stretch them fully. From my experience (using Pacific 16.2.9), once the stretch mode is enabled, the replicated pools switch to the stretch_rule with size 4, min_size 2, and require at least one replica in each datacenter.

[ceph-users] Re: disable stretch_mode possible?

Hi, I didn't have the time to play with it yet, but couldn't you just assign a different ruleset to the pool(s)? Or does ceph complain and prevent that? I'm not sure if stretch mode will be still active after changing the crush rule. But I'm not aware that there's a command to revert this

[ceph-users] Re: monitoring drives

I see a few (a priori) potential issues with this: - Given "disks" is THE key scaling dimension in a Ceph cluster, depending on how many metrics per device this exporter generates, it could negatively impact Prometheus performance (we already experienced such an issue when we explored

[ceph-users] Rgw compression any experience?

Hi, I’m looking in ceph octopus in my existing cluster to have object compression. Any feedback/experience appreciated. Also I’m curious is it possible to set after cluster setup or need to setup at the beginning? Thank you This message is confidential and is fo

[ceph-users] Re: 1 OSD laggy: log_latency_fn slow; heartbeat_map is_healthy had timed out after 15

Hi, In fact, a very stupid mistake. This is a CentOS 8 system where smartd was not installed. After installing and starting it, the OSD device is indeed in bad shape with many reported errors, explaining the behaviour observed. We managed to drain gracefully the sick OSD using the approach p

[ceph-users] Re: Slow OSD startup and slow ops

Hello, On Fri, Sep 30, 2022 at 8:12 AM Gauvain Pocentek wrote: > Hi Stefan, > > Thanks for your feedback! > > > On Thu, Sep 29, 2022 at 10:28 AM Stefan Kooman wrote: > >> On 9/26/22 18:04, Gauvain Pocentek wrote: >> >> > >> > >> > We are running a Ceph Octopus (15.2.16) cluster with similar

[ceph-users] Re: CephFS constant high write I/O to the metadata pool

Hi Patrick, With "objecter_ops" did you mean "ceph tell mds.pve-core-1 ops" and/or "ceph tell mds.pve-core-1 objecter_requests"? Both these show very few requests/ops - many times just returning empty lists. I'm pretty sure that this I/O isn't generated by any clients - I've earlier tried to isola

[ceph-users] Re: Spam on /var/log/messages due to config leftover?

I solved the problem by stopping prometheus on the problematic host, then removing the folder containing the prometheus config and storage (/var/lib/ceph//prometheus.) and then restarting the alertmanager and node-exporter units. Nicola ___ ceph-user