[ceph-users] Re: ceph-volume lvm zap destroyes up+in OSD

Hi, I can confirm the behavior for Pacific version 16.2.7. I checked with a Nautilus test cluster and there it seems to work as expected. I tried to zap a db device and then restarted one of the OSDs, successfully. So there seems to be a regression somewhere. I didn't search for tracker i

[ceph-users] Re: MDS internal op exportdir despite ephemeral pinning

Hi Patrick and everybody, I wrote a small script that pins the immediate children of 3 sub-dirs on our file system in a round-robin way to our 8 active ranks. I think the experience is worth reporting here. In any case, Patrick, if you can help me get distributed ephemeral pinning to work, this

[ceph-users] Re: *****SPAM***** Re: CephFS performance

> > That said, if you've been happy using CephFS with hard drives and > gigabit ethernet, it will be much faster if you store the metadata on > SSD and can increase the size of the MDS cache in memory Is using multiple adapters already being supported? That seems wanted using 1Gbit. https://www

[ceph-users] Re: ceph-volume lvm zap destroyes up+in OSD

Hi Eugen, can you confirm that the silent corruption happens also on a collocated OSDc (everything on the same device) on pacific? The zap command should simply exit with "osd not down+out" or at least not do anything. If this accidentally destructive behaviour is still present, I think it is

[ceph-users] radosgw-admin bucket check --fix returns a lot of errors (unable to find head object data)

Hi, we have a customer that got some _multipart_ files in his bucket, but the bucket got no unfinished multipart objects. So I tried to remove them via $ radosgw-admin object rm --bucket BUCKET --object=_multipart_OBJECT.qjqyT8bXiWW5jdbxpVqHxXnLWOG3koUi.1 ERROR: object remove returned: (2) No such

[ceph-users] filesystem became read only after Quincy upgrade

Hi, We upgraded this morning a Pacific Ceph cluster to the last Quincy version. The cluster was healthy before the upgrade, everything was done according to the upgrade procedure (non-cephadm) [1], all services have restarted correctly but the filesystem switched to read only mode when it beca

[ceph-users] RGW Forcing buckets to be encrypted (SSE-S3) by default (via a global bucket encryption policy)?

Hey ceph-users, loosely related to my question about client-side encryption in the Cloud Sync module (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/I366AIAGWGXG3YQZXP6GDQT4ZX2Y6BXM/) I am wondering if there are other options to ensure data is encrypted at rest and also only

[ceph-users] Re: RGW Forcing buckets to be encrypted (SSE-S3) by default (via a global bucket encryption policy)?

On 23/11/2022 13:36, Christian Rohmann wrote: I am wondering if there are other options to ensure data is encrypted at rest and also only replicated as encrypted data ... I should have referenced thread https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/TNA3MK2C744BN5OJQ4FMLWDK7

[ceph-users] Re: ceph-volume lvm zap destroyes up+in OSD

I can confirm the same behavior for all-in-one OSDs, it starts to wipe then aborts, but the OSD can't be restarted. I'll create a tracker issue, maybe not today though. Zitat von Frank Schilder : Hi Eugen, can you confirm that the silent corruption happens also on a collocated OSDc (ever

[ceph-users] Requesting recommendations for Ceph multi-cluster management

I'm looking for guidance/recommendations on how to approach below topic. As I'm fairly new to Ceph as a whole, I might be using or looking for terms/solutions incorrectly or simply missing some obvious puzzle pieces. Please do not assume advanced Ceph knowledge on my part (-: We are looking at

[ceph-users] Issues during Nautilus Pacific upgrade

Hi, We would like to share our experience upgrading one of our clusters from Nautilus (14.2.22-1bionic) to Pacific (16.2.10-1bionic) a few weeks ago. To start with, we had to convert our monitors databases to rockdb in order to continue with the upgrade. Also, we had to migrate all our OSDs to

[ceph-users] Multi site alternative

Hi, Due to the lack of documentation and issues with multisite bucket sync I’m looking for an alternative solution where I can put some sla around the sync like I can guarantee that the file will be available in x minutes. Which solution you guys are using which works fine with huge amount of o

[ceph-users] Re: filesystem became read only after Quincy upgrade

This bug looks very similar to this issue opened last year and closed without any solution : https://tracker.ceph.com/issues/52260 Adrien Le 23/11/2022 à 12:49, Adrien Georget a écrit : Hi, We upgraded this morning a Pacific Ceph cluster to the last Quincy version. The cluster was healthy be

[ceph-users] Re: hw failure, osd lost, stale+active+clean, pool size 1, recreate lost pgs?

Hi Jelle, did you try: ceph osd force-create-pg https://docs.ceph.com/en/quincy/rados/troubleshooting/troubleshooting-pg/#pool-size-1 Regards, Joachim ___ Clyso GmbH - Ceph Foundation Member Am 22.11.22 um 11:33 schrieb Jelle de Jong: Hello everybody, S

[ceph-users] Re: Multi site alternative

Hey Ivan, I think the answer would be multisite. I know there is a lot of effort currently to work out the last few kinks. This tracker might be of interest as it sounds like an already identified issue, https://tracker.ceph.com/issues/57562#change-228263 Matt From: istvan.sz...@agoda.com At:

[ceph-users] Re: CephFS performance

Hi David, First of all, thanks for your reply! The resiliency of BeeGFS is in doing RAID on disks (by hardware or software) at the same node as the storage. If there is a need for greater resilience, the maximum possible is through buddy (which would be another storage machine as a fail

[ceph-users] Re: CephFS performance

Hi Gregory, Thanks for your reply! We are evaluating possibilities to increase storage performance. I understand that Ceph has has better capability in data resiliency. This has been one of the arguments I use to keep this tool in our storage. I say this mainly in failure event

[ceph-users] failure resharding radosgw bucket

Hi list, I am completely lost trying to reshard a radosgw bucket which fails with the error: process_single_logshard: Error during resharding bucket 68ddc61c613a4e3096ca8c349ee37f56/snapshotnfs:(2) No such file or directory But let me start from the beginning. We are running a ceph cluster versio

[ceph-users] Re: *****SPAM***** Re: CephFS performance

> crashes). In the case of BeeGFS, if there is a problem on any machine, the > whole cluster becomes inconsistent (at the point of my tests, I'm not working > with that). > But the first question you should ask yourself is, can you afford to be having these down hours, or do you want to have a

[ceph-users] Re: failure resharding radosgw bucket

hi Jan, On Wed, Nov 23, 2022 at 12:45 PM Jan Horstmann wrote: > > Hi list, > I am completely lost trying to reshard a radosgw bucket which fails > with the error: > > process_single_logshard: Error during resharding bucket > 68ddc61c613a4e3096ca8c349ee37f56/snapshotnfs:(2) No such file or > direc

[ceph-users] Ceph Leadership Team Meeting 11-23-2022

Hi Cephers, Short meeting today: - The Sepia Lab is gradually coming back to life! Dan Mick & others managed to restore the testing environment (some issues could still remain, please ping Dan if you experience any). - Thanks to that, the release process for the Pacific 16.2.11 versio

[ceph-users] Re: Issues during Nautilus Pacific upgrade

> We would like to share our experience upgrading one of our clusters from > Nautilus (14.2.22-1bionic) to Pacific (16.2.10-1bionic) a few weeks ago. > To start with, we had to convert our monitors databases to rockdb in Weirdly I have just one monitor db in leveldb still. Is still recommend re

[ceph-users] Re: CephFS performance

Have you tested having the block.db and WAL for each OSD on a faster SSD/NVME device/ partition? I have a bit smaller environment, but was able to take a 2 Tb SSD, split it into 4 partitions and use it for the db and WAL for the 4 Drives. By Default if you move the block.db to a different

[ceph-users] Re: radosgw-admin bucket check --fix returns a lot of errors (unable to find head object data)

Hi, I was able to clean up the objects by hand. I leave my breadcrumbs here in case someone finds it useful. 1. Get all rados objects via `radosgw-admin bucket radoslist --bucket $BUCKET` and filter the ones the you need to remove 2. Remove the rados objects via `rados -p $RGWDATAPOOL rm $RADOSOBJ

[ceph-users] Re: filesystem became read only after Quincy upgrade

Hi Adrien, On 23/11/2022 19:49, Adrien Georget wrote: Hi, We upgraded this morning a Pacific Ceph cluster to the last Quincy version. The cluster was healthy before the upgrade, everything was done according to the upgrade procedure (non-cephadm) [1], all services have restarted correctly bu

[ceph-users] Re: filesystem became read only after Quincy upgrade

On 23/11/2022 19:49, Adrien Georget wrote: Hi, We upgraded this morning a Pacific Ceph cluster to the last Quincy version. The cluster was healthy before the upgrade, everything was done according to the upgrade procedure (non-cephadm) [1], all services have restarted correctly but the files