[ceph-users] Re: 16.2.11 branch

Hey Laura, Greg, all, On 31/10/2022 17:15, Gregory Farnum wrote: If you don't mind me asking Laura, have those issues regarding the testing lab been resolved yet? There are currently a lot of folks working to fix the testing lab issues. Essentially, disk corruption affected our ability to rea

[ceph-users] Re: ceph-volume inventory reports available devices as unavailable

On 12/14/22 15:18, Eugen Block wrote: Hi, I haven't been dealing with ceph-volume too much lately, but I remember seeing that when I have multiple DB devices on SSD and wanted to replace only one failed drive. Although ceph-volume inventory reported the disk as unavailable the actual create c

[ceph-users] Re: 16.2.11 branch

On 15/12/2022 10:31, Christian Rohmann wrote: May I kindly ask for an update on how things are progressing? Mostly I am interested on the (persisting) implications for testing new point releases (e.g. 16.2.11) with more and more bugfixes in them. I guess I just have not looked on the right M

[ceph-users] Cephadm recreating osd with multiple block devices

Hallo folks, i am encountering a weird behavior from Ceph when i try to remove an OSD to replace it with an encrypted one. Where the OSD is being directly recreated with an additional block device after removal. So, the idea is to remove an OSD which was created without encryption enabled and

[ceph-users] Re: SLOW_OPS

Eugen, thanks for answering. I understand that there is not enough memory, but I managed to recover a lot of the memory that was in use. Right now I can't upgrade to 48, but it's already planned. After yesterday's episode I managed to recover a lot of the memory that was in use. Until then every

[ceph-users] Re: mds stuck in standby, not one active

Hi, while upgrading to ceph pacific 6.2.7, the upgrade process stuck exactly at the mds daemons. Before, I have tried to increase/shrink the placement size of them, but nothing happens. Currently I have 4/3 running daemons. One daemon should be stopped and removed. Do you suggest to force re

[ceph-users] Re: cephfs snap-mirror stalled

Hi Holger, (sorry for the late reply) On Fri, Dec 9, 2022 at 6:22 PM Holger Naundorf wrote: > > As an update: > After the third restart now the mirror-daemon is running normal again - > only change to the restarts before was that during the restart > dbug_client was set to 20. (First restart was

[ceph-users] not all pgs not evicted after reweight

Hallo folks, i want to replace an OSD, so i reweight it to 0 `ceph osd reweight osd. 0`. The OSD hat 24 PGs and the number went down to 7, but stuck there. `ceph osd tree` shows: 22    hdd  0 0  0 B  0 B  0 B  0 B  0 B  0 B 0 0    7  up

[ceph-users] Re: Cephadm recreating osd with multiple block devices

I think the issue has been described in this note in the documentation. On 15.12.22 11:47, Ali Akil wrote: Hallo folks, i am encountering a weird behavior from Ceph when i try to remove an OSD to replace it with an encrypt

[ceph-users] Re: User + Dev Monthly Meeting happening tomorrow, December 15th!

Hi everyone, Since there are no topics on the agenda for today, the User + Dev Monthly Meeting is cancelled. We will reconvene next year! - Laura Flores On Wed, Dec 14, 2022 at 11:52 AM Laura Flores wrote: > Hi Ceph Users, > > The User + Dev Monthly Meeting is coming up tomorrow, *Thursday, De

[ceph-users] Re: MDS crashes to damaged metadata

Hi Patrick, we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05

[ceph-users] Removing OSD very slow (objects misplaced)

Hi, when removing some OSD with the command `ceph orch osd rm X`, the rebalancing starts very fast, but after a while it almost stalls with a very low recovering rate: Dec 15 18:47:17 … : cluster [DBG] pgmap v125312: 3361 pgs: 13 active+clean+scrubbing+deep, 4 active+remapped+backfilling, 3344 ac

[ceph-users] Re: mds stuck in standby, not one active

On Thu, Dec 15, 2022 at 7:24 AM Mevludin Blazevic wrote: > > Hi, > > while upgrading to ceph pacific 6.2.7, the upgrade process stuck exactly > at the mds daemons. Before, I have tried to increase/shrink the > placement size of them, but nothing happens. Currently I have 4/3 > running daemons. One

[ceph-users] Re: rgw: "failed to read header: bad method" after PutObject failed with 404 (NoSuchBucket)

Hi, I noticed that the error behavior is actually depending on the length of the PutObject request. The subsequent request * results in a 400 Bad Request error (with log message "failed to read header: bad method") for small files or * a read timeout on the connection is encountered by the clie

[ceph-users] Re: mds stuck in standby, not one active

Ceph fs dump: e62 enable_multiple, ever_enabled_multiple: 1,1 default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file

[ceph-users] Re: mds stuck in standby, not one active

On Thu, Dec 15, 2022 at 3:17 PM Mevludin Blazevic wrote: > > Ceph fs dump: > > e62 > enable_multiple, ever_enabled_multiple: 1,1 > default compat: compat={},rocompat={},incompat={1=base v0.20,2=client > writeable ranges,3=default file layouts on dirs,4=dir inode in separate > object,5=mds uses ver

[ceph-users] Re: 16.2.11 pacific QE validation status

On Fri, Dec 16, 2022 at 3:15 AM Yuri Weinstein wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/58257#note-1 > Release Notes - TBD > > Seeking approvals for: > > rados - Neha (https://github.com/ceph/ceph/pull/49431 is still being > tested and will be mer

[ceph-users] Re: 16.2.11 pacific QE validation status

I reviewed the upgrade runs: https://pulpito.ceph.com/yuriw-2022-12-13_15:57:57-upgrade:nautilus-x-pacific_16.2.11_RC-distro-default-smithi/ https://pulpito.ceph.com/yuriw-2022-12-13_21:47:46-upgrade:nautilus-x-pacific_16.2.11_RC-distro-default-smithi/ https://pulpito.ceph.com/yuriw-2022-12-13_15:

[ceph-users] Re: ceph-iscsi lock ping pong

On 15/12/2022 02:46, Joe Comeau wrote: That's correct - we use the kernel target not tcmu-runner Okay. There are some difference for the configurations between kernel target and the ceph-iscsi target. Thanks, - Xiubo >>> Xiubo Li 12/13/2022 6:02 PM >>> On 14/12/2022 06:54, Joe Comea

[ceph-users] Re: ceph-iscsi lock ping pong

On 14/12/2022 16:32, Stolte, Felix wrote: We have been using tgt for five years and switched to ceph-iscsi (LIO Framework) two months ago. We observed a massive performance boost. Can’t say though if the performance increase was only related to the different software or if our TGT configuratio

[ceph-users] max pool size (amount of data/number of OSDs)

Hi, There are various articles, case studies, etc about large ceph clusters, storing 10s of PiB,with CERN being the largest cluster as far as I know. Is there a largest pool capacity limit?  In other words, while you may have a 30PiB cluster,is there a limit or recommendation as to max pool capa

[ceph-users] Re: ceph-iscsi lock ping pong

On 14/12/2022 14:52, Stolte, Felix wrote: Issue is resolved now. After verifying that all esx hosts are configured for MRU, i took a closer look on the paths on each host. `gwcli` reported lun in question was owned by gateway A, but one esx host used the path to gateway B for I/O. I reconfigu

[ceph-users] Re: cephfs snap-mirror stalled

On 15.12.22 14:06, Venky Shankar wrote: Hi Holger, (sorry for the late reply) On Fri, Dec 9, 2022 at 6:22 PM Holger Naundorf wrote: As an update: After the third restart now the mirror-daemon is running normal again - only change to the restarts before was that during the restart dbug_clie