[ceph-users] Re: Multipath and cephadm
I did not really look deep, but by the last log it seems there's some utf
chars somewhere (Greek phi?) And the code is not handling it well when
logging, trying to use ASCII.
On Thu, 23 Dec 2021, 19:02 Michal Strnad, wrote:
> Hi all.
>
> We have problem using disks accessible via multipath. We are using
> cephadm for deployment, Pacific version for containers, CentOS 8 Stream
> on servers and following LVM configuration.
>
> devices {
> multipath_component_detection = 1
> }
>
>
>
> We tried several methods.
>
> 1.) Direct approach.
>
> cephadm shell ceph orch daemon add osd serverX:/dev/mapper/mpatha
>
> Errors are attached in 1.output file.
>
>
>
> 2. With the help of OSD specifications where they are mpathX devices used.
>
> service_type: osd
> service_id: osd-spec-serverX
> placement:
>host_pattern: 'serverX'
> spec:
>data_devices:
> paths:
>- /dev/mapper/mpathaj
>- /dev/mapper/mpathan
>- /dev/mapper/mpatham
>db_devices:
> paths:
>- /dev/sdc
> encrypted: true
>
> Errors are attached in 2.output file.
>
>
> 2. With the help of OSD specifications where they are dm-X devices used.
>
> service_type: osd
> service_id: osd-spec-serverX
> placement:
>host_pattern: 'serverX'
> spec:
>data_devices:
> paths:
>- /dev/dm-1
>- /dev/dm-2
>- /dev/dm-3
>- /dev/dm-X
>db_devices:
> size: ':2TB'
> encrypted: true
>
> Errors are attached in 3.output file.
>
> What is the right method for multipath deployments? I didn't find much
> on this topic.
>
> Thank you
>
> Michal
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: 3 OSDs can not be started after a server reboot - rocksdb Corruption
Hey Sebastian, On 12/22/2021 1:53 AM, Sebastian Mazza wrote: 9) Would you be able to run some long lasting (and potentially data corrupting) experiments at this cluster in an attempt to pin point the issue. I'm thinking about periodic OSD shutdown under the load to catch the corrupting event. With a raised debug level for that specific OSD. The major problem with this bug debugging is that we can see its consequences - but we have no clue about what was happening when actual corruption happened. Hence we need to reproduce that somehow. So please let me know if we can use your cluster/help for that... I want to help. Destroying the data on the cluster is not a problem. The question is, if I can find enough time, but I will do what I can. So, you are welcome to give me detailed instructions what I should test. One thing that could be important: I don’t think there was a significant load on the OSDs when this problem happened. So we want to repoduce the same issue with a more verbose logging. Hence my suggestion for the first attempt would be to try to restart the cluster in the same manner you did it before. With some preceeding steps: 1) Bring cluster back to the healthy state be redeplying broken OSDs. 2) inject verbose bluefs/bdev logging shortly before the restart (do not leave the cluster with these debug levels for a long time as it might consume tons of disk space for logging) ceph tell osd.* injectargs "--debug-bluefs 20 --debug-bdev 20" 3) bring some load to the cluster to force disk writing. 4) restart the cluster and check OSD status on completion. If someone is/are broken - save the relevant logs May be try the above multiple times if every OSD is fine on reboot. Thanks in advance, -- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Mounting cephfs on OSD hosts still a problem
Hi, we are planning a new cluster as backend for our new bareos backup setup. It currently supports ceph either via S3 or via filesystem (the rados plugin is deprecated in the current release). Since I would like to avoid the extra S3 hop (storage daemon <-> ceph rgw) our POC will use cephfs. To further isolate the cluster from all other system, I would also like to run the bareos storage daemon colocated to the CEPH OSDs. This will require mounting the cephfs instance on the hosts. In the past this was considered a problem since deadlocks may occur. Is this still the case? Or is mounting cephfs on the OSD hosts (kernel implementation) considered safe now? Best regards and happy holidays, Burkhard Linke ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Mounting cephfs on OSD hosts still a problem
If I remember correctly, mounting cephfs on osd hosts was never an issue. Mapping RBD images was where the issue would come up. I've got a few single node ceph clusters that I use for test and they have a cephfs mount to themselves that has never caused an issue (beyond systemd not knowing cephfs is a network filesystem thanks Redhat) -paul On 12/24/21 07:52, Burkhard Linke wrote: Hi, we are planning a new cluster as backend for our new bareos backup setup. It currently supports ceph either via S3 or via filesystem (the rados plugin is deprecated in the current release). Since I would like to avoid the extra S3 hop (storage daemon <-> ceph rgw) our POC will use cephfs. To further isolate the cluster from all other system, I would also like to run the bareos storage daemon colocated to the CEPH OSDs. This will require mounting the cephfs instance on the hosts. In the past this was considered a problem since deadlocks may occur. Is this still the case? Or is mounting cephfs on the OSD hosts (kernel implementation) considered safe now? Best regards and happy holidays, Burkhard Linke ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Mounting cephfs on OSD hosts still a problem
> > If I remember correctly, mounting cephfs on osd hosts was never an > issue. Mapping RBD images was where the issue would come up. I had exactly the opposite experience. I had to go back to ceph-fuse to mount a cephfs on osd nodes. Kernel mount cephfs was working fine up until some luminous release. I never had any issues with using rbd images on osd nodes. I think a lot of people are running some sort of hyper converged solution. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
